ssis xml source Archives | ZappySys Blog

How to read large XML / JSON file in SSIS (3 Million Rows in 3 Mins)

ZappySys — Mon, 08 Jan 2018 16:55:09 +0000

Introduction

In this post we will learn how to use ZappySys SSIS XML Source or ZappySys SSIS JSON Source to read large XML or JSON File (Process 3 Million rows in 3 minutes – 1.2 GB file).

If you use default settings to read data then it may result into OutOfMemory Exception so we will outline few techniques which will enable high performance Streaming Mode rather than In-memory load of entire file.

Prerequisites

Before we parse very large XML or JSON files using SSIS , you will need to make sure following prerequisites are met.

SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
You have at least two sample files… One file must be small dataset (less than 10 MB if possible). We will use small dataset file during design mode to get metadata and see data preview on the Component UI.
Second XML file is the big file with full dataset you like to parse at runtime.
Make sure SSIS PowerPack is installed. Click here to download.

Step-By-Step : Reading large XML file (SSIS XML Source)

Now let’s look at how to read large XML file (e.g. 3 Million rows or more) using ZappySys XML Source in SSIS. Below steps are identical for ZappySys JSON Source too (Except Step#7).

Open SSIS Designer and drag Data Flow from SSIS Toolbox
Double click Data Flow Task to switch to Data flow designer
Now drag ZS XML Source on the surface from SSIS Toolbox.
Double click ZS XML Source and specify
```
small dataset file
```
path you like to parse. (e.g. c:\data\customer_small.xml )
Click on Select Filter button to find Node which will be treated as Array. Once you close the Filter Browse Dialog. Append
```
--FAST
```
Your Expression may look like below.
```
$.Root.Row--FAST
```
Now uncheck
```
Include Parent Columns option
```
Check
```
Enable Performance Option
```
(For JSON Source Skip this step)
NOTE: Try to enable Performance mode setting after you Select Filter (by using smaller dataset file). Once filter is set you can check enable performance mode and make sure following two settings are correctly set (Option#2 needs new version – SSIS v4.1+ / ODBC v1.4+).
(1) On Array Handling Tab – Set Array node name (must be only one entry). For example your Filter is $.DATA.ORDER[*] then you can enter
```
ORDER
```
(2) On Advanced Filter Options Tab – Enter all unwanted Tag names you like to Skip. For example if you have other arrays like PRODUCT or CUSTOMER then enter
```
PRODUCT,CUSTOMER
```
Your setting may look like below

Configure XML source or JSON Source for Very Large Data File (Streaming mode for High Performance)
Click Preview to verify data (Adjust Filter if needed to extract correct Hierarchy)
Preview XML or JSON File data using SSIS XML Source or JSON Source
Click on Columns Tab
Change Scan Row count to 3000 or more and Click on
```
Refresh Column
```
.
Select Guess 4x , Check Lock, Check Reset and Click OK like below. At runtime if you ever get error about Data Type Issue you can always adjust this later on too. Make sure Lock column is set to avoid setting reset for manually changed columns. For more information on metadata changes check this article
SSIS Metadata Options – JSON / XML File Parsing
Click OK to save settings.
Now Right click on XML Component > Click Properties. Change
```
DirectPath
```
property to original file path (large file) (e.g.
```
c:\data\customers_large.xml
```
). Save Package.
Now you can run your SSIS Package from Designer or Command line. As you see in the below screenshot that there is virtually no memory pressure when you enable stream mode. Thanks to ZappySys unique XML / JSON Parsing Engine. When streaming mode is enabled file is not loaded into memory for parsing rather than that it only reads record by record to process very large JSON or XML file.In our below example we used Windows 7 Desktop, 16GB RAM, 4 Core i7 64 bit CPU. It took around
```
3 Minutes to Parse 3 Million Records (1.2 GB big XML file)
```
. If you Parse JSON file then it can be even faster due to compact size.
Reading Very Large XML or JSON File using SSIS (Stream Mode for High Performance)

Step-By-Step : Reading very large JSON file (SSIS JSON Source)

Reading very large JSON file using ZappySys JSON Source has exact same steps described in above section except two changes. You have to use ZS JSON Source and skip Step#7 (Check Enable Performance Mode – This option is not available JSON Source).

Parsing very large XML File with Multiple Arrays

Now let’s discuss a scenario which can result in OutOfMemory Exception unless you tweak some extra options. Assume you have file structure like below.
NOTE: This will only work in version 4.1.0 or later (in SSIS PowerPack) or 1.4.0 or later for ODBC PowerPack



	....
	....
	.... many more....
	....
	
	....
	....
	.... many more....
	....
	
	....
	....
	.... many more....
	....

Notice that in above XML it has 3 different nodes (For Product, Customer and Order). If you try to extract Orders it might fail with OutOfMemory Exception because it needs to scan large XML before it can hit First Order node. To solve this issue you can adjust following 2 settings.

On Array Handling Tab you have following
```
Row
```
On Advanced Filter Options tab enter Following two nodes which we dont want to extract.
```
Product,Customer
```

Thats it. This will avoid excessive Memory pressure to find very first node before it can start Stream.

Parse XML Array – Performance Mode Setting

Parse Large XML – Exclude nodes by name

Conclusion

As you saw in this article that ZappySys SSIS PowerPack is designed to handle very large dataset in JSON or XML. We also support very large CSV and Excel files too which are not covered in this article. Download SSIS PowerPack to explore 70+ more components by yourself to make your ETL simple and Fast.

The post How to read large XML / JSON file in SSIS (3 Million Rows in 3 Mins) appeared first on ZappySys Blog.

How to call Google DFP API with SSIS – DoubleClick

ZappySys — Tue, 14 Feb 2017 19:28:31 +0000

Introduction

Google DoubleClick is one of the most popular platforms for Advertisers. Many times you have need for custom integration or Automation for many operations. In this article we will learn how to Call Google DFP API (i.e. DoubleClick for Publishers) without coding using SSIS (Microsoft SQL Server Integration Services). In our previous article we discussed how to Integrate Google AdWords API using SSIS (Click here).

If you are new to SSIS then no worry there are plenty of tutorials for ssis

To achieve Custom integration for DoubleClick (DFP) API we will use SSIS XML Source. XML Source is part of SSIS PowerPack which has 45+ connectors and Tasks.

Prerequisite

Before you can finish tasks explained in this article you have to finish below steps. This article assumes you have basic knowledge of SSIS (SQL Server Integration Services)

Make sure you SSIS designer installed. Sometimes its referred as BIDS or SSDT (Get from here)
Download SSIS PowerPack
Download SoapUI (Its free third party tool to test SOAP API). SoapUI can help you to generate XML Request BODY easily from WSDL file provided by API Vendor. You can also test your service by supplying parameters.
Optional – Another very useful Free tool is Fiddler. You can use it to see raw request/response (Check this article)

Download DFP API Example SSIS Package

Click here to DFP API download Sample for SSIS SSIS Package (SSIS 2012, 2014, 2016)

Screenshot of Sample Package:

Example SSIS Package – Google DFP API Integration (Read / Write data in Google Double Click for Publisher) – No Coding required

Making your first DFP API Call using SSIS

Once you install PowerPack you are ready to execute your very first Google DFP API Call.

NOTE: If you don’t have any DFP network for test then see next section (Explains how to create test network for testing).

Lets look at how to call Google DFP API step-by-step.

Create new SSIS Project and open package designer
Drag ZS REST API Task from SSIS control flow toolbox and drop it on designer.

REST API Task is useful when you want to call SOAP/REST API but not necessarily parse response into rows and columns. We will look at JSON Source later in this article which can actually parse response into rows and columns (Useful to load Google DFP data into SQL Server or other RDBMS / flatfile )
Double click REST API Task to configure. Select Access mode to [Url from Connection]

Enter following URL in the URL Textbox

 https://ads.google.com/apis/ads/publisher/v201702/NetworkService

From the connection dialogbox select ZS-OAUTH to create new OAuth Connection for DFP API
On OAuth Connection dialogbox select Google from Provider dropdown. In this demo we will use Default OAuth App but you can register your own google OAuth app if you wish to use Custom OAuth option from UI.

Enter following Scopes in the Scopes textbox (or select manually by clicking Select Scopes button). Scope is nothing but permission for API (In our case View/Manage DFP data and Read/Write Report Files to Cloud Storage).

https://www.googleapis.com/auth/dfp
https://www.googleapis.com/auth/devstorage.read_only
https://www.googleapis.com/auth/devstorage.write_only
https://www.googleapis.com/auth/cloud-platform.read-only
https://www.googleapis.com/auth/devstorage.full_control
https://www.googleapis.com/auth/cloud-platform
https://www.googleapis.com/auth/devstorage.read_write

If you are not going call ReportService API to generate Reports in CSV format then you will need only one scope
```
https://www.googleapis.com/auth/dfp
```
Click Generate Token button. You will see browser popup for login and then Accept option to confirm permissions like below. Once you don’t it will populate tokens and prompt to save tokens to backup file. Secure token backup to safe place.
SSIS OAuth Connection – Connect to Google DFP API (DoubleClick SOAP Web Service)
Once done Click Test Connection and if its green then Click OK to save connection
Once you are back to REST API Task UI then select Method to POST
Select Request Body Content Type to XML (i.e. text/xml)
Click edit button next to Request Body and enter following XML.
```
   
      
   
   
      
   
```
If you are wondering how did we get above XML fragment then read next section about using 3rd party tool called SoapUI. Also we have detail article for calling SOAP request. Check this article on how to use SoapUI (free 3rd party tool) to create SOAP request Bodyfrom WSDL. DFP has many API endpoints for different actions (e.g. NetworkService , InventoryService , …). For each service you have different Api URL and different WSDL. In the next section you can learn how to create correct SOAP Body (for POST) using SoapUI tool.

For example if you using DFP NetworkService API for version v201702 then your help page URL would be like this
https://developers.google.com/doubleclick-publishers/docs/reference/v201702/NetworkService
And on the same page you will see WSDL link below
https://adwords.google.com/api/adwords/mcm/v201609/ManagedCustomerService?wsdl
(Just download that WSDL XML and save to local disk then use with SoapUI to generate XML body to submit for any DFP API call.)
Once you done with above steps, your Task Configuration will look like below.
SSIS Rest API Task – Call Google DoubleClick API – Get all DFP Networks

Now click Test Request button. You will see Response popup with below content if its successful (Scroll at the bottom on response form and you will see XML data) .. see below

------------------------------------
Request
------------------------------------
POST https 1.1 ==> /apis/ads/publisher/v201702/NetworkService
Host: ads.google.com

>>>> HEADERS <<<<<
Authorization: Bearer ya29.Glz-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: text/xml;charset=UTF-8
User-Agent: ZappySysApp
Host: ads.google.com
Content-Length: 291
Expect: 100-continue
Accept-Encoding: gzip, deflate
Connection: Keep-Alive

------------------------------------
Response
------------------------------------
POST 1.1 200 OK==> OK

>>>> HEADERS <<<<<
Content-Encoding: 
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Alt-Svc: quic=":443"; ma=2592000; v="35,34"
Transfer-Encoding: chunked
Cache-Control: private, max-age=0
Content-Type: text/xml; charset=UTF-8
Date: Sun, 26 Feb 2017 01:18:32 GMT
Expires: Sun, 26 Feb 2017 01:18:32 GMT
Server: GSE

>>>> Cookies <<<<<

>>>> CONTENT <<<<<


	
		
			27ab3d2e4e1fcb8f444b317d0b0f871d
			50
		
	
	
		
			
				551196
				XFP sandbox property
				238897396
				ca-pub-2799179143725683
				America/New_York
				USD
				237897516
				true

How to generate SOAP Request Body using SoapUI tool

This section describes how to create DoubleClick SOAP API Request Body (XML fragment) for any DFP SOAP API call described anywhere in this article.

Download SoapUI (Its free third party tool to test SOAP API). SoapUI can help you to generate XML Request BODY easily from WSDL file provided by API Vendor. You can also test your service by supplying parameters.
Now download WDSL xml for appropriate API calls you want to make. Assume that you want to call getAllNetworksResponse API found under NetworkService. For that first navigate to help file page and open help page. Make sure you select correct version from help navigation. Copy WSDL link and open in new browser window. Save XML to local disk (e.g. c:\api\dfp_networkservice_wsdl.xml (See below screenshot how to find DFP API WSDL link)
How to get Google DFP API WSDL URL (Used to generate POST Body)
Once SoapUI is downloaded and installed click File menu > Create New SOAP Project option
Name your project (e.g. DFP API) and specify WSDL URL or File Path if it was saved locally (e.g. c:\api\dfp_networkservice_wsdl.xml) and click OK
Now navigate to API Action for which you would like to get Body. Click Request node (If missing create new) and edit request. You may see XML like below (If some optional parameters not visible in XML then click Re-create request with optional parameters button from toolbar)
Call Google DFP API – Generate SOAP Request Body XML using SoapUI (Google DoubleClick Web Service)
Edit necessary parameters from above XML code and copy to Body of SSIS REST API Task.

Creating test network – DFP Sandbox

Very first step we recommend before testing DFP API call is create test network. You can call DFP API against LIVE network but not recommended if you are calling create/update API.

To create test network you can follow exact same steps described in the previous section except the Body step (Step#12). Use following Request body to call makeTestNetwork command.

API makeTestNetwork – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/NetworkService
Body (see below):

Once you click Test you may receive following response. Note your network code (e.g. 1122334455 from below). This network code is used in pretty much all DFP API to view or manage ad network related items or properties.

API makeTestNetwork – Response


	
		
			ba61d79efee325bf4b5aa45c46836021
			1021
		
	
	
		
			
				551196
				XFP sandbox property
				1122334455
				ca-pub-2799179143725683
				America/Los_Angeles
				USD
				237897516
				true

Above response contains test networkcode copy that code becuase you will need in almost every API call you make later. You can only have one test network. If test network is already created and you try to execute above code then you may receive error AuthenticationError.GOOGLE_ACCOUNT_ALREADY_ASSOCIATED_WITH_NETWORK

You can also get network code using two different ways.

Visit your DFP console homepage by visiting https://www.google.com/dfp/ and you will see network name and code in the top potion. Also its listed in the URL (e.g. https://www.google.com/dfp/1234567)
Another way to get available networks for your login is to visit DFP API Play ground here https://dfp-playground.appspot.com/ and you will see network list in the dropdown ( name and network code)

How to Create new Ad Units

Now lets look at an example which will create few Ad Units by calling createAdUnits API. If you are creating Ad Unit at the root level then you will need to specify correct Parent ID (i.e effectiveRootAdUnitId). To obtain that Parent ID you may have to call getCurrentNetwork API. In the response you will see effectiveRootAdUnitId.

There is another way to know effectiveRootAdUnitId or Id of any Ad Units which can be Parent for new AdUnit. Goto google.com/dfp > Click Inventory Tab > Click Ad Units Side menu > Click Download ad units hyper link. You can also download AdUnits as CSV file from DFP Portal and look for Ad Units which are created at the root. Parent ID column for Root level Ad Units is basically called effectiveRootAdUnitId.

Now lets look at how to get EffectiveRootID by calling getCurrentNetwork.

API getCurrentNetwork – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/NetworkService
Body (see below):




	
		1122334455
		ZappySysApp

API getCurrentNetwork – Response


	
		
			81d1b00e8fbfb862129eb2dd485cf7bb
			4507
		
	
	
		
			
				551196
				XFP sandbox property
				1122334455
				ca-pub-2799179143725683
				America/New_York
				USD
				237897516
				true

If you want to save single Element Value from Response into SSIS Variable then perform following steps (example of how to get just effectiveRootAdUnitId from above response)

On REST API Task > Go to Response Tab > Select Response Content Type = XML
Enter following expression in the XPath filter
```
//*[local-name() = 'effectiveRootAdUnitId']
```
Check Save response content option
Select Variable from Dropdown – Click New Variable > Name it RootAdUnitId.
Click Test Request Button to test…. In the content textbox you will now see only Numeric value extracted by Filter

SSIS REST Api Task – How to extract single vale from response and save to SSIS variable

Once you know Parent ID for your AdUnit now lets look at how to create Ad Units by calling createAdUnits

API createAdUnits – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/InventoryService
Body (see below):



	
		
			1122334455
			ZappySysApp
		
	
	
		
			
				237897516
				Ad_Unit_0
				Ad unit description #0.
				BLANK
				
					
						300
						250
					
					BROWSER
				
			
			
				237897516
				Ad_Unit_1
				Ad unit description #1.
				BLANK
				
					
						300
						250
					
					BROWSER

API createAdUnits – Response


	
		
			b244d7ba9b65e96fc42b849a7bfb16e4
			331
		
	
	
		
			
				252312396
				237897516
				false
				
					237897516
					ca-pub-2799179143725683
					ca-pub-2799179143725683
				
				Ad_Unit_0
				Ad unit description#0.
				BLANK
				ACTIVE
				253312396
				
					
						300
						250
						false
					
					BROWSER
					300x250
				
				SITE
				false
				
					
						true
						FFFFFF
						0000FF
						FFFFFF
						000000
						008000
						TEXT_AND_IMAGE
						DEFAULT
						DEFAULT
						DEFAULT
					
				
				
					
						2017
						2
						27
					
					7
					23
					32
					PST8PDT
				
				NONE
				false
				false
			
			
				252312516
				237897516
				false
				
					237897516
					ca-pub-2799179143725683
					ca-pub-2799179143725683
				
				Ad_Unit_1
				Ad unit description#2.
				BLANK
				ACTIVE
				253312516
				
					
						300
						250
						false
					
					BROWSER
					300x250
				
				SITE
				false
				
					
						true
						FFFFFF
						0000FF
						FFFFFF
						000000
						008000
						TEXT_AND_IMAGE
						DEFAULT
						DEFAULT
						DEFAULT
					
				
				
					
						2017
						2
						27
					
					7
					23
					32
					PST8PDT
				
				NONE
				false
				false
			
				252312876
				237897516
				false
				
					237897516
					ca-pub-2799179143725683
					ca-pub-2799179143725683
				
				Ad_Unit_4
				Ad unit description.
				BLANK
				ACTIVE
				253312876
				
					
						300
						250
						false
					
					BROWSER
					300x250
				
				SITE
				false
				
					
						true
						FFFFFF
						0000FF
						FFFFFF
						000000
						008000
						TEXT_AND_IMAGE
						DEFAULT
						DEFAULT
						DEFAULT
					
				
				
					
						2017
						2
						27
					
					7
					23
					32
					PST8PDT
				
				NONE
				false
				false

Loading Google DoubleClick data into SQL Server Table

So far we saw how to make simple DFP API calls without doing any parsing. Now lets look at how to use SSIS XML Source which not only make API calls but it will parse XML Response into rows and columns which can be loaded into target database such as SQL Server, Oracle, MySQL or even Flat File. XML Source also supports pagination so if you have many records it will automatically loop through all response untill all records are fetched.

Step-By-Step – Using XML Source to read Google DFP data (Parse into rows and columns)

Assuming you have tested your first DFP API Call (Explianed in the beginning of this article). You must have OAuth connection tested for API call.
Drag new Data flow task from SSIS Toolbox
Inside Data flow designer drag and drop ZS XML Source from toolbox
Double click XML Source to configure.
Set URL as below
https://ads.google.com/apis/ads/publisher/v201702/InventoryService
Check Use credentials option and from Drop down select OAuth connection manager (created in previous section)
Select Method = POST,
Select Request Content Type = XML (text/xml)

Enter Body as below (Click edit button).


   
      
         238897396
         ZappySysApp
      
   
   
      
         
            WHERE Status='ACTIVE'

Notice that in above we calling getAdUnitsByStatement API to fetch all Ad Units which are Active. Here is more information about Query Syntax for any API call getXXXXXXByStaement

Now click on Select Filter button and select results node and click OK. If prompted add array to list.
SSIS XML Source – Get data from Google DFP API using OAuth (Configure URL, Body and Filter)
Click Preview to see data. Click Columns tab to review data types (Auto detected). If you wish to change length or datatype then edit there and check Lock option (Last column)
SSIS XML Source – Preview Response for Google DoubleClick API Call – Parse into Rows/Columns
Click OK to save UI
Connect XML Source to same target (e.g. OLEDB Destination – SQL Server Connection)
Run data flow
SSIS Execution – Read from Google DFP and load into SQL Server Table

Configure Pagination for Google DFP API / PQL Query result

If you have large dataset to read from DoubleClick API ( when calling API such as getxxxxxxByStatement) then we recommend to supply LIMIT and OFFSET clause in your PQL query.

For example if you have 2000 ad units and you want to limit response size by maximum 300 rows then your query can be like below

--For first request (returns results 1-300)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 0

--For second request (returns results 301-600)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 300

--For third request  (returns results 601-900)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 600

........

This looping logic can be complex to implement if you do manually. But no worry if you are using SSIS XML Source. It comes with many pagination options (Click here to read more about pagination)

See below screenshot how to configure Pagination options for Google DoubleClick API. Basically two places you have to change. Inside body you have to set placeholder and change few settings on Pagination Tab.

SSIS XML Source – Configure Pagination for Google DoubleClick API data fetch – PQL LIMIT and OFFSET clause for Google DFP API

SSIS XML Source – Configure Pagination for Google DoubleClick API data fetch – Set Pagination Mode

Create CSV Report File and Download in GZip format (*.gz)

Sometime you have to download large amount of data (Possibly millions of rows) in that case Bulk approach would be better. DFP ReportService API allows to call following APIs which can be used to produce CSV report file in *.gz format and then you can download it using REST API Task.

Below are high level steps you have to perform to produce report file and download it.

Create the ReportJob by invoking runReportJob command .
Poll the ReportJob object using ReportService.getReportJob (This is required because file is not available right away) .
Continue to poll the ReportJob object until the reportJobStatus field is equal to COMPLETED or FAILED.
If successful, fetch the URL for downloading the report by invoking getReportDownloadURL.

Here is the sample SSIS Package to Perform this Action.

Click here to DFP API download Sample for SSIS SSIS Package (SSIS 2012, 2014, 2016)

See below screenshot how to generate Google DFP Report File and download / extract (Unzip) using Drag and Drop SSIS workflow.

Create Google DFP Report File (CSV / Gzip) and Download Using SSIS (Example of ReportService functions runReportJob, getReportJob and .getReportDownloadURL)

Call runReportJob (Start DFP Report JOB)

Very first step to produce DFP Report file is call ReportService >> runReportJob API. See below Command.

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract id of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for 111111 node anywhere in SOAP response XML and extract value (i.e. 111111 ) .. For screenshot see next section

//*[local-name() = 'id']

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxxxxAKqG89gM-TpReRXO
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 1018
Expect: 100-continue
Accept-Encoding: gzip, deflate


	
		
			238897396
			ZappySysApp
		
	
	
		
			
				
					DATE
					LINE_ITEM_ID
					LINE_ITEM_NAME
					AD_UNIT_NAME
					AD_SERVER_IMPRESSIONS
					AD_SERVER_CLICKS
					LINE_ITEM_COST_TYPE
					LINE_ITEM_GOAL_QUANTITY
					LAST_MONTH

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:41 GMT
Expires: Fri, 02 Jun 2017 21:23:41 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 999


	
		
			f03e7e474fde2a5f9c5d7c9d7eb83923
			846
		
	
	
		
			
				10000297829
				
					DATE
					LINE_ITEM_ID
					LINE_ITEM_NAME
					AD_UNIT_NAME
					TOP_LEVEL
					AD_SERVER_IMPRESSIONS
					AD_SERVER_CLICKS
					LINE_ITEM_COST_TYPE
					LINE_ITEM_GOAL_QUANTITY
					LAST_MONTH
					false
					false

Call getReportJob (Poll JOB Status)

Second step is to make sure is Report Status check. For large report it make take several seconds or minutes before you can get Download URL (See Next Step). If you try to call Next step before Report is ready then you may get error. So better to add Polling logic.

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract status of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for IN_PROGRESS node anywhere in SOAP response XML and extract value (i.e. IN_PROGRESS )

//*[local-name() = 'rval']

Extract Single XML Node Value from SOAP API Response – Use XPATH in SSIS

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxXmWqmq
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 524
Expect: 100-continue
Accept-Encoding: gzip, deflate


   
      
         238897396
         ZappySysApp
      
   
   
      
       10000297829

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:47 GMT
Expires: Fri, 02 Jun 2017 21:23:47 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 451


	
		
			d52cb3a3d68af24aec2f9b0284379700
			803
		
	
	
		
			COMPLETED

Call getReportDownloadURL (Get Download URL)

Once you get status = COMPLETED in above API Call (i.e. getReportJobStatus) then you ready to call getReportDownloadURL to fetch URL which you can download.

Here is the API sample Request and Response

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxBp84ltTNyVRb9hlUEWx
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 581
Expect: 100-continue
Accept-Encoding: gzip, deflate


   
      
         238897396
         ZappySysApp
      
   
   
      
       10000297829
       CSV_DUMP

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:49 GMT
Expires: Fri, 02 Jun 2017 21:23:49 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 1039


	
		
			8e5ac0bbeb8e88c180ea442fc5378d99
			1957
		
	
	
		
			https://storage.googleapis.com/dfp-report-export/caxxxxxxxx-xxxxxxxxxxxxxxb?GoogleAccessId=3769xxxxxxxxxxx-qxxxxxxxxxxxxxxxxsqdb5dvi8g@developer.gserviceaccount.com&Expires=1496438929&Signature=vnhCiAIqd%2BFgWCtAHo9roFOI3W%2BveUTYuSvyBQ9gtMI2HDTXOCVDh7wzUV%2BPBJZZghaL6k8wKYq33GA4NpjZVNYuaAgPLWgGwAUPdtioHRi%2Bt3Eymp%2FAPRGIoos0ekP3O7l%2FPNPxenC6UTP087HV3j06V%2Fuui9ixpKZWGCK8r56oUPj6S8q1hG95Ns3d3sEcuz3%2BkShu6GYG5mqir5vjtZPuzwsr%2F96G5Kk123mke%2B%2FIVRJGoqnCMghNWtIG1yR1IdZx%2FIpit%2FnaJ9HP6IvAr3oMIXkQ3cHv%2FtOhXGy9dTG34q%2FKDbtkSJAayHgjlSLPGKucygcXBhUrbywBu8Y77A%3D%3D

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract status of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for https://storage.googleapis.com/dfp-report-export/caxx………. node anywhere in SOAP response XML and extract value (i.e. URL)

Download File Using REST API Task

Once you get URL its time to download it. You can use REST API Task to download the file. On Response Setting Tab specify full file path (e.g. c:\report_csv.gz) and make sure Binary option is checked (Found next to the Save Path)

Making things dynamic

If you want to supply certain parameters at runtime (e.g. from SSIS Variable) rather than hard code then you can use Variable Placeholders any where in Request Body, URL or Headers. Click Insert Variable option found in Edit screen

Here is an example of dynamic URL (API Version stored in SSIS Variable)

https://ads.google.com/apis/ads/publisher/{{User::ApiVer}}/NetworkService

Deployment to Production

When you ready to deploy your SSIS Package to production make sure to change Test NetworkCode supplied inside Request Body to your Production Ad Network. Also Set RefreshToken Property of OAuth Connection Manager (Get it from Backup file saved when you created OAuth connection first time). Once you set RefreshToken of connection manager you won’t have to Login again to get a new token.

Conclusion

Traditionally if you wanted to integrate Google DFP API (DoubleClick API) calls inside your ETL workflow then it required some sort of coding effort (e.g. Use C# SDK, JAVA SDK or Python SDK). Coding effort is not only time consuming and expensive but also hard to maintain for any non-coder (e.g. ETL Developer or DBA).
Thanks to SSIS PowerPack which changed the way developers do API integration. Now you can reduce time to implement and total cost of you API integration project significantly by using drag and drop approach. This approach is not only easy to maintain but also fast and high quality. You can use ZappySys REST/SOAP SSIS Connectors and SSIS Tasks for any API integration project such as Google DFP API. Try SSIS PowerPack for free to explore possibilities. If you have any issue with your integration Contact Support

The post How to call Google DFP API with SSIS – DoubleClick appeared first on ZappySys Blog.

Convert CURL to SSIS PowerPack / ODBC for API Call

ZappySys — Thu, 10 Nov 2016 19:35:43 +0000

Introduction

Often, REST API documentation uses example syntax for the curl command-line tool because it’s one of the most popular tools for making API calls. However, suppose you are an SSIS developer using ZappySys API Drivers/Connectors from either the SSIS PowerPack or the ODBC PowerPack. In that case, this blog post will help you translate the curl command line syntax to similar options available in the ZappySys Product. For SSIS PowerPack products, you can refer to Connectors/Tasks such as ZappySys REST API Task / JSON Source Connector / XML Source. To make this article short and simple, we will show many screenshots from SSIS PowerPack UI, but the concepts are the same when you use other products, such as ODBC PowerPack JSON Driver / REST or XML Driver.

If you are using Postman to call the API, then refer to this article

If you are using Fiddler to call an API, then refer to this article

CheatSheet – Curl to SSIS PowerPack / ODBC PowerPack

Before we dig down more. Let’s look at how each command-line parameter from CURL maps to SSIS PowerPack settings.

CURL Parameter (Short / Long)	Purpose	SSIS PowerPack Setting
method	Pass HTTP method (e.g., GET, POST, PUT, DELETE)	Select Method from Request Method Dropdown on ZappySys REST API Task or JSON Source Connector, or XML Source, like this
-u userid:password, –user userid:password	Authenticate using Basic Authentication using UserID and Password	Use the HTTP Connection Manager to enable Basic Auth, then supply the username and password.
-H “name: value”, –header “name: value”	Supply HTTP Header(s)	Click on the Raw Edit option or enter key-value pairs of HTTP headers in the Headers grid like this
-d “key=value”, –data “key=value”	Supply BODY parameters using www-form-urlencoded Content Type	Click Edit next to the Request Data text box and supply Body. If you need to pass content in JSON / XML, then select the Request ContentType dropdown like this
-d “some data”, –data “some data” -d @filepath, –data @filepath	Supply BODY parameters using Content-Type other than www-form-urlencoded. You can also supply content from a file.	Click Edit next to the Request Data text box and supply Body. If you need to pass content in JSON / XML, select the Request ContentType dropdown as shown. To upload content stored from a local file, use this option.
-F key=value, –form key=value	Supply Content-Type multipart/form-data according to RFC 2388	Enable the Multipart option for Request on the ZappySys UI, like this
-k, –insecure	Ignore SSL Certificate-related errors (e.g., allow expired certificate or untrusted certificate)	Go to the Advanced Tab and check the ” ignore SSL Related Errors option like this.
-x, –proxy [protocol://]host[:port]	Specify Proxy Settings	You can enable System default proxy or a custom proxy using the Proxy Tab in the OAuth / HTTP Connection Manager. Check this.

Convert Curl Syntax to SSIS PowerPack – REST API Task / JSON / XML Connector

Take a look at the following example of CURL syntax

curl POST -u myusername:mypassword -H "Accept: application/json" -H "AppVersion:1.0" "https://mysite/v1/api/customers/" -d "customerid=AAA&OrderDate=2016-01-01"

Where :

-u

--user

= UserID and Password for server (This option sends credentials in BASIC Authentication mode)

-H

--header

= HTTP Header (e.g. Key: Value)

-d

--data

= Data for POST

For step-by-step information, check Article Learn how to POST data to a URL using SSIS

Here is the screenshot of HTTP POST in JSON format using the ZappySys REST API Task

Convert CURL Command Line Syntax to SSIS – Perform HTTP POST operation using SSIS REST API Task – Send data in JSON content type

If you wish to POST data and parse the JSON response into rows, you can use the SSIS JSON Source. JSON Source has a powerful JSON Parser that parses large JSON (stored in a file, an API URL, or a Direct string) into Rows and Columns. This is useful if you wish to save the response into a target such as SQL Server or a CSV file.

JSON Source example

HTTP Headers – Convert Curl Syntax to SSIS

Set up HTTP headers in ZappySys Task/Connect, or click on Raw Edit or Add/Edit items in Grid.

HTTP Credentials – Convert Curl Syntax to SSIS

Here is how you pass credentials in the curl command line.

curl.exe https://myserver.com/api -u myusername:mypassword

To setup HTTP credentials in SSIS for Basic / NTFS or Kerberos use HTTP Connection Manager (Access Mode must be changed to Url from Connection on ZappySys REST API Task and for JSON Source Connector or XML Source check Use Connection Option)

If you have OAuth 2.0 Credentials (Application ID, Application Secret) then use SSIS OAuth Connection Manager rather than HTTP Connection

Check this article to learn about how to pass HTTP Basic Credentials in SSIS PowerPack

Proxy Setting – Convert Curl Syntax to SSIS

Here is how you pass Proxy server info in cUrl command line (

-x

along with Proxy Server Host/IP and Port)

curl.exe -x 192.168.2.1:8888 https://myserver.com/api

Convert Curl Syntax to SSIS – Client Certificate Settings

HTTP Connection Manager UI

Passing client certificate along with Web request (i.e. pfx/pkcs12/p12)

Passing client certificate from file path (i.e. pfx/pkcs12/p12)

The post Convert CURL to SSIS PowerPack / ODBC for API Call appeared first on ZappySys Blog.

How to call Amazon MWS API using SSIS

ZappySys — Sat, 15 Oct 2016 04:42:21 +0000

Introduction

⚠️ Deprecation Notice: MWS API is deprecated
Amazon’s MWS (Marketplace Web Service) is being deprecated and replaced by the newer AWS Selling Partner API (SP-API). For a more robust and secure integration, we recommend using our AWS Selling Partner (SP-API) Connector. As Amazon is phasing out MWS functionality and eventually plans to fully deprecate it.

In this post you will learn how to call Amazon MWS API (Amazon Marketplace Web Service) or Amazon Product Advertising API using SSIS PowerPack.

Using drag and drop approach you can consume data from Amazon MWS XML Web service. In this post we will use ZappySys XML Source connector to read data from Amazon MWS API and load into SQL Server. We will also use REST API Task to call any API from Marketplace Web service and save output into Variable.

Prerequisites

This post assumes following things

Basic knowledge of SSIS and XML format.
Amazon MWS Account and and valid AWSAccessKeyId and Secret Key to call Amazon MWS API (How to obtain Key / Secret and SellerId ? ).
You have tested few API calls using Amazon MWS Scratchpad this will give you idea about various API you can call and parameters you need to pass.

Testing MWS Requests using ScratchPad

Very first step we recommend before you call MWS API in SSIS is to get familiar with Amazon MWS Scratchpad tool. Check this article for detailed steps. You can also use Fiddler to debug some requests.

Calling Amazon MWS API in Scratchpad Testing Tool (Response Details Tab) – ListMatchingProducts Example

Call Amazon MWS API using SSIS

To consume data from any XML Source (File, SOAP Web Service or XML API) you can use ZappySys XML Source
connector. or REST API Task. In below example you will see how to call Amazon MWS API and save output into Variable. With this approach you can call any API (GET / POST / DELETE / PUT)

Download and Install SSIS PowerPack
Create test package
From SSIS toolbox drag ZS REST API Task and double click on the task to configure it
Select Request URL Access Mode = [Url from Connection]
Enter URL as below
```
https://mws.amazonservices.com/Orders/2013-09-01
```
In above URL
** mws.amazonservices.com => This is your endpoint. If you are not in USA then Click here to find correct endpoint
** /Orders => That’s your API you want to call
** /2013-09-01 => That’s your API version (You have to add this same version in your POST Body too (see next section).
In Select Url Connection dropdown select New OAUTH connection option.
When prompted on OAuth connection UI select Provider=Amazon MWS, Enter AWSAccessKey and Secret key and Click OK (If you have MWSAuthToken then enter in Access Token field else leave it empty). Do not test because it may not work yet.
SSIS OAuth Connection – Call Amazon MWS API (Marketplace Web service API)
Another Setting you may turn on is Retry Handling. This allows to over come ThresholdLimitReached error when we call API too fast.
Retry Options
Now select Method=POST, and in the body enter following
```
Action=GetServiceStatus&SellerId=A77XXXXXXX&Version=2013-09-01
```
If your API requires pagination (more than 100 records from response) then refer next section (Sample Package). In the case of Pagination your second request may look like below
```
Action=YourActionNameWithNextToken&SellerId=A77XXXXXXX&Version=2013-09-01&NextToken={{User::vNextToken,FUN_URLENC}}
```
NOTE: Enter all required parameters in the following format, as shown below. All values need to be URL-encoded. If you use SSIS variable, use it along with FUN_URLENC format specifier, e.g. {{User::myVariable,FUN_URLENC}}. If you use a hard-coded value, and not sure if it is URL-encoded, then encode it with the same FUN_URLENC format specifier/function, e.g. <>. Always check if SSIS configuration matches the one you see in MWS ScratchPad Request Details.
Format:
param1=value1¶m2=<>¶m3={{User::myVar,FUN_URLENC}}
Now click Test and see content in the Preview window.
SSIS REST API Task – Call Amazon MWS API – Get XML data save into SSIS variable
If you wish to save response data into SSIS variable then goto response Tab and check save option and select variable name.

Amazon MWS API Pagination

Many Amazon MWS API calls by default don’t return all records (Explained here). Rather than returning all rows you may get partial response (e.g. Max 50 or 100 rows). If you want to get all records then you have to use NextToken found in your XML. For API which requires pagination, in that case your POST body may be different for 2nd or higher requests. Notice how we changed Action and appended NextToken=xxxxxxxxxxx in the second request. Also notice we used encoded token (URL encoded). This is very important because as per Amazon Specs this value has to be Url Encoded.

We also documented similar use case of Amazon MWS Pagination for Power BI (Check this one) its mostly same setup except 2-3 properties might be named different way.

Now lets look at sample request / response for paginated MWS requests (For clarity we have removed many common Parameters which are automatically added at runtime e.g. Signature, Timestamp). We use ListOrders (First Request) and ListOrderByNextToken (Second+ requests)

NOTE: Most of Amazon MWS API calls are heavily throttled means you cannot call more than X calls per minute or hour. So Refer to Throttling Limits for each API Endpoint.

First Request

POST https://mws.amazonservices.com/Orders/2013-09-01

Action=ListOrders&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z

First Response



    
        2YgYW55IxxxxxxxxxxxxVyZS4=
        2017-02-25T18%3A10%3A21.687Z
        
             ....... 
             ....... 
             .......

Second Request or higher (MWS API with Pagination using NextToken)

POST https://mws.amazonservices.com/Orders/2013-09-01

Action=ListOrdersByNextToken&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z&NextToken=YOUR_ENCODED_TOKEN_FROM_PREV_REQUEST

Second or higher Response



    
        2YgYW55IxxxxxxxxxxxxVyZS4=
        2017-02-25T18%3A10%3A21.687Z
        
             ....... 
             ....... 
             .......

Sample SSIS Package

If you want to see complete working package for pagination pattern then download from here. Change AccessKey, SecretKey, SellerID and Path as needed to make this package work.

Here is the screenshot of example SSIS package for Amazon MWS.

Amazon MWS APi Pagination (ListOrders , ListOrdersByNextToken Example)

Copy/Paste Properties

Here is copy/paste values for properties listed above.

DirectPath:  https://mws.amazonservices.com/Orders/2013-09-01
EnablePageTokenForBody: True
Filter:  $.ListOrders[$tag$]Response.ListOrders[$tag$]Result.Orders.Order[*]
HasDifferentNextPageInfo: True
HttpRequestData:  Action=ListOrders[$tag$]&SellerId=A1xxxxxxxx&CreatedAfter=2018-09-01T04%3A00%3A00Z&MarketplaceId.Id.1=ATVPDKIKX0DER
HttpRequestMethod:  POST
NextUrlAttribute: $.ListOrders[$tag$]Response.ListOrders[$tag$]Result.NextToken
NextUrlSuffix: &NextToken=<%nextlink_encoded%>
NetUrlWait: 10000 (ms) -- slow down to avoid throttling ... Max 6 requests per minute, increase if you get API Limit Error
PagePlaceholders:  body=|ByNextToken;filter=|ByNextToken

Understanding Pagination Properties

Here are various properties you have to set for Pagination.

Property	Description
Src	Change this to correct URL. Look at Scratchpad Response Details Tab and Findout first line after POST and before “?” . e.g. /Orders/2013-09-01 this will be used in your URL. Example: https://mws.amazonservices.com/Orders/2013-09-01
Body (i.e. RequestData)	Scratchpad shows HTTP POST section. You can copy that and remove System supplied parameters and then arrange all in one line to form your Body for request. Example: POST /Orders/2013-09-01?AWSAccessKeyId=AKxxxxxxxxxx &Action=ListOrders &SellerId=A10zzzzzzz &SignatureVersion=2 &Timestamp=2018-09-17T21%3A35%3A57Z &Version=2013-09-01 &Signature=3GigipfRsQgzzzzzzzzzzzzzs%3D &SignatureMethod=HmacSHA256 Above HTTP POST can be used as below for your Body in ZappySys Driver. Rest of the parameters are automatically supplied by system. See special placeholder named [$tag$] this gets replaced at runtime when you set EnablePageTokenForBody=True and HasDifferentNextPageInfo=True Action=ListOrders[$tag$]&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z
Filter	Change this parameter according to XML structure in the response. For example: If you see below response (first xml) in Scratchpad then your Filter will be *$.ListOrdersResponse.ListOrdersResult.Orders.Order[]** to enable pagination refer to Sample listed here (click Example response at the bottom of that page). In second response and onwards your Filter should be below (See Bold Part). $ListOrdersByNextTokenResponse.ListOrdersByNextTokenResult.Orders.Order[]However in Driver we replace ByNextToken* with [$tag$]. Like below. Using [$tag$] will automatically pick up the correct filter based on page number. You must set HasDifferentNextPageInfo=True to use [$tag$] feature.$.ListOrders[$tag$]Response.ListOrders[$tag$]Result.Orders.Order[]Example of first page response* 2YgYW55IxxxxxxxxxxxxVyZS4= 2017-02-25T18%3A10%3A21.687Z ....... ....... ....... Example of next page response 2YgYW55IxxxxxxxxxxxxVyZS4= 2017-02-25T18%3A10%3A21.687Z ....... ....... .......
NextUrlAttribute	This property defines which attributes indicate Token for next page. You can use this token in Body of next request (see NextUrlSuffix and EnablePageTokenForBody). You must set EnablePageTokenForBody=true to use Extracted token in Body.
StopIndicatorAttribute	Some APIs like GetReportRequestList stop pagination based on HasNext attribute value. (ListOrders doesnt need this property to be set). Use this property to extract HasNext from response by supplying correct Filter expression. You also need to set NextUrlEndIndicator property which defines static value which indicates the last page. Example properties as below. NextUrlEndIndicator=false StopIndicatorAttributeOrExpr= $.GetReportRequestList[$tag$]Response.GetReportRequestList[$tag$]Result.HasNext
NextUrlSuffix	This is used to create a string for NextToken attribute for next page request. If you want to just append extracted token to Body then you dont have to set this but in our case, we have to use &NextToken=EncodedValueOfExtractedToken so we have used &NextToken=<%nextlink_encoded%> expression.
PagePlaceholders	This property contains [$tag$] values for first page and next pages. You can define tags for filter, body or header. Each pair must be pipe delimited. First value or pair is [$tag$] for first request and second value of the pair is [$tag$] for any next request afterwards. Our first request use blank value for [$tag$]. Example: PagePlaceholders=’body=\|ByNextToken;filter=\|ByNextToken’
HasDifferentNextPageInfo	Set to True – This enables use of [$tag$] inside body, filter, headers to use different values of first request and second onwards requests.
EnablePageTokenForBody	Set to True – This appends value extracted from NextUrlSuffix inside body (e.g. NextToken).

Amazon MWS API Pagination (For Old Version)

So in previous section we saw how to achieve pagination in one step (for newer version of SSIS PowerPack). But there will be a case you cannot use latest SSIS PowerPack or you want to take complete control on each aspect of pagination. In such case use below method (Depreciated). In newer version we added many new properties so you dont have to do Loop pattern like below.

Sample SSIS Package

If you want to see complete working package for pagination pattern then download from here. Change AccessKey, SecretKey, SellerID and Path as needed to make this package work.

Here is the screenshot of example SSIS package for Amazon MWS.

SSIS Example Package – Read data from Amazon MWS Web service (API Call). Paginate API calls using NextToken

Load Amazon MWS API data into SQL Server using SSIS

In this section we will see how to use ZappySys XML Source connector to read data from Amazon MWS Web Service and load into SQL Server

For making things simple we are calling

Download and Install SSIS PowerPack
Create test package
From SSIS toolbox drag Data Flow task and double click task to go to Data Flow designer
Drag ZS XML Source from SSIS Toolbox
Configure XML Source as below
SSIS XML Source – Read data from Amazon MWS API (Amazon Marketplace Web service call)
Drag OLEDB destination from SSIS Toolbox.
Connect XML Source to OLEDB Destination and map input column to target table
Execute dataflow
Get data from Amazon MWS API – Save to File or SQL Server using SSIS

Working with Amazon MWS Feed API (e.g. Upload file)

Amazon MWS Provides Feed APIs which has slight different requirements. Important requirement in Feed file upload is you have to supply Content MD5 Hash. If you use SSIS PowerPack then you don’t have to worry about that complexity because calculating MD5 Hash is automatically done.

Upload Amazon MWS Feed File in TSV / CSV format (Tab separated)

Here is the screenshot for how to upload Tab separated values in CSV file. You can specify body from Variable or Read from file. If you read from file then must check File Upload/Multi-Part option and start path with @ symbol as below.

Call Amazon MWS Feed API – Upload CSV file format (Tab separated values – TSV)

Upload Amazon MWS Feed File in XML format

To upload feed file in XML format use same settings as above except Content Type should be XML (text/xml) (Select in the dropdown)

Calling Amazon Product Advertising API

If you have need to call Amazon Product Advertising API then also you can use above techniques (Use MWS Provider on OAuth connection) to call Product Advertise API. Both API (MWS API and Product Advertise API) use same signature hashing algorithm so its possible to use MWS Provider for Hashing. You may have to change your API URL parameters as needed when you call Amazon Product Advertising API. Basically when you call any MWS or Product Advertising API using GET method then OAuth connection manager automatically appends Signature to URL at runtime.

Import data from MWS Custom Reports (Inventory Report Example)

Now lets look at how to extract data from Custom Report API . Reading data from Custom Reports not single step process because report generation is Job style API. Which means you send report request and wait until its done. Once Report ready you have to read in CSV format or XML. Some Reports only available in CSV format. Check this post to understand how complex it can be to get data. We will make it simple for you to understand this in 3 steps. Basically calling reports requires minimum 3 API calls (see below)

Create a new report request – Cal RequestReport API . It returns ReportRequestId (You can use it in the next step)
Check Report Status see its done – Call GetReportRequestList API (Pass ReportRequestId got in the previous step to get data for only one Report request we care). Keep checking Report Status every few seconds until you get _DONE_ status in response.
Read Report Data – Call GetReport. This API call returns CSV or XML data. So use correct component for this step. If CSV data is returned then we must use CSV Source rather than XML Driver.

Now lets see how to read Amazon MWS custom report in SSIS. In this example, We will get Inventory by calling _GET_MERCHANT_LISTINGS_DATA_ report type.

Download Sample Package for MWS Report

To make things simple we have created a demo package – ready to go.
Download SSIS 2012 Demo – Amazon_MWS_API_GetReport_2012.

Prepare demo package for first run

Once you download and extract above sample package. Import to your existing SSIS Solution. Its SSIS 2012 format so it may upgrade to higher version if you using SSIS 2014, 2016 or 2017.

Once package is opened perform following steps in before you can execute the package.

Enter MarketplaceId, SellerId in Variables. Double click OAuth Connection and enter your MWS Account Key and Secret.
Create c:\temp directory (This is where we will save final report file)
Run this package
Go to Execution log and Find ReportId (This you can hardcode in Variables for Design time Preview / Metadata changes etc)

See below package screenshot

Read data from Amazon MWS Report in SSIS (Get Inventory Listing Example)

Steps Explained

Now lets see each steps in depth. We broken whole process in 3 groups.

Create Report Request (Step1)

This is the first step how you initiate report request. Use REST API Task to call this step. Here is raw request. Many items in body added at runtime (e.g. Signature, Timestamp).

On Request tab see how we used SSIS Variables to get many values at runtime dynamically.
On Response Tab we have to extract RequestId and save into variable.

POST 
https://mws.amazonservices.com/Reports/2009-01-01

AWSAccessKeyId=Axxxxxxxxx&Action=RequestReport&MarketplaceId=ATVPDKIKX0DER&ReportType=_GET_MERCHANT_LISTINGS_DATA_&SellerId=Axxxxxxxxx&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2019-06-05T21%3A23%3A31.090Z&Signature=2AbGQQ4xxxxxxxxxxxxxxxxxxxxjJzox%2FU%3D

RequestReport Action – Generate Amazon MWS Custom Report

Extract value from XML response using XPATH and save to Variable (Get RequestId Example)

Here is how to configure above task

Wait until data is ready (Step2)

Once we send RepotRequest and we have RequestId we can call step2 using REST API Task. In this Task we can use Status Check Feature. This is pretty handy feature because we dont have to implement Loop in SSIS. Basically we have to keep checking report until we get _DONE_ status in the response (ReportProcessingStatus node in XML)

Here is how to configure the task.

Request Tab Configuration.
URL:

https://mws.amazonservices.com/Reports/2009-01-01

Method: POST
Body:

Action=GetReportRequestList&ReportRequestIdList.Id.1={{User::ReportRequestId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}

Response Setting Tab Configuration
Response content type: Xml
Filter Expression:
```
//*[local-name() = 'ReportProcessingStatus']  
```
Status Check Tab
Check Enable Status
Success Value:
```
_DONE_
```
Check Every 5 Seconds

REST API Task – Status Check Feature (Wait until data is ready loop)

GetReportId (Step3)

Once above task finished we can call same request again but this time we have to configure slightly different way.

Here is how to configure the task.

Request Tab Configuration.
URL:

https://mws.amazonservices.com/Reports/2009-01-01

Method: POST
Body:

Action=GetReportRequestList&ReportRequestIdList.Id.1={{User::ReportRequestId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}

Response Setting Tab Configuration
Response content type: Xml
Filter Expression:
```
//*[local-name() = 'GeneratedReportId']  
```
Check Save to Variable: Save to User::ReportId
Status Check Tab
Make Enable Status is unchecked

Get Report Data (Step4,Step5 or Step6)

Now we are ready to ready our report. There are two ways we can do in SSIS.

First approach (step 4,5) is download the report and save to local disk and then in the next step we can use data flow (CSV Source) to parse the saved file.
Second approach (step6) is rather than saving to disk we can directly call API using CSV File source and parse content that way. Lets talk each approach briefly.

Step4,5 – Load Report From File

If you wish to save Report on disk then you can use REST API Task and call request like below.

Here is how to configure the task to download the file.

Request Tab Configuration.
URL:
```
https://mws.amazonservices.com/Reports/2009-01-01
```
Method: POST
Body: Action=GetReport&ReportId={{User::ReportId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}
Response Setting Tab Configuration
Response content type: Xml
Filter Expression:
```
//*[local-name() = 'GeneratedReportId']  
```
Check Save to Variable: Save to User::ReportId
On Response Settings Tab : Set Response charset to Western European [Windows 1252] (or directly set via Properties Grid to Windows-1252 under ResponseCharset Property)
Status Check Tab
Make Enable Status is unchecked

To read download CSV file use settings described in below section except URL change it with local file path.

Step6 – Load report from URL

If you wish to load directly from URL then use same setting as above but in CSV Source inside data flow.

Here is how to configure CSV Source.

Request Tab Configuration.
URL:

https://mws.amazonservices.com/Reports/2009-01-01

Method: POST
Body:

Action=GetReport&ReportId={{User::ReportId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}

General Tab
Column Delimiter: {TAB}
On Encoding Tab : Set to Western European [Windows 1252] (or directly set via Properties Grid to Windows-1252 under CharacterSet Property)

Conclusion

Amazon Marketplace Web service (MWS) provides great way to automate many functionality for Marketplace Seller. Using ZappySys SSIS PowerPack you con consume data from any REST API or Web service without replying on SDK / coding approach (e.g. C#, Java, Python, Ruby).

The post How to call Amazon MWS API using SSIS appeared first on ZappySys Blog.

SSIS – Extract single XML node using XPath from SOAP response

ZappySys — Wed, 07 Sep 2016 16:01:07 +0000

Introduction

In this post, you will learn how to extract a single XML node value from your XML web response (SOAP Web service call) using an XPath expression in the SSIS REST API Web Service Task and the XML Parser Task.

This article assumes you have basic knowledge of how to call a REST API or a SOAP Web service in SSIS. If you are not familiar with that, then refer to the following articles.

Calling SOAP Web Service in SSIS (XML Source)

Call REST API using SSIS Web Service Task / JSON / XML Source

Calling REST / SOAP Web service using the SSIS REST API Task

Refer to the following video on how to call a REST API web service.

Calling REST API in SSIS using REST API Task, Pass headers, Body, Url Parameters

Extract the REST API Response into a file

Here is how you can save the response into a variable or a file. Go to the Response settings tab and select the check Response content option, as shown below. You can also Filter Response content (e.g., Extract only a single value from a node).

SSIS Rest API Task – Extract data from a single XML node using XPath expression (Namespace used)

Extract a single XML Node using XPath (XML with Namespace)

To extract a single value from the Response XML, you can define an XPath expression on the REST API Response tab as follows. If your response XML document contains namespaces (e.g., the node looks like this: abcdefg), then you have to write XPath in a particular way, as shown below. Notice we have not specified the Namespace prefix ns1

Sample XML Document (Or web response from SOAP Service)


	abcdefg

XPath Expression Example:

//*[local-name()='Data']

XPath Output:

The above expression will produce the following result

abcdefg

Extract Attribute Value using XPath local-name() function

Now, let’s find out how to extract the attribute value rather than the element.

Consider the following XML. Let’s say we want to extract the id from the data node. In such a case, the previous XPath expression won’t work, so we need to use a slightly different XPath.


	abcdefg

Let’s use the XML Parser Task to test XPath.

Expression 1 – Get value by direct attribute name

Write the following expression as below and click Test

//@*[local-name()='id']

Extract XML Attribute Value using XPath expression (Use local-name function)

Expression 2 – Get specific element attribute value

If you want to be specific and read the value of the id attribute from a particular data element, you can use an expression like this:

//*[local-name()='Data']/@*[local-name()='id']

Extract XML Attribute Value using XPath expression 2

Configure the SSIS Rest API Task to filter XML Response using XPath

The screenshot below shows how to filter XML data using an XPath expression. You can save extracted data to an SSIS Variable or save it to a file.

SSIS Rest API Task – Extract data from a single XML node using XPath expression (Namespace used)

More Examples of XPath expressions

XPath expression examples:
============================

Example1: Extract value of any element by node name (without namespace specified) 

aaaaa

XPath Expression:  //*[local-name() = 'Token']
Returns: aaaaa
----------------------------
Example2: Extract text of element called Token 

aaaaa

XPath Expression:  ./AuthInfo/Token/text()
Returns: aaaaa
----------------------------
Example3: Extract value of attribute called Version 



XPath Expression:  ./Doc/Row/@Version
Returns: 1
----------------------------
Example4: Extract value of following-sibling node or use preceding-sibling 


	
		User
		admin
	
	
		SessionId
		s_123456
	


Expression:  //*[local-name()='PropName' and text()='SessionID']/following-sibling::node()[local-name()='PropValue']
Returns: s_123456
----------------------------
Example5: Extract node value with matching attribute 

s_1234

XPath Expression:  //*[local-name()='Prop' and @Name='SessionId']
Returns: s_1234
----------------------------
Example6: Extract attribute value by name (without namespace specified) 

s_1234

XPath Expression:  //@*[local-name()='token']
Returns: 123456
----------------------------

The post SSIS – Extract single XML node using XPath from SOAP response appeared first on ZappySys Blog.

How to import XML into SQL Server using SSIS

ZappySys — Tue, 30 Aug 2016 22:42:05 +0000

Introduction

In this post you will learn how to import XML file into target such as SQL Server, CSV, Excel, Oracle. Converting or Importing XML requires flattening XML structure (i.e. hierarchy) sometimes we refer this process as de-normalization. Parsing XML can be tricky depending on how many levels you have and from which level you need data. If you want to export XML rather than Import then check this article.

This article assumes you have installed SSIS PowerPack and watched introduction videos of SSIS XML Source Connector and SSIS XML Parser Transform

Import XML File using SSIS XML Source

ZappySys SSIS XML Source is a very powerful component. It supports not only reading from file but also reading from Direct XML String or REST API / SOAP Web Service.

How ZappySys XML Source parse/output data

XML Source created by ZappySys is very different than what Microsoft provides out the box because ZappySys component outputs one flat view of nested XML document (de-normalized) compared to many outputs what Microsoft XML Source produces which is hard to manage and join based on relation with parent node.

With ZappySys XML component you don’t have to worry about joins because by default parent columns included along with hierarchy you extracted (e.g. If you extract $.Customer.Orders[*].OrderDetails[*] then you get all columns listed under and nodes)

ZappySys XML Source (Read XML Files, Consume REST API, SOPA Web Service)

Importing single or multiple XML Files

To import single files specify full XML file path

Example: c:\ssis\File_001.xml

To import multiple files using XML source use wildcard pattern

Example: c:\ssis\File_*.xml

XML File Source – Specify File Path with wildcard pattern (Loop multiple files)

Importing data from multiple XML files

Using variable to make path dynamic

You can also use variable placeholders in your path to make it dynamic

Example: c:\ssis\{{User::FileName}}.xml

Importing data from XML Web Service or REST API

XML Source supports consuming REST API data or calling XML SOAP Web services. This article doesn’t include this feature in depth. But refer to learn more about this feature.

SSIS XML Source – Direct Mode – Call REST API web service or read from XML File

Parse XML string and split into columns using XML Parser Transform

Parsing XML Data – XML String into Columns using SSIS XML Parser Transform

Other Settings (XML Source, XML Parser Transform)

Preview Data

All access modes support preview data like below. Change your settings such as Filter expression, Source etc and click preview.

Preview XML Data

Filtering data using expression

You can filter XML data by specifying simple expressions (e.g. $.store.book[*]) to navigate various parts of XML where you can extract data (See above screenshot).

Things to remember for filter expressions

Expression must start with $ (means root from where parsing will start)
Use dot (.) to separate Hierarchy and use brackets to indicate array []
Use * to indicate all records

Sample XML for examples



  
    Bob
    2015-01-01
    
      Atlanta
      GA
    
  
  
    
      2005-02-01T10:00:09
      
        Tea
      
      
        Coffee
      
    
    
      2005-04-01T21:59:20
      
        Tea
      
      
        201
        Soda
      
    
  
  
    
      2005-02-01T21:59:20
      
        Apple
      
      
        Orange
      
    
    
      2005-01-01T21:59:20
      
        Soda
      
      
        Milk

Example of filter expression

Filter	Output
$	Get all records
$.Root.Customer[*]	Get all customers
$.Root.Customer[].Order[]	Get all orders for all customers
$.Root.Customer[].Order[](Include Parent Columns=True)	Get all orders for all customers (Include Parent Columns option checked)
$.Root.Customer[*].@CustId	Get only Customer Id for each customer record
$.Root.Customer[*].Order[:1]	Get first order for each customer
$.Root.Customer[*].Order[:3]	Get top 3 orders for each customer
$.Root.Customer[*].Order[:1]	Get last order of each customer
$.Root.Customer[*].Order[0,4]	Get orders starting from first row to fifth row (Zero based index). If you want 5th row to 10th row then use [4,9]
$.Root.Customer[*].Order[2]	Get 3rd order for each customer

Defining array elements

Any hierarchy you extracting with array indicator can be added to array element name list (On Array handling Tab). This helps when you only have one row for node specified as array.

For example if your filter expression is $.Customer.Order[*].OrderItem[*] in this case you indicated that you can have one or more Order nodes under Customer and you may have one or more OrderItem nodes under Order node. If this is the case then set array handling as below

Handling Rows/Array Elements (Repeated XML Elements)

Editing Column Metadata (Length and DataType)

XML Source and XML Parser both detects Datatype/Length by scanning sample data (Default 300 rows) but sometimes this is not accurate and it may produce invalid metadata. If that’s the case you can easily change it in Columns tab. Once you edit Length or DataType make sure you check Lock column (Last column in grid). Once Lock is applied metadata refresh wont affect your changes. You can also add new column manually if its not detected (Lets say your XML is very large and many columns not found in first 300 rows scanned). To add new column type in the last grid row or click new icon in the grid toolbar.

Edit XML File Source Column Metadata

Conclusion

In this post you have seen how to use SSIS XML Source Connector and SSIS XML Parser Transform to extract data from nested XML documents without any coding.

Related to: SSIS Import XML Files | SSIS Parse XML data | SSIS Extract from XML files | Read from XML Files

The post How to import XML into SQL Server using SSIS appeared first on ZappySys Blog.

Calling REST API in SSIS with Basic Authentication / Userid-Password (BASE64 Authorization header)

ZappySys — Thu, 27 Aug 2015 20:58:17 +0000

Introduction

In this blog post you will explain how to pass basic credentials (i.e. UserID/Password) along with your web request. You will also learn about setting up Authorization Header for HTTP Web Request in Base64 manually. To learn more about how to consume / call REST API in SSIS check this article.

HTTP Authorization Header basics

As per HTTP Standard you can pass credentials very simple way using basic Authorization header. Below is the sample of Basic Authorization header.

As you can see it consist of HeaderName=Authorization and Value=some base64 encoded string
Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==
As per standard Base64 encoded string is made up with two elements. your_userid:your_password

NOTE: Base64 is encoding and not encryption method. So never hand over your encoded Authorization header string to anyone. Anyone can decode Base64 encoded string easily by using tools like this

Passing Basic Credentials along with HTTP Web Request

You can pass credentials (i.e. UserID/Password) in two different ways.

Automatic approach – Using ZS-HTTP Connection Manager (BASE64 Encoded Header automatically generated)
Manual approach – Supplying Authorization where UserID/Password encoded as Base64

Automatic approach – Using ZS-HTTP Connection Manager

Now lets look at how to use automatic approach to pass Basic Authorization in following 3 different tasks/components

Passing basic credentials in SSIS REST API Task
Passing basic credentials in SSIS JSON Source
Passing basic credentials in SSIS XML Source

Using SSIS REST API Task (Pass Basic credentials using Connection Manager)

The simplest way to pass credentials is use HTTP Connection Manager. If you are using REST API Task then you can use Url from Connection Access Mode. And on connection manager specify UserId and password to pass Basic Authorization Header

Advantages of this method are..
1) Its very simple
2) Credentials are encrypted in SSIS Package

NOTE: We recommend using https:// Urls whenever possible otherwise your web request can be decoded in plain text by any servers sitting between you and target and possibly they can get your password by decoding base64 header.

Analyze Base64 Encoded HTTP Authorization Header generated by HTTP Connection Manager

You can analyze Webrequest in some 3rd party Free Debugging tools such as Fiddler (Check this article). As you see how its sending Authorization header in Base64 encoded string.

Base64 Encoded HTTP Basic Authorization Header – View in Fiddler

Using SSIS JSON/XML Source (Pass Basic credentials using Connection Manager)

You can use HTTP Connection manager in JSON Source and XML Source too. Only difference is you have to check Use Credentials checkbox and select HTTP connection manager from available connection managers dropdown.

Select AccessMode as “File Path or Web URL”
Enter API URL
Check “Use Credentials” option
Click “New” next to Connection dropdown or select New “ZS-HTTP” connection from dropdown
Configure HTTP Connection as below (i.e. Enter URL, Select Basic Auth, Enter UserID, Password)
Clock OK on HTTP connection manager UI to save and now Test JSON or XML Source UI by clicking Preview (Assuming no other configuration needed)

SSIS JSON Source – Passing Basic Credentials (Supply UserID / Password) using HTTP Connection

Manual approach – Supplying Authorization where UserID/Password encoded as Base64

Our JSON Source, XML Source and REST API Task they all support passing BASE64 Encoded String for Authorization header (see below screenshot). Setting up userid and password in HTTP Connection Manger would do same trick as setting up it manually.

To encode any Header value in BASE64 format you simply have to add BASE64ENC format specifier after variable name.

Example: {{User::varUserAndPassword,BASE64ENC}}

SSIS JSON Source Task – Pass Base64 Encoded HTTP Authorization Header

Reference

Wikipedia Article – HTTP Basic Authorization Header

Download FREE SSIS Components

The post Calling REST API in SSIS with Basic Authentication / Userid-Password (BASE64 Authorization header) appeared first on ZappySys Blog.