ssis rest api task Archives | ZappySys Blog

How to call Google DFP API with SSIS – DoubleClick

ZappySys — Tue, 14 Feb 2017 19:28:31 +0000

Introduction

Google DoubleClick is one of the most popular platforms for Advertisers. Many times you have need for custom integration or Automation for many operations. In this article we will learn how to Call Google DFP API (i.e. DoubleClick for Publishers) without coding using SSIS (Microsoft SQL Server Integration Services). In our previous article we discussed how to Integrate Google AdWords API using SSIS (Click here).

If you are new to SSIS then no worry there are plenty of tutorials for ssis

To achieve Custom integration for DoubleClick (DFP) API we will use SSIS XML Source. XML Source is part of SSIS PowerPack which has 45+ connectors and Tasks.

Prerequisite

Before you can finish tasks explained in this article you have to finish below steps. This article assumes you have basic knowledge of SSIS (SQL Server Integration Services)

Make sure you SSIS designer installed. Sometimes its referred as BIDS or SSDT (Get from here)
Download SSIS PowerPack
Download SoapUI (Its free third party tool to test SOAP API). SoapUI can help you to generate XML Request BODY easily from WSDL file provided by API Vendor. You can also test your service by supplying parameters.
Optional – Another very useful Free tool is Fiddler. You can use it to see raw request/response (Check this article)

Download DFP API Example SSIS Package

Click here to DFP API download Sample for SSIS SSIS Package (SSIS 2012, 2014, 2016)

Screenshot of Sample Package:

Example SSIS Package – Google DFP API Integration (Read / Write data in Google Double Click for Publisher) – No Coding required

Making your first DFP API Call using SSIS

Once you install PowerPack you are ready to execute your very first Google DFP API Call.

NOTE: If you don’t have any DFP network for test then see next section (Explains how to create test network for testing).

Lets look at how to call Google DFP API step-by-step.

Create new SSIS Project and open package designer
Drag ZS REST API Task from SSIS control flow toolbox and drop it on designer.

REST API Task is useful when you want to call SOAP/REST API but not necessarily parse response into rows and columns. We will look at JSON Source later in this article which can actually parse response into rows and columns (Useful to load Google DFP data into SQL Server or other RDBMS / flatfile )
Double click REST API Task to configure. Select Access mode to [Url from Connection]

Enter following URL in the URL Textbox

 https://ads.google.com/apis/ads/publisher/v201702/NetworkService

From the connection dialogbox select ZS-OAUTH to create new OAuth Connection for DFP API
On OAuth Connection dialogbox select Google from Provider dropdown. In this demo we will use Default OAuth App but you can register your own google OAuth app if you wish to use Custom OAuth option from UI.

Enter following Scopes in the Scopes textbox (or select manually by clicking Select Scopes button). Scope is nothing but permission for API (In our case View/Manage DFP data and Read/Write Report Files to Cloud Storage).

https://www.googleapis.com/auth/dfp
https://www.googleapis.com/auth/devstorage.read_only
https://www.googleapis.com/auth/devstorage.write_only
https://www.googleapis.com/auth/cloud-platform.read-only
https://www.googleapis.com/auth/devstorage.full_control
https://www.googleapis.com/auth/cloud-platform
https://www.googleapis.com/auth/devstorage.read_write

If you are not going call ReportService API to generate Reports in CSV format then you will need only one scope
```
https://www.googleapis.com/auth/dfp
```
Click Generate Token button. You will see browser popup for login and then Accept option to confirm permissions like below. Once you don’t it will populate tokens and prompt to save tokens to backup file. Secure token backup to safe place.
SSIS OAuth Connection – Connect to Google DFP API (DoubleClick SOAP Web Service)
Once done Click Test Connection and if its green then Click OK to save connection
Once you are back to REST API Task UI then select Method to POST
Select Request Body Content Type to XML (i.e. text/xml)
Click edit button next to Request Body and enter following XML.
```
   
      
   
   
      
   
```
If you are wondering how did we get above XML fragment then read next section about using 3rd party tool called SoapUI. Also we have detail article for calling SOAP request. Check this article on how to use SoapUI (free 3rd party tool) to create SOAP request Bodyfrom WSDL. DFP has many API endpoints for different actions (e.g. NetworkService , InventoryService , …). For each service you have different Api URL and different WSDL. In the next section you can learn how to create correct SOAP Body (for POST) using SoapUI tool.

For example if you using DFP NetworkService API for version v201702 then your help page URL would be like this
https://developers.google.com/doubleclick-publishers/docs/reference/v201702/NetworkService
And on the same page you will see WSDL link below
https://adwords.google.com/api/adwords/mcm/v201609/ManagedCustomerService?wsdl
(Just download that WSDL XML and save to local disk then use with SoapUI to generate XML body to submit for any DFP API call.)
Once you done with above steps, your Task Configuration will look like below.
SSIS Rest API Task – Call Google DoubleClick API – Get all DFP Networks

Now click Test Request button. You will see Response popup with below content if its successful (Scroll at the bottom on response form and you will see XML data) .. see below

------------------------------------
Request
------------------------------------
POST https 1.1 ==> /apis/ads/publisher/v201702/NetworkService
Host: ads.google.com

>>>> HEADERS <<<<<
Authorization: Bearer ya29.Glz-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: text/xml;charset=UTF-8
User-Agent: ZappySysApp
Host: ads.google.com
Content-Length: 291
Expect: 100-continue
Accept-Encoding: gzip, deflate
Connection: Keep-Alive

------------------------------------
Response
------------------------------------
POST 1.1 200 OK==> OK

>>>> HEADERS <<<<<
Content-Encoding: 
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Alt-Svc: quic=":443"; ma=2592000; v="35,34"
Transfer-Encoding: chunked
Cache-Control: private, max-age=0
Content-Type: text/xml; charset=UTF-8
Date: Sun, 26 Feb 2017 01:18:32 GMT
Expires: Sun, 26 Feb 2017 01:18:32 GMT
Server: GSE

>>>> Cookies <<<<<

>>>> CONTENT <<<<<


	
		
			27ab3d2e4e1fcb8f444b317d0b0f871d
			50
		
	
	
		
			
				551196
				XFP sandbox property
				238897396
				ca-pub-2799179143725683
				America/New_York
				USD
				237897516
				true

How to generate SOAP Request Body using SoapUI tool

This section describes how to create DoubleClick SOAP API Request Body (XML fragment) for any DFP SOAP API call described anywhere in this article.

Download SoapUI (Its free third party tool to test SOAP API). SoapUI can help you to generate XML Request BODY easily from WSDL file provided by API Vendor. You can also test your service by supplying parameters.
Now download WDSL xml for appropriate API calls you want to make. Assume that you want to call getAllNetworksResponse API found under NetworkService. For that first navigate to help file page and open help page. Make sure you select correct version from help navigation. Copy WSDL link and open in new browser window. Save XML to local disk (e.g. c:\api\dfp_networkservice_wsdl.xml (See below screenshot how to find DFP API WSDL link)
How to get Google DFP API WSDL URL (Used to generate POST Body)
Once SoapUI is downloaded and installed click File menu > Create New SOAP Project option
Name your project (e.g. DFP API) and specify WSDL URL or File Path if it was saved locally (e.g. c:\api\dfp_networkservice_wsdl.xml) and click OK
Now navigate to API Action for which you would like to get Body. Click Request node (If missing create new) and edit request. You may see XML like below (If some optional parameters not visible in XML then click Re-create request with optional parameters button from toolbar)
Call Google DFP API – Generate SOAP Request Body XML using SoapUI (Google DoubleClick Web Service)
Edit necessary parameters from above XML code and copy to Body of SSIS REST API Task.

Creating test network – DFP Sandbox

Very first step we recommend before testing DFP API call is create test network. You can call DFP API against LIVE network but not recommended if you are calling create/update API.

To create test network you can follow exact same steps described in the previous section except the Body step (Step#12). Use following Request body to call makeTestNetwork command.

API makeTestNetwork – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/NetworkService
Body (see below):

Once you click Test you may receive following response. Note your network code (e.g. 1122334455 from below). This network code is used in pretty much all DFP API to view or manage ad network related items or properties.

API makeTestNetwork – Response


	
		
			ba61d79efee325bf4b5aa45c46836021
			1021
		
	
	
		
			
				551196
				XFP sandbox property
				1122334455
				ca-pub-2799179143725683
				America/Los_Angeles
				USD
				237897516
				true

Above response contains test networkcode copy that code becuase you will need in almost every API call you make later. You can only have one test network. If test network is already created and you try to execute above code then you may receive error AuthenticationError.GOOGLE_ACCOUNT_ALREADY_ASSOCIATED_WITH_NETWORK

You can also get network code using two different ways.

Visit your DFP console homepage by visiting https://www.google.com/dfp/ and you will see network name and code in the top potion. Also its listed in the URL (e.g. https://www.google.com/dfp/1234567)
Another way to get available networks for your login is to visit DFP API Play ground here https://dfp-playground.appspot.com/ and you will see network list in the dropdown ( name and network code)

How to Create new Ad Units

Now lets look at an example which will create few Ad Units by calling createAdUnits API. If you are creating Ad Unit at the root level then you will need to specify correct Parent ID (i.e effectiveRootAdUnitId). To obtain that Parent ID you may have to call getCurrentNetwork API. In the response you will see effectiveRootAdUnitId.

There is another way to know effectiveRootAdUnitId or Id of any Ad Units which can be Parent for new AdUnit. Goto google.com/dfp > Click Inventory Tab > Click Ad Units Side menu > Click Download ad units hyper link. You can also download AdUnits as CSV file from DFP Portal and look for Ad Units which are created at the root. Parent ID column for Root level Ad Units is basically called effectiveRootAdUnitId.

Now lets look at how to get EffectiveRootID by calling getCurrentNetwork.

API getCurrentNetwork – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/NetworkService
Body (see below):




	
		1122334455
		ZappySysApp

API getCurrentNetwork – Response


	
		
			81d1b00e8fbfb862129eb2dd485cf7bb
			4507
		
	
	
		
			
				551196
				XFP sandbox property
				1122334455
				ca-pub-2799179143725683
				America/New_York
				USD
				237897516
				true

If you want to save single Element Value from Response into SSIS Variable then perform following steps (example of how to get just effectiveRootAdUnitId from above response)

On REST API Task > Go to Response Tab > Select Response Content Type = XML
Enter following expression in the XPath filter
```
//*[local-name() = 'effectiveRootAdUnitId']
```
Check Save response content option
Select Variable from Dropdown – Click New Variable > Name it RootAdUnitId.
Click Test Request Button to test…. In the content textbox you will now see only Numeric value extracted by Filter

SSIS REST Api Task – How to extract single vale from response and save to SSIS variable

Once you know Parent ID for your AdUnit now lets look at how to create Ad Units by calling createAdUnits

API createAdUnits – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/InventoryService
Body (see below):



	
		
			1122334455
			ZappySysApp
		
	
	
		
			
				237897516
				Ad_Unit_0
				Ad unit description #0.
				BLANK
				
					
						300
						250
					
					BROWSER
				
			
			
				237897516
				Ad_Unit_1
				Ad unit description #1.
				BLANK
				
					
						300
						250
					
					BROWSER

API createAdUnits – Response


	
		
			b244d7ba9b65e96fc42b849a7bfb16e4
			331
		
	
	
		
			
				252312396
				237897516
				false
				
					237897516
					ca-pub-2799179143725683
					ca-pub-2799179143725683
				
				Ad_Unit_0
				Ad unit description#0.
				BLANK
				ACTIVE
				253312396
				
					
						300
						250
						false
					
					BROWSER
					300x250
				
				SITE
				false
				
					
						true
						FFFFFF
						0000FF
						FFFFFF
						000000
						008000
						TEXT_AND_IMAGE
						DEFAULT
						DEFAULT
						DEFAULT
					
				
				
					
						2017
						2
						27
					
					7
					23
					32
					PST8PDT
				
				NONE
				false
				false
			
			
				252312516
				237897516
				false
				
					237897516
					ca-pub-2799179143725683
					ca-pub-2799179143725683
				
				Ad_Unit_1
				Ad unit description#2.
				BLANK
				ACTIVE
				253312516
				
					
						300
						250
						false
					
					BROWSER
					300x250
				
				SITE
				false
				
					
						true
						FFFFFF
						0000FF
						FFFFFF
						000000
						008000
						TEXT_AND_IMAGE
						DEFAULT
						DEFAULT
						DEFAULT
					
				
				
					
						2017
						2
						27
					
					7
					23
					32
					PST8PDT
				
				NONE
				false
				false
			
				252312876
				237897516
				false
				
					237897516
					ca-pub-2799179143725683
					ca-pub-2799179143725683
				
				Ad_Unit_4
				Ad unit description.
				BLANK
				ACTIVE
				253312876
				
					
						300
						250
						false
					
					BROWSER
					300x250
				
				SITE
				false
				
					
						true
						FFFFFF
						0000FF
						FFFFFF
						000000
						008000
						TEXT_AND_IMAGE
						DEFAULT
						DEFAULT
						DEFAULT
					
				
				
					
						2017
						2
						27
					
					7
					23
					32
					PST8PDT
				
				NONE
				false
				false

Loading Google DoubleClick data into SQL Server Table

So far we saw how to make simple DFP API calls without doing any parsing. Now lets look at how to use SSIS XML Source which not only make API calls but it will parse XML Response into rows and columns which can be loaded into target database such as SQL Server, Oracle, MySQL or even Flat File. XML Source also supports pagination so if you have many records it will automatically loop through all response untill all records are fetched.

Step-By-Step – Using XML Source to read Google DFP data (Parse into rows and columns)

Assuming you have tested your first DFP API Call (Explianed in the beginning of this article). You must have OAuth connection tested for API call.
Drag new Data flow task from SSIS Toolbox
Inside Data flow designer drag and drop ZS XML Source from toolbox
Double click XML Source to configure.
Set URL as below
https://ads.google.com/apis/ads/publisher/v201702/InventoryService
Check Use credentials option and from Drop down select OAuth connection manager (created in previous section)
Select Method = POST,
Select Request Content Type = XML (text/xml)

Enter Body as below (Click edit button).


   
      
         238897396
         ZappySysApp
      
   
   
      
         
            WHERE Status='ACTIVE'

Notice that in above we calling getAdUnitsByStatement API to fetch all Ad Units which are Active. Here is more information about Query Syntax for any API call getXXXXXXByStaement

Now click on Select Filter button and select results node and click OK. If prompted add array to list.
SSIS XML Source – Get data from Google DFP API using OAuth (Configure URL, Body and Filter)
Click Preview to see data. Click Columns tab to review data types (Auto detected). If you wish to change length or datatype then edit there and check Lock option (Last column)
SSIS XML Source – Preview Response for Google DoubleClick API Call – Parse into Rows/Columns
Click OK to save UI
Connect XML Source to same target (e.g. OLEDB Destination – SQL Server Connection)
Run data flow
SSIS Execution – Read from Google DFP and load into SQL Server Table

Configure Pagination for Google DFP API / PQL Query result

If you have large dataset to read from DoubleClick API ( when calling API such as getxxxxxxByStatement) then we recommend to supply LIMIT and OFFSET clause in your PQL query.

For example if you have 2000 ad units and you want to limit response size by maximum 300 rows then your query can be like below

--For first request (returns results 1-300)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 0

--For second request (returns results 301-600)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 300

--For third request  (returns results 601-900)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 600

........

This looping logic can be complex to implement if you do manually. But no worry if you are using SSIS XML Source. It comes with many pagination options (Click here to read more about pagination)

See below screenshot how to configure Pagination options for Google DoubleClick API. Basically two places you have to change. Inside body you have to set placeholder and change few settings on Pagination Tab.

SSIS XML Source – Configure Pagination for Google DoubleClick API data fetch – PQL LIMIT and OFFSET clause for Google DFP API

SSIS XML Source – Configure Pagination for Google DoubleClick API data fetch – Set Pagination Mode

Create CSV Report File and Download in GZip format (*.gz)

Sometime you have to download large amount of data (Possibly millions of rows) in that case Bulk approach would be better. DFP ReportService API allows to call following APIs which can be used to produce CSV report file in *.gz format and then you can download it using REST API Task.

Below are high level steps you have to perform to produce report file and download it.

Create the ReportJob by invoking runReportJob command .
Poll the ReportJob object using ReportService.getReportJob (This is required because file is not available right away) .
Continue to poll the ReportJob object until the reportJobStatus field is equal to COMPLETED or FAILED.
If successful, fetch the URL for downloading the report by invoking getReportDownloadURL.

Here is the sample SSIS Package to Perform this Action.

Click here to DFP API download Sample for SSIS SSIS Package (SSIS 2012, 2014, 2016)

See below screenshot how to generate Google DFP Report File and download / extract (Unzip) using Drag and Drop SSIS workflow.

Create Google DFP Report File (CSV / Gzip) and Download Using SSIS (Example of ReportService functions runReportJob, getReportJob and .getReportDownloadURL)

Call runReportJob (Start DFP Report JOB)

Very first step to produce DFP Report file is call ReportService >> runReportJob API. See below Command.

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract id of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for 111111 node anywhere in SOAP response XML and extract value (i.e. 111111 ) .. For screenshot see next section

//*[local-name() = 'id']

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxxxxAKqG89gM-TpReRXO
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 1018
Expect: 100-continue
Accept-Encoding: gzip, deflate


	
		
			238897396
			ZappySysApp
		
	
	
		
			
				
					DATE
					LINE_ITEM_ID
					LINE_ITEM_NAME
					AD_UNIT_NAME
					AD_SERVER_IMPRESSIONS
					AD_SERVER_CLICKS
					LINE_ITEM_COST_TYPE
					LINE_ITEM_GOAL_QUANTITY
					LAST_MONTH

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:41 GMT
Expires: Fri, 02 Jun 2017 21:23:41 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 999


	
		
			f03e7e474fde2a5f9c5d7c9d7eb83923
			846
		
	
	
		
			
				10000297829
				
					DATE
					LINE_ITEM_ID
					LINE_ITEM_NAME
					AD_UNIT_NAME
					TOP_LEVEL
					AD_SERVER_IMPRESSIONS
					AD_SERVER_CLICKS
					LINE_ITEM_COST_TYPE
					LINE_ITEM_GOAL_QUANTITY
					LAST_MONTH
					false
					false

Call getReportJob (Poll JOB Status)

Second step is to make sure is Report Status check. For large report it make take several seconds or minutes before you can get Download URL (See Next Step). If you try to call Next step before Report is ready then you may get error. So better to add Polling logic.

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract status of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for IN_PROGRESS node anywhere in SOAP response XML and extract value (i.e. IN_PROGRESS )

//*[local-name() = 'rval']

Extract Single XML Node Value from SOAP API Response – Use XPATH in SSIS

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxXmWqmq
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 524
Expect: 100-continue
Accept-Encoding: gzip, deflate


   
      
         238897396
         ZappySysApp
      
   
   
      
       10000297829

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:47 GMT
Expires: Fri, 02 Jun 2017 21:23:47 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 451


	
		
			d52cb3a3d68af24aec2f9b0284379700
			803
		
	
	
		
			COMPLETED

Call getReportDownloadURL (Get Download URL)

Once you get status = COMPLETED in above API Call (i.e. getReportJobStatus) then you ready to call getReportDownloadURL to fetch URL which you can download.

Here is the API sample Request and Response

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxBp84ltTNyVRb9hlUEWx
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 581
Expect: 100-continue
Accept-Encoding: gzip, deflate


   
      
         238897396
         ZappySysApp
      
   
   
      
       10000297829
       CSV_DUMP

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:49 GMT
Expires: Fri, 02 Jun 2017 21:23:49 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 1039


	
		
			8e5ac0bbeb8e88c180ea442fc5378d99
			1957
		
	
	
		
			https://storage.googleapis.com/dfp-report-export/caxxxxxxxx-xxxxxxxxxxxxxxb?GoogleAccessId=3769xxxxxxxxxxx-qxxxxxxxxxxxxxxxxsqdb5dvi8g@developer.gserviceaccount.com&Expires=1496438929&Signature=vnhCiAIqd%2BFgWCtAHo9roFOI3W%2BveUTYuSvyBQ9gtMI2HDTXOCVDh7wzUV%2BPBJZZghaL6k8wKYq33GA4NpjZVNYuaAgPLWgGwAUPdtioHRi%2Bt3Eymp%2FAPRGIoos0ekP3O7l%2FPNPxenC6UTP087HV3j06V%2Fuui9ixpKZWGCK8r56oUPj6S8q1hG95Ns3d3sEcuz3%2BkShu6GYG5mqir5vjtZPuzwsr%2F96G5Kk123mke%2B%2FIVRJGoqnCMghNWtIG1yR1IdZx%2FIpit%2FnaJ9HP6IvAr3oMIXkQ3cHv%2FtOhXGy9dTG34q%2FKDbtkSJAayHgjlSLPGKucygcXBhUrbywBu8Y77A%3D%3D

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract status of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for https://storage.googleapis.com/dfp-report-export/caxx………. node anywhere in SOAP response XML and extract value (i.e. URL)

Download File Using REST API Task

Once you get URL its time to download it. You can use REST API Task to download the file. On Response Setting Tab specify full file path (e.g. c:\report_csv.gz) and make sure Binary option is checked (Found next to the Save Path)

Making things dynamic

If you want to supply certain parameters at runtime (e.g. from SSIS Variable) rather than hard code then you can use Variable Placeholders any where in Request Body, URL or Headers. Click Insert Variable option found in Edit screen

Here is an example of dynamic URL (API Version stored in SSIS Variable)

https://ads.google.com/apis/ads/publisher/{{User::ApiVer}}/NetworkService

Deployment to Production

When you ready to deploy your SSIS Package to production make sure to change Test NetworkCode supplied inside Request Body to your Production Ad Network. Also Set RefreshToken Property of OAuth Connection Manager (Get it from Backup file saved when you created OAuth connection first time). Once you set RefreshToken of connection manager you won’t have to Login again to get a new token.

Conclusion

Traditionally if you wanted to integrate Google DFP API (DoubleClick API) calls inside your ETL workflow then it required some sort of coding effort (e.g. Use C# SDK, JAVA SDK or Python SDK). Coding effort is not only time consuming and expensive but also hard to maintain for any non-coder (e.g. ETL Developer or DBA).
Thanks to SSIS PowerPack which changed the way developers do API integration. Now you can reduce time to implement and total cost of you API integration project significantly by using drag and drop approach. This approach is not only easy to maintain but also fast and high quality. You can use ZappySys REST/SOAP SSIS Connectors and SSIS Tasks for any API integration project such as Google DFP API. Try SSIS PowerPack for free to explore possibilities. If you have any issue with your integration Contact Support

The post How to call Google DFP API with SSIS – DoubleClick appeared first on ZappySys Blog.

How to call eBay REST API in SQL Server with SSIS

ZappySys — Tue, 07 Feb 2017 03:36:07 +0000

Introduction

eBay recently launched brand new REST API with OAuth 2.0 support. This article will explain you how to call eBay REST API using SSIS to automate eBay operations (e.g. Buy/Sell) using SSIS REST API Integration Connectors / OAuth Connection Manager.

This article assumes that you have basic knowledge of SSIS (SQL Server Integration Services). We will use drag and drop approach without coding to do eBay REST API integration. We will use following custom tasks/connectors from SSIS PowerPack

SSIS PowerPack REST API Connectors can be used to consume/manage eBay data using API calls. Following connectors will be used in this article.

	REST API Task
	OAuth Connection Manager
	JSON Source (File, REST, OData)

Test Ebay API using Online API Testing Tool

Before you can dig deep down we suggest you to try few API calls using ebay online testing tool. Here is an example api call using their Online Testing Tool.

For example visit below URL to try sample Search API call.

https://developer.ebay.com/my/api_test_tool?index=0&env=production&api=browse&call=item_summary_search__GET&variation=json

Testing Ebay REST API using Online Tool

Call eBay API using SSIS – Create new inventory item

Now lets look at how to make simple eBay REST API call using SSIS. For demo purpose we will use Sandbox API rather than production.

First download and install SSIS PowerPack from here.
Create new SSIS Project and open SSIS Package
Drag new ZS REST API Task from SSIS Toolbox on control flow designer
Rename ZS REST API Task to Create eBay inventory item-1. Double click the task to configure it.
Select Request mode to [Url from Connection]
Select ZS-OAUTH connection type from connection dropdown.
On OAuth connection. Select eBay Sandbox from provider dropdown.
Enter App ID in Client ID field (Obtained from here)
Enter Cert ID in Client Secret field (Obtained from here)
Select desired Scopes or leave it blank to use default scopes. For this example we need following two scopes
https://api.ebay.com/oauth/api_scope/sell.inventory
https://api.ebay.com/oauth/api_scope/sell.inventory.readonly
Click on Advanced Tab. Enter RuName in the Return URL field. Don’t worry if your RuName doesn’t look like URL 🙂 (Click here to learn how to get RuName – Scroll to Getting the RuName value)
Now click on first tab again. Click Generate token button. When prompted login using Sandbox User Account (Click here to create test user for Sandbox).
NOTE: Production account or developer account wont work when you login for eBay Sandbox API Token
Click Accept button after login. If everything goes well you will see two tokens populated on your main screen (Access Token and Refresh Token). You will get prompt to save tokens to some backup file… Do that so same tokens can be used if you wish to deploy JOB to another machine.
Click Test to validate connection. Click OK to save.
SSIS OAuth Connection Manager – Connect to eBay Sandbox API
Now once you close connection you are back to REST API Task
Enter following URL to create or new eBay inventory item using PUT. Last part of following URL is ID we want to create
```
https://api.sandbox.ebay.com/sell/inventory/v1/inventory_item/GP-Cam-01
```
Select Method PUT from dropdown

Click Edit for Request Body. Enter following data for our sample product

{
	"availability" : {
		"shipToLocationAvailability" : {
			"quantity" : 50
		}
	},
	"condition" : "NEW",
	"product" : {
		"title" : "GoPro Hero4 Helmet Cam",
		"description" : "New GoPro Hero4 Helmet Cam. Unopened box.",
		"aspects" : {
			"Brand" : ["GoPro"],
			"Type" : ["Helmet/Action"],
			"Storage Type" : ["Removable"],
			"Recording Definition" : ["High Definition"],
			"Media Format" : ["Flash Drive (SSD)"],
			"Optical Zoom" : ["10x"]
		},
		"imageUrls" : [
			"http://i.ebayimg.com/images/i/182196556219-0-1/s-l1000.jpg",
			"http://i.ebayimg.com/images/i/182196556219-0-1/s-l1001.jpg",
			"http://i.ebayimg.com/images/i/182196556219-0-1/s-l1002.jpg"
		]
	}
}

Change Body content type to JSON from dropdown
Click on [Raw Edit] button in Headers grid toolbar. Enter following headers and click OK.
```
Content-Language: en-US
Accept: application/json
```
Click Test Request/Response – If you get success that means product got created.
You can create more products by simply changing SKU in URL and name/description in Body. Click Test to execute API call.

Read data from eBay and load into SQL Server using SSIS JSON Source

Now lets look at how to consume eBay REST API and load data into SQL Server. For this you can use JSON Source (File, REST API). For this demo we will Call eBay Search API . If you want to explore other API then check this page. Here is

Go to Data flow tab
Drag and drop ZS JSON Source from SSIS Toolbox
Double click to configure it.
Enter API URL like below. We added Search parameter and limit parameter (Max rows per response). Refer to API Help page to know more about other available parameters.
```
https://api.sandbox.ebay.com/buy/browse/v1/item_summary/search?q=phone&limit=100
```
Check Use Credentials option.
Select ZS-OAUTH connection or Create New (See previous section to create new one)
Select Filter (Look for Array Icon and select that node). For example if you calling search API then enter or select
```
$.itemSummaries[*]
```
like below.
SSIS JSON Source – Get data from eBay REST API (Use OAuth Connection)
Now click on pagination tab and enter
```
$.next
```
expression for next link attribute as below. This will enable pagination (It will keep reading in 100 rows per page until last page reached – Empty recordset)
Ebay REST API Pagination (Search API Example)
That’s it, now click Preview to see data
Once all looking good you can click OK to save JSON Source.
In Next Section we will see how to load Ebay Data into SQL Server.

Loading eBay data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

Drag and drop the Upsert Destination component from the SSIS Toolbox.
Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

Double-click the Upsert Destination component to open the configuration window.
Under Connection, select an existing target connection or click NEW to create a new connection.
- Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

In the Target Table dropdown, select the table where you want to load data.
Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

Go to the Mappings tab.
Click Auto Map to map source columns to target columns by name.
Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Pagination for eBay REST API Calls

In previous example we saw REST API Pagination using Next Link approach but some API may not use same approach in that case refer to this link. We looked at Next Link approach in previous section, you can also use Method #2 (URL parameter – Offset) but Next Link approach is more simpler.

Calling eBay XML format API – Finding / Trading / Shopping API

eBay converted most of their APIs to JSON format to use OAuth 2.0 standard but there few APIs still there which are not OAuth 2.0 standard. These includes eBay Trading API, eBay Finding API, eBay Shopping API. They all are still in XML format. Below section describe how to call these API. Use eBay API Test tool here to generate desired Request Body and Headers.

Now let’s look at an example to call eBay XML API (GetItem from Trading API). To support special case like eBay API ZappySys introduced OAuth connection property called CustomAuthHeader in v2.6.7 or higher (Only found in Properties grid view).

Right click on OAuth connection manager icon > Click Properties > Set CustomAuthHeader to X-EBAY-API-IAF-TOKEN
Now drag ZS REST API Task from Control flow SSIS Toolbox
Set Properties like below
1. Set URL as below (Sandbox Url is different than Production)
```
https://api.sandbox.ebay.com/ws/api.dll
```
2. Set Body As (this was obtained from eBay API Test tool here )
```
	en_US
	High
      
  1234
```
3. Set Headers as below (Click on Raw Edit Button and paste below heades). Again these header were also copied from eBay API Test tool here . Notice that API function name is included in the header. Also site id and compatibility level required.
```
X-EBAY-API-CALL-NAME: GetItem
X-EBAY-API-SITEID: 0
X-EBAY-API-COMPATIBILITY-LEVEL: 967
```
4. Now click OK to get response. If things working you will see some valid XML.
  Call eBay XML API using SSIS REST API Task (eBay Trading API, Shopping API, Finding API using OAuth Token)

User Token vs Application Token

Any time you call eBay API you have to supply token. eBay OAuth supports two types of token. In this article we used User Token. If you wish to use Application Token then you have to use OAuth Client Credentials grant. See this article

Difference between eBay Application Token vs User Token

Application Token
– Do not require user login
– Use this token if you need to perform basic operations which are supported by following permission https://api.ebay.com/oauth/api_scope
User Token
– Requires one time user login to create token and then refreshes automatically without login prompt (if you using SSIS PowerPack)
– Use this token if you need higher privileges behalf of actual use for update/write data using API call

Use Application token when you want to perform readonly operations and you don’t need higher privileges (such as admin access). Certain eBay API calls only possible if you supply user token.

POST data to Ebay (Insert or Update)

In previous section we have seen how to read data from ebay. Now let’s look at how to write data to ebay.

There are two ways you can achieve this.

Use Web API Destination in Data Flow
Use REST API Task in Control Flow

Read this article to learn more about API POST using SSIS

Using SSIS Web API Task (Write data from SQL Server to ebay)

Using REST API Task to POST data to ebay

If you have JSON/XML data already prepared and if you like to POST it to ebay API then REST API Task would be easy to use.

REST API Task Body can be direct string, variable (e.g. {{User::varSomeData}} or Body can come from File

SSIS REST API Task – POST data to Xero (Create contacts)

" data-medium-file="https://i2.wp.com/zappysys.com/blog/wp-content/uploads/2017/07/ssis-post-data-xero-api-create-contacts-example.png?fit=300%2C210" data-large-file="https://i2.wp.com/zappysys.com/blog/wp-content/uploads/2017/07/ssis-post-data-xero-api-create-contacts-example.png?fit=720%2C503" />

SSIS REST API Task – POST data to ebay (Create contacts)

Common Errors

Truncation related error

The most common error you may face when you run an SSIS package is truncation error. During the design time only 300 rows are scanned from a source (a file or a REST API call response) to detect datatypes but at runtime, it is likely you will retrieve far more records. So it is possible that you will get longer strings than initially expected. For detailed instructions on how to fix common metadata related errors read an article "How to handle SSIS errors (truncation, metadata issues)".

Authentication related error

Another frequent error you may get is an authentication error, which happens when you deploy/copy a package to another machine and run it there. Check the paragraph below to see why it happens and how to solve this problem.

Deployment to Production

In SSIS package sensitive data such as tokens and passwords are by default encrypted by SSIS with your Windows account which you use to create a package. So SSIS will fail to decrypt tokens/passwords when you run it from another machine using another Windows account. To circumvent this when you are creating an SSIS package which uses authentication components (e.g. an OAuth Connection Manager or an HTTP Connection Manager with credentials, etc.), consider using parameters/variables to pass tokens/passwords. In this way, you won’t face authentication related errors when a package is deployed to a production server.

Check our article on how to configure packages with sensitive data on your production or development server.

Conclusion

REST API integration typically requires coding and good understanding of protocol such as OAuth. SSIS PowerPack REST API connectors and OAuth Connection provides easy to use interface with many options to integrate virtually any API including eBay REST API inside your SQL Server Environment without any coding. Try SSIS PowerPack yourself see what is possible.

The post How to call eBay REST API in SQL Server with SSIS appeared first on ZappySys Blog.

How to call Elasticsearch API using SSIS

ZappySys — Tue, 17 Jan 2017 00:22:47 +0000

Introduction

UPDATE: ZappySys has released a brand new API Connector for ElasticSearch which makes it much simpler to Read/Write ElasticSearch Data in SSIS compared to the steps listed in this article. You can still use steps from this article but if you are new to API or want to avoid learning curve with API then use newer approach.

Please visit this page to see all Pre-Configured ready to use API connectors which you can use in SSIS API Source / SSIS API Destination OR API ODBC Driver (for non-SSIS Apps such as Excel, Power BI, Informatica).

Elasticsearch is a powerful engine that allows you to store, aggregate and, most importantly, search data in a very analytical way. In this tutorial, you will learn how to bulk load data from SQL Server to Elasticsearch with SSIS (part of SQL Server) and ZappySys PowerPack.

The scope of this article will be to show how to import records from SQL Server into Elasticsearch index as JSON documents using Elasticsearch Bulk API. In a nutshell, we will retrieve IDs and names of the products from a SQL table, transform each row into a JSON and lastly, index each JSON in Elasticsearch under record’s corresponding ID. This tutorial is going to be your stepping stone to use any Elasticsearch API as a destination.

We will be using these SSIS components of PowerPack to make things work:

	JSON Generator Transform
	Web API Destination

Let’s begin?

Prerequisites

SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
Northwind database deployed on your machine (we will use it to load data from; download it).
Elasticsearch instance up and running.
ZappySys SSIS PowerPack installed.

NOTE: If your ElasticSearch instance is hosted as AWS Managed ElasticSearch Instance then select

. rather than ZS-HTTP (Explained later in this article). When OAuth UI launches select AWS v4 Provider. For more information on calling REST API on AWS check this article.

Step-by-Step – Bulk loading data from SQL Server to Elasticsearch with SSIS

Load data from SQL Server database first

In this section you will perform several basic steps to start the package: load data from SQL Server and prepare it for further steps.

Create a new SSIS package and drag a Data Flow Task into the Control Flow from the SSIS Toolbox.
Dragging and dropping Data Flow Task into Control Flow
Open Data Flow Task and then add OLE DB Source.
Configure OLE DB Source to take data from Products table, Northwind database.
Choose ProductID and ProductName as columns you will use.

Load data to Elasticsearch using SSIS and ZappySys Web API Destination component.

Transform data into JSON documents

To add data into Elasticsearch firstly we need to prepare it with JSON Generator Transform. It will help us convert table rows into JSON documents.

JSON String to Create Index data for the Bulk API call

Close the window and then drag Derived Column component from the SSIS Toolbox.
Add two new columns and name them
```
RowHeader
```
and
```
RowFooter
```
:
Derived Column configuration to add prefix and header and footer to a JSON

In the Expression column give them values:
- RowHeader:
```
"{ \"index\": { \"_index\": \"shop\", \"_type\": \"products\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n"
```
- RowFooter:
```
"\n"
```
Now drag JSON Generator Transform from the SSIS Toolbox and open it.
Right mouse click on item Mappings and select
```
Add Element(s) (Below this node)
```
:
Convert SQL table rows into JSON using JSON Generator Transform
Once a dialog appears, select
```
ProductName
```
as Source Column and
```
name
```
as Output Alias:

Map table column to JSON property using JSON Generator Transform
You should get a similar view:
JSON Generator Transform configuration to create JSON from a SQL table
Then go to Header / Footer tab and:
- Select
```
Direct String
```
  option.
- Input
```
<%RowHeader%>
```
  in Header String option.
- Input
```
<%RowFooter%>
```
  in Footer String option.
JSON Generator Transform configuration to convert rows into JSON suitable for Elasticsearch Bulk operation
Close the window.

JSON String to Update Index data for the Bulk API call

From the above section “JSON String to Create Index data for the Bulk API call” follows all the steps From Step-1 to Step-8.

Just for (2nd step) Step-2 use below expression for

RowHeader

and

RowFooter

Derived Column configuration to add prefix and header and footer to a JSONIn the Expression column give them values:

RowHeader:

"{ \"update\": { \"_index\": \"shop\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n{ \"doc\" : "

RowFooter:
```
"}\n"
```

NOTE: To learn how to make more complicated transforms visit these links:

JSON String to Delete Index data for the Bulk API call

From the above section “JSON String to Create Index data for the Bulk API call” follows steps Step-1 and Step-2.

NOTE: No need to use JSON Generator Transform (From Step-3).

Just for (2nd step) Step-2 use below expression for

RowHeader

Derived Column configuration to add prefix and header and footer to a JSON

In the Expression column give them values:

RowHeader:

"{ \"delete\": { \"_index\": \"shop\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n"

Finally, Elasticsearch & SQL Server integration

Now drag Web API Destination SSIS component into the Data Flow and open it for editing.
In Select Connection section press
.
Once HTTP Connection Manager window opens configure connection to your Elasticsearch instance:
- Set Web Url, which points to your Elasticsearch instance.
- Set Credentials Type to
```
Basic - UserID/Password
```
  (or other appropriate authentication method).
- Finally, set User Name and Password:
  
  Configure SSIS HTTP Connection to connect to Elasticsearch
Close configuration window. Now it’s time to configure Web API Destination.
All that is separating you from getting those rows into Elasticsearch:
- Setting Input Column for Body to
```
ZS_JSON_OUT
```
  for Create and Update Index Data and for Delete Index Data Input Column for Body to
```
RowHeader [Derived Column]
```
  .
- Setting URL to
```
http://localhost:9200/_bulk
```
  .
- Setting HTTP Request Method to
```
POST
```
  .
- Setting Body content type Method to
```
JSON(application/json)
```

Index SQL Server Database data in Elasticsearch using SSIS component Web API Destination

Then open Batch Settings (For Body) tab:
- Enable batch submission.
- Set Body Batch Size, e.g. 1000. Make sure this number is even, otherwise you may get into problems (Elasticsearch Bulk request has special JSON request body format, as you perhaps noticed).
Web API Destination batch settings configuration
Close the window and run the package! You should see green lights, telling you everything is OK:
Successful data load from SQL Server to Elasticsearch
We can also query Elasticsearch by using one of its Search APIs – URI Search to see if we successfully indexed data:
SQL Server data index in Elasticsearch
Rows number match in SSIS and in search results, thus everything is good. You are done.

What if I want more? After upserting data use Web API Destination further

Let’s say you have a requirement to do something with rows that were freshly indexed in Elasticsearch. Then you will need somehow to distinguish between created and updated records in Elasticsearch. And that won’t be difficult because Web API Destination acts not only as Destination but as Transformation as well. So one thing you have to do is connect JSON Parser Transform downstream to Web API Destination. It will parse Elasticsearch HTTP JSON response – which is returned by Web API Destination – into columns, which you can later easily redirect using Conditional Split:

Add JSON Parser Transform and connect it to Web API Destination:
Use JSON Parser Transform to get Elasticsearch HTTP JSON response when integrating SQL Server & Elasticsearch
Make sure you have selected
```
ResponseText
```
as Select Input JSON Column.
Then go to Bulk API and copy/paste “the result of this bulk operation” JSON response:
Using JSON Parser Transform to parse JSON response from REST HTTP request
Set
```
$.items[*]
```
as the filter.
Optional step. You won’t find a response sample for every Elasticsearch API call. In that case, you may want to use Postman to make an HTTP request to Elasticsearch and get a sample response:
Use Postman to make a request to Elasticsearch and get a sample response to be used in JSON Parser Transform

NOTE: Don’t forget to set the username and password in Authorization section!

Then use a Conditional Split and Trash Destination to redirect the rows:

Input these clauses:

Inserted:

REPLACENULL([index.result],"") == "created" || REPLACENULL([create.result],"") == "created"

Updated:

REPLACENULL([index.result],"") == "updated" || REPLACENULL([create.result],"") == "updated"

As a result, new index records will be redirected to one destination, while updated records – to the other:

NOTE: Trash Destination is a handy component of ZappySys PowerPack which can be used as dummy destination when we don’t care about the destination (and we don’t care in this tutorial) 🙂

Overall you can use Web API Destination HTTP JSON response for other useful things as well, e.g. determine on how many replica shards record was indexed. JSON response will depend on which Elasticsearch API and which method you use.

Delete Index by making an API call.

If you want to delete the index by making Delete API call you can. Let’s make that call using the Rest API Task. Configure it like below screen and click on the Test Request button.

Rest API Delete Method

Things gone bad: Error handling & debugging

Incidentally, you may incorrectly construct JSON for Web API Destination body. Elasticsearch nodes may go offline or go out of memory. In any case you may want to know when that happens and take actions accordingly. For that purpose you have to redirect failed requests from Web API Destination to some other destination:

Add a Derived Column above Web API Destination with expression
```
(DT_WSTR,4000)ZS_JSON_OUT
```
and name it
```
JsonAsString
```
. This will let you see what JSON you are actually passing.
Then add database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don’t forget to set
```
Redirect row
```
option for both, Error and Truncation columns:
Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
Use Data Viewer to view HTTP requests that failed to be fulfilled in Elasticsearch

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Call ElasticSearch API hosted on AWS

If your ElasticSearch instance is hosted as AWS Managed ElasticSearch Instance then select

. rather than ZS-HTTP (Explained later in this article). When OAuth UI launches select AWS v4 Provider. For more information on calling REST API on AWS check this article.

Configure SSIS OAuth Connection – Use Amazon AWS API Provider, Enter Access Key, Secret Key

Call AWS Hosted ElasticSearch REST API in SSIS (V4 Request Signing)

Common Errors

Truncation related error

Authentication related error

Deployment to Production

Check our article on how to configure packages with sensitive data on your production or development server.

Download a sample package

Be sure to download a sample SQL Server 2008 SSIS package, in case you want to try it right away (you can upgrade it to a higher version).

NOTE: Once you open the package, you won’t be able to run it immediately, but don’t panic. Just configure OLE DB Source to point to your Northwind database and set URLs to point to your Elasticsearch instance.

Conclusion. What’s next?

In this article we have learned how to bulk load data from SQL Server and upsert it in Elasticsearch index. We used JSON Generator Transform to help us convert database table rows into JSON documents. In addition, Web API Destination was very helpful in automatically creating and making HTTP requests to Elasticsearch instance and indexing our data as a result. This tutorial was quite simple and straightforward, yet it is a good starter to use other Elasticsearch APIs and their features. From now on Elasticsearch reference is your best friend in learning what APIs exist, what are their methods and how HTTP requests should look like.

Keywords

How to import data from SQL Server to Elasticsearch | How to load data into Elasticsearch | Using Elasticsearch to index SQL Server | Elasticsearch and SQL Server integration | How to use Elasticsearch together with SQL Server | Upsert SQL Server data into Elasticsearch

The post How to call Elasticsearch API using SSIS appeared first on ZappySys Blog.

Using SSIS Regex Parser Task for Extracting HTML Content

ZappySys — Mon, 26 Dec 2016 17:09:57 +0000

Introduction

In this post you will learn how to use FREE SSIS Regex Parser Task along with REST API Task to extract HTML content in few clicks.

Scenario

Assume that you want to search certain keywords from Bing or google and want to know how many pages found for that keyword. Url for search would be something like http://www.bing.com/search?q=regex where regex is our search word.

When page is returned view source code of that page and you will find tag like below.

21,00,000 results

What we want is number 21,00,000 using Regular expression pattern search.

Step-By-Step : Extract HTML Tag value using Regex Expression

Download and Install SSIS PowerPack (It includes FREE SSIS Regex Parser Task )
Create new SSIS Package
Drag ZS REST API Task on Control flow designer from SSIS Toolbox
Double click to configure the task. Enter URL you like to fetch e.g. http://www.bing.com/search?q=regex
Click on Response Tab and check Save response option. Select Save to Variable. If needed create new variable.
Click Test (Scroll at the bottom to see html content)
Now drag ZS Regex Parser Task and connect with REST API Task
Select Variable which will hold HTML text you like to parse.
Enter following expression and map target to some Variable if you like to save extracted value. Below expression ends with {{0,1}} which means extract first match and 2nd group of that match (0 based Index). 2nd group of match will hold actual count of search result. If you omit {{x,y}} at the end then {{0,0}} is used.
```
\\s*(?[0-9,.]*){{0,1}}
```
See below screenshot

SSIS Regex Parser Task – Extract HTML Tag Value using Regular Expression
In the above step you can select Variable as Input or use placeholder in Direct string (e.g {{Use::varHtml}} )
You can also connect ZS Logging task to show extracted value

Here is final flow.

SSIS Regular expression parsing example

The post Using SSIS Regex Parser Task for Extracting HTML Content appeared first on ZappySys Blog.

Convert CURL to SSIS PowerPack / ODBC for API Call

ZappySys — Thu, 10 Nov 2016 19:35:43 +0000

Introduction

Often, REST API documentation uses example syntax for the curl command-line tool because it’s one of the most popular tools for making API calls. However, suppose you are an SSIS developer using ZappySys API Drivers/Connectors from either the SSIS PowerPack or the ODBC PowerPack. In that case, this blog post will help you translate the curl command line syntax to similar options available in the ZappySys Product. For SSIS PowerPack products, you can refer to Connectors/Tasks such as ZappySys REST API Task / JSON Source Connector / XML Source. To make this article short and simple, we will show many screenshots from SSIS PowerPack UI, but the concepts are the same when you use other products, such as ODBC PowerPack JSON Driver / REST or XML Driver.

If you are using Postman to call the API, then refer to this article

If you are using Fiddler to call an API, then refer to this article

CheatSheet – Curl to SSIS PowerPack / ODBC PowerPack

Before we dig down more. Let’s look at how each command-line parameter from CURL maps to SSIS PowerPack settings.

CURL Parameter (Short / Long)	Purpose	SSIS PowerPack Setting
method	Pass HTTP method (e.g., GET, POST, PUT, DELETE)	Select Method from Request Method Dropdown on ZappySys REST API Task or JSON Source Connector, or XML Source, like this
-u userid:password, –user userid:password	Authenticate using Basic Authentication using UserID and Password	Use the HTTP Connection Manager to enable Basic Auth, then supply the username and password.
-H “name: value”, –header “name: value”	Supply HTTP Header(s)	Click on the Raw Edit option or enter key-value pairs of HTTP headers in the Headers grid like this
-d “key=value”, –data “key=value”	Supply BODY parameters using www-form-urlencoded Content Type	Click Edit next to the Request Data text box and supply Body. If you need to pass content in JSON / XML, then select the Request ContentType dropdown like this
-d “some data”, –data “some data” -d @filepath, –data @filepath	Supply BODY parameters using Content-Type other than www-form-urlencoded. You can also supply content from a file.	Click Edit next to the Request Data text box and supply Body. If you need to pass content in JSON / XML, select the Request ContentType dropdown as shown. To upload content stored from a local file, use this option.
-F key=value, –form key=value	Supply Content-Type multipart/form-data according to RFC 2388	Enable the Multipart option for Request on the ZappySys UI, like this
-k, –insecure	Ignore SSL Certificate-related errors (e.g., allow expired certificate or untrusted certificate)	Go to the Advanced Tab and check the ” ignore SSL Related Errors option like this.
-x, –proxy [protocol://]host[:port]	Specify Proxy Settings	You can enable System default proxy or a custom proxy using the Proxy Tab in the OAuth / HTTP Connection Manager. Check this.

Convert Curl Syntax to SSIS PowerPack – REST API Task / JSON / XML Connector

Take a look at the following example of CURL syntax

curl POST -u myusername:mypassword -H "Accept: application/json" -H "AppVersion:1.0" "https://mysite/v1/api/customers/" -d "customerid=AAA&OrderDate=2016-01-01"

Where :

-u

--user

= UserID and Password for server (This option sends credentials in BASIC Authentication mode)

-H

--header

= HTTP Header (e.g. Key: Value)

-d

--data

= Data for POST

For step-by-step information, check Article Learn how to POST data to a URL using SSIS

Here is the screenshot of HTTP POST in JSON format using the ZappySys REST API Task

Convert CURL Command Line Syntax to SSIS – Perform HTTP POST operation using SSIS REST API Task – Send data in JSON content type

If you wish to POST data and parse the JSON response into rows, you can use the SSIS JSON Source. JSON Source has a powerful JSON Parser that parses large JSON (stored in a file, an API URL, or a Direct string) into Rows and Columns. This is useful if you wish to save the response into a target such as SQL Server or a CSV file.

JSON Source example

HTTP Headers – Convert Curl Syntax to SSIS

Set up HTTP headers in ZappySys Task/Connect, or click on Raw Edit or Add/Edit items in Grid.

HTTP Credentials – Convert Curl Syntax to SSIS

Here is how you pass credentials in the curl command line.

curl.exe https://myserver.com/api -u myusername:mypassword

To setup HTTP credentials in SSIS for Basic / NTFS or Kerberos use HTTP Connection Manager (Access Mode must be changed to Url from Connection on ZappySys REST API Task and for JSON Source Connector or XML Source check Use Connection Option)

If you have OAuth 2.0 Credentials (Application ID, Application Secret) then use SSIS OAuth Connection Manager rather than HTTP Connection

Check this article to learn about how to pass HTTP Basic Credentials in SSIS PowerPack

Proxy Setting – Convert Curl Syntax to SSIS

Here is how you pass Proxy server info in cUrl command line (

-x

along with Proxy Server Host/IP and Port)

curl.exe -x 192.168.2.1:8888 https://myserver.com/api

Convert Curl Syntax to SSIS – Client Certificate Settings

HTTP Connection Manager UI

Passing client certificate along with Web request (i.e. pfx/pkcs12/p12)

Passing client certificate from file path (i.e. pfx/pkcs12/p12)

The post Convert CURL to SSIS PowerPack / ODBC for API Call appeared first on ZappySys Blog.

Call Semantics3 REST API in SSIS use OAuth1

ZappySys — Sat, 15 Oct 2016 02:00:27 +0000

Introduction

In this post you will learn how to call Sementics3 REST API using SSIS PowerPack.

Using drag and drop approach you can consume data from Semantics3 REST API service. In this post we will use ZappySys JSON Source to read data from Amazon MWS API and load into SQL Server. We will also use REST API Task to call any API from Marketplace Web service and save output into Variable.

Assumption

This post assumes following things

You have basic knowledge of SSIS and JSON format.
You have registered for Semantics3 API Access. If not then it takes less than 5 minutes to register for free here to get started.
You have tested few API calls using Semantics3 REST API Playground this will give you idea about various API you can call and parameters you need to pass.

Step-By-Step – Call Semantics3 REST API using SSIS

To consume data from any JSON Source (File or REST API) you can use ZappySys JSON Source. or REST API Task. In below example you will see how to call Semantics3 REST API and save output into SSIS Variable. With this approach you can call any API (GET / POST / DELETE / PUT)

Download and Install SSIS PowerPack
Create test package
From SSIS toolbox drag ZS REST API Task and double click on the task to configure it
Select Request URL Access Mode = [Url from Connection]
Enter URL as below
```
https://api.semantics3.com/v1/products?q=%7b%22search%22%3a%22iphone%22%7d
```
In above URL
** q=%7b%22search%22%3a%22iphone%22%7d => This is your search query in URL encoded format. Data without URL encode may look like this q={“search” : “iphone” }
** If you have stored query in plain text (not encoded) in SSIS variable then you can use URLENC format specifier like this q={{User::varQ,URLENC}} . This will automatically encode data for you.
In Select Url Connection dropdown select New OAUTH connection option.
When prompted on OAuth connection UI select Provider=Custom, Enter ApiKey (i.e. Client ID) and Secret (i.e. Client Secret). If you not sure what is this then check your Developer console > Applications Tab. Select OAuth1 from Version dropdown.
Semantics3 API Call using SSIS OAuth1 Connection Manager
Click OK to save UI. Do not click test because it may not work yet.
Now click Test and see content in the Preview window.
Call Semantics3 REST API using SSIS REST API Task
If you wish to save response data into SSIS variable then goto response Tab and check save option and select variable name.

Load Semantics3 data into SQL Server using SSIS (REST API Call)

In this section we will see how to use ZappySys JSON Source to read data from Semantics3 REST API and load into SQL Server

For making things simple we are calling

Download and Install SSIS PowerPack
Create test package
From SSIS toolbox drag Data Flow task and double click task to go to Data Flow designer
Drag ZS JSON Source from SSIS Toolbox
Configure JSON Source as below. You can use Select Filter option to select hierarchy (JSON Array). Then click Preview to see data.
Get data from Semantics3 API using SSIS JSON Source
Drag OLEDB destination from SSIS Toolbox.
Connect JSON Source to OLEDB Destination and map input column to target table
Execute dataflow to load Semantics3 Data into SQL Server (or any destination such as File, Oracle, MySQL)
Read from Semantics3 REST API Call – Load into SQL Server using SSIS

Conclusion

Semantics3 API provides great way to automate many functionality for searching and ordering millions of products into your custom App. Using ZappySys SSIS PowerPack you can consume data from any REST API or Web service without replying on SDK / coding approach (e.g. C#, Java, Python, Ruby).

The post Call Semantics3 REST API in SSIS use OAuth1 appeared first on ZappySys Blog.

How to read / write Google SpreadSheet using SSIS

ZappySys — Sat, 24 Sep 2016 23:21:11 +0000

Introduction

UPDATE: ZappySys has released a brand new API Connector for Google Sheets Online which makes it much simpler to Read/Write Google Sheets Data in SSIS compared to the steps listed in this article. You can still use steps from this article but if you are new to API or want to avoid learning curve with API then use newer approach.

In this post you will learn how to get data from Google SpreadSheet (REST API) using SSIS. We will use drag and drop REST API connectors from SSIS PowerPack. No need to download any SDK or learn programming language (e.g. JAVA, C#, Ruby, Python) when you use SSIS PowerPack Connectors. We will use Google Drive API and Google Sheets API to get file list and export SpreadSheet as CSV file in few clicks (Using OAuth 2.0 connection in SSIS).

In this tutorial we will use REST API Task to call some ad-hoc API (e.g. get File List from Google Drive) and save output into Variable or File. We will use OAuth connection along with ZappySys JSON Source connector to read data from Google SpreadSheet (Use Drive API) and load into SQL Server (Export Google SpreadSheet to CSV).

Create Google API Project

First step to access any Google API is create an API Project in Google Console. If you don’t want to go through this then Skip this Step-1 and in the next section select Default OAuth App option on OAth Connection Manager (This is the most easiest option for now unless you want to use your own OAuth App).

Check this article for step-by-step instructions. When you follow these instructions make sure you enable Google Drive API (In the article it shows how to enable YouTube API as an example but you will need to enable Drive API for this article).

Create OAuth Connection Manager in SSIS

Once you create Google API project and obtained Client ID and Client Secret your next step is to create OAuth Connection Manager in SSIS. ZappySys OAuth connection manager comes with many predefined OAuth Providers (e.g. Facebook, Twitter, Google etc) but you can also define custom OAuth settings for any OAuth enabled API.

To create SSIS OAuth 2.0 Connection for Google API perform following steps.

Download and Install SSIS PowerPack
Create new SSIS Package
Right click in Connection Manager Area and Click “New Connection”
When prompted select ZS-OAUTH connection type
On the OAuth Connection Manager Select Provider=Google Sheets + Drive.
You can keep “Use Default App” selected or choose Custom App. Click here to learn – how to register Custom Google App
If you choose the Custom App option then enter the below scopes (or click Select Scopes). If you choose the default app then it’s not required.
This will allow read / write access to Drive Files or Sheet (E.g. Reading file content / Export file to CSV). You can use Scope browser to see many other available permissions.
```
https://www.googleapis.com/auth/drive
https://www.googleapis.com/auth/drive.file
https://www.googleapis.com/auth/drive.readonly
https://www.googleapis.com/auth/spreadsheets
https://www.googleapis.com/auth/spreadsheets.readonly
```
Click Generate Token button.

SSIS OAuth Connection Manager – Access Google Drive API using OAuth 2.0
You will see UI as below… Click Accept (You may have to scroll to see that button sometimes)
Click Test to see connection is working.

NOTE: If you don’t want to use Default OAuth App provided by ZappySys then select “Use Custom OAuth App” option (App created in previous section) specify your ClientID, ClientSecret.

Find Google Sheet ID for API call

Before we can read or write Sheet data in SSIS we need to know Google Driver File ID. We will use this ID in next few sections.

Easiest way to find File ID is look at the URL like below. If you need to know ID at runtime dynamically then refer last section.

Get Google Drive File ID for API call (Sheet ID)

Reading Google SpreadSheet Data in SSIS

Basically there are two ways you can export / read Google SpreadSheet data.

Method-1 : Read Google SpreadSheet using SSIS JSON / REST API Source
- In this approach we directly pull data from SpreadSheet using Core Google SpreadSheet.Values API to read particular Tab from Sheet and you can also specify Range. This is most effective way to read. Make sure you have v2.5.0.10807 or higher version (released after Aug 03 2017). New version added 2D array transformation options to parse multi-dimensional arrays found in SpreadSheet API JSON response.
Method-2 : Read Google SpreadSheet using Flat File Source (as CSV)
- Another approach is export SpreadSheet as CSV format. Then use Flat File Source to read that data and load into Target (E.g. SQL Server)

Method-1 : Read Google SpreadSheet using SSIS JSON / REST API Source

Now lets look at real steps to read Google SpreadSheet. To get values from specific tab and specific cell-row range you have to use Google SpreadSheet.Values API

In this example we will load below Google Spreadsheet into SQL Server

Step-By-Step : Loading Google SpreadSheet into SQL Server using SSIS

Drag and drop data flow on the control flow designer surface
Double click data flow and you will see SSIS Toolbox refreshed with new components
Drag ZS JSON Source from SSIS Toolbox onto Data flow designer surface
Double click JSON SOurce and configure below setting
1. Enter your Spreadsheet URL in the following format
```
https://sheets.googleapis.com/v4/spreadsheets/{your-file-id}/values/{cell-range}
```
  For example purpose we will use this public SpreadSheet provided by Google. If spreadsheet fileid is 1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms (see previous section how to get fileid) and your tab name is Class Data and you like to extract data from A to F columns starting from 2nd row then your Cell range in the URL can be ‘Class Data’!A2:F
  Here is the actual URL we will use for demo
```
https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/'Class Data'!A2:F
```
  — OR —
  You can also enter like below (If you omit Tab name then first table is used)
```
https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A1:F

--OR-- Use below URL for to supply Column names manually (Start from A2)

https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A2:F
```
  You can also enter like below, specify first row and last row
```
https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A1:F101
```
2. Check Use credentials and select OAuth connection we created in previous section
3. Click on Select Filter and select values node and click OK or you can directly enter following filter.
```
$.values[*]
```
  Configure SSIS JSON/REST API Source – Read data from Google SpreadSheet API
4. Now click on array Transformation tab. Now we will show you two different ways to configure column names.
5. Method-1: Auto Detect Column Names (added in version 3.1.2)
  In recent version we added column less array transformation where we have option to detect first record as column name.
  1. Select Transform Type as Column less array
  2. Check option First line has column names like below example (Ignore direct Sample JSON – We provided for demo only. In your case it will be URL)
    JSON With 2D Array Pattern – Column names in First Row (Google Sheets API Pattern)
6. Method-2: Manually Enter Column Names (For old version)
  1. Select Transform simple 2D array option from dropdown
  2. Check Specify columns list manually
  3. Enter column names (Must match the order and count based on range you selected)
```
Student Name,Gender,Class Level,Home State,Major,Extracurricular Activity
```
    Configure SSIS JSON/REST API Source 2D Array Transformation for Google SpreadSheet API
7. Now click Preview to see the data
  Configure SSIS JSON/REST API Source –
  Preview Google SpreadSheet Data
8. Click OK to save UI
9. Attach your JSON Source to target (e.g. OLEDB Destination for SQL Server Table).
10. Configure Destination
  Configure SQL Server destination for Google SpreadSheet to SQL Data load
11. Execute SSIS Package to load Google SpreadSheet data int SQL Server
  SSIS Example -Loading data from Google SpreadSheet into SQL Server Table

Method-2 : Read Google SpreadSheet using Flat File Source (as CSV)

Once we have Google Drive fileId we can perform following actions to export data into CSV file format. After that you can easily read CSV using SSIS FlatFile Source and load into SQL Server or other target.

Export Google SpreadSheet as CSV file using SSIS REST API Task

Here are the steps to export Google Drive SpreadSheet to CSV file

Just like previous section configure REST API Task

Enter following URL (See we used FileId from SSIS variable. You can hardcode too.

https://www.googleapis.com/drive/v3/files/{{User::varFileId}}/export?mimeType=text/csv

Goto response tab. Check Save response and select “Save to file option”.
Enter file path.
Response character set select utf-8 (This will allow Unicode characters in response)
Click Test to see preview
Export Google SpreadSheet to CSV file format using SSIS – Call Google Drive API to Export
Click OK to save

NOTE: New version now includes CSV Source for API. Use that if you want to consume Google Spread Sheet data in Data Flow. However we still recommend JSON Source (As per previous section to read data. Use CSV export method only if JSON API is not working for you for some reason.

Load Google SpreadSheet data into SQL Server using SSIS (REST API Call)

Once you export SpreadSheet to CSV file you can easily consume it to load into SQL Server or any Target (e.g. Oracle, MySQL) using Native SSIS FlatFile Source and OLEDB/ADO.net Destination.

NOTE: Using FlatFile Source is not covered in Article. You can find many articles online which shows how to use SSIS FlatFile Source.

Write data to Google Spread Sheet

Now lets look at how to write to Google SpreadSheet. To write multiple rows in a single request you may use batchUpdate API. Also check this link to learn more.

Below is sample request to write data to columns and some data to rows (set majorDimension = COLUMNS to write vertical and use ROWS to write horizontally. You can mix multiple ways just like below example)

POST 
https://sheets.googleapis.com/v4/spreadsheets/1lkMEgu0zm2Q-cnPeoNZp4hqeIuASxXSpsrxtDDV89tg/values:batchUpdate
Authorization: Bearer ya29.Gl3WBFb3xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: application/json
Accept: */*

{
  "valueInputOption": "USER_ENTERED",
  "data": [
    {
      "range": "Sheet2!A1:A4",
      "majorDimension": "COLUMNS",
      "values": [
        ["Item", "Wheel", "Door", "Engine"]
      ]
    },
    {
      "range": "Sheet2!B1:D2",
      "majorDimension": "ROWS",
      "values": [
        ["Cost", "Stocked", "Ship Date"],
        ["$20.50", "4", "3/1/2016"]
      ]
    }
  ]
}

Write data to Google SpreadSheet using REST API Task (OAuth 2.0 Connection)

Update Google Sheet Cells – Single Range (Set Values for multiple cells)

Now let’s look at the below example of updating multiple cells in google Sheet. For more information, you can see the example here. For parameter detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve Spread Sheet Update scenario. For simple example lets use REST API Task.

Drag and Drop ZS REST API Task on the control flow designer
Edit Task and select URL from Connection Method
Select OAuth Connection from the dropdown (Same connection we created in the previous section)
Enter the URL as below. Change Sheet ID, Range designator and API URL Parameters as per your need.
https://sheets.googleapis.com/v4/spreadsheets/[Sheet-ID-Here]/values/[Range-Here]?[Parameters-Here] Example URL (some part masked with xxxxxxxx):
```
https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4?valueInputOption=USER_ENTERED
```
Change Request Method to PUT
Change Request Content type to application/json

Enter Request Body as below (Assuming we need to update 4 cells)

{
"majorDimension": "ROWS",
  "values": [
   ["row1_cellA","row1_cellB"],
   ["row2_cellA","row2_cellB"],
  ]
}

Click Test Request see it works.
Calling Google Sheet API to Update Multiple Cell Values

Update Google Sheet Cells – Multiple Ranges

If you like to update values which needs to define multiple ranges then you need to use batchUpdate API. Import changes are you can’t include Range in the URL like previous example of Single Range Update. Also you have to use POST method and body with multiple ranges.

Example of sheet update with multiple range

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHxxxxxxxxxxxxxxxc/values:batchUpdate?valueInputOption=USER_ENTERED
Content-Type: application/json

>>>> BODY <<<<<

{
  "data": [
    {
      "range": "Sheet1!A2:B3",
      "majorDimension": "ROWS",
      "values": [
        [
          "row1_cellA",
          "row1_cellB"
        ],
        [
          "row2_cellA",
          "row2_cellB"
        ]
      ]
    },
    {
      "range": "Sheet1!D2:E3",
      "majorDimension": "ROWS",
      "values": [
        [
          "row1_cellD",
          "row1_cellE"
        ],
        [
          "row2_cellD",
          "row2_cellE"
        ]
      ]
    }
  ]
}

Write SQL Server data to Google Sheet

Now let’s look at real world example. How to write data from SQL Server table or other source and send to Google Sheet.

Drag and Drop SSIS Data Flow Task from SSIS Toolbox
Drag source (e.g. OLEDB Source) and configure it to read from source like SQL Server Table
OLE DB Source – Drag and Drop
Drag ZS JSON Generator.
1. Connect Source to JSON Generator (blue arrow)
2. Double click it to configure
3. Select Output Mode as Single Dataset Array option for Mode. Enter Some Batch value (e.g. 500) this will make sure we send 500 rows at a time to Google API call. This way if we have many rows its not rejected by API call.
4. Click Add element (Select Static Element) . Name: majorDimension and Value: ROWS
5. Now click Add Document Array icon, name as values
6. check Treat as 2D array option and click OK to save
7. Now under values node click Add elements > Select Multiple Columns and Add columns you like to add.
8. Click OK to save JSON GeneratorBelow is just an example How you can use Batch Settings to create JSON with N rows in each JSON document.
Now drag ZS Web API destination
1. Connect JSON Generator to Web API destination (blue arrow)
2. Double click it to configure
3. Select same OAuth connection manager created in earlier section (Assuming you had all scopes with write permission)
4. Now select Input column for Body (JSON Generator Output)
5. Enter API call URL as below (Replace ID with your own id)
```
https://sheets.googleapis.com/v4/spreadsheets/{your-file-id}/values/Sheet1!A2?valueInputOption=USER_ENTERED
```
6. Select Request Method as PUT
7. Select content type as application/json
8. You can enter sample Body As below.
```
{
"majorDimension": "ROWS",
  "values": [
   ["Bob","bob@mycompany.com",55],
   ["Sam","sam@mycompany.com",57]
  ]
}
```
9. Click Test Request / Response button. Check your Sheet in browser after it see it updates?
10. If you see 200 OK response means we are good to go.
11. Run package and see your real data updated in Sheet.

Format Cells using Google Sheet API

Now let’s look at an example to format cell using Google Sheet API (e.g. Change Font, Color, Border, Background). For more information review this API documentation.

To format Sheet you first need to get SheetID (keep in mind this is not same as spreadsheetId which you used in URL). SheetId is basically numeric ID of your Tab in WorkSheet.

Get SheetID

Request

GET https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHtxxxxxxxxxxxxxx

Response

{
  "spreadsheetId": "1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc",
  "properties": {
    "title": "Test.xlsx",
    "locale": "en",
    "autoRecalc": "ON_CHANGE",
    "timeZone": "America/Los_Angeles",
    "defaultFormat": {
      "backgroundColor": {
        "red": 1,
        "green": 1,
        "blue": 1
      },
      "padding": {
        "right": 3,
        "left": 3
      },
      "verticalAlignment": "BOTTOM",
      "wrapStrategy": "OVERFLOW_CELL",
      "textFormat": {
        "foregroundColor": {},
        "fontFamily": "Calibri",
        "fontSize": 11,
        "bold": false,
        "italic": false,
        "strikethrough": false,
        "underline": false
      }
    }
  },
  "sheets": [
    {
      "properties": {
        "sheetId": 1896331083,
        "title": "Sheet1",
        "index": 0,
        "sheetType": "GRID",
        "gridProperties": {
          "rowCount": 1000,
          "columnCount": 26,
          "rowGroupControlAfter": true,
          "columnGroupControlAfter": true
        }
      }
    }
  ],
  "spreadsheetUrl": "https://docs.google.com/spreadsheets/d/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc/edit"
}

Notice that in above response we see “sheetId”: 1896331083. This ID we will use in the Update API to format Cells.

Update Borders Example

Here is the example of Google Sheet Formatting API updateBorders Request

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSxxxxlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "updateBorders": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 0,
          "endRowIndex": 10,
          "startColumnIndex": 0,
          "endColumnIndex": 6
        },
        "top": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
        "bottom": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
        "innerHorizontal": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
      }
    }
  ]
}

Response

{
  "spreadsheetId": "1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc",
  "replies": [
    {}
  ]
}

Update Cell Font Style (Bold, Style, Color, Background)

Now let’s look at how to use repeateCellRequest to apply certain type of formatting to multiple cells. This may be color, font style, background color.

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3xxxxxx4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "repeatCell": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 0,
          "endRowIndex": 1
        },
        "cell": {
          "userEnteredFormat": {
            "backgroundColor": {
              "red": 0,
              "green": 0.5,
              "blue": 0.7,
              "alpha": 1.3
            },
            "textFormat": {
              "foregroundColor": {
                "red": 1
              },
              "bold": true,
              "italic": true,
              "strikethrough": true,
              "underline": true
            }
          }
        },
        "fields": "userEnteredFormat"
      }
    }
  ]
}

In above example we have applied textFormat and backgroundColor but you can use any of below sections.

{
“numberFormat”: {object(NumberFormat) },
“backgroundColor”: { object(Color) },
“borders”: { object(Borders) },
“padding”: { object(Padding) },
“horizontalAlignment”: enum(HorizontalAlign),
“verticalAlignment”: enum(VerticalAlign),
“wrapStrategy”: enum(WrapStrategy),
“textDirection”: enum(TextDirection),
“textFormat”: { object(TextFormat) },
“hyperlinkDisplayType”: enum(HyperlinkDisplayType),
“textRotation”: { object(TextRotation) }
}

Set Currency Formatting

Here is an example request to set currency formatting

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "repeatCell": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 1,
          "startColumnIndex": 4,
          "endColumnIndex": 5
        },
        "cell": {
          "userEnteredFormat": {
            "numberFormat": {
              "type": "CURRENCY",
              "pattern": "$#,##0.00"
            }
          }
        },
        "fields": "userEnteredFormat.numberFormat"
      }
    }
  ]
}

Set Cell Validation

You can also set cell validation by calling following API call. This will restrict user’s ability to enter wrong data in certain cells.

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "setDataValidation": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 1,
          "startColumnIndex": 0,
          "endColumnIndex": 6
        },
        "rule": {
          "condition": {
            "type": "ONE_OF_LIST",
            "values": [
              {
                "userEnteredValue": "Red"
              },
              {
                "userEnteredValue": "Green"
              },
              {
                "userEnteredValue": "Yellow"
              }
            ]
          },
          "inputMessage": "Color must be Red, Yellow or Green",
          "strict": true,
          "showCustomUi": true
        }
      }
    }
  ]
}

Dynamically obtain File ID for Google Drive REST API call

Once OAuth connection is defined you can now call any valid API from Google based on Permission you requested in OAuth Connection. OAuth connection takes care of renewing you Token and passing Authorization as per OAuth standard. All these protocol complexity is hidden from you.

First step to get data from Google SpreadSheet using REST API call is get fileId of the Google Drive File you want to read. The easiest way to know file ID is call following API. Below API call returns name, ID and other information about file(s). You can get only fileID and use that in Next Section to export Google SpreadSheet to CSV format.

https://www.googleapis.com/drive/v3/files

https://www.googleapis.com/drive/v3/files/?q=name%3D'Customers'

NOTE: For more information about searching files in Google drive / getting list of files using search expression language check this link. Make sure expression is URL encoded (e.g. name=’Customers’ should be passed as name%3D’Customers’)

Above API returns response as below. Notice the name and fileId. We will use fileId when calling export API in next section.

{
 "kind": "drive#fileList",
 "files": [
  {
   "kind": "drive#file",
   "id": "1lkMEgu0zm2Q-cnPeoNZp4hqeIuASxXSpsrxtDDV89tg",
   "name": "Customers",
   "mimeType": "application/vnd.google-apps.spreadsheet"
  },
  {
   "kind": "drive#file",
   "id": "0B6UyXr6ZlIFTc3RhcnRlcl9maWxl",
   "name": "How to get started with Drive",
   "mimeType": "application/pdf"
  }
 ]
}

If you want to automate retrieval of fileId from document name then perform the following steps.

In the SSIS package Control flow. Drag and drop ZS REST API Task from SSIS toolbox
In the Request Access mode, select “URL from Connection” option
Select OAuth connection manager (Created in Section-2)
Enter URL as below (It will return any file name equals to Customers (%3D is encoded value for = sign)
```
https://www.googleapis.com/drive/v3/files/?q=name%3D'Customers'
```
Click Test (You will see file information)
Get Google Drive File List in SSIS – Call Google Drive API to search file by name or expression
If you want to extract only fileID from full JSON then go to response tab
Select Format=JSON, Enter following JSONPath expression to extract only fileId from first record
```
$.files[0].id
```
Check save response content, select SSIS variable where you want to save fileId as below. Now click Test to see preview of response. You will notice now only fileId will be returned from full JSON response.
Get Google Drive fileid from file name using API call in SSIS
Click OK to save task

Conclusion

Google Drive API and Google Sheets API provides a great way to automate file related functionality. However, to call Google API you have to use SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Google API call via simple drag and drop approach without coding. Try out yourself see how long it takes to call virtually any REST API.

The post How to read / write Google SpreadSheet using SSIS appeared first on ZappySys Blog.

SSIS – Extract single XML node using XPath from SOAP response

ZappySys — Wed, 07 Sep 2016 16:01:07 +0000

Introduction

In this post, you will learn how to extract a single XML node value from your XML web response (SOAP Web service call) using an XPath expression in the SSIS REST API Web Service Task and the XML Parser Task.

This article assumes you have basic knowledge of how to call a REST API or a SOAP Web service in SSIS. If you are not familiar with that, then refer to the following articles.

Calling SOAP Web Service in SSIS (XML Source)

Call REST API using SSIS Web Service Task / JSON / XML Source

Calling REST / SOAP Web service using the SSIS REST API Task

Refer to the following video on how to call a REST API web service.

Calling REST API in SSIS using REST API Task, Pass headers, Body, Url Parameters

Extract the REST API Response into a file

Here is how you can save the response into a variable or a file. Go to the Response settings tab and select the check Response content option, as shown below. You can also Filter Response content (e.g., Extract only a single value from a node).

SSIS Rest API Task – Extract data from a single XML node using XPath expression (Namespace used)

Extract a single XML Node using XPath (XML with Namespace)

To extract a single value from the Response XML, you can define an XPath expression on the REST API Response tab as follows. If your response XML document contains namespaces (e.g., the node looks like this: abcdefg), then you have to write XPath in a particular way, as shown below. Notice we have not specified the Namespace prefix ns1

Sample XML Document (Or web response from SOAP Service)


	abcdefg

XPath Expression Example:

//*[local-name()='Data']

XPath Output:

The above expression will produce the following result

abcdefg

Extract Attribute Value using XPath local-name() function

Now, let’s find out how to extract the attribute value rather than the element.

Consider the following XML. Let’s say we want to extract the id from the data node. In such a case, the previous XPath expression won’t work, so we need to use a slightly different XPath.


	abcdefg

Let’s use the XML Parser Task to test XPath.

Expression 1 – Get value by direct attribute name

Write the following expression as below and click Test

//@*[local-name()='id']

Extract XML Attribute Value using XPath expression (Use local-name function)

Expression 2 – Get specific element attribute value

If you want to be specific and read the value of the id attribute from a particular data element, you can use an expression like this:

//*[local-name()='Data']/@*[local-name()='id']

Extract XML Attribute Value using XPath expression 2

Configure the SSIS Rest API Task to filter XML Response using XPath

The screenshot below shows how to filter XML data using an XPath expression. You can save extracted data to an SSIS Variable or save it to a file.

SSIS Rest API Task – Extract data from a single XML node using XPath expression (Namespace used)

More Examples of XPath expressions

XPath expression examples:
============================

Example1: Extract value of any element by node name (without namespace specified) 

aaaaa

XPath Expression:  //*[local-name() = 'Token']
Returns: aaaaa
----------------------------
Example2: Extract text of element called Token 

aaaaa

XPath Expression:  ./AuthInfo/Token/text()
Returns: aaaaa
----------------------------
Example3: Extract value of attribute called Version 



XPath Expression:  ./Doc/Row/@Version
Returns: 1
----------------------------
Example4: Extract value of following-sibling node or use preceding-sibling 


	
		User
		admin
	
	
		SessionId
		s_123456
	


Expression:  //*[local-name()='PropName' and text()='SessionID']/following-sibling::node()[local-name()='PropValue']
Returns: s_123456
----------------------------
Example5: Extract node value with matching attribute 

s_1234

XPath Expression:  //*[local-name()='Prop' and @Name='SessionId']
Returns: s_1234
----------------------------
Example6: Extract attribute value by name (without namespace specified) 

s_1234

XPath Expression:  //@*[local-name()='token']
Returns: 123456
----------------------------

The post SSIS – Extract single XML node using XPath from SOAP response appeared first on ZappySys Blog.

REST API File Upload using SSIS (Multi Part POST)

ZappySys — Thu, 02 Jun 2016 22:42:27 +0000

Introduction

In this post, you will learn how to use the SSIS REST API Task to perform REST API File Upload (i.e., RESTful file upload) to a web server. Typically, to access the REST API, you need to use a programming language, but in this post, we will use a drag-and-drop visual approach.

File upload using HTTP/HTTPS protocol, sometimes also known as multipart/form-data POST. Many REST APIs support file upload using HTTP methods such as POST or PUT.

Real-world scenarios:

Uploading images/media to Twitter or Facebook
Import Marketo Lead files using REST API

How to do REST API File Upload using SSIS

First, you will have to download a FREE Trial of SSIS PowerPack (It comes with many REST API Connectors and over 30+ functional tasks/components)

Download and install SSIS PowerPack from here
From the toolbox of the SSIS designer, drag the ZS SSIS REST API Task
Double-click the REST API Task to configure it
Specify URL where you wish to POST file/other data
Example: http://myserver/api/v1/uploaddata
Select HTTP Request Method as POST ( make sure your API Documentation for this. Sometimes it’s PUT rather than POST)
Check the File Upload option next to the Body text box (This is most important so don’t forget 🙂 )

Method 1 – Upload file using Raw Mode (Or POST data from a local file)

The first method of uploading a file or posting data to the server from a local file is a RAW upload. To enable this you simply have to make sure 2 things.

Check the File upload option next to Body.
In the Body, enter the local file path starting with the @ sign (e.g., @D:\Zappysys\CSV files\example1.csv)

When you don’t include a Key/Value pair and only the filepath is found, then the request is treated as a RAW Upload. In raw uploads, the default content type is application/octet-stream. If you want to put a custom Content-Type, then edit the Headers in the Grid.

The above technique can be used to read POST data content from a local file. Using this method, only one file can be uploaded. If the remote server supports multiple files (i.e., Multi-Part/Form-Data POST), use the following process. See the screenshot below of the file content upload using Raw mode. We used Fiddler to inspect the REST API Task’s Web requests.

POST file data using SSIS ( Raw file upload)

Method 2 – Upload file(s) using Multi-Part / Form Data mode (Mixed content, multiple files)

The second method for file upload is Multi-Part Upload. The advantage of using this method is that you can POST upload a key/value pair along with single or multiple files. To use this method, you must make sure the API supports it on the other end. Use the steps below to POST data using the Multi-Part method. SSIS PowerPack detects RAW mode vs Multi-Part POST based on BODY content. If the Body contains an Equal sign somewhere and the File Upload option is checked, then it’s treated as a Multi-Part upload. If File Upload Option is checked and Body contains only file path which starts with @ sign then its treated as Raw content upload.

Check the File upload option next to Body.
Now, click the edit button next to the Body text box. You may specify files/fields you wish to submit along with your request in the following format. It’s a key/value pair separated by a new line. An equal sign (=) must separate each key/value pair.NOTE: Any value starting with an at symbol (@) is treated as a File Path. If you don’t want that behavior, then use a slash, also known as an escape sequence (e.g., Somefield3StartsWithAt=\@MyTwitterHandle). You can do the same thing to add a new line character in your value. Use \r\n for a new line. Use \t for tab. Also, notice that the field name can be anything unless your API expects it in a specific way. Example: Twitter Image Upload API may expect your image file as a media field. In that case you would write …. media=@c:\data\someimage.png …. rather than File1=@c:\data\someimage.png
```
File1=@c:\data\somedata.txt
File1.Content-Type=text/plain
File2=@c:\data\someimage.png
File2.Content-Type=image/png
Somefield1=abcdef
Somefield2=123456
Extrafield3WithNewLine=This is line1\r\nThis is line2 with tab\t and more \t tab
Somefield3StartsWithAt=\@MyTwitterHandle
```
Click OK to Save above.
NOTE: If you do not specify Content-Type of file using
```
SomeFieldName.Content-Type
```
attribute, then the system automatically detects based on the file extension. To exclude content-type altogether, you have to set it as a blank string, like below
```
file1=@c:\temp\dump.json
file1.Content-Type=
```
Now, click the Test button in the UI to make sure everything looks good.
Uploading files using REST API in SSIS ( REST API File upload using Multipart form-data POST)

Understanding Multipart Upload / File Upload

Now, let’s take a look at what it looks like in Fiddler when an actual Multipart POST request is made for the REST API File upload. Notice that each field included in our HTTP POST request is separated by some random boundary. The REST API Task automatically generates this boundary. If you post an Image, then you will see binary data rather than text for file content. Here are some known issues you need to be aware of.

HTTP Multipart POST – File Upload Example – Check Fiddler Request

Here is a Sample Multi-Part request.

Notice a few things

The first line of the body content is “Boundary.”
Last line in the body is also a Boundary, but it has two extra dashes at the end after the boundary line (i.e. ———–xyzzzzz– )
Notice name=”something”. This is an essential attribute. This is how the target server knows which item to extract from your response.
Content-Type is automatically determined for a Multi-Part item when it is a file upload. Based on extension, ZappySys automatically selects the most common content type. See the next section.

POST https://somecompany.com/downloads/files/test/uploadtest.aspx HTTP/1.1
User-Agent: ZappySysApp/1.0.2019.10822
x-custom-hdr: 69c05bf5b6ef48dbb73c55020c50e027
Content-Type: multipart/form-data; boundary=---------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825
Host: somecompany.com
Content-Length: 3619
Expect: 100-continue
Accept-Encoding: gzip, deflate
Connection: Keep-Alive

-----------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825
Content-Disposition: form-data; name="File1"; filename="MultiPartFileUpload1.txt"
Content-Type: text/plain

A000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
-----------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825
Content-Disposition: form-data; name="File2"; filename="MultiPartFileUpload2.txt"
Content-Type: text/plain

B00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
-----------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825
Content-Disposition: form-data; name="Extrafield1"

aaaa
-----------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825
Content-Disposition: form-data; name="Extrafield2"

bbbb
-----------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825
Content-Disposition: form-data; name="Extrafield3WithNewLine"

This is line1
This is line2 with tab	tab	tab
-----------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825
Content-Disposition: form-data; name="Extrafield3WithAt"

@MyTwitterHandle
-----------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825--

Known Issues

Sometimes you may receive the following error if your server is expecting \r\n after the last Multi-Part boundary. Here is the fix for this error.

The remote server returned an error: (500) Internal Server Error.
Response Url: https://your-host/api

Status Code: InternalServerError 

Response Body: {"Message":"An error has occurred."
,"ExceptionMessage":"Unexpected end of MIME multipart stream. MIME multipart message is not complete.","ExceptionType":"System.IO.IOException","StackTrace":" at System.Net.Http.Formatting.Parsers.MimeMultipartBodyPartParser.d__0.MoveNext()\r\n at System.Net.Http.HttpContentMultipartExtensions.MoveNextPart(MultipartAsyncContext context)"
}

Most common extension and Content-Type

In the multi-part upload, you might have noticed that ZappySys adds a content type for some file extensions. Here is a full list of some common extensions and their content-type

Conclusion

If you need to perform REST API file upload or REST API integration without learning complex APIs or programming languages (e.g., Python, Java, C#), SSIS PowerPack provides a perfect way to integrate REST APIs with your existing system. You can uncover endless possibilities with a drag-and-drop approach that is not only simple to use but also offers many options for complex scenarios.

Keywords

The post REST API File Upload using SSIS (Multi Part POST) appeared first on ZappySys Blog.

Calling Web API in SSIS with Authorization and Redirect

ZappySys — Thu, 26 May 2016 17:15:38 +0000

Introduction

In some cases when you call API to http URL it may redirect you to different location (Also known as 301 or 302 redirect). Best example is when you type https://www.zappysys.com it will redirect you to https://zappysys.com. This can cause issue in some case if you are passing credentials (i.e. UserID / Password) because by default your credentials are not passed to redirected URL (This is security feature to avoid sending your credentials to different URL than you anticipated). So how to send Authorization to second URL if second URL expects same credentials too? Well not to worry… it’s just one extra step in that case. Lets see how to do that.

Pass Authorization to 301 / 302 Redirect URL

Assume that you have to call below API URL which requires Basic Authentication (UserID/Password) and it also redirects to some other URL which also needs same credentials. For this purpose we need to mainly perform below important changes.

http://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.com%2F

First drag ZS REST API Task from SSIS Control Flow Toolbox
Change Url mode to URL from connection. Click New Connection.
Configure your connection with API Url, UrserID and Password as below. Click OK to save connection.
SSIS REST API Example – Call with 301 / 302 Redirect
Now go to Advanced Tab and Check Prevent 301 redirect option
SSIS REST API Task – Disable automatic redirect option
Now go to Response Settings Tab and Map Location Header as below (Assuming you have created SSIS Variable named NewLocation (string type) )
SSIS REST API Task – Map response header to SSIS Variable
Click OK to save.

Call API URL from URL

Now drag another ZS REST API Task from SSIS Control Flow Toolbox
Change Url mode to URL from connection. Select connection created in previous section.
Now configure your Url as below so we read it from Variable.
SSIS REST API Task – Calling API URL from SSIS Variable
That’s it Save the task and run package.

Debug Web API call using Fiddler

To test how things looking behind the scene we strongly suggest to use tool like fiddler.

See how Fiddler shows 302 Status code for First API call and see how it passes Authorization (UserID/Password). You can also test Just first task without [Prevent 301/302 redirect] option see what happens]

Debugging Web API call using Fiddler in SSIS

The post Calling Web API in SSIS with Authorization and Redirect appeared first on ZappySys Blog.