xml source Archives | ZappySys Blog

How to make Path / URL dynamic in SSIS

ZappySys — Wed, 13 Sep 2017 13:27:33 +0000

Problem

Most of SSIS PowerPack components support simple text placeholders engine to make things dynamic without using Complex Steps of SSIS Expression Engine. So let’s look at how to make Path / URL or other input values dynamic in SSIS. To show you how to do that we will use JSON Source component, but basically, you can use it in any ZappySys PowerPack component that has URL option, including XML Source.

Create a variable

Don’t forget to set the default value, because it will be used to construct an URL in design time. A call to that URL will be made to figure out the columns.

Make an HTTP address dynamic

Creating a Dynamic Request Body

To make a dynamic body in ZappySys (e.g., in an HTTP request or API call using the REST API Task or JSON Source or XML Source), you typically use placeholders or expressions in the request body.

Here’s how to create a dynamic body:

Example: Dynamic JSON Body using Variables

{
  "name": "{{User::MyVariable}},
  "status": "active"
}

{{User::MyVariable}} is a variable placeholder.

Dynamic Body

That’s How You Make It Dynamic

By combining placeholders and expressions in the URL and body fields, ZappySys lets you build fully dynamic API requests. Whether you’re passing parameters, looping through records, or injecting runtime values—this approach keeps your integration flexible, scalable, and efficient.

The post How to make Path / URL dynamic in SSIS appeared first on ZappySys Blog.

Calling SOAP Web Service in SSIS (XML Source)

ZappySys — Thu, 23 Jun 2016 13:51:11 +0000

Introduction

In this article, you will learn how to call SOAP Web Service in SSIS. We will use drag and drop approach to consume SOAP Webservice using SSIS XML Source Connector. This same XML Connector can be used to consume data from local XML Files (wild card allowed e.g. *.xml) or you can consume XML stored in SSIS Variable. Great way to Parse XML Stored inside the database.

In this article, we will consume data from SSRS SOAP web service and load into SQL Server. You can apply a similar technique to call any web service.

Video Tutorial – Load SOAP Web Service data into SQL

Video Tutorial – Create XML and POST data to SOAP Web Service or REST API

Now lets look at examples of creating XML from multiple data sources and POST XML request to SOAP Web Service URL or any other XML Based REST API URL.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link

There are two important aspects in SOAP Web service.

Getting WSDL file or URL
Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

What is Service URL

Service URL is actual API URL you can call to get data. When you make a request to SOAP service url make sure following things.

Use correct service URL (Its not same as WSDL url). If you are not sure about service URL contact API vendor or refer API documentation.
Each service call you have to supply function you want to call. This can be done in 3 different ways.
1. Pass via SOAPAction HTTP Header
2. Pass via Url parameter or Url Path ( http://myserver/service/function1 — http://myserver/service/?method=function1 )
3. Pass via Request BODY
Find out how to pass credentials. It can be done via Basic Authorization or WSS Security Header in the Request body. Check this link to learn more

How to call SOAP API in SSIS (Summary)

To call SOAP API in SSIS you have to perform the following steps at a high level. Detailed steps are described in the next section

Download and Install SoapUI (It’s a Free tool to test SOAP / REST Services)
Import WSDL file or URL for your Service in SoapUI for testing
Add credentials for SOAP Web service (Basic Auth or WSS)
Create SOAP Request XML (Edit default parameters) and Click Play Button to test your request
Create SOAP Request Body from WSDL (Using SoapUI tool)
If your test works in SoapUI then you ready to move to SSIS part
Copy Service URL, Request Body and Headers to SSIS (For Header see Raw Tab on SoapUI > Copy two lines SOAPAction:***** and Content-Type:******)
Get ContentType and SOAPAction Headers from SoapUI Raw tab
Use SSIS REST API Task or SSIS XML Source or SSIS Web API Destination to call SOAP API (Use correct API URL, Body and Headers)

See the next section for detailed steps for calling SOAP web service in SSIS.

Step-By-Step : Calling SOAP Web Service in SSIS

In this section you will learn how to call SOAP Web service in SSIS without any type of coding. In few clicks you will be able to load data from XML SOAP Service to SQL Server. For demo purpose we will call SSRS Web service but you can pick any available SOAP Service.

Download and Install SSIS PowerPack FREE Trial from this link
Now Download SoapUI (Its popular free third-party tool by other vendor). SoapUI can help you to generate XML Request BODY easily by using WSDL file. You can also test your service by supplying parameters.
Now find out your SOAP Service WSDL URL. It can be URL or locally stored XML file. WSDL is standard to describes SOAP Service (e.g. Available functions, request/response format). For SSRS WSDL is located usually at below location. Sometime WSDL URL is not correctly downloaded in SoapUI in that case use web browser to download WSDL and save to local disk.
```
http://YourReportServer/reportserver/ReportService2010.asmx?wsdl
```
Now launch SoapUI. Click on File menu and Select New SOAP Project
Enter your WSDL file path or URL
Click OK to create Project. This will create some dummy requests for you. If you want to see all available parameter in your Request you can create New Request by right click on Method node and select New Request. When prompted to create optional parameters then click yes.
Create new SoapUI project to generate / test Soap Requests
Click on Request node as below and it will show you XML code for your SOAP request. You can edit parameters as needed. We will use this code in our XML Source to build SOAP Request. Some time your SOAP Service may require UserID/Password in SOAP Body or via Authorization Header like this one. If you passing credentials Authorization header then you can use ZS-HTTP connection (on XML Source UI check Use Credentials option and create new connection)
SOAP Request XML
Create new SSIS Project and add new package.
Open Package. Go to control flow. Drag and drop Data Flow task from SSIS Toolbox
Go to Data flow designer. Drag and drop ZS XML Source Component
While making SOAP Request you have to configure following items in your XML Source.
- URL
- Request Method (e.g. POST)
- Headers
- Request Body (for request SOAP XML)
Double click XML Source to configure it.
1. Set webservice URL
2. Set request method to POST
3. In the headers grid click Raw Edit to add 2 more headers as below.
  
  If you not sure about SOAP Action then find out from SoapUI > Raw Tab of Request Panel. Headers are only generated if you click the green Play button in SoapUI. Copy following two headers from SoapUI Raw tab and paste under SSIS XML Source > Headers Grid > Raw Edit.
```
Content-Type: text/xml;charset=UTF-8
SOAPAction: "http://some-url-of-your-api-call"
```
  SOAPAction URL is also listed under the following place in SoapUI.
  
  How to find SOAPAction Header value using SoapUI tool
Now click on Edit option next to Body Textbox. You can Copy sample Request XML generated in SoapUI and paste in Request Data (See below screenshot). You have to make sure to include all required fields in your XML Request (You may create new XML request in SoapUI with include Optional fields Turned ON to see all parameters).
XML Source supports using Placeholders in request URL, body and headers so its great way to make things dynamic. If you have need to use placeholders then you can use {Insert Variable} option like below screenshot and add variable anywhere in the text.
You can also edit Filter to control how data is returned. If you have Nested nodes in XML response you can select data node which you care about. Click select Filter button to navigate through hierarchy.
Filter SOAP Web Service – XML response data
Un-check include parent option if you don’t care about getting parent attributes.
Your final UI may look like below after setting Url, Method, Body and Headers.
Calling SOAP Web Service in SSIS using XML Source Connector
On Array handling tab you can type all the element names you know for sure that going to have one or more records. For example if your response looks like below you can enter a:row in the possible array fields on Array Handling tab. That way if you have just one record rather than multiple records Filter expression like this wont fail ( $.a:response.a:data.a:row[*] ) .
```
	
		....
		....
	
```
Now once all settings entered. Click Preview.
Set Filter and Preview SOAP Web Service Response in Table Format

Creating SOAP Request with Optional Parameters

Sometimes if you create SOAP request you may not see optional parameters. Here is how you can create SOAP Request with optional parameters.

Create SOAP Request XML (With Optional Parameters)

Load XML Data into SQL Server

Once you configure XML source now you can connect source to target such as SQL Server, MySQL, Oracle, Flat File etc. On target you can map various source columns to SQL Server Table columns.

SSIS SOAP Web Service Call – Loading XML data into SQL Server Table (or MySQL, Oracle, FlatFile)

Sample SOAP Webservices for Testing (FREE Public API)

If you like to test few more scenarios and you do not have SOAP service credentials then no worries. Try this link. It lists many FREE SOAP services which can be used for testing purpose without real credentials.

Example Request:

Here is sample SOAP API call to convert number into word (i.e. input=500, output=five hundred).
Enter Method, URL, Headers and Body like below on ZappySys UI

POST https://www.dataaccess.com/webservicesserver/NumberConversion.wso
Content-Type: text/xml; charset=utf-8



  
    
      500

Response



  
    
      five hundred

SOAP API Pagination (Loop through large dataset)

By default your SOAP API may not return full dataset if you have too many records to loop through then you can use Pagination techniques discussed in the following articles.

How to get data from Google AdWords using SSIS

Various Paging methods for REST API

Sending complex SOAP Request (Nested Parent-Child Array)

If you have to call more complex SOAP request which requires Parent-child structure (Nested Array) then you cant use Template Transform. In that case, check this article for real-world example.

Create nested XML for SOAP Request – Multiple inputs (POST data to API using SSIS Web API Destination)

Extract single XML node from SOAP Response

There will be a time when you need to extract just one value out of your response and save into SSIS variable / use it later on. If that’s the case then refer to this article.

Conclusion

Loading data from SOAP Api can be tricky and most of the time it requires use of SDKs or programming languages such as Java, C#, Python, Ruby etc. SSIS PowerPack comes with many RESTful API connectors including SOAP/XML Source and JSON Source. Try SSIS PowerPack yourself and discover many features not discussed in this article.

The post Calling SOAP Web Service in SSIS (XML Source) appeared first on ZappySys Blog.

How to get data from Google AdWords using SSIS

ZappySys — Mon, 13 Jun 2016 16:32:55 +0000

Introduction

UPDATE: ZappySys has released a brand new API Connector for Google AdWords which makes it much simpler to Read/Write Google AdWords Data in SSIS/ODBC compared to the steps listed in this article. You can still use steps from this article but if you are new to API or want to avoid learning curve with API then use newer approach.

Please visit this page to see all Pre-Configured ready to use API connectors which you can use in SSIS API Source / SSIS API Destination OR API ODBC Driver (for non-SSIS Apps such as Excel, Power BI, Informatica).

Google AdWords is one of the most popular Advertise Platforms. In this article, you will learn how to get data from Google AdWords by calling Google AdWords Web Service method and using drag and drop SSIS XML/SOAP connector (No coding required :)). Google AdWords Web Services allows retrieving various data such as account info, campaigns; as well as writing data, such as creating a campaign, add a keyword. Basically, it allows controlling your AdWords account programmatically instead of using the AdWords dashboard. For demo purposes, we will get data from Google AdWords Account and load it into Microsoft SQL Server (but you can use this technique to save any supported targets in SSIS (e.g. Excel, FlatFile, Oracle, MySQL, IBM DB2, etc.).

If you desire to download a report from AdWords, you must follow a different approach described in an article on how to download AdWords report. It describes how to download a report using ODBC and XML Driver but the concepts introduced there can be applied to SSIS as well.

If you are looking for a similar article for Google DoubleClick for Publisher (Google DFP) then check this article.

You will also learn how to create Google Ads or modify any other objects by calling SOAP API using SSIS REST API Task.

SSIS Tasks/Connectors discussed in this Article

In this article we will use following Connector(s) and Connection Manager to archive full data integration with Google AdWords.

Following SSIS Connectors support OAuth Connectivity. When you use below connectors no coding required to pull data from any OAuth enabled services (e.g. Twitter, Facebook, LinkedIn).

	REST API Task
	OAuth Connection Manager
	XML Source (REST API, SOAP Call or File data)

Two different ways to get Google AdWords Data in SSIS

There are two ways you can extract Google AdWords data in SSIS.

Use Google Analytics Source (For more info see this article) (Consume AdWords data using Google Analytics REST API)
Use XML Source along with OAuth connection manager (Consume AdWords data using SOAP API)

In this article we will only discuss Approach #2. For approach #1 refer see this link

Advantage of #2 is you get total control of full API set for AdWords but you have to do lot more manual work. For Approach #1 is lot simpler but you get access to limited reports (Dimensions and Metrics).

Google AdWords API Integration Example Screenshot

Below is screenshot of sample package discussed in this article (Download from here). As you can see we tried to demonstrate Reading data and Managing data (e.g. Create Ad) in the same package. You can extend these technique to perform any type of operations in Google AdWords or similar service such as DoubleClick for Publisher

Google AdWords API Integration Example – Create Ad, Get Accounts, Get Campaigns, Get Ad Groups, Get Ads (Extract AdWords data via SOAP API, Read data using Selector Method or use AWQL query language)

Step-By-Step Tutorial

Here are few high level steps you will have to perform to extract data from Google AdWords API using SSIS. This section assumes you have ClientID and Developer Token to access data. if you don’t have that then read next 2-3 sections.

Download and install SSIS PowerPack from here
Create new SSIS Project
From SSIS toolbox drag DataFlow Task and go to dataflow designer
From toolbox of drag ZS XML Source
Double click XML Source to configure it
Specify API URL from which you want to read data
Example : https://adwords.google.com/api/adwords/mcm/v201509/ManagedCustomerService
Now check Use Credentials check box and select New ZS-OAuth connection manager.
On connection manager UI you can select Google from Provider Dropdown.
Under Scopes enter following (Scope is basically permission)
```
https://www.googleapis.com/auth/adwords
```
Now click Generate Token and when prompted login and in the last step Accept.
If everything went ok then you will see AccessToken and RefreshToken populated.
Click Test connection and close the UI by clicking OK
Now on XML Source – Select HTTP Request Method as POST
In the Request Body click edit and enter request as below (Replace DEV_TOKEN and CLIENT_ID placeholders below)
```
	
		
			ENTER_YOUR_DEV_TOKEN_HERE
			ENTER_YOUR_CLIENT_ID_HERE
			ZappySys SSIS PowerPack v1
		
	
	
		
			
				CustomerId
				Name
			
		
	
```
If you are wondering how did we get above XML fragment for Request Body then Check this article on how to use SoapUI (free 3rd party tool) to create SOAP request Bodyfrom WSDL. Google AdWords has many API endpoints for different service. For each service you have different Api URL and different WSDL. In the next section you can learn how to create correct SOAP Body (for POST) using SoapUI tool.

For example if you are calling ManagedCustomerService API for version v201609 then your help page URL would be like this
https://developers.google.com/adwords/api/docs/reference/v201609/ManagedCustomerService

And on the same page you will see WSDL link below
https://ads.google.com/apis/ads/publisher/v201702/NetworkService?wsdl
(Just download that WSDL XML and save to local disk then use with SoapUI to generate XML body to submit for any DFP API call.)

See this link for more information (Its for Google DoubleClick Ad Platform but you will get an idea)
Go to Array handling tab at the bottom and enter text : ns2:entries – By doing this we will treat node as array eventhough only one entry found in response (By default content with multiple records treated as array).
Handling Array Element Nodes in SOAP Response
On the filter expression click edit and highlight entries node and click ok. This will produce something like below in filter textbox
```
$.soap:Envelope.soap:Body.ns2:getResponse.ns2:rval.ns2:entries[*]
```
Now click preview to see actual data.
Click OK to save UI. Now you can connect source to target such as SQL Server or Trash Destination.

NOTE: in above example we used v201605 API version but change it to latest if this API is no more valid. Google expires API version pretty often so keep that in mind if you get depreciated API version error.

Register for Developer Token to access AdWords account data using API

First important step to access any Google AdWords Account is request developer token. Click this link for step by step information. Watch short vide to get overview on entire process to access data via API.

Find your Client ID

Next thing you will need to know is your ClientID. Most of API Requests will require your ClientID. Perform following steps to find your ClientID.

How to find ClientID of your Google AdWords Account (Useful for making SOAP API Calls)

Connect to Google AdWords API using OAuth

Connecting to any Google API will require OAuth 2.0. SSIS PowerPack has Protocol so you can easily extract data from API without coding complex token extract/renew workflow. Such things usually requires some sort of programming but in our case you will do few clicks as described in Step-By-Step section earlier.

Here is the screenshot of OAuth Connection Manager.

Connecting to Google AdWords API using OAuth 2.0 in SSIS (Consuming XML/SOAP API)

Extract data from Google AdWords using SOAP API Calls

When you define filter expression you can check/uncheck “Include Parent Columns” setting. For AdWords SOAP Api you can uncheck so you only extract columns from rvals node in XML Response.

Extract data from Google AdWords SOAP/XML Response

SOAP Response Paging / Looping

Many time you will get response which is larger than default page size in AdWords API Call. In that case you can specify few pagination attributes on XML Source Paging tab. For more information on paging options you may refer this article . This article is talking about JSON Source but same options are applicable for XML Source too.

REST API Pagination – Passing PageNumber inside POST Body (Example: Google AdWords SOAP Api)

Configure REST API Pagination – using Page Number via POST Body Method

Read Google AdWords data and Load into SQL Server

Using XML Source you can extract data in tabular format and then you can map that to SQL Server Target (Use OLEDB Destination / ADO.net Destination).

Extract data from Google AdWords and load into SQL Server table using SSIS

Write data / Manage AdWords Account via REST API Calls

So far we have seen how to read data from AdWords account. But what if you have to create new records or write/update AdWords data via API Calls in automated manner? No worry – If you have need for that then use SSIS REST API Task which supports adhoc HTTP WebRequests including POST/DELETE/PUT etc. You can also upload files via REST API call using this task. This task supports rich error handling and saving your response into variable or files.

Create Ad under specific AdGroup

Below example shows how to create new ad under specific AdGroup using REST API Task

Notice we have used few placeholders to make things dynamic using SSIS Variables. Once REST API Task is executed you will see new ad created.

Creating Google AdWords Ad using SOAP/XML API in SSIS (REST API Task)

Verify Google Ad created programmatically using SOAP/XML API Call (SSIS REST API Task)

Methods for Creating new records

Google AdWords API supports mutate method for writing data. Many service endpoints supports this method. Below example request will create new Ad.



	
		
			{{User::varDevToken}}
			{{User::varClientId}}
			My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)
		
	
	
		
			
				ADD
				
	                30800928123
	                
	                    //zappysys.com/ad1
	                    //zappysys.com/ad1
	                    This is my Ad
	                    This is Line 1
	                    This is Line 2

Methods for Reading AdWords data

Google AdWords API supports two methods for reading data. Most of AdWords service endpoints supports GET method also known as Selector method (Its basically sending query in XML format). Seconds method is Query Method. It uses SQL like query language known as AWQL. (Read more about AWQL query language). If you are SQL Programmer then you may like AWQL Query method better than Selector. But either way you will be able to pass following information.

Columns you want to return
Filter (e.g. Where Clause)
Order By
Paging information (e.g. PageNumber, PageSize)

Get Method (XML Selector)

For Get Method using XML tags to define columns, filters and order by/paging information. Here is example Get method for below select query

SELECT Id, Heading
WHERE Status = 'ENABLED'
ORDER BY Heading DESC 
LIMIT 0,50

Get Request (Get all Active Ads)



	
		
			{{User::varDevToken}}
			{{User::varClientId}}
			My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)
		
	
	
		
			
			    Id
			    Headline
			    
			        Status
			        EQUALS
			        ENABLED
			    
			    
			        Headline
			        ASCENDING
			    
			    
			        0
			        50

Query Method (AWQL Query – SQL like query language)

Below is sample Soap request for AWQL Method to read Ads listed under from specific AdGroups. To learn more AWQL Query Syntax Click Here.

To learn more about which Columns you can use inside your AWQL Query Click here. Note that each service support different fields for Select Clause, Where Clause and Order By Clause. AWQL language can be also used to query Predefined AdWords Reports. The only difference is .. FROM clause is required when you query Reports.



	
		
			YOUR_DEV_TOKEN
			YOUR_CLIENT_ID
			My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)
		
	
	
		
			
SELECT Id, Name
WHERE Status = 'ENABLED'
LIMIT 0,5

Call AdWords Reports API – Download Campaign Performance data

There will be a time you like to download Reports from Adwords. For that read this article. Its showing how to use ZappySys ODBC Driver to call AdWords Reporting API but concept is same in SSIS Product too due to similar UI elements.

Download Example SSIS Package

Download SSIS 2012 / 2014/ 2016 Sample

Conclusion

Google AdWords data integration can be very challenging without right set of tools. SSIS PowerPack certainly makes it easy when you have to extract/manage Google AdWords data. Innovative drag and drop approach gives you various options to cover simple or most advanced scenarios without relying on any Programming language (e.g. C#, Java, Python, Ruby etc).

The post How to get data from Google AdWords using SSIS appeared first on ZappySys Blog.

How to register Google OAuth Application (Get ClientID and ClientSecret)

ZappySys — Sat, 09 Apr 2016 21:25:18 +0000

Introduction

In our previous article we understood little bit about OAuth 2.0 Protocol. We talked how to use Default App for ease of use. However In this post you will learn how to register Google OAuth Application so you can access various Google Services programmatically (e.g Youtube, Google Analytics, AdWords, DoubleClick etc.). Here is the link to find all services: Google API Services (Using OAuth 2.0)

Recently google started supporting OAuth 2.0 Protocol. There are many articles written explaining OAuth 2.0 protocol so we are not going to cover that in this article.

Following SSIS Connectors support OAuth Connectivity. When you use below connectors no coding required to pull data from OAuth enabled services.

SSIS Tasks / Components which Supports OAuth Connection / Google API

	API Source
	Google Analytics Source
	JSON Source Connector (File, REST API)
	REST API Webservice Task
	XML Source (File, REST, SOAP)
	Web API Destination

ODBC Drivers which Supports OAuth Connection / Google API (Call in Power BI, Excel, Informatica, MS Access)

	API ODBC Driver
	JSON ODBC Driver
	XML ODBC Driver
	CSV ODBC Driver

Create Google Project

Registering OAuth App in Google first requires you to create API Project. API Project is a container which can host many OAuth Apps or other API Keys.

To create API Project Visit https://console.developers.google.com/project
If you already have existing API Project then once you click above link you can pick desired API Project from dropdown (Found in the top bar). If you using existing API Project then you may skip next few steps and directly move to step which explains [how to create OAuth ClientId].
To create new project Click on “Create Project” button. Once you click Create it will take few seconds to create project for you.
Create New Google API Project (Google API – Register new OAuth App)
Now give some friendly name and change default Project ID (optional) for your project and click Create (See below).

NOTE: Before you click create you can change default Project ID suggested by Google by clicking Edit. This Project ID is used everywhere when calling many APIs like this one so copy this ID somewhere (You can also see this on project screen so dont worry if you miss it).Project ID is unique across google system so you may not get your preferred ID but if you can get one then its easy to remember for sure.

Create new Google Api Project for OAuth App (Set project name / edit default project ID)

Enable Google API for Integration

Once your Project is created, next step is enable APIs you like to call. By default most Google APIs are disabled so make sure you enable each APi category manually using below steps.

Once project is created it will redirect you to the API Home page where you will see many common Google API you can use.If you do not see that you can always click below option to enable API manually.Visit : https://console.cloud.google.com/apis/library
Enable Google APIs / Services
By default API you like to call might not be enabled. Search for API you wish to enable, click on the API name and click Enable.Some common Google API Examples:
1. To enable Google YouTube API
  > search for “YouTube“. It has few APIs so pick the one you need under this,
  > For example To manage/view channel use Data API, for Reporting use Analytics API)
2. To enable Google Drive API
  search for “Drive” and click Enable
3. To enable Google Sheets API
  search for “Sheet” (Some Sheets API operations need Drive API too see enable that as well)
4. To enable Google BigQuery API
  > search for “BigQuery“
5. To enable Google Analytics (GA3)
  > search “Analytics“ and enable Google Analytics API
6. To enable Google Analytics (GA4)
  > search for Google Analytics Data API and click Enable, Search for Google Analytics Admin API and click enable (needed this to query available properties)
  
  Search Google API for OAuth Application Access
  
  Enable Google API for OAuth Application Access

Create OAuth Credentials (Client ID / Secret)

Once we create project and enable APIs our next step is to obtain OAuth ClientID / Secret. Before you can obtain that you have to configure few properties of your OAuth Consent Screen. Most fields are optional on this screen but fill out them if you plan to publish this API App for other users (i.e. Public App).

Now click on the credentials tab and click “OAuth consent screen”. Fill out necessary information which will be visible to the user (such as App Name). Once you are done, continue on with credentials creation:Google OAuth API – Create OAuth App – Set Application Name on Consent Screen
Click on [Create credentials] button and select OAuth Client Id option.
Create OAuth Client ID Credentials for Google API access
Next you will be asked to select Application Type. Pick “Desktop app” if your Application is Desktop style application. If your app is web site or running browser then pick Web Application. Click on Create and you will see your ClientID and ClientSecret which you can use during OAuth Authentication. Keep these both information secret. This is used to call OAuth Approval Screen by your Desktop App or Web App… and then user can obtain Token which can be used to call desired Google API.
NOTE: Make sure you do not choose Web App and only select Desktop App. Desktop App doesnt have Redirect URL making it simple to setup and use with ZappySys software.

Create Google OAuth 2.0 Credentials – Desktop App

Generate ClientID and Client Secret for Google OAuth Application

Publish App

The last step in App creation is to go to the OAuth Consent Screen tab. Under Publishing Status, click PUBLISH APP to ensure your refresh token doesn’t expire often. If you plan to use the App for Private use, you do not have to worry about the Verification Status after Publishing (Which is the case for most users who access their own data).

Publish Google App (This avoids Refresh Token Expiration every 7 days)

Using Client ID and Client Secret in ZappySys OAuth Connection

You can use generated ClientID and ClientSecret to initiate OAuth Authorization process. We are not going to cover how that works, but below is a simple screenshot of how these two pieces of information are used to call the Consent screen with some desired permission requested by calling Application:

ZappySys OAuth Connection can be used by many components suggested in the beginning of this article. So refer them for more information.

Google OAuth Application: Generating Access and Refresh tokens using SSIS and OAuth Connection Manager.

Using System Default Browser to get the token

If embedded browser has some issue to extract your token then you can close the window by pressing [X] in the title bar. Once you do that you will be prompted to use System Default browser to finish the token extraction. If you click Yes then Full browser will be launched.

NOTE: Below feature is added in version published after Dec 31, 2019 (v3.0.0.11231 or later) so make sure you have latest version.

Using System Default Browser to extract OAuth Token

Untrusted App Warning

Sometimes Google might show you below warning. In order to continue the flow you might have to Click on Advanced and then click Go to YourAppName (unsafe) like below

How to ignore Google Unsafe App Warning (OAuth Flow)

OR, in some cases, you may see modern UI like below. In either case, choose continue.

Google App not Verified Warning (New UI)

Select the account

Using System Browser

Copy code and finish the flow

Manually Copy Code and Get OAuth Refresh / Access Token

Why I didn’t get refresh token back?

For some reason if you just see Access token and not refresh token then try following

Logout current session in browser and close all browser windows and try again
If you use Custom Provider rather than Google on OAuth connection then change Auth URL to include prompt and access_type like below
```
https://accounts.google.com/o/oauth2/auth?access_type=offline&prompt=consent
```

What to do if Google Refresh Token expires every 7 days

Technically, once you have a refresh token, you should not worry about renewing as long as the API call is made atleast once every few months (max 6 months idle time). However, in some cases, your refresh token can expire sooner, and you have to click Generate Token manually

One main reason that is not well documented is if the Google Oauth Private App you created to access your own data is still in “testing” mode and not “published,” then as per Google, you may get this error. Perform these steps to publish your app, which is in testing mode, so you do not have to renew the refresh token every 7 days.

Go to your Project in Google API console
Generally here
No API call has been made using the generated refresh_token for six months or longer
SOLUTION: Make at least some API in six months to keep refresh_token active
Your OAuth App is in “testing” status and not published yet
SOLUTION: Just click the Publish button on your App OAuth Consent screen. Do not worry about getting verified if you plan to use it for internal use; after this step, your refresh_token may not expire every 7 days See this thread.DO NOT WORRY ABOUT GETTING VERIFIED – Just publish it to avoid refresh token expiration every 7 days.

Publish Google App (This avoids Refresh Token Expiration every 7 days)

What is Google OAuth AccessToken

AccessToken is short lived token which you can pass along with each API Service call. once this token is expired you can get new AccessToken as long as you have obtained and saved RefreshToken (Returned to caller after very first time user go through Approval process on Consent screen). If you are using ZappySys OAuth connection manager

What is Google OAuth RefreshToken

May OAuth services support short lived token which expires after few minutes or hours. Google API by default expires token after 3600 seconds … So how to get new token ? Do you always go through that annoying Consent screen… what if your application is Automated Service? Well OAuth has solution for that too… its called RefreshToken.

First time to setup OAuth connection user have to go through Approval screen (i.e. Consent screen) once they click Approve button they may receive two types of token (AccessToken and RefreshToken). RefreshToken is used only when you need to get new AccessToken without going through Consent screen. So technically if OAuth supports RefreshToken (This is optional) then you can Save this piece of information in database somewhere and when you get token_expired error or you detect expiration time you can get new AccessToken without involving user consent (because you already did once).

Conclusion

Hope this article helped you to understand how to register Google OAuth Application. Download SSIS PowerPack to try various OAuth data integration scenarios using drag and drop high performance connectors. With SSIS PowerPack OAuth support you can virtually pull data from any API service (XML REST API or JSON REST API). Such as Facbook, Twitter, SurveyMonkey, LinkedIn and many more.

The post How to register Google OAuth Application (Get ClientID and ClientSecret) appeared first on ZappySys Blog.

How to do REST API Pagination in SSIS / ODBC Drivers

ZappySys — Fri, 08 Apr 2016 15:37:55 +0000

Introduction

In our previous blog, we saw how to call REST in SSIS including concepts of Authentication / Error Handling. Now in this post, we will cover API Pagination. You will learn concepts and patterns about REST API Pagination (for JSON / XML / SOAP or CSV API).

We will describe methods and steps to implement REST API Pagination in SSIS PowerPack or ODBC PowerPack API Drivers (for apps like Excel, Power BI, Informatica…) without coding. Both Products share a similar user interface with minor differences. For example purpose, we will show screenshots from SSIS PowerPack UI but concepts should remain same. You will also learn some techniques about how to implement an infinite loop in SSIS. Sometimes your REST API requires you to loop through all pages until no more data found then you can use pattern described in this article (sample package attached at the end).

In this article, we will use SSIS REST API Task and JSON Source Connector to process data from RESTful web service. If you have to process XML (or SOAP) data from REST API then simply use XML Source Connector for REST API, SOAP or File rather than JSON Source.

Debugging REST API Requests

Before we look at various aspects of REST API Pagination, we highly recommend you install FREE Web Debugging Tool Fiddler. Look at this article to understand how to debug REST API requests.

Method-1: REST API Pagination via URL Parameter

If you have paging requirement by specifying page number in URL then you can loop until no more pages found. Your REST API can indicate no more page found by any of the following ways.

Example URL: http://myserver.com/v1/api/getcustomers/?page=1

If no more data found for specified page number then

server may return an empty response
server may return the response with specific error and response status code

JSON Source Connector comes with flexible paging options to handle this scenario. See below screenshot how to configure this.

REST API Looping/Pagination via URL Page Number Parameter (Loop until last page detected)

URL with offset and limit

Sometimes you may have URLs with offset + limit parameters. as below. In this case, you can use the same technique as the above screenshot but just specify Page Num Indicator as offset and Increment by the same as your limit parameter (e.g. 1000 in this case). At runtime, your offset parameter will increment and generate URLs like below to fetch the next page.

If you don’t specify the offset parameter in the initial URL then offset=1 will be assumed as the default start. If your offset starts at 0 then refer to the next section for a workaround (i.e. Use URL Path Mode with StartAt variable = 0). Also if your increment for offset is other than the default page size then you must include the limit / pagesize parameter in the URL as below (e.g. limit=100 which is rows per page)

Example URLs:
http://myserver.com/v1/api/getcustomers/?offset=1&limit=1000
http://myserver.com/v1/api/getcustomers/?offset=1001&limit=1000
http://myserver.com/v1/api/getcustomers/?offset=2001&limit=1000

Method-2: REST API Pagination via URL Path

Sometimes your URL may contain a page or other identifier using path format.

Example URLs:
http://myserver.com/v1/api/getcustomers/1/
http://myserver.com/v1/api/getcustomers/2/
http://myserver.com/data/file1.json
http://myserver.com/data/file2.json

If no more data is found for the specified page number then

the server may return a 404 error (page not found)
server may return a response with a specific error and response status code

JSON Source Connector comes with flexible paging options to handle this scenario. See the below screenshot of how to configure this.

REST API Looping/Pagination via URL Path (Loop until last page detected)

URL with Skip Parameter (Offset starts at zero)

Now let’s look at another common pattern just like the offset we discussed earlier but in the Skip pattern, it starts at zero rather than offset=1. In the previous mode, we didn’t have the option to supply the initial value to start the counter but if you use Path mode then you can start the counter at some custom value. See the below screen to show how we can setup pagination with the skip method with custom increment.

If your increment for offset is other than the default page size then you must include page size parameter in the URL as below (e.g. pagesize=100 which is rows per page. Some APIs name this parameter in different way e.g. count=1000 or limit=100. Refer to the API documentation to know exact names)

Example URLs

http://mysite.com/api/getorders?skip=0&pagesize=100
http://mysite.com/api/getorders?skip=100&pagesize=100
http://mysite.com/api/getorders?skip=200&pagesize=100
http://mysite.com/api/getorders?skip=300&pagesize=100
....

REST API Pagination with Custom Skip / Increment Settings (Offset Starts from 0)

Method-3 : REST API Pagination via Response Attribute (Next Link or Cursor)

This method is not discussed in this article but click here to read more

There are mainly 3 ways you can get next link or cursor.

Full Link (e.g. nextPage : “https://mysite.com/api/v1/1/?filter=AAAA” )
Partial Link (e.g. nextPage : “/api/v1/getcustomer/1/?filter=AAAA” )
Cursor Continuation Token (e.g. nextCursor : “xxxxAAAbbbbCccc112233==” )

Next Link Method (Partial or Full URL)

Example response with next URL would be

{ 
  "value" : [....], 
  "odata.nextLink" : "http://abc.com/customers/2/" 
}

In this method you don’t specify page number in URL. If response include many records then you will get partial response along with indicator of next link.

For more information about this REST API Pagination please check below links. It explains in depth how to paginate REST API response with next page link found in response.

OData Paging using SSIS – REST API Paging Example

Read Twitter data in SSIS using REST API Task and JSON Source – OAuth2 Protocol

SSIS API Pagination – Next Link from Response Attribute

Next Link – Detect the last page based on empty rows (Regular Expression)

Now let’s look at a little bit different scenario where the Next Link is never empty even on the last page. This can cause an infinite loop if we configure it like the last section. So in such a case (see below example) when the last page still has the next link URL but the data row array is empty then use settings like the screenshot below.

First Page Example

{ 
  "value" : [....], 
  "odata.nextLink" : "http://abc.com/customers/2/" 
}

Last Page Example

{ 
  "value" : [], 
  "odata.nextLink" : "http://abc.com/customers/NNNN/" 
}

Example: Last Page

Basically, configure the following options

Set Stop indicator attribute to the same node which contains data rows (without array brackets). E.g. $.items
Set Stop indicator value / Regular Expression to something like
```
[]
```
assuming the last page has the same way. Check Fiddler Trace for exact text on the last page.

Last Page with non empty next link and empty data array (Detect Last Page based on zero row count)

Cursor Method

Many API returns cursor rather than link. Cursor method usually used to support a large number of records. Cursor is opened at server side with initial query information you submit and if cursor has more records you may get next cursor indicator in your response. In cursor method you have to supply Suffix for Next URL so after First URL your next URL will contain extra attribute which will include pagination information.

Suffix for next URL

Suffix for next URL can include special placeholders. There are two placeholders allowed. If you wish to append Next URL value in raw text then use <%nextlink%> but if your API needs value in encoded format e.g. A%20B rather than A+B then you have to use <%nextlink_encoded%>

Example URLs for Cursor:

Below is few sample URLs in the log when pagination occurs via Cursor method. As you see the first URL has Cursor=* means its first call, after first request server may send next cursor in the response body (see example)

{ 
  data: [....], 
  nextCursor: "Aooo/MxxxxxxxxxxxtdA==" 
}

First URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=*
Second URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=Aooo/MxxxxxxxxxxxtdA==
Third URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=Booo/MxxxxxxxxxxxtdA==
…..
…..

Method-4: Pass Page Number in Body Attribute

Some API allows you to pass your PageSize and Page number inside Request Body (i.e. POST data) in that case you may configure your options like below (We used Google AdWords Api as an example)

Configure REST API Pagination – using Page Number via POST Body Method

REST API Pagination – Passing PageNumber inside POST Body (Example: Google AdWords SOAP API)

Method-5: REST API Pagination via multiple URL parameters (e.g. Start and End)

In the previous section, you saw how to pass page numbers via URL (Single Parameter: http://abc.com/?page=1). However, sometimes you have a pagination requirement where you have to specify multiple parameters that need increment. See below example URL.

http://abc.com/?startRow=1&endRow=100
http://abc.com/?startRow=101&endRow=200
http://abc.com/?startRow=201&endRow=300
...

Pagination via Multiple URL Parameters

Method-6: REST API Pagination via Header (Next Link in Header)

Another popular pagination method is paginate via Header. In this method, each API response includes special Header which indicates next link for your response. In header method there are two approaches. [1] Custom Method [2] RFC 5988 Method. Lets look at both.

Next Link in Custom Header

In this method you have to specify which response header contains next URL. When no more NEXT url found in response reader stops.

Custom Header Pagination Method

Next Link in Header as per RFC 5988

In this method, HTTP response contains a special header (Link) which contains next link for our response. If no link found to read more data then reader stops. This method is described well here in RFC 5988. Check this GitHub API Example to understand more on this pagination method.

Link Header Pagination Method (RFC 5988)

Method-7: REST API Pagination using Cursor / Continuation Token from Response Header

There will be a time when your API sends some sort of Continuation Token in Response Header and you have to use that token to call next request. Some API needs this token passed as query string in Next URL and in some API you have to pass as request header. We will cover both cases in the following section.

Your API might call these tokens as “Cursor” too so we will use these both terms in the following section (They both mean same thing).

There are few ways these APIs pass Cursor to the next request

Pass Cursor via Request Header with same header name
Pass Cursor via Request Header with different header name (introduced in v4.2.3)
Pass multiple Response Headers via Request Headers (Azure DocumentDB style usecase)
Pass Cursor via URL Query String Parameter (introduced in v3.0 or later)

Lets see all approaches.

Pass Token via Request Header (Same header name)

Consider the API pagination use case in which you get some sort of continuation token in the response header and you take it and pass in the next request via request header (same header name).

Example:

First Request / Response:

------- Request ------
GET https://abc.com/api/getdata

------- Response ------
x-token: Nxdfabcd11111

{ some response data }

Second Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-token: Nxdfabcd11111

------- Response ------
x-token: Nxdfabcd22222

{ some response data }

……

Last Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-token: Nxdfabcd22222

------- Response ------

{ some response data }

For above pagination pattern that just enter the response header name in Page Num Indicator field as below

Pagination using response header continuation token – Pass as Request Header

All possible formats for Page Num Indicator is below

x-page-token
x-token1, x-token2
RequestUrlQueryName=ResponseHeaderName
header::RequestHeaderName=ResponseHeaderName
RequestUrlQueryName=ResponseHeaderName(regular-expression)

Pass Token via Request Header (Different header name)

Consider the API pagination use case just like previous one but only different is request header name is not same as response header name. For example rather than x-token in request we need to call it x-next-token.

Example:

First Request / Response:

------- Request ------
GET https://abc.com/api/getdata

------- Response ------
x-token: Nxdfabcd11111

{ some response data }

Second Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-next-token: Nxdfabcd11111

------- Response ------
x-token: Nxdfabcd22222

{ some response data }

……

Last Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-next-token: Nxdfabcd22222

------- Response ------

{ some response data }

For above pagination pattern that just enter the response header name in Page Num Indicator field as below

Pagination using response header continuation token – Pass as Request Header (Different name)

Pass multiple Response Headers via Request Headers (Azure DocumentDB style usecase)

Some API like Azure Cosmos DB uses Continuation Token via Header approach. In that case you can enter single header name or multiple headers like below (For multiple headers use comma)

REST API Pagination using Continuation Token via Header (Azure CosmosDB / DocumentDB API Example )

You can use use tools like Fiddler to see paginated requests as below.

First Request

View Paginated Requests in Fiddler (Azure CosmosDB API Example)

Second Request

Pass Next Page Token via Header (View Paginated Requests in Fiddler – Azure CosmosDB API Example)

Pass Token via URL Parameter (Query String)

In the new version we added a feature where you can actually read Continuation Token (i.e. Cursor) from Response Header and then pass to next request in the URL rather than Request Header. Here is the example API which uses such method.

First Request

--------- Request ---------
GET
http://test.com/api/getdata

--------- Response ---------
X-RESPONSE-TOKEN : Pg2xxxxxxxxxxxxxxxxx

{ data : [ 1,2,3.... 10 ] }

Second Request

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg2xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : Pg3xxxxxxxxxxxxxxxxx

{ data : [ 11,12,13.... 20 ] }

Here is how to configure above pagination pattern

Pass Response Header value to URL (Cursor)

Pass Token via URL Parameter (Query String) – with custom Regular expression (Usecase: Salesforce BULK query API)

Now consider previous scenario but with slightly different requirement. By default previous pagination pattern stops when specified response header is missing or response header value is blank. However is some cases header value contains custom string to indicate last page. For example x-token: NULL. In such case we need to use Page Num Indicator like below (must use v4.2.3 or higher in SSIS PowerPack / 1.5.2 or higher in ODBC PowerPack)

nextUrlParameterName=responseHeadername(Some_Regular_Expression)

For example to stop pagination in below case you have to enter Page Num Indicator as below. It says stop pagination when “NULL” word is detected

cursor=X-RESPONSE-TOKEN(^((?!NULL\b).)*$)

First Request

--------- Request ---------
GET
http://test.com/api/getdata

--------- Response ---------
X-RESPONSE-TOKEN : Pg2xxxxxxxxxxxxxxxxx

{ data : [ 1,2,3.... 10 ] }

Second Request

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg2xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : Pg3xxxxxxxxxxxxxxxxx

{ data : [ 11,12,13.... 20 ] }

……

Last Request

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg3xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : NULL

{ data : [ 21,22,23.... 30 ] }

Here is how to configure above pagination pattern

Pass Response Header value to URL (Cursor)

Method-8: Pass Pagination Token in Request Body

In previous section (Method-4: Pass Page Number in Body Attribute ) we saw how to pass simple numeric page counter in Body. This is fine for some API but if your API has base64 style text token then counter based method doesn’t work. Also if you have requirement to hide page attribute node in first request and send page counter node in second request body (Changing Request Body Pattern for First vs Next) then check next two sections.

Next two sections covers two API Pagination patterns to fetch 100 rows (1000 in each page) . Both requires to send token in Body but second section is more complex need (Different Body Attributes in first vs next requests )

Passing Page Token in Body (Blank Token in first request allowed)

Lets look at simple pattern where first request has blank token in Body but all other requests until we find last page we keep sending non-empty token. This approach works as long as API Provider doesn’t mind having Blank Token in first request. If you get error then check next section to handle more complex way.

Here is the example of API which needs token in body

First API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"" }

--------- Response ---------
{recordsFound: 5000, data: ["Acct1", "Acct2"..., "Acct1000"], nextToken:"AxBxCx112233001==" }

Second API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"AxBxCx112233001==" }

--------- Response ---------
{data: ["Acct1001", "Acct1002"..., "Acct2000"], nextToken:"AxBxCx112233002==" }

….. many more calls …

…. many more calls ….

Last API call (no more nextToken in response so STOP pagination)

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"AxBxCx112233009==" }

--------- Response ---------
{data: ["Acct9001", "Acct9002"..., "Acct10000"] }

So if you have such pattern then you can set few properties

For such issue you can use Changing few properties like below. Use Property Grid if you cant find on UI

Set EnablePageTokenForBody =
```
True
```
Set PagePlaceholders =
```
header=|;filter=|;
```

Set HttpRequestData =

{ searchFor: "Account", continueToken:"[$pagetoken$]" }

Setting Body with [$pagetoken$] to implement dynamic pagination

Setting Page Placeholders in Advanced Pagination tab.

Passing Page Token in Body (Blank Token in first request)

Now let’s look at more complex pattern. Here our first request body is different than other requests.

First API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account" }

--------- Response ---------
{recordsFound: 5000, data: ["Acct1", "Acct2"..., "Acct1000"], nextToken:"AxBxCx112233001==" }

Second API call

--------- Request ---------
POST http://test.com/api/getdataNext

{ continueToken:"AxBxCx112233001==" }

--------- Response ---------
{data: ["Acct1001", "Acct1002"..., "Acct2000"], nextToken:"AxBxCx112233002==" }

….. many more calls …

…. many more calls ….

Last API call (no more nextToken)

--------- Request ---------
POST http://test.com/api/getdataNext

{ continueToken:"AxBxCx112233009==" }

--------- Response ---------
{data: ["Acct9001", "Acct9002"..., "Acct10000"] }

So notice few things

First Request Body / URL is different than second / third and Nth API call
For first request we do not have token attribute but after first request we do have token attribute

We have this article and this article which describes similar pattern.

Now lets look at how to solve both problems

Passing PageToken or Counter in Next API Call Body

For such issue you can use Changing few properties like below. Use Property Grid if you cant find on UI

Set EnablePageTokenForBody =
```
True
```
Set PagePlaceholders =
```
header=|;filter=|;
```
Set HttpRequestData =
```
[$tag$]
```
Set HasDifferentNextPageInfo =
```
True
```
Set FirstPageBodyPart =
```
{ searchFor: "Account"}
```
Set NextPageBodyPart =
```
{ token:"[$pagetoken$]" }
```

So notice how we defined Body in two different properies as template. And in HttpRequestData we used

[$tag$]

Output Page Number as column

Sometimes you have use case where you want to Start Page Number from the last page loaded successfully. By default, you will see one additional column __FileName in the output. This column contains PageNumber from URL for each row. You can save this into database so when next time you run your SSIS you can start at MAX(PageNumber) + 1.

Outputting PageNumber option is not visible by default so Use Property Grid and look for OutputFileName property (Default=true)

Last Page Detection

Every API may have different way of telling you that you reached last page or no more data found so you should stop looping. SSIS PowerPack comes with many options so you can implement desired logic for last page detection.

Following approaches supported by last page detection. Below options are not applicable if you are using Method#3 (Response contains Next Link / Cursor)

Detect the last page by row count
Detect the last page by response status code
Detect the last page by error message (substring search)
Detect the last page by response size
Detect the last page based on response value or regular expression pattern

Now lets look at each setting. You should choose correct one based on your API behavior

Detect the last page by row count

If your API returns blank document or blank Array (if its JSON) then you can use this method to detect the last page. So when no content found in Response document it will stop pagination.

See below example of JSON API .. It shows sample first page and Sample last page (Assuming you entered Filter as $.Records[*] to extract data from Records array.

First Page

{ 
  Records : [ {...}, {...}, {...} ......... ]
}

Last Page

{ 
  Records : [  ]
}

SSIS API Pagination – Detect the last page by row count (Blank document or array)

Detect the last page by response status code

Sometimes you get error with status code when the last page is reached (Most common is 404). So rather than failure, you can treat that status code as last page indicator using this option (see below).

SSIS API Pagination – Detect last page by response status code

Detect the last page by the error message

If you cant use above two methods and want to detect the last page based on a certain error string in the response then use below approach.

SSIS API Pagination – Detect the last page by error/response string search

Detect the last page by response size

Sometimes you can detect the last page by response size. For example, your response contains “–” only if no more data is found then you cannot use any of the above approaches and you have to detect based on response size (e.g. less than 3 bytes). See below

SSIS API Pagination – Detect last page by response size

Detect the last page based on the response attribute value

There will be a time when you want to detect the last page based on some indicator value from your response. In that case, you can use the following settings to detect the last page.

Select Paging mode =Response Attribute Mode
Select Next Link Attribute (Example: $.next_page )
Select Stop Indicator Attribute (Example: $.hasMorePages )
Enter a value for Stop Indicator (Example: false )

If you don’t want to use Hardcoded value for stop indicator and want to use expression then see next section.

Detect the last page based on regular expression pattern

There will be a time when you want to detect the last page based on some indicator value from your response. In that case, you can use the following settings to detect the last page.

Select Paging mode =Response Attribute Mode
Select Next Link Attribute (Example: $.next_page )
Select Stop Indicator Attribute (Example: $.count )
Enter regex pattern (regular expression) for Stop Indicator (Example: regex=^\d{1,3}$ ). Stop when count attribute is less than 1000 in other word less than 4 digits.

Here is an example from Zendesk API (Stop pagination when count attribute from the response is less than 1000)

Pagination for Zendesk Incremental API – Next Link and Last Page detection using Stop Indicator Regular Expression

Handling API Rate Limit - Retry on too many API requests

Many API restricts you from calling too many API calls too soon (just to save their servers from being overwhelmed) - Example. Let's think this way if you fetching a million rows via paginated response and the very last page fails then you won't be happy because now you have to fetch everything again. To solve this ZappySys provides retry options on HTTP / OAuth and REST API connection managers.

Generally, when you exceed the rate limit, the server sends you error code 429 but it can be something else too or multiple codes are returned. In the below example, we use 503. Check your API documentation to know the exact code you should retry.

To set the retry options go to connection settings (e.g. HTTP / OAuth / REST connection). The below example is for OAuth connection.

To retry on multiple codes use Pipe as a delimiter. Example: 429|503

Retry Settings for API Rate Limit Exceeded Error (Status code 429 or other)

Performance Tips

You can improve performance of your API calls by doing a few changes (again not every API may support this)

Use the maximum page size possible (e.g. if you doing pagination you can set max possible page size – refer to your API documentation whats the default size and what is the max page size you can supply)
e.g. /myapi?pagesize=100 (parameter name can be something else like limit=100)

SSIS Infinite Loop

So now you know how to paginate automatically in JSON Source Connector but what if you want to use REST API Task for pagination? Some custom scenarios like web scraping require you to download all HTML pages until no more results found? In that case, you most likely need infinite looping pattern (Loop until the last page detected). To loop through pages until the last page is reached you can perform the following logic.

The last page can be detected based on any of the following ways
1. Check Response StatusCode
2. Check Error Message
3. Check Response Size (If less than X bytes then consider as the last page)

REST API Task support #1 and #2 scenarios. If you want to check bytes then use the expression in Expression Task or Script Task
(e.g. @exitloop = LEN(@responseData) <= 3 )

URL for Rest API Task (On Request Settings Tab)
When you configure REST API Task enter URL as below so its dynamic. You can use PageCounter variable anywhere in the URL.

//zappysys.com/downloads/files/test/cust-{{User::pagecounter}}.json

SSIS REST API Looping – Pass Page number in URL, SSIS Infinite Loop

Configure For Loop Task for Infinite Looping (Loop until last page found)

Download Sample SSIS Package

SSIS-Loop-Rest-API-2012.dtsx (Zip file)

Video Tutorial – Reading data from API in SSIS

In case you missed our previous article about calling REST API is SSIS then check below video.

Conclusion

It can be time consuming and steep learning curve to write C#/JAVA code to handle your rest api integration but using some Tasks and Connectors listed below can help you to solve big headache and provide clean drag and drop REST API integration in SSIS for virtually any REST API Source (e.g. Twitter, Facebook, Salesforce, office 365). Download SSIS PowerPack and try it out by yourself.

SSIS Components for REST API/JSON/XML/SOAP

	JSON Source Connector (File, REST API)
	Export JSON File Task
	JSON Parser Task
	XML Source Connector (File, SOAP, REST API)

The post How to do REST API Pagination in SSIS / ODBC Drivers appeared first on ZappySys Blog.