In this post you will learn how to call Amazon MWS API (Amazon Marketplace Web Service) or Amazon Product Advertising API using SSIS PowerPack.

Using drag and drop approach you can consume data from Amazon MWS XML Web service. In this post we will use ZappySys XML Source connector to read data from Amazon MWS API and load into SQL Server. We will also use REST API Task to call any API from Marketplace Web service and save output into Variable.

Prerequisites

This post assumes following things

1. Basic knowledge of SSIS and XML format.
2. Amazon MWS Account and and valid AWSAccessKeyId and Secret Key to call Amazon MWS API (How to obtain Key / Secret and SellerId ? ).
3. You have tested few API calls using Amazon MWS Scratchpad this will give you idea about various API you can call and parameters you need to pass.

Testing MWS Requests using ScratchPad

Very first step we recommend before you call MWS API in SSIS is to get familiar with Amazon MWS Scratchpad tool. Check this article for detailed steps.  You can also use Fiddler to debug some requests.

Calling Amazon MWS API in Scratchpad Testing Tool (Response Details Tab) – ListMatchingProducts Example

Call Amazon MWS API using SSIS

To consume data from any XML Source (File, SOAP Web Service or XML API) you can use  ZappySys XML Source
connector. or REST API Task. In below example you will see how to call Amazon MWS API and save output into Variable. With this approach you can call any API (GET / POST / DELETE / PUT)

1. Download and Install SSIS PowerPack
2. Create test package
3. From SSIS toolbox drag ZS REST API Task  and double click on the task to configure it
4. Select Request URL Access Mode = [Url from Connection]
5. Enter URL as below
   

   https://mws.amazonservices.com/Orders/2013-09-01

   In above URL
   ** mws.amazonservices.com => This is your endpoint. If you are not in USA then Click here to find correct endpoint
   ** /Orders => That’s your API you want to call
   ** /2013-09-01 => That’s your API version (You have to add this same version in your POST Body too (see next section).
6. In Select Url Connection dropdown select New OAUTH connection option.
7. When prompted on OAuth connection UI select Provider=Amazon MWS, Enter AWSAccessKey and Secret key and Click OK (If you have MWSAuthToken then enter in Access Token field else leave it empty). Do not test because it may not work yet.
   

   SSIS OAuth Connection – Call Amazon MWS API (Marketplace Web service API)

8. Another Setting you may turn on is Retry Handling. This allows to over come ThresholdLimitReached error when we call API too fast.
   

   Retry Options

9. Now select Method=POST, and in the body enter following
   

   Action=GetServiceStatus&SellerId=A77XXXXXXX&Version=2013-09-01

   If your API requires pagination (more than 100 records from response) then refer next section (Sample Package). In the case of Pagination your second request may look like below
   

   Action=YourActionNameWithNextToken&SellerId=A77XXXXXXX&Version=2013-09-01&NextToken={{User::vNextToken,FUN_URLENC}}

   NOTE: Enter all required parameters in the following format, as shown below. All values need to be URL-encoded. If you use SSIS variable, use it along with FUN_URLENC format specifier, e.g. {{User::myVariable,FUN_URLENC}}. If you use a hard-coded value, and not sure if it is URL-encoded, then encode it with the same FUN_URLENC format specifier/function, e.g. <<myValue,FUN_URLENC>>. Always check if SSIS configuration matches the one you see in MWS ScratchPad Request Details.
   Format:
   param1=value1&param2=<<value2,FUN_URLENC>>&param3={{User::myVar,FUN_URLENC}}
10. Now click Test and see content in the Preview window.
    

    SSIS REST API Task – Call Amazon MWS API – Get XML data save into SSIS variable

11. If you wish to save response data into SSIS variable then goto response Tab and check save option and select variable name.

Amazon MWS API Pagination

Many Amazon MWS API calls by default don’t return all records (Explained here). Rather than returning all rows you may get partial response (e.g. Max 50 or 100 rows). If you want to get all records then you have to use NextToken  found in your XML. For API which requires pagination, in that case your POST body may be different for 2nd or higher requests. Notice how we changed Action and appended NextToken=xxxxxxxxxxx in the second request. Also notice we used encoded token (URL encoded). This is very important because as per Amazon Specs this value has to be Url Encoded.

We also documented similar use case of Amazon MWS Pagination for Power BI (Check this one) its mostly same setup except 2-3 properties might be named different way.

Now lets look at sample request / response for paginated MWS requests (For clarity we have removed many common Parameters which are automatically added at runtime e.g. Signature, Timestamp). We use ListOrders (First Request) and ListOrderByNextToken (Second+ requests)

First Request

POST https://mws.amazonservices.com/Orders/2013-09-01

Action=ListOrders&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z

First Response

<?xml version="1.0"?>
<ListOrdersResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

Second Request or higher (MWS API with Pagination using NextToken)

POST https://mws.amazonservices.com/Orders/2013-09-01

Action=ListOrdersByNextToken&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z&NextToken=YOUR_ENCODED_TOKEN_FROM_PREV_REQUEST

Second or higher Response

<?xml version="1.0"?>
<ListOrdersByNextTokenResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersByNextTokenResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

Sample SSIS Package

If you want to see complete working package for pagination pattern then download from here. Change AccessKey, SecretKey, SellerID and Path as needed to make this package work.

Here is the screenshot of example SSIS package for Amazon MWS.

Amazon MWS APi Pagination (ListOrders , ListOrdersByNextToken Example)

Copy/Paste Properties

Here is copy/paste values for properties listed above.

DirectPath:  https://mws.amazonservices.com/Orders/2013-09-01
EnablePageTokenForBody: True
Filter:  $.ListOrders[$tag$]Response.ListOrders[$tag$]Result.Orders.Order[*]
HasDifferentNextPageInfo: True
HttpRequestData:  Action=ListOrders[$tag$]&SellerId=A1xxxxxxxx&CreatedAfter=2018-09-01T04%3A00%3A00Z&MarketplaceId.Id.1=ATVPDKIKX0DER
HttpRequestMethod:  POST
NextUrlAttribute: $.ListOrders[$tag$]Response.ListOrders[$tag$]Result.NextToken
NextUrlSuffix: &NextToken=<%nextlink_encoded%>
NetUrlWait: 10000 (ms) -- slow down to avoid throttling ... Max 6 requests per minute, increase if you get API Limit Error
PagePlaceholders:  body=|ByNextToken;filter=|ByNextToken

Understanding Pagination Properties

Here are various properties you have to set for Pagination.

POST /Orders/2013-09-01?AWSAccessKeyId=AKxxxxxxxxxx
  &Action=ListOrders
  &SellerId=A10zzzzzzz
  &SignatureVersion=2
  &Timestamp=2018-09-17T21%3A35%3A57Z
  &Version=2013-09-01
  &Signature=3GigipfRsQgzzzzzzzzzzzzzs%3D
  &SignatureMethod=HmacSHA256

Action=ListOrders[$tag$]&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z

<?xml version="1.0"?>
<ListOrdersResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

<?xml version="1.0"?>
<ListOrdersByNextTokenResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersByNextTokenResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

NextUrlEndIndicator=false
StopIndicatorAttributeOrExpr= $.GetReportRequestList[$tag$]Response.GetReportRequestList[$tag$]Result.HasNext

Amazon MWS API Pagination (For Old Version)

So in previous section we saw how to achieve pagination in one step (for newer version of SSIS PowerPack). But there will be a case you cannot use latest SSIS PowerPack or you want to take complete control on each aspect of pagination. In such case use below method (Depreciated).  In newer version we added many new properties so you dont have to do Loop pattern like below.

Sample SSIS Package

If you want to see complete working package for pagination pattern then download from here. Change AccessKey, SecretKey, SellerID and Path as needed to make this package work.

Here is the screenshot of example SSIS package for Amazon MWS.

SSIS Example Package – Read data from Amazon MWS Web service (API Call). Paginate API calls using NextToken

Load Amazon MWS API data into SQL Server using SSIS

In this section we will see how to use ZappySys XML Source connector to read data from Amazon MWS Web Service and load into SQL Server

For making things simple we are calling

1. Download and Install SSIS PowerPack
2. Create test package
3. From SSIS toolbox drag Data Flow task and double click task to go to Data Flow designer
4. Drag ZS XML Source from SSIS Toolbox
5. Configure XML Source as below
   

   SSIS XML Source – Read data from Amazon MWS API (Amazon Marketplace Web service call)

6. Drag OLEDB destination from SSIS Toolbox.
7. Connect XML Source to OLEDB Destination and map input column to target table
8. Execute dataflow
   

   Get data from Amazon MWS API – Save to File or SQL Server using SSIS

Working with Amazon MWS Feed API (e.g. Upload file)

Amazon MWS Provides Feed APIs which has slight different requirements. Important requirement in Feed file upload is you have to supply Content MD5 Hash. If you use SSIS PowerPack then you don’t have to worry about that complexity because calculating MD5 Hash is automatically done.

Upload Amazon MWS Feed File in TSV / CSV format (Tab separated)

Here is the screenshot for how to upload Tab separated values in CSV file. You can specify body from Variable or Read from file. If you read from file then must check File Upload/Multi-Part option and start path with @ symbol as below.

Call Amazon MWS Feed API – Upload CSV file format (Tab separated values – TSV)

Upload Amazon MWS Feed File in XML format

To upload feed file in XML format use same settings as above except Content Type should be XML (text/xml) (Select in the dropdown)

Calling Amazon Product Advertising API

If you have need to call Amazon Product Advertising API then also you can use above techniques (Use MWS Provider on OAuth connection) to call Product Advertise API. Both API (MWS API and Product Advertise API) use same signature hashing algorithm so its possible to use MWS Provider for Hashing. You may have to change your API URL parameters as needed when you call Amazon Product Advertising API. Basically when you call any MWS or Product Advertising API using GET method then OAuth connection manager automatically appends Signature to URL at runtime.

Import data from MWS Custom Reports (Inventory Report Example)

Now lets look at how to extract data from Custom Report API . Reading data from Custom Reports not single step process because report generation is Job style API. Which means you send report request and wait  until its done. Once Report ready you have to read in CSV format or XML. Some Reports only available in CSV format. Check this post to understand how complex it can be to get data. We will make it simple for you to understand this in 3 steps. Basically calling reports requires minimum 3 API calls (see below)

1. Create a new report request – Cal RequestReport API . It returns ReportRequestId (You can use it in the next step)
2. Check Report Status see its done – Call GetReportRequestList API (Pass ReportRequestId got in the previous step to get data for only one Report request we care). Keep checking Report Status every few seconds until you get _DONE_ status in response.
3. Read Report Data – Call GetReport. This API call returns CSV or XML data. So use correct component for this step. If CSV data is returned then we must use CSV Source rather than XML Driver.

Now lets see how to read Amazon MWS custom report in SSIS. In this example, We will get Inventory by calling _GET_MERCHANT_LISTINGS_DATA_ report type.

Download Sample Package for MWS Report

To make things simple we have created a demo package – ready to go.
Download SSIS 2012 Demo – Amazon_MWS_API_GetReport_2012.

Prepare demo package for first run

Once you download and extract above sample package. Import to your existing SSIS Solution. Its SSIS 2012 format so it may upgrade to higher version if you using SSIS 2014, 2016 or 2017.

Once package is opened perform following steps in before you can execute the package.

1. Enter MarketplaceId, SellerId in Variables. Double click OAuth Connection and enter your MWS Account Key and Secret.
2. Create c:\temp directory (This is where we will save final report file)
3. Run this package
4. Go to Execution log and Find ReportId (This you can hardcode in Variables for Design time Preview / Metadata changes etc)

See below package screenshot

Read data from Amazon MWS Report in SSIS (Get Inventory Listing Example)

Steps Explained

Now lets see each steps in depth. We broken whole process in 3 groups.

Create Report Request (Step1)

This is the first step how you initiate report request.  Use REST API Task to call this step. Here is raw request. Many items in body added at runtime (e.g. Signature, Timestamp).

• On Request tab see how we used SSIS Variables to get many values at runtime dynamically.
• On Response Tab we have to extract RequestId and save into variable.

POST 
https://mws.amazonservices.com/Reports/2009-01-01

AWSAccessKeyId=Axxxxxxxxx&Action=RequestReport&MarketplaceId=ATVPDKIKX0DER&ReportType=_GET_MERCHANT_LISTINGS_DATA_&SellerId=Axxxxxxxxx&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2019-06-05T21%3A23%3A31.090Z&Signature=2AbGQQ4xxxxxxxxxxxxxxxxxxxxjJzox%2FU%3D

RequestReport Action – Generate Amazon MWS Custom Report

Extract value from XML response using XPATH and save to Variable (Get RequestId Example)

Here is how to configure above task

Wait until data is ready (Step2)

Once we send RepotRequest and we have RequestId we can call step2 using REST API Task. In this Task we can use Status Check Feature. This is pretty handy feature because we dont have to implement Loop in SSIS. Basically we have to keep checking report until we get _DONE_ status in the response (ReportProcessingStatus node in XML)

Here is how to configure the task.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body:

   Action=GetReportRequestList&ReportRequestIdList.Id.1={{User::ReportRequestId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}

2. Response Setting Tab Configuration
   Response content type: Xml
   Filter Expression: 

   //*[local-name() = 'ReportProcessingStatus']  

3. Status Check Tab
   Check Enable Status
   Success Value: 

   _DONE_

   Check Every 5 Seconds

REST API Task – Status Check Feature (Wait until data is ready loop)

GetReportId (Step3)

Once above task finished we can call same request again but this time we have to configure slightly different way.

Here is how to configure the task.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body:

   Action=GetReportRequestList&ReportRequestIdList.Id.1={{User::ReportRequestId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}

2. Response Setting Tab Configuration
   Response content type: Xml
   Filter Expression: 

   //*[local-name() = 'GeneratedReportId']  

   Check Save to Variable: Save to User::ReportId
3. Status Check Tab
   Make Enable Status is unchecked

Get Report Data  (Step4,Step5 or Step6)

Now we are ready to ready our report. There are two ways we can do in SSIS.

• First approach (step 4,5) is download the report and save to local disk and then in the next step we can use data flow (CSV Source) to parse the saved file.
• Second approach (step6) is rather than saving to disk we can directly call API using CSV File source and parse content that way. Lets talk each approach briefly.

Step4,5 – Load Report From File

If you wish to save Report on disk then you can use REST API Task and call request like below.

Here is how to configure the task to download the file.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body: Action=GetReport&ReportId={{User::ReportId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}
2. Response Setting Tab Configuration
   Response content type: Xml
   Filter Expression: 

   //*[local-name() = 'GeneratedReportId']  

   Check Save to Variable: Save to User::ReportId
   
3. On Response Settings Tab : Set Response charset to Western European [Windows 1252]  (or directly set via Properties Grid to Windows-1252 under ResponseCharset Property)
   
4. Status Check Tab
   Make Enable Status is unchecked

To read download CSV file use settings described in below section except URL change it with local file path.

Step6 – Load report from URL

If you wish to load directly from URL then use same setting as above but in CSV Source inside data flow.

Here is how to configure CSV Source.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body:

   Action=GetReport&ReportId={{User::ReportId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}} 

2. General Tab
   Column Delimiter: {TAB}
3. On Encoding Tab : Set to Western European [Windows 1252]  (or directly set via Properties Grid to Windows-1252 under CharacterSet Property)

Conclusion

Amazon Marketplace Web service (MWS) provides great way to automate many functionality for Marketplace Seller. Using ZappySys SSIS PowerPack you con consume data from any REST API or Web service without replying on SDK / coding approach (e.g. C#, Java, Python, Ruby).

The post How to call Amazon MWS API using SSIS appeared first on ZappySys Blog.

In our previous blog, we saw how to call REST in SSIS including concepts of Authentication / Error Handling. Now in this post, we will cover API Pagination. You will learn concepts and patterns about REST API Pagination (for JSON / XML / SOAP or CSV API).

We will describe methods and steps to implement REST API Pagination in SSIS PowerPack or  ODBC PowerPack API Drivers (for apps like Excel, Power BI, Informatica…) without coding. Both Products share a similar user interface with minor differences. For example purpose, we will show screenshots from SSIS PowerPack UI but concepts should remain same. You will also learn some techniques about how to implement an infinite loop in SSIS. Sometimes your REST API requires you to loop through all pages until no more data found then you can use pattern described in this article (sample package attached at the end).

In this article, we will use SSIS REST API Task and JSON Source Connector to process data from RESTful web service. If you have to process XML (or SOAP) data from REST API then simply use XML Source Connector for REST API, SOAP or File rather than JSON Source.

Debugging REST API Requests

Before we look at various aspects of REST API Pagination, we highly recommend you install FREE Web Debugging Tool Fiddler. Look at this article to understand how to debug REST API requests.

Method-1: REST API Pagination via URL Parameter

If you have paging requirement by specifying page number in URL then you can loop until no more pages found. Your REST API can indicate no more page found by any of the following ways.

Example URL:  http://myserver.com/v1/api/getcustomers/?page=1

If no more data found for specified page number then

1. server may return an empty response
2. server may return the response with specific error and response status code

JSON Source Connector comes with flexible paging options to handle this scenario. See below screenshot how to configure this.

REST API Looping/Pagination via URL Page Number Parameter (Loop until last page detected)

URL with offset and limit

Sometimes you may have URLs with offset + limit parameters. as below. In this case, you can use the same technique as the above screenshot but just specify Page Num Indicator as offset and Increment by the same as your limit parameter (e.g. 1000 in this case). At runtime, your offset parameter will increment and generate URLs like below to fetch the next page.

Example URLs: 
http://myserver.com/v1/api/getcustomers/?offset=1&limit=1000
http://myserver.com/v1/api/getcustomers/?offset=1001&limit=1000
http://myserver.com/v1/api/getcustomers/?offset=2001&limit=1000

Method-2: REST API Pagination via URL Path

Sometimes your URL may contain a page or other identifier using path format.

Example URLs: 
http://myserver.com/v1/api/getcustomers/1/
http://myserver.com/v1/api/getcustomers/2/
http://myserver.com/data/file1.json
http://myserver.com/data/file2.json

If no more data is found for the specified page number then

1. the server may return a 404 error (page not found)
2. server may return a response with a specific error and response status code

JSON Source Connector comes with flexible paging options to handle this scenario. See the below screenshot of how to configure this.

REST API Looping/Pagination via URL Path (Loop until last page detected)

URL with Skip Parameter (Offset starts at zero)

Now let’s look at another common pattern just like the offset we discussed earlier but in the Skip pattern, it starts at zero rather than offset=1. In the previous mode, we didn’t have the option to supply the initial value to start the counter but if you use Path mode then you can start the counter at some custom value. See the below screen to show how we can setup pagination with the skip method with custom increment.

Example URLs

http://mysite.com/api/getorders?skip=0&pagesize=100
http://mysite.com/api/getorders?skip=100&pagesize=100
http://mysite.com/api/getorders?skip=200&pagesize=100
http://mysite.com/api/getorders?skip=300&pagesize=100
....

REST API Pagination with Custom Skip / Increment Settings (Offset Starts from 0)

Method-3 : REST API Pagination via Response Attribute (Next Link or Cursor)

This method is not discussed in this article but click here to read more

There are mainly 3 ways you can get next link or cursor.

1. Full Link (e.g. nextPage : “https://mysite.com/api/v1/1/?filter=AAAA” )
2. Partial Link  (e.g. nextPage : “/api/v1/getcustomer/1/?filter=AAAA” )
3. Cursor Continuation Token (e.g.   nextCursor : “xxxxAAAbbbbCccc112233==” )

Next Link Method (Partial or Full URL)

Example response with next URL would be

{ 
  "value" : [....], 
  "odata.nextLink" : "http://abc.com/customers/2/" 
}

In this method you don’t specify page number in URL. If response include many records then you will get partial response along with indicator of next link.

For more information about this REST API Pagination please check below links. It explains in depth how to paginate REST API response with next page link found in response.

OData Paging using SSIS – REST API Paging Example

Read Twitter data in SSIS using REST API Task and JSON Source – OAuth2 Protocol

SSIS API Pagination – Next Link from Response Attribute

Next Link – Detect the last page based on empty rows (Regular Expression)

Now let’s look at a little bit different scenario where the Next Link is never empty even on the last page. This can cause an infinite loop if we configure it like the last section. So in such a case (see below example) when the last page still has the next link URL but the data row array is empty then use settings like the screenshot below.

First Page Example

{ 
  "value" : [....], 
  "odata.nextLink" : "http://abc.com/customers/2/" 
}

Last Page Example

{ 
  "value" : [], 
  "odata.nextLink" : "http://abc.com/customers/NNNN/" 
}

Example: Last Page

Basically, configure the following options

• Set Stop indicator attribute to the same node which contains data rows (without array brackets). E.g. $.items
• Set Stop indicator value / Regular Expression to something like

  []

    assuming the last page has the same way. Check Fiddler Trace for exact text on the last page.

Last Page with non empty next link and empty data array (Detect Last Page based on zero row count)

Cursor Method

Many API returns cursor rather than link. Cursor method usually used to support a large number of records. Cursor is opened at server side with initial query information you submit and if cursor has more records you may get next cursor indicator in your response. In cursor method you have to supply Suffix for Next URL so after First URL your next URL will contain extra attribute which will include pagination information.

Suffix for next URL

Suffix for next URL can include special placeholders. There are two placeholders allowed. If you wish to append Next URL value in raw text then use <%nextlink%> but if your API needs value in encoded format e.g. A%20B rather than A+B then you have to use <%nextlink_encoded%>

Example URLs for Cursor:

Below is few sample URLs in the log when pagination occurs via Cursor method. As you see the first URL has Cursor=* means its first call, after first request server may send next cursor in the response body (see example)

{ 
  data: [....], 
  nextCursor: "Aooo/MxxxxxxxxxxxtdA==" 
}

First URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=*
Second URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=Aooo/MxxxxxxxxxxxtdA==
Third URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=Booo/MxxxxxxxxxxxtdA==
…..
…..

Method-4: Pass Page Number in Body Attribute

Some API allows you to pass your PageSize and Page number inside Request Body (i.e. POST data) in that case you may configure your options like below (We used Google AdWords Api as an example)

Configure REST API Pagination – using Page Number via POST Body Method

REST API Pagination – Passing PageNumber inside POST Body (Example: Google AdWords SOAP API)

Method-5: REST API Pagination via multiple URL parameters (e.g. Start and End)

In the previous section, you saw how to pass page numbers via URL (Single Parameter: http://abc.com/?page=1). However, sometimes you have a pagination requirement where you have to specify multiple parameters that need increment. See below example URL.

http://abc.com/?startRow=1&endRow=100
http://abc.com/?startRow=101&endRow=200
http://abc.com/?startRow=201&endRow=300
...

Pagination via Multiple URL Parameters

Method-6: REST API Pagination via Header (Next Link in Header)

Another popular pagination method is paginate via Header. In this method, each API response includes special Header which indicates next link for your response. In header method there are two approaches. [1] Custom Method [2] RFC 5988 Method. Lets look at both.

Next Link in Custom Header

In this method you have to specify which response header contains next URL. When no more NEXT url found in response reader stops.

Custom Header Pagination Method

Next Link in Header as per RFC 5988

In this method, HTTP response contains a special header (Link) which contains next link for our response. If no link found to read more data then reader stops. This method is described well here in RFC 5988. Check this GitHub API Example to understand more on this pagination method.

Link Header Pagination Method (RFC 5988)

Method-7: REST API Pagination using Cursor / Continuation Token from Response Header

There will be a time when your API sends some sort of Continuation Token in Response Header and you have to use that token to call next request. Some API needs this token passed as query string in Next URL and in some API you have to pass as request header. We will cover both cases in the following section.

Your API might call these tokens as “Cursor” too so we will use these both terms in the following section (They both mean same thing).

There are few ways these APIs pass Cursor to the next request

1. Pass Cursor via Request Header with same header name
2. Pass Cursor via Request Header with different header name (introduced in v4.2.3)
3. Pass multiple Response Headers via Request Headers (Azure DocumentDB style usecase)
4. Pass Cursor via URL Query String Parameter (introduced in v3.0 or later)

Lets see all approaches.

Pass Token via Request Header (Same header name)

Consider the API pagination use case in which you get some sort of continuation token in the response header and you take it and pass in the next request via request header (same header name).

Example:

First Request / Response:

------- Request ------
GET https://abc.com/api/getdata

------- Response ------
x-token: Nxdfabcd11111

{ some response data }

Second Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-token: Nxdfabcd11111

------- Response ------
x-token: Nxdfabcd22222

{ some response data }

……

……

Last Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-token: Nxdfabcd22222

------- Response ------

{ some response data }

For above pagination pattern  that just enter the response header name in Page Num Indicator field as below

Pagination using response header continuation token – Pass as Request Header

All possible formats for Page Num Indicator is below

x-page-token
x-token1, x-token2
RequestUrlQueryName=ResponseHeaderName
header::RequestHeaderName=ResponseHeaderName
RequestUrlQueryName=ResponseHeaderName(regular-expression)

Pass Token via Request Header (Different header name)

Consider the API pagination use case just like previous one but only different is request header name is not same as response header name. For example rather than x-token in request we need to call it x-next-token.

Example:

First Request / Response:

------- Request ------
GET https://abc.com/api/getdata

------- Response ------
x-token: Nxdfabcd11111

{ some response data }

Second Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-next-token: Nxdfabcd11111

------- Response ------
x-token: Nxdfabcd22222

{ some response data }

……

……

Last Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-next-token: Nxdfabcd22222

------- Response ------

{ some response data }

For above pagination pattern  that just enter the response header name in Page Num Indicator field as below

Pagination using response header continuation token – Pass as Request Header (Different name)

Pass multiple Response Headers via Request Headers (Azure DocumentDB style usecase)

Some API like Azure Cosmos DB uses Continuation Token via Header approach. In that case you can enter single header name or multiple headers like below (For multiple headers use comma)

REST API Pagination using Continuation Token via Header (Azure CosmosDB / DocumentDB API Example )

You can use use tools like Fiddler to see paginated requests as below.

First Request

View Paginated Requests in Fiddler (Azure CosmosDB API Example)

Second Request

Pass Next Page Token via Header (View Paginated Requests in Fiddler – Azure CosmosDB API Example)

Pass Token via URL Parameter (Query String)

In the new version we added a feature where you can actually read Continuation Token (i.e. Cursor) from Response Header and then pass to next request in the URL rather than Request Header. Here is the example API which uses such method.

First Request

--------- Request ---------
GET
http://test.com/api/getdata

--------- Response ---------
X-RESPONSE-TOKEN : Pg2xxxxxxxxxxxxxxxxx

{ data : [ 1,2,3.... 10 ] }

Second Request 

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg2xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : Pg3xxxxxxxxxxxxxxxxx

{ data : [ 11,12,13.... 20 ] }

Here is how to configure above pagination pattern

Pass Response Header value to URL (Cursor)

Pass Token via URL Parameter (Query String) – with custom Regular expression (Usecase: Salesforce BULK query API)

Now consider previous scenario but with slightly different requirement. By default previous pagination pattern stops when specified response header is missing or response header value is blank. However is some cases header value contains custom string to indicate last page. For example x-token: NULL. In such case we need to use Page Num Indicator like below (must use v4.2.3 or higher in SSIS PowerPack / 1.5.2 or higher in ODBC PowerPack)

nextUrlParameterName=responseHeadername(Some_Regular_Expression)

For example to stop pagination in below case you have to enter Page Num Indicator as below. It says stop pagination when  “NULL” word is detected

cursor=X-RESPONSE-TOKEN(^((?!NULL\b).)*$)

First Request

--------- Request ---------
GET
http://test.com/api/getdata

--------- Response ---------
X-RESPONSE-TOKEN : Pg2xxxxxxxxxxxxxxxxx

{ data : [ 1,2,3.... 10 ] }

Second Request 

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg2xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : Pg3xxxxxxxxxxxxxxxxx

{ data : [ 11,12,13.... 20 ] }

……

……

Last Request 

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg3xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : NULL

{ data : [ 21,22,23.... 30 ] }

Here is how to configure above pagination pattern

Pass Response Header value to URL (Cursor)

Method-8: Pass Pagination Token in Request Body

In previous section (Method-4: Pass Page Number in Body Attribute ) we saw how to pass simple numeric page counter in Body. This is fine for some API but if your API has base64 style text token then counter based method doesn’t work. Also if you have requirement to hide page attribute node in first request and send page counter node in second request body (Changing Request Body Pattern for First vs Next)  then check next two sections.

Next two sections covers two API Pagination patterns to fetch 100 rows (1000 in each page) . Both requires to send token in Body but second section is more complex need (Different Body Attributes in first vs next requests )

Passing Page Token in Body (Blank Token in first request allowed)

Lets look at simple pattern where first request has blank token in Body but all other requests until we find last page we keep sending non-empty token. This approach works as long as API Provider doesn’t mind having Blank Token in first request. If you get error then check next section to handle more complex way.

Here is the example of API which needs token in body

First API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"" }

--------- Response ---------
{recordsFound: 5000, data: ["Acct1", "Acct2"..., "Acct1000"], nextToken:"AxBxCx112233001==" }

Second API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"AxBxCx112233001==" }

--------- Response ---------
{data: ["Acct1001", "Acct1002"..., "Acct2000"], nextToken:"AxBxCx112233002==" }

….. many more calls …

…. many more calls ….

Last API call (no more nextToken in response so STOP pagination)

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"AxBxCx112233009==" }

--------- Response ---------
{data: ["Acct9001", "Acct9002"..., "Acct10000"] }

So if you have such pattern then you can set few properties

For such issue you can use Changing few properties like below. Use Property Grid if you cant find on UI

• Set EnablePageTokenForBody  =

  True

• Set PagePlaceholders =

  header=|;filter=|;

• Set HttpRequestData = 

  { searchFor: "Account", continueToken:"[$pagetoken$]" }

Setting Body with [$pagetoken$] to implement dynamic pagination

Setting Page Placeholders in Advanced Pagination tab.

Passing Page Token in Body (Blank Token in first request)

Now let’s look at more complex pattern. Here our first request body is different than other requests.

First API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account" }

--------- Response ---------
{recordsFound: 5000, data: ["Acct1", "Acct2"..., "Acct1000"], nextToken:"AxBxCx112233001==" }

Second API call

--------- Request ---------
POST http://test.com/api/getdataNext

{ continueToken:"AxBxCx112233001==" }

--------- Response ---------
{data: ["Acct1001", "Acct1002"..., "Acct2000"], nextToken:"AxBxCx112233002==" }

….. many more calls …

…. many more calls ….

Last API call (no more nextToken)

--------- Request ---------
POST http://test.com/api/getdataNext

{ continueToken:"AxBxCx112233009==" }

--------- Response ---------
{data: ["Acct9001", "Acct9002"..., "Acct10000"] }

So notice few things

• First Request Body / URL is different than second / third and Nth API call
• For first request we do not have token attribute but after first request we do have token attribute

We have this article and this article which describes similar pattern.

Now lets look at how to solve both problems

Passing PageToken or Counter in Next API Call Body

For such issue you can use Changing few properties like below. Use Property Grid if you cant find on UI

• Set EnablePageTokenForBody  =

  True

• Set PagePlaceholders =

  header=|;filter=|;

• Set HttpRequestData = 

  [$tag$]

• Set HasDifferentNextPageInfo = 

  True

• Set FirstPageBodyPart = 

  { searchFor: "Account"}

• Set NextPageBodyPart = 

  { token:"[$pagetoken$]" }

So notice how we defined Body in two different properies as template. And in HttpRequestData we used

[$tag$]

Output Page Number as column

Sometimes you have use case where you want to Start Page Number from the last page loaded successfully. By default, you will see one additional column __FileName in the output. This column contains PageNumber  from URL for each row. You can save this into database so when next time you run your SSIS you can start at MAX(PageNumber) + 1.

Outputting PageNumber option is not visible by default so Use Property Grid and look for OutputFileName property (Default=true)

Last Page Detection

Every API may have different way of telling you that you reached last page or no more data found so you should stop looping. SSIS PowerPack comes with many options so you can implement desired logic for last page detection.

Following approaches supported by last page detection. Below options are not applicable if you are using Method#3 (Response contains Next Link / Cursor)

1. Detect the last page by row count
2. Detect the last page by response status code
3. Detect the last page by error message (substring search)
4. Detect the last page by response size
5. Detect the last page based on response value or regular expression pattern

Now lets look at each setting. You should choose correct one based on your API behavior

Detect the last page by row count

If your API returns blank document or blank Array (if its JSON) then you can use this method to detect the last page. So when no content found in Response document it will stop pagination.

See below example of JSON API .. It shows sample first page and Sample last page (Assuming you entered Filter as $.Records[*] to extract data from Records array.

First Page

{ 
  Records : [ {...}, {...}, {...} ......... ]
}

Last Page

{ 
  Records : [  ]
}

SSIS API Pagination – Detect the last page by row count (Blank document or array)

Detect the last page by response status code

Sometimes you get error with status code when the last page is reached (Most common is 404). So rather than failure, you can treat that status code as last page indicator using this option (see below).

SSIS API Pagination – Detect last page by response status code

Detect the last page by the error message

If you cant use above two methods and want to detect the last page based on a certain error string in the response then use below approach.

SSIS API Pagination – Detect the last page by error/response string search

Detect the last page by response size

Sometimes you can detect the last page by response size. For example, your response contains “–” only if no more data is found then you cannot use any of the above approaches and you have to detect based on response size (e.g. less than 3 bytes). See below

SSIS API Pagination – Detect last page by response size

Detect the last page based on the response attribute value

There will be a time when you want to detect the last page based on some indicator value from your response. In that case, you can use the following settings to detect the last page.

• Select Paging mode =Response Attribute Mode
• Select Next Link Attribute (Example: $.next_page )
• Select Stop Indicator Attribute (Example: $.hasMorePages )
• Enter a value for Stop Indicator (Example: false )

If you don’t want to use Hardcoded value for stop indicator and want to use expression then see next section.

Detect the last page based on regular expression pattern

There will be a time when you want to detect the last page based on some indicator value from your response. In that case, you can use the following settings to detect the last page.

• Select Paging mode =Response Attribute Mode
• Select Next Link Attribute (Example: $.next_page )
• Select Stop Indicator Attribute (Example: $.count )
• Enter regex pattern (regular expression) for Stop Indicator (Example: regex=^\d{1,3}$ ). Stop when count attribute is less than 1000 in other word less than 4 digits.

Here is an example from Zendesk API (Stop pagination when count attribute from the response is less than 1000)

Pagination for Zendesk Incremental API – Next Link and Last Page detection using Stop Indicator Regular Expression

Handling API Rate Limit - Retry on too many API requests

Many API restricts you from calling too many API calls too soon (just to save their servers from being overwhelmed) - Example. Let's think this way if you fetching a million rows via paginated response and the very last page fails then you won't be happy because now you have to fetch everything again. To solve this ZappySys provides retry options on HTTP / OAuth and REST API connection managers.

Generally, when you exceed the rate limit, the server sends you error code 429 but it can be something else too or multiple codes are returned. In the below example, we use 503. Check your API documentation to know the exact code you should retry.

To set the retry options go to connection settings (e.g. HTTP / OAuth / REST connection). The below example is for OAuth connection.

To retry on multiple codes use Pipe as a delimiter. Example: 429|503

Retry Settings for API Rate Limit Exceeded Error (Status code 429 or other)

Performance Tips

You can improve performance of your API calls by doing a few changes (again not every API may support this)

• Use the maximum page size possible (e.g. if you doing pagination you can set max possible page size – refer to your API documentation whats the default size and what is the max page size you can supply)
  e.g.  /myapi?pagesize=100  (parameter name can be something else like limit=100)

SSIS Infinite Loop

So now you know how to paginate automatically in JSON Source Connector but what if you want to use REST API Task for pagination? Some custom scenarios like web scraping require you to download all HTML pages until no more results found? In that case, you most likely need infinite looping pattern (Loop until the last page detected). To loop through pages until the last page is reached you can perform the following logic.

The last page can be detected based on any of the following ways
1. Check Response StatusCode
2. Check Error Message
3. Check Response Size (If less than X bytes then consider as the last page)

REST API Task support #1 and #2 scenarios. If you want to check bytes then use the expression in Expression Task or Script Task
(e.g. @exitloop = LEN(@responseData) <= 3 )

URL for Rest API Task (On Request Settings Tab)
When you configure REST API Task enter URL as below so its dynamic. You can use PageCounter variable anywhere in the URL.

//zappysys.com/downloads/files/test/cust-{{User::pagecounter}}.json

SSIS REST API Looping – Pass Page number in URL, SSIS Infinite Loop

Configure For Loop Task for Infinite Looping (Loop until last page found)

Download Sample SSIS Package

SSIS-Loop-Rest-API-2012.dtsx (Zip file)

Video Tutorial – Reading data from API in SSIS

In case you missed our previous article about calling REST API is SSIS then check below video.

Conclusion

It can be time consuming and steep learning curve to write C#/JAVA code to handle your rest api integration but using some Tasks and Connectors listed below can help you to solve big headache and provide clean drag and drop REST API integration in SSIS for virtually any REST API Source (e.g. Twitter, Facebook, Salesforce, office 365). Download SSIS PowerPack and try it out by yourself.

SSIS Components for REST API/JSON/XML/SOAP

	JSON Source Connector (File, REST API)
	Export JSON File Task
	JSON Parser Task
	XML Source Connector (File, SOAP, REST API)

The post How to do REST API Pagination in SSIS / ODBC Drivers appeared first on ZappySys Blog.

In this article you will learn how to get data from Facebook using Graph API (OAuth 2.0) using SSIS PowerPack REST api Connectors. Facebook provides rich set of REST API to access data. Their REST API is also known as Facebook Graph API and it uses OAuth 2.0 Authorization. Traditionally any REST API Integration scenarios in SSIS requires you to use some sort of SDKs using popular programming languages (e.g. C#, Java, Ruby, Python). If you are DBA or SSIS Developer and you prefer to use clean drag and drop approach rather complex programming then this article will show you how to achieve it.

SSIS PowerPack REST API Connectors can be used to consume/manage Facebook data using REST API. Following connectors will be used in this article.

	REST API Task
	OAuth Connection Manager
	JSON Source (File, REST, OData)

Register Facebook OAuth App for Graph API access

Very first step to access Facebook API is register your custom OAuth App under Facebook developer portal. Check this article for more information on this topic.

Consume data from Facebook using REST API in SSIS

In this section we will guide you how to create OAuth connection and consume data from Facebook in few clicks using JSON Source (File, REST, OData) . We will load Facebook data into SQL Server.

Find Facebook PageID from its Name

In our case for example we want to Extract Page Insights for some page (e.g. facebook.com/MyPage ). To findout pageid perform following steps

1. Navigate to https://developers.facebook.com/tools/explorer
2. Click Get Token button, select permission you really care (To get PageID you don’t need to check anything).
3. Type /YourPageName in the API Url bar
4. Now click Submit. You will see response json which will have pagename and pageID
   

   Using Facebook Graph API Explorer, Test API, Find Object ID / PageID from Name

Create Facebook OAuth Connection

You can create SSIS OAuth connection manager two ways. 1) From Add Connection manager option from designer or Click Add ZS-OAuth from Component UI. Lets use first method.

1. In the SSIS Designer in connection manager pane .. right click and new connection … select ZS-OAUTH Connection from list
2. Select Facebook provider from dropdown
3. Select Custom OAuth App option
4. Enter Facebook OAuth App ID and OAuth App Secret
5. Select Permission (i.e. Scope) or type by hand (1 per each line)
6. Click generate Token. You will be redirected to login page and then it will show Authorize App confirmation. Click Accept.
7. Test Connection (If things go well it will show green success)
   

   Testing Facebook OAuth Connectivity in SSIS (Use Graph API)

Get Page Access Token (optional)

Accessing data of your Facebook page requires to generate a Page Access Token, follow the steps below to get it. If you are using non-page API methods, then you can freely skip this step. In SSIS Designer drag and drop REST API Task.

1. Then use the same OAuth connection we created in the previous step.
2. Proceed with setting the URL using this format:
   https://graph.facebook.com/v2.5/REPLACE_WITH_PAGE_ID?fields=access_token
3. Click Test Request/Response button and then copy the Page Access Token to a Notepad.

Getting Facebook Page Access Token using Facebook Graph API and SSIS

Get Long-Lived Access Token

Most likely, once finished, you will deploy the SSIS package to your SQL Server and then set a schedule to run it daily or hourly. With ordinary Access Tokens you will run into a problem after 30 or 60 days because these tokens will expire after a similar time and your SQL Job will fail. To avoid that, we can generate a Long-Lived Access Token which will live for 60 days, as long as you use that token (and running the package is using it). Perform these steps to generate a Long-Lived Access Token:

1. Firstly, you will need to find out your Facebook User ID. To do that, call this URL in Facebook Graph Explorer or using REST API Task with configured OAuth Connection Manager:
   https://graph.facebook.com/v2.5/me
2. Copy the ID from the response.
3. Then drag and drop another REST API Task and configure it like this, if you want to generate a Long-Lived Page Access Token:
   

   Getting Facebook Long-Lived Page Access Token using Facebook Graph API and SSIS

   Otherwise, if you want to generate a non-page Long-Lived Access Token, use this configuration:

   

   Getting Facebook Long-Lived Access Token using Facebook Graph API and SSIS

Use JSON/REST API Source to read from Graph API

Once you have OAuth connection manager. You can perform following steps to Extract data from Graph API Url. Facebook Graph API uses ObjectID in the URL so make sure you have that handy for whatever resource you want to access.

Configure JSON/REST API Source

Once you know pageid we can build our API URL. Lets assume your pageid is 12345678 and you want to extract page insight data via API call. In that case you may build your URL following way. You can change v2.5 to something else depending on API version you wish to use. If you omit version number from URL then default version will be used by Facebook API (Default API version can be found on Dashboard of your OAuth App).

https://graph.facebook.com/v2.5/YOUR_PAGE_ID/insights/page_impressions_organic_unique

Now perform the following steps to configure JSON Source to extract data from Facebook API.

NOTE: Below steps assume that you already Downloaded and Installed SSIS PowerPack

1. Drag Data Flow Task from SSIS Toolbox and double click to view Dataflow designer
2. Drag ZS JSON Source from SSIS Toolbox. Double click to configure.
3. In the AccessMode select “File Path or Web URL”
4. Enter URL:
   https://graph.facebook.com/v2.5/YOUR_PAGE_ID/insights/page_impressions_organic_unique?access_token=REPLACE_WITH_LONG_LIVED_ACCESS_TOKEN
   NOTE: You may use variables and parameters in the URL e.g. http://mydomain/{{User::MyPath}}:
   

   Getting Facebook Page data using Facebook Graph API and SSIS

5. Click on Select Filter and select values node and click OK. It may produce filter expression like below.
6. You can set Max Rows option to desired value if you don’t want to extract all rows (0=Unlimited)
7. Goto Paging Tab and enter $.paging.next in the “Next Cursor” field as below
   

   Getting page data from Facebook and paginating through the results using SSIS

8. Click on Columns tab if you wish to see/change metadata (You can change size / datatype there and Lock metadata to prevent auto refresh)
9. Now click Preview to see sample data and then exit preview and click OK to save UI changes

Load Facebook data into SQL Server

If you wish to save Facebook data into SQL Server or any other target like Oracle, MySQL, FlatFile, Excel, etc., then you can use some native SSIS Connectors (e.g. OLEDB Destination, ADO.net Destination). Connect JSON Source to Destination and map source columns to target.

Facebook API Paging through dataset (i.e. Loop, Cursor)

In the previous section, you saw how to adjust few attributes so JSON Source can paginate through Facebook API response. To learn more about REST API Paging Methods – Click this link

Here is another link from Facebook Documentation for Paging

Manage/Write/Update Facebook data using REST API

If you wish to create new POST  or manage data using REST API then you most likely need to use REST API Task rather than JSON Source. REST API Task also supports uploading media files.

Deployment to Production

You may also check the article on how to make URL dynamic/parametric, so that you can pass Long-Lived Access Token as an argument in a SQL Job.

Conclusion

Facebook data integration can be quite challenging without right set of tools. SSIS PowerPack provides complete toolset for any REST API integration related scenarios. Try it yourself and find out all hidden features not discussed in this article.

References

Facebook Graph REST API reference link

https://developers.facebook.com/tools/explorer

https://developers.facebook.com/docs/facebook-login/access-tokens/refreshing/

Similar articles

Read Twitter data in SSIS using REST API Task and JSON Source – OAuth2 Protocol

How to get data from Google AdWords using SSIS

The post How to get data from Facebook in SSIS using REST API appeared first on ZappySys Blog.

In this article, we will learn how to utilize the SSIS JSON Source Component to retrieve JSON data from an OData service and perform OData pagination on significant results. According to the OData specification, data can be presented in either JSON or XML format. For the XML format, refer to this article. This article primarily covers the OData JSON API example.

REST API Pagination concept

Most of the REST API Services limit the size of the dataset returned in a single request. When more data is found, it simply includes a pointer to the following result set. If you wish to fetch all data (e.g., loop through), you must check the following result set indicator. If it is not null, you can continue fetching until all data is retrieved. This article covers several pagination methods.

Reading OData API in SSIS using JSON Source

Let’s discuss how to read the OData API and Paginate Automatically. Below is a typical OData service Response (JSON Format). Notice how it includes the nextLink attribute, which points to the following URL to fetch more data. When you set Data Format to OData in the JSON Source, it will handle pagination for you.

Request URL:

https://services.odata.org/V3/Northwind/Northwind.svc/Order_Details?$format=json

Response:

{
  "odata.metadata": "https://services.odata.org/V3/Northwind/Northwind.svc/$metadata#Order_Details",
  "value": [
    {
      "OrderID": 10248,
      "ProductID": 11,
      "Discount": 0
    },
 ........
 ........
  ],
  "odata.nextLink": "Order_Details?$skiptoken=10436,75"
}

Step-By-Step

In this section, you will learn how to use the JSON Source Adapter to extract data from the OData API.

1. Firstly, you need to download and install SSIS ZappySys PowerPack.
2. Once you have finished the first step, Open Visual Studio and create a New SSIS Package Project.
3. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox.
   

   SSIS Data Flow Task – Drag and Drop

4. Double-click on the Data Flow task to see the Data Flow designer surface.
5. From the SSIS toolbox, drag and drop the JSON Source onto the Data Flow Designer surface.
   

   SSIS JSON Source – Drag and Drop

6. Double-click JSON Source and enter URL for OData API (You may use the below example URL for demo)
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Order_Details?$format=json

    
7. Now configure Array Filter as below (you may see $.value[*] or some other node if URL is different)
   

   OData Paging using SSIS JSON Source

8. Click Preview data and OK to save
   

   Read JSON File data from Web URL Example in SSIS

9. Now click OK to save.
10. Connect Your Source with a target like ZS Trash destination and execute the package. You will notice in the log that it paginates to return all rows
    

    Fetch OData in SSIS (REST API Example)

Loading OData API into SQL Server / Other Target using SSIS

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Twitter(Now X) REST API Paging Example using SSIS JSON Source

The Twitter (Now X) REST API is not an OData format, but you can see a similar concept there as well. On Twitter, you don’t have a partial URL like OData for the following link, but it provides a unique identifier for some sort of record. Read here for more information on the Twitter REST API Paging technique.

In our case, SSIS JSON Source Supports Paging very well, so we are covered. To loop through multiple result sets of Twitter data, simply configure the following four properties. See the screenshot below.

Twitter(X) REST API – Paging Example -Loop through resultset using cursor

Conclusion

Based on the above examples, you can see that handling Raging in REST API calls is not straightforward, but using Components like SSIS JSON Source can take some work off your plate.

The post How to read OData in SSIS – REST API Example appeared first on ZappySys Blog.

The REST API is becoming increasingly popular in the Data Integration landscape. If you are an SSIS Developer (not a coder) and stumbled upon a question What is REST API and how to consume API inside an SSIS package?” then you are at the right place :). We encourage you to read this article carefully and follow the various links we provided.

In this article, you will learn about many REST API Tips / Tools on the topic of how to call a REST API using the SSIS REST API Task, JSON Source, or XML Source. You will also learn how to perform HTTP GET and HTTP POST Requests without needing to know any programming languages (e.g., C#, Java, Python) – simply do a drag-and-drop operation in SSIS.

What is REST API / RESTful Web Service

Let’s first understand what a REST API is, also sometimes referred to as a RESTful Web Service. This is a new buzzword that you will likely hear frequently. More and more services are available in the Cloud, which makes it obvious to come up with some way for you to access data in a more firewall-friendly manner. What can be a better option than accessing it using a several-decade-old HTTP protocol, so no additional configuration is required for anybody accessing Cloud Services?

So, in short highlights for REST Web Service

• REST Web Service is a stateless client-server service model
• By passing an HTTP verb, you can perform server-side action over the standard HTTP protocol (e.g., GET, POST, LIST, DELETE, HEAD)
• You can pass parameters via URL query string and via HTTP Headers
• If you are doing HTTP POST, then you can pass additional data in the Request Body along with the other two methods described above
• REST API works by sending an HTTP Request and getting an HTTP Response
• The HTTP Response of a web service can contain Headers and Response Data, which can be in binary, text, JSON, XML, or other formats.
• You can secure your data by simply sending it over the HTTPS (Secure HTTP) protocol or encrypting values passed along with your request.

Tasks/Components in SSIS for Consuming RESTful API / WebService

	JSON Source (Read from REST API, JSON file or OData Service): Use this dataflow component when you have to fetch data from a REST API web service like a table. This component enables you to extract JSON data from web services and de-normalize nested structures, allowing you to save the data to a Relational database, such as SQL Server, or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local JSON files or direct JavaScript strings (Wildcard pattern support, e.g., c:\data\file*.json).
	Web API Destination (POST data to API URL): Use this data flow component when you need to load data into a target system via API calls (POST to URL). E.g., Loading Contacts into a Marketing System like MailChimp for email campaigns. Loading documents into CRM or document storage systems, e.g,. Couchbase or ElasticSearch.
	REST API Tak: Use this task when you don’t want to pull REST API data in tabular format but want to call the REST API for POST data to the server, DELETE data from the server, or things like download an HTML page, extract Authentication tokens, etc, where you are not necessarily dealing with data in tabular format. This task also allows you many other options, such as saving the RAW response into a variable or a file.
	XML Source (SOAP, File, REST): Use this dataflow component when you have to fetch data from an XML or SOAP web service and consume data like a table. This component allows you to extract data from a web service and save it to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local XML files or a direct XML string.
	CSV Source (File, REST): Use this dataflow component when you have to fetch data from File or REST API which returns data in CSV format . This component allows you to extract data from a web service and save it to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local CSV files or direct CSV string.

Using SSIS JSON Source to read from REST API and load into SQL Server

If you need to consume a REST API Service and store the result into SQL Server or any other RDBMS/FLAT File, then you can check this SSIS JSON Source. The JSON Source can deserialize your nested JSON (it also supports JSONPath filter expressions).

REST API Task – Ad-hoc web requests – Call REST API (POST, DELETE)

HTTP GET Request using SSIS Web Service Task or JSON/XML Source

In this example, we will retrieve JSON data from an OData web service.

• Download and Install SSIS PowerPack
• Create a new sample SSIS package in BIDS/SSDT
• From the SSIS Toolbox, look for items starting with “ZS”. Drag and Drop [ZS Rest API Web Service Task] to Designer Surface
• Configure HTTP Request properties as below
• Configure HTTP Response properties as below
• Run SSIS Package

Pass values in HTTP Headers

Calling REST API in SSIS using REST API Task, Pass headers, Body, Url Parameters

SSIS REST Api Task – HTTP GET, Test SSIS Web Service Call, Pass Custom Header

REST API response validation ( By status code, content, header)

Pass values in the Query String dynamically

HTTP POST Request using SSIS Web Service Task or JSON/XML Source

Passing Credentials to your Web Service

Basic Authentication – Set Authorization Header (Base64 Encoding)

Token-based approach – such as OAuth

Windows Authentication (NTLM)

By default, ZappySys REST API Tasks/Components use your default credentials. You can also use the HTTP Connection Manager with the “Use Credentials”> “Use Windows Authentication” option. See this article

Pass Client Certificate

Sometimes you may have to pass a client certificate along with your web request. The server checks the certificate passed along with your request and, if it matches, proceeds with your request. Here is the real-world use case How Azure Management Api uses Client Certificate.

ZappySys HTTP Connection Manager and OAuth Connection Manager both support passing a Client certificate.

Saving HTTP Web Service Response to a File or Variable

Let’s break down how to save the response from a ZappySys REST API Task into either an SSIS variable or a file within your SSIS package.

Scenario: Imagine you’re using the ZappySys REST API Task to retrieve JSON data from a URL. You want to store this data for further processing within your SSIS package.

1. Saving the REST API Response to an SSIS Variable

This is ideal when you need to manipulate the data within your SSIS package.

1. Add a ZappySys REST API Task to your Control Flow:
   Drag and drop the task onto your SSIS package’s Control Flow.
2. Configure the REST API Task:
   • Set the Connection Property to your REST API Connection Manager. (Optional)
   • Specify the URL of the API endpoint.
   • Configure any necessary headers, query parameters, or request body.
   • Go to the “Response Settings” tab.
3. Configure the Output Variable:
   • Check-mark Save Response content
   • Select Save Mode as “Save to Variable”
   • Select or create an SSIS variable of type String. This variable, named User::WeatherResponse will store the REST API response.
   

   Saving the REST API Response to an SSIS Variable

2. Saving the REST API Response to a File

This is useful for archiving the response or when you need to process the data outside of SSIS.

1. Add a ZappySys REST API Task to your Control Flow:
   Drag and drop the task onto your SSIS package’s Control Flow.
2. Configure the REST API Task:
   • Set the Connection Property to your REST API Connection Manager. (Optional)
   • Specify the URL of the API endpoint.
   • Configure any necessary headers, query parameters, or request body.
   • Go to the “Response Settings” tab.
3. Configure the Output File:
   • Check-mark Save Response content
   • Select Save Mode as “Save to File”
   • In the “Enter File Path” property, specify the path and filename where you want to save the REST API response. For example, if you are fetching JSON data from a product API, you can save it to the following file: C:\Data\Products.json.

Saving the REST API Response to a File

REST APIs enable array filtering to extract specific data from JSON or XML responses. This is achieved through techniques such as query parameters, JSONPath (for JSON), and XPath (for XML), allowing for the precise selection of elements. Additionally, regular expressions can be employed for string-based filtering and data validation.

REST APIs enable array filtering to extract specific data from JSON or XML responses.

Key Setting: “Treat response as binary”

As highlighted in the steps, the “Treat response as binary” option in the “Response” tab is the most important setting for correctly saving binary files. If you don’t check this, the task might try to interpret the binary data as text.

Example Scenario:

Let’s say you want to download a ZIP file from a REST API or a Public URL:

• API URL: https://example.com/api/download/archive.zip
• File Path: C:\Downloads\archive.zip
• In the ZS Rest API Task’s “Response” tab, you would:
  • Check-mark Save Response content
  • Select Save Mode as “Save to File”
  • Check “Treat response as binary”.
  • In the “Enter File Path” property, specify the path and filename where you want to save the REST API response. C:\Downloads\archive.zip.
  

  REST API Task – Download File

By following these steps, you can effectively use the ZappySys Rest API Task to download and save binary content from REST APIs to files within your SSIS workflows.

Save HTTP Web Response Headers / StatusCode

HTTP Web Response Validation

REST API Task supports validating your response for a specific header, status code, or content value. See the Validation tab for more information. Using this feature, you can throw an error if a certain header is missing from the response or reject the response if the StatusCode is other than 200.

Extract a single value from JSON/XML Web Response (e.g., Token)

The most common use case is if you are getting a token from your JSON REST API service. Assume that Service sends you a response in the following format, but you only care about the access_token property from the response below. In that case you can use JSON Path expression like this $.data.access_token

{
    data: {
        access_token: 111223344444,
        expires_in: 3600
    }
}

REST API Testing / Debugging Tools (3rd party)

Now, let’s look at a few popular tools that you will need to test/debug REST API calls during your development cycle.

Testing Rest API

There are a  few ways to test REST APIs, and each tool has its pros and cons, but the most popular tool to test REST API is Postman, so we will start from there

1. Postman – GUI
   Postman offers an excellent User interface to test the API. You can edit URLs, methods, and parameters, and add authentication with ease. See our article on how to migrate Postman settings to zappysys UI. Many API Providers offer easy-to-use Postman settings files for Testing API in Postman. If your API offers such a file, simply import it into Postman and start testing before using the ZappySys UI.
2. SoapUI – GUI
   SoapUI is another tool to test SOAP or REST APIs. They are primarily known for SOAP API Testing, but in recent versions, they have enhanced REST API Testing capability.
3. cURL – Command line
   If you want to call/test the API via the command line, then this is the most popular tool. Many API vendors use cURL syntax in their API documentation. Here is how to migrate cURL syntax to ZappySys UI.
4. Fiddler- GUI
   Fiddler has limited capability to test API call – See Replay in Composer option
5. ZappySys REST API Task
   You can use the ZappySys REST API Task to test the API inside Visual Studio. Edit URL / Headers / Body, etc, and click the Test Request Response option

Debugging REST API – Proxy

So far, we have seen client tools to test Request/Response for the URL you entered in the UI, but what if you want to Sniff Requests sent by some applications for which you have no way to know what Request Data / URL, or Body is sent behind the scenes? What response is it getting back and so on? Well, not to worry… Telerik Fiddler is the most popular tool at your fingertips and it’s FREE.

1. Fiddler – HTTP /  HTTPS Proxy / Debugger
2. Wireshark – Low-level Network Packet Sniffer

Conclusion

The REST API is becoming increasingly popular every day. With that, Data integration from RESTful services is going to be a challenge. The SSIS REST API Task and SSIS JSON Source can help resolve some of the issues you may encounter with SSIS. Use JSON Source if you need to store data into a SQL Table. Use REST API Task if you need to make an ad-hoc HTTP/REST request (such as get token, Delete Record, etc). ZappySys REST API Components gives you total control over your REST API Integration challenges without learning a programming language (e.g., Python, Ruby, C#, Java).

The post How to Call REST API in SSIS – Read JSON / XML / CSV appeared first on ZappySys Blog.