json source Archives | ZappySys Blog

How to convert JSON to CSV File using SSIS

ZappySys — Wed, 14 Feb 2018 16:52:37 +0000

Introduction

In this article, you will learn how to convert JSON to CSV File using SSIS. Sometimes you need to convert JSON file to a CSV, which resides in your local disk or somewhere on the Internet, be it a simple download link or a more sophisticated API call. Using ZappySys SSIS PowerPack together with SSIS will make it easy to happen.

We will be using this ZappySys SSIS PowerPack component to make things work:

JSON Source (REST API or File)

Prerequisites

SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
ZappySys SSIS PowerPack installed.

Step-by-Step – Export JSON to CSV File with SSIS

From SSIS Toolbox drag and drop Data Flow into the Control Flow.
Then drag and drop JSON Source (REST API or File) into the Data Flow.
Open JSON Source component for modifications.
Load JSON directly, from variable or URL:
Use JSONPath Expression in Filter section to select the data you need. We will use
```
$.store.book[*]
```
to extract all books from JSON array. Check Understanding JSONPath Expression with Examples article to see more examples how this filter works.
Close the window.
Drag and drop Flat File Destination into Data Flow and connect it to JSON Source:
Open Flat File Destination for configuration, configure it and map JSON Source columns with Flat File Destination columns.
You are done! JSON will be exported to a CSV (download a sample package).

Using API as JSON source

If your JSON happens to reside in a response of an API call which uses OAuth authentication, read this article how to configure JSON Source OAuth connection:

How to authenticate to an API with OAuth 2.0 using SSIS

Exporting multiple JSON arrays to a CSV

There may be a need to export two or more JSON arrays to several CSVs. In that case, you might be interested in this article:

Download a sample package

Be sure to download a sample SQL Server 2012 SSIS package in case you want to try it right away.

Conclusion

In this article we learned how to convert JSON to a CSV using SSIS PowerPack JSON Source in a very simple way. We also touched topics that delve into more complicated scenarios such as grabbing JSON from an API, exporting multiple JSON arrays and transforming them into columns. Check our Blog articles on JSON Source to find out what JSON Source is capable of more.

The post How to convert JSON to CSV File using SSIS appeared first on ZappySys Blog.

How to create BIML custom task or component

ZappySys — Thu, 21 Sep 2017 16:42:01 +0000

Introduction

BIML, or Business Intelligence Markup Language, is a nice technology that lets you generate SSIS packages using a bit of scripting and parameters. Check for more info if you are new to it:

In this article, we will cover how to create a BIML custom task or component. For this purpose, we will use the ZappySys PowerPack Task or Data Flow component as an example. At the end of this article you will find sample BIML files you can download and fiddle around.

Use BIML to create SSIS packages that use PowerPack tasks or components

BIML Elements

BIML uses

CustomTask

CustomComponent

, and

CustomSsisConnection

elements to create custom SSIS connectors. In each of them, you will need to specify

CreationName

or a similar attribute/element, which defines what kind of component you want to create. E.g., for LoggingTask it would look like this:

ZappySys.PowerPack.Task.LoggingTask.LoggingTask, ZappySys.PowerPack120, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502

You will find the connector name in DTSX. For Tasks, you have to specify the CreationName attribute; for components, it is UserComponentTypeName. For the Rest API Task, it looks similar to this:

NOTE: If your SSIS package is of a higher version than SQL Server 2012, convert it to SQL Server 2012 first.
For a different SQL Server SSIS package version, you will need to use a different DLL name, e.g.:

SQL Server 2008: ZappySys.PowerPack100
SQL Server 2012: ZappySys.PowerPack110
SQL Server 2014: ZappySys.PowerPack120
SQL Server 2016: ZappySys.PowerPack130
SQL Server 2017: ZappySys.PowerPack140

So, to generate a SQL Server 2017 SSIS package, the custom connector creation name becomes like this:
ZappySys.PowerPack.Task.LoggingTask.LoggingTask, ZappySys.PowerPack140, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502

Create a BIML custom task with CustomTask
You will need to use the
CustomTask
element to create PowerPack Tasks in your Control Flow. So open your DTSX where you use a PowerPack Task, find out the CreationName
and then copy child element in
element. (e.g., LoggingTask). You will need to escape it and use it in your BIML as well.
DTSX snippet
...

...
BIML snippet
A Logging Task under the Tasks element will look like this:
...

...

NOTE: Notice the escaped
XML element under
element.
Create BIML custom components using CustomComponent
Use the
CustomComponent
BIML element to generate these components in DataFlow:

Sources
Transformations
Destinations

If you need a connection, then use the CustomSsisConnection element.
Again, you will have to copy the
UserComponentTypeName
property value and use it in BIML to initialize a component. For a custom connection, it is a bit different: just use the CreationName attribute value from DTSX in the BIML element. Also, don’t forget to copy property values from DTSX and mirror them in BIML.
Below are DTSX and BIML snippets showing a Data Flow that contains a JSON Source using an HTTP Connection and is connected to a Trash Destination.
DTSX snippet
...

...

2
true
ZappySys.PowerPack.Adapter.JsonSource.JsonSource, ZappySys.PowerPack110, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502
...

...

ZappySys.PowerPack.Adapter.TrashDestination.TrashDestination, ZappySys.PowerPack110, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502
...

...
BIML snippet
...

2
true

...
Resources

BIML documentation
CustomTask documentation
CustomComponent documentation
CustomSsisConnection documentation

Download sample BIMLs

Sample BIML with PowerPack DataFlow Components
Sample BIML with PowerPack DataFlow Components with Connection
Sample BIML with PowerPack Task

The post How to create BIML custom task or component appeared first on ZappySys Blog.

How to make Path / URL dynamic in SSIS

ZappySys — Wed, 13 Sep 2017 13:27:33 +0000

Problem

Most of SSIS PowerPack components support simple text placeholders engine to make things dynamic without using Complex Steps of SSIS Expression Engine. So let’s look at how to make Path / URL or other input values dynamic in SSIS. To show you how to do that we will use JSON Source component, but basically, you can use it in any ZappySys PowerPack component that has URL option, including XML Source.

Create a variable

Don’t forget to set the default value, because it will be used to construct an URL in design time. A call to that URL will be made to figure out the columns.

Make an HTTP address dynamic

Creating a Dynamic Request Body

To make a dynamic body in ZappySys (e.g., in an HTTP request or API call using the REST API Task or JSON Source or XML Source), you typically use placeholders or expressions in the request body.

Here’s how to create a dynamic body:

Example: Dynamic JSON Body using Variables

{
  "name": "{{User::MyVariable}},
  "status": "active"
}

{{User::MyVariable}} is a variable placeholder.

Dynamic Body

That’s How You Make It Dynamic

By combining placeholders and expressions in the URL and body fields, ZappySys lets you build fully dynamic API requests. Whether you’re passing parameters, looping through records, or injecting runtime values—this approach keeps your integration flexible, scalable, and efficient.

The post How to make Path / URL dynamic in SSIS appeared first on ZappySys Blog.

How to register Facebook OAuth App for Graph API Access

ZappySys — Thu, 16 Jun 2016 18:27:52 +0000

Introduction

In this article, you will learn how to register Facebook OAuth App (Graph API) for data access using SSIS PowerPack REST API Connectors. Using SSIS PowerPack you can consume/manage Facebook data by calling Graph API. These connectors have drag and drop UI so you can do data integration without any coding (e.g. C#, Java, Python, Ruby).

JSON /REST Source Connector and REST API Task they both support a wide range of options for any type of REST API integration scenarios including Facebook REST API integration into your ETL Workflow. It also supports the use of OAuth 1.0 and 2.0 Protocols used by Twitter, Facebook, Google, LinkedIn, and many other popular Apis.

Facebook Graph API can be used to read or manage Facebook data via REST API Calls. To access Graph API you will have to create OAuth App inside Facebook Developer Portal. Let’s take a look step by step.

Step-By-Step: Create Facebook OAuth App (i.e Register OAuth App)

Go to www.facebook.com and log in with an account which you want to use to access Facebook data. You can access your own data using API without issue but if you have to access someone else data via App then it will require additional steps listed in this article in “Create Public Facebook OAuth App” section
Once you login to Facebook type https://developers.facebook.com/apps in the Address bar
Click on Create New App. If you never access developer portal then you may be prompted to Register as a developer.
Create Facebook OAuth App for Graph API access (For SSIS PowerPack REST Connectors)
It creates the app by given name and asked you to choose some options like ads and FB login. Click on skip to skip that step.
Now go to basic settings and there you can change app name, contact email and select category (e.g. Business). There you can also found OAuth App ID and App Secret store it somewhere for the next phase.
Register Facebook OAuth App for Custom Desktop Application (Basic Setup)
No Click on Add Products and add Facebook Login. Because our app is used for SSIS PowerPack which has custom login flow (using Embedded Browser). We have to Add Facebook Login support and check Embedded browser option (see below).
Adding Embedded browser support for Facebook OAuth Application
Now you ready to use your App to use OAuth App ID and App Secret.

Test Facebook OAuth App (Consume graph api in SSIS)

If you using SSIS PowerPack to manage your Facebook data using Graph API then perform the following steps to test your connectivity.

In the SSIS Designer in connection manager page .. right click and add ZS-OAUTH
Select Facebook provider, Configure App ID and OAuth App Secret
Select Permission (i.e. Scope) or type by hand (1 per each line)
Click generate Token. You will be redirected to login page and then it will show Authorize App confirmation. Click Accept.
Test Connection (If things go well it will show green success)
Testing Facebook OAuth Connectivity in SSIS (Use Graph API)

Fetch from Facebook Graph API (Load data into SQL Server)

Below is sample screenshot how you can use JSON Source to pull data from Facebook REST API and load into SQL Server. Refer this article for step-by-step information.

Extract data from Facebook using Graph API (SSIS JSON/REST API Connector) – Load into SQL Server Table

The post How to register Facebook OAuth App for Graph API Access appeared first on ZappySys Blog.

How to register Google OAuth Application (Get ClientID and ClientSecret)

ZappySys — Sat, 09 Apr 2016 21:25:18 +0000

Introduction

In our previous article we understood little bit about OAuth 2.0 Protocol. We talked how to use Default App for ease of use. However In this post you will learn how to register Google OAuth Application so you can access various Google Services programmatically (e.g Youtube, Google Analytics, AdWords, DoubleClick etc.). Here is the link to find all services: Google API Services (Using OAuth 2.0)

Recently google started supporting OAuth 2.0 Protocol. There are many articles written explaining OAuth 2.0 protocol so we are not going to cover that in this article.

Following SSIS Connectors support OAuth Connectivity. When you use below connectors no coding required to pull data from OAuth enabled services.

SSIS Tasks / Components which Supports OAuth Connection / Google API

	API Source
	Google Analytics Source
	JSON Source Connector (File, REST API)
	REST API Webservice Task
	XML Source (File, REST, SOAP)
	Web API Destination

ODBC Drivers which Supports OAuth Connection / Google API (Call in Power BI, Excel, Informatica, MS Access)

	API ODBC Driver
	JSON ODBC Driver
	XML ODBC Driver
	CSV ODBC Driver

Create Google Project

Registering OAuth App in Google first requires you to create API Project. API Project is a container which can host many OAuth Apps or other API Keys.

To create API Project Visit https://console.developers.google.com/project
If you already have existing API Project then once you click above link you can pick desired API Project from dropdown (Found in the top bar). If you using existing API Project then you may skip next few steps and directly move to step which explains [how to create OAuth ClientId].
To create new project Click on “Create Project” button. Once you click Create it will take few seconds to create project for you.
Create New Google API Project (Google API – Register new OAuth App)
Now give some friendly name and change default Project ID (optional) for your project and click Create (See below).

NOTE: Before you click create you can change default Project ID suggested by Google by clicking Edit. This Project ID is used everywhere when calling many APIs like this one so copy this ID somewhere (You can also see this on project screen so dont worry if you miss it).Project ID is unique across google system so you may not get your preferred ID but if you can get one then its easy to remember for sure.

Create new Google Api Project for OAuth App (Set project name / edit default project ID)

Enable Google API for Integration

Once your Project is created, next step is enable APIs you like to call. By default most Google APIs are disabled so make sure you enable each APi category manually using below steps.

Once project is created it will redirect you to the API Home page where you will see many common Google API you can use.If you do not see that you can always click below option to enable API manually.Visit : https://console.cloud.google.com/apis/library
Enable Google APIs / Services
By default API you like to call might not be enabled. Search for API you wish to enable, click on the API name and click Enable.Some common Google API Examples:
1. To enable Google YouTube API
  > search for “YouTube“. It has few APIs so pick the one you need under this,
  > For example To manage/view channel use Data API, for Reporting use Analytics API)
2. To enable Google Drive API
  search for “Drive” and click Enable
3. To enable Google Sheets API
  search for “Sheet” (Some Sheets API operations need Drive API too see enable that as well)
4. To enable Google BigQuery API
  > search for “BigQuery“
5. To enable Google Analytics (GA3)
  > search “Analytics“ and enable Google Analytics API
6. To enable Google Analytics (GA4)
  > search for Google Analytics Data API and click Enable, Search for Google Analytics Admin API and click enable (needed this to query available properties)
  
  Search Google API for OAuth Application Access
  
  Enable Google API for OAuth Application Access

Create OAuth Credentials (Client ID / Secret)

Once we create project and enable APIs our next step is to obtain OAuth ClientID / Secret. Before you can obtain that you have to configure few properties of your OAuth Consent Screen. Most fields are optional on this screen but fill out them if you plan to publish this API App for other users (i.e. Public App).

Now click on the credentials tab and click “OAuth consent screen”. Fill out necessary information which will be visible to the user (such as App Name). Once you are done, continue on with credentials creation:Google OAuth API – Create OAuth App – Set Application Name on Consent Screen
Click on [Create credentials] button and select OAuth Client Id option.
Create OAuth Client ID Credentials for Google API access
Next you will be asked to select Application Type. Pick “Desktop app” if your Application is Desktop style application. If your app is web site or running browser then pick Web Application. Click on Create and you will see your ClientID and ClientSecret which you can use during OAuth Authentication. Keep these both information secret. This is used to call OAuth Approval Screen by your Desktop App or Web App… and then user can obtain Token which can be used to call desired Google API.
NOTE: Make sure you do not choose Web App and only select Desktop App. Desktop App doesnt have Redirect URL making it simple to setup and use with ZappySys software.

Create Google OAuth 2.0 Credentials – Desktop App

Generate ClientID and Client Secret for Google OAuth Application

Publish App

The last step in App creation is to go to the OAuth Consent Screen tab. Under Publishing Status, click PUBLISH APP to ensure your refresh token doesn’t expire often. If you plan to use the App for Private use, you do not have to worry about the Verification Status after Publishing (Which is the case for most users who access their own data).

Publish Google App (This avoids Refresh Token Expiration every 7 days)

Using Client ID and Client Secret in ZappySys OAuth Connection

You can use generated ClientID and ClientSecret to initiate OAuth Authorization process. We are not going to cover how that works, but below is a simple screenshot of how these two pieces of information are used to call the Consent screen with some desired permission requested by calling Application:

ZappySys OAuth Connection can be used by many components suggested in the beginning of this article. So refer them for more information.

Google OAuth Application: Generating Access and Refresh tokens using SSIS and OAuth Connection Manager.

Using System Default Browser to get the token

If embedded browser has some issue to extract your token then you can close the window by pressing [X] in the title bar. Once you do that you will be prompted to use System Default browser to finish the token extraction. If you click Yes then Full browser will be launched.

NOTE: Below feature is added in version published after Dec 31, 2019 (v3.0.0.11231 or later) so make sure you have latest version.

Using System Default Browser to extract OAuth Token

Untrusted App Warning

Sometimes Google might show you below warning. In order to continue the flow you might have to Click on Advanced and then click Go to YourAppName (unsafe) like below

How to ignore Google Unsafe App Warning (OAuth Flow)

OR, in some cases, you may see modern UI like below. In either case, choose continue.

Google App not Verified Warning (New UI)

Select the account

Using System Browser

Copy code and finish the flow

Manually Copy Code and Get OAuth Refresh / Access Token

Why I didn’t get refresh token back?

For some reason if you just see Access token and not refresh token then try following

Logout current session in browser and close all browser windows and try again
If you use Custom Provider rather than Google on OAuth connection then change Auth URL to include prompt and access_type like below
```
https://accounts.google.com/o/oauth2/auth?access_type=offline&prompt=consent
```

What to do if Google Refresh Token expires every 7 days

Technically, once you have a refresh token, you should not worry about renewing as long as the API call is made atleast once every few months (max 6 months idle time). However, in some cases, your refresh token can expire sooner, and you have to click Generate Token manually

One main reason that is not well documented is if the Google Oauth Private App you created to access your own data is still in “testing” mode and not “published,” then as per Google, you may get this error. Perform these steps to publish your app, which is in testing mode, so you do not have to renew the refresh token every 7 days.

Go to your Project in Google API console
Generally here
No API call has been made using the generated refresh_token for six months or longer
SOLUTION: Make at least some API in six months to keep refresh_token active
Your OAuth App is in “testing” status and not published yet
SOLUTION: Just click the Publish button on your App OAuth Consent screen. Do not worry about getting verified if you plan to use it for internal use; after this step, your refresh_token may not expire every 7 days See this thread.DO NOT WORRY ABOUT GETTING VERIFIED – Just publish it to avoid refresh token expiration every 7 days.

Publish Google App (This avoids Refresh Token Expiration every 7 days)

What is Google OAuth AccessToken

AccessToken is short lived token which you can pass along with each API Service call. once this token is expired you can get new AccessToken as long as you have obtained and saved RefreshToken (Returned to caller after very first time user go through Approval process on Consent screen). If you are using ZappySys OAuth connection manager

What is Google OAuth RefreshToken

May OAuth services support short lived token which expires after few minutes or hours. Google API by default expires token after 3600 seconds … So how to get new token ? Do you always go through that annoying Consent screen… what if your application is Automated Service? Well OAuth has solution for that too… its called RefreshToken.

First time to setup OAuth connection user have to go through Approval screen (i.e. Consent screen) once they click Approve button they may receive two types of token (AccessToken and RefreshToken). RefreshToken is used only when you need to get new AccessToken without going through Consent screen. So technically if OAuth supports RefreshToken (This is optional) then you can Save this piece of information in database somewhere and when you get token_expired error or you detect expiration time you can get new AccessToken without involving user consent (because you already did once).

Conclusion

Hope this article helped you to understand how to register Google OAuth Application. Download SSIS PowerPack to try various OAuth data integration scenarios using drag and drop high performance connectors. With SSIS PowerPack OAuth support you can virtually pull data from any API service (XML REST API or JSON REST API). Such as Facbook, Twitter, SurveyMonkey, LinkedIn and many more.

The post How to register Google OAuth Application (Get ClientID and ClientSecret) appeared first on ZappySys Blog.

How to do REST API Pagination in SSIS / ODBC Drivers

ZappySys — Fri, 08 Apr 2016 15:37:55 +0000

Introduction

In our previous blog, we saw how to call REST in SSIS including concepts of Authentication / Error Handling. Now in this post, we will cover API Pagination. You will learn concepts and patterns about REST API Pagination (for JSON / XML / SOAP or CSV API).

We will describe methods and steps to implement REST API Pagination in SSIS PowerPack or ODBC PowerPack API Drivers (for apps like Excel, Power BI, Informatica…) without coding. Both Products share a similar user interface with minor differences. For example purpose, we will show screenshots from SSIS PowerPack UI but concepts should remain same. You will also learn some techniques about how to implement an infinite loop in SSIS. Sometimes your REST API requires you to loop through all pages until no more data found then you can use pattern described in this article (sample package attached at the end).

In this article, we will use SSIS REST API Task and JSON Source Connector to process data from RESTful web service. If you have to process XML (or SOAP) data from REST API then simply use XML Source Connector for REST API, SOAP or File rather than JSON Source.

Debugging REST API Requests

Before we look at various aspects of REST API Pagination, we highly recommend you install FREE Web Debugging Tool Fiddler. Look at this article to understand how to debug REST API requests.

Method-1: REST API Pagination via URL Parameter

If you have paging requirement by specifying page number in URL then you can loop until no more pages found. Your REST API can indicate no more page found by any of the following ways.

Example URL: http://myserver.com/v1/api/getcustomers/?page=1

If no more data found for specified page number then

server may return an empty response
server may return the response with specific error and response status code

JSON Source Connector comes with flexible paging options to handle this scenario. See below screenshot how to configure this.

REST API Looping/Pagination via URL Page Number Parameter (Loop until last page detected)

URL with offset and limit

Sometimes you may have URLs with offset + limit parameters. as below. In this case, you can use the same technique as the above screenshot but just specify Page Num Indicator as offset and Increment by the same as your limit parameter (e.g. 1000 in this case). At runtime, your offset parameter will increment and generate URLs like below to fetch the next page.

If you don’t specify the offset parameter in the initial URL then offset=1 will be assumed as the default start. If your offset starts at 0 then refer to the next section for a workaround (i.e. Use URL Path Mode with StartAt variable = 0). Also if your increment for offset is other than the default page size then you must include the limit / pagesize parameter in the URL as below (e.g. limit=100 which is rows per page)

Example URLs:
http://myserver.com/v1/api/getcustomers/?offset=1&limit=1000
http://myserver.com/v1/api/getcustomers/?offset=1001&limit=1000
http://myserver.com/v1/api/getcustomers/?offset=2001&limit=1000

Method-2: REST API Pagination via URL Path

Sometimes your URL may contain a page or other identifier using path format.

Example URLs:
http://myserver.com/v1/api/getcustomers/1/
http://myserver.com/v1/api/getcustomers/2/
http://myserver.com/data/file1.json
http://myserver.com/data/file2.json

If no more data is found for the specified page number then

the server may return a 404 error (page not found)
server may return a response with a specific error and response status code

JSON Source Connector comes with flexible paging options to handle this scenario. See the below screenshot of how to configure this.

REST API Looping/Pagination via URL Path (Loop until last page detected)

URL with Skip Parameter (Offset starts at zero)

Now let’s look at another common pattern just like the offset we discussed earlier but in the Skip pattern, it starts at zero rather than offset=1. In the previous mode, we didn’t have the option to supply the initial value to start the counter but if you use Path mode then you can start the counter at some custom value. See the below screen to show how we can setup pagination with the skip method with custom increment.

If your increment for offset is other than the default page size then you must include page size parameter in the URL as below (e.g. pagesize=100 which is rows per page. Some APIs name this parameter in different way e.g. count=1000 or limit=100. Refer to the API documentation to know exact names)

Example URLs

http://mysite.com/api/getorders?skip=0&pagesize=100
http://mysite.com/api/getorders?skip=100&pagesize=100
http://mysite.com/api/getorders?skip=200&pagesize=100
http://mysite.com/api/getorders?skip=300&pagesize=100
....

REST API Pagination with Custom Skip / Increment Settings (Offset Starts from 0)

Method-3 : REST API Pagination via Response Attribute (Next Link or Cursor)

This method is not discussed in this article but click here to read more

There are mainly 3 ways you can get next link or cursor.

Full Link (e.g. nextPage : “https://mysite.com/api/v1/1/?filter=AAAA” )
Partial Link (e.g. nextPage : “/api/v1/getcustomer/1/?filter=AAAA” )
Cursor Continuation Token (e.g. nextCursor : “xxxxAAAbbbbCccc112233==” )

Next Link Method (Partial or Full URL)

Example response with next URL would be

{ 
  "value" : [....], 
  "odata.nextLink" : "http://abc.com/customers/2/" 
}

In this method you don’t specify page number in URL. If response include many records then you will get partial response along with indicator of next link.

For more information about this REST API Pagination please check below links. It explains in depth how to paginate REST API response with next page link found in response.

OData Paging using SSIS – REST API Paging Example

Read Twitter data in SSIS using REST API Task and JSON Source – OAuth2 Protocol

SSIS API Pagination – Next Link from Response Attribute

Next Link – Detect the last page based on empty rows (Regular Expression)

Now let’s look at a little bit different scenario where the Next Link is never empty even on the last page. This can cause an infinite loop if we configure it like the last section. So in such a case (see below example) when the last page still has the next link URL but the data row array is empty then use settings like the screenshot below.

First Page Example

{ 
  "value" : [....], 
  "odata.nextLink" : "http://abc.com/customers/2/" 
}

Last Page Example

{ 
  "value" : [], 
  "odata.nextLink" : "http://abc.com/customers/NNNN/" 
}

Example: Last Page

Basically, configure the following options

Set Stop indicator attribute to the same node which contains data rows (without array brackets). E.g. $.items
Set Stop indicator value / Regular Expression to something like
```
[]
```
assuming the last page has the same way. Check Fiddler Trace for exact text on the last page.

Last Page with non empty next link and empty data array (Detect Last Page based on zero row count)

Cursor Method

Many API returns cursor rather than link. Cursor method usually used to support a large number of records. Cursor is opened at server side with initial query information you submit and if cursor has more records you may get next cursor indicator in your response. In cursor method you have to supply Suffix for Next URL so after First URL your next URL will contain extra attribute which will include pagination information.

Suffix for next URL

Suffix for next URL can include special placeholders. There are two placeholders allowed. If you wish to append Next URL value in raw text then use <%nextlink%> but if your API needs value in encoded format e.g. A%20B rather than A+B then you have to use <%nextlink_encoded%>

Example URLs for Cursor:

Below is few sample URLs in the log when pagination occurs via Cursor method. As you see the first URL has Cursor=* means its first call, after first request server may send next cursor in the response body (see example)

{ 
  data: [....], 
  nextCursor: "Aooo/MxxxxxxxxxxxtdA==" 
}

First URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=*
Second URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=Aooo/MxxxxxxxxxxxtdA==
Third URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=Booo/MxxxxxxxxxxxtdA==
…..
…..

Method-4: Pass Page Number in Body Attribute

Some API allows you to pass your PageSize and Page number inside Request Body (i.e. POST data) in that case you may configure your options like below (We used Google AdWords Api as an example)

Configure REST API Pagination – using Page Number via POST Body Method

REST API Pagination – Passing PageNumber inside POST Body (Example: Google AdWords SOAP API)

Method-5: REST API Pagination via multiple URL parameters (e.g. Start and End)

In the previous section, you saw how to pass page numbers via URL (Single Parameter: http://abc.com/?page=1). However, sometimes you have a pagination requirement where you have to specify multiple parameters that need increment. See below example URL.

http://abc.com/?startRow=1&endRow=100
http://abc.com/?startRow=101&endRow=200
http://abc.com/?startRow=201&endRow=300
...

Pagination via Multiple URL Parameters

Method-6: REST API Pagination via Header (Next Link in Header)

Another popular pagination method is paginate via Header. In this method, each API response includes special Header which indicates next link for your response. In header method there are two approaches. [1] Custom Method [2] RFC 5988 Method. Lets look at both.

Next Link in Custom Header

In this method you have to specify which response header contains next URL. When no more NEXT url found in response reader stops.

Custom Header Pagination Method

Next Link in Header as per RFC 5988

In this method, HTTP response contains a special header (Link) which contains next link for our response. If no link found to read more data then reader stops. This method is described well here in RFC 5988. Check this GitHub API Example to understand more on this pagination method.

Link Header Pagination Method (RFC 5988)

Method-7: REST API Pagination using Cursor / Continuation Token from Response Header

There will be a time when your API sends some sort of Continuation Token in Response Header and you have to use that token to call next request. Some API needs this token passed as query string in Next URL and in some API you have to pass as request header. We will cover both cases in the following section.

Your API might call these tokens as “Cursor” too so we will use these both terms in the following section (They both mean same thing).

There are few ways these APIs pass Cursor to the next request

Pass Cursor via Request Header with same header name
Pass Cursor via Request Header with different header name (introduced in v4.2.3)
Pass multiple Response Headers via Request Headers (Azure DocumentDB style usecase)
Pass Cursor via URL Query String Parameter (introduced in v3.0 or later)

Lets see all approaches.

Pass Token via Request Header (Same header name)

Consider the API pagination use case in which you get some sort of continuation token in the response header and you take it and pass in the next request via request header (same header name).

Example:

First Request / Response:

------- Request ------
GET https://abc.com/api/getdata

------- Response ------
x-token: Nxdfabcd11111

{ some response data }

Second Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-token: Nxdfabcd11111

------- Response ------
x-token: Nxdfabcd22222

{ some response data }

……

Last Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-token: Nxdfabcd22222

------- Response ------

{ some response data }

For above pagination pattern that just enter the response header name in Page Num Indicator field as below

Pagination using response header continuation token – Pass as Request Header

All possible formats for Page Num Indicator is below

x-page-token
x-token1, x-token2
RequestUrlQueryName=ResponseHeaderName
header::RequestHeaderName=ResponseHeaderName
RequestUrlQueryName=ResponseHeaderName(regular-expression)

Pass Token via Request Header (Different header name)

Consider the API pagination use case just like previous one but only different is request header name is not same as response header name. For example rather than x-token in request we need to call it x-next-token.

Example:

First Request / Response:

------- Request ------
GET https://abc.com/api/getdata

------- Response ------
x-token: Nxdfabcd11111

{ some response data }

Second Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-next-token: Nxdfabcd11111

------- Response ------
x-token: Nxdfabcd22222

{ some response data }

……

Last Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-next-token: Nxdfabcd22222

------- Response ------

{ some response data }

For above pagination pattern that just enter the response header name in Page Num Indicator field as below

Pagination using response header continuation token – Pass as Request Header (Different name)

Pass multiple Response Headers via Request Headers (Azure DocumentDB style usecase)

Some API like Azure Cosmos DB uses Continuation Token via Header approach. In that case you can enter single header name or multiple headers like below (For multiple headers use comma)

REST API Pagination using Continuation Token via Header (Azure CosmosDB / DocumentDB API Example )

You can use use tools like Fiddler to see paginated requests as below.

First Request

View Paginated Requests in Fiddler (Azure CosmosDB API Example)

Second Request

Pass Next Page Token via Header (View Paginated Requests in Fiddler – Azure CosmosDB API Example)

Pass Token via URL Parameter (Query String)

In the new version we added a feature where you can actually read Continuation Token (i.e. Cursor) from Response Header and then pass to next request in the URL rather than Request Header. Here is the example API which uses such method.

First Request

--------- Request ---------
GET
http://test.com/api/getdata

--------- Response ---------
X-RESPONSE-TOKEN : Pg2xxxxxxxxxxxxxxxxx

{ data : [ 1,2,3.... 10 ] }

Second Request

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg2xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : Pg3xxxxxxxxxxxxxxxxx

{ data : [ 11,12,13.... 20 ] }

Here is how to configure above pagination pattern

Pass Response Header value to URL (Cursor)

Pass Token via URL Parameter (Query String) – with custom Regular expression (Usecase: Salesforce BULK query API)

Now consider previous scenario but with slightly different requirement. By default previous pagination pattern stops when specified response header is missing or response header value is blank. However is some cases header value contains custom string to indicate last page. For example x-token: NULL. In such case we need to use Page Num Indicator like below (must use v4.2.3 or higher in SSIS PowerPack / 1.5.2 or higher in ODBC PowerPack)

nextUrlParameterName=responseHeadername(Some_Regular_Expression)

For example to stop pagination in below case you have to enter Page Num Indicator as below. It says stop pagination when “NULL” word is detected

cursor=X-RESPONSE-TOKEN(^((?!NULL\b).)*$)

First Request

--------- Request ---------
GET
http://test.com/api/getdata

--------- Response ---------
X-RESPONSE-TOKEN : Pg2xxxxxxxxxxxxxxxxx

{ data : [ 1,2,3.... 10 ] }

Second Request

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg2xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : Pg3xxxxxxxxxxxxxxxxx

{ data : [ 11,12,13.... 20 ] }

……

Last Request

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg3xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : NULL

{ data : [ 21,22,23.... 30 ] }

Here is how to configure above pagination pattern

Pass Response Header value to URL (Cursor)

Method-8: Pass Pagination Token in Request Body

In previous section (Method-4: Pass Page Number in Body Attribute ) we saw how to pass simple numeric page counter in Body. This is fine for some API but if your API has base64 style text token then counter based method doesn’t work. Also if you have requirement to hide page attribute node in first request and send page counter node in second request body (Changing Request Body Pattern for First vs Next) then check next two sections.

Next two sections covers two API Pagination patterns to fetch 100 rows (1000 in each page) . Both requires to send token in Body but second section is more complex need (Different Body Attributes in first vs next requests )

Passing Page Token in Body (Blank Token in first request allowed)

Lets look at simple pattern where first request has blank token in Body but all other requests until we find last page we keep sending non-empty token. This approach works as long as API Provider doesn’t mind having Blank Token in first request. If you get error then check next section to handle more complex way.

Here is the example of API which needs token in body

First API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"" }

--------- Response ---------
{recordsFound: 5000, data: ["Acct1", "Acct2"..., "Acct1000"], nextToken:"AxBxCx112233001==" }

Second API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"AxBxCx112233001==" }

--------- Response ---------
{data: ["Acct1001", "Acct1002"..., "Acct2000"], nextToken:"AxBxCx112233002==" }

….. many more calls …

…. many more calls ….

Last API call (no more nextToken in response so STOP pagination)

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"AxBxCx112233009==" }

--------- Response ---------
{data: ["Acct9001", "Acct9002"..., "Acct10000"] }

So if you have such pattern then you can set few properties

For such issue you can use Changing few properties like below. Use Property Grid if you cant find on UI

Set EnablePageTokenForBody =
```
True
```
Set PagePlaceholders =
```
header=|;filter=|;
```

Set HttpRequestData =

{ searchFor: "Account", continueToken:"[$pagetoken$]" }

Setting Body with [$pagetoken$] to implement dynamic pagination

Setting Page Placeholders in Advanced Pagination tab.

Passing Page Token in Body (Blank Token in first request)

Now let’s look at more complex pattern. Here our first request body is different than other requests.

First API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account" }

--------- Response ---------
{recordsFound: 5000, data: ["Acct1", "Acct2"..., "Acct1000"], nextToken:"AxBxCx112233001==" }

Second API call

--------- Request ---------
POST http://test.com/api/getdataNext

{ continueToken:"AxBxCx112233001==" }

--------- Response ---------
{data: ["Acct1001", "Acct1002"..., "Acct2000"], nextToken:"AxBxCx112233002==" }

….. many more calls …

…. many more calls ….

Last API call (no more nextToken)

--------- Request ---------
POST http://test.com/api/getdataNext

{ continueToken:"AxBxCx112233009==" }

--------- Response ---------
{data: ["Acct9001", "Acct9002"..., "Acct10000"] }

So notice few things

First Request Body / URL is different than second / third and Nth API call
For first request we do not have token attribute but after first request we do have token attribute

We have this article and this article which describes similar pattern.

Now lets look at how to solve both problems

Passing PageToken or Counter in Next API Call Body

For such issue you can use Changing few properties like below. Use Property Grid if you cant find on UI

Set EnablePageTokenForBody =
```
True
```
Set PagePlaceholders =
```
header=|;filter=|;
```
Set HttpRequestData =
```
[$tag$]
```
Set HasDifferentNextPageInfo =
```
True
```
Set FirstPageBodyPart =
```
{ searchFor: "Account"}
```
Set NextPageBodyPart =
```
{ token:"[$pagetoken$]" }
```

So notice how we defined Body in two different properies as template. And in HttpRequestData we used

[$tag$]

Output Page Number as column

Sometimes you have use case where you want to Start Page Number from the last page loaded successfully. By default, you will see one additional column __FileName in the output. This column contains PageNumber from URL for each row. You can save this into database so when next time you run your SSIS you can start at MAX(PageNumber) + 1.

Outputting PageNumber option is not visible by default so Use Property Grid and look for OutputFileName property (Default=true)

Last Page Detection

Every API may have different way of telling you that you reached last page or no more data found so you should stop looping. SSIS PowerPack comes with many options so you can implement desired logic for last page detection.

Following approaches supported by last page detection. Below options are not applicable if you are using Method#3 (Response contains Next Link / Cursor)

Detect the last page by row count
Detect the last page by response status code
Detect the last page by error message (substring search)
Detect the last page by response size
Detect the last page based on response value or regular expression pattern

Now lets look at each setting. You should choose correct one based on your API behavior

Detect the last page by row count

If your API returns blank document or blank Array (if its JSON) then you can use this method to detect the last page. So when no content found in Response document it will stop pagination.

See below example of JSON API .. It shows sample first page and Sample last page (Assuming you entered Filter as $.Records[*] to extract data from Records array.

First Page

{ 
  Records : [ {...}, {...}, {...} ......... ]
}

Last Page

{ 
  Records : [  ]
}

SSIS API Pagination – Detect the last page by row count (Blank document or array)

Detect the last page by response status code

Sometimes you get error with status code when the last page is reached (Most common is 404). So rather than failure, you can treat that status code as last page indicator using this option (see below).

SSIS API Pagination – Detect last page by response status code

Detect the last page by the error message

If you cant use above two methods and want to detect the last page based on a certain error string in the response then use below approach.

SSIS API Pagination – Detect the last page by error/response string search

Detect the last page by response size

Sometimes you can detect the last page by response size. For example, your response contains “–” only if no more data is found then you cannot use any of the above approaches and you have to detect based on response size (e.g. less than 3 bytes). See below

SSIS API Pagination – Detect last page by response size

Detect the last page based on the response attribute value

There will be a time when you want to detect the last page based on some indicator value from your response. In that case, you can use the following settings to detect the last page.

Select Paging mode =Response Attribute Mode
Select Next Link Attribute (Example: $.next_page )
Select Stop Indicator Attribute (Example: $.hasMorePages )
Enter a value for Stop Indicator (Example: false )

If you don’t want to use Hardcoded value for stop indicator and want to use expression then see next section.

Detect the last page based on regular expression pattern

There will be a time when you want to detect the last page based on some indicator value from your response. In that case, you can use the following settings to detect the last page.

Select Paging mode =Response Attribute Mode
Select Next Link Attribute (Example: $.next_page )
Select Stop Indicator Attribute (Example: $.count )
Enter regex pattern (regular expression) for Stop Indicator (Example: regex=^\d{1,3}$ ). Stop when count attribute is less than 1000 in other word less than 4 digits.

Here is an example from Zendesk API (Stop pagination when count attribute from the response is less than 1000)

Pagination for Zendesk Incremental API – Next Link and Last Page detection using Stop Indicator Regular Expression

Handling API Rate Limit - Retry on too many API requests

Many API restricts you from calling too many API calls too soon (just to save their servers from being overwhelmed) - Example. Let's think this way if you fetching a million rows via paginated response and the very last page fails then you won't be happy because now you have to fetch everything again. To solve this ZappySys provides retry options on HTTP / OAuth and REST API connection managers.

Generally, when you exceed the rate limit, the server sends you error code 429 but it can be something else too or multiple codes are returned. In the below example, we use 503. Check your API documentation to know the exact code you should retry.

To set the retry options go to connection settings (e.g. HTTP / OAuth / REST connection). The below example is for OAuth connection.

To retry on multiple codes use Pipe as a delimiter. Example: 429|503

Retry Settings for API Rate Limit Exceeded Error (Status code 429 or other)

Performance Tips

You can improve performance of your API calls by doing a few changes (again not every API may support this)

Use the maximum page size possible (e.g. if you doing pagination you can set max possible page size – refer to your API documentation whats the default size and what is the max page size you can supply)
e.g. /myapi?pagesize=100 (parameter name can be something else like limit=100)

SSIS Infinite Loop

So now you know how to paginate automatically in JSON Source Connector but what if you want to use REST API Task for pagination? Some custom scenarios like web scraping require you to download all HTML pages until no more results found? In that case, you most likely need infinite looping pattern (Loop until the last page detected). To loop through pages until the last page is reached you can perform the following logic.

The last page can be detected based on any of the following ways
1. Check Response StatusCode
2. Check Error Message
3. Check Response Size (If less than X bytes then consider as the last page)

REST API Task support #1 and #2 scenarios. If you want to check bytes then use the expression in Expression Task or Script Task
(e.g. @exitloop = LEN(@responseData) <= 3 )

URL for Rest API Task (On Request Settings Tab)
When you configure REST API Task enter URL as below so its dynamic. You can use PageCounter variable anywhere in the URL.

//zappysys.com/downloads/files/test/cust-{{User::pagecounter}}.json

SSIS REST API Looping – Pass Page number in URL, SSIS Infinite Loop

Configure For Loop Task for Infinite Looping (Loop until last page found)

Download Sample SSIS Package

SSIS-Loop-Rest-API-2012.dtsx (Zip file)

Video Tutorial – Reading data from API in SSIS

In case you missed our previous article about calling REST API is SSIS then check below video.

Conclusion

It can be time consuming and steep learning curve to write C#/JAVA code to handle your rest api integration but using some Tasks and Connectors listed below can help you to solve big headache and provide clean drag and drop REST API integration in SSIS for virtually any REST API Source (e.g. Twitter, Facebook, Salesforce, office 365). Download SSIS PowerPack and try it out by yourself.

SSIS Components for REST API/JSON/XML/SOAP

	JSON Source Connector (File, REST API)
	Export JSON File Task
	JSON Parser Task
	XML Source Connector (File, SOAP, REST API)

The post How to do REST API Pagination in SSIS / ODBC Drivers appeared first on ZappySys Blog.

Reading JSON Arrays from file / REST API

ZappySys — Mon, 11 Jan 2016 16:50:48 +0000

Introduction

REST API is becoming more and more common and with that you will see explosion in use of JSON data format. One of the questions we get a lot is “How to extract or read array from JSON data file” or “How to read multiple arrays from JSON data“. Example : Store JSON document has Customers[…] and Employees[…] arrays and lets say you want to extract both and save to target SQL server database. In this post you will learn how to make this possible and how to make it fast with some performance tips. We will use SSIS JSON Source (File, REST API Connector)

If you want to learn how to parse Multi-Dimensional JSON arrays then check this article.

Video Tutorial – Read from multiple arrays from JSON document

Read array from JSON document

To extract subdocument or array content from JSON document you need to use JSON Path expression. Lets check following example JSON document.

{
  "store": {
    "employees": [
      {
        "name": "bob",
        "hiredate": "2015-01-01"
      },
      {
        "name": "sam",
        "hiredate": "2015-01-02"
      },
      {
        "name": "ken",
        "hiredate": "2015-01-03"
      }
    ],
    "books": [
      {
        "category": "reference",
        "author": "bob",
        "title": "hellooo1",
        "price": 1.95,
        "sections": [
          "s1",
          "s2",
          "s3"
        ]
      },
      {
        "category": "fiction",
        "author": "sam",
        "title": "hellooo2",
        "price": 1.96,
        "sections": [
          "s4",
          "s1",
          "s3"
        ]
      },
      {
        "category": "science",
        "author": "steve",
        "title": "hellooo3",
        "tag": "1bcd",
        "price": 11,
        "sections": [
          "s9",
          "s2",
          "s3"
        ]
      }
    ],
    "location": {
      "street": "123 Main St.",
      "city": "Newyork",
      "state": "GA"
    }
  }
}

Below is example filter expression to extract common elements

$.store.books[*]       //get all books for store              
$.store.employees[*]       //get all employees for store                 
$.store.books[*].sections[*]       //get all sections from all books   
$.store.books[*].author //get all authors of all books for store           
$.store.books[*]    //get all books for store  
$.store.books[2]    //get 3rd book record
$.store.books[:2]   //get first 2 books from the top
$.store.books[-2:]  //get last 2 books
$.store.books[?(@.price < 10)] //get books where price is less than 10
$.store.books[?(@.tag)] //filter all books with tag

Read single array from JSON data (JSON Path expression)

SSIS PowerPack comes with powerful REST API Connectors for JSON and XML . In below example screenshot you will see how to extract single array section from your JSON File. But if you have multiple array sections (e.g. books and employees from above example) then you have to use Multiple JSON Source Connectors for same JSON File.

Read multiple arrays from JSON data

If you have scenario to extract multiple arrays from same JSON (e.g. extract Employees and Books from store ) then you have two options. You can pick any approach which suites your need (Method-1 is recommended).

Method-1: Use single JSON Source with [Output As Document] option and feed that to JSON Parser Transform
Method-2: Use multiple JSON Sources with different filter expressions
Method-3: Enable Array Flattening Option (Useful if you have smaller arrays – i.e. less than 10 items) – Version v2.7.6+

Now lets discuss each approach in detail.

Method-1: Use single JSON Source along with JSON Parser Transform

SSIS PowerPack v2.2 released new JSON Parser Transform and XML Parser Transform which can be used to parse any input JSON/XML data coming from upstream in pipeline. It parse input string column into multiple columns how JSON/XML Source does but the only difference is JSON Parser takes String as input from existing upstream column.

Here are the steps to parse multiple JSON arrays using JSON Parser Transform. For example purpose we will use sample store json listed above.

Drag new data flow in SSIS Control Flow designer. Double click on Data flow
Drag ZS JSON Source from SSIS Toolbox. Configure Source, Filter and other property.
Check [Output As Document] option (Found under Access Mode dropdown). Click preview. From grid copy one sample document from __DOCUMENT__ column. (Right click on Cell and Copy).This is raw JSON which we will use later to feed as sample in JSON Parser Transform (If you cant copy JSON then obtain some other way)
Enable Output As Raw Document Option – SSIS JSON Source
Click OK to save JSON Source.
Drag Multicast Transform (Native SSIS Transform) from Toolbox
Drag & Drop SSIS MultiCast Transform from Toolbox on designer surface
Connect JSON Source to Multicast Transform
Drag first JSON Parser Transform from Toolbox. Rename it to [Parse Employees]. Connect blue arrow coming from Multicast to JSON Parser Transform
Drag second JSON Parser Transform from Toolbox. Rename it to [Parse Books]. Connect blue arrow coming from Multicast to JSON Parser Transform
Double click [Parse Employees]. Select __DOCUMENT__ as upstream column. Enter sample JSON you obtained in step#3. Click on select Filter and highlight Employees node (i.e. $.store.employees[*] ). Click preview to see data.
Use JSON Parser Transform to parse Raw JSON document
Very Important:Click Columns tab and verify metadata (e.g. datatype, length). If you think you need extra columns or bigger column length then change in the grid and check [Lock] option found in the last grid column. If column length is smaller than output data then you will get error message when you run package. You can review error message carefully to determine which column needs to be adjusted (Repeat this until you clear all errors).
Click OK to save [Parse Employees].
Double click [Parse Books]. Select __DOCUMENT__ as upstream column. Enter sample JSON you obtained in step#3. Click on select Filter and highlight Books node (i.e. $.store.books[*] ). Click ok. Click preview to see data.
Perform same steps as #11 for [Parse Books] to set correct metadata.
Click OK to save [Parse Books].
Drag ZS Trash destination from Toolbox. Connect Trash destination to [Parse Employees]
Drag ZS Trash destination from Toolbox. Connect Trash destination to [Parse Books]
Run the package and you will see both 2 outputs from different JSON Parser with different row count.

Here is how it will look like

Extract multiple array output from JSON file / REST API response in (SSIS JSON Source, JSON Parser Transform)

Method-2: Use multiple JSON Sources with different filter expressions

You can consider using multiple JSON Source (each source with unique filter expression). Since one JSON source can only extract one array or Document.. You have to use below technique to read multiple arrays from same JSON document.

Read array from JSON data – Reading multiple array or single array

Method-3: Use Array Flattening Option (Preferred for smaller arrays)

If you have array inside extracted record and you like to flatten it so each items of array becomes column then use newly introduced feature [Enable Array Flattening]

Consider JSON like below. What if you don’t like to store every detail from below JSON in just one table (i.e. rather than splitting in 3 tables Customers, CustomerHobbies , CustomerAddresses ?) . To achieve full denormalization ZappySys offers powerful yet simple one click option.

{
	"Customer" : [{
			"Name" : "John",
			"Hobby" : ["Painting","Reading","Swimming"],
			"Addresses" : [
				{"Street":"100 Main St","ZipCode":"10062"}, 
				{"Street":"101 Fire St","ZipCode":"11121"}
			]
		}, {
			"Name" : "Peter",
			"Hobby" : ["Hiking","Golf"],
			"Addresses" : [
				{"Street":"200 Main St","ZipCode":"20062"}, 
				{"Street":"201 Fire St","ZipCode":"33321"}
			]
		}
	]
}

To enable flattening check below option on Filter

JSON Array Flattening Option in SSIS JSON Source

Column after Array Flattening Option

Using JSON REST API Connector to parse JSON data

If you want to access JSON data from REST API then you can use same JSON Source Connector. All you have to do is rather than local file path c:\data\xyz.json change it to URL as below and you will be able to access and filter JSON data using same technique as above.

Get JSON data from URL (REST API JSON response)

JSON Path for recursive data extract (Double dot expression)

There will be a time when you dont know how many levels deep you have to go for data extract. See below example. We will to get all employees id and name from hierarchy. For that you can use double dot like below. As you can see there are many nested levels and we want to extract id and name from each level so data can be

Expected Output

id      name
-------------------------
100	Jeff (CEO)
200	Bob (VP Sales)
300	Kim (VP HR)
400	Kumar (VP Dev)
210	Ron (Sales Mgr)	
220	Akash (Sales Mgr)	
410	Den (Sr. Dev)	
420	Den (Sr. Dev)
411	Den (Jr. Dev)
412	Sita (Jr. Dev)

Sample Data

{
  "employees": [
    {
      "id": 100,
      "name": "Jeff (CEO)",
      "employees": [
        {
          "id": 200,
          "name": "Bob (VP Sales)",
          "employees": [
            {
              "id": 210,
              "name": "Ron (Sales Mgr)"
            },
            {
              "id": 220,
              "name": "Akash (Sales Mgr)"
            }
          ]
        },
        {
          "id": 300,
          "name": "Kim (VP HR)"
        },
        {
          "id": 400,
          "name": "Kumar (VP Dev)",
          "employees": [
            {
              "id": 410,
              "name": "Den (Sr. Dev)",
              "employees": [
                {
                  "id": 411,
                  "name": "Den (Jr. Dev)"
                },
                {
                  "id": 412,
                  "name": "Sita (Jr. Dev)"
                }
              ]
            },
            {
              "id": 420,
              "name": "Den (Sr. Dev)"
            }
          ]
        }
      ]
    }
  ]
}

Configure JSON Source for Recursive Children Scan using Double Dot JSONPath

Open JSON Source and configure like below.

Set Filter as $..employees[*] (Notice how we used double dots in expression. This does recursive scan for N number of levels)

JSON Path for recursive scan of nested array (many levels deep)

Parse Multi-Dimensional Array

If you want to learn more how to parse Multi-Dimensional JSON arrays then check this article full length article.

Sample JSON with Multi Dimensional array

{
  "columns" : ["Id", "FirstName", "IsActive"],
  "rows" : [ [1,"bob",true], [2,"sam",false], [3,"joe",true] ]
}

Array Transformation Options for 2D array

Sample JSON with Multi Dimensional array (No columns in document)

{
  "rows" : [ [1,"bob",true], [2,"sam",false], [3,"joe",true] ]
}

Array Transformation Options for 2D array

Performance Tips

In this section we will describe how to make your things faster.

Reduce number of requests to server by changing page size

If you are doing REST API paging to get full resultset then make sure you adjust page size to appropriate number so total requests sent to server is reduced. Each request to server adds overhead. Most of API supports page size parameter. Refer to your API documentation how to adjust page size.

Avoid multiple REST API Round trips by caching response

If you extracting multiple arrays from REST API response using above technique then we recommend you to use REST API Task so save response to File or variable first and then use cached JSON data multiple times. This approach is not as friendly as others but it can certainly reduce your total API requests.

Use GZip compression

If your REST API supports GZip compression then check Enable Gzip on Advanced tab of SSIS JSON Source (File, REST API Connector). This will compress response in Gzip format from server and when it comes to your server it will be automatically decompressed by JSON Source connector. This seeds up thing significantly if you have lots of data to download.

Uncheck include parent columns if not needed

By default JSON Source connect includes parent column (Level above last property in filter) from the Filter path you specify. For Example if you specify $.store.employees[*] and “Include parent columns” checked then It will extract all employees and then also include parent level columns for store (Tip: Parent column name start with P_ by default) . If you don’t need parent columns then uncheck from output, this reduces many steps during data extract process.

Download sample package

Click here to download SSIS 2012 Package (Will work in SSIS 2014, 2016)

Conclusion

In this post you learned how to extract data from JSON array and how to make data extract fast. Download SSIS JSON Source (File, REST API Connector) to try everything listed above.

The post Reading JSON Arrays from file / REST API appeared first on ZappySys Blog.

SSIS Code – Check URL Exists using C# Script – 404 Error

ZappySys — Wed, 11 Nov 2015 15:56:18 +0000

In this post you will learn how to use SSIS Script Task (C# Code) to detect specific URL found or not (Detect HTTP 404 Error)

If you using SSIS JSON Source or our REST API Task then if URL/Document you trying to read not found then server may return 404 (Not Found) Exception. In order to handle this error gracefully you can use following Script to store status in a variable and then you can use conditional Control Flow to Skip data loading from Web Service.

Step1: Drag Script Task from toolbox
Step2: Select Language as C#
Step3: Select Readonly variable (In this case “User::url” – String Type)
Step4: Select ReadWrite variable (In this case “User::found” – Boolean Type)
Step5: Edit code and paste following script in your script task

SSIS C# Code – Check URL Exists (Catch 404 Not Found Error)

C# Code: Check URL Exists (Detect 404 Error)

public void Main()
	{
		// TODO: Add your code here
            Dts.Variables["found"].Value = UrlFound(Dts.Variables["url"].Value.ToString());
		Dts.TaskResult = (int)ScriptResults.Success;
	}

        private bool UrlFound(string url)
        {
            try
            {
                var request = WebRequest.Create(url);
                var response = request.GetResponse();
                return true;
            }
            catch (WebException we)
            {
                HttpWebResponse errorResponse = we.Response as HttpWebResponse;
                if (errorResponse != null)
                    if (errorResponse.StatusCode == HttpStatusCode.NotFound)
                    {
                        return false;
                    }
                    else
                        throw; //we only look for 404 Error
            }
            return false;
        }

The post SSIS Code – Check URL Exists using C# Script – 404 Error appeared first on ZappySys Blog.

How to read Twitter data in SSIS using REST API Call

ZappySys — Thu, 22 Oct 2015 16:09:00 +0000

Introduction

In this article you will how to Read Twitter data in SSIS using SSIS JSON Source and SSIS REST API Web Service Task. You will also learn about latest OAuth 2.0 Protocol to simplify REST API access.

Twitter REST API Authentication

In order to fetch any data from twitter using OAuth REST API calls you have to obtain two Keys (Consumer Key [Like UserID] and Consumer Secret [Like Password]). There are mainly two methods to read data from Twitter. Using Application-user authentication and Application Only authentication. Use correct method depending what type of information you want to pull from twitter. Most common method is Application-user authentication . If you want to read more about other methods then click here

Method 1 – Read Twitter data in SSIS using Application-user authentication

In this method you will need twitter user account to connect Twitter API. Once first time authorization is done you don’t have to re-authenticate. Certain type of API calls only allowed by this method (such as POST new tweet using API). For any API call in Twitter very first step is create OAuth App (i.e. Register new App in Twitter developer portal).

Using Default OAuth App Created by ZappySys

To make your life easy ZappySys provides default App for certain OAuth providers (e.g. Google, Twitter). If you decide to create your own app for whatever reason then check next section on how to register twitter OAuth Application. See below screenshot how to use Twitter OAuth connection using Default App. Once you click Generate Token it will prompt you to login using Twitter account and then you can grant permission.

Step-By-Step

Download and Install SSIS PowerPack FREE Trial from this link
Create new SSIS Project and add new package.
Open Package. Go to control flow. Drag and drop Data Flow task from SSIS Toolbox
Dragging and dropping Data Flow Task into Control Flow
Go to Data flow designer.Drag and drop ZS JSON Source Component and double on it to configure it.
Drag and Drop JSON Source Component

Enter any twitter API URL you want to call such as below

https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=zappysys&count=4

Now check use credentials option
Select new ZS-OAuth connection from dropdown
When new connection dialogbox pops up select Twitter from providers dropdown. Select Default App option for now.
Click Generate Token. It may ask you to login using your Twitter account credentials. If prompted click Approve.
If twitter OAuth grant approved then you will see Access Token and Access Token Secret both populated
Click Test Connection and if it works then click OK to close connection
On JSON Source you can now click Preview to see data from Twitter
You can optionally specify Filter expression [Click Select Filter button] to select result from specific JSON Node. For example if you want to select all Hashtags used inside each Tweet then select Filter like $.entities.hashtags[*]
Click OK to close the UI
Drag ZS Trash Destination from Toolbox (Or you can use OLEDB Destination if you want to load inside SQL Server)
Connect JSON Source to ZS Trash Destination
Execute SSIS package

Call Twitter REST API in SSIS – using OAuth Connection (Default App)

Using Custom OAuth App Created by you

For some reason if you dont want to use Default Twitter App then you can register Custom App. It requires few extra steps listed in next section but it wont take more than few minutes. In Custom App option you have to specify ClientId and Client Secret (in above screenshot). Once you click Generate Token it will prompt you to login using Twitter account and then you can grant permission.

Register/Create OAuth Application for twitter API Access

Now lets look at how to register OAuth Application for Twitter API access.

Goto https://apps.twitter.com/app/new and create new app by providing necessary information. Don’t get confused by calling it App, its just way in twitter to create multiple API Access Keys so you can grant different level of access to different users. Once app is created you will be taken to page where your Consumer Key and Secret Key will be listed.

Create Twitter Application – REST API Access for Developer

Obtaining Consumer Key and Consumer Secret from Twitter

Once app is created you can go to Keys and Access Tokens tab. Here you will find Consumer Key and Consumer Secret.
Consumer Key can be Public but Consumer Secret must not be shared (think it like a password).

Obtain Twitter API Consumer Key and Consumer Secret – REST API Access for Developer

Method 2 – Read Twitter data in SSIS – Application-only authentication

For some reason if you don’t want to use Twitter user account to access data (e.g. Giving access to consultant so he can access you company Twitter Account via API). In this scenario you have to use Application Only method. In this method you don’t authorize application using Login form.

Here is our basic flow to access twitter data.

Get Bearer Access Token by calling https://api.twitter.com/oauth2/token service (Use POST method and Pass BASE64 encoded ConsumerKey and Consumer Secret)
Call any twitter service by passing Bearer AccessToken we retrieved in previous step.NOTE: Any access to twitter service is over HTTPS so automatically your tokens passed along request is encrypted before sending over wire unless someone know how to hack SSL 🙂

Application Only Authentication using REST API task – Get Access Token

Once we create SSIS Package – First step to access any twitter data will be get Access Token. As you can see from below screenshot we have called https://api.twitter.com/oauth2/token service URL with POST method. Notice how we have supplied POST data and 2 headers. Authorization header contains BASE64 encoded value of YourConsumerKey:YourConsumerSecret

SSIS Twitter REST API Get Token

Fetch data from Twitter using JSON Source – Deformalize nested JSON

Once we have Authentication Token we are ready pull twitter data using JSON Source. Check below screenshot how we have supplied Token in Authorization Header. JSON Source can make your JSON look like normal table (It also de-normalize nested JSON into flat dataset. If you want to extract subset of JSON then simply specify JSON Path expression e.g. $.data.users[*]

Read JSON response of Twitter REST API in SSIS

Load Twitter JSON data to SQL Server

You can easily connect your SSIS JSON Source to OLEDB Destination if you want to load Twitter Data to some RDBMS such as SQL Server, MySQL.

Handling paging of large REST API result set with twitter data – looping/cursoring

Most of REST API limit total data sent in single response. So if you wish to get all records then you have to loop through multiple results. Twitter provides looping mechanism using CursorClick here to read more

In our case SSIS JSON Source Supports Paging very well so we are covered. To loop through multiple result sets of twitter data simply configure following 3 properties. See below screenshot.

Twitter REST API – Paging Example -Loop through resultset using cursor

Word of caution about too many requests

Twitter does not allow you request too much data too quickly so be careful how many requests you make :). Check their official page on twitter Rate Limit for REST API. You can add delay after each request if you doing pagination. Go to Throttling tab of JSON/XML Source. You can change setting there based on API restriction. For example if API allows only 30 requests per minute then adding 2 seconds delay will make sure you wont exceed 30 requests in 1 minute.

To learn more about rate limit of Twitter API check this table

Handling Twitter API Date Format (Parse Twitter Date/Time)

Twitter returns date in the following format e.g. Fri May 03 15:22:09 +0000 2013 to parse this to correct date/time datatype you can use Date/Time Handling Tab of JSON/XML Source.

Change Custom Date Format as below and preview Twitter data.

Parsing Twitter API date/time format

Download related files

Below is sample SSIS package
Twitter Demo SSIS 2012

Conclusion

So you have now seen how easy it is to access twitter data with OAuth 2.0 using SSIS JSON Source and SSIS REST API Web Service Task. We have also seen how to loop through large resultset using inbuilt Paging support of JSON Source.

The post How to read Twitter data in SSIS using REST API Call appeared first on ZappySys Blog.

How to Call REST API in SSIS – Read JSON / XML / CSV

ZappySys — Mon, 03 Aug 2015 04:41:10 +0000

Introduction

The REST API is becoming increasingly popular in the Data Integration landscape. If you are an SSIS Developer (not a coder) and stumbled upon a question What is REST API and how to consume API inside an SSIS package?” then you are at the right place :). We encourage you to read this article carefully and follow the various links we provided.

In this article, you will learn about many REST API Tips / Tools on the topic of how to call a REST API using the SSIS REST API Task, JSON Source, or XML Source. You will also learn how to perform HTTP GET and HTTP POST Requests without needing to know any programming languages (e.g., C#, Java, Python) – simply do a drag-and-drop operation in SSIS.

What is REST API / RESTful Web Service

Let’s first understand what a REST API is, also sometimes referred to as a RESTful Web Service. This is a new buzzword that you will likely hear frequently. More and more services are available in the Cloud, which makes it obvious to come up with some way for you to access data in a more firewall-friendly manner. What can be a better option than accessing it using a several-decade-old HTTP protocol, so no additional configuration is required for anybody accessing Cloud Services?

So, in short highlights for REST Web Service

REST Web Service is a stateless client-server service model
By passing an HTTP verb, you can perform server-side action over the standard HTTP protocol (e.g., GET, POST, LIST, DELETE, HEAD)
You can pass parameters via URL query string and via HTTP Headers
If you are doing HTTP POST, then you can pass additional data in the Request Body along with the other two methods described above
REST API works by sending an HTTP Request and getting an HTTP Response
The HTTP Response of a web service can contain Headers and Response Data, which can be in binary, text, JSON, XML, or other formats.
You can secure your data by simply sending it over the HTTPS (Secure HTTP) protocol or encrypting values passed along with your request.

Tasks/Components in SSIS for Consuming RESTful API / WebService

	JSON Source (Read from REST API, JSON file or OData Service): Use this dataflow component when you have to fetch data from a REST API web service like a table. This component enables you to extract JSON data from web services and de-normalize nested structures, allowing you to save the data to a Relational database, such as SQL Server, or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local JSON files or direct JavaScript strings (Wildcard pattern support, e.g., c:\data\file*.json).
	Web API Destination (POST data to API URL): Use this data flow component when you need to load data into a target system via API calls (POST to URL). E.g., Loading Contacts into a Marketing System like MailChimp for email campaigns. Loading documents into CRM or document storage systems, e.g,. Couchbase or ElasticSearch.
	REST API Tak: Use this task when you don’t want to pull REST API data in tabular format but want to call the REST API for POST data to the server, DELETE data from the server, or things like download an HTML page, extract Authentication tokens, etc, where you are not necessarily dealing with data in tabular format. This task also allows you many other options, such as saving the RAW response into a variable or a file.
	XML Source (SOAP, File, REST): Use this dataflow component when you have to fetch data from an XML or SOAP web service and consume data like a table. This component allows you to extract data from a web service and save it to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local XML files or a direct XML string.
	CSV Source (File, REST): Use this dataflow component when you have to fetch data from File or REST API which returns data in CSV format . This component allows you to extract data from a web service and save it to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local CSV files or direct CSV string.

Using SSIS JSON Source to read from REST API and load into SQL Server

If you need to consume a REST API Service and store the result into SQL Server or any other RDBMS/FLAT File, then you can check this SSIS JSON Source. The JSON Source can deserialize your nested JSON (it also supports JSONPath filter expressions).

REST API Task – Ad-hoc web requests – Call REST API (POST, DELETE)

HTTP GET Request using SSIS Web Service Task or JSON/XML Source

Calling the REST API from the SQL Server BI Stack usually requires programming skills. However, in this section, we will learn how easy it is to call a RESTful Web Service using the SSIS Web Service Task, JSON Source, or XML Source (all of which are drag-and-drop components).

In this example, we will retrieve JSON data from an OData web service.

Download and Install SSIS PowerPack
Create a new sample SSIS package in BIDS/SSDT
From the SSIS Toolbox, look for items starting with “ZS”. Drag and Drop [ZS Rest API Web Service Task] to Designer Surface
Configure HTTP Request properties as below
Configure HTTP Response properties as below
Run SSIS Package

Pass values in HTTP Headers

You can pass a value in HTTP Headers using a direct approach or a dynamic approach. If you wish to pass a value from an SSIS variable, then use a variable placeholder, e.g., {{User::varSomeVariable}}. You can also use a variable format specifier, e.g.

Format Date: {{User::varSomeVariable,yyyy-MM-dd}}
Encode into BASE64: {{User::varSomeVariable,BASE64ENC}}
Decode into plain text from BASE64: {{User::varSomeVariable,BASE64DEC}}

Calling REST API in SSIS using REST API Task, Pass headers, Body, Url Parameters

SSIS REST Api Task – HTTP GET, Test SSIS Web Service Call, Pass Custom Header

REST API response validation ( By status code, content, header)

Pass values in the Query String dynamically

Most web services usually accept parameters via the query string. You can also use variable placeholders to make your URL with Querystring dynamic (e.g., https://mysite.com/orderservice/?startrow={{User::varStart}}&endrow={{User::varEnd}}

HTTP POST Request using SSIS Web Service Task or JSON/XML Source

When you select the HTTP POST method, the content-type is set to application/x-www-form-urlencoded by default. With the POST method, you can send POST data. When the POST method is selected, the data textbox becomes editable. You can also use variable place holders in POST data to make it dynamic.

POST data is usually in key/value format (e.g., user=abcd&pass=mypass123), but sometimes the service requires data in JSON format or XML format. In that case, you can submit that way. Depending on the service requirements, you must set additional headers to indicate the content type (please refer to your service documentation).

Uploading files using HTTP Multipart/form-data POST Request

Many API support uploading files along with your POST request. Refer to this article to learn more about how to upload files using REST API calls.

Passing Credentials to your Web Service

If your service requires authentication, ensure that you pass the credentials needed in the correct format. Below is a list of some of the most common authentication techniques.

There are mainly three connections:

Using HTTP Connection

HTTP Connection supports multiple methods for authenticating to your API. Here are some popular modes.

Basic Authentication (UserID and Password Approach)
Dynamic Token approach (Two-step process – First get Token and then Call API (Pass the token extracted in the first step)
Static Token
HMAC / Hash Signature Authentication
SOAP WSS Security
Windows Authentication (NTLM)
Client Certificate

OAuth Connection

Many modern APIs now support OAuth Standard (i.e. v2.0, v1.0, v1.0a). ZappySys offers out-of-the-box support for this standard using OAuth Connection Manager

OAuth authorization
- Authorize Code Grant (3-legged – using Login Popup)
- Client Credentials Grant (Simple key/secret – No login Popup)
- Password Grant (Use of userid /password rather than only key/secret)

Basic Authentication – Set Authorization Header (Base64 Encoding)

The HTTP Standard supports BASIC Authorization mode, in which you can pass the username and password as a base64-encoded string. Check this article for more information on how to pass a BASE64 Encoded Authorization Header

Token-based approach – such as OAuth

This is becoming the most common approach, utilizing protocols such as OAuth, where the user first authenticates to the service using an AccountKey and SecretKey. Once authenticated, you receive a token that can be valid for a certain duration (or an infinite duration). After you receive the token, you can call services (each service call will include this token). Parsing a token from the initial response can be achieved through the REST API Response Filter Expression (e.g., $.token) or using the JSON Parser Task.

OAuth Authorization

All REST API Tasks/Components in Zappysys SSIS PowerPack support OAuth Authorization. OAuth is gaining popularity, and many larger companies have already adopted this standard (e.g., Facebook, Twitter, Google, Salesforce). With this method, you can connect to the REST API Service without storing your User ID/Password. Only the first time, you need to log in using your credentials to obtain an initial Token, and after that, the service will continue to use the Access Token rather than your username and password. Once the Token expires, it can automatically renew. Check this for more information about using OAuth.

Windows Authentication (NTLM)

By default, ZappySys REST API Tasks/Components use your default credentials. You can also use the HTTP Connection Manager with the “Use Credentials”> “Use Windows Authentication” option. See this article

Pass Client Certificate

Sometimes you may have to pass a client certificate along with your web request. The server checks the certificate passed along with your request and, if it matches, proceeds with your request. Here is the real-world use case How Azure Management Api uses Client Certificate.

ZappySys HTTP Connection Manager and OAuth Connection Manager both support passing a Client certificate.

Saving HTTP Web Service Response to a File or Variable

Let’s break down how to save the response from a ZappySys REST API Task into either an SSIS variable or a file within your SSIS package.

Scenario: Imagine you’re using the ZappySys REST API Task to retrieve JSON data from a URL. You want to store this data for further processing within your SSIS package.

1. Saving the REST API Response to an SSIS Variable

This is ideal when you need to manipulate the data within your SSIS package.

Add a ZappySys REST API Task to your Control Flow:
Drag and drop the task onto your SSIS package’s Control Flow.
Configure the REST API Task:
- Set the Connection Property to your REST API Connection Manager. (Optional)
- Specify the URL of the API endpoint.
- Configure any necessary headers, query parameters, or request body.
- Go to the “Response Settings” tab.
Configure the Output Variable:
- Check-mark Save Response content
- Select Save Mode as “Save to Variable”
- Select or create an SSIS variable of type String. This variable, named User::WeatherResponse will store the REST API response.
Saving the REST API Response to an SSIS Variable

2. Saving the REST API Response to a File

This is useful for archiving the response or when you need to process the data outside of SSIS.

Add a ZappySys REST API Task to your Control Flow:
Drag and drop the task onto your SSIS package’s Control Flow.
Configure the REST API Task:
- Set the Connection Property to your REST API Connection Manager. (Optional)
- Specify the URL of the API endpoint.
- Configure any necessary headers, query parameters, or request body.
- Go to the “Response Settings” tab.
Configure the Output File:
- Check-mark Save Response content
- Select Save Mode as “Save to File”
- In the “Enter File Path” property, specify the path and filename where you want to save the REST API response. For example, if you are fetching JSON data from a product API, you can save it to the following file: C:\Data\Products.json.

Saving the REST API Response to a File

REST APIs enable array filtering to extract specific data from JSON or XML responses. This is achieved through techniques such as query parameters, JSONPath (for JSON), and XPath (for XML), allowing for the precise selection of elements. Additionally, regular expressions can be employed for string-based filtering and data validation.

REST APIs enable array filtering to extract specific data from JSON or XML responses.

Key Setting: “Treat response as binary”

As highlighted in the steps, the “Treat response as binary” option in the “Response” tab is the most important setting for correctly saving binary files. If you don’t check this, the task might try to interpret the binary data as text.

Example Scenario:

Let’s say you want to download a ZIP file from a REST API or a Public URL:

API URL: https://example.com/api/download/archive.zip
File Path: C:\Downloads\archive.zip
In the ZS Rest API Task’s “Response” tab, you would:
- Check-mark Save Response content
- Select Save Mode as “Save to File”
- Check “Treat response as binary”.
- In the “Enter File Path” property, specify the path and filename where you want to save the REST API response. C:\Downloads\archive.zip.
REST API Task – Download File

By following these steps, you can effectively use the ZappySys Rest API Task to download and save binary content from REST APIs to files within your SSIS workflows.

Save HTTP Web Response Headers / StatusCode

If you wish to save the response to a file, then go to the response tab of the REST API Task and check the “Save Response” option, and you can select Save to File option from the dropdown

HTTP Web Response Validation

REST API Task supports validating your response for a specific header, status code, or content value. See the Validation tab for more information. Using this feature, you can throw an error if a certain header is missing from the response or reject the response if the StatusCode is other than 200.

REST API Pagination (Loop through multiple requests)

Another popular concept in REST API is paging. If your response is significant, the server often returns a partial response. You must then ensure that you consume the remaining data by requesting all subsequent URLs until the last page is returned.

Read this article to learn more about rest api paging

HTTP Web Response Error Handling

REST API Task supports rich error handling. You can ignore specific errors and save the error flag into a variable based on any of the following criteria.

Continue on any error
Continue on error with specific Response Code (e.g. 404)
Continue on error with a specific string in the message

Read/Write and Parse Cookies for HTTP Web Request/Response

Many times, you need to supply an authentication token via cookies, or any other information must be passed via cookies. In such cases, the REST API Task provides complete support for reading, writing, or parsing individual values from a cookie string. Look at the Cookies tab for more information. The Cookie Mapping grid can help you to map a cookie value to an SSIS variable (Write Cookie value to a variable).

Read this article to learn more about passing cookies with your web request

Changing Headers/URL or POST data Dynamically

Most of the fields on the REST API Task, JSON Source, and XML Source support placeholders.

Use a placeholder anywhere in the following fields using {{User::yourVariable}} format. If you edit the value, you may see the Insert PlaceHolder option.

Url
Body
Headers

You may format datetime using special syntax like this {{User::myDate,yyyy-MM-dd HHmmss.fff}}

Extract a single value from JSON/XML Web Response (e.g., Token)

If your web response is in JSON or XML format, then you can filter it using Expression (e.g., JSONPath or XPath). Go to the Response Tab and select Content Type Format from the dropdown. Enter an expression to select a nested sub-document or a value from your response. If you want to save individual properties from the response into multiple variables, then use JSON Parser Task

The most common use case is if you are getting a token from your JSON REST API service. Assume that Service sends you a response in the following format, but you only care about the access_token property from the response below. In that case you can use JSON Path expression like this $.data.access_token

{
    data: {
        access_token: 111223344444,
        expires_in: 3600
    }
}

REST API Testing / Debugging Tools (3rd party)

Now, let’s look at a few popular tools that you will need to test/debug REST API calls during your development cycle.

Testing Rest API

There are a few ways to test REST APIs, and each tool has its pros and cons, but the most popular tool to test REST API is Postman, so we will start from there

Postman – GUI
Postman offers an excellent User interface to test the API. You can edit URLs, methods, and parameters, and add authentication with ease. See our article on how to migrate Postman settings to zappysys UI. Many API Providers offer easy-to-use Postman settings files for Testing API in Postman. If your API offers such a file, simply import it into Postman and start testing before using the ZappySys UI.
SoapUI – GUI
SoapUI is another tool to test SOAP or REST APIs. They are primarily known for SOAP API Testing, but in recent versions, they have enhanced REST API Testing capability.
cURL – Command line
If you want to call/test the API via the command line, then this is the most popular tool. Many API vendors use cURL syntax in their API documentation. Here is how to migrate cURL syntax to ZappySys UI.
Fiddler- GUI
Fiddler has limited capability to test API call – See Replay in Composer option
ZappySys REST API Task
You can use the ZappySys REST API Task to test the API inside Visual Studio. Edit URL / Headers / Body, etc, and click the Test Request Response option

Debugging REST API – Proxy

So far, we have seen client tools to test Request/Response for the URL you entered in the UI, but what if you want to Sniff Requests sent by some applications for which you have no way to know what Request Data / URL, or Body is sent behind the scenes? What response is it getting back and so on? Well, not to worry… Telerik Fiddler is the most popular tool at your fingertips and it’s FREE.

Fiddler – HTTP / HTTPS Proxy / Debugger
Wireshark – Low-level Network Packet Sniffer

Conclusion

The REST API is becoming increasingly popular every day. With that, Data integration from RESTful services is going to be a challenge. The SSIS REST API Task and SSIS JSON Source can help resolve some of the issues you may encounter with SSIS. Use JSON Source if you need to store data into a SQL Table. Use REST API Task if you need to make an ad-hoc HTTP/REST request (such as get token, Delete Record, etc). ZappySys REST API Components gives you total control over your REST API Integration challenges without learning a programming language (e.g., Python, Ruby, C#, Java).

The post How to Call REST API in SSIS – Read JSON / XML / CSV appeared first on ZappySys Blog.

json source Archives | ZappySys Blog

How to convert JSON to CSV File using SSIS

Introduction

Prerequisites

Step-by-Step – Export JSON to CSV File with SSIS

Using API as JSON source

Exporting multiple JSON arrays to a CSV

Transforming JSON array

Common Errors

Truncation related error

Authentication related error

Deployment to Production

Download a sample package

Conclusion

How to create BIML custom task or component

Introduction

Use BIML to create SSIS packages that use PowerPack tasks or components

BIML Elements

Create a BIML custom task with CustomTask

DTSX snippet

BIML snippet

Create BIML custom components using CustomComponent

DTSX snippet

BIML snippet

Resources

Download sample BIMLs

How to make Path / URL dynamic in SSIS

Problem

Create a variable

Make an HTTP address dynamic

Creating a Dynamic Request Body

That’s How You Make It Dynamic

How to register Facebook OAuth App for Graph API Access

Introduction

Step-By-Step: Create Facebook OAuth App (i.e Register OAuth App)

Test Facebook OAuth App (Consume graph api in SSIS)

Fetch from Facebook Graph API (Load data into SQL Server)

How to register Google OAuth Application (Get ClientID and ClientSecret)

Introduction

Create Google Project

Enable Google API for Integration

Create OAuth Credentials (Client ID / Secret)

Publish App

Using Client ID and Client Secret in ZappySys OAuth Connection

Using System Default Browser to get the token

Untrusted App Warning

Select the account

Copy code and finish the flow

Why I didn’t get refresh token back?

What to do if Google Refresh Token expires every 7 days

What is Google OAuth AccessToken

What is Google OAuth RefreshToken

Conclusion

How to do REST API Pagination in SSIS / ODBC Drivers

Introduction

Debugging REST API Requests

Method-1: REST API Pagination via URL Parameter

Method-2: REST API Pagination via URL Path

URL with Skip Parameter (Offset starts at zero)

Method-3 : REST API Pagination via Response Attribute (Next Link or Cursor)

Next Link Method (Partial or Full URL)

Next Link – Detect the last page based on empty rows (Regular Expression)

Cursor Method

Method-4: Pass Page Number in Body Attribute

Method-5: REST API Pagination via multiple URL parameters (e.g. Start and End)

Method-6: REST API Pagination via Header (Next Link in Header)

Next Link in Custom Header

Next Link in Header as per RFC 5988

Method-7: REST API Pagination using Cursor / Continuation Token from Response Header

Pass Token via Request Header (Same header name)

Pass Token via Request Header (Different header name)

Pass multiple Response Headers via Request Headers (Azure DocumentDB style usecase)

Pass Token via URL Parameter (Query String)

Pass Token via URL Parameter (Query String) – with custom Regular expression (Usecase: Salesforce BULK query API)

Method-8: Pass Pagination Token in Request Body

Passing Page Token in Body (Blank Token in first request allowed)

Passing Page Token in Body (Blank Token in first request)

Passing PageToken or Counter in Next API Call Body

Output Page Number as column

Last Page Detection