In our previous post, we saw how to read Salesforce data using SSIS Salesforce Source (Which uses SOAP API under the hood). This may be slow if you have millions of rows because Salesforce Source paginates every 2000 rows, so it’s not a good option to read millions of rows to iterate on a large Salesforce dataset (Table or Query). To increase the read speed, you can use BULK API. This article explains how to read salesforce data using BULK API and increase read speed.

NEW Version (v5.5+)

In the new version (v5.5), we have added an option to query in Bulk Mode. This means if you have millions of Salesforce object rows to read, no worries; it may be very fast in the new version. Previously, it used to take a long time because it read in batches of 2000 rows in each request. With the Bulk API, the batch size is really large.

Here is how to set Salesforce Bulk API Mode in SELECT query (Read operation).

1. Install new version of SSIS PowerPack. Make sure it is v5.5 or higher.
2. Open existing Salesforce Source UI
3. Make sure you check [Use Bulk API] option like below. if you do not see this option then most likely you are using older version.
   -OR-
   Just add #bulkmode prefix in the SQL query like below, it will use Bulk API
   

   Bulk API mode for SSIS Salesforce Source (Read Large amount of data using SOQL Query)

OLD Version

If you are using an older version of SSIS Power Pack (v5.4.0 or older), you can follow the steps mentioned in the following section.

We will see how to query using Salesforce Bulk API in SSIS. It involves three simple steps like the one below.

1. Call BULK API – Submit SOQL Query and receive a JobId back (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

2. Keep checking Job Status Until Status is JobComplete (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

3. Read Query Result  (Use SSIS CSV Source)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Now, let’s look at everything in detail.

Obtain Salesforce Credentials

The first step to call Salesforce API is to obtain credentials (security token or OAuth ClientId /Secret)

Method#1 – Check this article to use UserId/ Password / SecretToken
Method#2 – Check this article to use the OAuth Connection Option

Create Salesforce Connection

Once we have Salesforce Credentials, we can create a new Salesforce Connection (You can do this from Task UI or the below way)

1. Open Visual Studio ( Assuming you already have SSIS installed along with ZappySys PowerPack )
2. Create a New SSIS Package Project.
3. Right-click on the Connection Manager panel and click on New Connection.
   
4. Select ZS-SALESFORCE Connection Manager from the Connection Managers list and Click on the Add Button.
   
5. Now in Connection Manager UI, Enter your SalesForce Credentials (OAuth Client Id/. Secret –or— User Name, Password and Security Token), and leave all other properties as it is.
   

Call Create Job – Retrieve / Save JobId

Once the connection is created, we can create our first step. Basically, we will call this API to create Job (Bulk Read)

1. Here, in Visual Studio, drag and drop the ZS Salesforce API Task on the design surface.
2. Double click the Task and configure Request Settings Tab like below
   – Select Salesforce Connection (Created in the previous section)
   – Check Use direct URL
   – Enter URL like below (Replace {your-instance} with your own instance (i.e. na9) . For more information on how to find out my
   salesforce instance id check this article
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

   – Request Method POST
   – Enter Body like below (You can use your own SOQL query). Keep in mind SQOL must be encoded for JSON format. This means the entire SQL must be in one line (replace new lines with \r\n , quotes with \” and tabs with \t)
   For example, to execute SELECT Id, Name FROM Accounts, you can write something like the one below in BODY.
   

   {
     "operation": "query",
     "query": "SELECT Id, Name\r\nFROM Account"
   }

   The above is for below (with a new line)
   

   SELECT Id, Name
   FROM Account

   

   Create Job – Call Bulk API to Read / Query using SOQL

3. Now Click Test Request (Copy the JobId which we will use in the Variable for Designtime Testing)
4. On the Response Tab, configure it like the below.
   – Select Response content Type = Json
   – Enter $.id as the expression
   – Check Save response option
   – Select Save to Variable and click New to create a variable. Name it JobId and enter a sample JobId value (copied from the previous step)
   

   Extract Salesforce JobId save to SSIS Variable

Do Status Check – Look for JobComplete

Once we create a job and store JobId in the variable we have to keep checking Job status every X seconds. Don’t worry you don’t have to use Loop or timer control for that. ZappySys made it so simple to check such thing without Loop /Timer.

1. Drag and drop the ZS Salesforce API Task in the design surface.
2. Double click and configure like below. URL will be different this time. Use variable as below and replace your instance ID
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

   

   Check Salesforce Job Status Periodically (Without Loop + Timer Controls)

3. Now go to the Status Check Tab and configure it, as shown below. It will keep looking for the words below in the response to decide whether it is a Success or Failure. If those words are not found, they will be checked every 5 seconds (Maximum 1200 secs).
   – Check Enable Status Check option
   – For Success Value – Enter JobComplete
   -Check Fail task option
   – Enter Cancel|Failed|Abort in the text box
   

   Configure Job Status Check Loop using Salesforce API Task

Read data (Compressed CSV Format)

Last step is read the result from completed Salesforce Job. Its in CSV format so we will use Advanced SSIS CSV Source which supports API connection to Salesforce. For that, we will use the same Salesforce API Task used before.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
2. Double click on the DataFlow task to see DataFlow designer surface.
3. Here, In Visual Studio, drag and drop the ZS CSV Source (Web API or File) in the design panel
   
4. Double click on ZS CSV Source (Web API or File) Configure it like below.
   – Enter URL to read result like this (You can supply OPTIONAL maxRecords parameters to control how much data comes in a single response)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results
   --OR--
   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results?maxRecords=50000

   – Check Use Credentials and select Salesforce Connection
   – Click Preview (assuming JobId variable has a valid completed JobId)
   

   Read Salesforce Bulk API Job Result – Using SSIS CSV / API Source

5. Now, set up Pagination as below to read all records using an automated loop. You will need VERSION Released after Jun 20, 2022 to use Regular Expression feature in Header Value extractionBy default, Salesforce doesn’t send all records in response so we need to setup pagination. In below example we are reading Sforce-Locator attribute from Response and sending it to the next URL as locator=<continue-token> value
   – Choose Response Header Contains Continue Token 
   – Enter below expression in Page Num Indicator
   

   locator=Sforce-Locator(^((?!null\b).)*$)

   Basically, the above expression will keep looping until we find Sforce-Locator : null in the response header.
   

   Pagination for Salesforce Bulk Query API (Get Header Value from Sforce-Locator)

6. Click OK to save and now run entire package like below.

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Conclusion

In this article we saw how to use various ZappySys Components / Task to achieve complex Salesforce API calls to read Query data using Bulk API. Download SSIS PowerPack and try yourself and also explore 70+ other tasks /components not discussed in this article.

The post How to query salesforce data using Bulk API in SSIS appeared first on ZappySys Blog.

In our last post (Regex Cheat Sheet) we explained use cases of SSIS Regular Expression Parser Task . Now lets look at some real world usecase. Basically in this article we are going to Extract data from audit log using SSIS. For demo purpose we will use log generated by SSIS Export CSV File Task output. When you export many tables dynamically using this task you want to know how many Row count exported for each Table. This information is logged in SSIS Output Log but no easy way to capture in SSIS variable. So what we will do we will use few tricks to capture that data using Regular Expression usecase.

So let’s get started.

Setup Export CSV Task (Output Multiple Tables to CSV Files Dynamically)

For example purpose we will use Export CSV File Task but you can use this technique for any other Tasks.

1. Drag and Drop ZS Export CSV Task from SSIS Toolbox and configure as below.
   
2.   Configure Export CSV Task to output multiple tables as explained here
3. Now create 2 SSIS User Variables
   1. FilePath (String Type)
   2. RowCount (String Type)

That’s it… now in the next section we will setup event handler to capture output from Log.

Setup EventHandle / Extract Audit Data using Regex

1. Now select task and click Event Handler tab
2. Select Executable Name from the dropdown, Select Event Name OnInformation from Handler Dropdown
3. Click Create Hyper Link to Create Event Handler like below
   

   Create Task Level Event Handler in SSIS

4. Now Drag below 3 tasks and connect like this
   1. Script Task (We will use as dummy start (No configuration needed)
   2. ZS Regular Expression Parser Task
   3. ZS Logging Task (Optional) but we will use to Output Extracted Values
      
5. Connect Dummy Script Task to ZS Regular Expression Parser Task and Right click on Green connected Arrow and Use Expression as below. Click OK to save
6. Configure ZS Regular Expression Parser Task as below
   1. Enter

      {{System::ErrorDescription}}

        as Direct Value
   2. Enter Below two mappings
      1. For RowCount set
         

         Total (\d+) records written to : (.*){{1,1}}

      2. For FilePath set
         

         Total (\d+) records written to : (.*){{1,2}}

7. Here is how it will look like
   

   Extract data from audit log using SSIS Regular Expression Parser Task

8. Thats it… Now connect 2nd step to last Logging Task…. in that Task you can set Text like below just to log extracted data. You can also use Execute SQL Task to log into Database Table
   

   {{User::FilePath}} ===> {{User::RowCount}}

9. Now run Package and you will see audit data we extracted for each file. Once File is exported we capture OnInformation event… and if Message contains substring like “records written to” then we extract data into variable for Row Count and Path then you can save to desired place for auditing purpose

The post Extract audit data from SSIS Execution Log (EventHandler and Regular Expression) appeared first on ZappySys Blog.

In this blog, we will learn how to call SOAP / REST API using Dynamic Token in SSIS (i.e. Two steps authentication approach – Fist Call Login API to get token and then call API).

In our previous blog post, we saw how to call Web API using some industry standards approaches, such as OAuth 2.0 Authorization or Basic Authorization. However, not all APIs follow the industry standard to authenticate API requests. After talking with many customers we noticed that very common non-standard technique to authenticate API is Dynamic Token Approach. In this approach first, you have to call Login request to get a temporary token (i.e. sessionid ). Once you get that temp token you can supply it along with your API requests. These temp token/session id usually expires in short time (i.e. expire in one hour or so).

Sounds complicated? Well not really if you use ZappySys SSIS PowerPack or ODBC Drivers for API. In this post, we will go through the steps, to authenticate using Dynamic Token approach. We will show you features of  ZappySys SSIS HTTP Connection Manager. For example purpose, we will call XML SOAP API using SSIS XML Source Connector but same concepts can be applied to read from JSON API using  SSIS JSON Source Connector or CSV API using CSV Source Connector.

So let’s get started.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS XML Source Connector and  SSIS JSON Source Connector that will help you to call SOAP / REST API, read various data from SOAP / REST API with SSIS.

To get token from login / authentication URL you have to gather a few pieces from your API documentation.

• How to call Login API for your API – (This is not the same URL as Vendor Site Login Page, but its usually API Login endpoint e.g. https://somecompany/api/v2/login)
• API Response Format (e.g. JSON, XML ,CSV or Random Text )

In our below example steps, we will use XML API which requires POST method first to pass Userid / password to authenticate. If Authentication is successful it may return a response in XML format which contains token.

In the next step, we will call other API (e.g. get-data) and pass token as Authorization Header. All these steps will be done automatically using  ZappySys SSIS HTTP Connection Manager.

Create HTTP Connection for Dynamic Token

To get API response data using SOAP / REST API call, you need to get an API Token from your login request URL. I am using ZappySys sample SOAP API to show you for the Demo purpose. Follow the steps mentioned below to get API Token:

Create in SSIS

1. Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
   

   Connection Manager Window: Create New Connection
2. Select ZS HTTP Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
   

   List of SSIS Connection Managers – Select HTTP Connection Manager

Create in ODBC

To use HTTP Connection in ODBC Drivers you can change connection Type to HTTP like below and it will show similar UI discussed in this article.

Create an HTTP connection inside an ODBC driver

Configure HTTP Connection Credential Type

Once you add new SSIS ZS HTTP Connection Manager, Go to General Tab and Configure HTTP Connection Manager for Two Step Authorization Flow. Enter API URL, Login Credentials, and configure some basic settings for the same. We used Login URL as just example but you can enter any valid API URL endpoint here. Actual Login URL goes on Dynamic Token Tab (see next step)

SSIS ZS HTTP Connection Manager: General Tab Configurations Settings

Configure Request Settings

Once you enter Username / password details on General Tab now we need to setup some information like Login Request URL, Body, Headers. Lets look at few options.

Passing User ID /Password in Body

1. Go to Dynamic Token tab, Enter the following pieces.
   1. Token/API Login URL
      For our example enter API Login URL as:
      

      https://zappysys.com/downloads/files/test/soap-login.aspx

   2. Request Method as POST
   3. Request Body as below. Notice we used 

      [$userid$]

        and 

      [$password$]

        placeholders. These are replaced automatically at runtime so you do not have to hard code in Body and keep it secure. (See General Tab to enter UserID/Password)
      Pass User ID / Password in Body
      

      <soapenv:Envelope xmlns:soapenv=""http://schemas.xmlsoap.org/soap/envelope/"" xmlns:urn=""urn:partner.soap.sforce.com"">
         <soapenv:Body>
            <urn:login>
               <urn:username>[$userid$]</urn:username>
               <urn:password>[$password$]</urn:password>
            </urn:login>
         </soapenv:Body>
      </soapenv:Envelope>

   4. Select Request Body Content Type. For JSON data it may be application/json. For Key/Value Pair  key1=value1&key2=value2 leave it Default.Passing User ID /Password in Request BODY
      

      SSIS ZS HTTP Connection Manager: Dynamic Token Tab Configurations Settings of Request Tab

Passing User ID / Password in URL (GET Request)

Our Example API pass credentials in the POST body but some API may pass via URL as GET request.

In such case, you can enter like this

https://mysite.com/api/v2/login?userid=[$userid$]&password=[$password$]

[$userid$]

[$password$]

Passing User ID /Password in Header as Basic Authentication

Another scenario is API requires HTTP Basic authentication, (Pass User ID / Password in Header as Base64 encoded value) then you can pass credentials like this:

Authorization: Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>


Here we have used FUN_BASE64ENC function but there are many other functions can be used anywhere in Body or Headers

Using Basic Authentication in Dynamic Token approach

Configure Response Settings

Now lets configure how to extract token (single value) from response. Which can be XML, JSON or sometimes Raw Text. We support 4 modes (default = None means full response text is treated as token). If you want to extract just single value and not full text then use below settings.

XML Expression Type (Use XPath)

1. Now, Go to Response Settings tab under a Dynamic Token tab, Enter Response Settings like Token Extract Mode, Expression Type, Content Filter Expression as below XPath Expression. The expression basically extracts content from sessionid XML node using below XPath Expression.

   //*[local-name()='sessionid']

   

   ODBC ZS HTTP Connection Manager: Dynamic Token Tab Configurations Settings of Response Tab

2. That’s it click on OK button to save your Connection Manager Configurations.

JSON Expression Type (Use JSONPath)

If you API returns JSON response (like Walmart API Example) then you can do something like below. Choose Expression Type=JSON and in Response Content Filter You can enter expression like $.some_property_name  (e.g. $.access_token). Check more about How to write JSON Path expression here.

SSIS ZS HTTP Connection Manager – Get a JSON response for the token

Regular Expression Type (Use Regex)

Another advanced mode to extract token from response is Regex Mode. You can write any valid regular expression like this one to extract just one value.

For example if your response returns raw string with token in double quotes but you want just token and no double quotes around it then try something like below.

Extract Token from response – Remove double quotes around value using Regular Expression

Call API in SSIS with Dynamic API Token

Now once you have setup HTTP connection manager to authenticate and get API token, we are ready for next step. Now we can use that token in SOAP / REST API Service call. So, let’s move to the SSIS Part.

1. Once SSIS Package is open, Goto Data Flow Section, For that Drag and Drop DataFlow Task from the SSIS Toolbox.
   

   Drag and drop Data flow task

2. Now, Drag and Drop SSIS ZS XML Source Component from the SSIS Toolbox (if your API is JSON based then drag ZS JSON Source).
   

   Drag and Drop ZS SSIS XML Source Component

3. Let’s double-click on XML Source Component to configure it’s Properties. Enter Request URL.
   

   https://zappysys.com/downloads/files/test/soap-getdata.aspx

   

   SSIS XML Source: Enter your SOAP Request URL

4. Check Use Credentials (Username/Password OR Token), Select HTTP Connection Manager from the drop-down list, we have created just before.
   

   ZS SSIS XML Source Component: Check Use Credentials and Select Connection Manager

5. Select the Request Method and Enter SOAP Request Body something like this. Notice that we used [$token$] as a placeholder. This is automatically replaced at runtime by ZS-HTTP Connection Manager.Passing Token in Body
   

   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:partner.soap.sforce.com">
      <soapenv:Body>
         <urn:sessionid>[$token$]</urn:sessionid>
      </soapenv:Body>
   </soapenv:Envelope>

   Passing Token in Header
   
   If your API expects token via HTTP Header (e.g. Authorization: xxxx-my-token-xxxxxxxx ) then make sure you configure Auth Scheme and Header name on HTTP Connection. By default it sends Token in Authorization header if you dont do anything.
   

   ZS SSIS XML Source Component: Enter SOAP Request Body

6. Select Content-Type as per your Request, here we are selecting

   text/XML;charset=UTF-8

     and adding 1 HTTP Header:

   SOAPAction: "https://zappysys.com/soap-getdata"

   

   ZS SSIS XML Source Component: Select Content-Type Add HTTP Request Header

7. select Filter Path: set as per your requirements. here for this example, we are setting:
   

   $.soapenv:Envelope.soapenv:Body.urn:Row[*]

   

   ZS SSIS XML Source Component: Select Filter

    

8. Check Include Parent Columns checkbox, for including parent Columns, and then click on Preview button to see the Preview data.
   

   SSIS XML Source Component: See the Preview data

9. That’s it. Click on OK button to Save your Configuration.
10. Finally drag any destination (i.e. OLEDB Destination) from your SSIS toolbox. If you want to load this data into SQL, then Drag OLE DB Destination Component, and configure it.

Conclusion

In this blog, we learned how to call any SOAP/REST API with Dynamic Token, retrieve Service response data with SSIS using XML Source Component OR JSON Source Component   in a very simple way. Furthermore, you can achieve many more functionalities with this tool. Check our blogs/articles on XML Source Component to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the call any SOAP / REST API with Dynamic Token feature with our tools:

• Landing Page for XML Source, you can also find Tutorial Video here.
• Help File: Documentation of XML Source.

The post How to call SOAP / REST API using Dynamic Token in SSIS / ODBC appeared first on ZappySys Blog.

In this short article, you will learn how to write Regular expressions in SSIS (i.e. Regex) and what tool to use to test them. You will also find helpful resources on how to write more sophisticated expressions and learn more about them. For demo purposes, we will use FREE SSIS Regex Parser Task to parse and extract the text using Regex.

You can use Regular expressions in several SSIS PowerPack connectors:

• SSIS Regex Parser Task (FREE),
• Azure Blob Storage Task,
• Advanced File System Task,
• Amazon S3 Storage Task and others.

Writing Regular Expressions in SSIS

Based on the SSIS component you use it will include the filename in filtering options or match the text: See below Syntax to write Regex in ZappySys tools. We support additional construct {{X, Y}} at the end of Regex to control two parameters. This additional construct is useful for data extracted from matching regex. If you want to Test patterns and not worry about data extraction then no need to use last {{

Occurance_Index

Group_Index_Or_Name

Syntax:

<your Regular Expression>[{{Occurance_Index|*[,Group_Index_Or_Name]}}]

Where:

Occurance_Index

Group_Index_Or_Name

How to name a group?

E.g. (\w+)@(?<domain>\w+.com)

How to use the group name in the match extract?

E.g. (\w+)@(?<domain>\w+.com){{0,domain}}

Example Input:

Let’s assume we have the following input text. We will test various Expressions.

Customer => AAA
Email => aaa@google.com
Phone => 101-222-3333
========
Customer => BBB
Email => bbb@yahoo.com
Phone => 102-222-3333
========
Customer => CCC
Email => ccc@hotmail.com
Phone => 103-222-3333
========
Customer => DDD
Email => ddd@outlook.com
Phone => 104-222-3333

Sample Regex Expressions

(.*)

More Regular Expression Examples

Regex Examples (Using SSIS Regular Expression Parser Task)

Here is an example how Regex

(.+)_(.+)_(.+){{1,2}}

Using Regular Expressions in SSIS Regex Parser Task (Extract Groups)

Using Groups / Occurrence Index

Some tasks like SSIS Regex Parser Task (FREE) supports extracting value from specific occurrence and specific part of matched pattern using special syntax at the end of your pattern (see below).

Your Regex Pattern Here{{OccuranceIndex,GroupIndex}}

Where :
OccuranceIndex is 0 based (0=extract first occurrence)
GroupIndex is 0 based (0=extract first matching group from pattern. First group is always entire text. )

See above screenshot in previous section for example.

Tools

The best tool we’ve found to write and test Regex is Regex Hero (will require IE with Silverlight if you want to use it online, in the browser):

Another great site for Regex testing is https://regex101.com (Works in any browser unlike previous one)

and few more sites as below

http://www.regexr.com/
http://www.regexlib.com/
http://www.regular-expressions.info/

Resources

Regular Expressions cheat-sheet to hang on the wall

Regular Expressions quick reference

The post Using Regular Expressions in SSIS appeared first on ZappySys Blog.

In this blog post, you will see how to remove invalid characters from XML using SSIS. We will use the search and replace feature of the Advanced File System Task.

Remove Invalid characters from XML.

The XML file specification specifies which characters can be part of XML data and which should be avoided. If you use an invalid character, the XML Parser can throw an error indicating that it found an invalid character. Here is a very useful link that explains which characters are allowed in XML and which are not.

Using SSIS and Regex to clean invalid XML Characters

To clean invalid characters, we can use the following regular expression. This basically removes any characters outside the allowed character range.

[^\x09\x0A\x0D\x20-\xD7FF\xE000-\xFFFD\x10000-x10FFFF]

The above expression is sufficient for most cases, but it is not the best solution. If you really want the most accurate expression per the XML Specs, use the expression below. It can be SLOW for large XML files.

[^\u0009\u000a\u000d\u0020-\ud7ff\ue000-\ufffd]|([\ud800-\udbff](?![\udc00-\udfff]))|((?<![\ud800-\udbff])[\udc00-\udfff])

Remove any characters

If you don’t have a range of valid characters but you know a list of invalid characters, use the expression below (without the ^). For example, if you want to remove A, C, and E; then use the below

[ACE]

--OR-- Use hex codes

[\x41\x43\x45]

You can also use a range

Step-By-Step Example

1. Download and install SSIS PowerPack from here.
2. Create a new SSIS Project.
3. Drag the ZS Advanced File System Task from the SSIS Toolbox to the control flow designer.
4. Double-click the task to edit its properties, as shown below. Enter the following expression for a Regular Expression.
   

   [^\x09\x0A\x0D\x20-\xD7FF\xE000-\xFFFD\x10000-x10FFFF]

5. Save the package and execute. After you run the task, it will remove (replace with blank) any invalid character in your XML file.

Here is the screenshot of the Task configuration.

SSIS Advanced File System Task – Search and Replace Option -Remove invalid XML characters using Regex (Regular Expression)

The post How to remove invalid characters from XML using SSIS and Regex appeared first on ZappySys Blog.

In this post you will learn how to use FREE SSIS Regex Parser Task along with REST API Task to extract HTML content in few clicks.

Scenario

Assume that you want to search certain keywords from Bing or google and want to know how many pages found for that keyword. Url for search would be something like http://www.bing.com/search?q=regex where regex is our search word.

When page is returned view source code of that page and you will find tag like below.

<span class="sb_count" data-bm="4">21,00,000 results</span>

What we want is number 21,00,000 using Regular expression pattern search.

Step-By-Step : Extract HTML Tag value using Regex Expression

1. Download and Install SSIS PowerPack (It includes FREE SSIS Regex Parser Task )
2. Create new SSIS Package
3. Drag ZS REST API Task on Control flow designer from SSIS Toolbox
4. Double click to configure the task. Enter URL you like to fetch e.g. http://www.bing.com/search?q=regex
5. Click on Response Tab and check Save response option. Select Save to Variable. If needed create new variable.
6. Click Test (Scroll at the bottom to see html content)
7. Now drag ZS Regex Parser Task and connect with REST API Task
8. Select Variable which will hold HTML text you like to parse.
9. Enter following expression and map target to some Variable if you like to save extracted value. Below expression ends with {{0,1}} which means extract first match and 2nd group of that match (0 based Index). 2nd group of match will hold actual count of search result. If you omit {{x,y}} at the end then {{0,0}} is used.
   

   \<span\s*\w*\s*class="sb_count"\s*>\s*(?<p2>[0-9,.]*){{0,1}}

   See below screenshot
   

   SSIS Regex Parser Task – Extract HTML Tag Value using Regular Expression

10. In the above step you can select Variable as Input or use placeholder in Direct string (e.g  {{Use::varHtml}} )
11.  You can also connect ZS Logging task to show extracted value

Here is final flow.

SSIS Regular expression parsing example

The post Using SSIS Regex Parser Task for Extracting HTML Content appeared first on ZappySys Blog.

In our previous blog, we saw how to call REST in SSIS including concepts of Authentication / Error Handling. Now in this post, we will cover API Pagination. You will learn concepts and patterns about REST API Pagination (for JSON / XML / SOAP or CSV API).

We will describe methods and steps to implement REST API Pagination in SSIS PowerPack or  ODBC PowerPack API Drivers (for apps like Excel, Power BI, Informatica…) without coding. Both Products share a similar user interface with minor differences. For example purpose, we will show screenshots from SSIS PowerPack UI but concepts should remain same. You will also learn some techniques about how to implement an infinite loop in SSIS. Sometimes your REST API requires you to loop through all pages until no more data found then you can use pattern described in this article (sample package attached at the end).

In this article, we will use SSIS REST API Task and JSON Source Connector to process data from RESTful web service. If you have to process XML (or SOAP) data from REST API then simply use XML Source Connector for REST API, SOAP or File rather than JSON Source.

Debugging REST API Requests

Before we look at various aspects of REST API Pagination, we highly recommend you install FREE Web Debugging Tool Fiddler. Look at this article to understand how to debug REST API requests.

Method-1: REST API Pagination via URL Parameter

If you have paging requirement by specifying page number in URL then you can loop until no more pages found. Your REST API can indicate no more page found by any of the following ways.

Example URL:  http://myserver.com/v1/api/getcustomers/?page=1

If no more data found for specified page number then

1. server may return an empty response
2. server may return the response with specific error and response status code

JSON Source Connector comes with flexible paging options to handle this scenario. See below screenshot how to configure this.

REST API Looping/Pagination via URL Page Number Parameter (Loop until last page detected)

URL with offset and limit

Sometimes you may have URLs with offset + limit parameters. as below. In this case, you can use the same technique as the above screenshot but just specify Page Num Indicator as offset and Increment by the same as your limit parameter (e.g. 1000 in this case). At runtime, your offset parameter will increment and generate URLs like below to fetch the next page.

Example URLs: 
http://myserver.com/v1/api/getcustomers/?offset=1&limit=1000
http://myserver.com/v1/api/getcustomers/?offset=1001&limit=1000
http://myserver.com/v1/api/getcustomers/?offset=2001&limit=1000

Method-2: REST API Pagination via URL Path

Sometimes your URL may contain a page or other identifier using path format.

Example URLs: 
http://myserver.com/v1/api/getcustomers/1/
http://myserver.com/v1/api/getcustomers/2/
http://myserver.com/data/file1.json
http://myserver.com/data/file2.json

If no more data is found for the specified page number then

1. the server may return a 404 error (page not found)
2. server may return a response with a specific error and response status code

JSON Source Connector comes with flexible paging options to handle this scenario. See the below screenshot of how to configure this.

REST API Looping/Pagination via URL Path (Loop until last page detected)

URL with Skip Parameter (Offset starts at zero)

Now let’s look at another common pattern just like the offset we discussed earlier but in the Skip pattern, it starts at zero rather than offset=1. In the previous mode, we didn’t have the option to supply the initial value to start the counter but if you use Path mode then you can start the counter at some custom value. See the below screen to show how we can setup pagination with the skip method with custom increment.

Example URLs

http://mysite.com/api/getorders?skip=0&pagesize=100
http://mysite.com/api/getorders?skip=100&pagesize=100
http://mysite.com/api/getorders?skip=200&pagesize=100
http://mysite.com/api/getorders?skip=300&pagesize=100
....

REST API Pagination with Custom Skip / Increment Settings (Offset Starts from 0)

Method-3 : REST API Pagination via Response Attribute (Next Link or Cursor)

This method is not discussed in this article but click here to read more

There are mainly 3 ways you can get next link or cursor.

1. Full Link (e.g. nextPage : “https://mysite.com/api/v1/1/?filter=AAAA” )
2. Partial Link  (e.g. nextPage : “/api/v1/getcustomer/1/?filter=AAAA” )
3. Cursor Continuation Token (e.g.   nextCursor : “xxxxAAAbbbbCccc112233==” )

Next Link Method (Partial or Full URL)

Example response with next URL would be

{ 
  "value" : [....], 
  "odata.nextLink" : "http://abc.com/customers/2/" 
}

In this method you don’t specify page number in URL. If response include many records then you will get partial response along with indicator of next link.

For more information about this REST API Pagination please check below links. It explains in depth how to paginate REST API response with next page link found in response.

OData Paging using SSIS – REST API Paging Example

Read Twitter data in SSIS using REST API Task and JSON Source – OAuth2 Protocol

SSIS API Pagination – Next Link from Response Attribute

Next Link – Detect the last page based on empty rows (Regular Expression)

Now let’s look at a little bit different scenario where the Next Link is never empty even on the last page. This can cause an infinite loop if we configure it like the last section. So in such a case (see below example) when the last page still has the next link URL but the data row array is empty then use settings like the screenshot below.

First Page Example

{ 
  "value" : [....], 
  "odata.nextLink" : "http://abc.com/customers/2/" 
}

Last Page Example

{ 
  "value" : [], 
  "odata.nextLink" : "http://abc.com/customers/NNNN/" 
}

Example: Last Page

Basically, configure the following options

• Set Stop indicator attribute to the same node which contains data rows (without array brackets). E.g. $.items
• Set Stop indicator value / Regular Expression to something like

  []

    assuming the last page has the same way. Check Fiddler Trace for exact text on the last page.

Last Page with non empty next link and empty data array (Detect Last Page based on zero row count)

Cursor Method

Many API returns cursor rather than link. Cursor method usually used to support a large number of records. Cursor is opened at server side with initial query information you submit and if cursor has more records you may get next cursor indicator in your response. In cursor method you have to supply Suffix for Next URL so after First URL your next URL will contain extra attribute which will include pagination information.

Suffix for next URL

Suffix for next URL can include special placeholders. There are two placeholders allowed. If you wish to append Next URL value in raw text then use <%nextlink%> but if your API needs value in encoded format e.g. A%20B rather than A+B then you have to use <%nextlink_encoded%>

Example URLs for Cursor:

Below is few sample URLs in the log when pagination occurs via Cursor method. As you see the first URL has Cursor=* means its first call, after first request server may send next cursor in the response body (see example)

{ 
  data: [....], 
  nextCursor: "Aooo/MxxxxxxxxxxxtdA==" 
}

First URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=*
Second URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=Aooo/MxxxxxxxxxxxtdA==
Third URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=Booo/MxxxxxxxxxxxtdA==
…..
…..

Method-4: Pass Page Number in Body Attribute

Some API allows you to pass your PageSize and Page number inside Request Body (i.e. POST data) in that case you may configure your options like below (We used Google AdWords Api as an example)

Configure REST API Pagination – using Page Number via POST Body Method

REST API Pagination – Passing PageNumber inside POST Body (Example: Google AdWords SOAP API)

Method-5: REST API Pagination via multiple URL parameters (e.g. Start and End)

In the previous section, you saw how to pass page numbers via URL (Single Parameter: http://abc.com/?page=1). However, sometimes you have a pagination requirement where you have to specify multiple parameters that need increment. See below example URL.

http://abc.com/?startRow=1&endRow=100
http://abc.com/?startRow=101&endRow=200
http://abc.com/?startRow=201&endRow=300
...

Pagination via Multiple URL Parameters

Method-6: REST API Pagination via Header (Next Link in Header)

Another popular pagination method is paginate via Header. In this method, each API response includes special Header which indicates next link for your response. In header method there are two approaches. [1] Custom Method [2] RFC 5988 Method. Lets look at both.

Next Link in Custom Header

In this method you have to specify which response header contains next URL. When no more NEXT url found in response reader stops.

Custom Header Pagination Method

Next Link in Header as per RFC 5988

In this method, HTTP response contains a special header (Link) which contains next link for our response. If no link found to read more data then reader stops. This method is described well here in RFC 5988. Check this GitHub API Example to understand more on this pagination method.

Link Header Pagination Method (RFC 5988)

Method-7: REST API Pagination using Cursor / Continuation Token from Response Header

There will be a time when your API sends some sort of Continuation Token in Response Header and you have to use that token to call next request. Some API needs this token passed as query string in Next URL and in some API you have to pass as request header. We will cover both cases in the following section.

Your API might call these tokens as “Cursor” too so we will use these both terms in the following section (They both mean same thing).

There are few ways these APIs pass Cursor to the next request

1. Pass Cursor via Request Header with same header name
2. Pass Cursor via Request Header with different header name (introduced in v4.2.3)
3. Pass multiple Response Headers via Request Headers (Azure DocumentDB style usecase)
4. Pass Cursor via URL Query String Parameter (introduced in v3.0 or later)

Lets see all approaches.

Pass Token via Request Header (Same header name)

Consider the API pagination use case in which you get some sort of continuation token in the response header and you take it and pass in the next request via request header (same header name).

Example:

First Request / Response:

------- Request ------
GET https://abc.com/api/getdata

------- Response ------
x-token: Nxdfabcd11111

{ some response data }

Second Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-token: Nxdfabcd11111

------- Response ------
x-token: Nxdfabcd22222

{ some response data }

……

……

Last Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-token: Nxdfabcd22222

------- Response ------

{ some response data }

For above pagination pattern  that just enter the response header name in Page Num Indicator field as below

Pagination using response header continuation token – Pass as Request Header

All possible formats for Page Num Indicator is below

x-page-token
x-token1, x-token2
RequestUrlQueryName=ResponseHeaderName
header::RequestHeaderName=ResponseHeaderName
RequestUrlQueryName=ResponseHeaderName(regular-expression)

Pass Token via Request Header (Different header name)

Consider the API pagination use case just like previous one but only different is request header name is not same as response header name. For example rather than x-token in request we need to call it x-next-token.

Example:

First Request / Response:

------- Request ------
GET https://abc.com/api/getdata

------- Response ------
x-token: Nxdfabcd11111

{ some response data }

Second Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-next-token: Nxdfabcd11111

------- Response ------
x-token: Nxdfabcd22222

{ some response data }

……

……

Last Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-next-token: Nxdfabcd22222

------- Response ------

{ some response data }

For above pagination pattern  that just enter the response header name in Page Num Indicator field as below

Pagination using response header continuation token – Pass as Request Header (Different name)

Pass multiple Response Headers via Request Headers (Azure DocumentDB style usecase)

Some API like Azure Cosmos DB uses Continuation Token via Header approach. In that case you can enter single header name or multiple headers like below (For multiple headers use comma)

REST API Pagination using Continuation Token via Header (Azure CosmosDB / DocumentDB API Example )

You can use use tools like Fiddler to see paginated requests as below.

First Request

View Paginated Requests in Fiddler (Azure CosmosDB API Example)

Second Request

Pass Next Page Token via Header (View Paginated Requests in Fiddler – Azure CosmosDB API Example)

Pass Token via URL Parameter (Query String)

In the new version we added a feature where you can actually read Continuation Token (i.e. Cursor) from Response Header and then pass to next request in the URL rather than Request Header. Here is the example API which uses such method.

First Request

--------- Request ---------
GET
http://test.com/api/getdata

--------- Response ---------
X-RESPONSE-TOKEN : Pg2xxxxxxxxxxxxxxxxx

{ data : [ 1,2,3.... 10 ] }

Second Request 

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg2xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : Pg3xxxxxxxxxxxxxxxxx

{ data : [ 11,12,13.... 20 ] }

Here is how to configure above pagination pattern

Pass Response Header value to URL (Cursor)

Pass Token via URL Parameter (Query String) – with custom Regular expression (Usecase: Salesforce BULK query API)

Now consider previous scenario but with slightly different requirement. By default previous pagination pattern stops when specified response header is missing or response header value is blank. However is some cases header value contains custom string to indicate last page. For example x-token: NULL. In such case we need to use Page Num Indicator like below (must use v4.2.3 or higher in SSIS PowerPack / 1.5.2 or higher in ODBC PowerPack)

nextUrlParameterName=responseHeadername(Some_Regular_Expression)

For example to stop pagination in below case you have to enter Page Num Indicator as below. It says stop pagination when  “NULL” word is detected

cursor=X-RESPONSE-TOKEN(^((?!NULL\b).)*$)

First Request

--------- Request ---------
GET
http://test.com/api/getdata

--------- Response ---------
X-RESPONSE-TOKEN : Pg2xxxxxxxxxxxxxxxxx

{ data : [ 1,2,3.... 10 ] }

Second Request 

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg2xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : Pg3xxxxxxxxxxxxxxxxx

{ data : [ 11,12,13.... 20 ] }

……

……

Last Request 

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg3xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : NULL

{ data : [ 21,22,23.... 30 ] }

Here is how to configure above pagination pattern

Pass Response Header value to URL (Cursor)

Method-8: Pass Pagination Token in Request Body

In previous section (Method-4: Pass Page Number in Body Attribute ) we saw how to pass simple numeric page counter in Body. This is fine for some API but if your API has base64 style text token then counter based method doesn’t work. Also if you have requirement to hide page attribute node in first request and send page counter node in second request body (Changing Request Body Pattern for First vs Next)  then check next two sections.

Next two sections covers two API Pagination patterns to fetch 100 rows (1000 in each page) . Both requires to send token in Body but second section is more complex need (Different Body Attributes in first vs next requests )

Passing Page Token in Body (Blank Token in first request allowed)

Lets look at simple pattern where first request has blank token in Body but all other requests until we find last page we keep sending non-empty token. This approach works as long as API Provider doesn’t mind having Blank Token in first request. If you get error then check next section to handle more complex way.

Here is the example of API which needs token in body

First API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"" }

--------- Response ---------
{recordsFound: 5000, data: ["Acct1", "Acct2"..., "Acct1000"], nextToken:"AxBxCx112233001==" }

Second API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"AxBxCx112233001==" }

--------- Response ---------
{data: ["Acct1001", "Acct1002"..., "Acct2000"], nextToken:"AxBxCx112233002==" }

….. many more calls …

…. many more calls ….

Last API call (no more nextToken in response so STOP pagination)

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"AxBxCx112233009==" }

--------- Response ---------
{data: ["Acct9001", "Acct9002"..., "Acct10000"] }

So if you have such pattern then you can set few properties

For such issue you can use Changing few properties like below. Use Property Grid if you cant find on UI

• Set EnablePageTokenForBody  =

  True

• Set PagePlaceholders =

  header=|;filter=|;

• Set HttpRequestData = 

  { searchFor: "Account", continueToken:"[$pagetoken$]" }

Setting Body with [$pagetoken$] to implement dynamic pagination

Setting Page Placeholders in Advanced Pagination tab.

Passing Page Token in Body (Blank Token in first request)

Now let’s look at more complex pattern. Here our first request body is different than other requests.

First API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account" }

--------- Response ---------
{recordsFound: 5000, data: ["Acct1", "Acct2"..., "Acct1000"], nextToken:"AxBxCx112233001==" }

Second API call

--------- Request ---------
POST http://test.com/api/getdataNext

{ continueToken:"AxBxCx112233001==" }

--------- Response ---------
{data: ["Acct1001", "Acct1002"..., "Acct2000"], nextToken:"AxBxCx112233002==" }

….. many more calls …

…. many more calls ….

Last API call (no more nextToken)

--------- Request ---------
POST http://test.com/api/getdataNext

{ continueToken:"AxBxCx112233009==" }

--------- Response ---------
{data: ["Acct9001", "Acct9002"..., "Acct10000"] }

So notice few things

• First Request Body / URL is different than second / third and Nth API call
• For first request we do not have token attribute but after first request we do have token attribute

We have this article and this article which describes similar pattern.

Now lets look at how to solve both problems

Passing PageToken or Counter in Next API Call Body

For such issue you can use Changing few properties like below. Use Property Grid if you cant find on UI

• Set EnablePageTokenForBody  =

  True

• Set PagePlaceholders =

  header=|;filter=|;

• Set HttpRequestData = 

  [$tag$]

• Set HasDifferentNextPageInfo = 

  True

• Set FirstPageBodyPart = 

  { searchFor: "Account"}

• Set NextPageBodyPart = 

  { token:"[$pagetoken$]" }

So notice how we defined Body in two different properies as template. And in HttpRequestData we used

[$tag$]

Output Page Number as column

Sometimes you have use case where you want to Start Page Number from the last page loaded successfully. By default, you will see one additional column __FileName in the output. This column contains PageNumber  from URL for each row. You can save this into database so when next time you run your SSIS you can start at MAX(PageNumber) + 1.

Outputting PageNumber option is not visible by default so Use Property Grid and look for OutputFileName property (Default=true)

Last Page Detection

Every API may have different way of telling you that you reached last page or no more data found so you should stop looping. SSIS PowerPack comes with many options so you can implement desired logic for last page detection.

Following approaches supported by last page detection. Below options are not applicable if you are using Method#3 (Response contains Next Link / Cursor)

1. Detect the last page by row count
2. Detect the last page by response status code
3. Detect the last page by error message (substring search)
4. Detect the last page by response size
5. Detect the last page based on response value or regular expression pattern

Now lets look at each setting. You should choose correct one based on your API behavior

Detect the last page by row count

If your API returns blank document or blank Array (if its JSON) then you can use this method to detect the last page. So when no content found in Response document it will stop pagination.

See below example of JSON API .. It shows sample first page and Sample last page (Assuming you entered Filter as $.Records[*] to extract data from Records array.

First Page

{ 
  Records : [ {...}, {...}, {...} ......... ]
}

Last Page

{ 
  Records : [  ]
}

SSIS API Pagination – Detect the last page by row count (Blank document or array)

Detect the last page by response status code

Sometimes you get error with status code when the last page is reached (Most common is 404). So rather than failure, you can treat that status code as last page indicator using this option (see below).

SSIS API Pagination – Detect last page by response status code

Detect the last page by the error message

If you cant use above two methods and want to detect the last page based on a certain error string in the response then use below approach.

SSIS API Pagination – Detect the last page by error/response string search

Detect the last page by response size

Sometimes you can detect the last page by response size. For example, your response contains “–” only if no more data is found then you cannot use any of the above approaches and you have to detect based on response size (e.g. less than 3 bytes). See below

SSIS API Pagination – Detect last page by response size

Detect the last page based on the response attribute value

There will be a time when you want to detect the last page based on some indicator value from your response. In that case, you can use the following settings to detect the last page.

• Select Paging mode =Response Attribute Mode
• Select Next Link Attribute (Example: $.next_page )
• Select Stop Indicator Attribute (Example: $.hasMorePages )
• Enter a value for Stop Indicator (Example: false )

If you don’t want to use Hardcoded value for stop indicator and want to use expression then see next section.

Detect the last page based on regular expression pattern

There will be a time when you want to detect the last page based on some indicator value from your response. In that case, you can use the following settings to detect the last page.

• Select Paging mode =Response Attribute Mode
• Select Next Link Attribute (Example: $.next_page )
• Select Stop Indicator Attribute (Example: $.count )
• Enter regex pattern (regular expression) for Stop Indicator (Example: regex=^\d{1,3}$ ). Stop when count attribute is less than 1000 in other word less than 4 digits.

Here is an example from Zendesk API (Stop pagination when count attribute from the response is less than 1000)

Pagination for Zendesk Incremental API – Next Link and Last Page detection using Stop Indicator Regular Expression

Handling API Rate Limit - Retry on too many API requests

Many API restricts you from calling too many API calls too soon (just to save their servers from being overwhelmed) - Example. Let's think this way if you fetching a million rows via paginated response and the very last page fails then you won't be happy because now you have to fetch everything again. To solve this ZappySys provides retry options on HTTP / OAuth and REST API connection managers.

Generally, when you exceed the rate limit, the server sends you error code 429 but it can be something else too or multiple codes are returned. In the below example, we use 503. Check your API documentation to know the exact code you should retry.

To set the retry options go to connection settings (e.g. HTTP / OAuth / REST connection). The below example is for OAuth connection.

To retry on multiple codes use Pipe as a delimiter. Example: 429|503

Retry Settings for API Rate Limit Exceeded Error (Status code 429 or other)

Performance Tips

You can improve performance of your API calls by doing a few changes (again not every API may support this)

• Use the maximum page size possible (e.g. if you doing pagination you can set max possible page size – refer to your API documentation whats the default size and what is the max page size you can supply)
  e.g.  /myapi?pagesize=100  (parameter name can be something else like limit=100)

SSIS Infinite Loop

So now you know how to paginate automatically in JSON Source Connector but what if you want to use REST API Task for pagination? Some custom scenarios like web scraping require you to download all HTML pages until no more results found? In that case, you most likely need infinite looping pattern (Loop until the last page detected). To loop through pages until the last page is reached you can perform the following logic.

The last page can be detected based on any of the following ways
1. Check Response StatusCode
2. Check Error Message
3. Check Response Size (If less than X bytes then consider as the last page)

REST API Task support #1 and #2 scenarios. If you want to check bytes then use the expression in Expression Task or Script Task
(e.g. @exitloop = LEN(@responseData) <= 3 )

URL for Rest API Task (On Request Settings Tab)
When you configure REST API Task enter URL as below so its dynamic. You can use PageCounter variable anywhere in the URL.

//zappysys.com/downloads/files/test/cust-{{User::pagecounter}}.json

SSIS REST API Looping – Pass Page number in URL, SSIS Infinite Loop

Configure For Loop Task for Infinite Looping (Loop until last page found)

Download Sample SSIS Package

SSIS-Loop-Rest-API-2012.dtsx (Zip file)

Video Tutorial – Reading data from API in SSIS

In case you missed our previous article about calling REST API is SSIS then check below video.

Conclusion

It can be time consuming and steep learning curve to write C#/JAVA code to handle your rest api integration but using some Tasks and Connectors listed below can help you to solve big headache and provide clean drag and drop REST API integration in SSIS for virtually any REST API Source (e.g. Twitter, Facebook, Salesforce, office 365). Download SSIS PowerPack and try it out by yourself.

SSIS Components for REST API/JSON/XML/SOAP

	JSON Source Connector (File, REST API)
	Export JSON File Task
	JSON Parser Task
	XML Source Connector (File, SOAP, REST API)

The post How to do REST API Pagination in SSIS / ODBC Drivers appeared first on ZappySys Blog.

Zendesk is one of the most popular customer service platforms available today. Zendesk offers a REST API to interact with its cloud-based service programmatically (to manage or read data). The problem is that not everyone is a programmer, and not everyone has time to learn how to write C# or JAVA code to access a REST API.

In this article, you will learn how to get Zendesk data using the SSIS JSON Source (REST API Connector). This connector allows you to read Zendesk data in SSIS using REST API Calls without any scripting/programming. Techniques mentioned in this article can also be used to consume data from other services (e.g., read data from Twitter). In this article, you will see step-by-step instructions to read Zendesk ticket information and load it into on-premises SQL Server.

Also, check our other article on how to call the Zendesk REST API to create a new ticket.

Different methods for Zendesk REST API Access

Zendesk allows you to access its data in three different ways.

1. REST API access using OAuth Access Token
2. REST API access using API Token
3. REST API access using Basic Authentication (Use userid (e.g., email) and Password)

Each method has pros and cons. The simplest method is the third, which is easy to use but requires passing your account password with each call. The second method is also easy to use but requires an additional step to create an access token. The first method is the most secure, but it has a few extra steps compared to the other methods.

Prerequisite

Before you can finish the tasks explained in this article, you have to finish the following steps. This article assumes you have basic knowledge of SSIS (SQL Server Integration Services)

1. Make sure you have the SSIS designer installed. Sometimes it’s referred to as BIDS or SSDT (Get from here)
2. Download SSIS PowerPack (FREE Trial – 30 days).
3. Optional – Another very useful Free tool is Fiddler. You can use it to see raw request/response (Check this article)

Method-1: Zendesk API Access using OAuth in SSIS

To read about the OAuth Method, click here

Method-2: Zendesk API Access using Token in SSIS

Now let’s look at the second method for accessing Zendesk data using SSIS.

Create Token in Zendesk Portal for REST API Access

1. Log in to your account https://{{your-subdomain}}.zendesk.com/login and then click on the settings icon
2. On the next page, click on the API link
3. On the API page, scroll to the API Access Tokens section. Make sure the Token Access method is checked.
4. Click on the ” add new token hyperlink. Once prompted, give a name to the Token token (this is just a label)
5. Once the above steps are done, a token should be created and listed under your API Access page. The token will be used as your Password to access the REST API (will be shown in the next section). For more information, check this link >>>  How to generate a token for ZenDesk API Call.
   

   Create a new Zendesk REST API Access Token

Configure SSIS HTTP Connection

Perform the following steps to extract data from Zendesk and load it into SQL Server. The steps below assume that you are using the API Token method (#2 from the supported methods to call the Zendesk API). This is better than using Userid and Password.

1. Open a new SSIS Project.
2. First of all, drag and drop the Data Flow Task from the SSIS Toolbox and double-click it to edit.
   

   Drag and drop Data flow.w

3. From the SSIS toolbox, drag and drop the ZS JSON Source onto the data flow designer surface, then double-click it to edit.
   

   SSIS JSON Source – Drag and Drop
4. From the AccessMode dropdown [Url From Connection]
5. Assume you want to extract all tickets from Zendesk, then enter the following URL (in the screenshot, we appended?page=1, which is optional)
   

   https://YOUR_SUB_DOMAIN.zendesk.com/api/v2/tickets.json

6. Check the Use Credentials option and select New ZS-HTTP connection. When prompted
   For Credential Type: select Basic Authentication  (Use of UserID and Token ).
   For Web URL: Enter any URL (This will be ignored anyway because the URL entered on the JSON Source UI will be used)
   For user name : enter YOUR_EMAIL/token  (e.g. bob@mycompany.com/token)
   For Password: enteToken token obtained from the Admin console > API section. For more information, check this link >>>  How to generate a token for ZenDesk API Call.
   
   NOTE: Appending /token command in Userid will allow us to pass the token rather than the Password of that account. This approach is more secure because you can disable the Token at any time without deactivating the user account. If you want to access data using your userid and Password rather than a token, then do not use /token after your email. By default, uToken Token and the use of Password are not allowed unless you enable these methods from the Admin console (See above section)
   

   Configure SSIS HTTP Connection – Use Zendesk Token to Call REST API

Adding Retry Settings for HTTP or OAuth connection

Zendesk enforces an API rate limit, which means you cannot issue too many API calls within a given time frame. Check this document for exact information.

ZappySys provides API retry settings on both the OAuth Connection and the HTTP Connection. For Zendesk, you can retry on a specific status code 429

1. Open connection manager UI
2. Go to the Retry-Settings page
3. Select Retry web error on matching response status code.
4. Enter 429 for the status code
5. Check multiple wait times
6. In the wait time, enter 5000 (wait 5 seconds or multiple of 5 seconds)
7. Set the max retry count to 6. See the example (It’s not for Zendesk, but will give you an idea)
   

   OAuth connection – Retry Options

Configure Zendesk API Pagination and Filter for SSIS JSON Source

1. Now click [Select Filter], select the tickets node (see the array icon), and click OK. It will generate the following Filter Expression to extract all records found under the tickets array node (e.g., tickets : [ {…} , {…} … ] )
   

   $.tickets[*]

   

   Select Filter – Extract Data from Zendesk API Response (Read Ticket)

2. N, let’sts configure pagination. By default, Zendesk API doesn’t return all records, so you have to specify pagination as below. Click on the Pagination Tab. Enter $.next_page for Next URL attribute as below (you can also Browse Path)
   

   SSIS JSON Source Zendesk Pagination

3. Now time to click Preview. If you followed everything correctly, then when you click the Preview Button, you should see some data in the preview (assuming you have at least one ticket).
   

   SSIS JSON Source – Data Preview – Zendesk REST API Call to extract tickets

4. Now click the Columns tab to review data types (you can change the default length and check the Lock option on the right side). This change is needed because the datatype guess happens based on the design-time setting (only 300 rows scanned by default). At runtime, if you get a truncation-related error, then copy the error message into Notepad to get the exact steps and column name that caused the error. If you need a length greater than 4000 chars, use the DT_NTEXT data type instead (change the datatype and click the column name cell on the right to save the change).
5. Click OK to save UI

How to extract Zendesk data using Incremental API

There will be a time when you need to extract data incrementally (i.e., Extract only the changed data since the last load). Zendesk API supports many Incremental API endpoints. If you are using this endpoint, you have to make sure of a few things.

• For the OAuth connection method, make sure the

  read

    scope is included (

  tickets:read

    won’t work, so just use

  read

   )
• Your URL must include /incremental path before the API type, and you must include start_time for the extract in Unix Epoch Format.  The following example will pull records modified after
  

  https://zappysys.zendesk.com/api/v2/incremental/tickets.json?start_time=1514764800

• On the pagination tab, you have to configure the following settings
  • Set Next Link as

    $.next_link

  • Set the Stop Indicator attribute as

    $.count

  • Set the Stop Indicator Value as

    regex=^\d{1,3}$

      Notice that using regex= prefix treats your end indicator value as a pattern check.—OR—
  • Set the Stop Indicator attribute as

    $.end_of_stream

       and the Stop indicator value as

    true

    

    Pagination for Zendesk Incremental API – Next Link and Last Page detection using Stop Indicator Regular Expression

Here is a simple SQL query to convert date/time to Unix time

select DATEDIFF(SECOND,'1970-01-01', GETUTCDATE() )

select DATEDIFF(SECOND,'1970-01-01', LastExtractDate ) as LastExtractDateAsUnix From dbo.ExtractStatus Where ExtractName='ZendeskExtract'

For C# code in the SSIS Script Task, you can use something like the following:

Int32 unixTimestamp = (Int32)(DateTime.UtcNow.Subtract(new DateTime(1970, 1, 1))).TotalSeconds;

Assuming you extracted Date/Time from the database and saved it to some SSIS Variable called User::varLastExtract. Now we like to convert that to Unix format using C#.

We assume you have created the following variables in the SSIS Package. Also, you have set varLastExtract to a date/time value you want to convert.

User::varLastExtractTimeUnix  (Int32)
User::varLastExtractTime (DateTime)

Use the steps below. to convert date time to Unix format in C#

1. Drag and drop Script Task in SSIS designer
2. Add both variable names in Read/Write Variable List (e.g., User::varLastExtractTimeUnix, User::varLastTimeExtract ).
3. Edit Script
4. Enter something like below
   

   DateTime dtUtc= Convert.ToDateTime( Dts.Variables["User::varLastExtractTime"].Value ).ToUniversalTime();
   Dts.Variables["User::varLastExtractTimeUnix"].Value = (Int32)(dtUtc.Subtract(new DateTime(1970, 1, 1))).TotalSeconds;

Now you can use this variable in your JSON Source URL to make things dynamic.

https://zappysys.zendesk.com/api/v2/incremental/tickets.json?start_time={{User::varLastExtractTimeUnix}}

Handling duplicate data during incremental load (DISTINCT)

As per this Zendesk document, they may include the last row of the previous page on the next page if the update timestamp. It is possible to have duplicate rows in your staging table. You can easily avoid this by using the DISTINCT keyword or another SQL query to remove duplicates. Make sure not to include P_next_page and P_count, as they can differ when duplicate rows are detected at page boundaries.

SELECT DISTINCT 
--[P_next_page] --- Do NOT include this... 
[P_previous_page]
--,[P_count] --- Do NOT include this... 
,[url]
,[id]
,[external_id]
,[via.channel]
,[via.source.rel]

,[assignee_id]
......
......
......
,[allow_attachments]
FROM [dbo]._stgZendeskTickets

Loading data from Zendesk to SQL Server using SSIS

1. Now, drag a new OLE DB Destination into the Data Flow Designer. Connect the JSON source with the destination. Now double-click Destination. At the destination, select connection manager (or create one), then click [New] next to the Table name dropdown. You will see default metadata for the table based on upstream columns. Notice how P_previous_link is 40 chars rather than 140. This can be edited in the source metadata under the Columns tab, as described in the previous step, so that Table datatypes are set correctly. Once you change the table name and datatypes, click OK to create the new table. Click on the Mapping Tab to map source columns to target columns. Once that’s done, click OK to save.
   

   Load Zendesk Data into SQL Server Using SSIS JSON Connector (REST API)

2. Now execute the package. If you have multiple pages in response, you will notice several requests in the log (e.g., &page=1 … &page=2 …).
   
   NOTE: In below screenshot we used per_page=2 for example purpose only. In reality, make it bigger, e.g., per_page=100 or per_page=200, so you don’t hit the server too often. If you’re not sure, then exclude the per_page parameter from the URL to use the default setting.
   

   Execute SSIS Package – Extract data from Zendesk and load into SQL Server

Method-3: Zendesk API Access using Username and Password in SSIS (Basic Authentication)

This method is the same as the second method outlined in the previous section. Except for a few changes. When you create an HTTP Connection, use your Username (without appenTokentoken ). For the Password, use your account password (the same one you use to log in to the Zendesk portal)

Zendesk API Rate Limit (Throttling)

Zendesk may limit how many calls you can make per second, so make sure to slow down if you get errors about api rate limit. You can slow down by entering Wait time on the Throttling tab of JSON Source (100 means wait 100 ms after each Request … only useful if pagination is enabled)

Making things Dynamic (e.g., URL)

You can use SSIS variables in your URL to make it dynamic. See below

https://mycompany.zendesk.com/api/v2/items/{{User::TicketNumber}}.json

Dealing with common errors

For detailed instructions to fix common metadata-related errors, read this article.

SSIS PowerPack – Column Metadata Editor (i.e., Change DataType, Column Length)

Deploy to production

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Conclusion

In this post, you saw how easy it is to read data from Zendesk using the SSIS JSON Source (REST API Connector). Click here to download and try it yourself.

Keywords

Loading Zendesk data into SQL Server
How to read Zendesk tickets in SSIS
Calling Zendesk REST API in SSIS
Fetch Zendesk data using a REST API call in SSIS
Extract Zendesk data using a REST API call in SSIS
Reading data from Zendesk using rest api call
Pulling Zendesk tickets using rest api call

The post How to read Zendesk data in SSIS with REST API appeared first on ZappySys Blog.

JSONPath expression is an expression language to filter JSON Data. Its a similar concept to XPath expression in XML but has limited features compared to XPath. If you are extracting data from REST API Services using SSIS JSON / XML Source or ODBC JSON / XML Driver then you will quickly realize that it’s very important to extract nested data by navigating to a certain level.

Sample data for JSONPath expression

All examples of JSONPath expression in this post will use the following sample JSON for reference.

{
  "store": {
    "books": [
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95,
        "sections": ["s1", "s2", "s3"],
        "publicationDay": "2023-05-10"
      },
      {
        "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99,
        "sections": ["s4", "s1", "s3"],
        "publicationDay": "2022-08-10"
      },
      {
        "category": "fiction",
        "author": "Herman Melville",
        "title": "Moby Dick",
        "isbn": "0-553-21311-3",
        "price": 8.99,
        "sections": ["s1"],
        "publicationDay": "1851-10-11"
      },
      {
        "category": "fiction",
        "author": "J. R. R. Tolkien",
        "title": "The Lord of the Rings",
        "isbn": "0-395-19395-8",
        "price": 22.99,
        "sections": ["s8", "s9"],
        "publicationDay": "1954-07-12"
      }
    ],
    "time": {
      "starttime": "8:00",
      "endtime": "18:00"
    }
  }
}

JSONPath Examples

//get all books for store
$.store.books[*]
                     
//get all sections from all books
$.store.books[*].sections[*]     

//get all authors of all books for store
$.store.books[*].author  
             
//get 3rd book record 
$.store.books[2]    

//get first 2 books from the top
$.store.books[:2]   

//get last 2 books
$.store.books[-2:]
  
//get all books where author attribute equals to 'sam'
$.store.books[?(@author=='Nigel Rees')] 

//get all documents / sub documents (any level) where author attribute equals to 'sam'
$..[?(@author=='Nigel Rees')] 

//get books where price is less than 10
$.store.books[?(@.price<10)] 

//filter all books with tag
$.store.books[?(@.tag)]

//find all books which contains section s1 or s2 (Use of Logical operator OR ( || )
$.store.books[?(@.sections[*]=='s1' || @.sections[*]=='s2' )]

//find all books where first section is s1 and second section is s2 (Logical operator OR ( && )
$.store.books[?(@.sections[0]=='s1' && @.sections[1]=='s2' )]

// Find all books where the publication day is before 2000-01-01
$.store.books[?(@.publicationDay < '2000-01-01')]

JSON Path – Using Regular Expressions (Regex)

The new version of SSIS PowerPack and ODBC PowerPack Supports Regular expressions in the path. Learn more about Regular expressions check this article

//Using Regular Expression: get all books where author attribute contains 'Nigel' or 'Waugh'
//Learn more about regular expressions here : https://zappysys.com/blog/using-regular-expressions-in-ssis/
$.store.books[?(@author=~ /Nigel|Waugh/  )] 

//Using Regular Expression: get all books where category is 'reference' OR author attribute contains 'Nigel' or 'Waugh'
$.store.books[?(@category=='reference' || @author=~ /Nigel|Waugh/ )] 

//Using Regular Expression: get all books where category is 'reference' AND author attribute contains 'Nigel' or 'Waugh'
$.store.books[?(@category=='reference' && @author=~ /Nigel|Waugh/ )] 

//Using Regular Expression: get all books where author name either start with 'Nigel' (see ^ prefix) or ends with 'Waugh' (see $ suffix)
//Learn more about regular expressions here : https://zappysys.com/blog/using-regular-expressions-in-ssis/
$.store.books[?(@author=~ /^Nigel|Waugh$/  )] 

//Using Regular Expression: get all books where author name starts with 'Nigel'
//Learn more about regular expressions here : https://zappysys.com/blog/using-regular-expressions-in-ssis/
$.store.books[?(@author=~ /^Nigel/  )] 

//Using Regular Expression: get all books where author name ends with 'Waugh'
//Learn more about regular expressions here : https://zappysys.com/blog/using-regular-expressions-in-ssis/
$.store.books[?(@author=~ /Waugh$/  )]

JSON Path – Nested Scan for any level (double dots)

Now lets look at example of below JSON. Assume that you like to extract information from all nested “children”  nodes (basically extract id and name of every child from any level).

You can use double dots without specifying parent heirarchy like below to scan recusrsivly.

$..children[*]

{
  "children": [
    {
      "id": 1,
      "name": "c1",
      "children": [
        {
          "id": 2,
          "name": "c2",
          "children": [
            {
              "id": 3,
              "name": "c3",
              "children": [
                { "id": 4, "name": "c4" },
                { "id": 5, "name": "c5" }
              ]
            }
          ]
        }
      ]
    }
  ]
}

For this you can write following JSON Path

$..children[*]

OR (below to find all children where name is c1 or c2 or c3)

$..children[?(@name=~ /c1|c2|c3/ )]

Here is an example of a JSON Driver in ODBC. The same expression can be used in SSIS too.

ODBC JSON Driver – Using Filter with Regular Expressions / JSONPath

JSONPath for Root Objects in Array

Now lets look at another pattern of JSON where your documents are on the first level rather than under some array property. Root filter only works if you have a version released after Dec-29-2021. Previous version didn’t support filtering on Root Array.

Example JSON

[
 { id:1 , color:'red'},
 { id:2 , color:'red'},
 { id:3 , color:'blue'},
 { id:4 , color:'green'}
]

Example JSON Path Filters

$.[0]  //Extract first document
$..[?(@color=='red')]   //Recursive scan (any level) - Extract all documents where color=red use 2 dots after $ for filter

NOTE: using two dots after $ (e.g. $..[zzzz]) invokes nested scan so any level array will return documents, this may not be the acceptable case sometimes but if you want to filter the root array then this is the only way for now in ZappySys.

If you must not allow recursive scan then you can use the following two properties

EnableRawOutputModeSingleRow = true;
RawOutputDataRowTemplate = “{data: [$1] }”  –where [$1] is replaced at runtime with Document Text you like to parse (can be static or URL response)

OR

Use functions in RawOutputDataRowTemplate if needed

RawOutputDataRowTemplate = “{data: <<[$1],FUN_BASE64DEC>>}”

The above will wrap around the actual JSON / XML you need to parse so now after this setting you can use JSONPath = $.data[?(@id==1)] to avoid two dots to prevent a recursive scan

The post JSONPath Expression Examples – JSON Cheatsheet (SSIS / ODBC) appeared first on ZappySys Blog.