In this post, you will learn how to read / write ManageEngine ServiceDesk data in SSIS (Drag and drop approach without any coding).  We will use SSIS JSON/ REST API Source to extract data from Servicedesk API and use SSIS Web API Destination to write data to Zoho.

We will look at step-by-step instructions to read ServiceDesk  data (e.g. Request, Problems, Contacts) and load into SQL Server Database Table.

Authentication ServiceDesk REST API in SSIS (Use OAuth 2.0)

Very first step to call ServiceDesk API is you need to register OAuth App in Zoho Portal. Don’t get confused if you see Zoho Portal for ServiceDesk a Product from ManageEngine is in fact Zoho subsidiary (in other words Zoho is the parent company). ServiceDesk has On-Premises version too but this article is only about Cloud version.

Register Zoho OAuth App and get ClientID/ Secret (*** MUST READ ***)

Check this article to register your OAuth App and obtain Client ID  /Client Secret

Create ServiceDesk API Connection in SSIS

Once you create Zoho OAuth App now its time to call some Zoho API. Let’s create the OAuth Connection first. In the next section we will see how to load data in SQL Server.

1. Right click on Connection Panel and choose Choose “New Connection..”. Select ZS-OAUTH type
   

   Create new SSIS OAuth API Connection Manager

2. Configure your OAuth connection General Tab as below
3. OAuth Provider: Custom
4. OAuth Version: OAuth2
5. Client Id , Client Secret
6. Enter One Scope Per Line (For more info on scope read here) . For write you can change READ to WRITE or use ALL as per the help link.
   

   SDPOnDemand.requests.READ

7. Auth URL : https://accounts.zoho.com/oauth/v2/auth
8. Token URL:  https://accounts.zoho.com/oauth/v2/token
9. Configure Advanced tab as below
10. Enter Callback URL : https://zappysys.com/oauth
11. Enter Extra Attribute for /Auth : access_type=offline&prompt=consent
    This is needed to obtain refresh_token (if you ommit this then only access_token returned which cannot be renewed once expired after 1hr)

    

12. Now go back to general tab and Click Generate Token. Close the Popup and use Full browser instead if you see prompt.
13. Click OK to save the connection UI

Read data from ServiceDesk API in SSIS JSON Source

In this section we will learn how to read data from ServiceDesk using SSIS JSON Source. We will use OAuth connection created in earlier section.

Now lets look at step by step example of reading ManageEngine ServiceDesk data using SSIS (e.g. Requests).

ManageEngine provides JSON API for read action so we will Use SSIS JSON Source for this purpose.

Configure JSON Source

So below steps shows example of reading from ManageEngine Requests API

1. Open SSIS Package
2. Drag data flow task from Control flow SSIS Toolbox
   

   Drag data flow from SSIS Toolbox

3. Go to data flow and drag ZS JSON Source from SSIS Toolbox
   

   Drag and drop SSIS JSON Source (REST API or File)

4. Double click JSON Source to configure. Enter URL as below. You can supply few more parameters. Click Here to get full list of parameters and usage. In below example we are reading records from Leads Module but you can replace Leads to something else (e.g. Accounts). Refer to above help link for available Modules.
   

   https://sdpondemand.manageengine.com/api/v3/requests

5. Check Use Credentials. Select OAuth connection manager we created in the previous section.
6. Add the Accept header in HTTP Headers.
   

   Accept: application/vnd.manageengine.sdp.v3+json

7. Under filter options tab, enter Array Filter as  

   $.requests[*]

8. Now click Preview to see your data
9. Click OK to save UI. In the next section, we will see how to setup pagination and incremental extract of Servicedesk records. After that we will see how to load data into SQL Server Table.

Read ManageEngine ServiceDesk Plus data in SSIS (Cloud Version) – Zoho API

ServiceDesk API Pagination (Read all Requests)

Just like most API ServiceDesk also uses pagination. See below to learn how to configure pagination in JSON Source for ServiceDesk API.

1. Double click JSON Source
2. Change URL to include two things (Page size (how many rows per page and start row). Note <%page%> must match with Page Num Indicator you going enter in the next screen
   

   https://sdpondemand.manageengine.com/api/v3/requests?input_data={"list_info":{"row_count":100,"start_index":<%page%>}}

3. Go to Pagination Tab and select URL Path Mode
4. For Page Num Indicator : Enter

   <%page%>

5. For Increment By : Enter

   100

6. Page Start Number select New variable call it StartIndex and start default value 1

Once you configure this way now your API will return all records until last page is detected. For more information on pagination check this article

How to paginate ManageEngine ServiceDesk API

Loading ServiceDesk data into SQL Server Table / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Write data to ServiceDesk using OAuth 2.0

If you like to load data from any source (i.e. SQL Server) to Zoho CRM then you can use Records API (POST). Use a combination of the following Transforms  / Components. Later in this article we have some more examples (Older API) which may give you one more ways to load data into Zoho (Use of Template Transform).

Basically you have to build POST Body using JSON Generator Transform or Template Transform and then use that data in Web API Destination to call POST API request to push data to ServiceDesk. Basically screnshot below is just an example using some other Zoho API but you get the idea.

	Web API Destination
	JSON Generator Transform
	JSON Parser Transform

Create JSON for POST request (Single dataset pattern)

Insert data to Zoho Table using SSIS (Lead, Account Modules) – Bulk Load using POST API call

Write data to ServiceDesk using SSIS (Import SQL Server Table to ServiceDesk)

Now you know how to read data from Zoho CRM using JSON API Source next step is to load data to Zoho CRM. For writing data we will use following three components

	Web API Destination
	Template Transform
	XML Parser Transform

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Conclusion

Manageengine  Servicedesk API provides a great way to automate data read/write operations. However, to call Service API  you have to use SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Servicedesk API call via a  simple drag and drop approach without coding. Try  SSIS PowerPack for free and call virtually any REST API in a few clicks.

The post Read/Write ManageEngine ServiceDesk Data in SSIS (Zoho API) appeared first on ZappySys Blog.

In our previous post, we saw how to read Salesforce data using SSIS Salesforce Source (Which uses SOAP API under the hood). This may be slow if you have millions of rows because Salesforce Source paginates every 2000 rows, so it’s not a good option to read millions of rows to iterate on a large Salesforce dataset (Table or Query). To increase the read speed, you can use BULK API. This article explains how to read salesforce data using BULK API and increase read speed.

NEW Version (v5.5+)

In the new version (v5.5), we have added an option to query in Bulk Mode. This means if you have millions of Salesforce object rows to read, no worries; it may be very fast in the new version. Previously, it used to take a long time because it read in batches of 2000 rows in each request. With the Bulk API, the batch size is really large.

Here is how to set Salesforce Bulk API Mode in SELECT query (Read operation).

1. Install new version of SSIS PowerPack. Make sure it is v5.5 or higher.
2. Open existing Salesforce Source UI
3. Make sure you check [Use Bulk API] option like below. if you do not see this option then most likely you are using older version.
   -OR-
   Just add #bulkmode prefix in the SQL query like below, it will use Bulk API
   

   Bulk API mode for SSIS Salesforce Source (Read Large amount of data using SOQL Query)

OLD Version

If you are using an older version of SSIS Power Pack (v5.4.0 or older), you can follow the steps mentioned in the following section.

We will see how to query using Salesforce Bulk API in SSIS. It involves three simple steps like the one below.

1. Call BULK API – Submit SOQL Query and receive a JobId back (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

2. Keep checking Job Status Until Status is JobComplete (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

3. Read Query Result  (Use SSIS CSV Source)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Now, let’s look at everything in detail.

Obtain Salesforce Credentials

The first step to call Salesforce API is to obtain credentials (security token or OAuth ClientId /Secret)

Method#1 – Check this article to use UserId/ Password / SecretToken
Method#2 – Check this article to use the OAuth Connection Option

Create Salesforce Connection

Once we have Salesforce Credentials, we can create a new Salesforce Connection (You can do this from Task UI or the below way)

1. Open Visual Studio ( Assuming you already have SSIS installed along with ZappySys PowerPack )
2. Create a New SSIS Package Project.
3. Right-click on the Connection Manager panel and click on New Connection.
   
4. Select ZS-SALESFORCE Connection Manager from the Connection Managers list and Click on the Add Button.
   
5. Now in Connection Manager UI, Enter your SalesForce Credentials (OAuth Client Id/. Secret –or— User Name, Password and Security Token), and leave all other properties as it is.
   

Call Create Job – Retrieve / Save JobId

Once the connection is created, we can create our first step. Basically, we will call this API to create Job (Bulk Read)

1. Here, in Visual Studio, drag and drop the ZS Salesforce API Task on the design surface.
2. Double click the Task and configure Request Settings Tab like below
   – Select Salesforce Connection (Created in the previous section)
   – Check Use direct URL
   – Enter URL like below (Replace {your-instance} with your own instance (i.e. na9) . For more information on how to find out my
   salesforce instance id check this article
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

   – Request Method POST
   – Enter Body like below (You can use your own SOQL query). Keep in mind SQOL must be encoded for JSON format. This means the entire SQL must be in one line (replace new lines with \r\n , quotes with \” and tabs with \t)
   For example, to execute SELECT Id, Name FROM Accounts, you can write something like the one below in BODY.
   

   {
     "operation": "query",
     "query": "SELECT Id, Name\r\nFROM Account"
   }

   The above is for below (with a new line)
   

   SELECT Id, Name
   FROM Account

   

   Create Job – Call Bulk API to Read / Query using SOQL

3. Now Click Test Request (Copy the JobId which we will use in the Variable for Designtime Testing)
4. On the Response Tab, configure it like the below.
   – Select Response content Type = Json
   – Enter $.id as the expression
   – Check Save response option
   – Select Save to Variable and click New to create a variable. Name it JobId and enter a sample JobId value (copied from the previous step)
   

   Extract Salesforce JobId save to SSIS Variable

Do Status Check – Look for JobComplete

Once we create a job and store JobId in the variable we have to keep checking Job status every X seconds. Don’t worry you don’t have to use Loop or timer control for that. ZappySys made it so simple to check such thing without Loop /Timer.

1. Drag and drop the ZS Salesforce API Task in the design surface.
2. Double click and configure like below. URL will be different this time. Use variable as below and replace your instance ID
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

   

   Check Salesforce Job Status Periodically (Without Loop + Timer Controls)

3. Now go to the Status Check Tab and configure it, as shown below. It will keep looking for the words below in the response to decide whether it is a Success or Failure. If those words are not found, they will be checked every 5 seconds (Maximum 1200 secs).
   – Check Enable Status Check option
   – For Success Value – Enter JobComplete
   -Check Fail task option
   – Enter Cancel|Failed|Abort in the text box
   

   Configure Job Status Check Loop using Salesforce API Task

Read data (Compressed CSV Format)

Last step is read the result from completed Salesforce Job. Its in CSV format so we will use Advanced SSIS CSV Source which supports API connection to Salesforce. For that, we will use the same Salesforce API Task used before.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
2. Double click on the DataFlow task to see DataFlow designer surface.
3. Here, In Visual Studio, drag and drop the ZS CSV Source (Web API or File) in the design panel
   
4. Double click on ZS CSV Source (Web API or File) Configure it like below.
   – Enter URL to read result like this (You can supply OPTIONAL maxRecords parameters to control how much data comes in a single response)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results
   --OR--
   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results?maxRecords=50000

   – Check Use Credentials and select Salesforce Connection
   – Click Preview (assuming JobId variable has a valid completed JobId)
   

   Read Salesforce Bulk API Job Result – Using SSIS CSV / API Source

5. Now, set up Pagination as below to read all records using an automated loop. You will need VERSION Released after Jun 20, 2022 to use Regular Expression feature in Header Value extractionBy default, Salesforce doesn’t send all records in response so we need to setup pagination. In below example we are reading Sforce-Locator attribute from Response and sending it to the next URL as locator=<continue-token> value
   – Choose Response Header Contains Continue Token 
   – Enter below expression in Page Num Indicator
   

   locator=Sforce-Locator(^((?!null\b).)*$)

   Basically, the above expression will keep looping until we find Sforce-Locator : null in the response header.
   

   Pagination for Salesforce Bulk Query API (Get Header Value from Sforce-Locator)

6. Click OK to save and now run entire package like below.

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Conclusion

In this article we saw how to use various ZappySys Components / Task to achieve complex Salesforce API calls to read Query data using Bulk API. Download SSIS PowerPack and try yourself and also explore 70+ other tasks /components not discussed in this article.

The post How to query salesforce data using Bulk API in SSIS appeared first on ZappySys Blog.

In this post we will lean SSIS Magento data read / write operations. Magento is a very popular eCommerce platform and they offer JSON based REST API and XML based SOAP API. You can use either API based on your need to automate common integration needs.

We recommend using REST API (JSON API) if possible because they are simpler and faster. We will focus only on JSON API in this article but we do have blog post on how to call SOAP API.

About Calling Magento REST API

If you are new to Magento API we highly recommend to get familiar with Magento REST API here. Magento REST API  offers two ways to authenticate.

1. Token based Authentication
2. OAuth Based Authentication (OAuth 1.0a)

In this article we will look at only Token based approach. In token based approach we have to get new token using admin / customer account information.

We will cover later how to access admin or customer level API later in this article (See HTTP connection configuration).

Configure Connection / Call Magento REST API in SSIS

Now lets look at very simple example how to use Token based method to call Magento REST API in SSIS. We will first configure HTTP connection and then call REST API.

Let’s get started.

1. First, open SSIS Package after installing SSIS PowerPack
2. Drag ZS REST API Task from SSIS Toolbox on Control Flow Designer. Let’s rename task to Get Magento Categories
3. Enter REST API URL you like to call. For example we can use below URL. Replace **your-magento-host** with your own name.
   

   http://**your-magento-host**/index.php/rest/V1/categories

4. Select Url from Connection mode
5. From Connection Dropdown select new ZS-HTTP connection
   

   Create new connection for SSIS REST API Task

6. Configure General Tab of HTTP Connection like below
   URL can be anything for now because we will use override option in next step (On REST API Task).
7. Select Credentials Type as Dynamic Token.
8. Enter Userid / password
   

   Configure Magento REST API Connect for Token based Authentication

9. Now goto Dynamic Token Tab (Or click Configure Link) and configure like below.
10. Enter Token URL for API Login URL field.
    NOTE: If you are a customer and want to access your own account (e.g. view your orders) then replace /admin with /customer  in URL)
    Replace **my-host** with your host name.
    

    http://***my-host***/index.php/rest/V1/integration/admin/token

11. Change Method to POST.
12. Enter Body as below (Placeholders are automatically replaced when /token endpoint is called.
    

    {"username":"[$userid$]", "password":"[$password$]"}

13. Select Content Type as Application/Json
    

    Configure Magento Token Request

14. Now click Dynamic Token – Response Tab and configure like below.
    Select Regex and enter Expression as below. This will remove double quotes around token in response.
    

    "(.*)"{{0,1}}

    

    Extract Token from response – Remove double quotes around value using Regular Expression

15. Click OK to save connection
16. Now on REST API Task check Use Direct URL Option
17. Click Test Request to confirm its working
    

    Call Magento REST API using SSIS REST API Task

Read data from Magento and Load into SQL Server

Now lets look at how to read Magento data and load into SQL Server. Assume that you like to read all products.

Configure JSON Source (Read Magento REST API)

1. Drag Data flow and double click to edit:
   

   Dragging and dropping Data Flow Task into Control Flow

2. Drag ZS JSON Source on data flow and double click to edit
3. Configure JSON Source like below if you like to get product list for example
   1. Enter URL as below (change http to https if your host support secure channel).
      

      http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=1000
      --- to extract only specific fields do below way --
      http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=1000&fields=items[sku,name]

   2. Check Use Credentials. Use same HTTP connection we created in previous section
   3. In the Filter enter $.items[*] or click Select Filter to Browse and select items node.
   4. Click Preview to confirm
4. Here is how your setup will look like after configuration.
   

   SSIS JSON Source Configuration – Read data from Magento REST API

5. If you expecting more than 1000 rows then you have to set up pagination. Now go to pagination tab and configure as below (Below options are only available in v2.9.1 or higher). If you have older version then check next section for workaround.
   1. Pagination Mode =URL Parameter Mode
   2. Page Num Indicator = searchCriteria[currentPage]
   3. Max Rows Expression = $.total_count
   4. Page Data Expression = $.items[*]
6. Here is how your Magento REST API Pagination Setup will look like.
   

   REST API Pagination using Max Row count Mode

7. Click OK to save JSON Source. In next section we will see how to load data into Target like SQL Server.

Configure OLEDB Destination (Load into SQL Server)

1. Drag OLEDB Destination.
2. Select / Create Connection
3. Select or create NEW target Table (Click NEW button)
4. Click Mapping Tab to map source columns to target table
5. Click OK to save
6. Execute Package

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Configure Magento Pagination (For Old Version)

In previous section we configured pagination using newer version (v2.9.1 or higher). But what if you have older version and you dont see that feature. Use below steps for workaround.

Now let’s look at another common scenario when you real lots of data. Let’s say you have 5000 products but API call we saw earlier returns max 1000 rows. In that case we have to keep reading next page until all rows are returned. ZappySys provides many settings to configure various REST APIs out there, unfortunately there is no standard around pagination methods. Since Magento API has some bug in API pagination we have to do following hack. Magento keep sending same data of last page if you try to access page which doesn’t exists. Here is how to overcome that bug.

Configure REST API task to get total row count

First step to configure magento pagination is get total rows returned in our request which you like to paginate (in our case /products).

1. Go to Control Flow designer
2. Drag ZS REST API Task from SSIS Toolbox
3. Let’s rename task to Get Magento Product Count
4. Double click it and configure like below (Change HTTP to HTTPS if needed)
   

   http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=10

   
5. Go to Response Tab and configure like below. Enter expression as $.total_count
6. Click OK to save

Define Expression for JSON Source – MaxRows Property

Now next step is to define expression on MaxRows property of JSON Rows so we stop once all rows consumed from Paginated response. If you don’t do this it will keep paginating forever 🙁

1. On Control Flow designer select Data Flow which contains JSON Source for Magento
2. Right click and go to Properties of that data flow
3. Find Expression and click Button to add new expression
4. On Expression dialogbox select [Your JSON Source].[MaxRows] and for expression enter variable name which holds total count (e.g. @[User::MaxRows] )
5. Click OK to Save

Configure JSON Source for Magento Pagination

Now last thing we have to do is go to JSON Source and configure few things for pagination.

1. Go to data flow and double click JSON Source
2. Click on Pagination tab and select Pagination by URL Parameter Mode
3. Enter Page name as below
   

   searchCriteria[currentPage]

4. Click OK to save UI
5. Now run entire page to test (NOTE: Do not execute just data flow because if you do that way, it will not extract total row count)
   

Write data to Magento using SSIS Web API destination

Now let’s look at example to load data into Magento. In below example we will create few sample products.

1. Create new data flow and go to data flow designer
2. Drag and drop ZS CSV Source from SSIS toolbox. You can use any Data source but for simple demo we will use it.
3. Double click CSV source and select Direct Value mode from dropdown. Enter following sample data (Lets create 12 sample products.)
   

   SKU,ProductName
   PROD-1,Product 1
   PROD-2,Product 2
   PROD-3,Product 3
   PROD-4,Product 4
   PROD-5,Product 5
   PROD-6,Product 6
   PROD-7,Product 7
   PROD-8,Product 8
   PROD-9,Product 9
   PROD-10,Product 10
   PROD-11,Product 11
   PROD-12,Product 12

4. Now click OK to save UI
5. Now drag ZS Template Transform from toolbox.
6. Connect CSV Source to Template Transform
7. Enter following sample text to build request for new product. Refer to Magento API help file / tutorial to learn more.
   

   {
     "product": {
       "sku": "<%SKU%>",
       "name": "<%ProductName%>",
       "attribute_set_id": 4,
       "price": 25,
       "status": 1,
       "visibility": 1
      }
   }

8. Notice that when you click Insert Variable > Columns > select desired upstream column placeholder. You can also encode special characters in your data (e.g. new lines) using function like <%MyColumn,JSONENCODE%>
9. Click OK and save UI
10. Now drag ZS Web API Destination from SSIS Toolbox. Configure like below.Enter URL same as before (used to read product), change Method to POST, change content type to Application/JSON.
11. Now run entire flow (You can watch your API requests in Fiddler for debugging)

Debug Magento Web Requests using Fiddler

Click on below article to learn how to debug web requests

How to use Fiddler to analyze HTTP Web Requests

Magento Integration in Other Apps (e.g. Power BI / Informatica / SQL Server)

You can use techniques listed on this page with ODBC Drivers too for integration in other tools like Power BI, SQL Server code, Informatica etc.

Conclusion

In this post we saw how easy it is to perform Magento Integration in SSIS without doing any programming. In few steps you can read or write data in Magento using SSIS PowerPack REST API components. Download FREE Trial and explore many other scenarios.

The post SSIS Magento data Read / Write using REST API Call appeared first on ZappySys Blog.

In this blog, we will learn How to read data from Azure DevOps in SSIS and load into SQL Server Table. Azure DevOps was formally known as TFS Online / VSTS.

In this article we will learn, how to read Issues (Work Items) using Azure DevOps API (i.e. VSTS API). In last few years Microsoft has changed marketing names few times for TFS Service. For example everything started with  Visual Source Safe (VSS) > TFS > TFS Online > Visual Studio Online > VSTS (Visual Studio Team Services)) and finally now its called Azure DevOps. In this article we might use terms like VSTS or TFS  Online or Azure DevOps but not to get confused its all about Calling Azure Dev Ops API.

In this article we will try to cover few topics such as.

• Register an OAuth App to call REST API for Azure DevOps API (i.e. VSTS / TFS Online)
• Read all Issues (Work Items) data from VSTS with SSIS and finally
• Create new Issue/Bug in Visual Studio Team Service Project WorkItems.
• Authenticate and Call Azure DevOps using PAT (Personal Access Tokens) rather than OAuth 2.0

We will go through the steps to read Issues data from Visual Studio Team Service and Load into SQL Server using  SSIS REST API Task and SSIS JSON Source.

In nutshell, this post will focus on how to call VSTS API using SSIS.

So let’s get started.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Finally, do not forget to install ZappySys SSIS PowerPack

Call Azure DevOps API Using PAT (Personal Access Token)

There are two ways to call Azure DevOps API. You can use OAuth 2.0 or use PAT (i.e. Personal Access Token) In the next section we will see how to use OAuth 2.0 if you like more secure way to authenticate (OAuth 2.0 requires more steps).

Microsoft offers Static API Token (referred as PAT – Personal Access Token) which you can use like a password (it remains valid for 1 year before you have regenerate).

So let’s get started.

Create PAT (Personal Access Token)

1. Visit to

   https://dev.azure.com/YOUR_COMPANY_NAME/_usersSettings/tokens

    . Create Personal Access Token (i.e. PAT ) by clicking User Settings icon > Personal Access Token (see below)  (detailed steps listed here )
   

   Call Azure DevOps API (formally known as VSTS / TFS Online) – Create Personal Access Token (i.e. PAT)

2. On Token Configuration Screen change Expiration to maximum (i.e. one year from today)
3. Now select permission – We will call Analytics API below (but you can also check Work Items section if you need to call some other API for read / write purpose)
   

   Change Permission / Expiration Date for Azure DevOps PAT (Personal Access Token) – Read / Write Work Items, Call Analytics API Example

Call Azure DevOps OData API in SSIS – Read WorkItems  Example (Analytics API Service)

Now lets configure JSON Source to read WorkItems by calling  Azure DevOps Analytics API (in Previous few sections we call different set of APIs). Check this link for more information on AzureDevOps Analytics API (its OData format).

For demo purpose we are calling WorkItems entity but you can read from many other entities like (Read Areas, Iterations, Projects, Teams, Tags etc). Check this link to see which Entity (i.e. Table) you can query for specific OData API version.

So let’s get started

1. Go to SSIS designer and drag Data Flow Task on the Designer Surface
   
2. Then double click Data Flow and from toolbox drag ZS JSON Source
   
3. Double click JSON Source to configure like below
   1. First change Data Format to OData (Analytics API we will call in this section is OData format)
   2. Enter URL to Query data like below. Change YOUR_COMPANY_NAME with your own name. Below API URL will call Analytics will
      

      https://analytics.dev.azure.com/YOUR_COMPANY_NAME/Odbc/_odata/v3.0/WorkItems?$select=WorkItemId,Title,WorkItemType,State,CreatedDate&$orderby=CreatedDate desc&$top=11

   3. Check Use Credentials and Select new ZS-HTTP connection (instead of OAUTH)
4. On HTTP Connection UI Configure below
   1. Enter Some URL – This will be ignored anyways (e.g. https://dev.azure.com )
   2. Change Credentials Type to Basic (User ID / Password)
   3. Enter some random word in User ID -i.e. fakeusername (this is ignored anyways but needs some non-empty value for older version of SSIS PowerPack else validation fails)
   4. Enter PAT (i.e. Personal Access Token) obtained in the previous section
5. Click OK to Save Connection UI and go back to JSON Source
6. On JSON Source, Enter or Select Array Filter as 

   $.value[*]

7. Click Preview data see its working using HTTP connection method

Configure JSON Source – Call Azure DevOps Api using PAT (Personal Access Token)

Calling Azure DevOps API in SSIS using OAuth 2.0

Now in this section we will see how to use more secure but little harder approach to authenticate. We will use OAuth 2.0 this time rather than static token (PAT) which we used in previous section.

In order to start, we will show several examples. ZappySys includes an SSIS JSON Source that will help you to call VSTS API, read various data from VSTS with SSIS, create a new issue on VSTS with SSIS and do many more actions using REST API. To learn more about VSTS API check this help file.

Create VSTS App (for OAuth 2.0 API Call)

To access VSTS data using REST API call you need to create a VSTS app. Follow the mentioned steps below to register VSTS OAuth App:

1. you can directly login to the Visual Studio Team Service – TFS Online site with your credentials, and redirect to the Visual Studio Team Service App Registration for creating an app.
2. you can open MyApps Page.
   

   Visual Studio Team Service: My Apps

   Visual Studio Team Service: My Apps[/caption]

3. Click on Create new application link, and you will be redirected to the Visual Studio Team Service App Registration Page.
4. Make sure you enter Authorization call base URL. We suggest to below call back URL if you are unsure. We will use exact same URL on OAuth connection UI (Advanced Tab)

   https://zappysys.com/oauth

5. 

   Visual Studio Team Service: New OAuth App Registration

6. Select the Authorization scopes as per your requirements, and click on Create Application button.
   For example to read Work Items or Create new Issues programmatically you can check following scope (i.e. OAuth permission).Check – Work items (full)  (i.e. vso.work_full scope)
   Note: The scopes selected on your registered application (below screenshot) must match the scopes entered in the ZappySys OAuth connection scopes field (see below). If they do not match exactly, an InvalidScope error can be produced.
   
   
   

   Visual Studio Team Service: Select Authorization Scopes

7. Once you are done with the Creation of an App certainly, you can edit the Application If you want to change Redirect URL or something.
8. furthermore, Go to MyApps Page and you have your App’s Lists there with App’s App ID as Client ID and App Secret as Client Secret.
   

   Visual Studio Team Service: Your Apps with App’s Credentials

finally, we are ready with an App. Let´s start with an example. In this example, we will show how to retrieve all Issues data from Visual Studio Team Service – TFS Online and create a new Issue in Visual Studio Team Service – TFS Online Work Items List under your Project with SSIS.

Load VSTS data to SQL Server using SSIS JSON Source

Now once you registered OAuth app in the Visual Studio Team Service – TFS Online website we can move to SSIS Part.

Configure OAuth Connection Manager

1. Right click on Connection Managers Panel to Create New OAuth Connection, and Context Menu will appear, Select New Connection from the Context Menu.
   

   Connection Manager Panel: Select Connection

2. Select ZS-OAUTH Connection Manager from the Connection Managers list.
   

   SSIS Connection Mangers List: Select O-AUTH Connection Manager

3. Configure the new OAuth Connection for VSTS(Visual Studio Team Service – TFS Online) as below.
4. First lets enter Call back URL (Same URL you entered when you created App earlier)SSIS OAuth Connection Manager: Advanced Tab Configurations
5. For VSTS API you need to set the Refresh Token File Path as below because VSTS API Token’s Refresh Token keeps changing.
6. Then configure the App on the General tab like below (i.e. OAuth Provider, Client ID, Client Secret, Scopes, etc.)
   Scope must be valid one from this list (only those scope allowed which are checked during app creation). For example during app creation you check Work items (full) then you can enter vso.work_full in scope.
   Note: The scopes selected on your registered application must match the scopes entered in the scopes field (below screenshot). If they do not match exactly, an InvalidScope error can be produced.
   
7. finally, you are done with the OAuth Configurations. Click on Generate Token button to Generate token and then click on Test Connection button and click on OK to save the connection manager configurations settings.

How to get ProjectID?

If you don’t know the ProjectID, then you can also enter ProjectName in place of ProjectID in Request URL. you can also call GET Request for getting Lists of Projects. the Request URL for getting the list of Projects is:

https://{accountName}.visualstudio.com/_apis/projects?api-version=4.1

For this, we will use SSIS REST API Task

1. First of All, Goto Control Flow Section, Drag and Drop SSIS REST API Task from the SSIS Toolbox.
   

   Drag and Drop REST API Task

2. furthermore, Double click on that Component to Edit the Configurations and Configure the task like the below-attached screesnshot.
   Some basic configurations need to be done in SSIS REST API Task.
   # Request Tab: 

   * Select Request URL Access Mode to Url from Connection.
   * Select Url Connection to your configured ZS-OAuth Connection Manager.
   * Enter Request URL: https://zappysys.visualstudio.com/_apis/projects?api-version=4.1
   * Select HTTP Request Method: GET

   

   SSIS REST API Tsk: Request Tab Configurations

   Create New Variable from Variable Menu

   

   SSIS User Variables: Create a new Variable and set Value

   # Response Tab: 

   * Select Response Content Type: Json
   * Set Response Content Filter Expression: $.value[1].id
   * check the checkbox of Save Response Content
   * Set Save Mode to Save to Variable
   * Select your Variable: {{User::ProjectID}}

   

   SSIS REST API Task: Response Tab Configurations

3.  finally, you are done with the configurations and do Test Request/Response and your ProjectID will be saved in {{User:: ProjectID}}

Configure JSON Source Component

1. First of all, Create a new SSIS Project or Open Existing Project in Visual Studio  (File > New > Project > Business Intelligence > Integration Services)
2. Once SSIS Package is open, Goto Data Flow Section, Drag and Drop ZS JSON Source Component from the SSIS Toolbox.
   

   Drag and Drop JSON Source Component

3. furthermore, Double click on that Component to Edit the Configurations. Select File path or web URL from the Access Mode drop-down.
   Enter Web URL like this:
   

   https://{accountName}.almsearch.visualstudio.com/{project}/_apis/search/workitemsearchresults?api-version=4.1-preview.1

   here, We have used 2 Parameters.

   * {accountName} is the name of the Visual Studio Team Services – TFS Online account.
   * {project} is Project ID or project name (For more See this Visual Studio Team Service API Documentation for getting filtered Issues data Request API)
   We have used Search Work Items for Fetching Paginated Work Items(Issues, Bugs)

   You can use Variable too in your Request URL, to make URL Dynamic, See the below-attached Screenshot.
   To make URL dynamic first create an SSIS variable like below screenshot and then change URL to use Variable Placeholder like this

   https://zappysys.almsearch.visualstudio.com/{{User::ProjectID}}/_apis/search/workitemsearchresults?api-version=4.1-preview.1

    

SSIS JSON Source Component: Enter the Request URL

JSON Source Configuration for Pagination

1. now, we are going to set Pagination related configurations of Visual Studio Team Service – TFS Online for Getting all WorkItems with Search POST REST Request API. Here in our example,
   Request Method: POST
   Body Content-Type: JSON (application/json)
   Request Body:
   
   

   {
     "searchText": "a",
     "$skip": <%page%>,
     "$top": 100,
     "filters": {
       "System.TeamProject": [
         "ProductTesting"
       ]
     }
   }

   here, SearchText: The SearchText for filtering WorkItems based on the keywords match.
   $skip: Number of results to be skipped
   $top: Number of results to be returned.
   
   you can take any number of the page size. for example, you can set 100 in both $skip and $top properties (I have set 10 for the Demo Purpose only)
   filters: Filters to be applied. Set it to null if there are no filters to be applied.
   System.TeamProject: ProjectName in which you want to do filtering of WorkItems.
   

   SSIS JSON Source Component: Configurations of Settings Tab

2.  You can Configure the Pagination Tab something like this:
   

   SSIS JSON Source Component: Pagination Tab Settings

3. furthermore, Select the Filter to set the Filter.
   

   SSIS JSON Source Component Configurations: Select Filter

4. It seems like you are all OK with the configurations, Click on the Preview button to see the response data.
   

   SSIS JSON Source Component: See the Preview

5. finally, Click on OK button to save JSON Source settings.

Configure OLE DB Destination Component for dumping Contacts data into SQL

1. first of all, Drag any destination (i.e. OLEDB Destination) from your SSIS toolbox. For demo purpose, we will load Contacts data into SQL Server Table.
   

   Drag and Drop OLE DB Destination Component

2. Configure OLEDB Destination (Select Connection and Target Table). You can Create a new Table for the destination by clicking a NEW button next to Table drop-down.
   

   OLE DB Destination Component – Configure for dumping WorkItems of Project from VSTS data in SQL Table

3. almost you are done with the configurations, just need to configure mappings, So for that Click on the Columns Tab to configure Mappings. also, Click on OK button to Save the configurations of OLE DB Destination Component.
   

   OLE DB Destination – Column Configurations

4. furthermore, You can see the requests in Fiddler to see the Paginated requests flows.
   

   Fiddler – See All The Requests

5. finally, Execute your package to Load Contacts data from Visual Studio Team Service – TFS Online to SQL Server.

Create a new WorkItem on Visual Studio Team Service – TFS Online (Use SSIS REST API Task)

Now let’s look at one more example to call Visual Studio Team Service(VSTS) – TFS Online REST API. This time we will create a new WorkItem using REST API call in SSIS. For this, we will use SSIS REST API Task

1. First of All, Goto Control Flow Section, Drag and Drop SSIS REST API Task from the SSIS Toolbox.
   

   Drag and Drop REST API Task

2. furthermore, Double click on that Component to Edit the Configurations and Configure the task like the below-attached screesnshot.
   

   ::::: Request Settings :::::
   
   * Select Request URL Access Mode to select Url from Connection.
   * Select Url Connection: Select your created OAUTH Connection Manager.
   * Enter Reqeust URL
         Request URL: POST https://{accountName}.visualstudio.com/{project}/_apis/wit/workitems/${type}?api-version=4.1
         here, 
         {accountName} - The name of the Visual Studio Team Services account.
         {project} - Project ID or project name
         {type} - WorkItems Type (Task, Bug, Issue, Requirement etc)
   * Select HTTP Request Method as PATCH method
   * Enter Request Body.
         Request Body:
         [
            {
                "op": "add",
                "path": "/fields/System.Title",
                "value": "Sample task"
            }
         ]
   * Select Body Content Type: Text/Plain
   *Enter Request Header:
           Content-Type: application/json-patch+json
   (For more Information See this Visual Studio Team Service - TFS Online API Documentation for Creating a new WorkItem
   https://docs.microsoft.com/en-us/rest/api/vsts/wit/work%20items/create?view=vsts-rest-4.1)

   

   REST API Task: Configurations

3. So all good with the Configurations let’s do Test Request/Response by clicking on Test Request/Response button. See the below-attached request/response screen.
   

   Create New WorkItem in VSTS Account: Do Test Request/Response

4. finally, you have created a WorkItem in VSTS successfully using VSTS REST API and REST API Task, See in VSTS Site.
   

   VSTS Site: See the Newly Created WorkItem

5. And that’s it. similarly, you can Update/Delete particular WorkItem in WorkItems List in VSTS.

Conclusion

In this blog, we learned how to read Visual Studio Team Services – TFS(Team Foundation Server) Online data in SSIS, How to register an OAuth App for Visual Studio Team Service REST API, How to get all WorkItems data from Visual Studio Team Service  – TFS(Team Foundation Server) Online and create a new WorkItem on VSTS with SSIS using ZS JSON Source Component and SSIS REST API Task in a very simple way. You can achieve many more functionalities with the use of both of these tools. Check our blogs/articles on REST API Task https://zappysys.com/blog/tag/ssis-rest-api-task/ and JSON Source Component https://zappysys.com/blog/tag/ssis-json-source/ to find out what these tools are capable of more.

References

Finally, you can use the following links for more information about the use of VSTS API with our tools:

• Visual Studio Team Service Documentation for API
• Landing Page for REST API Task, you can also find Tutorial Video here.
• Landing Page for JSON Source Component, you can also find Tutorial Video here.
• Help File: Documentation of SSIS REST API Task and JSON Source Component.

The post Read Azure DevOps data in SSIS (VSTS / TFS Online) – Call REST API / Load to SQL Server appeared first on ZappySys Blog.

In this blog, we will learn How to read Constant Contact data in SSIS and load into SQL Server Table, along with few other topics such as how to generate an API Token using Mashery Developer Account for Constant Contact REST API Call, how to read all Contacts from Constant Contact with SSIS and finally How to add new Contact in Constant Contact Contact List Collection.

We will go through the steps to read Contact data from Constant Contact and Load into SQL Server using SSIS JSON Source.

In nutshell, this post will focus on how to call Constant Contact API using SSIS. If you are MailChimp user and curious how to perform data integration in SSIS then check this article about reading MailChimp data in SSIS.

So let’s get started.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Finally, do not forget to install ZappySys SSIS PowerPack

What is Constant Contact

Keeping in touch with customers on a consistent basis is one key to maintaining current clients and bringing in new ones. A number of companies exist that provide businesses with a way to contact clients regularly through snail mail, email or social media. Constant Contact offers online marketing services for businesses to stay in touch with clients.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS REST API Task and SSIS JSON Source that will help you to call Constant Contact API to read Constant Contact data, retrieve lists of Contacts data from Constant Contact, Add new Contact in Constant Contact with SSIS and do many more actions using REST API. To learn more about Constant Contact API check this help file.

Setup your Mashery Developer Account

To access Constant Contact data using Mashery Developer Account you need to register a MasheryID with Constant Contact. Follow the steps mentioned below to register Mashery Developer Account for Constant Contact Access:

1. first of all, you can directly login to the Constant Contact site with your credentials, and redirect to the Mashrey Developer Console to register a MasheryID.
2. thereafter Fillup the form and click on Register button
   

   Register Mashery ID to Constant Contact

3. If you have already registered on Mashery Developer Account, then you just have to log in using your Credentials.
   

   Login to Mashery Developer Account

4. So, when you are done with the Mashery, you can go to MyApplication Page to create a new application.

Create Constant Contact App (for OAuth)

To access Constant Contact data using REST API call you need to create a Mashery app. Follow the steps mentioned below to register Constant Contact OAuth App:

1. First of all, you can directly login to the Mashery Developer Account with your credentials, and redirect to the Constant Contact My Application’s Page for creating an app.
2. also, you can open the MyApps Page.
   

   Constant Contact: My Apps

3. Click on Create a New Application button while, the new Application form appears, Fill it up.
   

   Constant Contact: Create New Application Form

4. Once you are done with the Creation of an App certainly, you can edit the Application.
   Enter the Redirect URL there.
   

   Constant Contact: Edit Application

5. furthermore, Go to Keys tab and you have your App’s Lists there with App’s Key as Client ID and Secret as Client Secret.
   

   Constant Contact: Apps Keys

finally, we are ready with an App. Let´s start with an example. In this example, we will show how to retrieve all Contacts data from Constant Contact and add a new Contact in Contact Collection List on Constant contact with SSIS.

Load Constant Contact data to SQL Server using SSIS JSON Source

Now once you registered OAuth app in the Constant Contact website we can move to SSIS Part.

Configure JSON Source Component

1. First of all, Create a new SSIS Project or Open Existing Project in Visual Studio  (File > New > Project > Business Intelligence > Integration Services)
2. Once SSIS Package is open, Goto Data Flow Section, Drag and Drop ZS JSON Source Component from the SSIS Toolbox.
   

   Drag and Drop JSON Source Component

3. furthermore, Double click on that Component to Edit the Configurations. Select File path or web URL from the Access Mode drop-down.
   Enter Web URL like this:
   

   https://api.constantcontact.com/v2/contacts?status=ALL&limit=100&api_key=<<<Your_App_Key>>>

   here, We have used 2 Parameters.

   * status is the filter the response based on the contact status property: ALL, ACTIVE, UNCONFIRMED, OPTOUT, or REMOVED.
   * limit is the number of contacts to return per page, 1-500, default 50 (For more See this Constant Contact API Documentation for getting Contacts Request API)
   * <<<Your_App_Key>>> is your App Key, So you need to add your app key.

   You can use Variable too in your Request URL, to make URL Dynamic, See the below-attached Screenshot.
   To make URL dynamic first create a SSIS variable like below screenshot and then change URL to use Variable Placeholder like this

   https://api.constantcontact.com/v2/contacts?status=ALL&limit=100&api_key={{User::AppKey}}

   

   SSIS User Variables: Create a new Variable and set Value

   

   SSIS JSON Source Component: Enter the Request URL

Configure OAuth Connection Manager

1. Check Use Credentials Option and select a new ZS-OAuth Connection from the drop-down.
2. Configure the new OAuth Connection for Constant Contact as below.
   Basically, you need to configure the configurations of App on the General tab (i.e. Client ID, Client Secret, Authorization URL, Access Token URL)
   

   SSIS OAuth Connection Manager: General Tab Configurations

   also, most importantly, you need to Enter your Callback URL in Advanced Tab.

   

   SSIS OAuth Connection Manager: Advance Tab Configurations.

JSON Source Configuration for Pagination

1.  now, we are going to set Pagination related configurations for Constant Contact Getting all Contacts request. Here in our example, we are setting Page size as 5. while We need to read the next page information from the response.
   So, we can set the status=ALL, Limit=100 and also we can specify Next Link Expression something like this: $.meta.pagination.next_link and Suffix for Next URL: ?api_key={{User::AppKey}}
   
   

   SSIS JSON Source – Constant Contact request Pagination Configurations Settings

2. furthermore Select the Filter to set the Filter. seems like you are all OK with the configurations, Click on Preview button to see the response data.
   

   JSON Source: Preview Contacts data of Constant Contact

3. finally, Click on OK button to save JSON Source settings.

Configure OLE DB Destination Component for dumping Contacts data into SQL

1. first of all, Drag any destination (i.e. OLEDB Destination) from your SSIS toolbox. For demo purpose, we will load Contacts data into SQL Server Table.
   

   Drag and Drop OLE DB Destination Component

2. Configure OLEDB Destination (Select Connection and Target Table). You can Create a new Table for the destination by clicking a NEW button next to Table drop-down.
   

   OLE DB Destination Component – Configure for dumping Constant Contact data in SQL Table

3. almost you are done with the configurations, just need to configure mappings, So for that Click on the Columns Tab to configure Mappings. also, Click on OK button to Save the configurations of OLE DB Destination Component.
   

   OLE DB Destination – Column Configurations

4. furthermore, You can see the requests in Fiddler to see the Paginated requests flows.
   

   Fiddler – See All The Requests

5. finally, Execute your package to Load Contacts data from Constant Contact to SQL Server.

Add a new Contact on Constant Contact (Use SSIS REST API Task)

Now let’s look at one more example to call Constant Contact REST API. This time we will add a new contact using REST API call in SSIS. For this, we will use SSIS REST API Task

1. First of All, Goto Control Flow Section, Drag and Drop SSIS REST API Task from the SSIS Toolbox.
   

   Drag and Drop REST API Task

2. furthermore, Double click on that Component to Edit the Configurations and Configure the task like the below-attached screesnshot.
   

   Request URL: 
   https://api.constantcontact.com/v2/contacts?action_by=ACTION_BY_OWNER&api_key=<<<YOUR_APP_KEY>>>
   
   here, action_by Can be ACTION_BY_OWNER or ACTION_BY_VISITOR (from the API Document,Click here for more Informaion)
   Sample Request Body: 
   {
   	"addresses": [{
   		"line1": "4928 Small Street",
   		"city": "New York",
   		"address_type": "PERSONAL",
   		"state_code": "MA",
   		"country_code": "us",
   		"postal_code": "10022"
   	}],
   	"lists": [{
   		"id": "1652428712"
   	}],
   	"email_addresses": [{
   		"email_address": "hbshah@zappysys.com"
   	}],
   	"prefix_name": "Mr.",
   	"first_name": "Hardik",
   	"last_name": "Shah",
   	"job_title": "Tech Supporter",
   	"company_name": "ZappySys",
   	"home_phone": "555-555-5555",
   	"work_phone": "555-555-5555",
   	"cell_phone": "555-555-5555",
   	"fax": "555-555-5555",
   	"custom_fields": [{
   		"name": "CustomField1",
   		"value": "Has control of $25 million budget"
   	}]
   }

   

   REST API Task: Configurations

3. now you are done with the configurations. you can do Test Request/Response. See the below-attached request/response screen.
   

   Add New Contact in Constant Contact Account: Do Test Request/Response

4. finally, you have successfully added a contact. furthermore, Goto your Constant Contact Account on its site, you can see the latest Contact added which is added by us with the use of ZappySys and SSIS.
   

   Constant Contact Site: See the Newly Added Contact

5. though, You are done with adding a contact in Constant Contact. similarly, you can Update/Delete particular Contact in Contact Collection List.

Conclusion

In this blog, we learned how to Setup your Mashery Developer Account, Create Constant Contact App for OAuth, Get all Contacts data from Constant Contact and add a new contact on Constant Contact with SSIS using ZS JSON Source Component and SSIS REST API Task in a very simple way. You can achieve many more functionalities with the use of both of these tools. Check our blogs/articles on REST API Task https://zappysys.com/blog/tag/ssis-rest-api-task/ and JSON Source Component https://zappysys.com/blog/tag/ssis-json-source/ to find out what these tools are capable of more.

References

Finally, you can use the following links for more information about the use of Constant Contact API with our tools:

• Constant Contact Documentation for API
• Landing Page for REST API Task, you can also find Tutorial Video here.
• Landing Page for JSON Source Component, you can also find Tutorial Video here.
• Help File: Documentation of SSIS REST API Task and JSON Source Component.

The post How to read Constant Contact data in SSIS – Call REST API / Load to SQL Server appeared first on ZappySys Blog.

Introduction

In this post, we will learn how to call Zuora API using SSIS.  We will go through steps to read data from Zuora and load into SQL Server using  SSIS REST API Task  and  SSIS JSON Source . At the end of this article, you will find a complete sample (dtsx file).

Tasks/Components in SSIS for Consuming RESTful API / WebService

Below are few components you can use to read/write data using REST API for any web service.

	JSON Source (Read from REST API, JSON File or OData Service): Use this dataflow component when you have to fetch data from REST API webservice like a table. This component allows you to extract JSON data from webservice and de-normalize nested structure so you can save to Relational database such as SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local JSON files or direct JSON string (Wildcard pattern supported too e.g. c:\data\file*.json).
	Web API Destination (POST data to API URL) : Use this dataflow component when you have to load data into target system via API calls (POST to URL). E.g. Loading Contacts into Marketing System like MailChimp for email campaign. Loading documents into CRM or document storage systems e.g. Couchbase or ElasticSearch.
	REST API Task : Use this task when you don’t want to pull REST API data in tabular format but want to call rest API for POST data to server, DELETE data from server or things like download HTML page, extract Authentication tokens etc where you not necessarily dealing data in tabular format. This task also allows you many other options such as saving RAW response into variable or file.
	XML Source (SOAP, File, REST) : Use this dataflow component when you have to fetch data from XML or SOAP webservice and consume data like a table. This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local XML files or direct XML string.

Screenshot of Complete SSIS Package (Zuora to SQL Server)

Here is screenshot of our full process (we going to outline in next sections)

SSIS Zuora API Integration – Read/Write Data (Load Zuora Data to SQL Server Table)

Step-By-Step : Using SSIS JSON Source / REST API Task to read Zuora API 

Zuora API is somewhat different than other APIs we documented here because Zuora API uses very different Pagination Technique. Some common pagination techniques listed here but it’s not used by Zuora API so we have to do few extra steps.

Let’s look at step by step.

Configure Zuora OAuth Connection

The first step to connect to Zuora API is to configure  the OAuth App (Ask your Zuora Admin or Follow Steps listed here). Once you create Oauth App get ClientId and ClientSecret.

Now let’s create new SSIS

1. Right click in the Connection Manager Panel > Click New Connection…
2. Select ZS-OAUTH from the list
3. Configure OAuth Connection as below
   1. Set OAuth Version

      OAuth2

   2. Set OAuth Grant Type to

      Client Credentials Grant

   3. Enter Client ID and Client Secret
   4. For Token URL enter URL

      https://rest.zuora.com/oauth/token

   5. Goto Advanced Tab. Select SSL/TLS Protocol

      v3.0 or Higher

4. Click Test (Confirm Green Checkmark)
5. Click OK to Save

SSIS OAuth Connection For Zuora API

Configure Variables

Now lets configure few SSIS variables.

SSIS Variables for Zuora API Integration

@counter > 0 
   ? "https://rest.zuora.com/v1/action/queryMore" 
   : "https://rest.zuora.com/v1/action/query"

{"queryString": "SELECT Id,Name,AccountNumber,Status FROM Account"}

Configure API Pagination

Now let’s configure For Loop such a way that it keeps looping until @continue flag is True

1. Drag For Loop Task from SSIS Toolbox
2. Configure as below
   1. Enter EvalExpression as

      @continue==true

   2. Enter AssignExpression as

      @counter=@counter+1

Configure API Pagination using For Loop Task

Configure API Call using SSIS REST API Task

Now let’s configure REST API Task to make a call to get data from Zuora API and save JSON response into File (which we will parse in next step)

So if we making the first request will call API via

https://rest.zuora.com/v1/action/query

https://rest.zuora.com/v1/action/queryMore

This is controlled by Expression we defined in URL variable.

Configure SSIS REST API Task – Call Zuora Query API (ZOQL – SQL Query)

Configure SSIS REST API Response Settings – Save Zuora Query to JSON File

Configure Data Flow (Read Zuora Data and Load into SQL Server Table)

We will skip this configuration on how to read JSON file and Save to SQL Server. Because it will be covered in next section.

Inside Data Flow when you configure ZS JSON Source

Make sure to set Filter to $.records[*]

Parse Pagination Attributes (done flag, queryLocator for next page)

Once that done you can parse the response from saved File. We need to check for 2 attributes (done and queryLocator ). Parse JSON as below.

SSIS JSON Parser Task –
Parse Multiple Fields from JSON FIle and Save to Variable

Detect Last Page (Set Variables)

Drag two Set Expression Tasks from SSIS Toolbox.

For First Task [Set Continue Flag] Enter Expression as 

@continue=!@doneFlag

For First Task [Set Query Locator ] Enter Expression as 

@[User::bodyString]="{ \"queryLocator\" : \"" + @[User::queryLocator] + "\" }"

Save and execute Package

That’s all now you can save package and execute. Check Progress.

Debug Web API call using Fiddler

To test how things looking behind the scene we strongly suggest to use tool like  fiddler. You can double click the URL entry (Right side) to see Request and Response Panels. Top panel is Request (URL, Headers, Body) and Bottom Panel is Response.

Debugging Web API call using Fiddler in SSIS

Loading data from REST API to SQL Server

Once you setup HTTP / OAUTH connection we can use JSON Source or XML Source to extract data from API and  load into SQL Server. If you are API is XML based API then use XML Source in below examples else use JSON Source. For example purpose we will load data into SQL Server but you can load into any Target (e.g. Flat file, Oracle, Excel) using Microsoft or ZappySys Destination connectors To load REST API data into SQL Server perform the following steps (Screenshots are generic and used for demonstration for purpose, your values can be different than below)

1. Go to SSIS Data Flow tab
2. Drag ZS JSON Source from the SSIS Toolbox (Use ZS XML Source if your API is XML based API)
3. Double click to edit Source
4. Set API URL you like to call, Check Use Credentials if you need to use HTTP or OAuth connection
5. Set Method / Body if its other than GET. For method other than GET make sure you select correct Content-Type from Dropdown (e.g. application/json). This indicate input data format in Body.
6. Set HTTP Headers if needed (in Most cases you won't need to set this)
7. Select Filter (many times this is not needed). If your API response has array node then select it here. For example if your response looks like this { rows: [ {...}, {...}, {...} ] } then filter would be $.rows[*]. Select Array node you like to extract in the Filter selection UI.
8. Configure Pagination if needed on the pagination tab. Check your API documentation see it supports pagination.
9. Click OK to Save component settings and generate desired metadata / columns for the output.
10. Now drag OLEDB destination from the SSIS toolbox and drop on the Data Flow surface (technically you can use any valid Destination but for demo we will use OLEDB Destination)
11. Connect Source to Destination Component
12. On OLEDB destination select / create new SQL Connection and then Click "New Table"
13. Click on Mappings tab to map columns by names and click OK to save
14. Execute the Package

Configure SSIS OLEDB Destination – Loading REST API Data into SQL Server Table

REST API to SQL Server Column Mappings for OLEDB Destination

Loading REST API to SQL Server in SSIS

POST / Write Data to API (Insert, Update or Delete)

There will be a time you like to automate POST actions (e.g. create new incident via API call). Check this article to learn more on how to POST , DELETE, PUT data using API calls

Common Errors

There are many error can occur during runtime. Here are most common errors you may face at runtime.

Truncation related error

The most common error you may face when you run a SSIS package is truncation error. During the design time only 300 rows are scanned from a source (a file or a REST API call response) to detect datatypes but at runtime it is likely you will retrieve far more records. So it is possible that you will get longer strings than initially expected. For detailed instructions how to fix common metadata related errors read an article "How to handle SSIS errors (truncation, metadata issues)".

Authentication related error

Another frequent error you may get is authentication error, which happens when you deploy/copy a package to another machine and run it there. Check Deployment to Production below to see why it happens and how to solve this problem.

Deployment to Production

In SSIS package sensitive data such as tokens and passwords are by default encrypted by SSIS with your Windows account which you use to create a package. So SSIS will fail to decrypt tokens/passwords, when you run it from another machine using another Windows account. To circumvent this when you are creating a SSIS package which uses authentication components (e.g. an OAuth Connection Manager or an HTTP Connection Manager with credentials, etc.), consider using parameters/variables to pass tokens/passwords. In this way you won’t face authentication related errors when package is deployed to a production server. Check our article on how to configure packages with sensitive data on your production or development server.

Download Sample SSIS Package (2012 or Higher Version)

Click here to download SSIS Sample Package – ZuoraApiPagination.dtsx

Conclusion. What’s next?

In this article, we have learned how to load data from Zuora and load into SQL Server using SSIS ( drag and drop approach without coding). We used SSIS JSON Source/REST API to extract data from Zuora REST API using OAuth. JSON Source makes it super simple to parsing complex / large JSON Files or any Web API Response into rows and column so you can load into a database like SQL Server. Download SSIS PowerPack to try many other automation scenarios not discussed in this article.

The post How to Import / Export data from Zuora API using SSIS appeared first on ZappySys Blog.

In our previous blog, we saw how to call REST in SSIS including concepts of Authentication / Error Handling. Now in this post, we will cover API Pagination. You will learn concepts and patterns about REST API Pagination (for JSON / XML / SOAP or CSV API).

We will describe methods and steps to implement REST API Pagination in SSIS PowerPack or  ODBC PowerPack API Drivers (for apps like Excel, Power BI, Informatica…) without coding. Both Products share a similar user interface with minor differences. For example purpose, we will show screenshots from SSIS PowerPack UI but concepts should remain same. You will also learn some techniques about how to implement an infinite loop in SSIS. Sometimes your REST API requires you to loop through all pages until no more data found then you can use pattern described in this article (sample package attached at the end).

In this article, we will use SSIS REST API Task and JSON Source Connector to process data from RESTful web service. If you have to process XML (or SOAP) data from REST API then simply use XML Source Connector for REST API, SOAP or File rather than JSON Source.

Debugging REST API Requests

Before we look at various aspects of REST API Pagination, we highly recommend you install FREE Web Debugging Tool Fiddler. Look at this article to understand how to debug REST API requests.

Method-1: REST API Pagination via URL Parameter

If you have paging requirement by specifying page number in URL then you can loop until no more pages found. Your REST API can indicate no more page found by any of the following ways.

Example URL:  http://myserver.com/v1/api/getcustomers/?page=1

If no more data found for specified page number then

1. server may return an empty response
2. server may return the response with specific error and response status code

JSON Source Connector comes with flexible paging options to handle this scenario. See below screenshot how to configure this.

REST API Looping/Pagination via URL Page Number Parameter (Loop until last page detected)

URL with offset and limit

Sometimes you may have URLs with offset + limit parameters. as below. In this case, you can use the same technique as the above screenshot but just specify Page Num Indicator as offset and Increment by the same as your limit parameter (e.g. 1000 in this case). At runtime, your offset parameter will increment and generate URLs like below to fetch the next page.

Example URLs: 
http://myserver.com/v1/api/getcustomers/?offset=1&limit=1000
http://myserver.com/v1/api/getcustomers/?offset=1001&limit=1000
http://myserver.com/v1/api/getcustomers/?offset=2001&limit=1000

Method-2: REST API Pagination via URL Path

Sometimes your URL may contain a page or other identifier using path format.

Example URLs: 
http://myserver.com/v1/api/getcustomers/1/
http://myserver.com/v1/api/getcustomers/2/
http://myserver.com/data/file1.json
http://myserver.com/data/file2.json

If no more data is found for the specified page number then

1. the server may return a 404 error (page not found)
2. server may return a response with a specific error and response status code

JSON Source Connector comes with flexible paging options to handle this scenario. See the below screenshot of how to configure this.

REST API Looping/Pagination via URL Path (Loop until last page detected)

URL with Skip Parameter (Offset starts at zero)

Now let’s look at another common pattern just like the offset we discussed earlier but in the Skip pattern, it starts at zero rather than offset=1. In the previous mode, we didn’t have the option to supply the initial value to start the counter but if you use Path mode then you can start the counter at some custom value. See the below screen to show how we can setup pagination with the skip method with custom increment.

Example URLs

http://mysite.com/api/getorders?skip=0&pagesize=100
http://mysite.com/api/getorders?skip=100&pagesize=100
http://mysite.com/api/getorders?skip=200&pagesize=100
http://mysite.com/api/getorders?skip=300&pagesize=100
....

REST API Pagination with Custom Skip / Increment Settings (Offset Starts from 0)

Method-3 : REST API Pagination via Response Attribute (Next Link or Cursor)

This method is not discussed in this article but click here to read more

There are mainly 3 ways you can get next link or cursor.

1. Full Link (e.g. nextPage : “https://mysite.com/api/v1/1/?filter=AAAA” )
2. Partial Link  (e.g. nextPage : “/api/v1/getcustomer/1/?filter=AAAA” )
3. Cursor Continuation Token (e.g.   nextCursor : “xxxxAAAbbbbCccc112233==” )

Next Link Method (Partial or Full URL)

Example response with next URL would be

{ 
  "value" : [....], 
  "odata.nextLink" : "http://abc.com/customers/2/" 
}

In this method you don’t specify page number in URL. If response include many records then you will get partial response along with indicator of next link.

For more information about this REST API Pagination please check below links. It explains in depth how to paginate REST API response with next page link found in response.

OData Paging using SSIS – REST API Paging Example

Read Twitter data in SSIS using REST API Task and JSON Source – OAuth2 Protocol

SSIS API Pagination – Next Link from Response Attribute

Next Link – Detect the last page based on empty rows (Regular Expression)

Now let’s look at a little bit different scenario where the Next Link is never empty even on the last page. This can cause an infinite loop if we configure it like the last section. So in such a case (see below example) when the last page still has the next link URL but the data row array is empty then use settings like the screenshot below.

First Page Example

{ 
  "value" : [....], 
  "odata.nextLink" : "http://abc.com/customers/2/" 
}

Last Page Example

{ 
  "value" : [], 
  "odata.nextLink" : "http://abc.com/customers/NNNN/" 
}

Example: Last Page

Basically, configure the following options

• Set Stop indicator attribute to the same node which contains data rows (without array brackets). E.g. $.items
• Set Stop indicator value / Regular Expression to something like

  []

    assuming the last page has the same way. Check Fiddler Trace for exact text on the last page.

Last Page with non empty next link and empty data array (Detect Last Page based on zero row count)

Cursor Method

Many API returns cursor rather than link. Cursor method usually used to support a large number of records. Cursor is opened at server side with initial query information you submit and if cursor has more records you may get next cursor indicator in your response. In cursor method you have to supply Suffix for Next URL so after First URL your next URL will contain extra attribute which will include pagination information.

Suffix for next URL

Suffix for next URL can include special placeholders. There are two placeholders allowed. If you wish to append Next URL value in raw text then use <%nextlink%> but if your API needs value in encoded format e.g. A%20B rather than A+B then you have to use <%nextlink_encoded%>

Example URLs for Cursor:

Below is few sample URLs in the log when pagination occurs via Cursor method. As you see the first URL has Cursor=* means its first call, after first request server may send next cursor in the response body (see example)

{ 
  data: [....], 
  nextCursor: "Aooo/MxxxxxxxxxxxtdA==" 
}

First URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=*
Second URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=Aooo/MxxxxxxxxxxxtdA==
Third URL: http://api.crossref.org/members/311/works?filter=type:journal-article&rows=100&cursor=Booo/MxxxxxxxxxxxtdA==
…..
…..

Method-4: Pass Page Number in Body Attribute

Some API allows you to pass your PageSize and Page number inside Request Body (i.e. POST data) in that case you may configure your options like below (We used Google AdWords Api as an example)

Configure REST API Pagination – using Page Number via POST Body Method

REST API Pagination – Passing PageNumber inside POST Body (Example: Google AdWords SOAP API)

Method-5: REST API Pagination via multiple URL parameters (e.g. Start and End)

In the previous section, you saw how to pass page numbers via URL (Single Parameter: http://abc.com/?page=1). However, sometimes you have a pagination requirement where you have to specify multiple parameters that need increment. See below example URL.

http://abc.com/?startRow=1&endRow=100
http://abc.com/?startRow=101&endRow=200
http://abc.com/?startRow=201&endRow=300
...

Pagination via Multiple URL Parameters

Method-6: REST API Pagination via Header (Next Link in Header)

Another popular pagination method is paginate via Header. In this method, each API response includes special Header which indicates next link for your response. In header method there are two approaches. [1] Custom Method [2] RFC 5988 Method. Lets look at both.

Next Link in Custom Header

In this method you have to specify which response header contains next URL. When no more NEXT url found in response reader stops.

Custom Header Pagination Method

Next Link in Header as per RFC 5988

In this method, HTTP response contains a special header (Link) which contains next link for our response. If no link found to read more data then reader stops. This method is described well here in RFC 5988. Check this GitHub API Example to understand more on this pagination method.

Link Header Pagination Method (RFC 5988)

Method-7: REST API Pagination using Cursor / Continuation Token from Response Header

There will be a time when your API sends some sort of Continuation Token in Response Header and you have to use that token to call next request. Some API needs this token passed as query string in Next URL and in some API you have to pass as request header. We will cover both cases in the following section.

Your API might call these tokens as “Cursor” too so we will use these both terms in the following section (They both mean same thing).

There are few ways these APIs pass Cursor to the next request

1. Pass Cursor via Request Header with same header name
2. Pass Cursor via Request Header with different header name (introduced in v4.2.3)
3. Pass multiple Response Headers via Request Headers (Azure DocumentDB style usecase)
4. Pass Cursor via URL Query String Parameter (introduced in v3.0 or later)

Lets see all approaches.

Pass Token via Request Header (Same header name)

Consider the API pagination use case in which you get some sort of continuation token in the response header and you take it and pass in the next request via request header (same header name).

Example:

First Request / Response:

------- Request ------
GET https://abc.com/api/getdata

------- Response ------
x-token: Nxdfabcd11111

{ some response data }

Second Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-token: Nxdfabcd11111

------- Response ------
x-token: Nxdfabcd22222

{ some response data }

……

……

Last Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-token: Nxdfabcd22222

------- Response ------

{ some response data }

For above pagination pattern  that just enter the response header name in Page Num Indicator field as below

Pagination using response header continuation token – Pass as Request Header

All possible formats for Page Num Indicator is below

x-page-token
x-token1, x-token2
RequestUrlQueryName=ResponseHeaderName
header::RequestHeaderName=ResponseHeaderName
RequestUrlQueryName=ResponseHeaderName(regular-expression)

Pass Token via Request Header (Different header name)

Consider the API pagination use case just like previous one but only different is request header name is not same as response header name. For example rather than x-token in request we need to call it x-next-token.

Example:

First Request / Response:

------- Request ------
GET https://abc.com/api/getdata

------- Response ------
x-token: Nxdfabcd11111

{ some response data }

Second Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-next-token: Nxdfabcd11111

------- Response ------
x-token: Nxdfabcd22222

{ some response data }

……

……

Last Request / Response:

------- Request ------
GET https://abc.com/api/getdata
x-next-token: Nxdfabcd22222

------- Response ------

{ some response data }

For above pagination pattern  that just enter the response header name in Page Num Indicator field as below

Pagination using response header continuation token – Pass as Request Header (Different name)

Pass multiple Response Headers via Request Headers (Azure DocumentDB style usecase)

Some API like Azure Cosmos DB uses Continuation Token via Header approach. In that case you can enter single header name or multiple headers like below (For multiple headers use comma)

REST API Pagination using Continuation Token via Header (Azure CosmosDB / DocumentDB API Example )

You can use use tools like Fiddler to see paginated requests as below.

First Request

View Paginated Requests in Fiddler (Azure CosmosDB API Example)

Second Request

Pass Next Page Token via Header (View Paginated Requests in Fiddler – Azure CosmosDB API Example)

Pass Token via URL Parameter (Query String)

In the new version we added a feature where you can actually read Continuation Token (i.e. Cursor) from Response Header and then pass to next request in the URL rather than Request Header. Here is the example API which uses such method.

First Request

--------- Request ---------
GET
http://test.com/api/getdata

--------- Response ---------
X-RESPONSE-TOKEN : Pg2xxxxxxxxxxxxxxxxx

{ data : [ 1,2,3.... 10 ] }

Second Request 

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg2xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : Pg3xxxxxxxxxxxxxxxxx

{ data : [ 11,12,13.... 20 ] }

Here is how to configure above pagination pattern

Pass Response Header value to URL (Cursor)

Pass Token via URL Parameter (Query String) – with custom Regular expression (Usecase: Salesforce BULK query API)

Now consider previous scenario but with slightly different requirement. By default previous pagination pattern stops when specified response header is missing or response header value is blank. However is some cases header value contains custom string to indicate last page. For example x-token: NULL. In such case we need to use Page Num Indicator like below (must use v4.2.3 or higher in SSIS PowerPack / 1.5.2 or higher in ODBC PowerPack)

nextUrlParameterName=responseHeadername(Some_Regular_Expression)

For example to stop pagination in below case you have to enter Page Num Indicator as below. It says stop pagination when  “NULL” word is detected

cursor=X-RESPONSE-TOKEN(^((?!NULL\b).)*$)

First Request

--------- Request ---------
GET
http://test.com/api/getdata

--------- Response ---------
X-RESPONSE-TOKEN : Pg2xxxxxxxxxxxxxxxxx

{ data : [ 1,2,3.... 10 ] }

Second Request 

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg2xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : Pg3xxxxxxxxxxxxxxxxx

{ data : [ 11,12,13.... 20 ] }

……

……

Last Request 

--------- Request ---------
GET
http://test.com/api/getdata?cursor=Pg3xxxxxxxxxxxxxxxxx

--------- Response ---------
X-RESPONSE-TOKEN : NULL

{ data : [ 21,22,23.... 30 ] }

Here is how to configure above pagination pattern

Pass Response Header value to URL (Cursor)

Method-8: Pass Pagination Token in Request Body

In previous section (Method-4: Pass Page Number in Body Attribute ) we saw how to pass simple numeric page counter in Body. This is fine for some API but if your API has base64 style text token then counter based method doesn’t work. Also if you have requirement to hide page attribute node in first request and send page counter node in second request body (Changing Request Body Pattern for First vs Next)  then check next two sections.

Next two sections covers two API Pagination patterns to fetch 100 rows (1000 in each page) . Both requires to send token in Body but second section is more complex need (Different Body Attributes in first vs next requests )

Passing Page Token in Body (Blank Token in first request allowed)

Lets look at simple pattern where first request has blank token in Body but all other requests until we find last page we keep sending non-empty token. This approach works as long as API Provider doesn’t mind having Blank Token in first request. If you get error then check next section to handle more complex way.

Here is the example of API which needs token in body

First API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"" }

--------- Response ---------
{recordsFound: 5000, data: ["Acct1", "Acct2"..., "Acct1000"], nextToken:"AxBxCx112233001==" }

Second API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"AxBxCx112233001==" }

--------- Response ---------
{data: ["Acct1001", "Acct1002"..., "Acct2000"], nextToken:"AxBxCx112233002==" }

….. many more calls …

…. many more calls ….

Last API call (no more nextToken in response so STOP pagination)

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account", continueToken:"AxBxCx112233009==" }

--------- Response ---------
{data: ["Acct9001", "Acct9002"..., "Acct10000"] }

So if you have such pattern then you can set few properties

For such issue you can use Changing few properties like below. Use Property Grid if you cant find on UI

• Set EnablePageTokenForBody  =

  True

• Set PagePlaceholders =

  header=|;filter=|;

• Set HttpRequestData = 

  { searchFor: "Account", continueToken:"[$pagetoken$]" }

Setting Body with [$pagetoken$] to implement dynamic pagination

Setting Page Placeholders in Advanced Pagination tab.

Passing Page Token in Body (Blank Token in first request)

Now let’s look at more complex pattern. Here our first request body is different than other requests.

First API call

--------- Request ---------
POST http://test.com/api/getdata

{ searchFor: "Account" }

--------- Response ---------
{recordsFound: 5000, data: ["Acct1", "Acct2"..., "Acct1000"], nextToken:"AxBxCx112233001==" }

Second API call

--------- Request ---------
POST http://test.com/api/getdataNext

{ continueToken:"AxBxCx112233001==" }

--------- Response ---------
{data: ["Acct1001", "Acct1002"..., "Acct2000"], nextToken:"AxBxCx112233002==" }

….. many more calls …

…. many more calls ….

Last API call (no more nextToken)

--------- Request ---------
POST http://test.com/api/getdataNext

{ continueToken:"AxBxCx112233009==" }

--------- Response ---------
{data: ["Acct9001", "Acct9002"..., "Acct10000"] }

So notice few things

• First Request Body / URL is different than second / third and Nth API call
• For first request we do not have token attribute but after first request we do have token attribute

We have this article and this article which describes similar pattern.

Now lets look at how to solve both problems

Passing PageToken or Counter in Next API Call Body

For such issue you can use Changing few properties like below. Use Property Grid if you cant find on UI

• Set EnablePageTokenForBody  =

  True

• Set PagePlaceholders =

  header=|;filter=|;

• Set HttpRequestData = 

  [$tag$]

• Set HasDifferentNextPageInfo = 

  True

• Set FirstPageBodyPart = 

  { searchFor: "Account"}

• Set NextPageBodyPart = 

  { token:"[$pagetoken$]" }

So notice how we defined Body in two different properies as template. And in HttpRequestData we used

[$tag$]

Output Page Number as column

Sometimes you have use case where you want to Start Page Number from the last page loaded successfully. By default, you will see one additional column __FileName in the output. This column contains PageNumber  from URL for each row. You can save this into database so when next time you run your SSIS you can start at MAX(PageNumber) + 1.

Outputting PageNumber option is not visible by default so Use Property Grid and look for OutputFileName property (Default=true)

Last Page Detection

Every API may have different way of telling you that you reached last page or no more data found so you should stop looping. SSIS PowerPack comes with many options so you can implement desired logic for last page detection.

Following approaches supported by last page detection. Below options are not applicable if you are using Method#3 (Response contains Next Link / Cursor)

1. Detect the last page by row count
2. Detect the last page by response status code
3. Detect the last page by error message (substring search)
4. Detect the last page by response size
5. Detect the last page based on response value or regular expression pattern

Now lets look at each setting. You should choose correct one based on your API behavior

Detect the last page by row count

If your API returns blank document or blank Array (if its JSON) then you can use this method to detect the last page. So when no content found in Response document it will stop pagination.

See below example of JSON API .. It shows sample first page and Sample last page (Assuming you entered Filter as $.Records[*] to extract data from Records array.

First Page

{ 
  Records : [ {...}, {...}, {...} ......... ]
}

Last Page

{ 
  Records : [  ]
}

SSIS API Pagination – Detect the last page by row count (Blank document or array)

Detect the last page by response status code

Sometimes you get error with status code when the last page is reached (Most common is 404). So rather than failure, you can treat that status code as last page indicator using this option (see below).

SSIS API Pagination – Detect last page by response status code

Detect the last page by the error message

If you cant use above two methods and want to detect the last page based on a certain error string in the response then use below approach.

SSIS API Pagination – Detect the last page by error/response string search

Detect the last page by response size

Sometimes you can detect the last page by response size. For example, your response contains “–” only if no more data is found then you cannot use any of the above approaches and you have to detect based on response size (e.g. less than 3 bytes). See below

SSIS API Pagination – Detect last page by response size

Detect the last page based on the response attribute value

There will be a time when you want to detect the last page based on some indicator value from your response. In that case, you can use the following settings to detect the last page.

• Select Paging mode =Response Attribute Mode
• Select Next Link Attribute (Example: $.next_page )
• Select Stop Indicator Attribute (Example: $.hasMorePages )
• Enter a value for Stop Indicator (Example: false )

If you don’t want to use Hardcoded value for stop indicator and want to use expression then see next section.

Detect the last page based on regular expression pattern

There will be a time when you want to detect the last page based on some indicator value from your response. In that case, you can use the following settings to detect the last page.

• Select Paging mode =Response Attribute Mode
• Select Next Link Attribute (Example: $.next_page )
• Select Stop Indicator Attribute (Example: $.count )
• Enter regex pattern (regular expression) for Stop Indicator (Example: regex=^\d{1,3}$ ). Stop when count attribute is less than 1000 in other word less than 4 digits.

Here is an example from Zendesk API (Stop pagination when count attribute from the response is less than 1000)

Pagination for Zendesk Incremental API – Next Link and Last Page detection using Stop Indicator Regular Expression

Handling API Rate Limit - Retry on too many API requests

Many API restricts you from calling too many API calls too soon (just to save their servers from being overwhelmed) - Example. Let's think this way if you fetching a million rows via paginated response and the very last page fails then you won't be happy because now you have to fetch everything again. To solve this ZappySys provides retry options on HTTP / OAuth and REST API connection managers.

Generally, when you exceed the rate limit, the server sends you error code 429 but it can be something else too or multiple codes are returned. In the below example, we use 503. Check your API documentation to know the exact code you should retry.

To set the retry options go to connection settings (e.g. HTTP / OAuth / REST connection). The below example is for OAuth connection.

To retry on multiple codes use Pipe as a delimiter. Example: 429|503

Retry Settings for API Rate Limit Exceeded Error (Status code 429 or other)

Performance Tips

You can improve performance of your API calls by doing a few changes (again not every API may support this)

• Use the maximum page size possible (e.g. if you doing pagination you can set max possible page size – refer to your API documentation whats the default size and what is the max page size you can supply)
  e.g.  /myapi?pagesize=100  (parameter name can be something else like limit=100)

SSIS Infinite Loop

So now you know how to paginate automatically in JSON Source Connector but what if you want to use REST API Task for pagination? Some custom scenarios like web scraping require you to download all HTML pages until no more results found? In that case, you most likely need infinite looping pattern (Loop until the last page detected). To loop through pages until the last page is reached you can perform the following logic.

The last page can be detected based on any of the following ways
1. Check Response StatusCode
2. Check Error Message
3. Check Response Size (If less than X bytes then consider as the last page)

REST API Task support #1 and #2 scenarios. If you want to check bytes then use the expression in Expression Task or Script Task
(e.g. @exitloop = LEN(@responseData) <= 3 )

URL for Rest API Task (On Request Settings Tab)
When you configure REST API Task enter URL as below so its dynamic. You can use PageCounter variable anywhere in the URL.

//zappysys.com/downloads/files/test/cust-{{User::pagecounter}}.json

SSIS REST API Looping – Pass Page number in URL, SSIS Infinite Loop

Configure For Loop Task for Infinite Looping (Loop until last page found)

Download Sample SSIS Package

SSIS-Loop-Rest-API-2012.dtsx (Zip file)

Video Tutorial – Reading data from API in SSIS

In case you missed our previous article about calling REST API is SSIS then check below video.

Conclusion

It can be time consuming and steep learning curve to write C#/JAVA code to handle your rest api integration but using some Tasks and Connectors listed below can help you to solve big headache and provide clean drag and drop REST API integration in SSIS for virtually any REST API Source (e.g. Twitter, Facebook, Salesforce, office 365). Download SSIS PowerPack and try it out by yourself.

SSIS Components for REST API/JSON/XML/SOAP

	JSON Source Connector (File, REST API)
	Export JSON File Task
	JSON Parser Task
	XML Source Connector (File, SOAP, REST API)

The post How to do REST API Pagination in SSIS / ODBC Drivers appeared first on ZappySys Blog.

In this article you will learn how to automate/integrate MailChimp API calls inside SSIS without any coding efforts. Historically anytime you want to access API or Automate workflow you have to use SDKs or write custom script which can be very time consuming and expensive effort. If you are looking for drag and drop approach rather than coding for your MailChimp API integration then this article is perfect for your need.

To call MailChimp REST API inside SSIS we you will need to download SSIS PowerPack. Following SSIS task/components can help you to call any API and parse JSON response into table format (i.e. Rows and Columns) which you can write to any Destination later (i.e. Load MailChimp data into SQL Server or other way around).

	JSON Source Connector (File, REST API)
	REST API Webservice Task

Calling MailChimp API two different ways in SSIS

MailChimp API offers two different ways to authenticate your API requests.

1. Use OAuth 2.0 Authorization
2. Use Basic Authorization (Use API Key)

Method-1 : Use OAuth 2.0 Authentication – Call MailChimp API using SSIS REST API Task

This section will show you how to call MailChimp API using OAuth 2.0 protocol. Simplest way to test our MailChimp API call is use REST API Webservice Task.

1. Download and Install SSIS PowerPack (Click here to download). It includes 40+ components which can help you with any JSON or API Integration scenario inside SSIS
2. Open / Create SSIS Project
3. Drag ZS REST API Task from SSIS Toolbox
4. Double click REST API Task to configure it
5. Set Url Access mode to [Url from Connection]
6. From connection dropdown select New OAuth Connection
7. Select Provider = MailChimp
8. Click Generate Token and once done Click Test and OK to save UI
   

   SSIS MailChimp OAuth Provider – OAuth connection manager

9. Once you are back to REST API Task enter following partial URL in URL box. Notice that we have not supplied full URL because depending on which region you are MailChimp may be using different URL for API access (e.g. https://us14.api.mailchimp.com/3.0/campaigns ). If you don’t specify full URL then OAuth connection manager automatically determines correct Data center for you and prepares correct URL at runtime.
   

   /3.0/campaigns

10. Click Test Request – Scroll to see Content (you should see JSON like below)
    

    Call MailChimp API using SSIS REST API Task (Use OAuth 2.0 Connection Manager)

Method-2 : Use Basic Authentication – Call MailChimp API using SSIS REST API Task

Simplest way to test our MailChimp API call is use REST API Webservice Task.

1. Download and Install SSIS PowerPack (Click here to download). It includes 40+ components which can help you with any JSON or API Integration scenario inside SSIS
2. Open / Create SSIS Project
3. Drag ZS REST API Task from SSIS Toolbox
4. Double click REST API Task to configure it
5. Set Url Access mode to [Url from Connection]
6. From connection dropdown select New HTTP Connection
7. Enter any valid URL e.g. https://us14.api.mailchimp.com/3.0/campaigns (If you are not sure what is your DC then try to call https://login.mailchimp.com/oauth2/metadata URL first and get response via REST API Task – Test to see api_endpoint URL)
   

   https://us14.api.mailchimp.com/3.0/campaigns

8. Click use credentials. Click Use userid/password option
9. Enter any fake userid (this is ignored) and Valid API Key in Password field. If you are not sure how to get Mailchimp API Key then check this article
10. Click OK to save HTTP connection
11. Click Test Request – Scroll to see Content (you should see JSON like below)

Get data from MailChimp API and load into SQL Server

In previous section we saw how to use REST API Task to call any MailChimp API. It can do POST / PUT / DELETE try of requests. Another scenario could be read data from MailChimp and load into SQL Server (i.e. your local Data warehouse). For this purpose you can use JSON Source. It can parse JSON response into Rows and columns. Also it supports pagination so you can loop through response.

1. Download and Install SSIS PowerPack (Click here to download). It includes 40+ components which can help you with any JSON or API Integration scenario inside SSIS
2. Open / Create SSIS Project
3. Drag and drop Data flow task in SSIS designer from SSIS Toolbox
   

   Dragging and dropping Data Flow Task into Control Flow

4. Double click Data flow to open Data flow designer
5. From SSIS toolbox find ZS JSON Source and drag on the surface
6. Double click to configure JSON source
7. Set Access mode to [Path or URL from Connection]
8. Before entering URL check option [Use credentials] and Select existing OAuth connection manager created in Previous section (i.e. Method-1 : Use OAuth)
9. Now we have to enter API URL. First determine endpoint assigned to your account. For that enter https://login.mailchimp.com/oauth2/metadata and click preview. Copy url from api_endpoint column found on the preview window. Once copied click OK to close
10. Now Use API endpoint URL (copied from previous step) and append version number (i.e. /3.0) and API (i.e. /campaigns) you like to call. Your final URL may be something like below. Our DC was us14 yours can be different.
    

    https://us14.api.mailchimp.com/3.0/campaigns

11. Once you enter correct API URL click on Select Filter button to select correct hierarchy you want to extract. If API response contains records inside JSON array then select that as your filter. For example if you getting list of all campaigns then your Filter may be like $.campaigns[*]
12. Once filter is specified, click Preview.
13. Click on columns tab to review columns and metadata. You can change default length or datatype. If you change default metadata then make sure to check Lock option (last column – right side) so it stays there when metadata refresh occurs
14. Click OK to save UI
15. Now you can connect your JSON source to any destination such as SQL Server (Use OLEDB or ADO.net Destination component)
16. Execute SSIS Package to see data in destination

Load MailChimp data into SQL Server – Use SSIS JSON Source to call REST API

MailChimp REST API Pagination

If you have many records then most likely your response will be partial. To fetch all records from MailChimp you have to use Pagination technique. Read this article for more information

Page number is indicated by offset query string parameter in URL as below. 0 mean first page. When pagination method is used on JSON source it will automatically increment offset and fetch data from next URL until no more records found for selected expression in JSON (e.g. $.campaigns[*] )

MailChimp API Pagination using SSIS JSON Source

Conclusion

As you saw how easy it is to consume MailChimp API using SSIS without any coding. SSIS PowerPack is one of the finest toolset for virtually any API or JSON integration project. Click here to Try SSIS PowerPack for FREE.

The post How to get data from MailChimp in SSIS with REST API appeared first on ZappySys Blog.

Zendesk is one of the most popular customer service platforms available today. Zendesk offers a REST API to interact with its cloud-based service programmatically (to manage or read data). The problem is that not everyone is a programmer, and not everyone has time to learn how to write C# or JAVA code to access a REST API.

In this article, you will learn how to get Zendesk data using the SSIS JSON Source (REST API Connector). This connector allows you to read Zendesk data in SSIS using REST API Calls without any scripting/programming. Techniques mentioned in this article can also be used to consume data from other services (e.g., read data from Twitter). In this article, you will see step-by-step instructions to read Zendesk ticket information and load it into on-premises SQL Server.

Also, check our other article on how to call the Zendesk REST API to create a new ticket.

Different methods for Zendesk REST API Access

Zendesk allows you to access its data in three different ways.

1. REST API access using OAuth Access Token
2. REST API access using API Token
3. REST API access using Basic Authentication (Use userid (e.g., email) and Password)

Each method has pros and cons. The simplest method is the third, which is easy to use but requires passing your account password with each call. The second method is also easy to use but requires an additional step to create an access token. The first method is the most secure, but it has a few extra steps compared to the other methods.

Prerequisite

Before you can finish the tasks explained in this article, you have to finish the following steps. This article assumes you have basic knowledge of SSIS (SQL Server Integration Services)

1. Make sure you have the SSIS designer installed. Sometimes it’s referred to as BIDS or SSDT (Get from here)
2. Download SSIS PowerPack (FREE Trial – 30 days).
3. Optional – Another very useful Free tool is Fiddler. You can use it to see raw request/response (Check this article)

Method-1: Zendesk API Access using OAuth in SSIS

To read about the OAuth Method, click here

Method-2: Zendesk API Access using Token in SSIS

Now let’s look at the second method for accessing Zendesk data using SSIS.

Create Token in Zendesk Portal for REST API Access

1. Log in to your account https://{{your-subdomain}}.zendesk.com/login and then click on the settings icon
2. On the next page, click on the API link
3. On the API page, scroll to the API Access Tokens section. Make sure the Token Access method is checked.
4. Click on the ” add new token hyperlink. Once prompted, give a name to the Token token (this is just a label)
5. Once the above steps are done, a token should be created and listed under your API Access page. The token will be used as your Password to access the REST API (will be shown in the next section). For more information, check this link >>>  How to generate a token for ZenDesk API Call.
   

   Create a new Zendesk REST API Access Token

Configure SSIS HTTP Connection

Perform the following steps to extract data from Zendesk and load it into SQL Server. The steps below assume that you are using the API Token method (#2 from the supported methods to call the Zendesk API). This is better than using Userid and Password.

1. Open a new SSIS Project.
2. First of all, drag and drop the Data Flow Task from the SSIS Toolbox and double-click it to edit.
   

   Drag and drop Data flow.w

3. From the SSIS toolbox, drag and drop the ZS JSON Source onto the data flow designer surface, then double-click it to edit.
   

   SSIS JSON Source – Drag and Drop
4. From the AccessMode dropdown [Url From Connection]
5. Assume you want to extract all tickets from Zendesk, then enter the following URL (in the screenshot, we appended?page=1, which is optional)
   

   https://YOUR_SUB_DOMAIN.zendesk.com/api/v2/tickets.json

6. Check the Use Credentials option and select New ZS-HTTP connection. When prompted
   For Credential Type: select Basic Authentication  (Use of UserID and Token ).
   For Web URL: Enter any URL (This will be ignored anyway because the URL entered on the JSON Source UI will be used)
   For user name : enter YOUR_EMAIL/token  (e.g. bob@mycompany.com/token)
   For Password: enteToken token obtained from the Admin console > API section. For more information, check this link >>>  How to generate a token for ZenDesk API Call.
   
   NOTE: Appending /token command in Userid will allow us to pass the token rather than the Password of that account. This approach is more secure because you can disable the Token at any time without deactivating the user account. If you want to access data using your userid and Password rather than a token, then do not use /token after your email. By default, uToken Token and the use of Password are not allowed unless you enable these methods from the Admin console (See above section)
   

   Configure SSIS HTTP Connection – Use Zendesk Token to Call REST API

Adding Retry Settings for HTTP or OAuth connection

Zendesk enforces an API rate limit, which means you cannot issue too many API calls within a given time frame. Check this document for exact information.

ZappySys provides API retry settings on both the OAuth Connection and the HTTP Connection. For Zendesk, you can retry on a specific status code 429

1. Open connection manager UI
2. Go to the Retry-Settings page
3. Select Retry web error on matching response status code.
4. Enter 429 for the status code
5. Check multiple wait times
6. In the wait time, enter 5000 (wait 5 seconds or multiple of 5 seconds)
7. Set the max retry count to 6. See the example (It’s not for Zendesk, but will give you an idea)
   

   OAuth connection – Retry Options

Configure Zendesk API Pagination and Filter for SSIS JSON Source

1. Now click [Select Filter], select the tickets node (see the array icon), and click OK. It will generate the following Filter Expression to extract all records found under the tickets array node (e.g., tickets : [ {…} , {…} … ] )
   

   $.tickets[*]

   

   Select Filter – Extract Data from Zendesk API Response (Read Ticket)

2. N, let’sts configure pagination. By default, Zendesk API doesn’t return all records, so you have to specify pagination as below. Click on the Pagination Tab. Enter $.next_page for Next URL attribute as below (you can also Browse Path)
   

   SSIS JSON Source Zendesk Pagination

3. Now time to click Preview. If you followed everything correctly, then when you click the Preview Button, you should see some data in the preview (assuming you have at least one ticket).
   

   SSIS JSON Source – Data Preview – Zendesk REST API Call to extract tickets

4. Now click the Columns tab to review data types (you can change the default length and check the Lock option on the right side). This change is needed because the datatype guess happens based on the design-time setting (only 300 rows scanned by default). At runtime, if you get a truncation-related error, then copy the error message into Notepad to get the exact steps and column name that caused the error. If you need a length greater than 4000 chars, use the DT_NTEXT data type instead (change the datatype and click the column name cell on the right to save the change).
5. Click OK to save UI

How to extract Zendesk data using Incremental API

There will be a time when you need to extract data incrementally (i.e., Extract only the changed data since the last load). Zendesk API supports many Incremental API endpoints. If you are using this endpoint, you have to make sure of a few things.

• For the OAuth connection method, make sure the

  read

    scope is included (

  tickets:read

    won’t work, so just use

  read

   )
• Your URL must include /incremental path before the API type, and you must include start_time for the extract in Unix Epoch Format.  The following example will pull records modified after
  

  https://zappysys.zendesk.com/api/v2/incremental/tickets.json?start_time=1514764800

• On the pagination tab, you have to configure the following settings
  • Set Next Link as

    $.next_link

  • Set the Stop Indicator attribute as

    $.count

  • Set the Stop Indicator Value as

    regex=^\d{1,3}$

      Notice that using regex= prefix treats your end indicator value as a pattern check.—OR—
  • Set the Stop Indicator attribute as

    $.end_of_stream

       and the Stop indicator value as

    true

    

    Pagination for Zendesk Incremental API – Next Link and Last Page detection using Stop Indicator Regular Expression

Here is a simple SQL query to convert date/time to Unix time

select DATEDIFF(SECOND,'1970-01-01', GETUTCDATE() )

select DATEDIFF(SECOND,'1970-01-01', LastExtractDate ) as LastExtractDateAsUnix From dbo.ExtractStatus Where ExtractName='ZendeskExtract'

For C# code in the SSIS Script Task, you can use something like the following:

Int32 unixTimestamp = (Int32)(DateTime.UtcNow.Subtract(new DateTime(1970, 1, 1))).TotalSeconds;

Assuming you extracted Date/Time from the database and saved it to some SSIS Variable called User::varLastExtract. Now we like to convert that to Unix format using C#.

We assume you have created the following variables in the SSIS Package. Also, you have set varLastExtract to a date/time value you want to convert.

User::varLastExtractTimeUnix  (Int32)
User::varLastExtractTime (DateTime)

Use the steps below. to convert date time to Unix format in C#

1. Drag and drop Script Task in SSIS designer
2. Add both variable names in Read/Write Variable List (e.g., User::varLastExtractTimeUnix, User::varLastTimeExtract ).
3. Edit Script
4. Enter something like below
   

   DateTime dtUtc= Convert.ToDateTime( Dts.Variables["User::varLastExtractTime"].Value ).ToUniversalTime();
   Dts.Variables["User::varLastExtractTimeUnix"].Value = (Int32)(dtUtc.Subtract(new DateTime(1970, 1, 1))).TotalSeconds;

Now you can use this variable in your JSON Source URL to make things dynamic.

https://zappysys.zendesk.com/api/v2/incremental/tickets.json?start_time={{User::varLastExtractTimeUnix}}

Handling duplicate data during incremental load (DISTINCT)

As per this Zendesk document, they may include the last row of the previous page on the next page if the update timestamp. It is possible to have duplicate rows in your staging table. You can easily avoid this by using the DISTINCT keyword or another SQL query to remove duplicates. Make sure not to include P_next_page and P_count, as they can differ when duplicate rows are detected at page boundaries.

SELECT DISTINCT 
--[P_next_page] --- Do NOT include this... 
[P_previous_page]
--,[P_count] --- Do NOT include this... 
,[url]
,[id]
,[external_id]
,[via.channel]
,[via.source.rel]

,[assignee_id]
......
......
......
,[allow_attachments]
FROM [dbo]._stgZendeskTickets

Loading data from Zendesk to SQL Server using SSIS

1. Now, drag a new OLE DB Destination into the Data Flow Designer. Connect the JSON source with the destination. Now double-click Destination. At the destination, select connection manager (or create one), then click [New] next to the Table name dropdown. You will see default metadata for the table based on upstream columns. Notice how P_previous_link is 40 chars rather than 140. This can be edited in the source metadata under the Columns tab, as described in the previous step, so that Table datatypes are set correctly. Once you change the table name and datatypes, click OK to create the new table. Click on the Mapping Tab to map source columns to target columns. Once that’s done, click OK to save.
   

   Load Zendesk Data into SQL Server Using SSIS JSON Connector (REST API)

2. Now execute the package. If you have multiple pages in response, you will notice several requests in the log (e.g., &page=1 … &page=2 …).
   
   NOTE: In below screenshot we used per_page=2 for example purpose only. In reality, make it bigger, e.g., per_page=100 or per_page=200, so you don’t hit the server too often. If you’re not sure, then exclude the per_page parameter from the URL to use the default setting.
   

   Execute SSIS Package – Extract data from Zendesk and load into SQL Server

Method-3: Zendesk API Access using Username and Password in SSIS (Basic Authentication)

This method is the same as the second method outlined in the previous section. Except for a few changes. When you create an HTTP Connection, use your Username (without appenTokentoken ). For the Password, use your account password (the same one you use to log in to the Zendesk portal)

Zendesk API Rate Limit (Throttling)

Zendesk may limit how many calls you can make per second, so make sure to slow down if you get errors about api rate limit. You can slow down by entering Wait time on the Throttling tab of JSON Source (100 means wait 100 ms after each Request … only useful if pagination is enabled)

Making things Dynamic (e.g., URL)

You can use SSIS variables in your URL to make it dynamic. See below

https://mycompany.zendesk.com/api/v2/items/{{User::TicketNumber}}.json

Dealing with common errors

For detailed instructions to fix common metadata-related errors, read this article.

SSIS PowerPack – Column Metadata Editor (i.e., Change DataType, Column Length)

Deploy to production

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Conclusion

In this post, you saw how easy it is to read data from Zendesk using the SSIS JSON Source (REST API Connector). Click here to download and try it yourself.

Keywords

Loading Zendesk data into SQL Server
How to read Zendesk tickets in SSIS
Calling Zendesk REST API in SSIS
Fetch Zendesk data using a REST API call in SSIS
Extract Zendesk data using a REST API call in SSIS
Reading data from Zendesk using rest api call
Pulling Zendesk tickets using rest api call

The post How to read Zendesk data in SSIS with REST API appeared first on ZappySys Blog.