In this post we will learn how to use ZappySys SSIS XML Source or ZappySys SSIS JSON Source  to read large XML or JSON File (Process 3 Million rows in 3 minutes – 1.2 GB file).

If you use default settings to read data then it may result into OutOfMemory Exception so we will outline few techniques which will enable high performance Streaming Mode rather than In-memory load of entire file.

Prerequisites

Before we parse very large XML or JSON files using SSIS , you will need to make sure following prerequisites are met.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. You have at least two sample files… One file must be small dataset (less than 10 MB if possible). We will  use small dataset file during design mode to get metadata and see data preview on the Component UI.
4. Second XML file is the big file with full dataset you like to parse at runtime.
5. Make sure SSIS PowerPack is installed. Click here to download.

Step-By-Step : Reading large XML file (SSIS XML Source)

Now let’s look at how to read large XML file (e.g. 3 Million rows or more) using ZappySys XML Source in SSIS. Below steps are identical for  ZappySys JSON Source  too (Except Step#7).

1. Open SSIS Designer and drag Data Flow from SSIS Toolbox
2. Double click Data Flow Task to switch to Data flow designer
3. Now drag ZS XML Source on the surface from SSIS Toolbox.
4. Double click ZS XML Source and specify

   small dataset file

     path you like to parse. (e.g.  c:\data\customer_small.xml )
5. Click on Select Filter button to find Node which will be treated as Array. Once you close the Filter Browse Dialog. Append

   --FAST

     Your Expression may look like below.
   

   $.Root.Row--FAST

6. Now uncheck  

   Include Parent Columns option

7. Check 

   Enable Performance Option

     (For JSON Source Skip this step)
   NOTE: Try to enable Performance mode setting after you Select Filter (by using smaller dataset file). Once filter is set you can check enable performance mode and make sure following two settings are correctly set (Option#2 needs new version – SSIS v4.1+ / ODBC v1.4+).
   (1) On Array Handling Tab – Set Array node name (must be only one entry). For example your Filter is $.DATA.ORDER[*] then you can enter

   ORDER

   (2) On Advanced Filter Options Tab – Enter all unwanted Tag names you like to Skip. For example if you have other arrays like PRODUCT or CUSTOMER then enter

   PRODUCT,CUSTOMER

   
   
8. Your setting may look like below
   
   
   

   Configure XML source or JSON Source for Very Large Data File (Streaming mode for High Performance)

9. Click Preview to verify data (Adjust Filter if needed to extract correct Hierarchy)
   

   Preview XML or JSON File data using SSIS XML Source or JSON Source

10. Click on Columns Tab
11. Change Scan Row count to 3000 or more and Click on

    Refresh Column

     .
12. Select Guess 4x , Check Lock, Check Reset and Click OK like below. At runtime if you ever get error about Data Type Issue you can always adjust this later on too. Make sure Lock column is set to avoid setting reset for manually changed columns. For more information on metadata changes check this article
    

    SSIS Metadata Options – JSON / XML File Parsing

13. Click OK to save settings.
14. Now Right click on XML Component > Click Properties. Change

    DirectPath

      property to original file path (large file) (e.g.

    c:\data\customers_large.xml

      ). Save Package.
15. Now you can run your SSIS Package from Designer or Command line. As you see in the below screenshot that there is virtually no memory pressure when you enable stream mode. Thanks to ZappySys unique XML / JSON Parsing Engine. When streaming mode is enabled file is not loaded into memory for parsing rather than that it only reads record by record to process very large JSON or XML file.In our below example we used Windows 7 Desktop, 16GB RAM, 4 Core i7 64 bit CPU. It took around

    3 Minutes to Parse 3 Million Records (1.2 GB big XML file)

     . If you Parse JSON file then it can be even faster due to compact size.
    

    Reading Very Large XML or JSON File using SSIS (Stream Mode for High Performance)

Step-By-Step : Reading very large JSON file (SSIS JSON Source)

Reading very large JSON file using ZappySys JSON Source  has exact same steps described in above section except two changes. You have to use ZS JSON Source and skip Step#7 (Check Enable Performance Mode – This option is not available JSON Source).

Parsing very large XML File with Multiple Arrays

Now let’s discuss a scenario which can result in OutOfMemory Exception unless you tweak some extra options. Assume you have file structure like below.
NOTE: This will only work in version 4.1.0 or later (in SSIS PowerPack) or 1.4.0 or later for ODBC PowerPack

<?xml version="1.0" encoding="UTF-8"?>
<Data>
	<Product>....<Product>
	<Product>....<Product>
	.... many more....
	<Product>....<Product>
	
	<Customer>....<Customer>
	<Customer>....<Customer>
	.... many more....
	<Customer>....<Customer>
	
	<Row>....<Row>
	<Row>....<Row>
	.... many more....
	<Row>....<Row>
</Data>

Notice that in above XML it has 3 different nodes (For Product, Customer and Order). If you try to extract Orders it might fail with OutOfMemory Exception because it needs to scan large XML before it can hit First Order node. To solve this issue you can adjust following 2 settings.

1. On Array Handling Tab you have following
   

   Row

2. On Advanced Filter Options tab enter Following two nodes which we dont want to extract.
   

   Product,Customer

Thats it. This will avoid excessive Memory pressure to find very first node before it can start Stream.

Parse XML Array – Performance Mode Setting

Parse Large XML – Exclude nodes by name

Conclusion

As you saw in this article that ZappySys SSIS PowerPack is designed to handle very large dataset in JSON or XML. We also support very large CSV and Excel files too which are not covered in this article. Download SSIS PowerPack to explore 70+ more components by yourself to make your ETL simple and Fast.

The post How to read large XML / JSON file in SSIS (3 Million Rows in 3 Mins) appeared first on ZappySys Blog.

To consume Zendesk data using the REST API, you can use different methods. Now, let’s see how to create an OAuth Application so you don’t have to rely on a static Token or a UserID/Password to access data.

The OAuth method is more secure and recommended over other methods, so whenever possible, use it.

Create Zendesk OAuth Application

To create an OAuth App for Zendesk, perform the following steps.

1. Log in to your Zendesk Portal. And go tothe  following URL (This step assumes you have admin rights to createan  OAuth App)
   

   https://zappysys.zendesk.com/agent/admin/api/oauth_clients

   

   How to create a Zendesk OAuth Application for REST API Access

2. After clicking on the plus sign and entering app information (e.g., Display name, Unique ID, also known as ClientId or AppId )
   

   Configure Zendesk OAuth App

3. All the way at the bottom of that page, you will see Secret (i.e., Client Secret). Save that along with a unique identifier for later use.
4. That’s it. Now you can start using the Zendesk OAuth App to access the REST API

Understanding Scopes

In the next section, you will learn how to configure an OAuth connection. During that, you will need to specify scopes (Token Permissions you want to grant).

The syntax to define the scope is as follows. Scope can be read-only, write-only, or read-write. The resource is an optional part. If you omit the scope, it will be applied to all.

resource:scope

You can use the following resource scope for the following resources:

read
write
read write
users:read
users:read users:write
users:read users:write organizations:write

Examples of scopes for Zendesk Chat APIs

read
chat

Calling Zendesk REST API using SSIS PowerPack OAuth Connection

Here is how you can use Zerdesk OAuth to access the API using the SSIS PowerPack JSON / REST Source

1. First, drag the ZS JSON Source from the SSIS Toolbox into the Data Flow designer
2. Configure JSON Source as below. Make sure you update the URL to your domain.
   URL for Zendesk Tickets API
   

   https://{YOUR-DOMAIN}.zendesk.com/api/v2/tickets

   URL for Zendesk Chat API
   

   https://{YOUR-DOMAIN}.zopim.com/api/v2/chats

3. Configure the OAuth Connection As below for the Zendesk Ticket API. Scopes differ for the Ticket and Chat APIs; please enter the appropriate ones.
   

   Configure SSIS OAuth connection for Zendesk API Access

   

   Configure OAuth Redirect URL and Content Type

4. Configure the OAuth Connection As below for the Zendesk Chat API. Scopes differ for the Ticket and Chat APIs; please enter the appropriate ones.
   

   Zendesk Chat OAuth Settings – General Tab

   

   Zendesk Chat OAuth Settings – Advanced Tab

   

   Zendesk Chat OAuth Settings – OAuth2 Grant Options Tab

5. After you set the necessary parameters, click Generate Token (Login using your account with the required permission needed to access data)
   As of now, Zendesk doesn’t support Refresh Token. So once you click Generate Token, it will only create an Access Token (which doesn’t expire until you revoke it explicitly). The Refresh Token will be blank, as expected.
   OAuth 2.0 Flow (Google API Example)
   

   Connect to Google API using OAuth 2.0 (YouTube API Example)

6. Configure Pagination parameters as below
   

   SSIS JSON Source –
   Configure Zendesk REST API Pagination

7. Select Filter
   

   Select Filter – Extract Data from Zendesk API Response (Read Tickets)

8. Click Preview to test
   

   SSIS JSON Source – Data Preview – Zendesk REST API Call to extract tickets

Adding Retry Settings for HTTP or OAuth connection

Zendesk enforces an API rate limit, which means you cannot issue too many API calls in a given time frame. Check this document for exact information.

ZappySys provides API retry settings on both the OAuth Connection and the HTTP Connection. For Zendesk you can retry on a specific status code 429

1. Open connection manager UI
2. Go to Retry-Settings page
3. Select Retry web error on matching response status code.
4. Enter 429 for status code
5. Check multiply wait time
6. In the wait time enter 5000 (wait 5 seconds or multiple of 5 seconds)
7. Set Max retry count as 6

See below example (Its not for Zendesk but will give you an idea)

Retry Options

Configure Zendesk API Pagination and Filter for SSIS JSON Source

The post How to configure Zendesk OAuth Application and Connection for REST API appeared first on ZappySys Blog.

In this post you will learn how to access Microsoft Graph API  (Office 365 REST API / Sharepoint API) inside SSIS without any coding. Microsoft Graph API is a unified way to access many Microsoft services API including Office 365 API. In this post we will use SSIS JSON Source / REST API Connector to load data from Office 365 to SQL Server.

Use Case of Microsoft Graph API

Here are some use cases why you want to use Microsoft Graph API

• Read/Write events from Outlook Calendar
• Send email  / Read emails
• Get list of files from OneDrive
• Upload/ Download files to OneDrive
• Read/Write Excel Sheet (Use range or specific range of specific sheet)
• Read Sharepoint Document library
• Search content from Sharepoint document library
• Get AD users for your Organization (AD Accounts)
• Update OneNote
• Track changes to Users, Events, Calendar Items using delta API

Step-By-Step : Access Outlook.com Emails using SSIS

Now lets look at examples to access your outlook.com emails using SSIS. You can use same techniques to call pretty much any Office 365 API.

Configure SSIS OAuth Connection for Graph REST API

First step to access any Graph API is configure OAuth connection

1. Open Visual Studio
2. Open existing SSIS Project or create new using File > New > Project > Choose “Integration Services Project” Type under Business Intelligence template category
3. Open Package
4. Right click inside Connection Managers area and click “New Connection…”
5. From the connection type list select “ZS-OAUTH” connection type.
   

   Create new SSIS OAuth API Connection Manager

6. On the OAuth Connection Manager configure following options
   1. Select “Custom” from Provider dropdown
   2. Select OAuth2 from OAuth version
   3. Enter your Application Id and Secret Key (i.e. App Password) obtained in the previous section from here.
   4. Enter following URL in the Authorization URL
      

      https://login.microsoftonline.com/common/oauth2/v2.0/authorize

   5. Enter the following URL in the token URL field
      

      https://login.microsoftonline.com/common/oauth2/v2.0/token

   6. Enter the following Graph API Scopes (each scope must be entered in a new line).
      

      user.read
      mail.read
      offline_access

      NOTE: Always include offline_access scope returns refreh_token which allows you to renew token without going through login process again. For more information about which scopes needed always refer to API help page (e.g. if you want to read mail using this API then you will see it says you can use Mail.Read or Mail.ReadWrite scope).
   7. Go to Advanced tab and enter the following URL in the Callback/Return URL (Assuming exact same URL used when you registered App in the previous section)
      

      https://zappysys.com/oauth

   8. Now go back to first tab and Click Generate Token. When prompted Login using your Personal Microsoft Account or Work Account (Office 365 or AD login)
   9. After login you will see Accept option just click it. If things go right then you will see Access Token and Refresh Token fields will be populated.
7. Once all configured you can click Test to make sure connection works.
   

   SSIS OAuth Connection Manager for Office 365 REST API / Microsoft Graph API

Configure SSIS JSON/REST API Source

Once you configure OAuth connection you are ready to read data from Office 365 API. Lets look at step by step how to read emails using Mail API and save to SQL Server.

1. Drag and drop Data flow task from SSIS toolbox
2. Goto data flow designer and drag and drop ZS JSON Source/REST API Connector on the designer
3. Double click to edit JSON Source.
4. In the URL field enter below URL
   

   https://graph.microsoft.com/v1.0/me/messages

   To customize URL with additional parameters check this help link. It’s a standard OData Protocol so you can leverage many common OData features (e.g. Use $top parameters to set pageSize. Default is 10 for List Mail but you can include more records per response by changing it such as …/messages?$top=50).
5. Check Use credentials option
6. From the connections dropdown select OAuth Connection manager created in the previous section
7. Change Data Format option to OData 
8. In the Filter enter following or click “Select Filter” and select Value Array node > Click OK.
   

   $.value[*]

9. In the Max rows set to 200 so we don’t pull all data for now 🙂 … This is our hello world example.
10. Now click Preview to see data. If all is well then Save UI by clicking OK
    

    SSIS JSON Source – Get data from Microsoft Graph API (Office 365 API – Read Mail Example)

Loading Graph API / Office 365 data to SQL Server

Now last step is loading Outlook data to SQL Server. For example purpose we will load data into SQL Server but you can load into any Target (e.g. Flat file, Oracle, Excel) using Microsoft or ZappySys Destination connectors

To load Office 365 data into SQL Server perform the following steps

1. Drag OLEDB destination on Data Flow surface
2. Connect ZS JSON Source to Destination
3. On OLEDB destination select / create new SQL Connection and then Click “New Table”
4. Click on Mappings tab and click OK to save
5. Execute Package

Configure SSIS OLEDB Destination – Loading Xero Data into SQL Server Table

Graph API / Office 365 to SQL Server Column Mappings for OLEDB Destination

Loading Graph REST API  / Office 365 data to SQL Server in SSIS

POST data to Graph API (Insert or Update)

So far we have seen how to read data from Office 365 API. Now let’s look at how to write data to Office 365.

There are two ways you can achieve this.

1. Use Web API Destination in Data Flow
2. Use REST API Task in Control Flow

Read this article to learn more about API POST using SSIS

Using SSIS Web API Task (Write data from SQL Server to Office 365 / Graph API)

Using REST API Task to POST data to Graph API

If you have JSON/XML data already prepared and if you like to POST it to Graph API then REST API Task would be easy to use.

REST API Task Body can be direct string, variable (e.g. {{User::varSomeData}} or Body can come from File 

SSIS REST API Task – POST data to Graph API

Download file from OneDrive

To download file from OneDrive perform these steps

https://zappysys.zendesk.com/hc/en-us/articles/115005010573-How-to-download-file-from-OneDrive-using-Graph-API

Upload file to OneDrive

https://zappysys.zendesk.com/hc/en-us/articles/115004893713-How-to-upload-file-to-OneDrive-using-Office-365-Graph-API

Reading Office Excel Sheets from OneDrive

JSON Source also support reading API response which returns 2D arrays. Such as Office Excel Spreadsheet API to read excel data. Check this blog post to learn similar scenario to parse 2D arrays. There are few other options for parsing 2D arrays which are documented here.

https://graph.microsoft.com/v1.0/me/drive/items/01CYZLFJDYxxxxxxx/workbook/worksheets('test')/range(address='A1:B200')

Testing Office 365 API using Graph API Explorer

If you like to test API request/response inside browser then Microsoft provides really great tool to called Graph API Explorer. Click here to try Graph API Explorer.

See below screenshot

Common Errors

Deployment to Production

Conclusion. What’s next?

In this article we have learned how to load data from Microsoft Graph API to SQL Server using SSIS ( drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from Outlook Mail using OAuth. JSON Source makes it super simple to parsing complex / large JSON Files or any Web API Response into rows and column so you can load into database like SQL Server. Download SSIS PowerPack to try many other automation scenarios not discussed in this article.

Keywords:

Office 365 API Integration with SQL Server | How to extract Office 365 data in SSIS? | How to read outlook mail, calendar event and contacts? | Calling Microsoft Graph API using SSIS. | Office 365 to SQL Server | SQL Server to Office 365 | SSIS Office 365 API Integration | SSIS Graph API Integration

The post How to use Office 365 API with SSIS (Mail, Calendar, Contacts, OneDrive, Excel) appeared first on ZappySys Blog.

In our previous post we saw how to parse JSON arrays. Now let’s look at more advanced techniques to parse multi-dimensional JSON array in SSIS (  e.g. 2D – JSON array inside array). We will use SSIS JSON Source to parse complex nested JSON in few clicks. Tips and Tricks mentioned in this article also apply to ODBC PowerPack API Drivers.

2D arrays are used to stuff more data in most compact way. Since you don’t have to repeat column names you save great amount of space. 2D array JSON format is close to CSV format in data size but it gives advantage and structure of JSON without adding extra fat.

Parsing JSON Arrays – Simple hierarchy

If you have simple JSON / XML Structure then you can use Select Filter option to parse array and flatten the hierarchy. You can select top level array or nested array to de-normalize the structure.

For example you can select $.orders[*] to extract all order records or use $.orders[*].items[*] to get each order items for all orders. There is also possibility to use light weight expression (like WHERE clause). Check this article how to use JSONPath but it has its own limitation,

Select Filter

Parsing Multi-Dimensional JSON Arrays

Now lets look at how to parse various multi-dimensional JSON array patterns in SSIS using SSIS JSON Source  or ODBC API drivers like JSON Driver

1. 2D Value Array with schema information (column names in a separate array) – Pattern 1
2. 2D Value Array without schema information – Pattern 2
3. Parse JSON Array using Complex Transformation – Pattern 3
4. Key/Value Transformation – Pattern 4
5. Multiple columns using JSONPath Expression (Google Geocoding API Usecase) – Pattern 5
6. Columnless Array – Pattern 6

Pattern 1 – JSON Array using Simple Transformation (Column names found in JSON)

This is the most common pattern of multi-dimensional JSON array where column names are stored in a separate array and values are stored in another 2D array

{
  "columns" : ["Id", "FirstName", "IsActive"],
  "rows" : [ [1,"bob",true], [2,"sam",false], [3,"joe",true] ]
}

Pattern 2 – JSON Array using Simple Transformation (Column names not found in JSON)

This is another common pattern of multi-dimensional JSON array where there are no column names. Values are stored in 2D array. Google SpreadSheet API uses this approach. In this pattern you don’t have column names specified so we have to enter manually (This like a CSV file without column names header). On the 2D array Tab check “Specify columns list manually” option. Then enter column names separated by comma. Enter same number of columns in same order as value array.

{
  "rows" : [ [1,"bob",true], [2,"sam",false], [3,"joe",true] ]
}

Pattern 3 – JSON Array using Complex Transformation

Below is more complex version where column names are found in array of column schema documents. Values are also stored in 2D array with complex structure. Google BigQuery uses similar complex structure

{
  "schema" : { 
    "columns" : [
      {"Name":"Id", "Type":"int"}, 
      {"Name":"FirstName", "Type":"string"}, 
      {"Name":"IsActive", "Type":"bool"}
    ]
  },
  "rows" : [ 
    { "rowid":1 , "values" : [1,"bob",true] }, 
    { "rowid":2 , "values" : [2,"sam",false] },
    { "rowid":3 , "values" : [3,"joe",true] }
  ]
}

Here is another complex variation involving multiple JSON 2D arrays, wherein column names are present within an array of column schema documents. Additionally, values are stored in a 2D array with a sophisticated structure. The Google Analytics Data API also employs a similar intricate structure for data representation.

{
	"dimensionHeaders": [{
		"name": "transactionid"
	}, {
		"name": "date"
	}, {
		"name": "CampaignId"
	}, {
		"name": "CampaignName"
	}, {
		"name": "GoogleAdsAdGroupName"
	}, {
		"name": "googleAdsAdGroupId"
	}, {
		"name": "defaultChannelGroup"
	}],
	"metricHeaders": [{
		"name": "sessions",
		"type": "TYPE_INTEGER"
	}],
	"rows": [{
			"dimensionValues": [{
				"value": "(not set)"
			}, {
				"value": "20230709"
			}, {
				"value": "(not set)"
			}, {
				"value": "(referral)"
			}, {
				"value": "(not set)"
			}, {
				"value": ""
			}, {
				"value": "Referral"
			}],
			"metricValues": [{
				"value": "5"
			}]
		},
		{
			"dimensionValues": [{
				"value": "(not set)"
			}, {
				"value": "20230711"
			}, {
				"value": "(not set)"
			}, {
				"value": "(referral)"
			}, {
				"value": "(not set)"
			}, {
				"value": ""
			}, {
				"value": "Referral"
			}],
			"metricValues": [{
				"value": "6"
			}]
		}
	]
}

In the Column name and Values Filter, instead of specifying individual arrays (e.g., dimensionHeaders and metricHeaders), we need to use the ‘*’ wildcard symbol to retrieve data from all arrays in a single output. This allows us to gather all the arrays’ data efficiently and conveniently.

Pattern 4 – Key/Value Transformation

SSIS PowerPack v2.6.6 and later introduced new transformation type called Key/Value transformation. This option Pivots dynamic Key/Value pairs of some attributes into Columns. Zoho CRM is best use case. Fully described here.

{
  "rows": [
    {
      "rownum": 1,
      "fields": [
        {
          "name": "FirstName",
          "value": "Bob"
        },
        {
          "name": "LastName",
          "value": "Tylor"
        },
        {
          "name": "Phone",
          "value": "111-111-1111"
        }
      ]
    },
    {
      "rownum": 2,
      "fields": [
        {
          "name": "FirstName",
          "value": "Sam"
        },
        {
          "name": "LastName",
          "value": "Smith"
        },
        {
          "name": "Phone",
          "value": "222-222-2222"
        }
      ]
    }
  ]
}

To transform above JSON where fields appear as column (e..g FirstName, LastName and Email) you need to apply following settings on JSON Source UI.

1. Select Filter $.rows[*]
2. Goto 2D Array Tab and select following settings
   1. Transform Type = Key/Value
   2. Column Name Filter =  $.fields[*].name
   3. Row Value Filter =  $.fields[*].value

Pattern 4.1 – Rootless JSON Array Key/Value Transformation

Sometimes you might get a rootless JSON array is a JSON structure where the top level is an array, not an object. This format is common when representing collections like lists of items or Custom Columns. Parsing it requires direct access to the array and its elements without a surrounding parent object.

[
    {
        "name": "FirstName",
        "value": "Bob"
    },
    {
        "name": "LastName",
        "value": "Tylor"
    },
    {
        "name": "Phone",
        "value": "111-111-1111"
    }
]

Output:
From this JSON array’s element name should be used as the column header, and the corresponding element value should populate the relevant column.

FirstName  |  LasteName  |  Phone
Bob        |  Tylor      |  111-111-1111

 
To parse this type of rootless JSON array, we need to use the “EnableSingleRowOutput” and “RawOutputDataTemplate” settings. These settings ensure that the array is treated as individual rows of data and that the raw JSON structure is maintained during parsing, allowing for direct access to each element within the array.
 
To do so Right-click on the JSON Source or JSON Parser Transform, go to “Properties”

• Search for “EnableSingleRowOutput“, and set its value to True
• Next, search for the property “RawOutputDataTemplate” and set its value as {data: [$1]}

SSIS JSON Source EnableSingleRowOutput and RawOutputDataTemplate Property

• Navigate to the “Array Transform” tab.
• Set the Transform Type to “Key/Value to Columns“
• Then select the desired “Column Name Filter“
• And select the desired “Row Values Filter“

That’s it! Click the ‘Preview’ button in the JSON source, and at runtime, you’ll get the desired output like this:

Parse Rootless JSON Array Using JSON Source

You can find a sample article and a video tutorial here: Reading Multiple Arrays from JSON/XML in SSIS.
Multicast approach (the recommended way):

Pattern 5 – Multiple Columns with Expressions (Google Geocoding API Usecase)

(Version 2.7.4 or higher) If you have a use case to include additional columns in the output based on multiple JSONPath expressions then you can use below option. This is useful for API such as Google GeoCoding API

Assume that you have following JSON as input (For demo use Direct Value Mode). You want to extract Address Components into Columns. Perform the following steps to paste such JSON.

{
   "results" : [
      {
         "address_components" : [
            {
               "long_name" : "1455",
               "short_name" : "1455",
               "types" : [ "street_number" ]
            },
            {
               "long_name" : "Gateview Circle Northeast",
               "short_name" : "Gateview Cir NE",
               "types" : [ "route" ]
            },
            {
               "long_name" : "Marietta",
               "short_name" : "Marietta",
               "types" : [ "locality", "political" ]
            },
            {
               "long_name" : "Cobb County",
               "short_name" : "Cobb County",
               "types" : [ "administrative_area_level_2", "political" ]
            },
            {
               "long_name" : "Georgia",
               "short_name" : "GA",
               "types" : [ "administrative_area_level_1", "political" ]
            },
            {
               "long_name" : "United States",
               "short_name" : "US",
               "types" : [ "country", "political" ]
            },
            {
               "long_name" : "30062",
               "short_name" : "30062",
               "types" : [ "postal_code" ]
            },
            {
               "long_name" : "2176",
               "short_name" : "2176",
               "types" : [ "postal_code_suffix" ]
            }
         ],
         "formatted_address" : "1455 Gateview Cir NE, Marietta, GA 30062, USA",
         "geometry" : {
            "bounds" : {
               "northeast" : {
                  "lat" : 33.9863736,
                  "lng" : -84.5125815
               },
               "southwest" : {
                  "lat" : 33.9862069,
                  "lng" : -84.5127193
               }
            },
            "location" : {
               "lat" : 33.9863117,
               "lng" : -84.51265389999999
            },
            "location_type" : "ROOFTOP",
            "viewport" : {
               "northeast" : {
                  "lat" : 33.98763923029151,
                  "lng" : -84.51130141970849
               },
               "southwest" : {
                  "lat" : 33.98494126970851,
                  "lng" : -84.51399938029151
               }
            }
         },
         "place_id" : "ChIJhUujpoIT9YgRVnSiltZkv0U",
         "types" : [ "premise" ]
      },
      
      {
         "address_components" : [
            {
               "long_name" : "1485",
               "short_name" : "1485",
               "types" : [ "street_number" ]
            },
            {
               "long_name" : "Gateview Way Northeast",
               "short_name" : "Gateview Way NE",
               "types" : [ "route" ]
            },
            {
               "long_name" : "Marietta",
               "short_name" : "Marietta",
               "types" : [ "locality", "political" ]
            },
            {
               "long_name" : "Cobb County",
               "short_name" : "Cobb County",
               "types" : [ "administrative_area_level_2", "political" ]
            },
            {
               "long_name" : "Georgia",
               "short_name" : "GA",
               "types" : [ "administrative_area_level_1", "political" ]
            },
            {
               "long_name" : "United States",
               "short_name" : "US",
               "types" : [ "country", "political" ]
            },
            {
               "long_name" : "30066",
               "short_name" : "30062",
               "types" : [ "postal_code" ]
            },
            {
               "long_name" : "2176",
               "short_name" : "2176",
               "types" : [ "postal_code_suffix" ]
            }
         ],
         "formatted_address" : "1481 Gateview Cir NE, Marietta, GA 30062, USA",
         "geometry" : {
            "bounds" : {
               "northeast" : {
                  "lat" : 33.9863736,
                  "lng" : -84.5125815
               },
               "southwest" : {
                  "lat" : 33.9862069,
                  "lng" : -84.5127193
               }
            },
            "location" : {
               "lat" : 33.9863117,
               "lng" : -84.51265389999999
            },
            "location_type" : "ROOFTOP",
            "viewport" : {
               "northeast" : {
                  "lat" : 33.98763923029151,
                  "lng" : -84.51130141970849
               },
               "southwest" : {
                  "lat" : 33.12233444,
                  "lng" : -84.3434343434
               }
            }
         },
         "place_id" : "DuIJhUujpoIT9YgRVnSiltZkvCH",
         "types" : [ "premise" ]
      }      
   ],
   "status" : "OK"
}

Steps to Parse JSON using Multiple Expressions

1. Drag and drop ZS JSON Source inside Data Flow Designer surface
2. Double click to edit component
3. Select Direct Value Mode and enter sample JSON (See above example)
4. Select Filter or enter manually  $.results[*]
5. Go to 2D Array Transformation Tab
6. Select Transformation Type as Multiple columns using Expressions
7. Click Configure Columns. You can now enter column name and expression to extract value for that column. Your expression must return only a single value.
   For Example, we entered expressions like below
   

   Column => street_number
   Expression => $.address_components[?(@.types[0]=='street_number')].long_name
   --------------------------------------------------------------------------
   Column => street_name
   Expression => $.address_components[?(@.types[0]=='street_name')].long_name
   --------------------------------------------------------------------------
   Column => city
   Expression => $.address_components[?(@.types[0]=='city')].long_name
   ...............
   ...............
   ...............

   

   Pattern 5- Multiple Columns Using JSONPath Expressions (Google Geocoding API example)

8. Now click Preview to see parsed data. As you see along with other columns we also got additional columns using complex JSONPath expressions.
   

   Preview Cleaned / Parsed Addresses from Google Geocoding API response

Pattern 6 – Columnless array

In this pattern you have 2D array without any property name in JSON like below. For this select Columnless Array option on 2D Transform Tab. You can also define custom columns if you dont like default names.

[
 [1, "AAA", "2012-01-01"],
 [2, "BBB", "2015-01-21"],
 [3, "CCC", "2017-01-31"]
]

OR API Patterns like Google Sheet API

{
  "range": "Sheet1!A1:D5",
  "majorDimension": "ROWS",
  "values": [
    ["Item", "Cost", "Stocked", "Ship Date"],
    ["Wheel", "$20.50", "4", "3/1/2016"],
    ["Door", "$15", "2", "3/15/2016"],
    ["Engine", "$100", "1", "30/20/2016"],
    ["Totals", "$135.5", "7", "3/20/2016"]
  ],
}

Manually supply column names

If you want to enter column names manually then use below setting. If you do not enter comma separated list of columns then system will auto generate names based on total columns found.

Parse JSON Array without any column names / property

Auto detect column names from first line

As we mentioned that API like Google Sheets might send you column names as Row data in first line. In such case you can use new option provided in v3.1.2 or later as below. If you 2D array is nested like below screenshot (e.g. under Values node) then you have to supply Array Filter on Filter Options Tab. In this case its

$.values[*]

JSON With 2D Array with Column names in First Row (Google Sheets API Pattern)

Pattern 7 – JSON Line format (JSONL) – New line separated

Version 3.1.2 and later you can use option to parse JSON Line format. In JSON Line (also known as JSONL ) record separator is \n or \r\n. There are two versions of JSONL format.

1. JSON Object Format – Using

   { }

     brackets around record (Also referred as Multi-Content Format). For This format you do not need any special settings in ZappySys. Its automatically detected and parsed correctly.
   Example:
   

   { Id:1, Name: "AAA" }
   { Id:2, Name: "BBB" }
   { Id:3, Name: "CCC" }

2. JSON Array Format – Using

   [ ]

     brackets around record. Typically this format is used with First line as Columns and other lines as Data.
   Example:
   

   ["Id","Name"]
   [1,"AAA"]
   [2,"BBB"]
   [3,"CCC"]

As we said First Example is automatically Parsed by ZappySys but for 2nd example (Array format) you have to use below Settings.

Parsing JSON Lines / JSONL Data Format (JSON Array Separated by new line)

Pattern 8 – Columnless Value Array with Pivot

If you have data like below without any property name and all in single dimension  array then you can use new option introduced after v3.1.3 “Pivot – Columnless Array” on array transformation tab. This option is not same as Pivot Option found on Pivot Tab (See next section). This option only works when you have no property name for array you like to transform. If you have property name for array then use option as described in the next section.

["one","two",null]

Output:

data
------------
one
two
null

Pattern 8 – Pivot Columnless Array

Pivot JSON / XML data

Now let’s consider little different scenario. If you don’t have JSON Array (See below sample) and your values are part of property name (in below case ProductCode – P001, P002… is actually Property name rather than value).  In this case you need to use Pivot Option. Check this article for full detail.

{
	"version": 1.0,
	"products": {
		"P001": {
			"name": "Product 1",
			"price": 10
		},
		"P002": {
			"name": "Product 2",
			"price": 11
		},
		"P003": {
			"name": "Product 3",
			"price": 12
		}
	}
}

Pivot JSON / XML Data option for ZappySys API Connectors / Drivers

Pivot JSON Data using SSIS JSON Source

Conclusion

Using SSIS PowerPack or ODBC PowerPack you can transform complex JSON in a few clicks. Both products support all options listed in this article. Based on your use case you can either use ZappySys Drivers or use SSIS Connectors.

The post Parse JSON array in SSIS or ODBC Drivers appeared first on ZappySys Blog.

Xero is well known Cloud based Accounting Software. In this post you will learn how to implement Xero API Integration with SQL Server or any other RDBMS (e.g. Oracle, MySQL, Postgresql) using SSIS in few clicks. We will use SSIS JSON Source to Read data from Xero and Load into SQL Server / other target (Using OAuth Connection). We will also discuss reverse scenario to Write data to Xero (API POST for Insert or Update in Xero) using SSIS Web API Destination and SSIS JSON Generator Transform

Prerequisites

Before we look into Step-By-Step section to extract and load data from Xero to SQL Server let’s make sure you met following requirements.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download FREE trial.
4. You have basic familiarity with REST API concepts and Xero API.

Concepts of Xero REST API

API Authorization

Before you can access Xero API you must register account and App. There are 3 types of Apps to access Xero API.

1. Private App (X509 Certificate used to sign requests)
2. Public App (Use 3-legged OAuth. Token expires in 30 mins and then you have to authorize again from UI)
3. Partner App (Use 3-legged OAuth. Token expires in 30 mins.. but refresh token supported to get new token for unlimited times without Authorization via Login UI)
4. OAuth2.0 Authentication (Recommended) 

In this article we will use Private App to access your company data in Xero.

Data formats

Xero API supports Xml and JSON both formats over the same URL (endpoints). If you want data in JSON format then you have to pass following Header (else data in Xml format will be returned). If you are using SSIS JSON Source then make sure following header is added. For SSIS XML Source no need to add this header.

Accept : application/json

Step-By-Step : Xero API Integration in SSIS

Now lets look at steps needed to fetch data from Xero and load into SQL Server.

Configure OAuth2.0 Connection for Xero API (Recommended)

So let’s how to create the Xero OAuth2.0 connection. For more on OAuth2.0 refer to this article.

1. Right-click inside the Connection Managers area again and click “New Connection…”
2. From the connection type list select “ZS-OAUTH” connection type.
   

   Create new SSIS OAuth API Connection Manager

3. Select OAuth Provider which Custom, enter the CleintID and Secret and set the URLs as below and desired scopes that are required for the API Call.
   Auth URL:
   

   https://login.xero.com/identity/connect/authorize

   Token URL:
   

   https://identity.xero.com/connect/token

   Scopes: Add your desired scopes based on your API call requirements.
   
   Note: To get a refresh token, you must request the offline_access scope. A refresh token allows you to refresh your access token and maintain an offline connection.
   offline_access

   https://developer.xero.com/documentation/guides/oauth2/scopes/#offline-access

   

   OAuth2.0 Connection

4.  Now go to the Advanced tab and set the redirect URL as below and also you need to set the same URL in the Xero API App.
   

   https://zappysys.com/oauth2

   

   Set Callback/Return Url in OAuth Connection

5. Now Click on the Generate Token Button and logged in with Xero username and password and grant permission.
6. That’s it your token is generated. If you got the blank Screen like this after following all the steps. Close that window.
   

   OAuth2.0 connection : Blank Window at last

7. And it will ask you to use the Different Method to generate the Token using system default browser click on YES.
   

   Different Method to generate the Token using system default browser

8. It will open the URL in the browser there you need to log in and grant the permission.
9. After that, you will able to view the access token in the browser. you can copy from there also.
   

   Generated Token

10. At the last, it will again focus on the connection and there it will ask you to save the token if you want to store it. if not then you can manually copy and paste that token.
    

    OAuth Connection : Store the Tokens

11. That’s it now click on the Test Connection button to test it.
    

    Generate Token and Test Connection

Register OAuth App Private for Xero (Private App) OAuth Connection (Depreciated)

Here is how to configure Xero to use Private App (click on below link).

https://developer.xero.com/documentation/auth-and-limits/private-applications

In the above link during these steps openssl command may throw error about openssl.cnf is missing or cannot load. If you face such error then don’t worry just refer to following workaround.

https://stackoverflow.com/questions/14459078/unable-to-load-config-info-from-usr-local-ssl-openssl-cnf

Basically you can either supply config file path in each command OR set path by issuing SET command before any other command. See below examples (Assuming you installed OpenSSL to c:\OpenSSL-Win64 folder)

c:\OpenSSL-Win64>openssl req -x509 -config "C:\OpenSSL-Win64\bin\openssl.cnf"

— OR —  type before command before any other openssl commands

c:\OpenSSL-Win64>set OPENSSL_CONF=c:\OpenSSL-Win64\bin\openssl.cnf

c:\OpenSSL-Win64>openssl req -x509 -config "C:\OpenSSL-Win64\bin\openssl.cnf"

Once you generate public key, private key and, pfx file we are ready to call Xero API in SSIS

Configure Xero (Private App) OAuth Connection

Once you generate certificate using openssl and register Private App for Xero we are ready to move to SSIS piece. Lets see how to do that.

1. Open Visual Studio
2. Open existing SSIS Project or create new using File > New > Project > Choose “Integration Services Project” Type under Business Intelligence template category
3. Open Package
4. Right click inside Connection Managers area and click “New Connection…”
5. From the connection type list select “ZS-OAUTH” connection type.

   

   Create new SSIS OAuth API Connection Manager

6. On the connection select “Xero (Private App)” from Service Provider Dropdown
7. Enter Client ID (Obtain this from App created in Previous section) App Selection leave “Use Default OAuth App” selected. If you created custom OAuth App then select “Use Custom OAuth App” Option
8. Enter some fake value in Client Secret (not needed when you use Private App). Secret will come from certificate file like below.
   

   SSIS OAuth Connection – Configure Xero API Connectivity using Private App (Certificate File Approach)

9. Now go to Certificate tab and Check Use Certificate Enter, Enter certificate location and password like below.
   

   SSIS OAuth Connection – Configure X509 Certificate File Option for Xero Private App (Select PFX file)

10. Click OK to save OAuth connection manager

So in the next section from the upper connections, we need to use any of the one connection only for Xero API now let’s try to read data from Xero using SSIS JSON Source.

Xero recently added OAuth 2.0 support so that now recommended way to use as Connection.

Configure JSON / REST API Source – Read data from Xero

Now let’s look at how to configure SSIS JSON Source to read data from Xero API

1. In the control flow ssis toolbox Drag and drop data flow task
2. Double click Data flow
3. Drag ZS JSON Source (REST API or File) from SSIS Toolbox and drop on Data flow designer
4. Double click ZS JSON Source to configure it.
5. Set the following properties
   1. Enter URL
      

      https://api.xero.com/api.xro/2.0/Invoices

   2. Check Use Credentials and select OAuth connection we created in the previous section
   3. For Headers grid. Click on Raw Edit button and remove enter following (remove default headers)
      

      Accept: application/json

   4. Click on Select filter and select an array node as below to generate filter expression (e.g. $.Invoices[*] )

Configure SSIS JSON /REST API Source – Read Xero Invoices

Also, add following header in case of multiple tenants are being used in XERO environment –

xero-tenant-id: <TENANT ID FROM XERO>

Configure Xero REST API Pagination in SSIS JSON Source

By default, Xero may not send you all records and you may have to setup pagination to read more data after the first response.

For example, you like to read LineItems from Invoices. For that make sure you use below URL and include page=1 to start with. If you do not include page=1 then Line items are not returned (Yes its ODD but clearly documented here).

1. Enter URL as
   

   https://api.xero.com/api.xro/2.0/Invoices?page=1

2. Select Filter as

   $.Invoices[*].LineItems[*]

3. Go to Pagination tab
4. Select Pagination by URL Parameter
5. Enter page as pagination parameter name like below
   

   REST API Looping/Pagination via URL Page Number Parameter (Loop until last page detected)

6. Click OK to save

Now when you run package you will see all LineItems for each Invoice will be returned.

Loading Xero data to SQL Server

Now last step is loading Xero data to SQL Server. For example purpose we will load data into SQL Server but you can load into any Target (e.g. Flat file, Oracle, Excel) using Microsoft or ZappySys Destination connectors

To load Xero data into SQL Server perform following steps

1. Drag OLEDB destination on Data Flow surface
2. Connect ZS JSON Source to Destination
3. On OLEDB destination select / create new SQL Connection and then Click “New Table”
4. Click on Mappings tab and click OK to save
5. Execute Package

Configure SSIS OLEDB Destination – Loading Xero Data into SQL Server Table

Xero to SQL Server Column Mappings for OLEDB Destination

Loading Xero data to SQL Server in SSIS

POST data to Xero (Insert or Update)

So far we have seen how to read data from Xero. Now let’s look at how to write data to Xero.

There are two ways you can achieve this.

1. Use Web API Destination in Data Flow
2. Use REST API Task in Control Flow

Read this article to learn more about API POST using SSIS

Using SSIS Web API Task (Write data from SQL Server to Xero)

Using REST API Task to POST data to Xero

If you have JSON/XML data already prepared and if you like to POST it to Xero API then REST API Task would be easy to use.

REST API Task Body can be direct string, variable (e.g. {{User::varSomeData}} or Body can come from File 

SSIS REST API Task – POST data to Xero (Create contacts)

Read data from Xero Report API

In previous examples we saw how to read data from various Xero API endpoints but now lets look at how to call Xero Report API. For example if you like to extract data displayed in below then you can extract it in Xml or JSON format. Below report is in HTML format but we will show you how to get similar information in JSON format and load into SQL Server Table.

Xero Report Example – View Profit and Loss Report in Xero Portal

To extract above information you can call below API Endpoint (e.g. Profit & Loss Report)

GET 
https://api.xero.com/api.xro/2.0/Reports/ProfitAndLoss?fromDate=2017-02-01&toDate=2017-02-28

You can use Variables to make above URL dynamic in JSON Source as below

https://api.xero.com/api.xro/2.0/Reports/ProfitAndLoss?fromDate={{User::StartDate,yyyy-MM-dd}}&toDate={{User::EndDate,yyyy-MM-dd}}

Each report has its own API URL so make sure you refer to the connect URL for calling different report.

Here is sample response from API

{
  "Reports": [
    {
      "ReportID": "ProfitAndLoss",
      "ReportName": "Profit and Loss",
      "ReportType": "ProfitAndLoss",
      "ReportTitles": [
        "Profit & Loss",
        "Demo Company (AU)",
        "1 February 2018 to 28 February 2018"
      ],
      "ReportDate": "25 February 2018",
      "UpdatedDateUTC": "\/Date(1519593468971)\/",
      "Rows": [
        {
          "RowType": "Header",
          "Cells": [
            { "Value": "" },
            { "Value": "28 Feb 18" },
            { "Value": "28 Jan 18" }
          ]
        },
        {
          "RowType": "Section",
          "Title": " Income",
          "Rows": [
            {
              "RowType": "Row",
              "Cells": [
                {
                  "Value": "Sales",
                  "Attributes": [
                    {
                      "Value": "e2bacdc6-2006-43c2-a5da-3c0e5f43b452",
                      "Id": "account"
                    }
                  ]
                },{
                  "Value": "9220.05",
                  "Attributes": [
                    {
                      "Value": "e2bacdc6-2006-43c2-a5da-3c0e5f43b452",
                      "Id": "account"
                    }
                  ]
                },{
                  "Value": "5120.05",
                  "Attributes": [
                    {
                      "Value": "e2bacdc6-2006-43c2-a5da-3c0e5f43b452",
                      "Id": "account"
                    }
                  ]
                }
              ]
            },
            {
              "RowType": "SummaryRow",
              "Cells": [
                { "Value": "Total Income" },
                { "Value": "9220.05" },
                { "Value": "1250.09" }
              ]
            }
          ]
        },{
          "RowType": "Section",
          "Rows": [
            {
              "RowType": "Row",
              "Cells": [
                { "Value": "NET PROFIT" },
                { "Value": "-6250.09" },
                { "Value": "-7250.09" }
              ]
            }
          ]
        }
      ]
    }
  ]
}

Now lets look at steps to configure JSON Source to extract your Xero Report Data.

1. Drag new JSON Source from SSIS Toolbox
2. Double click JSON Source to configure it
3. In the Path / Web URL, enter API Report URL to call desired Report Start / End Date or any other expected Parameter by report (Refer to API Documentation). For our example we will enter below URL
   

   https://api.xero.com/api.xro/2.0/Reports/ProfitAndLoss?fromDate=2017-02-01&toDate=2017-02-28

4. Check Use Credentials and Select Connection we created in the Previous Section
5. In the Filter enter

   $.Reports[*].Rows[*].Rows[*]

6. Now go to 2D Array Transform Tab and configure like below. If your UI varies then use Properties window to edit property manually.
   Basically you need to edit
   Transformation Type:

   2-dimensional array

   Column Name Filter:

   $.Reports[0].Rows[*].Cells[*].Value

   Row Values Filter:

   $.Cells[*].Value

7. Now Click Preview Data. If you get error Column ” does not belong to table nosqldata. Then most likely you using old version with Known Issue (i.e. v2.9.7.10822 or older). Get latest version for fix. For workaround of this issue you must follow next section to specify columns manually. Columns must match count.

Here is how it will look like.

Configure Transform Options for Xero Report API (Dynamic Column Title)

Read from Xero Report API using JSON Source in SSIS (Dynamic Columns Example)

Fixing Dynamic Columns Problem in for Xero Report API

Now let’s look at one common scenario which you have to think. SSIS data flow has fixed metadata so if your Report is generating Dynamic Columns each time based on Date Range then it will be trouble for you because all your columns may come as null if initial date range changes.

To address this issue you have to use Static Column Names as below.

Account (or sometimes blank) | Oct 08 | Sep 08 | Aug 08 | Jul 08 | YTD

NOTE: Use REST API Task to obtain Raw JSON and review First Row which contains Number of columns
For example if your API return some thing like below then you can use 5 columns (one Blank header + 4 Months)

"Rows": [
        {
          "RowType": "Header",
          "Cells": [
            { "Value": "" },
            { "Value": "28 Feb 18" }
            { "Value": "28 Mar 18" }
            { "Value": "28 Apr 18" }
            { "Value": "28 May 18" }
          ]
        },

Once we know how many columns we should expect from API response, we can perform following steps. We will now configure our JSON Source component to use Static Columns.

1. Drag new JSON Source from SSIS Toolbox
2. Right click JSON Source > Click on Properties (You can do below using UI too but in old version no option for that – See below this section)
3. Enter following values for property
   1. Filter: 

      $.Reports[*].Rows[*].Rows[*]

   2. ArrayTransformationType:

      TransformComplexTwoDimensionalArray

   3. ArrayTransCustomColumns:

      Col1,Col2,Col3,Col4,Col5,Col6,Col7,Col8,Col9,Col10,Col11,Col12,Col13

          (Assuming you have 12 months to compare)
   4. ArrayEnableCustomColumn:

      True

   5. ArrayTransRowValueFilter:

      $.Cells[*].Value

4. Now double click JSON Source to configure
5. In the Path / Web URL, enter API Report URL to call desired Report Start / End Date or any other expected Parameter by report (Refer to API Documentation). For our example we will enter below URL
   

   https://api.xero.com/api.xro/2.0/Reports/ProfitAndLoss?fromDate=2017-02-01&toDate=2017-02-28

6. Check Use Credentials and Select Connection we created in the Previous Section
7. In the Filter enter

   $.Reports[*].Rows[*].Rows[*]

8. Now click Preview.

For Old Version use Properties Grid

Configure Xero Report Data Extract using SSIS Property Grid (For Older Version)

For New Version Use UI (It has Specify Columns Manually Option)

Read from Xero Report API using parameters in JSON Source in SSIS (Static Columns Example)

Common Errors

Deployment to Production

Conclusion. What’s next?

In this article we have learned how to load data from Xero to SQL Server using SSIS ( drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from Xero REST API using OAuth. JSON Source makes it super simple to parsing complex / large JSON Files or any Web API Response into rows and column so you can load into database like SQL Server. Download SSIS PowerPack to try many other automation scenarios not discussed in this article.

Keywords:

Xero Integration with SQL Server | How to extract data from Xero in SSIS? | How to read data from Xero API? | Loading Xero Data into SQL Server. | Xero to SQL Server | SQL Server to Xero | SSIS Xero Integration

The post How to read/load data in Xero using SSIS appeared first on ZappySys Blog.

Google BigQuery is a fully managed Big Data platform to run queries against large scale data. In this article you will learn how to integrate Google BigQuery data into Microsoft SQL Server using SSIS. We will leverage highly flexible JSON based REST API Connector and OAuth Connection to import / export data from Google BigQuery API just in a few clicks.

If you are looking for a similar product inside Amazon AWS Cloud then check an article about Amazon Athena.

Prerequisites

Before we look into a Step-By-Step section to extract and load data from BigQuery to SQL Server let’s make sure you meet the following requirements:

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have basic familiarity with REST API concepts and Google BigQuery API. This post uses free Public Dataset to query BigQuery table so no billing is required to get your hands dirty 🙂 However, if you like to query your own dataset then make sure you have at least one BigQuery dataset and one table with some sample data created (this part does require billing enabled). Read a Google Quickstart article for more information on how to create a new BigQuery dataset and a table.

Understanding Google BigQuery Object Heirarchy

Google BigQuery has 3 main concepts below.

• Project
  • Dataset
    • Table
      • Query requests (each query creates a unique JobID – valid for 24 hours from which you can read data)

So in order to create BigQuery table you always need Project and then Dataset under that project. You can group objects around Project / Datasets. When you query you can supply fully qualified name of your table in FROM clause (e.g.

select count(*) from `bigquery-public-data.usa_names.usa_1910_2013`

So let’s get started.

Read data from Google BigQuery using SSIS

Basically you can query Google BigQuery data in two ways: In this article we will not cover 2nd method.

• Method-1: Query data using jobs/query method in BigQuery API. Use it if you expect to get a result in a fairly short amount of time. This API method generates a temp table which gets deleted after 24 hours. You can read data within that time frame using newly created JobId reference.
  
  NOTE: If you have records that sum up to more than 10MB of data or ~10,000 rows (assuming 1 row uses 1KB of data) then you need to proceed with two-step process as explained below.
• Method-2: Export SQL query result using jobs/insert method in BigQuery API. Use this method if you expect a query to take a long time to finish.

Now let’s see how to query Google BigQuery data using SSIS 🙂

Create Google API Project and Register OAuth App

To use any Google API firstly you have to finish two tasks:

1. Create Google API Project and obtain projectId (see next section).
2. Enable BigQuery API
3. Register your own Google OAuth App and obtain ClientId and Client Secret.

Check step-by-step instructions on how to create Google API Project and create OAuth App for Google. During instructions make sure you enable BigQuery API (screenshots may differ).

Get Google API Project ID

Once you have Google API Project created you can grab Project ID (see screenshot below). You will need Project ID in the next step when building API URL.

How to find Google API Project ID?

Configure OAuth Connection for Google BigQuery API

In order to call most of Google APIs you will need an OAuth App. If you want to use your own app then refer to how to register Google OAuth Application (Get ClientID and ClientSecret) article. For simplicity we will use Default OAuth App:

1. Right click inside Connection Managers area and click “New Connection…”
2. From the connection type list select “ZS-OAUTH” connection type.
   

   Create new SSIS OAuth API Connection Manager

3. For OAuth Provider select “Google”.
4. For OAuth App type options leave “Use Default OAuth App” selected. If you created custom OAuth App then you would need to select “Use Custom OAuth App” option.
5. For Scopes enter or select the following URLs (URL is just a permission name). Each URL must be in a separate line. For demo purposes we use only the first 3 scopes but we included a few more in case you like to test API in depth with a different permission set:
   

   https://www.googleapis.com/auth/bigquery
   https://www.googleapis.com/auth/cloud-platform
   https://www.googleapis.com/auth/cloud-platform.read-only
   https://www.googleapis.com/auth/bigquery.insertdata
   https://www.googleapis.com/auth/devstorage.full_control
   https://www.googleapis.com/auth/devstorage.read_only
   https://www.googleapis.com/auth/devstorage.read_write

6. Click Generate Token. Login using correct account if needed and then you will be prompted to click “Approve” OAuth App.
7. Once you click OK on the login screen you will see new tokens populated on the connection screen.
8. Click Test to make sure connection / tokens are valid and then click OK to save the connection.
   NOTE: Below screenshot uses Default App but we recommend you to to use Custom OAuth App (Your own Clientid / secret – Obtained in previous section)
   

   Create SSIS OAuth API Connection for Google BigQuery API

Start BigQuery Job and get JobId (Submit Query)

Once you have SSIS OAuth connection created for BigQuery API it’s time to read data from BigQuery. So, basically, there are two ways you can read BigQuery data: using query or insert method. For demo purposes we will use jobs/query method. If you want fire complex queries which can run for many minutes then refer to jobs/insert method.

If you are expecting more than 10MB of data (~10K records) then you have to split data extract in two steps:

1. Step-1 : Call jobs/query method using REST API Task to submit SQL query. This steps returns jobComplete: true if supplied query execution is done within timeoutMs parameter you included. By default its 10 seconds so if your query is going to take longer than 10 seconds then its good idea to add bigger timeout that way you dont need step#2. Once API call is done you will Get JobId in the response. Save that JobId in a variable for later use. This job result is valid for 24-hours only.
2. Step-2 (Optional) – You can add another optional REST API Task after previous step to Wait until Job is completed. For simplicity, this article will Omit setting up Status Check but its very simple….Use below settings
   1. Drag new REST API Task from toolbox. Connect it to Step-1 and double click to configure as below.
   2. Select URL From Connection mode,
   3. Select OAuth connection
   4. Enter URL as https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/jobs/{{User::JobId}} , Method: GET
   5. On Response Settings Tab, Select Response Content Type JSON and for content filter enter $.status.state
   6. On Status Check Tab, check Enable Status Check Loop and enter DONE in SuccessValue field
   7. Doing this setup will make sure we do not query data until Job Status is DONE (System keeps checking every 5 seconds)
3. Step-2 : Read Job result using JSON Source in a Data Flow Task.

Now lets see how to configure REST API Task to submit query and extract JobId (which will be used in next section).

1. In the Control Flow designer drag and drop ZS REST API Task from the SSIS toolbox.
2. Double click the task to configure it.
3. Change Request URL Access Mode to [Url from connection].
4. From the connection dropdown select OAuth connection created in the previous section.
5. Enter URL as below (replace YOUR-PROJECT-ID with a valid Project ID obtained in the previous section):
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-PROJECT-ID/queries

6. Select Method as POST.
7. Enter Body (Request Data) as below (change your query if needed; we used Public dataset for the demo): If you do not specify timeout then default is 10 seconds only. Also in this call we only care about JobID from response so we just added maxResults=10 because in 2nd step we will get all rows by doing pagination.
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000"
   }

   To use a Standard SQL query instead of Legacy SQL, add useLegacySql property and set it to false:
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000",
      "useLegacySql": false
   }

   For all possible parameters refer to jobs/query method documentation.
   NOTE: You can also supply location Parameter in the Body JSON to indicate where job should run. For non EU / US datacenters we suggest you to supply this parameter. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location.
   Example Use Of Location
   

   { "location":"us-east1",  "maxResults": 10, "query": "SELECT title FROM [bigquery-public-data:samples.wikipedia] LIMIT 10" }

8. Select Body Content Type as application/json.
9. Click Test Request/Response. If things go well you will see JSON content. Then just copy JobId from the response and save for later use:
   
10. Now go to Response Settings tab.
11. Select ContentType=Json.
12. Enter below expression to extract JobId from the response. You will need this Id to read SQL query output (this JobId is valid for 24 hrs):
    

    $.jobReference.jobId

13. Choose save response content to save to a variable. Select <New Variable…> option. When prompted give a name to your variable (i.e. vJobId) and for the value field paste JobId copied from the above step:
    
    
    NOTE: This JobId expires after 24-hrs so while you are designing a package and not running it then design time value is used for testing/previewing. So make sure your JobId is valid. If needed click Test Request/Response to grab a new JobId and update the variable with a new value so you can preview data without running a full package.
14. Click OK to save the UI.

Read BigQuery SQL result (for specified JobID)

Now let’s look at how to read data from BigQuery

Configure Google BigQuery Web Request (URL, Method, ContentType, Body etc.)

1. In the Control Flow designer drag and drop Data Flow Task from SSIS toolbox.
   
2. Double click Data Flow Task and drag and drop ZS JSON Source (For API/File) from SSIS toolbox.
   
3. Double click JSON Source to edit and configure as below
4. Enter URL as below. Replace YOUR-API-PROJECT-ID with the API Project ID obtained in the previous section.
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?maxResults=10000

   –OR– (Use below if your Job ran outside US / EU data center)
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?location=YOUR_REGION_ID&maxResults=10000

   NOTE: location Parameter indicates the geographic location of the job. This is Required parameter except for US and EU. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location. You can supply same location parameter for first step when you submit the query

   Failure to supply location parameter for non US /EU users may result in 404 NotFound Error.
   

5. Check Use Credentials option and select OAuth Connection Manager created in the previous section.
6. For HTTP Request Method select GET.
   

Read data from Google BigQuery from Temp Job result

Configure Filter

We need to transform a single JSON response into multiple rows so we need to apply a correct filter.

1. On JSON Source go to Filter Options tab.
2. In the Filter field enter $.rows[*] or click [Select Filter] button to browse hierarchy you want to extract.

Select Filter for JSON Response

Configure BigQuery API Pagination Settings

Most of modern APIs usually implement some sort of pagination technique so you get a part of data in each request rather than all in one go. Thus if you want more you can paginate through pages until the last page is reached. You can also read Understanding REST API Pagination in SSIS article to learn more about pagination in SSIS.

BigQuery API returns pageToken attribute in response JSON if more data is found for requested query result. You can then pass pageToken in the next URL in this format: http://my-api-url/?pageToken=xxxxxxxxxx

Now lets configure JSON Source to automate this pagination for us. On JSON Source go to Pagination Tab and enter the following two settings:

1. Set Next Link as $.pageToken.
2. Set Suffix for Next URL as &pageToken=<%nextlink%>.

See the screenshot below to get more clarity:

Configure BigQuery API Pagination on SSIS JSON Source

Configure Array Transformation

Now the last thing we have to configure is special 2-dimensional JSON array format used by BigQuery API:

1. On the JSON Source UI go to 2D Array Transformation tab.
2. Enter the following settings:
   1. For Transformation Type select Transform complex 2-dimensional array.
   2. For Column Name filter enter $.schema.fields[*].name.
   3. For Row Values Filter enter $.f[*].v.

JSON Array Transformation Options

Preview Data and Save UI

That’s it click Preview to see some data. If you entered sample JobID for your User::vJobID variable then you will see some data. In the next section we will see how to load Google BigQuery data into SQL Server. You can click Columns Tab to review Metadata.

Click OK to save UI and generate Metadata.

Configure Target – Load Google BigQuery data into SQL Server

Now you can connect your JSON Source to any target such as ZS Trash Destination or a real database destination such as OLEDB Destination.

If you wish to dump data from Google BigQuery to a SQL Server table then just perform the following steps:

1. Drag and drop OLEDB Destination from SSIS Toolbox. Rename it to something like SQL Server Table.
2. Double click on OLEDB Destination.
3. Click New to create a new connection > Configure connection > Click OK.
4. Click on New to create a new table > Rename default table name > Click OK.
5. Click on Mappings tab and click OK to save UI with default mappings.

Configure SSIS OLEDB Destination – Google BigQuery to SQL Server Import

Execute Package – Loading BigQuery data into SQL Server

SSIS Package Execution –
Loading Google BigQuery Data into SQL Server

Create / Delete Google BigQuery Dataset using API call

As we mentioned before, you need dataset before you can create a table. Most common way to create dataset is via User Interface but what if you like to automate from your SSIS Package or other workflow? Here is how you can create or delete. Basically you can use REST API Task to send CREATE or DROP command for Dataset object

Create Google Dataset

To create dataset configure REST API Task using below settings (We will call this API)

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/datasets

4. Request Method as POST
5. Enter Request Body as below (change YOUR_DATASET_NAME – e.g. TestDataset )
   

   {"datasetReference": { "datasetId": "YOUR_DATASET_NAME", "projectId": "{{User::ProjectId}}"} }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Dataset already exists then you can go to Error Handling Tab (See next section for screenshot) and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Delete Google BigQuery Dataset

For deleing dataset you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Create / Delete Google BigQuery Table using API call

Now let’s look at how to create /drop Table in Google BigQuery using API calls.

Create Google BigQuery Table

To send CREATE TABLE  SQL statement (DDL) we have to use same approach as we send normal SQL Query using this API call. So notice we used Standard SQL for this call by supplying  useLegacySql: false . DDL Statement must be all in one line

POST https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

Content-Type: application/json

>>>> BODY <<<<<

{
 "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
 "useLegacySql": false,
 "timeoutMs": 100000 
}

To create dataset configure REST API Task using below settings

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

4. Request Method as POST
5. Enter Request Body as below
   

   {
    "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
    "useLegacySql": false,
    "timeoutMs": 100000
   }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Table exists then you can go to enable Error Handling Tab and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Create Google BigQuery Table – API Call (Continue On Error Setting – Skip CREATE if Table / Dataset Already Exists)

Delete Google BigQuery Table

For deleing table you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Write data to Google BigQuery using SSIS – 1 Million row insert test (FAST)

Now let’s look at how easy it is to import data into Google BigQuery using SSIS. We will use the same OAuth connection we created before. To learn more about inserting data into BigQuery check tabledata/insertAll method documentation.

Make sure billing is enabled

Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

Streaming is not available via the free tier. If you attempt to use streaming without enabling billing, you receive the following error: BigQuery: Streaming insert is not allowed in the free tier.

As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call (Write  demo).

Configure OAuth Connection / Permissions

If you want to perform data insert operation in BigQuery using API calls then include the following scopes in your OAuth Connection Manager and generate a token (see our first section of this article – We already included scopes for Write operation but incase you didnt do then regenerate token with below scopes):

https://www.googleapis.com/auth/bigquery
https://www.googleapis.com/auth/cloud-platform
https://www.googleapis.com/auth/bigquery.insertdata

Create BigQuery Dataset (From UI)

For this demo first create a test dataset and one table under it like shown below (billing must be enabled on your Google API Project). To do that go to https://bigquery.cloud.google.com/welcome and configure them:

Insert data into BigQuery using API call

And here is a REST API example to insert data into a BigQuery Table:

Request URL:

https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

Request Headers:

Authorization: Bearer ya29.Gl0cBxxxxxxxxxxxxxxxxxxuEIhJIEnxE6GsQPHI
Content-Type: application/json

Request Body:

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
     {"json": {"RowId": 2,"CustomerName": "BBB"} }
   ]
}

Here is our data flow setup to achive very high throughput for Google BigQuery Data Load. We will show you how to insert one million rows in Google BigQuery in less than a minute based on below setup. We will use Multi Threading option and New Compression Option (Added in v3.1.4)

1. Dummy Data Source – Generate sample records
2. JSON Generator Transform – Generates JSON documents to send as POST request for above /insertAll API call.
3. Web API destination  – Call /insertAll API call to submit our data to BigQuery
4. (Optional) JSON Parser Transform – Parse Error Message for any response
5. (Optional) Trash Destination – Save any errors to text file for review

Google BigQuery Data Load Demo in SSIS – 1 Million Rows Insert with Multi Threads and Compression ON (Fast Upload)

Configure SSIS JSON Generator – Generate JSON for BigQuery Table Insert

Now let’s see how to build an HTTP request with JSON body and send it to BigQuery:

1. Drag Data Flow Task and double click on it.
   
2. Drag and configure your Source (for this demo we use Dummy Data Source with Customer Template). See previous section for configuration of Dummy Data Source.
   
3. Drag and drop ZS JSON Generator Transform to produce JSON from a database or file records. If your source is already sending a valid JSON then you can skip this step (e.g. SQL query is
   returning JSON). You can also read raw JSON from a very large file (new-line separated JSON) using JSON Source with Output as Raw Data option checked.
4. Connect Source to JSON Generator. Double click JSON Generator Transform to start configuring it like below.
5. Select Output Mode as Single Dataset Array and enter Batch Size 10000 (This is Max limit allowed by Google BigQuery API insertAll)
6. First right click on Mappings node and select Add Static Element
   

   Generate JSON for Google BigQuery InsertAll API request – Batch 10000 rows in a single API call

7. Enter Name as kind and value as bigquery#tableDataInsertAllRequest
   
   

   JSON Generator – Add Static Element

8. Now right click on Mappings node and select Add Document Array option
   

   JSON Generator – Add Document Array

9. Enter rows as array title and click OK
   

   JSON Generator – Name array

10. Select newly added array node and right click > Add unbound nested element  enter Output alias as json and click OK.
    

    JSON Generator – Add unbound nested element

11. Select json node and right click > Select Add Elements below this node and select multiple columns you like to send to BigQuery. Click OK to save.
    

    JSON Generator – Add Multiple Elements

12. Now let’s preview our JSON (Copy preview JSON to try in the next step – Web API destination)
    NOTE: Table name and column names are case-sensitive so make sure your JSON attribute matches exact same way.
    Here is the finished JSON Structure for next Step
    

    Sample JSON Request body for Google BigQuery insertAll API request

13. Click OK to save UI.

Configure SSIS Web API destination – Insert data into BigQuery Table

Once you have Input JSON prepared,  now let’s configure destination for BigQuery.

1. Drag and drop ZS Web API Destination.
2. Connect your JSON Generator Transform to Web API destination.
3. Configure general properties:
   

   SSIS Web API Destination –
   Configure for BigQuery Data load

4. Make sure to enter URL in this format:
   

   https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

   Make sure to replace 3 parts in above URL (MY_PROJECT_ID, MY_DATASET_ID, MY_TABLE_ID) with actual values from your Google Project and BigQuery dataset/table configuration.
5. Now you can enable Compression and Multiple Threads for higher throughput as below.
   NOTE: Compression Property was added in v3.1.4 so you may not see it if you have older version.
   

   Google BigQuery Data Loading Performance Optimization Options – Enable Multiple Threads and Compression Options

6. If you want to try test insert request from UI without running full package then go back to first tab and edit Body (Use Sample JSON generated by previous JSON Transform – You can grab from JSON Preview Panel on Generator Transform).Click Test Request / Response and confirm Success as below. You can go back to your BigQuery Portal and check one row is inserted after our test click. If everything looking good then run full package to insert all records.Sample JSON for Body:
   

   {
     "kind": "bigquery#tableDataInsertAllRequest",
     "rows": [
        {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
        {"json": {"RowId": 2,"CustomerName": "BBB"} }
      ]
   }

   

   Test Google BigQuery Table Insert – SSIS Web API Destination UI

7. Hit OK to save UI.
8. Run the package and verify data in Google BigQuery Console:
   

   Loading data into Google BigQuery using SSIS

Error Handling for BigQuery Data Load (Bulk Insert API Calls)

There will be a time when some records you insert may not go well in Google BigQuery. In such case you can read output from Web API destination and parse further with JSON Parser Transform. Check for certain values in the output. You must use JSON Parser after Web API destination (Connect Blue Arrow from Web API destination – Since its Soft Error it won’t redirect in Red Arrow ).

For example here is the sample JSON in POST Body for testing which produces error due to bad column name. When bad row found in batch all records will be rejected. Notice that error returns index of record in batch so you can identify which row went bad. It also returns column name in location attribute.

NOTE: Bad column name in 2nd record

Test Body (Bad):

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RecordID": 1,"CustomerID": "X1"} }, 
     {"json": {"Bad_Column": 2,"CustomerID": "X2"} }
     {"json": {"RecordID": 3,"CustomerID": "X3"} }, 
   ]
}

Response (For Bad Input):

{
  "kind": "bigquery#tableDataInsertAllResponse",
  "insertErrors": [
    {
      "index": 1,
      "errors": [
        {
          "reason": "invalid",
          "location": "bad_column",
          "debugInfo": "",
          "message": "no such field."
        }
      ]
    },
    {
      "index": 0,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    },
    {
      "index": 2,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    }
  ]
}

To configure error detection perform following steps.

1. Drag and drop ZS JSON Parser Transform after Web API destination
2. Click on Web API destination. Connect Blue arrow  to JSON Parser Transform
3. Configure JSON Parser Transform like below
4. Connect JSON Parser TRansform to some Destination to save error information (e.g. SQL Table or Trans destination)

Handling BigQuery Insert Errors in SSIS

Other Common Errors in BigQuery API calls

In this section we will talk about many common API errors in BigQuery.

Error: The project XXXXXXX has not enabled BigQuery

Sometimes you might get below error. To fix this error make sure you go to your Project and Enable BigQuery API

Status Code: BadRequest

Response Body: {
  "error": {
    "code": 400,
    "message": "The project bigquerytest-281915 has not enabled BigQuery.",
    "errors": [
      {
        "message": "The project bigquerytest-281915 has not enabled BigQuery.",
        "domain": "global",
        "reason": "invalid"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}

Error: 404 – Not Found: Table / Job xxxxxx

• Make sure you are setting GET request and not POST.
• Also make sure your jobId is valid because it expires after 24 hours.
• Make sure project-id supplied in URL is valid ID  (DO NOT Specify Alias, use internal ID for project)
• Make sure you supplied location parameter if your outside EU / US region. Reading data using this API call might fail if you failed to supply location in URL

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "notFound",
    "message": "Not Found: Table xxxxxx"
   }
  ],
  "code": 404,
  "message": "Not Found: Table xxxxxxx
 }
}

Error: 403 – Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier

If you trying call certain APIs on sandbox mode or free tier then you might get below error. To overcome this error enable billing on google bigquery. As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call.

Response Url: https://www.googleapis.com/bigquery/v2/projects/bigquerytest-281915/datasets/TestDataset/tables/Table1/insertAll

Status Code: Forbidden

Response Body: {
  "error": {
    "code": 403,
    "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
    "errors": [
      {
        "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
        "domain": "global",
        "reason": "accessDenied"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}

Debugging Web API Requests

If you need to debug actual requests made to Google server then you can use a tool like Fiddler. It’s a very handy tool to troubleshoot JSON format issues. It will allow to see how a request is made to a server.

Using Fiddler to debug Google BigQuery API requests in SSIS

Common Errors

Deployment to Production

Download Sample Package

Click Here to Download SSIS Sample Package – Google BigQuery API Read Write Create Delete (SSIS 2019, 2017, 2012)

Conclusion. What’s next?

In this article we have learned how to load data from Google BigQuery into SQL Server using SSIS (drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from Google BigQuery REST API using OAuth. JSON Source Connector makes it super simple to parse complex/large JSON files or any Web API response into rows and columns so you can load data into a database, e.g. SQL Server database. Download SSIS PowerPack to try many other automation scenarios that were not discussed in this article.

Keywords:

Google BigQuery Integration with SQL Server | How to extract data from google bigquery in SSIS? | How to read data from Google BigQuery API? | Loading BigQuery Data into SQL Server. | BigQuery to SQL Server | SSIS Google Big Query Integration | SSIS Google BigQuery Import  JSON File | SSIS Google BigQuery Export data

The post How to read / write data in Google BigQuery using SSIS appeared first on ZappySys Blog.

In this article you will learn how to edit SSIS Component Metadata and handle most common errors related to SSIS PowerPack Components. If you never heard about SSIS PowerPack then it’s addon pack to get additional 45+  SSIS components and Task (e.g. Connectivity to REST API, JSON, XML, Azure, AWS, Google, Salesforce etc).

SSIS Truncation Error (BufferException)

By default SSIS PowerPack components scan 300 rows to determine most accurate datatypes. However in many case you need to scan lot more rows to determine correct length and datatypes. If you go with default scan option to determine metadata then there is a chance that it may fail with the following error when you run SSIS package.

[JSON Source [2]] Error: System.Exception: BufferException: Truncation error with column [tickets[0].collaborator_ids] (to fix this error - see full message >> copy/paste this to text editor)
Error: The value is too large to fit in the column data area of the buffer. >> Source: tickets[0].collaborator_ids, Actual Length: 13, Value: [18726645887]
How to fix=> There are two ways to fix this error (see below steps).
>>> Method 1 <<<
- Goto dataflow designer and double click on the component which failed.
- Once component UI is open, go to [Columns] tab and change column length and DataType so data is not truncated. 
- Optionally you can click [Refresh Columns] and select bigger length for all string columns by changing [Metadata Scan Mode] to more wider option (e.g. Guess2x to Guess4x). Check [Overwrite locked columns] option if needed so all columns are refreshed.
- After you change length/datatype you can check [Lock] option so when metadata is refreshed for some reason your changes are preserved. 
>>> Method 2 <<<
- To know which records causing failure you can also redirect error output by dragging red arrow from component to downstream component (When prompted set Truncate Action=Redirect).
- If you simply dont care about such errors then Set Truncation action=Ignore in above step.
. RowNumber=11

SSIS Runtime Error – BufferException / Truncation error

Possible Reasons

Such error can occur due to following reasons

• You used default metadata scanning option to detect metadata. Default option scans only 300 rows to guess datatype and length. If you have more than 300 rows then possible to have data which has longer length than found in first 300 rows.
• Underlying data is changed after you configured the component so now you have bigger length for one or more string columns

Possible Fix

To fix truncation related errors you can use any of the following approach.

Method-1: Change default scan options and re-scan

Perform following steps to scan records with custom options.

• Edit component and goto columns tab
• On the columns tab change default scan row count to larger number this will cover bigger scan range (Default is 300).
• Click [Refresh Columns] button to scan again.
• When prompted change Auto mode to Guess4x (i.e. Set length to 4 times bigger than maximum detected length).
• Also check Set lock flag so any future scan wont override your changes. If you want to wipeout existing metadata and start again then check [Overwrite metdadata] option too
• Click OK to start scanning. Depending on how many scan rows selected it may take several seconds.
• Once scan is Done you will see Grid refresh with new column length and datatype

SSIS PowerPack – Advanced Metadata Scan Options

Method-2: Fix metadata for the column(s) which are causing runtime error

Another approach to fix metadata related errors you get at runtime is only fix those columns which caused the issue. Carefully read the error message to find which column is causing BufferException (sometimes you have to copy full message into notepad to see it).

Once you know the column name which caused the error, you have two options.

1. Remove that column from output : Uncheck from Left side list
— OR —
2. Increase column length : For this you can go to Columns tab as below screenshot and fix the length (Increase by 3-4 times or more if you expect longer string). If you need more than 4000 chars long string then you have to DT_NTEXT ( or DT_TEXT). After fixing column length (or datatype) click OK and run package again. Keep doing this until you fix all errors.

SSIS PowerPack – Column Metadata Editor (i.e. Change DataType, Column Length)

Method-3: Redirect bad rows to error output OR Ignore bad rows

If you want to detect records which are causing errors then you can redirect bad rows to error output using following way.

Redirecting bad rows to error output

To know which records causing failure you can redirect error output by dragging red arrow from component to downstream component (When prompted set Error and Truncation options to [Redirect row]).

SSIS Component Error Handling – Redirect bad rows to error output (Attach to destination)

SSIS Component Error Output – Add data viewer to see bad records and error message

Ignore bad records

If you simply don’t care about such errors and want to ignore them rather redirect then Set Error and Truncation options to [Ignore failure] in above step.

The post Handling SSIS Component Metadata Issues appeared first on ZappySys Blog.

HubSpot is a popular platform where CRM, Marketing and Sales meet in one place. It’s a good piece of service, yet sometimes you need to integrate it with your other system. In this article you will learn how to pull contacts from HubSpot API and load them into a SQL Server Database using SSIS and ZappySys SSIS PowerPack.

Prerequisites

1. HubSpot Account registered (an account you will want to get data from).
2. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
3. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
4. An existing SQL Server Database and table created (we will use it to load data into).
5. ZappySys SSIS PowerPack installed.

Various methods to call HubSpot API

HubSpot API supports two kinds of authentication:

1. OAuth 2.0.
2. API Key in URL.

API Key in URL is simple, with minimal configuration, but it ought to be used only for development and testing purposes.

In this tutorial we will show how to authenticate using OAuth 2.0, which you should use in your production environment.

Step-by-Step – Get data from HubSpot API into SQL Server database

1. Create HubSpot Developer Account (https://app.hubspot.com/signup/developers). It’s an account you will need to use to create HubSpot Apps.
2. Login with your HubSpot Developer Account (https://app.hubspot.com/login).
3. Select your newly created HubSpot Developer Account:
   

   Logging into HubSpot – Select HubSpot developer account

   NOTE: Account name should contain “hubspot-developers” text.
4. Register HubSpot OAuth App for REST API Call for use with OAuth 2.0 by simply pressing “Create application” button:
   

   Register HubSpot OAuth App for REST API Call for use with OAuth 2.0

5. Name the application (e.g. “ContactsApp”) and choose “Private” to make the application hidden from the outer and dangerous world 🙂
   

   Name HubSpot REST API App which will be used from within SSIS

6. Press “Edit” button on the app you just created.
7. Write down Client ID and Client secret. We will need them later for authentication:
   

   Write down Client ID and Client Secret which will be used in OAuth Connection SSIS component

8. Open SSIS Designer and create a new package.
9. Drag a new Data Flow task from SSIS Toolbox into the Control Flow:
   
   

   Dragging and dropping Data Flow Task into Control Flow

10. Edit Data Flow task by double clicking. Then drag ZS JSON Source and OLE DB Destination components from SSIS Toolbox into the Data Flow, but don’t connect them yet:
    
    

    Drag ZappySys JSON Source SSIS component into the Data Flow

11. Start editing JSON Source component:
    1. Set Access Mode to File path or web URL.
    2. Set Web URL to 

       https://api.hubapi.com/contacts/v1/lists/all/contacts/all?count=100

       .
       
       NOTE: We are adding URL parameter with value

       ?count=100

       , because by default HubSpot API returns 20 contacts and we want to maximize result set and minimize the number of web requests.
    3. Press Select Filter and select contacts node or simply enter

       $.contacts[*]

         into the text box.
    4. Check Use Credentials checkbox.
    5. Select <New ZS-OAUTH Connection> menu item to create a new connection:
       

       JSON Source SSIS component configuration – Get data from HubSpot API using OAuth authentication

12. Configure OAuth 2.0 connection for HubSpot API:
    1. Select HubSpot as OAuth Provider.
    2. Select “Use Custom OAuth (App created by you)” option.
       
       NOTE: You can also use Default OAuth App option for a quick test. Select it and press “Generate Token” button after you enter the scope.
    3. Paste Client Id and Client Secret you also saved in Step #7.
    4. Input

       contacts

        as our scope.
    5. Finally, press “Generate Token” button:
       

       OAuth Connection SSIS component configuration – Connect to HubSpot API using OAuth 2.0

13. Select your ordinary HubSpot account you want to get contacts from:
    

    OAuth Connection SSIS component configuration – Select a HubSpot account to get data from

    NOTE: Account name should NOT contain “hubspot-developers” text.
14. Grant access to your HubSpot App:
    

    OAuth Connection SSIS component configuration – Grant access to HubSpot API App

15. You will get a prompt to save Access and Refresh tokens to a backup file. Press “Yes” to save them.
    
    NOTE: You will need to pass

    Refresh Token

     as the parameter, when you run the package in another machine. Check Deployment to Production section for more info.
16. Configure HubSpot API pagination
    1. Set Next Link/Cursor Expression to

       $.vid-offset

         by pressing “…” button or simply pasting the value into the text box.
    2. In the same way set Stop Indicator/Attribute to 

       $.has-more

       .
    3. Input 

       false

        as Stop indicator value/Regular expression.
    4. Set Suffix for Next Url to

       &vidOffset=<%nextlink%>

       .
       

       JSON Source SSIS component pagination configuration for HubSpot API call

17. Press “Preview” button to… well, obviously to preview the results 🙂
    

    JSON Source SSIS component data preview of HubSpot API call result

18. Close “Data Preview” dialog and click “OK” button to save and close JSON Source component.
19. A dialog Metadata Scan Options should appear, to which you should respond “Ok” 🙂
20. Now you can connect JSON Source component with OLE DB Destination:
    
    

    Connect JSON Source SSIS component with OLE DB Destination for HubSpot data.

21. Now modify OLE DB Destination to have a connection to a SQL Server and point to a table, where you want contacts to be stored.
22. It’s time to map the columns of JSON Source with destination table columns. With your mouse simply connect corresponding columns or simply map Input Column with Destination Column. Press “OK” to close the window.
    

    Map JSON Source SSIS component columns with destination table columns you will store HubSpot data in

23. Finally, funtime! Press “F5” to execute the package. You should see a similar view as below. The number you see is the number of contacts were loaded from HubSpot API to your table:
    

    The result of getting HubSpot API data with SSIS

API Rate Limit (Throttling)

HubSpot places limits on using its API. Basically, it doesn’t allow you to make endless number of calls per second. Also it has a ceiling for requests made per day. So if you have paging enabled in JSON Source component, it will make as many requests as it has to retrieve all data. And in this step you may get an error, telling that you exceeded calling rate limit of the API. To avoid that, simply go to the Throttling tab and set the amount of milliseconds component has to sleep after making each call:

HubSpot API calls throttling configuration

Making things Dynamic (e.g. URL)

The URL we used (https://api.hubapi.com/contacts/v1/lists/all/contacts/all) in addition to

count

&showListMemberships=true

true

https://api.hubapi.com/contacts/v1/lists/all/contacts/all?showListMemberships={{User::ShowMemberships}}

Common Errors

For detailed instruction to fix common metadata related errors read this article.

Truncation related error

Most common error you may face when you run SSIS Package is truncation error. During design time only 300 rows scanned to detect datatype but at runtime you may extract far more records than 300. So its possible that you will get longer strings than initially expected. If you get error just copy that to notepad and read error carefully. It will explain the steps needed to fix the problem. Basically you can edit Columns on JSON Source UI > Change Data Length or Datatype (if you need more than 4000 chars change to the type to DT_NTEXT and click on Column Name cell to save change). After you edit default settings then make sure to check Lock option (found all on the right side in same row).

Authentication related error (happens when you run package from SQL Job Agent)

So many times when you run from Visual Studio it works fine but when you deploy to Production and run from SQL Job it fails. If this is your issue then check you passing Token (i.e. Password Property of HTTP connection) using some sort of parameter approach (e.g. from command line or from config file). By default password is encrypted using your account so when you run it from SQL Job it will fail to decrypt it. See more on Package Protection Level.

Deployment to Production

When you are ready to deploy the package to Production consider passing Refresh Tokens and other sensitive data as passwords via SSIS parameters. In this way you don’t face authentication related error described in above section. See more on Package Protection Level.

Download Sample Package

Be sure to download a sample SQL Server 2012 SSIS package, in case you encounter problems that you can’t go through in the step-by-step part.

Conclusion

In this article we learned how to call HubSpot API RESTful method and retrieve all contacts. Actually, HubSpot API consists of many small APIs and we used Contacts API method particularly. If you need anything else than simply retrieving the contacts of your account, delve into the vastness of HubSpot API methods!

Also it’s worth mentioning that JSON Source and OAuth Connection SSIS components saved us some valuable time, because we avoided using default SSIS Script Component and doing all the hard work ourselves.

Keywords

How to fetch data from HubSpot API | Pull data from HubSpot API into SQL Server | Call HubSpot API with SSIS | Sync HubSpot data to SQL Server | HubSpot data integration with SQL Server database | Load data from HubSpot REST API using SSIS

The post How to get data from HubSpot API with SSIS appeared first on ZappySys Blog.

ServiceNow one of the most popular Incident management platform used by many. In this blog post you will learn how to get data from ServiceNow REST API using SSIS JSON Source (No coding required). You will also learn other API integration scenarios such as POST, DELETE, PUT API calls.

You will also learn how to paginate response if you have large amount of data (E.g. more than 10000 rows)

Prerequisites

Before we look into Step-By-Step section to calling ServiceNow API in SSIS let’s make sure you met following requirements.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download FREE trial.
4. You have basic familiarity with ServiceNow API and you have UserID and Password handy.

Step-By-Step : Accessing ServiceNow data using SSIS

Now lets look at how to fetch data from ServiceNow API using SSIS JSON Source.

Configure SSIS HTTP Connection for ServiceNow REST API

First step to access ServiceNow data is configure HTTP connection on JSON Source. In below example we will use ServiceNow REST API. There are many endpoints for ServiceNow REST API but we will use Table API Service to query table. Now lets look at steps to configure ServiceNow API connection

1. Open SSIS Package and go to Data Flow designer (Click new data flow if missing)
2. Drag and drop  ZS JSON Source from SSIS Data Flow Toolbox
3. Double click JSON Source to configure it as below
   1. First Enter Full URL to access desired API. For example you can enter URL like below (We entered sysparm_limit=2000 to fetch 2000 records in each response. Change this to higher limit to speed up data extract. This number must match on Pagination tab – see next section). Change instance001 to your own instance name.
      

      https://instance001.service-now.com/api/now/v2/table/incident?sysparm_limit=2000

   2. Check Use Credentials option
   3. Click New Connection and configure HTTP connection like below. Once done click OK
      

      Configure ServiceNow REST API Connection in SSIS (Basic Auth using UserID, Password)

   4. Click on Select Filter or enter $.result[*] in the filter textbox
      

      Configure SSIS JSON for ServiceNow REST API – Read data from Table

4. Click Preview to see some data. In next section we will configure Pagination to fetch many records.
5. If you like to load data to SQL Server then Simply connect your JSON Source to OLEDB Destination and Create New Table > Click Mappings and Click OK
6. Execute Package

Configure JSON Source for ServiceNow API Pagination

By default ServiceNow API doesn’t return all records from your table. So if you wish to get all records then you must configure Pagination settings like below. For more information on pagination in SSIS read this article.

ServiceNow REST API RFC-5988 Pagination in SSIS

Once you configure API Pagination it will keep fetching data from new URL automatically until no more data found or it hit Max Row limit you specify on UI (In our case we did max 100 rows). Use tool like Fiddler to view HTTP requests made by JSON Source.

Debugging Web API requests using Fiddler

Configure JSON Source for ServiceNow JSONv2 Pagination (older version API)

In this section you will learn how to use older version of API known as ServiceNow JSONv2 Service (another help link here). Syntax for ServiceNow JSONv2 API is below to access table data.

https://<instance name>.service-now.com/mytable.do?JSONv2
Or
https://<instance name>.service-now.com/mytable.do?JSONv2&param1=val1&param2=val2....

Now lets look at steps to configure ServiceNow API connection

To configure JSONv2 API use above steps for most of the things except following settings

1. Configure URL like below (assuming you want to query sys_db_object table, with 15 record max in each response)
   

   https://instance001.service-now.com/sys_db_object.do?JSONv2&sysparm_record_count=15&sysparm_query=ws_access=true^sys_mod_count=1^ORDERBYsys_id

2. Configure Pagination Settings like below
   1. Select Mode = Response Attribute Mode
   2. For NextLink Enter $.records[-1:].sys_id
   3. For Suffix for Next URL enter &sysparm_query=sys_id><%nextlink%>^ws_access=true^sys_mod_count=1^ORDERBYsys_id
   4. For Filter enter $.records[*]

Loading ServiceNow data to SQL Server

Once you done configuring JSON Source you can  load data to SQL Server. For example purpose we will load data into SQL Server but you can load into any Target (e.g. Flat file, Oracle, Excel) using Microsoft or ZappySys Destination connectors

To load ServiceNow data into SQL Server perform the following steps (Screenshot is used for demonstration for purpose, your values can be different than below)

1. Drag OLEDB destination on Data Flow surface
2. Connect ZS JSON Source to Destination
3. On OLEDB destination select / create new SQL Connection and then Click “New Table”
4. Click on Mappings tab and click OK to save
5. Execute Package

POST Data to ServiceNow API (Insert, Update or Delete)

There will be a time you like to automate POST actions (e.g. create new incident via API call). Check this article to learn more on how to POST , DELETE, PUT data using API calls

Common Errors

Deployment to Production

Conclusion. What’s next?

In this article we have learned how to load data from ServiceNow API to SQL Server using SSIS ( drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from ServiceNow table. JSON Source Connector makes it super simple to parsing complex / large JSON Files or any Web API Response into rows and column so you can load into database like SQL Server. Download SSIS PowerPack to try many other automation scenarios not discussed in this article.

Keywords:

ServiceNow Integration with SQL Server | How to extract ServiceNow data in SSIS? | How to read ServiceNow table? | Calling ServiceNow REST API using SSIS. | ServiceNow to SQL Server | SQL Server to ServiceNow | SSIS ServiceNow API Integration | ServiceNow SOAP API Integration

The post How to read data from ServiceNow REST API using SSIS appeared first on ZappySys Blog.

eBay recently launched brand new REST API with OAuth 2.0 support. This article will explain you how to call eBay REST API using SSIS to automate eBay operations (e.g. Buy/Sell) using SSIS REST API Integration Connectors / OAuth Connection Manager.

This article assumes that you have basic knowledge of SSIS (SQL Server Integration Services). We will use drag and drop approach without coding to do eBay REST API integration. We will use following custom tasks/connectors from SSIS PowerPack

SSIS PowerPack REST API Connectors can be used to consume/manage eBay data using API calls. Following connectors will be used in this article.

	REST API Task
	OAuth Connection Manager
	JSON Source (File, REST, OData)

Test Ebay API using Online API Testing Tool

Before you can dig deep down we suggest you to try few API calls using ebay online testing tool. Here is an example api call using their Online Testing Tool.

For example visit below URL to try sample Search API call.

https://developer.ebay.com/my/api_test_tool?index=0&env=production&api=browse&call=item_summary_search__GET&variation=json

Testing Ebay REST API using Online Tool

Call eBay API using SSIS – Create new inventory item

Now lets look at how to make simple eBay REST API call using SSIS. For demo purpose we will use Sandbox API rather than production.

1. First download and install SSIS PowerPack from here.
2. Create new SSIS Project and open SSIS Package
3. Drag new ZS REST API Task from SSIS Toolbox on control flow designer
4. Rename ZS REST API Task to Create eBay inventory item-1. Double click the task to configure it.
5. Select Request mode to [Url from Connection]
6. Select ZS-OAUTH connection type from connection dropdown.
7. On OAuth connection. Select eBay Sandbox from provider dropdown.
8. Enter App ID in Client ID field (Obtained from here)
9. Enter Cert ID in Client Secret field  (Obtained from here)
10. Select desired Scopes or leave it blank to use default scopes. For this example we need following two scopes
    https://api.ebay.com/oauth/api_scope/sell.inventory
    https://api.ebay.com/oauth/api_scope/sell.inventory.readonly
11. Click on Advanced Tab. Enter RuName in the Return URL field. Don’t worry if your RuName doesn’t look like URL 🙂 (Click here to learn how to get RuName – Scroll to Getting the RuName value)
12. Now click on first tab again. Click Generate token button. When prompted login using Sandbox User Account (Click here to create test user for Sandbox).
    NOTE: Production account or developer account wont work when you login for eBay Sandbox API Token
13. Click Accept button after login. If everything goes well you will see two tokens populated on your main screen (Access Token and Refresh Token). You will get prompt to save tokens to some backup file… Do that so same tokens can be used if you wish to deploy JOB to another machine.
14. Click Test to validate connection. Click OK to save.
    

    SSIS OAuth Connection Manager – Connect to eBay Sandbox API

15. Now once you close connection you are back to REST API Task
16. Enter following URL to create or new eBay inventory item using PUT. Last part of following URL is ID we want to create
    

    https://api.sandbox.ebay.com/sell/inventory/v1/inventory_item/GP-Cam-01

17. Select Method PUT from dropdown
18. Click Edit for Request Body. Enter following data for our sample product
    

    {
    	"availability" : {
    		"shipToLocationAvailability" : {
    			"quantity" : 50
    		}
    	},
    	"condition" : "NEW",
    	"product" : {
    		"title" : "GoPro Hero4 Helmet Cam",
    		"description" : "New GoPro Hero4 Helmet Cam. Unopened box.",
    		"aspects" : {
    			"Brand" : ["GoPro"],
    			"Type" : ["Helmet/Action"],
    			"Storage Type" : ["Removable"],
    			"Recording Definition" : ["High Definition"],
    			"Media Format" : ["Flash Drive (SSD)"],
    			"Optical Zoom" : ["10x"]
    		},
    		"imageUrls" : [
    			"http://i.ebayimg.com/images/i/182196556219-0-1/s-l1000.jpg",
    			"http://i.ebayimg.com/images/i/182196556219-0-1/s-l1001.jpg",
    			"http://i.ebayimg.com/images/i/182196556219-0-1/s-l1002.jpg"
    		]
    	}
    }

19. Change Body content type to JSON from dropdown
20. Click on [Raw Edit] button in Headers grid toolbar. Enter following headers and click OK.
    

    Content-Language: en-US
    Accept: application/json

     
21. Click Test Request/Response – If you get success that means product got created.
22. You can create more products by simply changing SKU in URL and name/description in Body. Click Test to execute API call.

Read data from eBay and load into SQL Server using SSIS JSON Source

Now lets look at how to consume eBay REST API and load data into SQL Server. For this you can use JSON Source (File, REST API). For this demo we will Call eBay Search API . If you want to explore other API then check this page. Here is

1. Go to Data flow tab
2. Drag and drop ZS JSON Source from SSIS Toolbox
3. Double click to configure it.
4. Enter API URL like below. We added Search parameter and limit parameter (Max rows per response). Refer to API Help page to know more about other available parameters.
   

   https://api.sandbox.ebay.com/buy/browse/v1/item_summary/search?q=phone&limit=100

5. Check Use Credentials option.
6. Select ZS-OAUTH connection or Create New (See previous section to create new one)
7. Select Filter (Look for Array Icon and select that node). For example if you calling search API then  enter or select 

   $.itemSummaries[*]

     like below.
   

   SSIS JSON Source – Get data from eBay REST API (Use OAuth Connection)

8. Now click on pagination tab and enter

   $.next

     expression for next link attribute as below. This will enable pagination (It will keep reading in 100 rows per page until last page reached – Empty  recordset)
   

   Ebay REST API Pagination (Search API Example)

9. That’s it, now click Preview to see data
10. Once all looking good you can click OK to save JSON Source.
11. In Next Section we will see how to load Ebay Data into SQL Server.

Loading eBay data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Pagination for eBay REST API Calls

In previous example we saw REST API Pagination using Next Link approach but some API may not use same approach in that case refer to this link.  We looked at Next Link approach in previous section, you can also use Method #2 (URL parameter  – Offset) but Next Link approach is more simpler.

Calling eBay XML format API – Finding / Trading / Shopping API

eBay converted most of their APIs to JSON format to use OAuth 2.0 standard but there few APIs still there which are not OAuth 2.0 standard. These includes eBay Trading API, eBay Finding API, eBay Shopping API. They all are still in XML format. Below section describe how to call these API. Use eBay API Test tool here to generate desired Request Body and Headers.

Now let’s look at an example to call eBay XML API (GetItem from Trading API). To support special case like eBay API ZappySys introduced OAuth connection property called CustomAuthHeader in v2.6.7 or higher (Only found in Properties grid view).

1. Right click on OAuth connection manager icon > Click Properties > Set CustomAuthHeader to X-EBAY-API-IAF-TOKEN
   
2. Now drag ZS REST API Task from Control flow SSIS Toolbox
3. Set Properties like below
   1. Set URL as below (Sandbox Url is different than Production)
      

      https://api.sandbox.ebay.com/ws/api.dll

   2. Set Body As (this was obtained from  eBay API Test tool here )
      

      <?xml version="1.0" encoding="utf-8"?>
      <GetItemRequest xmlns="urn:ebay:apis:eBLBaseComponents">
      	<ErrorLanguage>en_US</ErrorLanguage>
      	<WarningLevel>High</WarningLevel>
            <!--Enter an ItemID-->
        <ItemID>1234</ItemID>
      </GetItemRequest>

   3. Set Headers as below (Click on Raw Edit Button and paste below heades). Again these header were also copied from eBay API Test tool here . Notice that API function name is included in the header. Also site id and compatibility level required.
      

      X-EBAY-API-CALL-NAME: GetItem
      X-EBAY-API-SITEID: 0
      X-EBAY-API-COMPATIBILITY-LEVEL: 967

   4. Now click OK to get response. If things working you will see some valid XML.
      

      Call eBay XML API using SSIS REST API Task (eBay Trading API, Shopping API, Finding API using OAuth Token)

User Token vs Application Token

Any time you call eBay API you have to supply token. eBay OAuth supports two types of token. In this article we used User Token. If you wish to use Application Token then you have to use OAuth Client Credentials grant. See this article

Difference between eBay Application Token vs User Token

1. Application Token
   – Do not require user login
   – Use this token if you need to perform basic operations which are supported by following permission   https://api.ebay.com/oauth/api_scope
2. User Token
   – Requires one time user login to create token and then refreshes automatically without login prompt (if you using SSIS PowerPack)
   – Use this token if you need higher privileges behalf of actual use for update/write data using API call

Use Application token when you want to perform readonly operations and you don’t need higher privileges (such as admin access). Certain eBay API calls only possible if you supply user token.

POST data to Ebay (Insert or Update)

In previous section we have seen how to read data from ebay. Now let’s look at how to write data to ebay.

There are two ways you can achieve this.

1. Use Web API Destination in Data Flow
2. Use REST API Task in Control Flow

Read this article to learn more about API POST using SSIS

Using SSIS Web API Task (Write data from SQL Server to ebay)

Using REST API Task to POST data to ebay

If you have JSON/XML data already prepared and if you like to POST it to ebay API then REST API Task would be easy to use.

REST API Task Body can be direct string, variable (e.g. {{User::varSomeData}} or Body can come from File 

Common Errors

Deployment to Production

Conclusion

REST API integration typically requires coding and good understanding of protocol such as OAuth. SSIS PowerPack REST API connectors and OAuth Connection provides easy to use interface with many options to integrate virtually any API including eBay REST API inside your SQL Server Environment without any coding. Try SSIS PowerPack yourself see what is possible.

The post How to call eBay REST API in SQL Server with SSIS appeared first on ZappySys Blog.