In our previous article, we see how to load data into Elastic Search using SSIS. Now let’s look at how to read data from ElasticSearch using SSIS and load response into SQL Server.

Elasticsearch is a powerful engine that allows you to store, aggregate, and, most importantly, search data in a very analytical way. In this tutorial, you will learn how to load Elasticsearch data to SQL Server with SSIS (part of SQL Server) and ZappySys PowerPack. We also see how to set the maximum result window of the index using the Rest API Task.

In a nutshell, this post will focus on how to Make Elasticsearch Search API call using SSIS.

Step-By-Step to Call Elasticsearch API call using SSIS

Read from Elasticsearch Search API call

Let’s make the Elasticsearch Search API call using the JSON source to get all the records of the index with pagination.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Dragging and dropping Data Flow Task into Control Flow

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to make Search API Call on the index.
   

   http://{elasticsearch-site-url}/{index}/{index-type}/_search

4. Now check Use credentials and Select Connection section press

   <New ZS-HTTP Connection>

   .
   

   SSIS JSON Source – Passing Basic Credentials (Supply UserID / Password) using HTTP Connection

5. Once the HTTP Connection Manager window opens configure the connection to your Elasticsearch instance:
   • Set Web Url, which points to your Elasticsearch instance.
   • Set Credentials Type to

     Basic - UserID/Password

      (or other appropriate authentication methods).
   • Finally, set User Name and Password:
     
     

     Configure SSIS HTTP Connection to connect to Elasticsearch

6. Now, select method as POST and set for and size value in the body to get data using pagination. Set the content type as JSON (application/json).
   

   {"from" : <%page%>, "size" : 100}

7. Furthermore, go to the pagination tab and configure pagination like the below screenshot makes sure we need to enter increment size the same as given in body size value therefor 100.
   

   Pagination Mode: POST data in the body

8. That’s it now select the desired filter and click on the Preview button.
9. That’s it we are ready to load the data in the SQL Server.

For more than 10000 rows

When you try to get more than 100000 rows you will get the next error message

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [11000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
            }
        ]
    },
    "status": 400
}

Here is how to configure your JSON Source to pull more than 10K rows using Scroll Method.

1. In Settings Tab: In the Body and URL rather than Index name use

   [$tag$]

2. In the filter type:

   $.hits.hits[*]._source

3. Uncheck Include Parent
   

   Elasticsearch example for more than 10000 rows

4. In Pagination Tab, Set Next Link as below

   $._scroll_id

5. Stop Indicator Attribute as below

   $.hits.hits[0]._id

6. Stop Indicator Value as

   regex=^$

   

   Elasticsearch pagination tab configuration

7. In Advanced Pagination Tab, Check Has Different Page Info and Enable Page Token
8. Enter Page Place Holders as below (change YOUR_INDEX_NAME)
   

   url=YOUR_INDEX_NAME/_search?scroll=10m|_search/scroll

9. First Page Body as below – Change query as needed else keep it match all to fetch all rows from Index
   

   { "size":1000, "query": { "match_all": { }  } }

10. Next Page Body as below 

    {  "scroll": "5m", "scroll_id": "[$pagetoken$]" }

    

    Configuration for Elasticsearch advanced pagination tab

Loading Elasticsearch API data into SQL Server / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Enable Deep Pagination Setting

By default index, the maximum result window size is 10,000 means you can paginate upto 10000 rows. If you ever want to get more than 10000 result then you have to perform the following steps to change default setting from 10000 to 500000 (that’s max).

We need to send PUT request to below URL:

http://{elasticsearch-site-url}/{your-index}/_settings

Need to pass below body as I am setting max result window size value as 500000.

{ "index" : { "max_result_window" : 500000 } }

And set content-type as JSON (application/json) and click on Test Request/Response and it will make the API call and set the max window size.

Rest API Task PUT Call

That’s it we have successfully made the call to set the maximum result window size of the index.

Common Errors

Deployment to Production

Conclusion

After all, we saw you How to Make Elasticsearch Search API call using SSIS JSON Source and load response into SQL Server. In this article, we have seen that how to make a setting call to set the maximum result window of the index. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File)
• Elasticsearch Search APIs

The post How to read data from ElasticSearch using SSIS appeared first on ZappySys Blog.

Xero is well known Cloud based Accounting Software. In this post you will learn how to implement Xero API Integration with SQL Server or any other RDBMS (e.g. Oracle, MySQL, Postgresql) using SSIS in few clicks. We will use SSIS JSON Source to Read data from Xero and Load into SQL Server / other target (Using OAuth Connection). We will also discuss reverse scenario to Write data to Xero (API POST for Insert or Update in Xero) using SSIS Web API Destination and SSIS JSON Generator Transform

Prerequisites

Before we look into Step-By-Step section to extract and load data from Xero to SQL Server let’s make sure you met following requirements.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download FREE trial.
4. You have basic familiarity with REST API concepts and Xero API.

Concepts of Xero REST API

API Authorization

Before you can access Xero API you must register account and App. There are 3 types of Apps to access Xero API.

1. Private App (X509 Certificate used to sign requests)
2. Public App (Use 3-legged OAuth. Token expires in 30 mins and then you have to authorize again from UI)
3. Partner App (Use 3-legged OAuth. Token expires in 30 mins.. but refresh token supported to get new token for unlimited times without Authorization via Login UI)
4. OAuth2.0 Authentication (Recommended) 

In this article we will use Private App to access your company data in Xero.

Data formats

Xero API supports Xml and JSON both formats over the same URL (endpoints). If you want data in JSON format then you have to pass following Header (else data in Xml format will be returned). If you are using SSIS JSON Source then make sure following header is added. For SSIS XML Source no need to add this header.

Accept : application/json

Step-By-Step : Xero API Integration in SSIS

Now lets look at steps needed to fetch data from Xero and load into SQL Server.

Configure OAuth2.0 Connection for Xero API (Recommended)

So let’s how to create the Xero OAuth2.0 connection. For more on OAuth2.0 refer to this article.

1. Right-click inside the Connection Managers area again and click “New Connection…”
2. From the connection type list select “ZS-OAUTH” connection type.
   

   Create new SSIS OAuth API Connection Manager

3. Select OAuth Provider which Custom, enter the CleintID and Secret and set the URLs as below and desired scopes that are required for the API Call.
   Auth URL:
   

   https://login.xero.com/identity/connect/authorize

   Token URL:
   

   https://identity.xero.com/connect/token

   Scopes: Add your desired scopes based on your API call requirements.
   
   Note: To get a refresh token, you must request the offline_access scope. A refresh token allows you to refresh your access token and maintain an offline connection.
   offline_access

   https://developer.xero.com/documentation/guides/oauth2/scopes/#offline-access

   

   OAuth2.0 Connection

4.  Now go to the Advanced tab and set the redirect URL as below and also you need to set the same URL in the Xero API App.
   

   https://zappysys.com/oauth2

   

   Set Callback/Return Url in OAuth Connection

5. Now Click on the Generate Token Button and logged in with Xero username and password and grant permission.
6. That’s it your token is generated. If you got the blank Screen like this after following all the steps. Close that window.
   

   OAuth2.0 connection : Blank Window at last

7. And it will ask you to use the Different Method to generate the Token using system default browser click on YES.
   

   Different Method to generate the Token using system default browser

8. It will open the URL in the browser there you need to log in and grant the permission.
9. After that, you will able to view the access token in the browser. you can copy from there also.
   

   Generated Token

10. At the last, it will again focus on the connection and there it will ask you to save the token if you want to store it. if not then you can manually copy and paste that token.
    

    OAuth Connection : Store the Tokens

11. That’s it now click on the Test Connection button to test it.
    

    Generate Token and Test Connection

Register OAuth App Private for Xero (Private App) OAuth Connection (Depreciated)

Here is how to configure Xero to use Private App (click on below link).

https://developer.xero.com/documentation/auth-and-limits/private-applications

In the above link during these steps openssl command may throw error about openssl.cnf is missing or cannot load. If you face such error then don’t worry just refer to following workaround.

https://stackoverflow.com/questions/14459078/unable-to-load-config-info-from-usr-local-ssl-openssl-cnf

Basically you can either supply config file path in each command OR set path by issuing SET command before any other command. See below examples (Assuming you installed OpenSSL to c:\OpenSSL-Win64 folder)

c:\OpenSSL-Win64>openssl req -x509 -config "C:\OpenSSL-Win64\bin\openssl.cnf"

— OR —  type before command before any other openssl commands

c:\OpenSSL-Win64>set OPENSSL_CONF=c:\OpenSSL-Win64\bin\openssl.cnf

c:\OpenSSL-Win64>openssl req -x509 -config "C:\OpenSSL-Win64\bin\openssl.cnf"

Once you generate public key, private key and, pfx file we are ready to call Xero API in SSIS

Configure Xero (Private App) OAuth Connection

Once you generate certificate using openssl and register Private App for Xero we are ready to move to SSIS piece. Lets see how to do that.

1. Open Visual Studio
2. Open existing SSIS Project or create new using File > New > Project > Choose “Integration Services Project” Type under Business Intelligence template category
3. Open Package
4. Right click inside Connection Managers area and click “New Connection…”
5. From the connection type list select “ZS-OAUTH” connection type.

   

   Create new SSIS OAuth API Connection Manager

6. On the connection select “Xero (Private App)” from Service Provider Dropdown
7. Enter Client ID (Obtain this from App created in Previous section) App Selection leave “Use Default OAuth App” selected. If you created custom OAuth App then select “Use Custom OAuth App” Option
8. Enter some fake value in Client Secret (not needed when you use Private App). Secret will come from certificate file like below.
   

   SSIS OAuth Connection – Configure Xero API Connectivity using Private App (Certificate File Approach)

9. Now go to Certificate tab and Check Use Certificate Enter, Enter certificate location and password like below.
   

   SSIS OAuth Connection – Configure X509 Certificate File Option for Xero Private App (Select PFX file)

10. Click OK to save OAuth connection manager

So in the next section from the upper connections, we need to use any of the one connection only for Xero API now let’s try to read data from Xero using SSIS JSON Source.

Xero recently added OAuth 2.0 support so that now recommended way to use as Connection.

Configure JSON / REST API Source – Read data from Xero

Now let’s look at how to configure SSIS JSON Source to read data from Xero API

1. In the control flow ssis toolbox Drag and drop data flow task
2. Double click Data flow
3. Drag ZS JSON Source (REST API or File) from SSIS Toolbox and drop on Data flow designer
4. Double click ZS JSON Source to configure it.
5. Set the following properties
   1. Enter URL
      

      https://api.xero.com/api.xro/2.0/Invoices

   2. Check Use Credentials and select OAuth connection we created in the previous section
   3. For Headers grid. Click on Raw Edit button and remove enter following (remove default headers)
      

      Accept: application/json

   4. Click on Select filter and select an array node as below to generate filter expression (e.g. $.Invoices[*] )

Configure SSIS JSON /REST API Source – Read Xero Invoices

Also, add following header in case of multiple tenants are being used in XERO environment –

xero-tenant-id: <TENANT ID FROM XERO>

Configure Xero REST API Pagination in SSIS JSON Source

By default, Xero may not send you all records and you may have to setup pagination to read more data after the first response.

For example, you like to read LineItems from Invoices. For that make sure you use below URL and include page=1 to start with. If you do not include page=1 then Line items are not returned (Yes its ODD but clearly documented here).

1. Enter URL as
   

   https://api.xero.com/api.xro/2.0/Invoices?page=1

2. Select Filter as

   $.Invoices[*].LineItems[*]

3. Go to Pagination tab
4. Select Pagination by URL Parameter
5. Enter page as pagination parameter name like below
   

   REST API Looping/Pagination via URL Page Number Parameter (Loop until last page detected)

6. Click OK to save

Now when you run package you will see all LineItems for each Invoice will be returned.

Loading Xero data to SQL Server

Now last step is loading Xero data to SQL Server. For example purpose we will load data into SQL Server but you can load into any Target (e.g. Flat file, Oracle, Excel) using Microsoft or ZappySys Destination connectors

To load Xero data into SQL Server perform following steps

1. Drag OLEDB destination on Data Flow surface
2. Connect ZS JSON Source to Destination
3. On OLEDB destination select / create new SQL Connection and then Click “New Table”
4. Click on Mappings tab and click OK to save
5. Execute Package

Configure SSIS OLEDB Destination – Loading Xero Data into SQL Server Table

Xero to SQL Server Column Mappings for OLEDB Destination

Loading Xero data to SQL Server in SSIS

POST data to Xero (Insert or Update)

So far we have seen how to read data from Xero. Now let’s look at how to write data to Xero.

There are two ways you can achieve this.

1. Use Web API Destination in Data Flow
2. Use REST API Task in Control Flow

Read this article to learn more about API POST using SSIS

Using SSIS Web API Task (Write data from SQL Server to Xero)

Using REST API Task to POST data to Xero

If you have JSON/XML data already prepared and if you like to POST it to Xero API then REST API Task would be easy to use.

REST API Task Body can be direct string, variable (e.g. {{User::varSomeData}} or Body can come from File 

SSIS REST API Task – POST data to Xero (Create contacts)

Read data from Xero Report API

In previous examples we saw how to read data from various Xero API endpoints but now lets look at how to call Xero Report API. For example if you like to extract data displayed in below then you can extract it in Xml or JSON format. Below report is in HTML format but we will show you how to get similar information in JSON format and load into SQL Server Table.

Xero Report Example – View Profit and Loss Report in Xero Portal

To extract above information you can call below API Endpoint (e.g. Profit & Loss Report)

GET 
https://api.xero.com/api.xro/2.0/Reports/ProfitAndLoss?fromDate=2017-02-01&toDate=2017-02-28

You can use Variables to make above URL dynamic in JSON Source as below

https://api.xero.com/api.xro/2.0/Reports/ProfitAndLoss?fromDate={{User::StartDate,yyyy-MM-dd}}&toDate={{User::EndDate,yyyy-MM-dd}}

Each report has its own API URL so make sure you refer to the connect URL for calling different report.

Here is sample response from API

{
  "Reports": [
    {
      "ReportID": "ProfitAndLoss",
      "ReportName": "Profit and Loss",
      "ReportType": "ProfitAndLoss",
      "ReportTitles": [
        "Profit & Loss",
        "Demo Company (AU)",
        "1 February 2018 to 28 February 2018"
      ],
      "ReportDate": "25 February 2018",
      "UpdatedDateUTC": "\/Date(1519593468971)\/",
      "Rows": [
        {
          "RowType": "Header",
          "Cells": [
            { "Value": "" },
            { "Value": "28 Feb 18" },
            { "Value": "28 Jan 18" }
          ]
        },
        {
          "RowType": "Section",
          "Title": " Income",
          "Rows": [
            {
              "RowType": "Row",
              "Cells": [
                {
                  "Value": "Sales",
                  "Attributes": [
                    {
                      "Value": "e2bacdc6-2006-43c2-a5da-3c0e5f43b452",
                      "Id": "account"
                    }
                  ]
                },{
                  "Value": "9220.05",
                  "Attributes": [
                    {
                      "Value": "e2bacdc6-2006-43c2-a5da-3c0e5f43b452",
                      "Id": "account"
                    }
                  ]
                },{
                  "Value": "5120.05",
                  "Attributes": [
                    {
                      "Value": "e2bacdc6-2006-43c2-a5da-3c0e5f43b452",
                      "Id": "account"
                    }
                  ]
                }
              ]
            },
            {
              "RowType": "SummaryRow",
              "Cells": [
                { "Value": "Total Income" },
                { "Value": "9220.05" },
                { "Value": "1250.09" }
              ]
            }
          ]
        },{
          "RowType": "Section",
          "Rows": [
            {
              "RowType": "Row",
              "Cells": [
                { "Value": "NET PROFIT" },
                { "Value": "-6250.09" },
                { "Value": "-7250.09" }
              ]
            }
          ]
        }
      ]
    }
  ]
}

Now lets look at steps to configure JSON Source to extract your Xero Report Data.

1. Drag new JSON Source from SSIS Toolbox
2. Double click JSON Source to configure it
3. In the Path / Web URL, enter API Report URL to call desired Report Start / End Date or any other expected Parameter by report (Refer to API Documentation). For our example we will enter below URL
   

   https://api.xero.com/api.xro/2.0/Reports/ProfitAndLoss?fromDate=2017-02-01&toDate=2017-02-28

4. Check Use Credentials and Select Connection we created in the Previous Section
5. In the Filter enter

   $.Reports[*].Rows[*].Rows[*]

6. Now go to 2D Array Transform Tab and configure like below. If your UI varies then use Properties window to edit property manually.
   Basically you need to edit
   Transformation Type:

   2-dimensional array

   Column Name Filter:

   $.Reports[0].Rows[*].Cells[*].Value

   Row Values Filter:

   $.Cells[*].Value

7. Now Click Preview Data. If you get error Column ” does not belong to table nosqldata. Then most likely you using old version with Known Issue (i.e. v2.9.7.10822 or older). Get latest version for fix. For workaround of this issue you must follow next section to specify columns manually. Columns must match count.

Here is how it will look like.

Configure Transform Options for Xero Report API (Dynamic Column Title)

Read from Xero Report API using JSON Source in SSIS (Dynamic Columns Example)

Fixing Dynamic Columns Problem in for Xero Report API

Now let’s look at one common scenario which you have to think. SSIS data flow has fixed metadata so if your Report is generating Dynamic Columns each time based on Date Range then it will be trouble for you because all your columns may come as null if initial date range changes.

To address this issue you have to use Static Column Names as below.

Account (or sometimes blank) | Oct 08 | Sep 08 | Aug 08 | Jul 08 | YTD

NOTE: Use REST API Task to obtain Raw JSON and review First Row which contains Number of columns
For example if your API return some thing like below then you can use 5 columns (one Blank header + 4 Months)

"Rows": [
        {
          "RowType": "Header",
          "Cells": [
            { "Value": "" },
            { "Value": "28 Feb 18" }
            { "Value": "28 Mar 18" }
            { "Value": "28 Apr 18" }
            { "Value": "28 May 18" }
          ]
        },

Once we know how many columns we should expect from API response, we can perform following steps. We will now configure our JSON Source component to use Static Columns.

1. Drag new JSON Source from SSIS Toolbox
2. Right click JSON Source > Click on Properties (You can do below using UI too but in old version no option for that – See below this section)
3. Enter following values for property
   1. Filter: 

      $.Reports[*].Rows[*].Rows[*]

   2. ArrayTransformationType:

      TransformComplexTwoDimensionalArray

   3. ArrayTransCustomColumns:

      Col1,Col2,Col3,Col4,Col5,Col6,Col7,Col8,Col9,Col10,Col11,Col12,Col13

          (Assuming you have 12 months to compare)
   4. ArrayEnableCustomColumn:

      True

   5. ArrayTransRowValueFilter:

      $.Cells[*].Value

4. Now double click JSON Source to configure
5. In the Path / Web URL, enter API Report URL to call desired Report Start / End Date or any other expected Parameter by report (Refer to API Documentation). For our example we will enter below URL
   

   https://api.xero.com/api.xro/2.0/Reports/ProfitAndLoss?fromDate=2017-02-01&toDate=2017-02-28

6. Check Use Credentials and Select Connection we created in the Previous Section
7. In the Filter enter

   $.Reports[*].Rows[*].Rows[*]

8. Now click Preview.

For Old Version use Properties Grid

Configure Xero Report Data Extract using SSIS Property Grid (For Older Version)

For New Version Use UI (It has Specify Columns Manually Option)

Read from Xero Report API using parameters in JSON Source in SSIS (Static Columns Example)

Common Errors

Deployment to Production

Conclusion. What’s next?

In this article we have learned how to load data from Xero to SQL Server using SSIS ( drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from Xero REST API using OAuth. JSON Source makes it super simple to parsing complex / large JSON Files or any Web API Response into rows and column so you can load into database like SQL Server. Download SSIS PowerPack to try many other automation scenarios not discussed in this article.

Keywords:

Xero Integration with SQL Server | How to extract data from Xero in SSIS? | How to read data from Xero API? | Loading Xero Data into SQL Server. | Xero to SQL Server | SQL Server to Xero | SSIS Xero Integration

The post How to read/load data in Xero using SSIS appeared first on ZappySys Blog.

Google BigQuery is a fully managed Big Data platform to run queries against large scale data. In this article you will learn how to integrate Google BigQuery data into Microsoft SQL Server using SSIS. We will leverage highly flexible JSON based REST API Connector and OAuth Connection to import / export data from Google BigQuery API just in a few clicks.

If you are looking for a similar product inside Amazon AWS Cloud then check an article about Amazon Athena.

Prerequisites

Before we look into a Step-By-Step section to extract and load data from BigQuery to SQL Server let’s make sure you meet the following requirements:

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have basic familiarity with REST API concepts and Google BigQuery API. This post uses free Public Dataset to query BigQuery table so no billing is required to get your hands dirty 🙂 However, if you like to query your own dataset then make sure you have at least one BigQuery dataset and one table with some sample data created (this part does require billing enabled). Read a Google Quickstart article for more information on how to create a new BigQuery dataset and a table.

Understanding Google BigQuery Object Heirarchy

Google BigQuery has 3 main concepts below.

• Project
  • Dataset
    • Table
      • Query requests (each query creates a unique JobID – valid for 24 hours from which you can read data)

So in order to create BigQuery table you always need Project and then Dataset under that project. You can group objects around Project / Datasets. When you query you can supply fully qualified name of your table in FROM clause (e.g.

select count(*) from `bigquery-public-data.usa_names.usa_1910_2013`

So let’s get started.

Read data from Google BigQuery using SSIS

Basically you can query Google BigQuery data in two ways: In this article we will not cover 2nd method.

• Method-1: Query data using jobs/query method in BigQuery API. Use it if you expect to get a result in a fairly short amount of time. This API method generates a temp table which gets deleted after 24 hours. You can read data within that time frame using newly created JobId reference.
  
  NOTE: If you have records that sum up to more than 10MB of data or ~10,000 rows (assuming 1 row uses 1KB of data) then you need to proceed with two-step process as explained below.
• Method-2: Export SQL query result using jobs/insert method in BigQuery API. Use this method if you expect a query to take a long time to finish.

Now let’s see how to query Google BigQuery data using SSIS 🙂

Create Google API Project and Register OAuth App

To use any Google API firstly you have to finish two tasks:

1. Create Google API Project and obtain projectId (see next section).
2. Enable BigQuery API
3. Register your own Google OAuth App and obtain ClientId and Client Secret.

Check step-by-step instructions on how to create Google API Project and create OAuth App for Google. During instructions make sure you enable BigQuery API (screenshots may differ).

Get Google API Project ID

Once you have Google API Project created you can grab Project ID (see screenshot below). You will need Project ID in the next step when building API URL.

How to find Google API Project ID?

Configure OAuth Connection for Google BigQuery API

In order to call most of Google APIs you will need an OAuth App. If you want to use your own app then refer to how to register Google OAuth Application (Get ClientID and ClientSecret) article. For simplicity we will use Default OAuth App:

1. Right click inside Connection Managers area and click “New Connection…”
2. From the connection type list select “ZS-OAUTH” connection type.
   

   Create new SSIS OAuth API Connection Manager

3. For OAuth Provider select “Google”.
4. For OAuth App type options leave “Use Default OAuth App” selected. If you created custom OAuth App then you would need to select “Use Custom OAuth App” option.
5. For Scopes enter or select the following URLs (URL is just a permission name). Each URL must be in a separate line. For demo purposes we use only the first 3 scopes but we included a few more in case you like to test API in depth with a different permission set:
   

   https://www.googleapis.com/auth/bigquery
   https://www.googleapis.com/auth/cloud-platform
   https://www.googleapis.com/auth/cloud-platform.read-only
   https://www.googleapis.com/auth/bigquery.insertdata
   https://www.googleapis.com/auth/devstorage.full_control
   https://www.googleapis.com/auth/devstorage.read_only
   https://www.googleapis.com/auth/devstorage.read_write

6. Click Generate Token. Login using correct account if needed and then you will be prompted to click “Approve” OAuth App.
7. Once you click OK on the login screen you will see new tokens populated on the connection screen.
8. Click Test to make sure connection / tokens are valid and then click OK to save the connection.
   NOTE: Below screenshot uses Default App but we recommend you to to use Custom OAuth App (Your own Clientid / secret – Obtained in previous section)
   

   Create SSIS OAuth API Connection for Google BigQuery API

Start BigQuery Job and get JobId (Submit Query)

Once you have SSIS OAuth connection created for BigQuery API it’s time to read data from BigQuery. So, basically, there are two ways you can read BigQuery data: using query or insert method. For demo purposes we will use jobs/query method. If you want fire complex queries which can run for many minutes then refer to jobs/insert method.

If you are expecting more than 10MB of data (~10K records) then you have to split data extract in two steps:

1. Step-1 : Call jobs/query method using REST API Task to submit SQL query. This steps returns jobComplete: true if supplied query execution is done within timeoutMs parameter you included. By default its 10 seconds so if your query is going to take longer than 10 seconds then its good idea to add bigger timeout that way you dont need step#2. Once API call is done you will Get JobId in the response. Save that JobId in a variable for later use. This job result is valid for 24-hours only.
2. Step-2 (Optional) – You can add another optional REST API Task after previous step to Wait until Job is completed. For simplicity, this article will Omit setting up Status Check but its very simple….Use below settings
   1. Drag new REST API Task from toolbox. Connect it to Step-1 and double click to configure as below.
   2. Select URL From Connection mode,
   3. Select OAuth connection
   4. Enter URL as https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/jobs/{{User::JobId}} , Method: GET
   5. On Response Settings Tab, Select Response Content Type JSON and for content filter enter $.status.state
   6. On Status Check Tab, check Enable Status Check Loop and enter DONE in SuccessValue field
   7. Doing this setup will make sure we do not query data until Job Status is DONE (System keeps checking every 5 seconds)
3. Step-2 : Read Job result using JSON Source in a Data Flow Task.

Now lets see how to configure REST API Task to submit query and extract JobId (which will be used in next section).

1. In the Control Flow designer drag and drop ZS REST API Task from the SSIS toolbox.
2. Double click the task to configure it.
3. Change Request URL Access Mode to [Url from connection].
4. From the connection dropdown select OAuth connection created in the previous section.
5. Enter URL as below (replace YOUR-PROJECT-ID with a valid Project ID obtained in the previous section):
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-PROJECT-ID/queries

6. Select Method as POST.
7. Enter Body (Request Data) as below (change your query if needed; we used Public dataset for the demo): If you do not specify timeout then default is 10 seconds only. Also in this call we only care about JobID from response so we just added maxResults=10 because in 2nd step we will get all rows by doing pagination.
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000"
   }

   To use a Standard SQL query instead of Legacy SQL, add useLegacySql property and set it to false:
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000",
      "useLegacySql": false
   }

   For all possible parameters refer to jobs/query method documentation.
   NOTE: You can also supply location Parameter in the Body JSON to indicate where job should run. For non EU / US datacenters we suggest you to supply this parameter. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location.
   Example Use Of Location
   

   { "location":"us-east1",  "maxResults": 10, "query": "SELECT title FROM [bigquery-public-data:samples.wikipedia] LIMIT 10" }

8. Select Body Content Type as application/json.
9. Click Test Request/Response. If things go well you will see JSON content. Then just copy JobId from the response and save for later use:
   
10. Now go to Response Settings tab.
11. Select ContentType=Json.
12. Enter below expression to extract JobId from the response. You will need this Id to read SQL query output (this JobId is valid for 24 hrs):
    

    $.jobReference.jobId

13. Choose save response content to save to a variable. Select <New Variable…> option. When prompted give a name to your variable (i.e. vJobId) and for the value field paste JobId copied from the above step:
    
    
    NOTE: This JobId expires after 24-hrs so while you are designing a package and not running it then design time value is used for testing/previewing. So make sure your JobId is valid. If needed click Test Request/Response to grab a new JobId and update the variable with a new value so you can preview data without running a full package.
14. Click OK to save the UI.

Read BigQuery SQL result (for specified JobID)

Now let’s look at how to read data from BigQuery

Configure Google BigQuery Web Request (URL, Method, ContentType, Body etc.)

1. In the Control Flow designer drag and drop Data Flow Task from SSIS toolbox.
   
2. Double click Data Flow Task and drag and drop ZS JSON Source (For API/File) from SSIS toolbox.
   
3. Double click JSON Source to edit and configure as below
4. Enter URL as below. Replace YOUR-API-PROJECT-ID with the API Project ID obtained in the previous section.
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?maxResults=10000

   –OR– (Use below if your Job ran outside US / EU data center)
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?location=YOUR_REGION_ID&maxResults=10000

   NOTE: location Parameter indicates the geographic location of the job. This is Required parameter except for US and EU. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location. You can supply same location parameter for first step when you submit the query

   Failure to supply location parameter for non US /EU users may result in 404 NotFound Error.
   

5. Check Use Credentials option and select OAuth Connection Manager created in the previous section.
6. For HTTP Request Method select GET.
   

Read data from Google BigQuery from Temp Job result

Configure Filter

We need to transform a single JSON response into multiple rows so we need to apply a correct filter.

1. On JSON Source go to Filter Options tab.
2. In the Filter field enter $.rows[*] or click [Select Filter] button to browse hierarchy you want to extract.

Select Filter for JSON Response

Configure BigQuery API Pagination Settings

Most of modern APIs usually implement some sort of pagination technique so you get a part of data in each request rather than all in one go. Thus if you want more you can paginate through pages until the last page is reached. You can also read Understanding REST API Pagination in SSIS article to learn more about pagination in SSIS.

BigQuery API returns pageToken attribute in response JSON if more data is found for requested query result. You can then pass pageToken in the next URL in this format: http://my-api-url/?pageToken=xxxxxxxxxx

Now lets configure JSON Source to automate this pagination for us. On JSON Source go to Pagination Tab and enter the following two settings:

1. Set Next Link as $.pageToken.
2. Set Suffix for Next URL as &pageToken=<%nextlink%>.

See the screenshot below to get more clarity:

Configure BigQuery API Pagination on SSIS JSON Source

Configure Array Transformation

Now the last thing we have to configure is special 2-dimensional JSON array format used by BigQuery API:

1. On the JSON Source UI go to 2D Array Transformation tab.
2. Enter the following settings:
   1. For Transformation Type select Transform complex 2-dimensional array.
   2. For Column Name filter enter $.schema.fields[*].name.
   3. For Row Values Filter enter $.f[*].v.

JSON Array Transformation Options

Preview Data and Save UI

That’s it click Preview to see some data. If you entered sample JobID for your User::vJobID variable then you will see some data. In the next section we will see how to load Google BigQuery data into SQL Server. You can click Columns Tab to review Metadata.

Click OK to save UI and generate Metadata.

Configure Target – Load Google BigQuery data into SQL Server

Now you can connect your JSON Source to any target such as ZS Trash Destination or a real database destination such as OLEDB Destination.

If you wish to dump data from Google BigQuery to a SQL Server table then just perform the following steps:

1. Drag and drop OLEDB Destination from SSIS Toolbox. Rename it to something like SQL Server Table.
2. Double click on OLEDB Destination.
3. Click New to create a new connection > Configure connection > Click OK.
4. Click on New to create a new table > Rename default table name > Click OK.
5. Click on Mappings tab and click OK to save UI with default mappings.

Configure SSIS OLEDB Destination – Google BigQuery to SQL Server Import

Execute Package – Loading BigQuery data into SQL Server

SSIS Package Execution –
Loading Google BigQuery Data into SQL Server

Create / Delete Google BigQuery Dataset using API call

As we mentioned before, you need dataset before you can create a table. Most common way to create dataset is via User Interface but what if you like to automate from your SSIS Package or other workflow? Here is how you can create or delete. Basically you can use REST API Task to send CREATE or DROP command for Dataset object

Create Google Dataset

To create dataset configure REST API Task using below settings (We will call this API)

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/datasets

4. Request Method as POST
5. Enter Request Body as below (change YOUR_DATASET_NAME – e.g. TestDataset )
   

   {"datasetReference": { "datasetId": "YOUR_DATASET_NAME", "projectId": "{{User::ProjectId}}"} }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Dataset already exists then you can go to Error Handling Tab (See next section for screenshot) and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Delete Google BigQuery Dataset

For deleing dataset you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Create / Delete Google BigQuery Table using API call

Now let’s look at how to create /drop Table in Google BigQuery using API calls.

Create Google BigQuery Table

To send CREATE TABLE  SQL statement (DDL) we have to use same approach as we send normal SQL Query using this API call. So notice we used Standard SQL for this call by supplying  useLegacySql: false . DDL Statement must be all in one line

POST https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

Content-Type: application/json

>>>> BODY <<<<<

{
 "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
 "useLegacySql": false,
 "timeoutMs": 100000 
}

To create dataset configure REST API Task using below settings

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

4. Request Method as POST
5. Enter Request Body as below
   

   {
    "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
    "useLegacySql": false,
    "timeoutMs": 100000
   }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Table exists then you can go to enable Error Handling Tab and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Create Google BigQuery Table – API Call (Continue On Error Setting – Skip CREATE if Table / Dataset Already Exists)

Delete Google BigQuery Table

For deleing table you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Write data to Google BigQuery using SSIS – 1 Million row insert test (FAST)

Now let’s look at how easy it is to import data into Google BigQuery using SSIS. We will use the same OAuth connection we created before. To learn more about inserting data into BigQuery check tabledata/insertAll method documentation.

Make sure billing is enabled

Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

Streaming is not available via the free tier. If you attempt to use streaming without enabling billing, you receive the following error: BigQuery: Streaming insert is not allowed in the free tier.

As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call (Write  demo).

Configure OAuth Connection / Permissions

If you want to perform data insert operation in BigQuery using API calls then include the following scopes in your OAuth Connection Manager and generate a token (see our first section of this article – We already included scopes for Write operation but incase you didnt do then regenerate token with below scopes):

https://www.googleapis.com/auth/bigquery
https://www.googleapis.com/auth/cloud-platform
https://www.googleapis.com/auth/bigquery.insertdata

Create BigQuery Dataset (From UI)

For this demo first create a test dataset and one table under it like shown below (billing must be enabled on your Google API Project). To do that go to https://bigquery.cloud.google.com/welcome and configure them:

Insert data into BigQuery using API call

And here is a REST API example to insert data into a BigQuery Table:

Request URL:

https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

Request Headers:

Authorization: Bearer ya29.Gl0cBxxxxxxxxxxxxxxxxxxuEIhJIEnxE6GsQPHI
Content-Type: application/json

Request Body:

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
     {"json": {"RowId": 2,"CustomerName": "BBB"} }
   ]
}

Here is our data flow setup to achive very high throughput for Google BigQuery Data Load. We will show you how to insert one million rows in Google BigQuery in less than a minute based on below setup. We will use Multi Threading option and New Compression Option (Added in v3.1.4)

1. Dummy Data Source – Generate sample records
2. JSON Generator Transform – Generates JSON documents to send as POST request for above /insertAll API call.
3. Web API destination  – Call /insertAll API call to submit our data to BigQuery
4. (Optional) JSON Parser Transform – Parse Error Message for any response
5. (Optional) Trash Destination – Save any errors to text file for review

Google BigQuery Data Load Demo in SSIS – 1 Million Rows Insert with Multi Threads and Compression ON (Fast Upload)

Configure SSIS JSON Generator – Generate JSON for BigQuery Table Insert

Now let’s see how to build an HTTP request with JSON body and send it to BigQuery:

1. Drag Data Flow Task and double click on it.
   
2. Drag and configure your Source (for this demo we use Dummy Data Source with Customer Template). See previous section for configuration of Dummy Data Source.
   
3. Drag and drop ZS JSON Generator Transform to produce JSON from a database or file records. If your source is already sending a valid JSON then you can skip this step (e.g. SQL query is
   returning JSON). You can also read raw JSON from a very large file (new-line separated JSON) using JSON Source with Output as Raw Data option checked.
4. Connect Source to JSON Generator. Double click JSON Generator Transform to start configuring it like below.
5. Select Output Mode as Single Dataset Array and enter Batch Size 10000 (This is Max limit allowed by Google BigQuery API insertAll)
6. First right click on Mappings node and select Add Static Element
   

   Generate JSON for Google BigQuery InsertAll API request – Batch 10000 rows in a single API call

7. Enter Name as kind and value as bigquery#tableDataInsertAllRequest
   
   

   JSON Generator – Add Static Element

8. Now right click on Mappings node and select Add Document Array option
   

   JSON Generator – Add Document Array

9. Enter rows as array title and click OK
   

   JSON Generator – Name array

10. Select newly added array node and right click > Add unbound nested element  enter Output alias as json and click OK.
    

    JSON Generator – Add unbound nested element

11. Select json node and right click > Select Add Elements below this node and select multiple columns you like to send to BigQuery. Click OK to save.
    

    JSON Generator – Add Multiple Elements

12. Now let’s preview our JSON (Copy preview JSON to try in the next step – Web API destination)
    NOTE: Table name and column names are case-sensitive so make sure your JSON attribute matches exact same way.
    Here is the finished JSON Structure for next Step
    

    Sample JSON Request body for Google BigQuery insertAll API request

13. Click OK to save UI.

Configure SSIS Web API destination – Insert data into BigQuery Table

Once you have Input JSON prepared,  now let’s configure destination for BigQuery.

1. Drag and drop ZS Web API Destination.
2. Connect your JSON Generator Transform to Web API destination.
3. Configure general properties:
   

   SSIS Web API Destination –
   Configure for BigQuery Data load

4. Make sure to enter URL in this format:
   

   https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

   Make sure to replace 3 parts in above URL (MY_PROJECT_ID, MY_DATASET_ID, MY_TABLE_ID) with actual values from your Google Project and BigQuery dataset/table configuration.
5. Now you can enable Compression and Multiple Threads for higher throughput as below.
   NOTE: Compression Property was added in v3.1.4 so you may not see it if you have older version.
   

   Google BigQuery Data Loading Performance Optimization Options – Enable Multiple Threads and Compression Options

6. If you want to try test insert request from UI without running full package then go back to first tab and edit Body (Use Sample JSON generated by previous JSON Transform – You can grab from JSON Preview Panel on Generator Transform).Click Test Request / Response and confirm Success as below. You can go back to your BigQuery Portal and check one row is inserted after our test click. If everything looking good then run full package to insert all records.Sample JSON for Body:
   

   {
     "kind": "bigquery#tableDataInsertAllRequest",
     "rows": [
        {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
        {"json": {"RowId": 2,"CustomerName": "BBB"} }
      ]
   }

   

   Test Google BigQuery Table Insert – SSIS Web API Destination UI

7. Hit OK to save UI.
8. Run the package and verify data in Google BigQuery Console:
   

   Loading data into Google BigQuery using SSIS

Error Handling for BigQuery Data Load (Bulk Insert API Calls)

There will be a time when some records you insert may not go well in Google BigQuery. In such case you can read output from Web API destination and parse further with JSON Parser Transform. Check for certain values in the output. You must use JSON Parser after Web API destination (Connect Blue Arrow from Web API destination – Since its Soft Error it won’t redirect in Red Arrow ).

For example here is the sample JSON in POST Body for testing which produces error due to bad column name. When bad row found in batch all records will be rejected. Notice that error returns index of record in batch so you can identify which row went bad. It also returns column name in location attribute.

NOTE: Bad column name in 2nd record

Test Body (Bad):

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RecordID": 1,"CustomerID": "X1"} }, 
     {"json": {"Bad_Column": 2,"CustomerID": "X2"} }
     {"json": {"RecordID": 3,"CustomerID": "X3"} }, 
   ]
}

Response (For Bad Input):

{
  "kind": "bigquery#tableDataInsertAllResponse",
  "insertErrors": [
    {
      "index": 1,
      "errors": [
        {
          "reason": "invalid",
          "location": "bad_column",
          "debugInfo": "",
          "message": "no such field."
        }
      ]
    },
    {
      "index": 0,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    },
    {
      "index": 2,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    }
  ]
}

To configure error detection perform following steps.

1. Drag and drop ZS JSON Parser Transform after Web API destination
2. Click on Web API destination. Connect Blue arrow  to JSON Parser Transform
3. Configure JSON Parser Transform like below
4. Connect JSON Parser TRansform to some Destination to save error information (e.g. SQL Table or Trans destination)

Handling BigQuery Insert Errors in SSIS

Other Common Errors in BigQuery API calls

In this section we will talk about many common API errors in BigQuery.

Error: The project XXXXXXX has not enabled BigQuery

Sometimes you might get below error. To fix this error make sure you go to your Project and Enable BigQuery API

Status Code: BadRequest

Response Body: {
  "error": {
    "code": 400,
    "message": "The project bigquerytest-281915 has not enabled BigQuery.",
    "errors": [
      {
        "message": "The project bigquerytest-281915 has not enabled BigQuery.",
        "domain": "global",
        "reason": "invalid"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}

Error: 404 – Not Found: Table / Job xxxxxx

• Make sure you are setting GET request and not POST.
• Also make sure your jobId is valid because it expires after 24 hours.
• Make sure project-id supplied in URL is valid ID  (DO NOT Specify Alias, use internal ID for project)
• Make sure you supplied location parameter if your outside EU / US region. Reading data using this API call might fail if you failed to supply location in URL

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "notFound",
    "message": "Not Found: Table xxxxxx"
   }
  ],
  "code": 404,
  "message": "Not Found: Table xxxxxxx
 }
}

Error: 403 – Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier

If you trying call certain APIs on sandbox mode or free tier then you might get below error. To overcome this error enable billing on google bigquery. As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call.

Response Url: https://www.googleapis.com/bigquery/v2/projects/bigquerytest-281915/datasets/TestDataset/tables/Table1/insertAll

Status Code: Forbidden

Response Body: {
  "error": {
    "code": 403,
    "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
    "errors": [
      {
        "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
        "domain": "global",
        "reason": "accessDenied"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}

Debugging Web API Requests

If you need to debug actual requests made to Google server then you can use a tool like Fiddler. It’s a very handy tool to troubleshoot JSON format issues. It will allow to see how a request is made to a server.

Using Fiddler to debug Google BigQuery API requests in SSIS

Common Errors

Deployment to Production

Download Sample Package

Click Here to Download SSIS Sample Package – Google BigQuery API Read Write Create Delete (SSIS 2019, 2017, 2012)

Conclusion. What’s next?

In this article we have learned how to load data from Google BigQuery into SQL Server using SSIS (drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from Google BigQuery REST API using OAuth. JSON Source Connector makes it super simple to parse complex/large JSON files or any Web API response into rows and columns so you can load data into a database, e.g. SQL Server database. Download SSIS PowerPack to try many other automation scenarios that were not discussed in this article.

Keywords:

Google BigQuery Integration with SQL Server | How to extract data from google bigquery in SSIS? | How to read data from Google BigQuery API? | Loading BigQuery Data into SQL Server. | BigQuery to SQL Server | SSIS Google Big Query Integration | SSIS Google BigQuery Import  JSON File | SSIS Google BigQuery Export data

The post How to read / write data in Google BigQuery using SSIS appeared first on ZappySys Blog.

Elasticsearch is a powerful engine that allows you to store, aggregate, and, most importantly, search data in a very analytical way. In this tutorial, you will learn how to bulk load data from SQL Server to Elasticsearch with SSIS (part of SQL Server) and ZappySys PowerPack.

The scope of this article will be to show how to import records from SQL Server into Elasticsearch index as JSON documents using Elasticsearch Bulk API. In a nutshell, we will retrieve IDs and names of the products from a SQL table, transform each row into a JSON and lastly, index each JSON in Elasticsearch under record’s corresponding ID. This tutorial is going to be your stepping stone to use any Elasticsearch API as a destination.

We will be using these SSIS components of PowerPack to make things work:

	JSON Generator Transform
	Web API Destination

Let’s begin?

Prerequisites

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Northwind database deployed on your machine (we will use it to load data from; download it).
4. Elasticsearch instance up and running.
5. ZappySys SSIS PowerPack installed.

<New ZS-OAUTH Connection>

Step-by-Step – Bulk loading data from SQL Server to Elasticsearch with SSIS

Load data from SQL Server database first

In this section, you will perform several basic steps to start the package: load data from SQL Server and prepare it for further steps.

1. Create a new SSIS package and drag a Data Flow Task into the Control Flow from the SSIS Toolbox.
   

   Dragging and dropping Data Flow Task into Control Flow

2. Open Data Flow Task and then add OLE DB Source.
3. Configure OLE DB Source to take data from Products table, Northwind database.
4. Choose ProductID and ProductName as columns you will use.
   

   Load data to Elasticsearch using SSIS and ZappySys Web API Destination component.

Transform data into JSON documents

To add data into Elasticsearch firstly we need to prepare it with JSON Generator Transform. It will help us convert table rows into JSON documents.

JSON String to Create Index data for the Bulk API call

1. Close the window and then drag Derived Column component from the SSIS Toolbox.
2. Furthermore, Add two new columns and name them 

   RowHeader

    and 

   RowFooter

   :
   

   Derived Column configuration to add prefix and header and footer to a JSON

   In the Expression column give them values:

   • RowHeader:
     

     "{ \"index\": { \"_index\": \"shop\", \"_type\": \"products\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n"

   • RowFooter:  

     "\n"

3. Now drag JSON Generator Transform from the SSIS Toolbox and open it.
4. Right mouse click on item Mappings and select

   Add Element(s) (Below this node)

   :
   

   Convert SQL table rows into JSON using JSON Generator Transform

5. Once a dialog appears, select

   ProductName

    as Source Column and

   name

    as Output Alias:
   
   

   Map table column to JSON property using JSON Generator Transform

6. You should get a similar view:
   

   JSON Generator Transform configuration to create JSON from a SQL table

7. Then go to Header / Footer tab and:
   • Select

     Direct String

      option.
   • Input

     <%RowHeader%>

      in Header String option.
   • Input

     <%RowFooter%>

      in Footer String option.
   

   JSON Generator Transform configuration to convert rows into JSON suitable for Elasticsearch Bulk operation

8. Close the window.

JSON String to Update Index data for the Bulk API call

From the above section “JSON String to Create Index data for the Bulk API call” follows all the steps From Step-1 to Step-8.

Just for (2nd step) Step-2 use below expression for  

RowHeader

RowFooter

Derived Column configuration to add prefix and header and footer to a JSONIn the Expression column give them values:

• RowHeader:
  

  "{ \"update\": { \"_index\": \"shop\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n{ \"doc\" : "

• RowFooter:  

  "}\n"

JSON String to Delete Index data for the Bulk API call

From the above section “JSON String to Create Index data for the Bulk API call” follows steps Step-1 and Step-2.

Just for (2nd step) Step-2 use below expression for  

RowHeader

Derived Column configuration to add prefix and header and footer to a JSON

In the Expression column give them values:

• RowHeader:
  

  "{ \"delete\": { \"_index\": \"shop\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n"

Finally, Elasticsearch & SQL Server integration

1. Now drag Web API Destination SSIS component into the Data Flow and open it for editing.
2. In Select Connection section press

   <New ZS-HTTP Connection>

   .
   NOTE: If your ElasticSearch instance is hosted as AWS Managed Service then select  

   <New ZS-OAUTH Connection>

   . rather than ZS-HTTP. When OAuth UI launches select AWS v4 Provider. For more information on calling REST API on AWS check this article.
3. Once HTTP Connection Manager window opens configure connection to your Elasticsearch instance:
   • Set Web Url, which points to your Elasticsearch instance.
   • Set Credentials Type to

     Basic - UserID/Password

      (or other appropriate authentication method).
   • Finally, set User Name and Password:
     
     

     Configure SSIS HTTP Connection to connect to Elasticsearch

4. Close configuration window. Now it’s time to configure Web API Destination.
5. All that is separating you from getting those rows into Elasticsearch is:
   • Setting Input Column for Body to

     ZS_JSON_OUT

      for Create and Update Index Data and for Delete Index Data Input Column for Body to

     RowHeader [Derived Column]

     .
   • Setting URL to

     http://localhost:9200/_bulk

     .
   • Setting HTTP Request Method to

     POST

     .
   

   Index SQL Server Database data in Elasticsearch using SSIS component Web API Destination

6. Then open Batch Settings (For Body) tab:
   • Enable batch submission.
   • Set Body Batch Size, e.g. 1000. Make sure this number is even, otherwise you may get into problems (Elasticsearch Bulk request has special JSON request body format, as you perhaps noticed).
   

   Web API Destination batch settings configuration

7. Close the window and run the package! You should see green lights, telling you everything is OK:
   

   Successful data load from SQL Server to Elasticsearch

8. We can also query Elasticsearch by using one of its Search APIs – URI Search to see if we successfully indexed data:
   

   SQL Server data index in Elasticsearch

9. Rows number match in SSIS and in search results, thus everything is good. You are done.

What if I want more? After upserting data use Web API Destination further

Let’s say you have a requirement to do something with rows that were freshly indexed in Elasticsearch. Then you will need somehow to distinguish between created and updated records in Elasticsearch. And that won’t be difficult because Web API Destination acts not only as Destination but as Transformation as well. So one thing you have to do is connect JSON Parser Transform downstream to Web API Destination. It will parse Elasticsearch HTTP JSON response – which is returned by Web API Destination – into columns, which you can later easily redirect using Conditional Split:

1. Add JSON Parser Transform and connect it to Web API Destination:
   

   Use JSON Parser Transform to get Elasticsearch HTTP JSON response when integrating SQL Server & Elasticsearch

2. Make sure you have selected

   ResponseText

    as Select Input JSON Column.
3. Then go to Bulk API and copy/paste “the result of this bulk operation” JSON response:
   

   Using JSON Parser Transform to parse JSON response from REST HTTP request

4. Set

   $.items[*]

    as the filter.
5. Optional step. You won’t find a response sample for every Elasticsearch API call. In that case, you may want to use Postman to make an HTTP request to Elasticsearch and get a sample response:
   

   Use Postman to make a request to Elasticsearch and get a sample response to be used in JSON Parser Transform

   NOTE: Don’t forget to set the username and password in Authorization section!
6. Then use a Conditional Split and Trash Destination to redirect the rows:
   
   Input these clauses:
   • Inserted:
     

     REPLACENULL([index.result],"") == "created" || REPLACENULL([create.result],"") == "created"

   • Updated:
     

     REPLACENULL([index.result],"") == "updated" || REPLACENULL([create.result],"") == "updated"

7. As a result, new index records will be redirected to one destination, while updated records – to the other:
   

Overall you can use Web API Destination HTTP JSON response for other useful things as well, e.g. determine on how many replica shards record was indexed. JSON response will depend on which Elasticsearch API and which method you use.

Delete Index by making an API call.

If you want to delete the index by making Delete API call you can. Let’s make that call using the Rest API Task. Configure it like below screen and click on the Test Request button.

Rest API Delete Method

Things went bad: Error handling & debugging

Incidentally, you may incorrectly construct JSON for Web API Destination body. Elasticsearch nodes may go offline or go out of memory. In any case you may want to know when that happens and take actions accordingly. For that purpose you have to redirect failed requests from Web API Destination to some other destination:

1. Add a Derived Column above Web API Destination with expression

   (DT_WSTR,4000)ZS_JSON_OUT

    and name it

   JsonAsString

   . This will let you see what JSON you are actually passing.
2. After that, add database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don’t forget to set

   Redirect row

    option for both, Error and Truncation columns:
   
3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled in Elasticsearch

Call ElasticSearch API hosted on AWS

If your ElasticSearch instance is hosted as AWS Managed ElasticSearch Instance then select  

<New ZS-OAUTH Connection>

Call AWS Hosted ElasticSearch REST API in SSIS (V4 Request Signing)

Common Errors

Deployment to Production

Download a sample package

Be sure to download a sample SQL Server 2008 SSIS package, in case you want to try it right away (you can upgrade it to a higher version).

Conclusion. What’s next?

In conclusion, in this article we have learned how to bulk load data from SQL Server and upsert it in Elasticsearch index. We used JSON Generator Transform to help us convert database table rows into JSON documents. In addition, Web API Destination was very helpful in automatically creating and making HTTP requests to Elasticsearch instance and indexing our data as a result. This tutorial was quite simple and straightforward, yet it is a good starter to use other Elasticsearch APIs and their features. From now on Elasticsearch reference is your best friend in learning what APIs exist, what are their methods and how HTTP requests should look like.

Keywords

How to import data from SQL Server to Elasticsearch | How to load data into Elasticsearch | Using Elasticsearch to index SQL Server | Elasticsearch and SQL Server integration | How to use Elasticsearch together with SQL Server | Upsert SQL Server data into Elasticsearch

The post Loading data from SQL Server to Elasticsearch with SSIS appeared first on ZappySys Blog.

HubSpot is a popular platform where CRM, Marketing and Sales meet in one place. It’s a good piece of service, yet sometimes you need to integrate it with your other system. In this article you will learn how to pull contacts from HubSpot API and load them into a SQL Server Database using SSIS and ZappySys SSIS PowerPack.

Prerequisites

1. HubSpot Account registered (an account you will want to get data from).
2. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
3. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
4. An existing SQL Server Database and table created (we will use it to load data into).
5. ZappySys SSIS PowerPack installed.

Various methods to call HubSpot API

HubSpot API supports two kinds of authentication:

1. OAuth 2.0.
2. API Key in URL.

API Key in URL is simple, with minimal configuration, but it ought to be used only for development and testing purposes.

In this tutorial we will show how to authenticate using OAuth 2.0, which you should use in your production environment.

Step-by-Step – Get data from HubSpot API into SQL Server database

1. Create HubSpot Developer Account (https://app.hubspot.com/signup/developers). It’s an account you will need to use to create HubSpot Apps.
2. Login with your HubSpot Developer Account (https://app.hubspot.com/login).
3. Select your newly created HubSpot Developer Account:
   

   Logging into HubSpot – Select HubSpot developer account

   NOTE: Account name should contain “hubspot-developers” text.
4. Register HubSpot OAuth App for REST API Call for use with OAuth 2.0 by simply pressing “Create application” button:
   

   Register HubSpot OAuth App for REST API Call for use with OAuth 2.0

5. Name the application (e.g. “ContactsApp”) and choose “Private” to make the application hidden from the outer and dangerous world 🙂
   

   Name HubSpot REST API App which will be used from within SSIS

6. Press “Edit” button on the app you just created.
7. Write down Client ID and Client secret. We will need them later for authentication:
   

   Write down Client ID and Client Secret which will be used in OAuth Connection SSIS component

8. Open SSIS Designer and create a new package.
9. Drag a new Data Flow task from SSIS Toolbox into the Control Flow:
   
   

   Dragging and dropping Data Flow Task into Control Flow

10. Edit Data Flow task by double clicking. Then drag ZS JSON Source and OLE DB Destination components from SSIS Toolbox into the Data Flow, but don’t connect them yet:
    
    

    Drag ZappySys JSON Source SSIS component into the Data Flow

11. Start editing JSON Source component:
    1. Set Access Mode to File path or web URL.
    2. Set Web URL to 

       https://api.hubapi.com/contacts/v1/lists/all/contacts/all?count=100

       .
       
       NOTE: We are adding URL parameter with value

       ?count=100

       , because by default HubSpot API returns 20 contacts and we want to maximize result set and minimize the number of web requests.
    3. Press Select Filter and select contacts node or simply enter

       $.contacts[*]

         into the text box.
    4. Check Use Credentials checkbox.
    5. Select <New ZS-OAUTH Connection> menu item to create a new connection:
       

       JSON Source SSIS component configuration – Get data from HubSpot API using OAuth authentication

12. Configure OAuth 2.0 connection for HubSpot API:
    1. Select HubSpot as OAuth Provider.
    2. Select “Use Custom OAuth (App created by you)” option.
       
       NOTE: You can also use Default OAuth App option for a quick test. Select it and press “Generate Token” button after you enter the scope.
    3. Paste Client Id and Client Secret you also saved in Step #7.
    4. Input

       contacts

        as our scope.
    5. Finally, press “Generate Token” button:
       

       OAuth Connection SSIS component configuration – Connect to HubSpot API using OAuth 2.0

13. Select your ordinary HubSpot account you want to get contacts from:
    

    OAuth Connection SSIS component configuration – Select a HubSpot account to get data from

    NOTE: Account name should NOT contain “hubspot-developers” text.
14. Grant access to your HubSpot App:
    

    OAuth Connection SSIS component configuration – Grant access to HubSpot API App

15. You will get a prompt to save Access and Refresh tokens to a backup file. Press “Yes” to save them.
    
    NOTE: You will need to pass

    Refresh Token

     as the parameter, when you run the package in another machine. Check Deployment to Production section for more info.
16. Configure HubSpot API pagination
    1. Set Next Link/Cursor Expression to

       $.vid-offset

         by pressing “…” button or simply pasting the value into the text box.
    2. In the same way set Stop Indicator/Attribute to 

       $.has-more

       .
    3. Input 

       false

        as Stop indicator value/Regular expression.
    4. Set Suffix for Next Url to

       &vidOffset=<%nextlink%>

       .
       

       JSON Source SSIS component pagination configuration for HubSpot API call

17. Press “Preview” button to… well, obviously to preview the results 🙂
    

    JSON Source SSIS component data preview of HubSpot API call result

18. Close “Data Preview” dialog and click “OK” button to save and close JSON Source component.
19. A dialog Metadata Scan Options should appear, to which you should respond “Ok” 🙂
20. Now you can connect JSON Source component with OLE DB Destination:
    
    

    Connect JSON Source SSIS component with OLE DB Destination for HubSpot data.

21. Now modify OLE DB Destination to have a connection to a SQL Server and point to a table, where you want contacts to be stored.
22. It’s time to map the columns of JSON Source with destination table columns. With your mouse simply connect corresponding columns or simply map Input Column with Destination Column. Press “OK” to close the window.
    

    Map JSON Source SSIS component columns with destination table columns you will store HubSpot data in

23. Finally, funtime! Press “F5” to execute the package. You should see a similar view as below. The number you see is the number of contacts were loaded from HubSpot API to your table:
    

    The result of getting HubSpot API data with SSIS

API Rate Limit (Throttling)

HubSpot places limits on using its API. Basically, it doesn’t allow you to make endless number of calls per second. Also it has a ceiling for requests made per day. So if you have paging enabled in JSON Source component, it will make as many requests as it has to retrieve all data. And in this step you may get an error, telling that you exceeded calling rate limit of the API. To avoid that, simply go to the Throttling tab and set the amount of milliseconds component has to sleep after making each call:

HubSpot API calls throttling configuration

Making things Dynamic (e.g. URL)

The URL we used (https://api.hubapi.com/contacts/v1/lists/all/contacts/all) in addition to

count

&showListMemberships=true

true

https://api.hubapi.com/contacts/v1/lists/all/contacts/all?showListMemberships={{User::ShowMemberships}}

Common Errors

For detailed instruction to fix common metadata related errors read this article.

Truncation related error

Most common error you may face when you run SSIS Package is truncation error. During design time only 300 rows scanned to detect datatype but at runtime you may extract far more records than 300. So its possible that you will get longer strings than initially expected. If you get error just copy that to notepad and read error carefully. It will explain the steps needed to fix the problem. Basically you can edit Columns on JSON Source UI > Change Data Length or Datatype (if you need more than 4000 chars change to the type to DT_NTEXT and click on Column Name cell to save change). After you edit default settings then make sure to check Lock option (found all on the right side in same row).

Authentication related error (happens when you run package from SQL Job Agent)

So many times when you run from Visual Studio it works fine but when you deploy to Production and run from SQL Job it fails. If this is your issue then check you passing Token (i.e. Password Property of HTTP connection) using some sort of parameter approach (e.g. from command line or from config file). By default password is encrypted using your account so when you run it from SQL Job it will fail to decrypt it. See more on Package Protection Level.

Deployment to Production

When you are ready to deploy the package to Production consider passing Refresh Tokens and other sensitive data as passwords via SSIS parameters. In this way you don’t face authentication related error described in above section. See more on Package Protection Level.

Download Sample Package

Be sure to download a sample SQL Server 2012 SSIS package, in case you encounter problems that you can’t go through in the step-by-step part.

Conclusion

In this article we learned how to call HubSpot API RESTful method and retrieve all contacts. Actually, HubSpot API consists of many small APIs and we used Contacts API method particularly. If you need anything else than simply retrieving the contacts of your account, delve into the vastness of HubSpot API methods!

Also it’s worth mentioning that JSON Source and OAuth Connection SSIS components saved us some valuable time, because we avoided using default SSIS Script Component and doing all the hard work ourselves.

Keywords

How to fetch data from HubSpot API | Pull data from HubSpot API into SQL Server | Call HubSpot API with SSIS | Sync HubSpot data to SQL Server | HubSpot data integration with SQL Server database | Load data from HubSpot REST API using SSIS

The post How to get data from HubSpot API with SSIS appeared first on ZappySys Blog.

Google DoubleClick is one of the most popular platforms for Advertisers. Many times you have need for custom integration or Automation for many operations. In this article we will learn how to Call Google DFP API (i.e. DoubleClick for Publishers) without coding using SSIS (Microsoft SQL Server Integration Services). In our previous article we discussed how to Integrate Google AdWords API using SSIS (Click here).

If you are new to SSIS then no worry there are plenty of tutorials for ssis

To achieve Custom integration for DoubleClick (DFP) API we will use SSIS XML Source. XML Source is part of SSIS PowerPack which has 45+ connectors and Tasks.

Prerequisite

Before you can finish tasks explained in this article you have to finish below steps. This article assumes you have basic knowledge of SSIS (SQL Server Integration Services)

1. Make sure you SSIS designer installed. Sometimes its referred as BIDS or SSDT (Get from here)
2. Download SSIS PowerPack
3. Download SoapUI (Its free third party tool to test SOAP API). SoapUI can help you to generate XML Request BODY easily from WSDL file provided by API Vendor. You can also test your service by supplying parameters.
4. Optional – Another very useful Free tool is Fiddler. You can use it to see raw request/response (Check this article)

Download DFP API Example SSIS Package

Click here to DFP API download Sample for SSIS SSIS Package (SSIS 2012, 2014, 2016)

Screenshot of Sample Package:

Example SSIS Package – Google DFP API Integration (Read / Write data in Google Double Click for Publisher) – No Coding required

Making your first DFP API Call using SSIS

Once you install PowerPack you are ready to execute your very first Google DFP API Call.

NOTE: If you don’t have any DFP network for test then see next section (Explains how to create test network for testing).

Lets look at how to call Google DFP API step-by-step.

1. Create new SSIS Project and open package designer
2. Drag ZS REST API Task from SSIS control flow toolbox and drop it on designer.
   
   REST API Task is useful when you want to call SOAP/REST API but not necessarily parse response into rows and columns. We will look at JSON Source later in this article which can actually parse response into rows and columns (Useful to load Google DFP data into SQL Server or other RDBMS / flatfile )
3. Double click REST API Task to configure. Select Access mode to [Url from Connection]
4. Enter following URL in the URL Textbox
   

    https://ads.google.com/apis/ads/publisher/v201702/NetworkService

5. From the connection dialogbox select ZS-OAUTH to create new OAuth Connection for DFP API
6. On OAuth Connection dialogbox select Google from Provider dropdown. In this demo we will use Default OAuth App but you can register your own google OAuth app if you wish to use Custom OAuth option from UI.
7. Enter following Scopes in the Scopes textbox (or select manually by clicking Select Scopes button). Scope is nothing but permission for API (In our case View/Manage DFP data and Read/Write Report Files to Cloud Storage).
   

   https://www.googleapis.com/auth/dfp
   https://www.googleapis.com/auth/devstorage.read_only
   https://www.googleapis.com/auth/devstorage.write_only
   https://www.googleapis.com/auth/cloud-platform.read-only
   https://www.googleapis.com/auth/devstorage.full_control
   https://www.googleapis.com/auth/cloud-platform
   https://www.googleapis.com/auth/devstorage.read_write

8. If you are not going call ReportService API to generate Reports in CSV format then you will need only one scope
   

   https://www.googleapis.com/auth/dfp

9. Click Generate Token button. You will see browser popup for login and then Accept option to confirm permissions like below. Once you don’t it will populate tokens and prompt to save tokens to backup file. Secure token backup to safe place.
   

   SSIS OAuth Connection – Connect to Google DFP API (DoubleClick SOAP Web Service)

10. Once done Click Test Connection and if its green then Click OK to save connection
11. Once you are back to REST API Task UI then select Method to POST
12. Select Request Body Content Type to XML (i.e. text/xml)
13. Click edit button next to Request Body and enter following XML.
    

    <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
       <soapenv:Header>
          <v20:RequestHeader/>
       </soapenv:Header>
       <soapenv:Body>
          <v20:getAllNetworks/>
       </soapenv:Body>
    </soapenv:Envelope>

    If you are wondering how did we get above XML fragment then read next section about using 3rd party tool called SoapUI. Also we have detail article for calling SOAP request. Check this article on how to use SoapUI (free 3rd party tool) to create SOAP request Bodyfrom WSDL. DFP has many API endpoints for different actions (e.g. NetworkService , InventoryService , …). For each service you have different Api URL and different WSDL. In the next section you can learn how to create correct SOAP Body (for POST) using SoapUI tool.

    For example if you using DFP NetworkService API for version v201702 then your help page URL would be like this
    https://developers.google.com/doubleclick-publishers/docs/reference/v201702/NetworkService
    And on the same page you will see WSDL link below
    https://adwords.google.com/api/adwords/mcm/v201609/ManagedCustomerService?wsdl
    (Just download that WSDL XML and save to local disk then use with SoapUI to generate XML body to submit for any DFP API call.)

14. Once you done with above steps, your Task Configuration will look like below.
    

    SSIS Rest API Task – Call Google DoubleClick API – Get all DFP Networks

15. Now click Test Request button. You will see Response popup with below content if its successful (Scroll at the bottom on response form and you will see XML data) .. see below
    

    ------------------------------------
    Request
    ------------------------------------
    POST https 1.1 ==> /apis/ads/publisher/v201702/NetworkService
    Host: ads.google.com
    
    >>>> HEADERS <<<<<
    Authorization: Bearer ya29.Glz-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    Content-Type: text/xml;charset=UTF-8
    User-Agent: ZappySysApp
    Host: ads.google.com
    Content-Length: 291
    Expect: 100-continue
    Accept-Encoding: gzip, deflate
    Connection: Keep-Alive
    
    ------------------------------------
    Response
    ------------------------------------
    POST 1.1 200 OK==> OK
    
    >>>> HEADERS <<<<<
    Content-Encoding: 
    X-Content-Type-Options: nosniff
    X-Frame-Options: SAMEORIGIN
    X-XSS-Protection: 1; mode=block
    Alt-Svc: quic=":443"; ma=2592000; v="35,34"
    Transfer-Encoding: chunked
    Cache-Control: private, max-age=0
    Content-Type: text/xml; charset=UTF-8
    Date: Sun, 26 Feb 2017 01:18:32 GMT
    Expires: Sun, 26 Feb 2017 01:18:32 GMT
    Server: GSE
    
    >>>> Cookies <<<<<
    
    >>>> CONTENT <<<<<
    
    <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    	<soap:Header>
    		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
    			<requestId>27ab3d2e4e1fcb8f444b317d0b0f871d</requestId>
    			<responseTime>50</responseTime>
    		</ResponseHeader>
    	</soap:Header>
    	<soap:Body>
    		<getAllNetworksResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
    			<rval>
    				<id>551196</id>
    				<displayName>XFP sandbox property</displayName>
    				<networkCode>238897396</networkCode>
    				<propertyCode>ca-pub-2799179143725683</propertyCode>
    				<timeZone>America/New_York</timeZone>
    				<currencyCode>USD</currencyCode>
    				<effectiveRootAdUnitId>237897516</effectiveRootAdUnitId>
    				<isTest>true</isTest>
    			</rval>
    		</getAllNetworksResponse>
    	</soap:Body>
    </soap:Envelope>

     

How to generate SOAP Request Body using SoapUI tool

This section describes how to create DoubleClick SOAP API Request Body (XML fragment) for any DFP SOAP API call described anywhere in this article.

1. Download SoapUI (Its free third party tool to test SOAP API). SoapUI can help you to generate XML Request BODY easily from WSDL file provided by API Vendor. You can also test your service by supplying parameters.
2. Now download WDSL xml for appropriate API calls you want to make. Assume that you want to call  getAllNetworksResponse API found under NetworkService. For that first navigate to help file page and open help page. Make sure you select correct version from help navigation. Copy WSDL link and open in new browser window. Save XML to local disk (e.g. c:\api\dfp_networkservice_wsdl.xml (See below screenshot how to find DFP API WSDL link)
   

   How to get Google DFP API WSDL URL (Used to generate POST Body)

3. Once SoapUI is downloaded and installed click File menu > Create New SOAP Project option
4. Name your project (e.g. DFP API) and specify WSDL URL or File Path if it was saved locally (e.g. c:\api\dfp_networkservice_wsdl.xml) and click OK
5. Now navigate to API Action for which you would like to get Body. Click Request node (If missing create new) and edit request. You may see XML like below (If some optional parameters not visible in XML then click Re-create request with optional parameters button from toolbar)
   

   Call Google DFP API – Generate SOAP Request Body XML using SoapUI (Google DoubleClick Web Service)

6. Edit necessary parameters from above XML code and copy to Body of SSIS REST API Task.

Creating test network – DFP Sandbox

Very first step we recommend before testing DFP API call is create test network. You can call DFP API against LIVE network but not recommended if you are calling create/update API.

To create test network you can follow exact same steps described in the previous section except the Body step (Step#12). Use following Request body to call makeTestNetwork command.

API makeTestNetwork – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/NetworkService
Body (see below):

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
   <soapenv:Header>
      <v20:RequestHeader/>
   </soapenv:Header>
   <soapenv:Body>
      <v20:makeTestNetwork/>
   </soapenv:Body>
</soapenv:Envelope>

Once you click Test you may receive following response. Note your network code (e.g. 1122334455 from below). This network code is used in pretty much all DFP API to view or manage ad network related items or properties.

API makeTestNetwork – Response

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<requestId>ba61d79efee325bf4b5aa45c46836021</requestId>
			<responseTime>1021</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<makeTestNetworkResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<rval>
				<id>551196</id>
				<displayName>XFP sandbox property</displayName>
				<networkCode>1122334455</networkCode>
				<propertyCode>ca-pub-2799179143725683</propertyCode>
				<timeZone>America/Los_Angeles</timeZone>
				<currencyCode>USD</currencyCode>
				<effectiveRootAdUnitId>237897516</effectiveRootAdUnitId>
				<isTest>true</isTest>
			</rval>
		</makeTestNetworkResponse>
	</soap:Body>
</soap:Envelope>

You can also get network code using two different ways.

• Visit your DFP console homepage by visiting https://www.google.com/dfp/ and you will see network name and code in the top potion. Also its listed in the URL (e.g. https://www.google.com/dfp/1234567)
• Another way to get available networks for your login is to visit DFP API Play ground here https://dfp-playground.appspot.com/ and you will see network list in the dropdown ( name and network code)

How to Create new Ad Units

Now lets look at an example which will create few Ad Units by calling createAdUnits API. If you are creating Ad Unit at the root level then you will need to specify correct Parent ID (i.e  effectiveRootAdUnitId). To obtain that Parent ID you may have to call  getCurrentNetwork API. In the response you will see effectiveRootAdUnitId.

There is another way to know effectiveRootAdUnitId or Id of any Ad Units which can be Parent for new AdUnit. Goto google.com/dfp > Click Inventory Tab > Click Ad Units Side menu > Click Download ad units hyper link. You can also download AdUnits as CSV file from DFP Portal and look for Ad Units which are created at the root. Parent ID column for Root level Ad Units is basically called effectiveRootAdUnitId.

Now lets look at how to get EffectiveRootID by calling getCurrentNetwork.

API getCurrentNetwork – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/NetworkService
Body (see below):

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Header>
	<RequestHeader xmlns="https://www.google.com/apis/ads/publisher/v201608">
		<networkCode xmlns="https://www.google.com/apis/ads/publisher/v201608">1122334455</networkCode>
		<applicationName xmlns="https://www.google.com/apis/ads/publisher/v201608">ZappySysApp</applicationName>
	</RequestHeader>
	</soap:Header>
	<soap:Body>
		<getCurrentNetwork xmlns="https://www.google.com/apis/ads/publisher/v201608" />
	</soap:Body>
</soap:Envelope>

API getCurrentNetwork – Response

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<requestId>81d1b00e8fbfb862129eb2dd485cf7bb</requestId>
			<responseTime>4507</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<getCurrentNetworkResponse xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<rval>
				<id>551196</id>
				<displayName>XFP sandbox property</displayName>
				<networkCode>1122334455</networkCode>
				<propertyCode>ca-pub-2799179143725683</propertyCode>
				<timeZone>America/New_York</timeZone>
				<currencyCode>USD</currencyCode>
				<effectiveRootAdUnitId>237897516</effectiveRootAdUnitId>
				<isTest>true</isTest>
			</rval>
		</getCurrentNetworkResponse>
	</soap:Body>
</soap:Envelope>

If you want to save single Element Value from Response into SSIS Variable then perform following steps (example of how to get just effectiveRootAdUnitId from above response)

1. On REST API Task > Go to Response Tab > Select Response Content Type = XML
2. Enter following expression in the XPath filter
   

   //*[local-name() = 'effectiveRootAdUnitId']

3. Check Save response content option
4. Select Variable from Dropdown – Click New Variable > Name it RootAdUnitId.
5. Click Test Request Button to test…. In the content textbox you will now see only Numeric value extracted by Filter

SSIS REST Api Task – How to extract single vale from response and save to SSIS variable

Once you know Parent ID for your AdUnit now lets look at how to create Ad Units by calling createAdUnits

API createAdUnits – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/InventoryService
Body (see below):

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<networkCode xmlns="https://www.google.com/apis/ads/publisher/v201608">1122334455</networkCode>
			<applicationName xmlns="https://www.google.com/apis/ads/publisher/v201608">ZappySysApp</applicationName>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<createAdUnits xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<adUnits>
				<parentId>237897516</parentId>
				<name>Ad_Unit_0</name>
				<description>Ad unit description #0.</description>
				<targetWindow>BLANK</targetWindow>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
					</size>
					<environmentType>BROWSER</environmentType>
				</adUnitSizes>
			</adUnits>
			<adUnits>
				<parentId>237897516</parentId>
				<name>Ad_Unit_1</name>
				<description>Ad unit description #1.</description>
				<targetWindow>BLANK</targetWindow>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
					</size>
					<environmentType>BROWSER</environmentType>
				</adUnitSizes>
			</adUnits>			
		</createAdUnits>
	</soap:Body>
</soap:Envelope>

API createAdUnits – Response

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<requestId>b244d7ba9b65e96fc42b849a7bfb16e4</requestId>
			<responseTime>331</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<createAdUnitsResponse xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<rval>
				<id>252312396</id>
				<parentId>237897516</parentId>
				<hasChildren>false</hasChildren>
				<parentPath>
					<id>237897516</id>
					<name>ca-pub-2799179143725683</name>
					<adUnitCode>ca-pub-2799179143725683</adUnitCode>
				</parentPath>
				<name>Ad_Unit_0</name>
				<description>Ad unit description#0.</description>
				<targetWindow>BLANK</targetWindow>
				<status>ACTIVE</status>
				<adUnitCode>253312396</adUnitCode>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
						<isAspectRatio>false</isAspectRatio>
					</size>
					<environmentType>BROWSER</environmentType>
					<fullDisplayString>300x250</fullDisplayString>
				</adUnitSizes>
				<mobilePlatform>SITE</mobilePlatform>
				<explicitlyTargeted>false</explicitlyTargeted>
				<inheritedAdSenseSettings>
					<value>
						<adSenseEnabled>true</adSenseEnabled>
						<borderColor>FFFFFF</borderColor>
						<titleColor>0000FF</titleColor>
						<backgroundColor>FFFFFF</backgroundColor>
						<textColor>000000</textColor>
						<urlColor>008000</urlColor>
						<adType>TEXT_AND_IMAGE</adType>
						<borderStyle>DEFAULT</borderStyle>
						<fontFamily>DEFAULT</fontFamily>
						<fontSize>DEFAULT</fontSize>
					</value>
				</inheritedAdSenseSettings>
				<lastModifiedDateTime>
					<date>
						<year>2017</year>
						<month>2</month>
						<day>27</day>
					</date>
					<hour>7</hour>
					<minute>23</minute>
					<second>32</second>
					<timeZoneID>PST8PDT</timeZoneID>
				</lastModifiedDateTime>
				<smartSizeMode>NONE</smartSizeMode>
				<isSharedByDistributor>false</isSharedByDistributor>
				<isSetTopBoxEnabled>false</isSetTopBoxEnabled>
			</rval>
			<rval>
				<id>252312516</id>
				<parentId>237897516</parentId>
				<hasChildren>false</hasChildren>
				<parentPath>
					<id>237897516</id>
					<name>ca-pub-2799179143725683</name>
					<adUnitCode>ca-pub-2799179143725683</adUnitCode>
				</parentPath>
				<name>Ad_Unit_1</name>
				<description>Ad unit description#2.</description>
				<targetWindow>BLANK</targetWindow>
				<status>ACTIVE</status>
				<adUnitCode>253312516</adUnitCode>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
						<isAspectRatio>false</isAspectRatio>
					</size>
					<environmentType>BROWSER</environmentType>
					<fullDisplayString>300x250</fullDisplayString>
				</adUnitSizes>
				<mobilePlatform>SITE</mobilePlatform>
				<explicitlyTargeted>false</explicitlyTargeted>
				<inheritedAdSenseSettings>
					<value>
						<adSenseEnabled>true</adSenseEnabled>
						<borderColor>FFFFFF</borderColor>
						<titleColor>0000FF</titleColor>
						<backgroundColor>FFFFFF</backgroundColor>
						<textColor>000000</textColor>
						<urlColor>008000</urlColor>
						<adType>TEXT_AND_IMAGE</adType>
						<borderStyle>DEFAULT</borderStyle>
						<fontFamily>DEFAULT</fontFamily>
						<fontSize>DEFAULT</fontSize>
					</value>
				</inheritedAdSenseSettings>
				<lastModifiedDateTime>
					<date>
						<year>2017</year>
						<month>2</month>
						<day>27</day>
					</date>
					<hour>7</hour>
					<minute>23</minute>
					<second>32</second>
					<timeZoneID>PST8PDT</timeZoneID>
				</lastModifiedDateTime>
				<smartSizeMode>NONE</smartSizeMode>
				<isSharedByDistributor>false</isSharedByDistributor>
				<isSetTopBoxEnabled>false</isSetTopBoxEnabled>
			</rval>
				<id>252312876</id>
				<parentId>237897516</parentId>
				<hasChildren>false</hasChildren>
				<parentPath>
					<id>237897516</id>
					<name>ca-pub-2799179143725683</name>
					<adUnitCode>ca-pub-2799179143725683</adUnitCode>
				</parentPath>
				<name>Ad_Unit_4</name>
				<description>Ad unit description.</description>
				<targetWindow>BLANK</targetWindow>
				<status>ACTIVE</status>
				<adUnitCode>253312876</adUnitCode>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
						<isAspectRatio>false</isAspectRatio>
					</size>
					<environmentType>BROWSER</environmentType>
					<fullDisplayString>300x250</fullDisplayString>
				</adUnitSizes>
				<mobilePlatform>SITE</mobilePlatform>
				<explicitlyTargeted>false</explicitlyTargeted>
				<inheritedAdSenseSettings>
					<value>
						<adSenseEnabled>true</adSenseEnabled>
						<borderColor>FFFFFF</borderColor>
						<titleColor>0000FF</titleColor>
						<backgroundColor>FFFFFF</backgroundColor>
						<textColor>000000</textColor>
						<urlColor>008000</urlColor>
						<adType>TEXT_AND_IMAGE</adType>
						<borderStyle>DEFAULT</borderStyle>
						<fontFamily>DEFAULT</fontFamily>
						<fontSize>DEFAULT</fontSize>
					</value>
				</inheritedAdSenseSettings>
				<lastModifiedDateTime>
					<date>
						<year>2017</year>
						<month>2</month>
						<day>27</day>
					</date>
					<hour>7</hour>
					<minute>23</minute>
					<second>32</second>
					<timeZoneID>PST8PDT</timeZoneID>
				</lastModifiedDateTime>
				<smartSizeMode>NONE</smartSizeMode>
				<isSharedByDistributor>false</isSharedByDistributor>
				<isSetTopBoxEnabled>false</isSetTopBoxEnabled>
			</rval>
		</createAdUnitsResponse>
	</soap:Body>
</soap:Envelope>

Loading Google DoubleClick data into SQL Server Table

So far we saw how to make simple DFP API calls without doing any parsing. Now lets look at how to use SSIS XML Source which not only make API calls but it will parse XML Response into rows and columns which can be loaded into target database such as SQL Server, Oracle, MySQL or even Flat File. XML Source also supports pagination so if you have many records it will automatically loop through all response untill all records are fetched.

Step-By-Step – Using XML Source to read Google DFP data (Parse into rows and columns)

1. Assuming you have tested your first DFP API Call (Explianed in the beginning of this article). You must have OAuth connection tested for API call.
2. Drag new Data flow task from SSIS Toolbox
3. Inside Data flow designer drag and drop ZS XML Source from toolbox
4. Double click XML Source to configure.
5. Set URL as below
   https://ads.google.com/apis/ads/publisher/v201702/InventoryService
6. Check Use credentials option and from Drop down select OAuth connection manager (created in previous section)
7. Select Method = POST,
8. Select Request Content Type = XML (text/xml)
9. Enter Body as below (Click edit button).
   

   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
      <soapenv:Header>
         <v20:RequestHeader>
            <v20:networkCode>238897396</v20:networkCode>
            <v20:applicationName>ZappySysApp</v20:applicationName>
         </v20:RequestHeader>
      </soapenv:Header>
      <soapenv:Body>
         <v20:getAdUnitsByStatement>
            <v20:filterStatement>
               <v20:query>WHERE Status='ACTIVE'</v20:query>
            </v20:filterStatement>
         </v20:getAdUnitsByStatement>
      </soapenv:Body>
   </soapenv:Envelope>

   Notice that in above we calling getAdUnitsByStatement API to fetch all Ad Units which are Active. Here is more information about Query Syntax for any API call getXXXXXXByStaement
10. Now click on Select Filter button and select results node and click OK. If prompted add array to list.
    

    SSIS XML Source – Get data from Google DFP API using OAuth (Configure URL, Body and Filter)

11. Click Preview to see data. Click Columns tab to review data types (Auto detected). If you wish to change length or datatype then edit there and check Lock option (Last column)
    

    SSIS XML Source – Preview Response for Google DoubleClick API Call – Parse into Rows/Columns

12. Click OK to save UI
13. Connect XML Source to same target (e.g. OLEDB Destination – SQL Server Connection)
14. Run data flow
    

    SSIS Execution – Read from Google DFP and load into SQL Server Table

Configure Pagination for Google DFP API / PQL Query result

If you have large dataset to read from DoubleClick API ( when calling API such as getxxxxxxByStatement) then we recommend to supply LIMIT and OFFSET clause in your PQL query.

For example if you have 2000 ad units and you want to limit response size by maximum 300 rows then your query can be like below

--For first request (returns results 1-300)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 0

--For second request (returns results 301-600)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 300

--For third request  (returns results 601-900)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 600

........

This looping logic can be complex to implement if you do manually. But no worry if you are using SSIS XML Source. It comes with many pagination options (Click here to read more about pagination)

See below screenshot how to configure Pagination options for Google DoubleClick API. Basically two places you have to change. Inside body you have to set placeholder and change few settings on Pagination Tab.

SSIS XML Source – Configure Pagination for Google DoubleClick API data fetch – PQL LIMIT and OFFSET clause for Google DFP API

SSIS XML Source – Configure Pagination for Google DoubleClick API data fetch – Set Pagination Mode

Create CSV Report File and Download in GZip format (*.gz)

Sometime you have to download large amount of data (Possibly millions of rows) in that case Bulk approach would be better. DFP ReportService API allows to call following APIs which can be used to produce CSV report file in *.gz format and then you can download it using REST API Task.

Below are high level steps you have to perform to produce report file and download it.

• Create the ReportJob by invoking runReportJob command .
• Poll the ReportJob object using ReportService.getReportJob (This is required because file is not available right away) .
• Continue to poll the ReportJob object until the reportJobStatus field is equal to COMPLETED or FAILED.
• If successful, fetch the URL for downloading the report by invoking getReportDownloadURL.

Here is the sample SSIS Package to Perform this Action.

Click here to DFP API download Sample for SSIS SSIS Package (SSIS 2012, 2014, 2016)

See below screenshot how to generate Google DFP Report File and download / extract (Unzip) using Drag and Drop SSIS workflow.

Create Google DFP Report File (CSV / Gzip) and Download Using SSIS (Example of ReportService functions runReportJob, getReportJob and .getReportDownloadURL)

Call runReportJob (Start DFP Report JOB)

Very first step to produce DFP Report file is call ReportService >> runReportJob API. See below Command.

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract id of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for <id>111111</id> node anywhere in SOAP response XML and extract value (i.e. 111111 )  .. For screenshot see next section

//*[local-name() = 'id']

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxxxxAKqG89gM-TpReRXO
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 1018
Expect: 100-continue
Accept-Encoding: gzip, deflate

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
	<soapenv:Header>
		<v20:RequestHeader>
			<v20:networkCode>238897396</v20:networkCode>
			<v20:applicationName>ZappySysApp</v20:applicationName>
		</v20:RequestHeader>
	</soapenv:Header>
	<soapenv:Body>
		<v20:runReportJob>
			<v20:reportJob>
				<v20:reportQuery>
					<v20:dimensions>DATE</v20:dimensions>
					<v20:dimensions>LINE_ITEM_ID</v20:dimensions>
					<v20:dimensions>LINE_ITEM_NAME</v20:dimensions>
					<v20:dimensions>AD_UNIT_NAME</v20:dimensions>
					<v20:columns>AD_SERVER_IMPRESSIONS</v20:columns>
					<v20:columns>AD_SERVER_CLICKS</v20:columns>
					<v20:dimensionAttributes>LINE_ITEM_COST_TYPE</v20:dimensionAttributes>
					<v20:dimensionAttributes>LINE_ITEM_GOAL_QUANTITY</v20:dimensionAttributes>
					<v20:dateRangeType>LAST_MONTH</v20:dateRangeType>
				</v20:reportQuery>
			</v20:reportJob>
		</v20:runReportJob>
	</soapenv:Body>
</soapenv:Envelope>

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:41 GMT
Expires: Fri, 02 Jun 2017 21:23:41 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 999

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<requestId>f03e7e474fde2a5f9c5d7c9d7eb83923</requestId>
			<responseTime>846</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<runReportJobResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<rval>
				<id>10000297829</id>
				<reportQuery>
					<dimensions>DATE</dimensions>
					<dimensions>LINE_ITEM_ID</dimensions>
					<dimensions>LINE_ITEM_NAME</dimensions>
					<dimensions>AD_UNIT_NAME</dimensions>
					<adUnitView>TOP_LEVEL</adUnitView>
					<columns>AD_SERVER_IMPRESSIONS</columns>
					<columns>AD_SERVER_CLICKS</columns>
					<dimensionAttributes>LINE_ITEM_COST_TYPE</dimensionAttributes>
					<dimensionAttributes>LINE_ITEM_GOAL_QUANTITY</dimensionAttributes>
					<dateRangeType>LAST_MONTH</dateRangeType>
					<useSalesLocalTimeZone>false</useSalesLocalTimeZone>
					<includeZeroSalesRows>false</includeZeroSalesRows>
				</reportQuery>
			</rval>
		</runReportJobResponse>
	</soap:Body>
</soap:Envelope>

Call getReportJob (Poll JOB Status)

Second step is to make sure is Report Status check. For large report it make take several seconds or minutes before you can get Download URL (See Next Step). If you try to call Next step before Report is ready then you may get error. So better to add Polling logic.

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract status of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for <rval>IN_PROGRESS</rval> node anywhere in SOAP response XML and extract value (i.e. IN_PROGRESS )

//*[local-name() = 'rval']

Extract Single XML Node Value from SOAP API Response – Use XPATH in SSIS

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxXmWqmq
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 524
Expect: 100-continue
Accept-Encoding: gzip, deflate

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
   <soapenv:Header>
      <v20:RequestHeader>
         <v20:networkCode>238897396</v20:networkCode>
         <v20:applicationName>ZappySysApp</v20:applicationName>
      </v20:RequestHeader>
   </soapenv:Header>
   <soapenv:Body>
      <v20:getReportJobStatus>
       <v20:reportJobId>10000297829</v20:reportJobId>
      </v20:getReportJobStatus>
   </soapenv:Body>
</soapenv:Envelope>

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:47 GMT
Expires: Fri, 02 Jun 2017 21:23:47 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 451

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<requestId>d52cb3a3d68af24aec2f9b0284379700</requestId>
			<responseTime>803</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<getReportJobStatusResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<rval>COMPLETED</rval>
		</getReportJobStatusResponse>
	</soap:Body>
</soap:Envelope>

Call getReportDownloadURL (Get Download URL)

Once you get status = COMPLETED in above API Call (i.e.  getReportJobStatus) then you ready to call getReportDownloadURL  to fetch URL which you can download.

Here is the API sample Request and Response

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxBp84ltTNyVRb9hlUEWx
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 581
Expect: 100-continue
Accept-Encoding: gzip, deflate

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
   <soapenv:Header>
      <v20:RequestHeader>
         <v20:networkCode>238897396</v20:networkCode>
         <v20:applicationName>ZappySysApp</v20:applicationName>
      </v20:RequestHeader>
   </soapenv:Header>
   <soapenv:Body>
      <v20:getReportDownloadURL>
       <v20:reportJobId>10000297829</v20:reportJobId>
       <v20:exportFormat>CSV_DUMP</v20:exportFormat>
      </v20:getReportDownloadURL>
   </soapenv:Body>
</soapenv:Envelope>

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:49 GMT
Expires: Fri, 02 Jun 2017 21:23:49 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 1039

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<requestId>8e5ac0bbeb8e88c180ea442fc5378d99</requestId>
			<responseTime>1957</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<getReportDownloadURLResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<rval>https://storage.googleapis.com/dfp-report-export/caxxxxxxxx-xxxxxxxxxxxxxxb?GoogleAccessId=3769xxxxxxxxxxx-qxxxxxxxxxxxxxxxxsqdb5dvi8g@developer.gserviceaccount.com&amp;Expires=1496438929&amp;Signature=vnhCiAIqd%2BFgWCtAHo9roFOI3W%2BveUTYuSvyBQ9gtMI2HDTXOCVDh7wzUV%2BPBJZZghaL6k8wKYq33GA4NpjZVNYuaAgPLWgGwAUPdtioHRi%2Bt3Eymp%2FAPRGIoos0ekP3O7l%2FPNPxenC6UTP087HV3j06V%2Fuui9ixpKZWGCK8r56oUPj6S8q1hG95Ns3d3sEcuz3%2BkShu6GYG5mqir5vjtZPuzwsr%2F96G5Kk123mke%2B%2FIVRJGoqnCMghNWtIG1yR1IdZx%2FIpit%2FnaJ9HP6IvAr3oMIXkQ3cHv%2FtOhXGy9dTG34q%2FKDbtkSJAayHgjlSLPGKucygcXBhUrbywBu8Y77A%3D%3D</rval>
		</getReportDownloadURLResponse>
	</soap:Body>
</soap:Envelope>

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract status of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for <rval>https://storage.googleapis.com/dfp-report-export/caxx……….</rval> node anywhere in SOAP response XML and extract value (i.e. URL)

Download File Using REST API Task

Once you get URL its time to download it. You can use REST API Task to download the file. On Response Setting Tab specify full file path (e.g. c:\report_csv.gz) and make sure Binary option is checked (Found next to the Save Path)

Making things dynamic

If you want to supply certain parameters at runtime (e.g. from SSIS Variable) rather than hard code then you can use Variable Placeholders any where in Request Body, URL or Headers. Click Insert Variable option found in Edit screen

Here is an example of dynamic URL (API Version stored in SSIS Variable)

https://ads.google.com/apis/ads/publisher/{{User::ApiVer}}/NetworkService

Deployment to Production

When you ready to deploy your SSIS Package to production make sure to change Test NetworkCode supplied inside Request Body to your Production Ad Network. Also Set RefreshToken Property of OAuth Connection Manager (Get it from Backup file saved when you created OAuth connection first time). Once you set RefreshToken of connection manager you won’t have to Login again to get a new token.

Conclusion

Traditionally if you wanted to integrate Google DFP API (DoubleClick API) calls inside your ETL workflow then it required some sort of coding effort (e.g. Use C# SDK, JAVA SDK or Python SDK). Coding effort is not only time consuming and expensive but also hard to maintain for any non-coder (e.g. ETL Developer or DBA).
Thanks to SSIS PowerPack which changed the way developers do API integration. Now you can reduce time to implement and total cost of you API integration project significantly by using drag and drop approach. This approach is not only easy to maintain but also fast and high quality. You can use ZappySys REST/SOAP SSIS Connectors and SSIS Tasks for any API integration project such as Google DFP API. Try SSIS PowerPack for free to explore possibilities. If you have any issue with your integration Contact Support

The post How to call Google DFP API with SSIS – DoubleClick appeared first on ZappySys Blog.

eBay recently launched brand new REST API with OAuth 2.0 support. This article will explain you how to call eBay REST API using SSIS to automate eBay operations (e.g. Buy/Sell) using SSIS REST API Integration Connectors / OAuth Connection Manager.

This article assumes that you have basic knowledge of SSIS (SQL Server Integration Services). We will use drag and drop approach without coding to do eBay REST API integration. We will use following custom tasks/connectors from SSIS PowerPack

SSIS PowerPack REST API Connectors can be used to consume/manage eBay data using API calls. Following connectors will be used in this article.

	REST API Task
	OAuth Connection Manager
	JSON Source (File, REST, OData)

Test Ebay API using Online API Testing Tool

Before you can dig deep down we suggest you to try few API calls using ebay online testing tool. Here is an example api call using their Online Testing Tool.

For example visit below URL to try sample Search API call.

https://developer.ebay.com/my/api_test_tool?index=0&env=production&api=browse&call=item_summary_search__GET&variation=json

Testing Ebay REST API using Online Tool

Call eBay API using SSIS – Create new inventory item

Now lets look at how to make simple eBay REST API call using SSIS. For demo purpose we will use Sandbox API rather than production.

1. First download and install SSIS PowerPack from here.
2. Create new SSIS Project and open SSIS Package
3. Drag new ZS REST API Task from SSIS Toolbox on control flow designer
4. Rename ZS REST API Task to Create eBay inventory item-1. Double click the task to configure it.
5. Select Request mode to [Url from Connection]
6. Select ZS-OAUTH connection type from connection dropdown.
7. On OAuth connection. Select eBay Sandbox from provider dropdown.
8. Enter App ID in Client ID field (Obtained from here)
9. Enter Cert ID in Client Secret field  (Obtained from here)
10. Select desired Scopes or leave it blank to use default scopes. For this example we need following two scopes
    https://api.ebay.com/oauth/api_scope/sell.inventory
    https://api.ebay.com/oauth/api_scope/sell.inventory.readonly
11. Click on Advanced Tab. Enter RuName in the Return URL field. Don’t worry if your RuName doesn’t look like URL 🙂 (Click here to learn how to get RuName – Scroll to Getting the RuName value)
12. Now click on first tab again. Click Generate token button. When prompted login using Sandbox User Account (Click here to create test user for Sandbox).
    NOTE: Production account or developer account wont work when you login for eBay Sandbox API Token
13. Click Accept button after login. If everything goes well you will see two tokens populated on your main screen (Access Token and Refresh Token). You will get prompt to save tokens to some backup file… Do that so same tokens can be used if you wish to deploy JOB to another machine.
14. Click Test to validate connection. Click OK to save.
    

    SSIS OAuth Connection Manager – Connect to eBay Sandbox API

15. Now once you close connection you are back to REST API Task
16. Enter following URL to create or new eBay inventory item using PUT. Last part of following URL is ID we want to create
    

    https://api.sandbox.ebay.com/sell/inventory/v1/inventory_item/GP-Cam-01

17. Select Method PUT from dropdown
18. Click Edit for Request Body. Enter following data for our sample product
    

    {
    	"availability" : {
    		"shipToLocationAvailability" : {
    			"quantity" : 50
    		}
    	},
    	"condition" : "NEW",
    	"product" : {
    		"title" : "GoPro Hero4 Helmet Cam",
    		"description" : "New GoPro Hero4 Helmet Cam. Unopened box.",
    		"aspects" : {
    			"Brand" : ["GoPro"],
    			"Type" : ["Helmet/Action"],
    			"Storage Type" : ["Removable"],
    			"Recording Definition" : ["High Definition"],
    			"Media Format" : ["Flash Drive (SSD)"],
    			"Optical Zoom" : ["10x"]
    		},
    		"imageUrls" : [
    			"http://i.ebayimg.com/images/i/182196556219-0-1/s-l1000.jpg",
    			"http://i.ebayimg.com/images/i/182196556219-0-1/s-l1001.jpg",
    			"http://i.ebayimg.com/images/i/182196556219-0-1/s-l1002.jpg"
    		]
    	}
    }

19. Change Body content type to JSON from dropdown
20. Click on [Raw Edit] button in Headers grid toolbar. Enter following headers and click OK.
    

    Content-Language: en-US
    Accept: application/json

     
21. Click Test Request/Response – If you get success that means product got created.
22. You can create more products by simply changing SKU in URL and name/description in Body. Click Test to execute API call.

Read data from eBay and load into SQL Server using SSIS JSON Source

Now lets look at how to consume eBay REST API and load data into SQL Server. For this you can use JSON Source (File, REST API). For this demo we will Call eBay Search API . If you want to explore other API then check this page. Here is

1. Go to Data flow tab
2. Drag and drop ZS JSON Source from SSIS Toolbox
3. Double click to configure it.
4. Enter API URL like below. We added Search parameter and limit parameter (Max rows per response). Refer to API Help page to know more about other available parameters.
   

   https://api.sandbox.ebay.com/buy/browse/v1/item_summary/search?q=phone&limit=100

5. Check Use Credentials option.
6. Select ZS-OAUTH connection or Create New (See previous section to create new one)
7. Select Filter (Look for Array Icon and select that node). For example if you calling search API then  enter or select 

   $.itemSummaries[*]

     like below.
   

   SSIS JSON Source – Get data from eBay REST API (Use OAuth Connection)

8. Now click on pagination tab and enter

   $.next

     expression for next link attribute as below. This will enable pagination (It will keep reading in 100 rows per page until last page reached – Empty  recordset)
   

   Ebay REST API Pagination (Search API Example)

9. That’s it, now click Preview to see data
10. Once all looking good you can click OK to save JSON Source.
11. In Next Section we will see how to load Ebay Data into SQL Server.

Loading eBay data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Pagination for eBay REST API Calls

In previous example we saw REST API Pagination using Next Link approach but some API may not use same approach in that case refer to this link.  We looked at Next Link approach in previous section, you can also use Method #2 (URL parameter  – Offset) but Next Link approach is more simpler.

Calling eBay XML format API – Finding / Trading / Shopping API

eBay converted most of their APIs to JSON format to use OAuth 2.0 standard but there few APIs still there which are not OAuth 2.0 standard. These includes eBay Trading API, eBay Finding API, eBay Shopping API. They all are still in XML format. Below section describe how to call these API. Use eBay API Test tool here to generate desired Request Body and Headers.

Now let’s look at an example to call eBay XML API (GetItem from Trading API). To support special case like eBay API ZappySys introduced OAuth connection property called CustomAuthHeader in v2.6.7 or higher (Only found in Properties grid view).

1. Right click on OAuth connection manager icon > Click Properties > Set CustomAuthHeader to X-EBAY-API-IAF-TOKEN
   
2. Now drag ZS REST API Task from Control flow SSIS Toolbox
3. Set Properties like below
   1. Set URL as below (Sandbox Url is different than Production)
      

      https://api.sandbox.ebay.com/ws/api.dll

   2. Set Body As (this was obtained from  eBay API Test tool here )
      

      <?xml version="1.0" encoding="utf-8"?>
      <GetItemRequest xmlns="urn:ebay:apis:eBLBaseComponents">
      	<ErrorLanguage>en_US</ErrorLanguage>
      	<WarningLevel>High</WarningLevel>
            <!--Enter an ItemID-->
        <ItemID>1234</ItemID>
      </GetItemRequest>

   3. Set Headers as below (Click on Raw Edit Button and paste below heades). Again these header were also copied from eBay API Test tool here . Notice that API function name is included in the header. Also site id and compatibility level required.
      

      X-EBAY-API-CALL-NAME: GetItem
      X-EBAY-API-SITEID: 0
      X-EBAY-API-COMPATIBILITY-LEVEL: 967

   4. Now click OK to get response. If things working you will see some valid XML.
      

      Call eBay XML API using SSIS REST API Task (eBay Trading API, Shopping API, Finding API using OAuth Token)

User Token vs Application Token

Any time you call eBay API you have to supply token. eBay OAuth supports two types of token. In this article we used User Token. If you wish to use Application Token then you have to use OAuth Client Credentials grant. See this article

Difference between eBay Application Token vs User Token

1. Application Token
   – Do not require user login
   – Use this token if you need to perform basic operations which are supported by following permission   https://api.ebay.com/oauth/api_scope
2. User Token
   – Requires one time user login to create token and then refreshes automatically without login prompt (if you using SSIS PowerPack)
   – Use this token if you need higher privileges behalf of actual use for update/write data using API call

Use Application token when you want to perform readonly operations and you don’t need higher privileges (such as admin access). Certain eBay API calls only possible if you supply user token.

POST data to Ebay (Insert or Update)

In previous section we have seen how to read data from ebay. Now let’s look at how to write data to ebay.

There are two ways you can achieve this.

1. Use Web API Destination in Data Flow
2. Use REST API Task in Control Flow

Read this article to learn more about API POST using SSIS

Using SSIS Web API Task (Write data from SQL Server to ebay)

Using REST API Task to POST data to ebay

If you have JSON/XML data already prepared and if you like to POST it to ebay API then REST API Task would be easy to use.

REST API Task Body can be direct string, variable (e.g. {{User::varSomeData}} or Body can come from File 

Common Errors

Deployment to Production

Conclusion

REST API integration typically requires coding and good understanding of protocol such as OAuth. SSIS PowerPack REST API connectors and OAuth Connection provides easy to use interface with many options to integrate virtually any API including eBay REST API inside your SQL Server Environment without any coding. Try SSIS PowerPack yourself see what is possible.

The post How to call eBay REST API in SQL Server with SSIS appeared first on ZappySys Blog.

In this post you will learn how to call Amazon MWS API (Amazon Marketplace Web Service) or Amazon Product Advertising API using SSIS PowerPack.

Using drag and drop approach you can consume data from Amazon MWS XML Web service. In this post we will use ZappySys XML Source connector to read data from Amazon MWS API and load into SQL Server. We will also use REST API Task to call any API from Marketplace Web service and save output into Variable.

Prerequisites

This post assumes following things

1. Basic knowledge of SSIS and XML format.
2. Amazon MWS Account and and valid AWSAccessKeyId and Secret Key to call Amazon MWS API (How to obtain Key / Secret and SellerId ? ).
3. You have tested few API calls using Amazon MWS Scratchpad this will give you idea about various API you can call and parameters you need to pass.

Testing MWS Requests using ScratchPad

Very first step we recommend before you call MWS API in SSIS is to get familiar with Amazon MWS Scratchpad tool. Check this article for detailed steps.  You can also use Fiddler to debug some requests.

Calling Amazon MWS API in Scratchpad Testing Tool (Response Details Tab) – ListMatchingProducts Example

Call Amazon MWS API using SSIS

To consume data from any XML Source (File, SOAP Web Service or XML API) you can use  ZappySys XML Source
connector. or REST API Task. In below example you will see how to call Amazon MWS API and save output into Variable. With this approach you can call any API (GET / POST / DELETE / PUT)

1. Download and Install SSIS PowerPack
2. Create test package
3. From SSIS toolbox drag ZS REST API Task  and double click on the task to configure it
4. Select Request URL Access Mode = [Url from Connection]
5. Enter URL as below
   

   https://mws.amazonservices.com/Orders/2013-09-01

   In above URL
   ** mws.amazonservices.com => This is your endpoint. If you are not in USA then Click here to find correct endpoint
   ** /Orders => That’s your API you want to call
   ** /2013-09-01 => That’s your API version (You have to add this same version in your POST Body too (see next section).
6. In Select Url Connection dropdown select New OAUTH connection option.
7. When prompted on OAuth connection UI select Provider=Amazon MWS, Enter AWSAccessKey and Secret key and Click OK (If you have MWSAuthToken then enter in Access Token field else leave it empty). Do not test because it may not work yet.
   

   SSIS OAuth Connection – Call Amazon MWS API (Marketplace Web service API)

8. Another Setting you may turn on is Retry Handling. This allows to over come ThresholdLimitReached error when we call API too fast.
   

   Retry Options

9. Now select Method=POST, and in the body enter following
   

   Action=GetServiceStatus&SellerId=A77XXXXXXX&Version=2013-09-01

   If your API requires pagination (more than 100 records from response) then refer next section (Sample Package). In the case of Pagination your second request may look like below
   

   Action=YourActionNameWithNextToken&SellerId=A77XXXXXXX&Version=2013-09-01&NextToken={{User::vNextToken,FUN_URLENC}}

   NOTE: Enter all required parameters in the following format, as shown below. All values need to be URL-encoded. If you use SSIS variable, use it along with FUN_URLENC format specifier, e.g. {{User::myVariable,FUN_URLENC}}. If you use a hard-coded value, and not sure if it is URL-encoded, then encode it with the same FUN_URLENC format specifier/function, e.g. <<myValue,FUN_URLENC>>. Always check if SSIS configuration matches the one you see in MWS ScratchPad Request Details.
   Format:
   param1=value1&param2=<<value2,FUN_URLENC>>&param3={{User::myVar,FUN_URLENC}}
10. Now click Test and see content in the Preview window.
    

    SSIS REST API Task – Call Amazon MWS API – Get XML data save into SSIS variable

11. If you wish to save response data into SSIS variable then goto response Tab and check save option and select variable name.

Amazon MWS API Pagination

Many Amazon MWS API calls by default don’t return all records (Explained here). Rather than returning all rows you may get partial response (e.g. Max 50 or 100 rows). If you want to get all records then you have to use NextToken  found in your XML. For API which requires pagination, in that case your POST body may be different for 2nd or higher requests. Notice how we changed Action and appended NextToken=xxxxxxxxxxx in the second request. Also notice we used encoded token (URL encoded). This is very important because as per Amazon Specs this value has to be Url Encoded.

We also documented similar use case of Amazon MWS Pagination for Power BI (Check this one) its mostly same setup except 2-3 properties might be named different way.

Now lets look at sample request / response for paginated MWS requests (For clarity we have removed many common Parameters which are automatically added at runtime e.g. Signature, Timestamp). We use ListOrders (First Request) and ListOrderByNextToken (Second+ requests)

First Request

POST https://mws.amazonservices.com/Orders/2013-09-01

Action=ListOrders&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z

First Response

<?xml version="1.0"?>
<ListOrdersResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

Second Request or higher (MWS API with Pagination using NextToken)

POST https://mws.amazonservices.com/Orders/2013-09-01

Action=ListOrdersByNextToken&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z&NextToken=YOUR_ENCODED_TOKEN_FROM_PREV_REQUEST

Second or higher Response

<?xml version="1.0"?>
<ListOrdersByNextTokenResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersByNextTokenResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

Sample SSIS Package

If you want to see complete working package for pagination pattern then download from here. Change AccessKey, SecretKey, SellerID and Path as needed to make this package work.

Here is the screenshot of example SSIS package for Amazon MWS.

Amazon MWS APi Pagination (ListOrders , ListOrdersByNextToken Example)

Copy/Paste Properties

Here is copy/paste values for properties listed above.

DirectPath:  https://mws.amazonservices.com/Orders/2013-09-01
EnablePageTokenForBody: True
Filter:  $.ListOrders[$tag$]Response.ListOrders[$tag$]Result.Orders.Order[*]
HasDifferentNextPageInfo: True
HttpRequestData:  Action=ListOrders[$tag$]&SellerId=A1xxxxxxxx&CreatedAfter=2018-09-01T04%3A00%3A00Z&MarketplaceId.Id.1=ATVPDKIKX0DER
HttpRequestMethod:  POST
NextUrlAttribute: $.ListOrders[$tag$]Response.ListOrders[$tag$]Result.NextToken
NextUrlSuffix: &NextToken=<%nextlink_encoded%>
NetUrlWait: 10000 (ms) -- slow down to avoid throttling ... Max 6 requests per minute, increase if you get API Limit Error
PagePlaceholders:  body=|ByNextToken;filter=|ByNextToken

Understanding Pagination Properties

Here are various properties you have to set for Pagination.

POST /Orders/2013-09-01?AWSAccessKeyId=AKxxxxxxxxxx
  &Action=ListOrders
  &SellerId=A10zzzzzzz
  &SignatureVersion=2
  &Timestamp=2018-09-17T21%3A35%3A57Z
  &Version=2013-09-01
  &Signature=3GigipfRsQgzzzzzzzzzzzzzs%3D
  &SignatureMethod=HmacSHA256

Action=ListOrders[$tag$]&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z

<?xml version="1.0"?>
<ListOrdersResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

<?xml version="1.0"?>
<ListOrdersByNextTokenResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersByNextTokenResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

NextUrlEndIndicator=false
StopIndicatorAttributeOrExpr= $.GetReportRequestList[$tag$]Response.GetReportRequestList[$tag$]Result.HasNext

Amazon MWS API Pagination (For Old Version)

So in previous section we saw how to achieve pagination in one step (for newer version of SSIS PowerPack). But there will be a case you cannot use latest SSIS PowerPack or you want to take complete control on each aspect of pagination. In such case use below method (Depreciated).  In newer version we added many new properties so you dont have to do Loop pattern like below.

Sample SSIS Package

If you want to see complete working package for pagination pattern then download from here. Change AccessKey, SecretKey, SellerID and Path as needed to make this package work.

Here is the screenshot of example SSIS package for Amazon MWS.

SSIS Example Package – Read data from Amazon MWS Web service (API Call). Paginate API calls using NextToken

Load Amazon MWS API data into SQL Server using SSIS

In this section we will see how to use ZappySys XML Source connector to read data from Amazon MWS Web Service and load into SQL Server

For making things simple we are calling

1. Download and Install SSIS PowerPack
2. Create test package
3. From SSIS toolbox drag Data Flow task and double click task to go to Data Flow designer
4. Drag ZS XML Source from SSIS Toolbox
5. Configure XML Source as below
   

   SSIS XML Source – Read data from Amazon MWS API (Amazon Marketplace Web service call)

6. Drag OLEDB destination from SSIS Toolbox.
7. Connect XML Source to OLEDB Destination and map input column to target table
8. Execute dataflow
   

   Get data from Amazon MWS API – Save to File or SQL Server using SSIS

Working with Amazon MWS Feed API (e.g. Upload file)

Amazon MWS Provides Feed APIs which has slight different requirements. Important requirement in Feed file upload is you have to supply Content MD5 Hash. If you use SSIS PowerPack then you don’t have to worry about that complexity because calculating MD5 Hash is automatically done.

Upload Amazon MWS Feed File in TSV / CSV format (Tab separated)

Here is the screenshot for how to upload Tab separated values in CSV file. You can specify body from Variable or Read from file. If you read from file then must check File Upload/Multi-Part option and start path with @ symbol as below.

Call Amazon MWS Feed API – Upload CSV file format (Tab separated values – TSV)

Upload Amazon MWS Feed File in XML format

To upload feed file in XML format use same settings as above except Content Type should be XML (text/xml) (Select in the dropdown)

Calling Amazon Product Advertising API

If you have need to call Amazon Product Advertising API then also you can use above techniques (Use MWS Provider on OAuth connection) to call Product Advertise API. Both API (MWS API and Product Advertise API) use same signature hashing algorithm so its possible to use MWS Provider for Hashing. You may have to change your API URL parameters as needed when you call Amazon Product Advertising API. Basically when you call any MWS or Product Advertising API using GET method then OAuth connection manager automatically appends Signature to URL at runtime.

Import data from MWS Custom Reports (Inventory Report Example)

Now lets look at how to extract data from Custom Report API . Reading data from Custom Reports not single step process because report generation is Job style API. Which means you send report request and wait  until its done. Once Report ready you have to read in CSV format or XML. Some Reports only available in CSV format. Check this post to understand how complex it can be to get data. We will make it simple for you to understand this in 3 steps. Basically calling reports requires minimum 3 API calls (see below)

1. Create a new report request – Cal RequestReport API . It returns ReportRequestId (You can use it in the next step)
2. Check Report Status see its done – Call GetReportRequestList API (Pass ReportRequestId got in the previous step to get data for only one Report request we care). Keep checking Report Status every few seconds until you get _DONE_ status in response.
3. Read Report Data – Call GetReport. This API call returns CSV or XML data. So use correct component for this step. If CSV data is returned then we must use CSV Source rather than XML Driver.

Now lets see how to read Amazon MWS custom report in SSIS. In this example, We will get Inventory by calling _GET_MERCHANT_LISTINGS_DATA_ report type.

Download Sample Package for MWS Report

To make things simple we have created a demo package – ready to go.
Download SSIS 2012 Demo – Amazon_MWS_API_GetReport_2012.

Prepare demo package for first run

Once you download and extract above sample package. Import to your existing SSIS Solution. Its SSIS 2012 format so it may upgrade to higher version if you using SSIS 2014, 2016 or 2017.

Once package is opened perform following steps in before you can execute the package.

1. Enter MarketplaceId, SellerId in Variables. Double click OAuth Connection and enter your MWS Account Key and Secret.
2. Create c:\temp directory (This is where we will save final report file)
3. Run this package
4. Go to Execution log and Find ReportId (This you can hardcode in Variables for Design time Preview / Metadata changes etc)

See below package screenshot

Read data from Amazon MWS Report in SSIS (Get Inventory Listing Example)

Steps Explained

Now lets see each steps in depth. We broken whole process in 3 groups.

Create Report Request (Step1)

This is the first step how you initiate report request.  Use REST API Task to call this step. Here is raw request. Many items in body added at runtime (e.g. Signature, Timestamp).

• On Request tab see how we used SSIS Variables to get many values at runtime dynamically.
• On Response Tab we have to extract RequestId and save into variable.

POST 
https://mws.amazonservices.com/Reports/2009-01-01

AWSAccessKeyId=Axxxxxxxxx&Action=RequestReport&MarketplaceId=ATVPDKIKX0DER&ReportType=_GET_MERCHANT_LISTINGS_DATA_&SellerId=Axxxxxxxxx&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2019-06-05T21%3A23%3A31.090Z&Signature=2AbGQQ4xxxxxxxxxxxxxxxxxxxxjJzox%2FU%3D

RequestReport Action – Generate Amazon MWS Custom Report

Extract value from XML response using XPATH and save to Variable (Get RequestId Example)

Here is how to configure above task

Wait until data is ready (Step2)

Once we send RepotRequest and we have RequestId we can call step2 using REST API Task. In this Task we can use Status Check Feature. This is pretty handy feature because we dont have to implement Loop in SSIS. Basically we have to keep checking report until we get _DONE_ status in the response (ReportProcessingStatus node in XML)

Here is how to configure the task.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body:

   Action=GetReportRequestList&ReportRequestIdList.Id.1={{User::ReportRequestId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}

2. Response Setting Tab Configuration
   Response content type: Xml
   Filter Expression: 

   //*[local-name() = 'ReportProcessingStatus']  

3. Status Check Tab
   Check Enable Status
   Success Value: 

   _DONE_

   Check Every 5 Seconds

REST API Task – Status Check Feature (Wait until data is ready loop)

GetReportId (Step3)

Once above task finished we can call same request again but this time we have to configure slightly different way.

Here is how to configure the task.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body:

   Action=GetReportRequestList&ReportRequestIdList.Id.1={{User::ReportRequestId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}

2. Response Setting Tab Configuration
   Response content type: Xml
   Filter Expression: 

   //*[local-name() = 'GeneratedReportId']  

   Check Save to Variable: Save to User::ReportId
3. Status Check Tab
   Make Enable Status is unchecked

Get Report Data  (Step4,Step5 or Step6)

Now we are ready to ready our report. There are two ways we can do in SSIS.

• First approach (step 4,5) is download the report and save to local disk and then in the next step we can use data flow (CSV Source) to parse the saved file.
• Second approach (step6) is rather than saving to disk we can directly call API using CSV File source and parse content that way. Lets talk each approach briefly.

Step4,5 – Load Report From File

If you wish to save Report on disk then you can use REST API Task and call request like below.

Here is how to configure the task to download the file.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body: Action=GetReport&ReportId={{User::ReportId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}}
2. Response Setting Tab Configuration
   Response content type: Xml
   Filter Expression: 

   //*[local-name() = 'GeneratedReportId']  

   Check Save to Variable: Save to User::ReportId
   
3. On Response Settings Tab : Set Response charset to Western European [Windows 1252]  (or directly set via Properties Grid to Windows-1252 under ResponseCharset Property)
   
4. Status Check Tab
   Make Enable Status is unchecked

To read download CSV file use settings described in below section except URL change it with local file path.

Step6 – Load report from URL

If you wish to load directly from URL then use same setting as above but in CSV Source inside data flow.

Here is how to configure CSV Source.

1. Request Tab Configuration.
   URL:

   https://mws.amazonservices.com/Reports/2009-01-01

   Method: POST
   Body:

   Action=GetReport&ReportId={{User::ReportId}}&MarketplaceId={{User::MarketplaceId}}&SellerId={{User::SellerId}} 

2. General Tab
   Column Delimiter: {TAB}
3. On Encoding Tab : Set to Western European [Windows 1252]  (or directly set via Properties Grid to Windows-1252 under CharacterSet Property)

Conclusion

Amazon Marketplace Web service (MWS) provides great way to automate many functionality for Marketplace Seller. Using ZappySys SSIS PowerPack you con consume data from any REST API or Web service without replying on SDK / coding approach (e.g. C#, Java, Python, Ruby).

The post How to call Amazon MWS API using SSIS appeared first on ZappySys Blog.

In this tutorial we will use REST API Task to call some ad-hoc API (e.g. get File List from Google Drive) and save output into Variable or File. We will use OAuth connection along with ZappySys JSON Source connector to read data from Google SpreadSheet (Use Drive API) and load into SQL Server (Export Google SpreadSheet to CSV).

Create Google API Project

First step to access any Google API is create an API Project in Google Console. If you don’t want to go through this then Skip this Step-1 and in the next section select Default OAuth App option on OAth Connection Manager (This is the most easiest option for now unless you want to use your own OAuth App).

Check this article for step-by-step instructions. When you follow these instructions make sure you enable Google Drive API (In the article it shows how to enable YouTube API as an example but you will need to enable Drive API for this article).

Create OAuth Connection Manager in SSIS

Once you create Google API project and obtained Client ID and Client Secret your next step is to create OAuth Connection Manager in SSIS. ZappySys OAuth connection manager comes with many predefined OAuth Providers (e.g. Facebook, Twitter, Google etc) but you can also define custom OAuth settings for any OAuth enabled API.

To create SSIS OAuth 2.0 Connection for Google API perform following steps.

1. Download and Install SSIS PowerPack
2. Create new SSIS Package
3. Right click in Connection Manager Area and Click “New Connection”
4. When prompted select ZS-OAUTH connection type
5. On the OAuth Connection Manager Select Provider=Google Sheets + Drive.
6. You can keep “Use Default App” selected or choose Custom App. Click here to learn – how to register Custom Google App
7. If you choose the Custom App option then enter the below scopes (or click Select Scopes). If you choose the default app then it’s not required.

   This will allow read / write access to Drive Files or Sheet (E.g. Reading file content / Export file to CSV). You can use Scope browser to see many other available permissions.

   https://www.googleapis.com/auth/drive
   https://www.googleapis.com/auth/drive.file
   https://www.googleapis.com/auth/drive.readonly
   https://www.googleapis.com/auth/spreadsheets
   https://www.googleapis.com/auth/spreadsheets.readonly

   Click Generate Token button.

   

   SSIS OAuth Connection Manager – Access Google Drive API using OAuth 2.0

8. You will see UI as below… Click Accept (You may have to scroll to see that button sometimes)
9. Click Test to see connection is working.

NOTE: If you don’t want to use Default OAuth App provided by ZappySys then select “Use Custom OAuth App” option (App created in previous section) specify your ClientID, ClientSecret.

Find Google Sheet ID for API call

Before we can read or write Sheet data in SSIS we need to know Google Driver File ID. We will use this ID in next few sections.

Easiest way to find File ID is look at the URL like below. If you need to know ID at runtime dynamically then refer last section.

Get Google Drive File ID for API call (Sheet ID)

Reading Google SpreadSheet Data in SSIS

Basically there are two ways you can export / read Google SpreadSheet data.

• Method-1 : Read Google SpreadSheet using SSIS JSON / REST API Source
  • In this approach we directly pull data from SpreadSheet using Core Google SpreadSheet.Values API to read particular Tab from Sheet and you can also specify Range. This is most effective way to read. Make sure you have v2.5.0.10807 or higher version (released after Aug 03 2017). New version added 2D array transformation options to parse multi-dimensional arrays found in SpreadSheet API JSON response.
• Method-2 : Read Google SpreadSheet using Flat File Source (as CSV)
  • Another approach is export SpreadSheet as CSV format. Then use Flat File Source to read that data and load into Target (E.g. SQL Server)

Method-1 : Read Google SpreadSheet using SSIS JSON / REST API Source

Now lets look at real steps to read Google SpreadSheet. To get values from specific tab and specific cell-row range you have to use Google SpreadSheet.Values API

In this example we will load below Google Spreadsheet into SQL Server

Step-By-Step : Loading Google SpreadSheet into SQL Server using SSIS

1. Drag and drop data flow on the control flow designer surface
2. Double click data flow and you will see SSIS Toolbox refreshed with new components
3. Drag ZS JSON Source from SSIS Toolbox onto Data flow designer surface
4. Double click JSON SOurce and configure below setting
   1. Enter your Spreadsheet URL in the following format
      

      https://sheets.googleapis.com/v4/spreadsheets/{your-file-id}/values/{cell-range}

      For example purpose we will use this public SpreadSheet provided by Google. If spreadsheet fileid is 1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms (see previous section how to get fileid) and your tab name is Class Data and you like to extract data from A to F columns starting from 2nd row then your Cell range in the URL can be ‘Class Data’!A2:F
      Here is the actual URL we will use for demo
      

      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/'Class Data'!A2:F

      — OR —
      You can also enter like below (If you omit Tab name then first table is used)
      

      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A1:F
      
      --OR-- Use below URL for to supply Column names manually (Start from A2)
      
      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A2:F

      You can also enter like below, specify first row and last row
      

      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A1:F101

   2. Check Use credentials and select OAuth connection we created in previous section
   3. Click on Select Filter and select values node and click OK  or you can directly enter following filter.
      

      $.values[*]

      

      Configure SSIS JSON/REST API Source – Read data from Google SpreadSheet API

   4. Now click on array Transformation tab. Now we will show you two different ways to configure column names.
   5. Method-1: Auto Detect Column Names (added in version 3.1.2)
      In recent version we added column less array transformation where we have option to detect first record as column name.
      1. Select Transform Type as Column less array
      2. Check option First line has column names like below example (Ignore direct Sample JSON – We provided for demo only. In your case it will be URL)
         

         JSON With 2D Array Pattern – Column names in First Row (Google Sheets API Pattern)

   6. Method-2: Manually Enter Column Names (For old version)
      1. Select Transform simple 2D array option from dropdown
      2. Check Specify columns list manually
      3. Enter column names (Must match the order and count based on range you selected)
         

         Student Name,Gender,Class Level,Home State,Major,Extracurricular Activity

         

         Configure SSIS JSON/REST API Source 2D Array Transformation for Google SpreadSheet API

   7. Now click Preview to see the data
      

      Configure SSIS JSON/REST API Source –
      Preview Google SpreadSheet Data

   8. Click OK to save UI
   9. Attach your JSON Source to target (e.g. OLEDB Destination for SQL Server Table).
   10. Configure Destination
       

       Configure SQL Server destination for Google SpreadSheet to SQL Data load

   11. Execute SSIS Package to load Google SpreadSheet data int SQL Server
       

       SSIS Example -Loading data from Google SpreadSheet into SQL Server Table

Method-2 : Read Google SpreadSheet using Flat File Source (as CSV)

Once we have Google Drive fileId we can perform following actions to export data into CSV file format. After that you can easily read CSV using SSIS FlatFile Source and load into SQL Server or other target.

Export Google SpreadSheet as CSV file using SSIS REST API Task

Here are the steps to export Google Drive SpreadSheet to CSV file

1. Just like previous section configure REST API Task
2. Enter following URL (See we used FileId from SSIS variable. You can hardcode too.
   

   https://www.googleapis.com/drive/v3/files/{{User::varFileId}}/export?mimeType=text/csv

3. Goto response tab. Check Save response and select “Save to file option”.
4. Enter file path.
5. Response character set select utf-8 (This will allow Unicode characters in response)
6. Click Test to see preview
   

   Export Google SpreadSheet to CSV file format using SSIS – Call Google Drive API to Export

7. Click OK to save

Load Google SpreadSheet data into SQL Server using SSIS (REST API Call)

Once you export SpreadSheet to CSV file you can easily consume it to load into SQL Server or any Target (e.g. Oracle, MySQL) using Native SSIS FlatFile Source and OLEDB/ADO.net Destination.

Write data to Google Spread Sheet

Now lets look at how to write to Google SpreadSheet. To write multiple rows in a single request you may use batchUpdate API. Also check this link to learn more.

Below is sample request to write data to columns and some data to rows (set majorDimension = COLUMNS to write vertical and use ROWS to write horizontally. You can mix multiple ways just like below example)

POST 
https://sheets.googleapis.com/v4/spreadsheets/1lkMEgu0zm2Q-cnPeoNZp4hqeIuASxXSpsrxtDDV89tg/values:batchUpdate
Authorization: Bearer ya29.Gl3WBFb3xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: application/json
Accept: */*

{
  "valueInputOption": "USER_ENTERED",
  "data": [
    {
      "range": "Sheet2!A1:A4",
      "majorDimension": "COLUMNS",
      "values": [
        ["Item", "Wheel", "Door", "Engine"]
      ]
    },
    {
      "range": "Sheet2!B1:D2",
      "majorDimension": "ROWS",
      "values": [
        ["Cost", "Stocked", "Ship Date"],
        ["$20.50", "4", "3/1/2016"]
      ]
    }
  ]
}

Write data to Google SpreadSheet using REST API Task (OAuth 2.0 Connection)

Update Google Sheet Cells – Single Range (Set Values for multiple cells)

Now let’s look at the below example of updating multiple cells in google Sheet. For more information, you can see the example here. For parameter detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve Spread Sheet Update scenario. For simple example lets use REST API Task.

1. Drag and Drop ZS REST API Task on the control flow designer
2. Edit Task and select URL from Connection Method
3. Select OAuth Connection from the dropdown (Same connection we created in the previous section)
4. Enter the URL as below. Change Sheet ID, Range designator and API URL Parameters as per your need.
   https://sheets.googleapis.com/v4/spreadsheets/[Sheet-ID-Here]/values/[Range-Here]?[Parameters-Here] Example URL (some part masked with xxxxxxxx):  
   

   https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4?valueInputOption=USER_ENTERED

5. Change Request Method to PUT
6. Change Request Content type to application/json
7. Enter Request Body as below (Assuming we need to update 4 cells)
   

   {
   "majorDimension": "ROWS",
     "values": [
      ["row1_cellA","row1_cellB"],
      ["row2_cellA","row2_cellB"],
     ]
   }

8. Click Test Request see it works.
   

   Calling Google Sheet API to Update Multiple Cell Values

Update Google Sheet Cells – Multiple Ranges

If you like to update values which needs to define multiple ranges then you need to use batchUpdate API. Import changes are you can’t include Range in the URL like previous example of Single Range Update. Also you have to use POST method and body with multiple ranges.

Example of sheet update with multiple range

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHxxxxxxxxxxxxxxxc/values:batchUpdate?valueInputOption=USER_ENTERED
Content-Type: application/json

>>>> BODY <<<<<

{
  "data": [
    {
      "range": "Sheet1!A2:B3",
      "majorDimension": "ROWS",
      "values": [
        [
          "row1_cellA",
          "row1_cellB"
        ],
        [
          "row2_cellA",
          "row2_cellB"
        ]
      ]
    },
    {
      "range": "Sheet1!D2:E3",
      "majorDimension": "ROWS",
      "values": [
        [
          "row1_cellD",
          "row1_cellE"
        ],
        [
          "row2_cellD",
          "row2_cellE"
        ]
      ]
    }
  ]
}

Write SQL Server data to Google Sheet

Now let’s look at real world example. How to write data from SQL Server table or other source and send to Google Sheet.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox
   
2. Drag source (e.g. OLEDB Source) and configure it to read from source like SQL Server Table
   

   OLE DB Source – Drag and Drop

3. Drag ZS JSON Generator.
   1. Connect Source to JSON Generator (blue arrow)
   2. Double click it to configure
   3. Select Output Mode as Single Dataset Array option for Mode. Enter Some Batch value (e.g. 500) this will make sure we send 500 rows at a time to Google API call. This way if we have many rows its not rejected by API call.
   4. Click Add element (Select Static Element) . Name: majorDimension and Value: ROWS
   5. Now click Add Document Array icon, name as values
   6. check Treat as 2D array option and click OK to save
   7. Now under values node click Add elements > Select Multiple Columns and Add columns you like to add.
   8. Click OK to save JSON GeneratorBelow is just an example How you can use Batch Settings to create JSON with N rows in each JSON document.
      
4. Now drag ZS Web API destination
   1. Connect JSON Generator to Web API destination (blue arrow)
   2. Double click it to configure
   3. Select same OAuth connection manager created in earlier section (Assuming you had all scopes with write permission)
   4. Now select Input column for Body (JSON Generator Output)
   5. Enter API call URL as below (Replace ID with your own id)
      

      https://sheets.googleapis.com/v4/spreadsheets/{your-file-id}/values/Sheet1!A2?valueInputOption=USER_ENTERED

   6. Select Request Method as PUT
   7. Select content type as application/json
   8. You can enter sample Body As below.
      

      {
      "majorDimension": "ROWS",
        "values": [
         ["Bob","bob@mycompany.com",55],
         ["Sam","sam@mycompany.com",57]
        ]
      }

   9. Click Test Request / Response button. Check your Sheet in browser after it see it updates?
   10. If you see 200 OK response means we are good to go.
   11. Run package and see your real data updated in Sheet.

Format Cells using Google Sheet API

Now let’s look at an example to format cell using Google Sheet API (e.g. Change Font, Color, Border, Background). For more information review this API documentation.

To format Sheet you first need to get SheetID  (keep in mind this is not same as spreadsheetId which you used in URL). SheetId is basically numeric ID of your Tab in WorkSheet.

Get SheetID

Request

GET https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHtxxxxxxxxxxxxxx

Response

{
  "spreadsheetId": "1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc",
  "properties": {
    "title": "Test.xlsx",
    "locale": "en",
    "autoRecalc": "ON_CHANGE",
    "timeZone": "America/Los_Angeles",
    "defaultFormat": {
      "backgroundColor": {
        "red": 1,
        "green": 1,
        "blue": 1
      },
      "padding": {
        "right": 3,
        "left": 3
      },
      "verticalAlignment": "BOTTOM",
      "wrapStrategy": "OVERFLOW_CELL",
      "textFormat": {
        "foregroundColor": {},
        "fontFamily": "Calibri",
        "fontSize": 11,
        "bold": false,
        "italic": false,
        "strikethrough": false,
        "underline": false
      }
    }
  },
  "sheets": [
    {
      "properties": {
        "sheetId": 1896331083,
        "title": "Sheet1",
        "index": 0,
        "sheetType": "GRID",
        "gridProperties": {
          "rowCount": 1000,
          "columnCount": 26,
          "rowGroupControlAfter": true,
          "columnGroupControlAfter": true
        }
      }
    }
  ],
  "spreadsheetUrl": "https://docs.google.com/spreadsheets/d/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc/edit"
}

Notice that in above response we see “sheetId”: 1896331083. This ID we will use in the Update API to format Cells.

Update Borders Example

Here is the example of Google Sheet Formatting API updateBorders Request

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSxxxxlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "updateBorders": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 0,
          "endRowIndex": 10,
          "startColumnIndex": 0,
          "endColumnIndex": 6
        },
        "top": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
        "bottom": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
        "innerHorizontal": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
      }
    }
  ]
}

Response

{
  "spreadsheetId": "1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc",
  "replies": [
    {}
  ]
}

Update Cell Font Style (Bold, Style, Color, Background)

Now let’s look at how to use repeateCellRequest to apply certain type of formatting to multiple cells. This may be color, font style, background color.

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3xxxxxx4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "repeatCell": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 0,
          "endRowIndex": 1
        },
        "cell": {
          "userEnteredFormat": {
            "backgroundColor": {
              "red": 0,
              "green": 0.5,
              "blue": 0.7,
              "alpha": 1.3
            },
            "textFormat": {
              "foregroundColor": {
                "red": 1
              },
              "bold": true,
              "italic": true,
              "strikethrough": true,
              "underline": true
            }
          }
        },
        "fields": "userEnteredFormat"
      }
    }
  ]
}

In above example we have applied textFormat and backgroundColor but you can use any of below sections.

Set Currency Formatting

Here is an example request to set currency formatting

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "repeatCell": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 1,
          "startColumnIndex": 4,
          "endColumnIndex": 5
        },
        "cell": {
          "userEnteredFormat": {
            "numberFormat": {
              "type": "CURRENCY",
              "pattern": "$#,##0.00"
            }
          }
        },
        "fields": "userEnteredFormat.numberFormat"
      }
    }
  ]
}

Set Cell Validation

You can also set cell validation by calling following API call. This will restrict user’s ability to enter wrong data in certain cells.

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "setDataValidation": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 1,
          "startColumnIndex": 0,
          "endColumnIndex": 6
        },
        "rule": {
          "condition": {
            "type": "ONE_OF_LIST",
            "values": [
              {
                "userEnteredValue": "Red"
              },
              {
                "userEnteredValue": "Green"
              },
              {
                "userEnteredValue": "Yellow"
              }
            ]
          },
          "inputMessage": "Color must be Red, Yellow or Green",
          "strict": true,
          "showCustomUi": true
        }
      }
    }
  ]
}

Dynamically obtain File ID for Google Drive REST API call

Once OAuth connection is defined you can now call any valid API from Google based on Permission you requested in OAuth Connection. OAuth connection takes care of renewing you Token and passing Authorization as per OAuth standard. All these protocol complexity is hidden from you.

First step to get data from Google SpreadSheet using REST API call is get fileId of the Google Drive File you want to read. The easiest way to know file ID is call following API. Below API call returns name, ID and other information about file(s). You can get only fileID and use that in Next Section to export Google SpreadSheet to CSV format.

https://www.googleapis.com/drive/v3/files

OR

https://www.googleapis.com/drive/v3/files/?q=name%3D'Customers'

NOTE: For more information about searching files in Google drive / getting list of files using search expression language check this link. Make sure expression is URL encoded (e.g. name=’Customers’ should be passed as name%3D’Customers’)

Above API returns response as below. Notice the name and fileId. We will use fileId when calling export API in next section.

{
 "kind": "drive#fileList",
 "files": [
  {
   "kind": "drive#file",
   "id": "1lkMEgu0zm2Q-cnPeoNZp4hqeIuASxXSpsrxtDDV89tg",
   "name": "Customers",
   "mimeType": "application/vnd.google-apps.spreadsheet"
  },
  {
   "kind": "drive#file",
   "id": "0B6UyXr6ZlIFTc3RhcnRlcl9maWxl",
   "name": "How to get started with Drive",
   "mimeType": "application/pdf"
  }
 ]
}

If you want to automate retrieval of fileId from document name then perform the following steps.

1. In the SSIS package Control flow. Drag and drop ZS REST API Task from SSIS toolbox
2. In the Request Access mode, select “URL from Connection” option
3. Select OAuth connection manager (Created in Section-2)
4. Enter URL as below (It will return any file name equals to Customers (%3D is encoded value for = sign)
   

   https://www.googleapis.com/drive/v3/files/?q=name%3D'Customers'

5. Click Test (You will see file information)
   

   Get Google Drive File List in SSIS – Call Google Drive API to search file by name or expression

6. If you want to extract only fileID from full JSON then go to response tab
7. Select Format=JSON, Enter following JSONPath expression to extract only fileId from first record
   

   $.files[0].id

8. Check save response content, select SSIS variable where you want to save fileId as below. Now click Test to see preview of response. You will notice now only fileId will be returned from full JSON response.
   

   Get Google Drive fileid from file name using API call in SSIS

9. Click OK to save task

Conclusion

Google Drive API and Google Sheets API provides a great way to automate file related functionality. However, to call Google API  you have to use SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Google API call via simple drag and drop approach without coding. Try out yourself see how long it takes to call virtually any REST API.

The post How to read / write Google SpreadSheet using SSIS appeared first on ZappySys Blog.

Google AdWords is one of the most popular Advertise Platforms. In this article, you will learn how to get data from Google AdWords by calling Google AdWords Web Service method and using drag and drop SSIS XML/SOAP connector (No coding required :)). Google AdWords Web Services allows retrieving various data such as account info, campaigns; as well as writing data, such as creating a campaign, add a keyword. Basically, it allows controlling your AdWords account programmatically instead of using the AdWords dashboard. For demo purposes, we will get data from Google AdWords Account and load it into Microsoft SQL Server (but you can use this technique to save any supported targets in SSIS (e.g. Excel, FlatFile, Oracle, MySQL, IBM DB2, etc.).

If you desire to download a report from AdWords, you must follow a different approach described in an article on how to download AdWords report. It describes how to download a report using ODBC and XML Driver but the concepts introduced there can be applied to SSIS as well.

If you are looking for a similar article for Google DoubleClick for Publisher (Google DFP) then check this article.

You will also learn how to create Google Ads or modify any other objects by calling SOAP API using SSIS REST API Task.

SSIS Tasks/Connectors discussed in this Article

In this article we will use following Connector(s) and Connection Manager to archive full data integration with Google AdWords.

Following SSIS Connectors support OAuth Connectivity. When you use below connectors no coding required to pull data from any OAuth enabled services (e.g. Twitter, Facebook, LinkedIn).

	REST API Task
	OAuth Connection Manager
	XML Source (REST API, SOAP Call or File data)

Two different ways to get Google AdWords Data in SSIS

There are two ways you can extract Google AdWords data in SSIS.

1. Use Google Analytics Source (For more info see this article)  (Consume AdWords data using Google Analytics REST API)
2. Use XML Source along with OAuth connection manager (Consume AdWords data using SOAP API)

In this article we will only discuss Approach #2. For approach #1 refer see this link

Advantage of #2 is you get total control of full API set for AdWords but you have to do lot more manual work. For Approach #1 is lot simpler but you get access to limited reports (Dimensions and Metrics).

Google AdWords API Integration Example Screenshot

Below is screenshot of sample package discussed in this article (Download from here). As you can see we tried to demonstrate Reading data and Managing data (e.g. Create Ad) in the same package. You can extend these technique to perform any type of operations in Google AdWords or similar service such as DoubleClick for Publisher

Google AdWords API Integration Example – Create Ad, Get Accounts, Get Campaigns, Get Ad Groups, Get Ads (Extract AdWords data via SOAP API, Read data using Selector Method or use AWQL query language)

Step-By-Step Tutorial

Here are few high level steps you will have to perform to extract data from Google AdWords API using SSIS. This section assumes you have ClientID and Developer Token to access data. if you don’t have that then read next 2-3 sections.

1. Download and install SSIS PowerPack from here
2. Create new SSIS Project
3. From SSIS toolbox drag DataFlow Task and go to dataflow designer
4. From toolbox of drag ZS XML Source
5. Double click XML Source to configure it
6. Specify API URL from which you want to read data
   Example : https://adwords.google.com/api/adwords/mcm/v201509/ManagedCustomerService
7. Now check Use Credentials check box and select New ZS-OAuth connection manager.
8. On connection manager UI you can select Google from Provider Dropdown.
9. Under Scopes enter following (Scope is basically permission)
   

   https://www.googleapis.com/auth/adwords

    
10. Now click Generate Token and when prompted login and in the last step Accept.
11. If everything went ok then you will see AccessToken and RefreshToken populated.
12. Click Test connection and close the UI by clicking OK
13. Now on XML Source – Select HTTP Request Method as POST
14. In the Request Body click edit and enter request as below (Replace DEV_TOKEN and CLIENT_ID placeholders below)
    

    <?xml version="1.0" encoding="utf-8"?>
    <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    	<soap:Header>
    		<RequestHeader xmlns="https://adwords.google.com/api/adwords/mcm/v201605">
    			<developerToken>ENTER_YOUR_DEV_TOKEN_HERE</developerToken>
    			<clientCustomerId>ENTER_YOUR_CLIENT_ID_HERE</clientCustomerId>
    			<userAgent>ZappySys SSIS PowerPack v1</userAgent>
    		</RequestHeader>
    	</soap:Header>
    	<soap:Body>
    		<get xmlns="https://adwords.google.com/api/adwords/mcm/{{User::ApiVersion}}">
    			<serviceSelector>
    				<fields>CustomerId</fields>
    				<fields>Name</fields>
    			</serviceSelector>
    		</get>
    	</soap:Body>
    </soap:Envelope>

     

    If you are wondering how did we get above XML fragment for Request Body then Check this article on how to use SoapUI (free 3rd party tool) to create SOAP request Bodyfrom WSDL. Google AdWords has many API endpoints for different service. For each service you have different Api URL and different WSDL. In the next section you can learn how to create correct SOAP Body (for POST) using SoapUI tool.

    For example if you are calling ManagedCustomerService API for version v201609 then your help page URL would be like this
    https://developers.google.com/adwords/api/docs/reference/v201609/ManagedCustomerService

    And on the same page you will see WSDL link below
    https://ads.google.com/apis/ads/publisher/v201702/NetworkService?wsdl
    (Just download that WSDL XML and save to local disk then use with SoapUI to generate XML body to submit for any DFP API call.)

    See this link for more information (Its for Google DoubleClick Ad Platform but you will get an idea)

15. Go to Array handling tab at the bottom and enter text : ns2:entries – By doing this we will treat <ns2:entries> node as array eventhough only one entry found in response (By default content with multiple records treated as array).
    

    Handling Array Element Nodes in SOAP Response

16. On the filter expression click edit and highlight entries node and click ok. This will produce something like below in filter textbox
    

    $.soap:Envelope.soap:Body.ns2:getResponse.ns2:rval.ns2:entries[*]

     
17. Now click preview to see actual data.
18. Click OK to save UI. Now you can connect source to target such as SQL Server or Trash Destination.

NOTE: in above example we used v201605 API version but change it to latest if this API is no more valid. Google expires API version pretty often so keep that in mind if you get depreciated API version error.

Register for Developer Token to access AdWords account data using API

First important step to access any Google AdWords Account is request developer token. Click this link for step by step information. Watch short vide to get overview on entire process to access data via API.

Find your Client ID

Next thing you will need to know is your ClientID. Most of API Requests will require your ClientID. Perform following steps to find your ClientID.

Login to your AdWords Account using email address. At the top you will see your client ID as below screenshot.

How to find ClientID of your Google AdWords Account (Useful for making SOAP API Calls)

Connect to Google AdWords API using OAuth

Connecting to any Google API will require OAuth 2.0. SSIS PowerPack has Protocol so you can easily extract data from API without coding complex token extract/renew workflow. Such things usually requires some sort of programming but in our case you will do few clicks as described in Step-By-Step section earlier.

Here is the screenshot of OAuth Connection Manager.

Connecting to Google AdWords API using OAuth 2.0 in SSIS (Consuming XML/SOAP API)

Extract data from Google AdWords using SOAP API Calls

When you define filter expression you can check/uncheck “Include Parent Columns” setting. For AdWords SOAP Api you can uncheck so you only extract columns from rvals node in XML Response.

Extract data from Google AdWords SOAP/XML Response

SOAP Response Paging / Looping

Many time you will get response which is larger than default page size in AdWords API Call. In that case you can specify few pagination attributes on XML Source Paging tab. For more information on paging options you may refer this article . This article is talking about JSON Source but same options are applicable for XML Source too.

REST API Pagination – Passing PageNumber inside POST Body (Example: Google AdWords SOAP Api)

Configure REST API Pagination – using Page Number via POST Body Method

Read Google AdWords data and Load into SQL Server

Using XML Source you can extract data in tabular format and then you can map that to SQL Server Target (Use OLEDB Destination / ADO.net Destination).

Extract data from Google AdWords and load into SQL Server table using SSIS

Write data / Manage AdWords Account via REST API Calls

So far we have seen how to read data from AdWords account. But what if you have to create new records or write/update AdWords data via API Calls in automated manner? No worry – If you have need for that then use SSIS REST API Task which supports adhoc HTTP WebRequests including POST/DELETE/PUT etc. You can also upload files via REST API call using this task. This task supports rich error handling and saving your response into variable or files.

Create Ad under specific AdGroup

Below example shows how to create new ad under specific AdGroup using REST API Task

Notice we have used few placeholders to make things dynamic using SSIS Variables. Once REST API Task is executed you will see new ad created.

Creating Google AdWords Ad using SOAP/XML API in SSIS (REST API Task)

Verify Google Ad created programmatically using SOAP/XML API Call (SSIS REST API Task)

Methods for Creating new records

Google AdWords API supports mutate method for writing data. Many service endpoints supports this method. Below example request will create new Ad.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<developerToken>{{User::varDevToken}}</developerToken>
			<clientCustomerId>{{User::varClientId}}</clientCustomerId>
			<userAgent>My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)</userAgent>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<mutate xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<operations>
				<operator>ADD</operator>
				<operand>
	                <adGroupId>30800928123</adGroupId>
	                <ad xsi:type="TextAd">
	                    <displayUrl>//zappysys.com/ad1</displayUrl>
	                    <finalUrls>//zappysys.com/ad1</finalUrls>
	                    <headline>This is my Ad</headline>
	                    <description1>This is Line 1</description1>
	                    <description2>This is Line 2</description2>
	                </ad>            
				</operand>
			</operations>
		</mutate>
	</soap:Body>
</soap:Envelope>

Methods for Reading AdWords data

Google AdWords API supports two methods for reading data. Most of AdWords service endpoints supports GET method also known as Selector method (Its basically sending query in XML format). Seconds method is Query Method. It uses SQL like query language known as AWQL. (Read more about AWQL query language). If you are SQL Programmer then you may like AWQL Query method better than Selector. But either way you will be able to pass following information.

• Columns you want to return
• Filter (e.g. Where Clause)
• Order By
• Paging information (e.g. PageNumber, PageSize)

Get Method (XML Selector)

For Get Method using XML tags to define columns, filters and order by/paging information. Here is example Get method for below select query

SELECT Id, Heading
WHERE Status = 'ENABLED'
ORDER BY Heading DESC 
LIMIT 0,50

Get Request (Get all Active Ads)

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<developerToken xmlns="https://adwords.google.com/api/adwords/cm/v201605">{{User::varDevToken}}</developerToken>
			<clientCustomerId xmlns="https://adwords.google.com/api/adwords/cm/v201605">{{User::varClientId}}</clientCustomerId>
			<userAgent xmlns="https://adwords.google.com/api/adwords/cm/v201605">My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)</userAgent>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<get xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<serviceSelector>
			    <fields>Id</fields>
			    <fields>Headline</fields>
			    <predicates>
			        <field>Status</field>
			        <operator>EQUALS</operator>
			        <values>ENABLED</values>
			    </predicates>
			    <ordering>
			        <field>Headline</field>
			        <sortOrder>ASCENDING</sortOrder>
			    </ordering>
			    <paging>
			        <startIndex>0</startIndex>
			        <numberResults>50</numberResults>
			    </paging>
			</serviceSelector>
		</get>
	</soap:Body>
</soap:Envelope>

Query Method (AWQL Query – SQL like query language)

Below is sample Soap request for AWQL Method to read Ads listed under from specific AdGroups. To learn more AWQL Query Syntax Click Here.

To learn more about which Columns you can use inside your AWQL Query Click here. Note that each service support different fields for Select Clause, Where Clause and Order By Clause. AWQL language can be also used to query Predefined AdWords Reports. The only difference is .. FROM clause is required when you query Reports.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<developerToken xmlns="https://adwords.google.com/api/adwords/cm/v201605">YOUR_DEV_TOKEN</developerToken>
			<clientCustomerId xmlns="https://adwords.google.com/api/adwords/cm/v201605">YOUR_CLIENT_ID</clientCustomerId>
			<userAgent xmlns="https://adwords.google.com/api/adwords/cm/v201605">My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)</userAgent>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<query xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<query>
SELECT Id, Name
WHERE Status = 'ENABLED'
LIMIT 0,5
			</query>
		</query>
	</soap:Body>
</soap:Envelope>

Call AdWords Reports API – Download Campaign Performance data

There will be a time you like to download Reports from Adwords. For that  read this article. Its showing how to use ZappySys ODBC Driver to call AdWords Reporting API but concept is same in SSIS Product too due to similar UI elements.

Download Example SSIS Package

Download SSIS 2012 / 2014/ 2016 Sample

Conclusion

Google AdWords data integration can be very challenging without right set of tools. SSIS PowerPack certainly makes it easy when you have to extract/manage Google AdWords data. Innovative drag and drop approach gives you various options to cover simple or most advanced scenarios without relying on any Programming language (e.g. C#, Java, Python, Ruby etc).

The post How to get data from Google AdWords using SSIS appeared first on ZappySys Blog.