In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate Google Drive into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful Google Drive Connector that makes uploading files to Google Drive a breeze. In this guide, we’ll walk you through the process of uploading files to Google Drive using SSIS and the ZappySys SSIS PowerPack Google Drive Connector.

Upload files to Google Drive with SSIS:

In this section, we will learn how to configure and use Google Drive Connector in the API Source to upload files on Google Drive.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom
   

   SSIS Toolbox – ZappySys Tasks

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the Google Drive Connector

1. Use a preinstalled Google Drive Connector from the Popular Connector List or press the Search Online radio button to download Google Drive Connector. Once downloaded simply use it in the configuration:
   
2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to Google Drive using User Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with User Account

   Method2: Connect to Google Drive using Service Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with Service Account

How to upload a local file to Google Drive

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to upload a local file to Google Drive.

1. Choose the Upload a file endpoint from the “Select Table/Endpoint” dropdown menu.
2. To access detailed API documentation for uploading files, you can click on the following link: Help for: Upload File.
3. Please specify the desired filename for the uploaded file by setting the FileName parameter.
4. Specify the desired local file path by setting the DiskFilePath parameter.
5. If you wish to upload the local file to a specific folder in Google Drive, please choose the folder ID from the dropdown menu labeled ParentFolderId. Otherwise, you can skip this parameter as it’s optional.

Please refer to the sample screenshot below.

SSIS API Source – Google Drive – Upload Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for uploading a local file to Google Drive.

How to upload multiple files to Google Drive

In the API Source Component, we’ve selected the API Connection Manager. Next, we need to configure the remaining properties of the component to upload multiple files to Google Drive. Additionally, we should utilize a Foreach Loop Container to iterate through each file one by one in a loop.

1. Create 3 SSIS variables as described below.
   

   Variable Name	Datatype	Value
   varFileLists	Object	System.Object
   varFileName	String	fake.extension
   varFullFilePath	String	C:/fake/fake.extension

2. Go to Control Flow, From the SSIS toolbox drag and drop ZS Advanced File System Task on the control flow designer surface, and double click on it to edit it:
   
3. Select the “Get file list as ADO.NET Data Table” action from the Action drop-down menu. Set the file path in the Path textbox. Choose the variable (Object type) where you want to store the file list, and then click the OK button to save the settings.
   

   SSIS Advanced File System Task – Get file list as ADO.net Data Table

4. Now, drag and drop the “Foreach Loop Container” from the SSIS Toolbox and connect it with the “ZS Advanced File System Task.” Double-click on the “Foreach Loop Container” to edit its properties.
   

   SSIS Foreach Loop Container – Drag & Drop

5. In the Foreach Loop Editor’s “Collection” tab, choose the Enumerator as “Foreach ADO Enumerator.” In the “ADO object source variable” field, select your SSIS object-typed variable. Select the “Rows in the first table” radio option.
   

   SSIS Foreach Loop Editor – Collection – Select Variable & Enumeration mode

6. In the Foreach Loop Editor’s “Variable Mappings” tab, select the variables that you want to map to the values from the collection, and then click the “OK” button to save your selections.
   

   SSIS Foreach Loop Editor – Variable Mappings

7. Inside the Foreach Loop Container, drag and drop the Data Flow Task, and then rename it according to your preference.
   

   SSIS Data Flow Task – Drag & Drop inside Foreach Loop Container

8. Double-click on the Data Flow Task to see the Data Flow Designer, From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double click on it to edit it:
   
9. Double-click on the API Source component to configure it for uploading multiple files to Google Drive.
   1. Choose the Upload a file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. To access detailed API documentation for uploading files, you can click on the following link: Help for: Upload File.
   3. Please specify the desired SSIS Variable [varFileName] which contains file name for the uploaded file by setting the FileName parameter.
   4. To specify the desired local SSIS variable [varFullFilePath], which holds the file path, set the DiskFilePath parameter.
      When configuring the SSIS variable directly in the Raw Edit, JSON may become invalid due to the requirement of slashes within the local file path. To address this, we use a JSONENC Placeholder Function for proper escaping.
      

      <<{{User::varUserFullFilePath}},FUN_JSONENC>>

   5. If you wish to upload the local file to a specific folder in Google Drive, please choose the folder ID from the dropdown menu labeled ParentFolderId. Otherwise, you can skip this parameter as it’s optional.

   Please refer to the sample screenshot below.

   

   SSIS API Source – Google Drive – Upload Multiple files configuration

   To configure dynamic filenames and file paths, you can set variables. Click on “Raw Edit,” and in the “Edit Parameters” popup, you can use SSIS variables, as shown in the image below.

   Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for uploading multiple files to Google Drive.

Conclusion

In this blog, we learned how to connect with Google Drive using SSIS API Connection Manager and upload single or multiple files to Google Drive using a combination of SSIS components, including the SSIS API Source component, SSIS ZS Advanced File System Task, Foreach Loop Container, and SSIS variables.

The post How to upload files on Google Drive using SSIS appeared first on ZappySys Blog.

In our previous few posts we saw how to call various google apis in SSIS.  In this post lets learn how to call Google Search Console API in SSIS or other ODBC Compatible Apps such as Power BI, Informatica, SSRS using API Drivers for ODBC

About Google Search Console API (Google Webmaster API)

If you are new to Google Search Console API (i.e. Webmaster API) then start from this link. You can also check API Explorer here to test API requests.

Search API can be called by supplying either API key in URL  or use OAuth

Search Console V3 Covers following APIs (May add more in future)

Search Console API Call Example

Here is simple example of Search Console API call. Below example returns all sites you added under your search console account.

Request

GET https://www.googleapis.com/webmasters/v3/sites?key={YOUR_API_KEY}

Response

{
 "siteEntry": [
  {
   "siteUrl": "https://my-google-search-api-1.com/",
   "permissionLevel": "siteOwner"
  },
  {
   "siteUrl": "https://www.my-google-search-api-1.com/",
   "permissionLevel": "siteOwner"
  },
  {
   "siteUrl": "https://my-google-search-api-2.com/",
   "permissionLevel": "siteOwner"
  },
  {
   "siteUrl": "https://www.my-google-search-api-2.com/",
   "permissionLevel": "siteOwner"
  }
 ]
}

Step-By-Step – Call Search Console API in SSIS

Now lets look at very simple API call to list search console sites

1. Open SSIS Package
2. Drag and drop ZS REST API Task from SSIS tool box
   

   Drag and Drop ZS REST API Task from SSIS Toolbox

3. Now double click REST API Task and configure (Skip Step 5 if you use API Key instead of OAuth).
4. Enter API URL you like to call. For This example we will use simple URL as below
   OAuth based Authentication
   

   https://www.googleapis.com/webmasters/v3/sites/

   API Key based Authentication
   

   https://www.googleapis.com/webmasters/v3/sites/?key={YOUR_API_KEY}

5. If you want to use OAuth based credentials then perform the following steps
   1. Select URL from Connection 
   2. Click New ZS-OAUTH connection from Dropdown
   3. On the OAuth UI select Google as Provider
   4. Enter Scopes as below
      

      https://www.googleapis.com/auth/webmasters
      https://www.googleapis.com/auth/webmasters.readonly

   5. Click Generate Token. When you get Login Prompt enter your Google Account information and click Accept
   6. Click OK to Save OAuth UI
6. This is how it will look like if you use OAuth Connection
   

   Call Google Search Console API using SSIS REST API Task (OAuth Authentication)

7. Now click Test Request / Response to check if its working. You should get response like below

Read from Search Console Analytics API (query)

Now let’s look at how to call Search Console Analytics API (i.e. query) for Specified Site and extract data in tabular format using ZS JSON Source. In below example we will obtain search Impression, clicks, position etc for specified site. We will aggregate this over date and country (dimensions).

In this section you will learn how to use JSON Source Adapter to extract data from JSON file (In this case its Web URL).

1. Firstly, You need to Download and Install SSIS ZappySys PowerPack.
2. Once you finished first step, Open Visual Studio and Create New SSIS Package Project.
3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
4. Double click on the Data Flow task to see Data Flow designer surface.
5. From the SSIS toolbox drag and drop JSON Source on the Data Flow designer surface.
   
6. Double click JSON Source and configure like below
   1. Enter URL as below. Notice two things, first we used <<somedata,FUN_URLENC>> placeholder function to encode : and // in the URL so it become %2F %3A. We use

      --dont-escape--

        suffix at the end.
      For OAuth based Credentials
      

      https://www.googleapis.com/webmasters/v3/sites/<<https://mysite.com,FUN_URLENC>>/searchAnalytics/query--dont-escape--

      For API Key based Credentials
      

      https://www.googleapis.com/webmasters/v3/sites/<<https://mysite.com,FUN_URLENC>>/searchAnalytics/query/?key={YOUR_API_KEY}--dont-escape--

   2. Check Use Credentials and select OAuth connection (created earlier) (if you are using API key in URL then skip this step).
   3. Select POST as HTTP Method
   4. Enter Body for Query. Here are some examples for various queries you can use. For our case we want to group by date and country so use below
      

      {
      "startDate": "2019-01-01",
      "endDate": "2019-01-10",
      "dimensions" : ["date","country"]
      }

   5. Select Content Type as

      application/json

        from Content Type drop down.
   6. Select Filter or enter as 

      $.rows[*]

   7. Now go to Extract Multiple Arrays Tab and Check Array Flattening Option (on older version it was on the different Tab)
7. Click Preview and you will see your Google Search Console API data as below.
   

   Read from Google Search API Console (Analytics Query Endpoint) – JSON Source

Loading Google Search Console API data into SQL Server / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Import Google Search Console API data in Reporting / other ETL tools (ODBC Usecase)

There will be a time when you dont want to use SSIS connector like mentioned in previous sections but extract Google Search API data in Reporting Tools / Other ETL Platform.  Good news is you can use ZappySys ODBC Driver for JSON / REST API in ODBC Apps. See few popular ODBC Apps below.

• Power BI connection for Google Search Console API
• Excel connection for Google Search Console API
• Informatica connection for Google Search Console API
• MS Access connection for Google Search Console API
• C# , PowerShell, VB.net connection for Google Search Console API

Preview / Generate Query in JSON Driver / XML Driver

Load Google Search Console API in SQL Server without any ETL

Now let’s look at even more interesting integration scenario. If you have SQL Server and like to load Google Search Console API without any ETL then you can use Data Gateway

Read this article to learn how to load REST API in SQL Server without any coding (Use just T-SQL) .

You can write query like below to load data inside SQL Table.

SELECT * FROM OPENQUERY([MY_LINKED_SERVER]
, 'SELECT * FROM $
WITH(
	 Src=''https://www.googleapis.com/webmasters/v3/sites/<<https://mysite.com,FUN_URLENC>>/searchAnalytics/query/?key={YOUR_API_KEY}--dont-escape--''
	,Filter=''$.rows[*]''
	,RequestData=''{
"startDate": "2019-01-01",
"endDate": "2019-01-10",
"dimensions" : ["date","country"]
}''
	,RequestContentTypeCode=''ApplicationJson''
	,Header=''cache-control: no-cache || Accept: */*''
	,RequestMethod=''POST''
	,EnableArrayFlattening=''True''
)')

SSMS Output – Query REST API via Linked Server OPENQUERY statement (Connect to ZappySys Data Gateway)

Conclusion

So in this post we saw how easy it is to achieve total REST API integration in SSIS using JSON Connector. We also saw how to use ODBC Drivers to query REST API data inside Apps like Excel, MS Access, Informatica and other ODBC Apps using ODBC JSON / REST API driver. If you are SSIS User download SSIS PowerPack and try for FREE and if you don’t have SSIS in house and want to try more generic approach then Download ODBC PowerPack Drivers for FREE Trial.

The post Get data from Google Search Console API in SSIS and ODBC Apps appeared first on ZappySys Blog.

Google BigQuery is a fully managed Big Data platform to run queries against large scale data. In this article you will learn how to integrate Google BigQuery data into Microsoft SQL Server using SSIS. We will leverage highly flexible JSON based REST API Connector and OAuth Connection to import / export data from Google BigQuery API just in a few clicks.

If you are looking for a similar product inside Amazon AWS Cloud then check an article about Amazon Athena.

Prerequisites

Before we look into a Step-By-Step section to extract and load data from BigQuery to SQL Server let’s make sure you meet the following requirements:

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have basic familiarity with REST API concepts and Google BigQuery API. This post uses free Public Dataset to query BigQuery table so no billing is required to get your hands dirty 🙂 However, if you like to query your own dataset then make sure you have at least one BigQuery dataset and one table with some sample data created (this part does require billing enabled). Read a Google Quickstart article for more information on how to create a new BigQuery dataset and a table.

Understanding Google BigQuery Object Heirarchy

Google BigQuery has 3 main concepts below.

• Project
  • Dataset
    • Table
      • Query requests (each query creates a unique JobID – valid for 24 hours from which you can read data)

So in order to create BigQuery table you always need Project and then Dataset under that project. You can group objects around Project / Datasets. When you query you can supply fully qualified name of your table in FROM clause (e.g.

select count(*) from `bigquery-public-data.usa_names.usa_1910_2013`

So let’s get started.

Read data from Google BigQuery using SSIS

Basically you can query Google BigQuery data in two ways: In this article we will not cover 2nd method.

• Method-1: Query data using jobs/query method in BigQuery API. Use it if you expect to get a result in a fairly short amount of time. This API method generates a temp table which gets deleted after 24 hours. You can read data within that time frame using newly created JobId reference.
  
  NOTE: If you have records that sum up to more than 10MB of data or ~10,000 rows (assuming 1 row uses 1KB of data) then you need to proceed with two-step process as explained below.
• Method-2: Export SQL query result using jobs/insert method in BigQuery API. Use this method if you expect a query to take a long time to finish.

Now let’s see how to query Google BigQuery data using SSIS 🙂

Create Google API Project and Register OAuth App

To use any Google API firstly you have to finish two tasks:

1. Create Google API Project and obtain projectId (see next section).
2. Enable BigQuery API
3. Register your own Google OAuth App and obtain ClientId and Client Secret.

Check step-by-step instructions on how to create Google API Project and create OAuth App for Google. During instructions make sure you enable BigQuery API (screenshots may differ).

Get Google API Project ID

Once you have Google API Project created you can grab Project ID (see screenshot below). You will need Project ID in the next step when building API URL.

How to find Google API Project ID?

Configure OAuth Connection for Google BigQuery API

In order to call most of Google APIs you will need an OAuth App. If you want to use your own app then refer to how to register Google OAuth Application (Get ClientID and ClientSecret) article. For simplicity we will use Default OAuth App:

1. Right click inside Connection Managers area and click “New Connection…”
2. From the connection type list select “ZS-OAUTH” connection type.
   

   Create new SSIS OAuth API Connection Manager

3. For OAuth Provider select “Google”.
4. For OAuth App type options leave “Use Default OAuth App” selected. If you created custom OAuth App then you would need to select “Use Custom OAuth App” option.
5. For Scopes enter or select the following URLs (URL is just a permission name). Each URL must be in a separate line. For demo purposes we use only the first 3 scopes but we included a few more in case you like to test API in depth with a different permission set:
   

   https://www.googleapis.com/auth/bigquery
   https://www.googleapis.com/auth/cloud-platform
   https://www.googleapis.com/auth/cloud-platform.read-only
   https://www.googleapis.com/auth/bigquery.insertdata
   https://www.googleapis.com/auth/devstorage.full_control
   https://www.googleapis.com/auth/devstorage.read_only
   https://www.googleapis.com/auth/devstorage.read_write

6. Click Generate Token. Login using correct account if needed and then you will be prompted to click “Approve” OAuth App.
7. Once you click OK on the login screen you will see new tokens populated on the connection screen.
8. Click Test to make sure connection / tokens are valid and then click OK to save the connection.
   NOTE: Below screenshot uses Default App but we recommend you to to use Custom OAuth App (Your own Clientid / secret – Obtained in previous section)
   

   Create SSIS OAuth API Connection for Google BigQuery API

Start BigQuery Job and get JobId (Submit Query)

Once you have SSIS OAuth connection created for BigQuery API it’s time to read data from BigQuery. So, basically, there are two ways you can read BigQuery data: using query or insert method. For demo purposes we will use jobs/query method. If you want fire complex queries which can run for many minutes then refer to jobs/insert method.

If you are expecting more than 10MB of data (~10K records) then you have to split data extract in two steps:

1. Step-1 : Call jobs/query method using REST API Task to submit SQL query. This steps returns jobComplete: true if supplied query execution is done within timeoutMs parameter you included. By default its 10 seconds so if your query is going to take longer than 10 seconds then its good idea to add bigger timeout that way you dont need step#2. Once API call is done you will Get JobId in the response. Save that JobId in a variable for later use. This job result is valid for 24-hours only.
2. Step-2 (Optional) – You can add another optional REST API Task after previous step to Wait until Job is completed. For simplicity, this article will Omit setting up Status Check but its very simple….Use below settings
   1. Drag new REST API Task from toolbox. Connect it to Step-1 and double click to configure as below.
   2. Select URL From Connection mode,
   3. Select OAuth connection
   4. Enter URL as https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/jobs/{{User::JobId}} , Method: GET
   5. On Response Settings Tab, Select Response Content Type JSON and for content filter enter $.status.state
   6. On Status Check Tab, check Enable Status Check Loop and enter DONE in SuccessValue field
   7. Doing this setup will make sure we do not query data until Job Status is DONE (System keeps checking every 5 seconds)
3. Step-2 : Read Job result using JSON Source in a Data Flow Task.

Now lets see how to configure REST API Task to submit query and extract JobId (which will be used in next section).

1. In the Control Flow designer drag and drop ZS REST API Task from the SSIS toolbox.
2. Double click the task to configure it.
3. Change Request URL Access Mode to [Url from connection].
4. From the connection dropdown select OAuth connection created in the previous section.
5. Enter URL as below (replace YOUR-PROJECT-ID with a valid Project ID obtained in the previous section):
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-PROJECT-ID/queries

6. Select Method as POST.
7. Enter Body (Request Data) as below (change your query if needed; we used Public dataset for the demo): If you do not specify timeout then default is 10 seconds only. Also in this call we only care about JobID from response so we just added maxResults=10 because in 2nd step we will get all rows by doing pagination.
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000"
   }

   To use a Standard SQL query instead of Legacy SQL, add useLegacySql property and set it to false:
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000",
      "useLegacySql": false
   }

   For all possible parameters refer to jobs/query method documentation.
   NOTE: You can also supply location Parameter in the Body JSON to indicate where job should run. For non EU / US datacenters we suggest you to supply this parameter. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location.
   Example Use Of Location
   

   { "location":"us-east1",  "maxResults": 10, "query": "SELECT title FROM [bigquery-public-data:samples.wikipedia] LIMIT 10" }

8. Select Body Content Type as application/json.
9. Click Test Request/Response. If things go well you will see JSON content. Then just copy JobId from the response and save for later use:
   
10. Now go to Response Settings tab.
11. Select ContentType=Json.
12. Enter below expression to extract JobId from the response. You will need this Id to read SQL query output (this JobId is valid for 24 hrs):
    

    $.jobReference.jobId

13. Choose save response content to save to a variable. Select <New Variable…> option. When prompted give a name to your variable (i.e. vJobId) and for the value field paste JobId copied from the above step:
    
    
    NOTE: This JobId expires after 24-hrs so while you are designing a package and not running it then design time value is used for testing/previewing. So make sure your JobId is valid. If needed click Test Request/Response to grab a new JobId and update the variable with a new value so you can preview data without running a full package.
14. Click OK to save the UI.

Read BigQuery SQL result (for specified JobID)

Now let’s look at how to read data from BigQuery

Configure Google BigQuery Web Request (URL, Method, ContentType, Body etc.)

1. In the Control Flow designer drag and drop Data Flow Task from SSIS toolbox.
   
2. Double click Data Flow Task and drag and drop ZS JSON Source (For API/File) from SSIS toolbox.
   
3. Double click JSON Source to edit and configure as below
4. Enter URL as below. Replace YOUR-API-PROJECT-ID with the API Project ID obtained in the previous section.
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?maxResults=10000

   –OR– (Use below if your Job ran outside US / EU data center)
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?location=YOUR_REGION_ID&maxResults=10000

   NOTE: location Parameter indicates the geographic location of the job. This is Required parameter except for US and EU. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location. You can supply same location parameter for first step when you submit the query

   Failure to supply location parameter for non US /EU users may result in 404 NotFound Error.
   

5. Check Use Credentials option and select OAuth Connection Manager created in the previous section.
6. For HTTP Request Method select GET.
   

Read data from Google BigQuery from Temp Job result

Configure Filter

We need to transform a single JSON response into multiple rows so we need to apply a correct filter.

1. On JSON Source go to Filter Options tab.
2. In the Filter field enter $.rows[*] or click [Select Filter] button to browse hierarchy you want to extract.

Select Filter for JSON Response

Configure BigQuery API Pagination Settings

Most of modern APIs usually implement some sort of pagination technique so you get a part of data in each request rather than all in one go. Thus if you want more you can paginate through pages until the last page is reached. You can also read Understanding REST API Pagination in SSIS article to learn more about pagination in SSIS.

BigQuery API returns pageToken attribute in response JSON if more data is found for requested query result. You can then pass pageToken in the next URL in this format: http://my-api-url/?pageToken=xxxxxxxxxx

Now lets configure JSON Source to automate this pagination for us. On JSON Source go to Pagination Tab and enter the following two settings:

1. Set Next Link as $.pageToken.
2. Set Suffix for Next URL as &pageToken=<%nextlink%>.

See the screenshot below to get more clarity:

Configure BigQuery API Pagination on SSIS JSON Source

Configure Array Transformation

Now the last thing we have to configure is special 2-dimensional JSON array format used by BigQuery API:

1. On the JSON Source UI go to 2D Array Transformation tab.
2. Enter the following settings:
   1. For Transformation Type select Transform complex 2-dimensional array.
   2. For Column Name filter enter $.schema.fields[*].name.
   3. For Row Values Filter enter $.f[*].v.

JSON Array Transformation Options

Preview Data and Save UI

That’s it click Preview to see some data. If you entered sample JobID for your User::vJobID variable then you will see some data. In the next section we will see how to load Google BigQuery data into SQL Server. You can click Columns Tab to review Metadata.

Click OK to save UI and generate Metadata.

Configure Target – Load Google BigQuery data into SQL Server

Now you can connect your JSON Source to any target such as ZS Trash Destination or a real database destination such as OLEDB Destination.

If you wish to dump data from Google BigQuery to a SQL Server table then just perform the following steps:

1. Drag and drop OLEDB Destination from SSIS Toolbox. Rename it to something like SQL Server Table.
2. Double click on OLEDB Destination.
3. Click New to create a new connection > Configure connection > Click OK.
4. Click on New to create a new table > Rename default table name > Click OK.
5. Click on Mappings tab and click OK to save UI with default mappings.

Configure SSIS OLEDB Destination – Google BigQuery to SQL Server Import

Execute Package – Loading BigQuery data into SQL Server

SSIS Package Execution –
Loading Google BigQuery Data into SQL Server

Create / Delete Google BigQuery Dataset using API call

As we mentioned before, you need dataset before you can create a table. Most common way to create dataset is via User Interface but what if you like to automate from your SSIS Package or other workflow? Here is how you can create or delete. Basically you can use REST API Task to send CREATE or DROP command for Dataset object

Create Google Dataset

To create dataset configure REST API Task using below settings (We will call this API)

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/datasets

4. Request Method as POST
5. Enter Request Body as below (change YOUR_DATASET_NAME – e.g. TestDataset )
   

   {"datasetReference": { "datasetId": "YOUR_DATASET_NAME", "projectId": "{{User::ProjectId}}"} }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Dataset already exists then you can go to Error Handling Tab (See next section for screenshot) and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Delete Google BigQuery Dataset

For deleing dataset you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Create / Delete Google BigQuery Table using API call

Now let’s look at how to create /drop Table in Google BigQuery using API calls.

Create Google BigQuery Table

To send CREATE TABLE  SQL statement (DDL) we have to use same approach as we send normal SQL Query using this API call. So notice we used Standard SQL for this call by supplying  useLegacySql: false . DDL Statement must be all in one line

POST https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

Content-Type: application/json

>>>> BODY <<<<<

{
 "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
 "useLegacySql": false,
 "timeoutMs": 100000 
}

To create dataset configure REST API Task using below settings

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

4. Request Method as POST
5. Enter Request Body as below
   

   {
    "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
    "useLegacySql": false,
    "timeoutMs": 100000
   }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Table exists then you can go to enable Error Handling Tab and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Create Google BigQuery Table – API Call (Continue On Error Setting – Skip CREATE if Table / Dataset Already Exists)

Delete Google BigQuery Table

For deleing table you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Write data to Google BigQuery using SSIS – 1 Million row insert test (FAST)

Now let’s look at how easy it is to import data into Google BigQuery using SSIS. We will use the same OAuth connection we created before. To learn more about inserting data into BigQuery check tabledata/insertAll method documentation.

Make sure billing is enabled

Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

Streaming is not available via the free tier. If you attempt to use streaming without enabling billing, you receive the following error: BigQuery: Streaming insert is not allowed in the free tier.

As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call (Write  demo).

Configure OAuth Connection / Permissions

If you want to perform data insert operation in BigQuery using API calls then include the following scopes in your OAuth Connection Manager and generate a token (see our first section of this article – We already included scopes for Write operation but incase you didnt do then regenerate token with below scopes):

https://www.googleapis.com/auth/bigquery
https://www.googleapis.com/auth/cloud-platform
https://www.googleapis.com/auth/bigquery.insertdata

Create BigQuery Dataset (From UI)

For this demo first create a test dataset and one table under it like shown below (billing must be enabled on your Google API Project). To do that go to https://bigquery.cloud.google.com/welcome and configure them:

Insert data into BigQuery using API call

And here is a REST API example to insert data into a BigQuery Table:

Request URL:

https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

Request Headers:

Authorization: Bearer ya29.Gl0cBxxxxxxxxxxxxxxxxxxuEIhJIEnxE6GsQPHI
Content-Type: application/json

Request Body:

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
     {"json": {"RowId": 2,"CustomerName": "BBB"} }
   ]
}

Here is our data flow setup to achive very high throughput for Google BigQuery Data Load. We will show you how to insert one million rows in Google BigQuery in less than a minute based on below setup. We will use Multi Threading option and New Compression Option (Added in v3.1.4)

1. Dummy Data Source – Generate sample records
2. JSON Generator Transform – Generates JSON documents to send as POST request for above /insertAll API call.
3. Web API destination  – Call /insertAll API call to submit our data to BigQuery
4. (Optional) JSON Parser Transform – Parse Error Message for any response
5. (Optional) Trash Destination – Save any errors to text file for review

Google BigQuery Data Load Demo in SSIS – 1 Million Rows Insert with Multi Threads and Compression ON (Fast Upload)

Configure SSIS JSON Generator – Generate JSON for BigQuery Table Insert

Now let’s see how to build an HTTP request with JSON body and send it to BigQuery:

1. Drag Data Flow Task and double click on it.
   
2. Drag and configure your Source (for this demo we use Dummy Data Source with Customer Template). See previous section for configuration of Dummy Data Source.
   
3. Drag and drop ZS JSON Generator Transform to produce JSON from a database or file records. If your source is already sending a valid JSON then you can skip this step (e.g. SQL query is
   returning JSON). You can also read raw JSON from a very large file (new-line separated JSON) using JSON Source with Output as Raw Data option checked.
4. Connect Source to JSON Generator. Double click JSON Generator Transform to start configuring it like below.
5. Select Output Mode as Single Dataset Array and enter Batch Size 10000 (This is Max limit allowed by Google BigQuery API insertAll)
6. First right click on Mappings node and select Add Static Element
   

   Generate JSON for Google BigQuery InsertAll API request – Batch 10000 rows in a single API call

7. Enter Name as kind and value as bigquery#tableDataInsertAllRequest
   
   

   JSON Generator – Add Static Element

8. Now right click on Mappings node and select Add Document Array option
   

   JSON Generator – Add Document Array

9. Enter rows as array title and click OK
   

   JSON Generator – Name array

10. Select newly added array node and right click > Add unbound nested element  enter Output alias as json and click OK.
    

    JSON Generator – Add unbound nested element

11. Select json node and right click > Select Add Elements below this node and select multiple columns you like to send to BigQuery. Click OK to save.
    

    JSON Generator – Add Multiple Elements

12. Now let’s preview our JSON (Copy preview JSON to try in the next step – Web API destination)
    NOTE: Table name and column names are case-sensitive so make sure your JSON attribute matches exact same way.
    Here is the finished JSON Structure for next Step
    

    Sample JSON Request body for Google BigQuery insertAll API request

13. Click OK to save UI.

Configure SSIS Web API destination – Insert data into BigQuery Table

Once you have Input JSON prepared,  now let’s configure destination for BigQuery.

1. Drag and drop ZS Web API Destination.
2. Connect your JSON Generator Transform to Web API destination.
3. Configure general properties:
   

   SSIS Web API Destination –
   Configure for BigQuery Data load

4. Make sure to enter URL in this format:
   

   https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

   Make sure to replace 3 parts in above URL (MY_PROJECT_ID, MY_DATASET_ID, MY_TABLE_ID) with actual values from your Google Project and BigQuery dataset/table configuration.
5. Now you can enable Compression and Multiple Threads for higher throughput as below.
   NOTE: Compression Property was added in v3.1.4 so you may not see it if you have older version.
   

   Google BigQuery Data Loading Performance Optimization Options – Enable Multiple Threads and Compression Options

6. If you want to try test insert request from UI without running full package then go back to first tab and edit Body (Use Sample JSON generated by previous JSON Transform – You can grab from JSON Preview Panel on Generator Transform).Click Test Request / Response and confirm Success as below. You can go back to your BigQuery Portal and check one row is inserted after our test click. If everything looking good then run full package to insert all records.Sample JSON for Body:
   

   {
     "kind": "bigquery#tableDataInsertAllRequest",
     "rows": [
        {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
        {"json": {"RowId": 2,"CustomerName": "BBB"} }
      ]
   }

   

   Test Google BigQuery Table Insert – SSIS Web API Destination UI

7. Hit OK to save UI.
8. Run the package and verify data in Google BigQuery Console:
   

   Loading data into Google BigQuery using SSIS

Error Handling for BigQuery Data Load (Bulk Insert API Calls)

There will be a time when some records you insert may not go well in Google BigQuery. In such case you can read output from Web API destination and parse further with JSON Parser Transform. Check for certain values in the output. You must use JSON Parser after Web API destination (Connect Blue Arrow from Web API destination – Since its Soft Error it won’t redirect in Red Arrow ).

For example here is the sample JSON in POST Body for testing which produces error due to bad column name. When bad row found in batch all records will be rejected. Notice that error returns index of record in batch so you can identify which row went bad. It also returns column name in location attribute.

NOTE: Bad column name in 2nd record

Test Body (Bad):

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RecordID": 1,"CustomerID": "X1"} }, 
     {"json": {"Bad_Column": 2,"CustomerID": "X2"} }
     {"json": {"RecordID": 3,"CustomerID": "X3"} }, 
   ]
}

Response (For Bad Input):

{
  "kind": "bigquery#tableDataInsertAllResponse",
  "insertErrors": [
    {
      "index": 1,
      "errors": [
        {
          "reason": "invalid",
          "location": "bad_column",
          "debugInfo": "",
          "message": "no such field."
        }
      ]
    },
    {
      "index": 0,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    },
    {
      "index": 2,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    }
  ]
}

To configure error detection perform following steps.

1. Drag and drop ZS JSON Parser Transform after Web API destination
2. Click on Web API destination. Connect Blue arrow  to JSON Parser Transform
3. Configure JSON Parser Transform like below
4. Connect JSON Parser TRansform to some Destination to save error information (e.g. SQL Table or Trans destination)

Handling BigQuery Insert Errors in SSIS

Other Common Errors in BigQuery API calls

In this section we will talk about many common API errors in BigQuery.

Error: The project XXXXXXX has not enabled BigQuery

Sometimes you might get below error. To fix this error make sure you go to your Project and Enable BigQuery API

Status Code: BadRequest

Response Body: {
  "error": {
    "code": 400,
    "message": "The project bigquerytest-281915 has not enabled BigQuery.",
    "errors": [
      {
        "message": "The project bigquerytest-281915 has not enabled BigQuery.",
        "domain": "global",
        "reason": "invalid"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}

Error: 404 – Not Found: Table / Job xxxxxx

• Make sure you are setting GET request and not POST.
• Also make sure your jobId is valid because it expires after 24 hours.
• Make sure project-id supplied in URL is valid ID  (DO NOT Specify Alias, use internal ID for project)
• Make sure you supplied location parameter if your outside EU / US region. Reading data using this API call might fail if you failed to supply location in URL

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "notFound",
    "message": "Not Found: Table xxxxxx"
   }
  ],
  "code": 404,
  "message": "Not Found: Table xxxxxxx
 }
}

Error: 403 – Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier

If you trying call certain APIs on sandbox mode or free tier then you might get below error. To overcome this error enable billing on google bigquery. As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call.

Response Url: https://www.googleapis.com/bigquery/v2/projects/bigquerytest-281915/datasets/TestDataset/tables/Table1/insertAll

Status Code: Forbidden

Response Body: {
  "error": {
    "code": 403,
    "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
    "errors": [
      {
        "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
        "domain": "global",
        "reason": "accessDenied"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}

Debugging Web API Requests

If you need to debug actual requests made to Google server then you can use a tool like Fiddler. It’s a very handy tool to troubleshoot JSON format issues. It will allow to see how a request is made to a server.

Using Fiddler to debug Google BigQuery API requests in SSIS

Common Errors

Deployment to Production

Download Sample Package

Click Here to Download SSIS Sample Package – Google BigQuery API Read Write Create Delete (SSIS 2019, 2017, 2012)

Conclusion. What’s next?

In this article we have learned how to load data from Google BigQuery into SQL Server using SSIS (drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from Google BigQuery REST API using OAuth. JSON Source Connector makes it super simple to parse complex/large JSON files or any Web API response into rows and columns so you can load data into a database, e.g. SQL Server database. Download SSIS PowerPack to try many other automation scenarios that were not discussed in this article.

Keywords:

Google BigQuery Integration with SQL Server | How to extract data from google bigquery in SSIS? | How to read data from Google BigQuery API? | Loading BigQuery Data into SQL Server. | BigQuery to SQL Server | SSIS Google Big Query Integration | SSIS Google BigQuery Import  JSON File | SSIS Google BigQuery Export data

The post How to read / write data in Google BigQuery using SSIS appeared first on ZappySys Blog.

In our previous post, we saw how to get data from Google Analytics using SSIS. Once you get going with the drag and drop SSIS Google Analytics Connector, you may bump up to one interesting challenge. If your request includes sessions above a certain threshold, Google may start sampling your data to provide an average for many Metrics. So, how do you make sure your request includes all sessions in the calculation? The following sections will describe a few ways to achieve this.

Google Analytics Data Sampling Tips

There is no way to guarantee 100% Unsampled data in the Standard Google Analytics API call due to restrictions imposed by Google. However, there are a few workarounds that may help you achieve a very high Data Sampling rate.

The most common solution is that if you set the Sampling option to HIGHER_PRECISION, then Google may honor your request up to a certain threshold, but as the dataset grows (e.g., if you select a larger date range), Google may sample your data even though you set HIGHER_PRECISION.

Extract data from Google Analytics in SSIS: Use the Sample Size option to get Unsampled data.

Other ways to get unsampled data from the Google Analytics API

Now, let’s look at a few other ways if the previous approach doesn’t work.

1. Narrow down your Date range… e.g., Process day by day rather than a full extract. If day by day is also sampling, they try Hour-by-Hour.
2. Use the Google API to generate and download Unsampled data (We don’t have an article on this, but see the reference below).
3. Use a Google Analytics 360 Premium Account with the unsampled report feature. You can get this via the Web Interface or call the API to automate the extraction. https://support.google.com/analytics/answer/2601061?hl=en

https://developers.google.com/analytics/devguides/config/mgmt/v3/unsampled-reports
https://support.google.com/analytics/answer/2637192?hl=en&visit_id=1-636288218914624195-2113255108&rd=1

Downloading Unsampled report using Google Analytics 360 API – ZappySys REST API Task

If you are a Google Analytics 360 user and wish to use the API to generate a file and download it in SSIS, refer to the following article. It shows how to call the Google API. It’s not about the Analytics API, but you will get a high-level idea.

https://zappysys.com/blog/get-data-google-spreadsheet-using-ssis/

The post 3 ways to get Google Analytics unsampled data in SSIS appeared first on ZappySys Blog.

Google DoubleClick is one of the most popular platforms for Advertisers. Many times you have need for custom integration or Automation for many operations. In this article we will learn how to Call Google DFP API (i.e. DoubleClick for Publishers) without coding using SSIS (Microsoft SQL Server Integration Services). In our previous article we discussed how to Integrate Google AdWords API using SSIS (Click here).

If you are new to SSIS then no worry there are plenty of tutorials for ssis

To achieve Custom integration for DoubleClick (DFP) API we will use SSIS XML Source. XML Source is part of SSIS PowerPack which has 45+ connectors and Tasks.

Prerequisite

Before you can finish tasks explained in this article you have to finish below steps. This article assumes you have basic knowledge of SSIS (SQL Server Integration Services)

1. Make sure you SSIS designer installed. Sometimes its referred as BIDS or SSDT (Get from here)
2. Download SSIS PowerPack
3. Download SoapUI (Its free third party tool to test SOAP API). SoapUI can help you to generate XML Request BODY easily from WSDL file provided by API Vendor. You can also test your service by supplying parameters.
4. Optional – Another very useful Free tool is Fiddler. You can use it to see raw request/response (Check this article)

Download DFP API Example SSIS Package

Click here to DFP API download Sample for SSIS SSIS Package (SSIS 2012, 2014, 2016)

Screenshot of Sample Package:

Example SSIS Package – Google DFP API Integration (Read / Write data in Google Double Click for Publisher) – No Coding required

Making your first DFP API Call using SSIS

Once you install PowerPack you are ready to execute your very first Google DFP API Call.

NOTE: If you don’t have any DFP network for test then see next section (Explains how to create test network for testing).

Lets look at how to call Google DFP API step-by-step.

1. Create new SSIS Project and open package designer
2. Drag ZS REST API Task from SSIS control flow toolbox and drop it on designer.
   
   REST API Task is useful when you want to call SOAP/REST API but not necessarily parse response into rows and columns. We will look at JSON Source later in this article which can actually parse response into rows and columns (Useful to load Google DFP data into SQL Server or other RDBMS / flatfile )
3. Double click REST API Task to configure. Select Access mode to [Url from Connection]
4. Enter following URL in the URL Textbox
   

    https://ads.google.com/apis/ads/publisher/v201702/NetworkService

5. From the connection dialogbox select ZS-OAUTH to create new OAuth Connection for DFP API
6. On OAuth Connection dialogbox select Google from Provider dropdown. In this demo we will use Default OAuth App but you can register your own google OAuth app if you wish to use Custom OAuth option from UI.
7. Enter following Scopes in the Scopes textbox (or select manually by clicking Select Scopes button). Scope is nothing but permission for API (In our case View/Manage DFP data and Read/Write Report Files to Cloud Storage).
   

   https://www.googleapis.com/auth/dfp
   https://www.googleapis.com/auth/devstorage.read_only
   https://www.googleapis.com/auth/devstorage.write_only
   https://www.googleapis.com/auth/cloud-platform.read-only
   https://www.googleapis.com/auth/devstorage.full_control
   https://www.googleapis.com/auth/cloud-platform
   https://www.googleapis.com/auth/devstorage.read_write

8. If you are not going call ReportService API to generate Reports in CSV format then you will need only one scope
   

   https://www.googleapis.com/auth/dfp

9. Click Generate Token button. You will see browser popup for login and then Accept option to confirm permissions like below. Once you don’t it will populate tokens and prompt to save tokens to backup file. Secure token backup to safe place.
   

   SSIS OAuth Connection – Connect to Google DFP API (DoubleClick SOAP Web Service)

10. Once done Click Test Connection and if its green then Click OK to save connection
11. Once you are back to REST API Task UI then select Method to POST
12. Select Request Body Content Type to XML (i.e. text/xml)
13. Click edit button next to Request Body and enter following XML.
    

    <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
       <soapenv:Header>
          <v20:RequestHeader/>
       </soapenv:Header>
       <soapenv:Body>
          <v20:getAllNetworks/>
       </soapenv:Body>
    </soapenv:Envelope>

    If you are wondering how did we get above XML fragment then read next section about using 3rd party tool called SoapUI. Also we have detail article for calling SOAP request. Check this article on how to use SoapUI (free 3rd party tool) to create SOAP request Bodyfrom WSDL. DFP has many API endpoints for different actions (e.g. NetworkService , InventoryService , …). For each service you have different Api URL and different WSDL. In the next section you can learn how to create correct SOAP Body (for POST) using SoapUI tool.

    For example if you using DFP NetworkService API for version v201702 then your help page URL would be like this
    https://developers.google.com/doubleclick-publishers/docs/reference/v201702/NetworkService
    And on the same page you will see WSDL link below
    https://adwords.google.com/api/adwords/mcm/v201609/ManagedCustomerService?wsdl
    (Just download that WSDL XML and save to local disk then use with SoapUI to generate XML body to submit for any DFP API call.)

14. Once you done with above steps, your Task Configuration will look like below.
    

    SSIS Rest API Task – Call Google DoubleClick API – Get all DFP Networks

15. Now click Test Request button. You will see Response popup with below content if its successful (Scroll at the bottom on response form and you will see XML data) .. see below
    

    ------------------------------------
    Request
    ------------------------------------
    POST https 1.1 ==> /apis/ads/publisher/v201702/NetworkService
    Host: ads.google.com
    
    >>>> HEADERS <<<<<
    Authorization: Bearer ya29.Glz-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    Content-Type: text/xml;charset=UTF-8
    User-Agent: ZappySysApp
    Host: ads.google.com
    Content-Length: 291
    Expect: 100-continue
    Accept-Encoding: gzip, deflate
    Connection: Keep-Alive
    
    ------------------------------------
    Response
    ------------------------------------
    POST 1.1 200 OK==> OK
    
    >>>> HEADERS <<<<<
    Content-Encoding: 
    X-Content-Type-Options: nosniff
    X-Frame-Options: SAMEORIGIN
    X-XSS-Protection: 1; mode=block
    Alt-Svc: quic=":443"; ma=2592000; v="35,34"
    Transfer-Encoding: chunked
    Cache-Control: private, max-age=0
    Content-Type: text/xml; charset=UTF-8
    Date: Sun, 26 Feb 2017 01:18:32 GMT
    Expires: Sun, 26 Feb 2017 01:18:32 GMT
    Server: GSE
    
    >>>> Cookies <<<<<
    
    >>>> CONTENT <<<<<
    
    <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    	<soap:Header>
    		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
    			<requestId>27ab3d2e4e1fcb8f444b317d0b0f871d</requestId>
    			<responseTime>50</responseTime>
    		</ResponseHeader>
    	</soap:Header>
    	<soap:Body>
    		<getAllNetworksResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
    			<rval>
    				<id>551196</id>
    				<displayName>XFP sandbox property</displayName>
    				<networkCode>238897396</networkCode>
    				<propertyCode>ca-pub-2799179143725683</propertyCode>
    				<timeZone>America/New_York</timeZone>
    				<currencyCode>USD</currencyCode>
    				<effectiveRootAdUnitId>237897516</effectiveRootAdUnitId>
    				<isTest>true</isTest>
    			</rval>
    		</getAllNetworksResponse>
    	</soap:Body>
    </soap:Envelope>

     

How to generate SOAP Request Body using SoapUI tool

This section describes how to create DoubleClick SOAP API Request Body (XML fragment) for any DFP SOAP API call described anywhere in this article.

1. Download SoapUI (Its free third party tool to test SOAP API). SoapUI can help you to generate XML Request BODY easily from WSDL file provided by API Vendor. You can also test your service by supplying parameters.
2. Now download WDSL xml for appropriate API calls you want to make. Assume that you want to call  getAllNetworksResponse API found under NetworkService. For that first navigate to help file page and open help page. Make sure you select correct version from help navigation. Copy WSDL link and open in new browser window. Save XML to local disk (e.g. c:\api\dfp_networkservice_wsdl.xml (See below screenshot how to find DFP API WSDL link)
   

   How to get Google DFP API WSDL URL (Used to generate POST Body)

3. Once SoapUI is downloaded and installed click File menu > Create New SOAP Project option
4. Name your project (e.g. DFP API) and specify WSDL URL or File Path if it was saved locally (e.g. c:\api\dfp_networkservice_wsdl.xml) and click OK
5. Now navigate to API Action for which you would like to get Body. Click Request node (If missing create new) and edit request. You may see XML like below (If some optional parameters not visible in XML then click Re-create request with optional parameters button from toolbar)
   

   Call Google DFP API – Generate SOAP Request Body XML using SoapUI (Google DoubleClick Web Service)

6. Edit necessary parameters from above XML code and copy to Body of SSIS REST API Task.

Creating test network – DFP Sandbox

Very first step we recommend before testing DFP API call is create test network. You can call DFP API against LIVE network but not recommended if you are calling create/update API.

To create test network you can follow exact same steps described in the previous section except the Body step (Step#12). Use following Request body to call makeTestNetwork command.

API makeTestNetwork – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/NetworkService
Body (see below):

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
   <soapenv:Header>
      <v20:RequestHeader/>
   </soapenv:Header>
   <soapenv:Body>
      <v20:makeTestNetwork/>
   </soapenv:Body>
</soapenv:Envelope>

Once you click Test you may receive following response. Note your network code (e.g. 1122334455 from below). This network code is used in pretty much all DFP API to view or manage ad network related items or properties.

API makeTestNetwork – Response

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<requestId>ba61d79efee325bf4b5aa45c46836021</requestId>
			<responseTime>1021</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<makeTestNetworkResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<rval>
				<id>551196</id>
				<displayName>XFP sandbox property</displayName>
				<networkCode>1122334455</networkCode>
				<propertyCode>ca-pub-2799179143725683</propertyCode>
				<timeZone>America/Los_Angeles</timeZone>
				<currencyCode>USD</currencyCode>
				<effectiveRootAdUnitId>237897516</effectiveRootAdUnitId>
				<isTest>true</isTest>
			</rval>
		</makeTestNetworkResponse>
	</soap:Body>
</soap:Envelope>

You can also get network code using two different ways.

• Visit your DFP console homepage by visiting https://www.google.com/dfp/ and you will see network name and code in the top potion. Also its listed in the URL (e.g. https://www.google.com/dfp/1234567)
• Another way to get available networks for your login is to visit DFP API Play ground here https://dfp-playground.appspot.com/ and you will see network list in the dropdown ( name and network code)

How to Create new Ad Units

Now lets look at an example which will create few Ad Units by calling createAdUnits API. If you are creating Ad Unit at the root level then you will need to specify correct Parent ID (i.e  effectiveRootAdUnitId). To obtain that Parent ID you may have to call  getCurrentNetwork API. In the response you will see effectiveRootAdUnitId.

There is another way to know effectiveRootAdUnitId or Id of any Ad Units which can be Parent for new AdUnit. Goto google.com/dfp > Click Inventory Tab > Click Ad Units Side menu > Click Download ad units hyper link. You can also download AdUnits as CSV file from DFP Portal and look for Ad Units which are created at the root. Parent ID column for Root level Ad Units is basically called effectiveRootAdUnitId.

Now lets look at how to get EffectiveRootID by calling getCurrentNetwork.

API getCurrentNetwork – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/NetworkService
Body (see below):

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Header>
	<RequestHeader xmlns="https://www.google.com/apis/ads/publisher/v201608">
		<networkCode xmlns="https://www.google.com/apis/ads/publisher/v201608">1122334455</networkCode>
		<applicationName xmlns="https://www.google.com/apis/ads/publisher/v201608">ZappySysApp</applicationName>
	</RequestHeader>
	</soap:Header>
	<soap:Body>
		<getCurrentNetwork xmlns="https://www.google.com/apis/ads/publisher/v201608" />
	</soap:Body>
</soap:Envelope>

API getCurrentNetwork – Response

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<requestId>81d1b00e8fbfb862129eb2dd485cf7bb</requestId>
			<responseTime>4507</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<getCurrentNetworkResponse xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<rval>
				<id>551196</id>
				<displayName>XFP sandbox property</displayName>
				<networkCode>1122334455</networkCode>
				<propertyCode>ca-pub-2799179143725683</propertyCode>
				<timeZone>America/New_York</timeZone>
				<currencyCode>USD</currencyCode>
				<effectiveRootAdUnitId>237897516</effectiveRootAdUnitId>
				<isTest>true</isTest>
			</rval>
		</getCurrentNetworkResponse>
	</soap:Body>
</soap:Envelope>

If you want to save single Element Value from Response into SSIS Variable then perform following steps (example of how to get just effectiveRootAdUnitId from above response)

1. On REST API Task > Go to Response Tab > Select Response Content Type = XML
2. Enter following expression in the XPath filter
   

   //*[local-name() = 'effectiveRootAdUnitId']

3. Check Save response content option
4. Select Variable from Dropdown – Click New Variable > Name it RootAdUnitId.
5. Click Test Request Button to test…. In the content textbox you will now see only Numeric value extracted by Filter

SSIS REST Api Task – How to extract single vale from response and save to SSIS variable

Once you know Parent ID for your AdUnit now lets look at how to create Ad Units by calling createAdUnits

API createAdUnits – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/InventoryService
Body (see below):

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<networkCode xmlns="https://www.google.com/apis/ads/publisher/v201608">1122334455</networkCode>
			<applicationName xmlns="https://www.google.com/apis/ads/publisher/v201608">ZappySysApp</applicationName>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<createAdUnits xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<adUnits>
				<parentId>237897516</parentId>
				<name>Ad_Unit_0</name>
				<description>Ad unit description #0.</description>
				<targetWindow>BLANK</targetWindow>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
					</size>
					<environmentType>BROWSER</environmentType>
				</adUnitSizes>
			</adUnits>
			<adUnits>
				<parentId>237897516</parentId>
				<name>Ad_Unit_1</name>
				<description>Ad unit description #1.</description>
				<targetWindow>BLANK</targetWindow>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
					</size>
					<environmentType>BROWSER</environmentType>
				</adUnitSizes>
			</adUnits>			
		</createAdUnits>
	</soap:Body>
</soap:Envelope>

API createAdUnits – Response

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<requestId>b244d7ba9b65e96fc42b849a7bfb16e4</requestId>
			<responseTime>331</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<createAdUnitsResponse xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<rval>
				<id>252312396</id>
				<parentId>237897516</parentId>
				<hasChildren>false</hasChildren>
				<parentPath>
					<id>237897516</id>
					<name>ca-pub-2799179143725683</name>
					<adUnitCode>ca-pub-2799179143725683</adUnitCode>
				</parentPath>
				<name>Ad_Unit_0</name>
				<description>Ad unit description#0.</description>
				<targetWindow>BLANK</targetWindow>
				<status>ACTIVE</status>
				<adUnitCode>253312396</adUnitCode>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
						<isAspectRatio>false</isAspectRatio>
					</size>
					<environmentType>BROWSER</environmentType>
					<fullDisplayString>300x250</fullDisplayString>
				</adUnitSizes>
				<mobilePlatform>SITE</mobilePlatform>
				<explicitlyTargeted>false</explicitlyTargeted>
				<inheritedAdSenseSettings>
					<value>
						<adSenseEnabled>true</adSenseEnabled>
						<borderColor>FFFFFF</borderColor>
						<titleColor>0000FF</titleColor>
						<backgroundColor>FFFFFF</backgroundColor>
						<textColor>000000</textColor>
						<urlColor>008000</urlColor>
						<adType>TEXT_AND_IMAGE</adType>
						<borderStyle>DEFAULT</borderStyle>
						<fontFamily>DEFAULT</fontFamily>
						<fontSize>DEFAULT</fontSize>
					</value>
				</inheritedAdSenseSettings>
				<lastModifiedDateTime>
					<date>
						<year>2017</year>
						<month>2</month>
						<day>27</day>
					</date>
					<hour>7</hour>
					<minute>23</minute>
					<second>32</second>
					<timeZoneID>PST8PDT</timeZoneID>
				</lastModifiedDateTime>
				<smartSizeMode>NONE</smartSizeMode>
				<isSharedByDistributor>false</isSharedByDistributor>
				<isSetTopBoxEnabled>false</isSetTopBoxEnabled>
			</rval>
			<rval>
				<id>252312516</id>
				<parentId>237897516</parentId>
				<hasChildren>false</hasChildren>
				<parentPath>
					<id>237897516</id>
					<name>ca-pub-2799179143725683</name>
					<adUnitCode>ca-pub-2799179143725683</adUnitCode>
				</parentPath>
				<name>Ad_Unit_1</name>
				<description>Ad unit description#2.</description>
				<targetWindow>BLANK</targetWindow>
				<status>ACTIVE</status>
				<adUnitCode>253312516</adUnitCode>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
						<isAspectRatio>false</isAspectRatio>
					</size>
					<environmentType>BROWSER</environmentType>
					<fullDisplayString>300x250</fullDisplayString>
				</adUnitSizes>
				<mobilePlatform>SITE</mobilePlatform>
				<explicitlyTargeted>false</explicitlyTargeted>
				<inheritedAdSenseSettings>
					<value>
						<adSenseEnabled>true</adSenseEnabled>
						<borderColor>FFFFFF</borderColor>
						<titleColor>0000FF</titleColor>
						<backgroundColor>FFFFFF</backgroundColor>
						<textColor>000000</textColor>
						<urlColor>008000</urlColor>
						<adType>TEXT_AND_IMAGE</adType>
						<borderStyle>DEFAULT</borderStyle>
						<fontFamily>DEFAULT</fontFamily>
						<fontSize>DEFAULT</fontSize>
					</value>
				</inheritedAdSenseSettings>
				<lastModifiedDateTime>
					<date>
						<year>2017</year>
						<month>2</month>
						<day>27</day>
					</date>
					<hour>7</hour>
					<minute>23</minute>
					<second>32</second>
					<timeZoneID>PST8PDT</timeZoneID>
				</lastModifiedDateTime>
				<smartSizeMode>NONE</smartSizeMode>
				<isSharedByDistributor>false</isSharedByDistributor>
				<isSetTopBoxEnabled>false</isSetTopBoxEnabled>
			</rval>
				<id>252312876</id>
				<parentId>237897516</parentId>
				<hasChildren>false</hasChildren>
				<parentPath>
					<id>237897516</id>
					<name>ca-pub-2799179143725683</name>
					<adUnitCode>ca-pub-2799179143725683</adUnitCode>
				</parentPath>
				<name>Ad_Unit_4</name>
				<description>Ad unit description.</description>
				<targetWindow>BLANK</targetWindow>
				<status>ACTIVE</status>
				<adUnitCode>253312876</adUnitCode>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
						<isAspectRatio>false</isAspectRatio>
					</size>
					<environmentType>BROWSER</environmentType>
					<fullDisplayString>300x250</fullDisplayString>
				</adUnitSizes>
				<mobilePlatform>SITE</mobilePlatform>
				<explicitlyTargeted>false</explicitlyTargeted>
				<inheritedAdSenseSettings>
					<value>
						<adSenseEnabled>true</adSenseEnabled>
						<borderColor>FFFFFF</borderColor>
						<titleColor>0000FF</titleColor>
						<backgroundColor>FFFFFF</backgroundColor>
						<textColor>000000</textColor>
						<urlColor>008000</urlColor>
						<adType>TEXT_AND_IMAGE</adType>
						<borderStyle>DEFAULT</borderStyle>
						<fontFamily>DEFAULT</fontFamily>
						<fontSize>DEFAULT</fontSize>
					</value>
				</inheritedAdSenseSettings>
				<lastModifiedDateTime>
					<date>
						<year>2017</year>
						<month>2</month>
						<day>27</day>
					</date>
					<hour>7</hour>
					<minute>23</minute>
					<second>32</second>
					<timeZoneID>PST8PDT</timeZoneID>
				</lastModifiedDateTime>
				<smartSizeMode>NONE</smartSizeMode>
				<isSharedByDistributor>false</isSharedByDistributor>
				<isSetTopBoxEnabled>false</isSetTopBoxEnabled>
			</rval>
		</createAdUnitsResponse>
	</soap:Body>
</soap:Envelope>

Loading Google DoubleClick data into SQL Server Table

So far we saw how to make simple DFP API calls without doing any parsing. Now lets look at how to use SSIS XML Source which not only make API calls but it will parse XML Response into rows and columns which can be loaded into target database such as SQL Server, Oracle, MySQL or even Flat File. XML Source also supports pagination so if you have many records it will automatically loop through all response untill all records are fetched.

Step-By-Step – Using XML Source to read Google DFP data (Parse into rows and columns)

1. Assuming you have tested your first DFP API Call (Explianed in the beginning of this article). You must have OAuth connection tested for API call.
2. Drag new Data flow task from SSIS Toolbox
3. Inside Data flow designer drag and drop ZS XML Source from toolbox
4. Double click XML Source to configure.
5. Set URL as below
   https://ads.google.com/apis/ads/publisher/v201702/InventoryService
6. Check Use credentials option and from Drop down select OAuth connection manager (created in previous section)
7. Select Method = POST,
8. Select Request Content Type = XML (text/xml)
9. Enter Body as below (Click edit button).
   

   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
      <soapenv:Header>
         <v20:RequestHeader>
            <v20:networkCode>238897396</v20:networkCode>
            <v20:applicationName>ZappySysApp</v20:applicationName>
         </v20:RequestHeader>
      </soapenv:Header>
      <soapenv:Body>
         <v20:getAdUnitsByStatement>
            <v20:filterStatement>
               <v20:query>WHERE Status='ACTIVE'</v20:query>
            </v20:filterStatement>
         </v20:getAdUnitsByStatement>
      </soapenv:Body>
   </soapenv:Envelope>

   Notice that in above we calling getAdUnitsByStatement API to fetch all Ad Units which are Active. Here is more information about Query Syntax for any API call getXXXXXXByStaement
10. Now click on Select Filter button and select results node and click OK. If prompted add array to list.
    

    SSIS XML Source – Get data from Google DFP API using OAuth (Configure URL, Body and Filter)

11. Click Preview to see data. Click Columns tab to review data types (Auto detected). If you wish to change length or datatype then edit there and check Lock option (Last column)
    

    SSIS XML Source – Preview Response for Google DoubleClick API Call – Parse into Rows/Columns

12. Click OK to save UI
13. Connect XML Source to same target (e.g. OLEDB Destination – SQL Server Connection)
14. Run data flow
    

    SSIS Execution – Read from Google DFP and load into SQL Server Table

Configure Pagination for Google DFP API / PQL Query result

If you have large dataset to read from DoubleClick API ( when calling API such as getxxxxxxByStatement) then we recommend to supply LIMIT and OFFSET clause in your PQL query.

For example if you have 2000 ad units and you want to limit response size by maximum 300 rows then your query can be like below

--For first request (returns results 1-300)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 0

--For second request (returns results 301-600)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 300

--For third request  (returns results 601-900)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 600

........

This looping logic can be complex to implement if you do manually. But no worry if you are using SSIS XML Source. It comes with many pagination options (Click here to read more about pagination)

See below screenshot how to configure Pagination options for Google DoubleClick API. Basically two places you have to change. Inside body you have to set placeholder and change few settings on Pagination Tab.

SSIS XML Source – Configure Pagination for Google DoubleClick API data fetch – PQL LIMIT and OFFSET clause for Google DFP API

SSIS XML Source – Configure Pagination for Google DoubleClick API data fetch – Set Pagination Mode

Create CSV Report File and Download in GZip format (*.gz)

Sometime you have to download large amount of data (Possibly millions of rows) in that case Bulk approach would be better. DFP ReportService API allows to call following APIs which can be used to produce CSV report file in *.gz format and then you can download it using REST API Task.

Below are high level steps you have to perform to produce report file and download it.

• Create the ReportJob by invoking runReportJob command .
• Poll the ReportJob object using ReportService.getReportJob (This is required because file is not available right away) .
• Continue to poll the ReportJob object until the reportJobStatus field is equal to COMPLETED or FAILED.
• If successful, fetch the URL for downloading the report by invoking getReportDownloadURL.

Here is the sample SSIS Package to Perform this Action.

Click here to DFP API download Sample for SSIS SSIS Package (SSIS 2012, 2014, 2016)

See below screenshot how to generate Google DFP Report File and download / extract (Unzip) using Drag and Drop SSIS workflow.

Create Google DFP Report File (CSV / Gzip) and Download Using SSIS (Example of ReportService functions runReportJob, getReportJob and .getReportDownloadURL)

Call runReportJob (Start DFP Report JOB)

Very first step to produce DFP Report file is call ReportService >> runReportJob API. See below Command.

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract id of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for <id>111111</id> node anywhere in SOAP response XML and extract value (i.e. 111111 )  .. For screenshot see next section

//*[local-name() = 'id']

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxxxxAKqG89gM-TpReRXO
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 1018
Expect: 100-continue
Accept-Encoding: gzip, deflate

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
	<soapenv:Header>
		<v20:RequestHeader>
			<v20:networkCode>238897396</v20:networkCode>
			<v20:applicationName>ZappySysApp</v20:applicationName>
		</v20:RequestHeader>
	</soapenv:Header>
	<soapenv:Body>
		<v20:runReportJob>
			<v20:reportJob>
				<v20:reportQuery>
					<v20:dimensions>DATE</v20:dimensions>
					<v20:dimensions>LINE_ITEM_ID</v20:dimensions>
					<v20:dimensions>LINE_ITEM_NAME</v20:dimensions>
					<v20:dimensions>AD_UNIT_NAME</v20:dimensions>
					<v20:columns>AD_SERVER_IMPRESSIONS</v20:columns>
					<v20:columns>AD_SERVER_CLICKS</v20:columns>
					<v20:dimensionAttributes>LINE_ITEM_COST_TYPE</v20:dimensionAttributes>
					<v20:dimensionAttributes>LINE_ITEM_GOAL_QUANTITY</v20:dimensionAttributes>
					<v20:dateRangeType>LAST_MONTH</v20:dateRangeType>
				</v20:reportQuery>
			</v20:reportJob>
		</v20:runReportJob>
	</soapenv:Body>
</soapenv:Envelope>

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:41 GMT
Expires: Fri, 02 Jun 2017 21:23:41 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 999

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<requestId>f03e7e474fde2a5f9c5d7c9d7eb83923</requestId>
			<responseTime>846</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<runReportJobResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<rval>
				<id>10000297829</id>
				<reportQuery>
					<dimensions>DATE</dimensions>
					<dimensions>LINE_ITEM_ID</dimensions>
					<dimensions>LINE_ITEM_NAME</dimensions>
					<dimensions>AD_UNIT_NAME</dimensions>
					<adUnitView>TOP_LEVEL</adUnitView>
					<columns>AD_SERVER_IMPRESSIONS</columns>
					<columns>AD_SERVER_CLICKS</columns>
					<dimensionAttributes>LINE_ITEM_COST_TYPE</dimensionAttributes>
					<dimensionAttributes>LINE_ITEM_GOAL_QUANTITY</dimensionAttributes>
					<dateRangeType>LAST_MONTH</dateRangeType>
					<useSalesLocalTimeZone>false</useSalesLocalTimeZone>
					<includeZeroSalesRows>false</includeZeroSalesRows>
				</reportQuery>
			</rval>
		</runReportJobResponse>
	</soap:Body>
</soap:Envelope>

Call getReportJob (Poll JOB Status)

Second step is to make sure is Report Status check. For large report it make take several seconds or minutes before you can get Download URL (See Next Step). If you try to call Next step before Report is ready then you may get error. So better to add Polling logic.

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract status of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for <rval>IN_PROGRESS</rval> node anywhere in SOAP response XML and extract value (i.e. IN_PROGRESS )

//*[local-name() = 'rval']

Extract Single XML Node Value from SOAP API Response – Use XPATH in SSIS

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxXmWqmq
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 524
Expect: 100-continue
Accept-Encoding: gzip, deflate

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
   <soapenv:Header>
      <v20:RequestHeader>
         <v20:networkCode>238897396</v20:networkCode>
         <v20:applicationName>ZappySysApp</v20:applicationName>
      </v20:RequestHeader>
   </soapenv:Header>
   <soapenv:Body>
      <v20:getReportJobStatus>
       <v20:reportJobId>10000297829</v20:reportJobId>
      </v20:getReportJobStatus>
   </soapenv:Body>
</soapenv:Envelope>

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:47 GMT
Expires: Fri, 02 Jun 2017 21:23:47 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 451

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<requestId>d52cb3a3d68af24aec2f9b0284379700</requestId>
			<responseTime>803</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<getReportJobStatusResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<rval>COMPLETED</rval>
		</getReportJobStatusResponse>
	</soap:Body>
</soap:Envelope>

Call getReportDownloadURL (Get Download URL)

Once you get status = COMPLETED in above API Call (i.e.  getReportJobStatus) then you ready to call getReportDownloadURL  to fetch URL which you can download.

Here is the API sample Request and Response

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxBp84ltTNyVRb9hlUEWx
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 581
Expect: 100-continue
Accept-Encoding: gzip, deflate

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
   <soapenv:Header>
      <v20:RequestHeader>
         <v20:networkCode>238897396</v20:networkCode>
         <v20:applicationName>ZappySysApp</v20:applicationName>
      </v20:RequestHeader>
   </soapenv:Header>
   <soapenv:Body>
      <v20:getReportDownloadURL>
       <v20:reportJobId>10000297829</v20:reportJobId>
       <v20:exportFormat>CSV_DUMP</v20:exportFormat>
      </v20:getReportDownloadURL>
   </soapenv:Body>
</soapenv:Envelope>

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:49 GMT
Expires: Fri, 02 Jun 2017 21:23:49 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 1039

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<requestId>8e5ac0bbeb8e88c180ea442fc5378d99</requestId>
			<responseTime>1957</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<getReportDownloadURLResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<rval>https://storage.googleapis.com/dfp-report-export/caxxxxxxxx-xxxxxxxxxxxxxxb?GoogleAccessId=3769xxxxxxxxxxx-qxxxxxxxxxxxxxxxxsqdb5dvi8g@developer.gserviceaccount.com&amp;Expires=1496438929&amp;Signature=vnhCiAIqd%2BFgWCtAHo9roFOI3W%2BveUTYuSvyBQ9gtMI2HDTXOCVDh7wzUV%2BPBJZZghaL6k8wKYq33GA4NpjZVNYuaAgPLWgGwAUPdtioHRi%2Bt3Eymp%2FAPRGIoos0ekP3O7l%2FPNPxenC6UTP087HV3j06V%2Fuui9ixpKZWGCK8r56oUPj6S8q1hG95Ns3d3sEcuz3%2BkShu6GYG5mqir5vjtZPuzwsr%2F96G5Kk123mke%2B%2FIVRJGoqnCMghNWtIG1yR1IdZx%2FIpit%2FnaJ9HP6IvAr3oMIXkQ3cHv%2FtOhXGy9dTG34q%2FKDbtkSJAayHgjlSLPGKucygcXBhUrbywBu8Y77A%3D%3D</rval>
		</getReportDownloadURLResponse>
	</soap:Body>
</soap:Envelope>

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract status of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for <rval>https://storage.googleapis.com/dfp-report-export/caxx……….</rval> node anywhere in SOAP response XML and extract value (i.e. URL)

Download File Using REST API Task

Once you get URL its time to download it. You can use REST API Task to download the file. On Response Setting Tab specify full file path (e.g. c:\report_csv.gz) and make sure Binary option is checked (Found next to the Save Path)

Making things dynamic

If you want to supply certain parameters at runtime (e.g. from SSIS Variable) rather than hard code then you can use Variable Placeholders any where in Request Body, URL or Headers. Click Insert Variable option found in Edit screen

Here is an example of dynamic URL (API Version stored in SSIS Variable)

https://ads.google.com/apis/ads/publisher/{{User::ApiVer}}/NetworkService

Deployment to Production

When you ready to deploy your SSIS Package to production make sure to change Test NetworkCode supplied inside Request Body to your Production Ad Network. Also Set RefreshToken Property of OAuth Connection Manager (Get it from Backup file saved when you created OAuth connection first time). Once you set RefreshToken of connection manager you won’t have to Login again to get a new token.

Conclusion

Traditionally if you wanted to integrate Google DFP API (DoubleClick API) calls inside your ETL workflow then it required some sort of coding effort (e.g. Use C# SDK, JAVA SDK or Python SDK). Coding effort is not only time consuming and expensive but also hard to maintain for any non-coder (e.g. ETL Developer or DBA).
Thanks to SSIS PowerPack which changed the way developers do API integration. Now you can reduce time to implement and total cost of you API integration project significantly by using drag and drop approach. This approach is not only easy to maintain but also fast and high quality. You can use ZappySys REST/SOAP SSIS Connectors and SSIS Tasks for any API integration project such as Google DFP API. Try SSIS PowerPack for free to explore possibilities. If you have any issue with your integration Contact Support

The post How to call Google DFP API with SSIS – DoubleClick appeared first on ZappySys Blog.

In this tutorial we will use REST API Task to call some ad-hoc API (e.g. get File List from Google Drive) and save output into Variable or File. We will use OAuth connection along with ZappySys JSON Source connector to read data from Google SpreadSheet (Use Drive API) and load into SQL Server (Export Google SpreadSheet to CSV).

Create Google API Project

First step to access any Google API is create an API Project in Google Console. If you don’t want to go through this then Skip this Step-1 and in the next section select Default OAuth App option on OAth Connection Manager (This is the most easiest option for now unless you want to use your own OAuth App).

Check this article for step-by-step instructions. When you follow these instructions make sure you enable Google Drive API (In the article it shows how to enable YouTube API as an example but you will need to enable Drive API for this article).

Create OAuth Connection Manager in SSIS

Once you create Google API project and obtained Client ID and Client Secret your next step is to create OAuth Connection Manager in SSIS. ZappySys OAuth connection manager comes with many predefined OAuth Providers (e.g. Facebook, Twitter, Google etc) but you can also define custom OAuth settings for any OAuth enabled API.

To create SSIS OAuth 2.0 Connection for Google API perform following steps.

1. Download and Install SSIS PowerPack
2. Create new SSIS Package
3. Right click in Connection Manager Area and Click “New Connection”
4. When prompted select ZS-OAUTH connection type
5. On the OAuth Connection Manager Select Provider=Google Sheets + Drive.
6. You can keep “Use Default App” selected or choose Custom App. Click here to learn – how to register Custom Google App
7. If you choose the Custom App option then enter the below scopes (or click Select Scopes). If you choose the default app then it’s not required.

   This will allow read / write access to Drive Files or Sheet (E.g. Reading file content / Export file to CSV). You can use Scope browser to see many other available permissions.

   https://www.googleapis.com/auth/drive
   https://www.googleapis.com/auth/drive.file
   https://www.googleapis.com/auth/drive.readonly
   https://www.googleapis.com/auth/spreadsheets
   https://www.googleapis.com/auth/spreadsheets.readonly

   Click Generate Token button.

   

   SSIS OAuth Connection Manager – Access Google Drive API using OAuth 2.0

8. You will see UI as below… Click Accept (You may have to scroll to see that button sometimes)
9. Click Test to see connection is working.

NOTE: If you don’t want to use Default OAuth App provided by ZappySys then select “Use Custom OAuth App” option (App created in previous section) specify your ClientID, ClientSecret.

Find Google Sheet ID for API call

Before we can read or write Sheet data in SSIS we need to know Google Driver File ID. We will use this ID in next few sections.

Easiest way to find File ID is look at the URL like below. If you need to know ID at runtime dynamically then refer last section.

Get Google Drive File ID for API call (Sheet ID)

Reading Google SpreadSheet Data in SSIS

Basically there are two ways you can export / read Google SpreadSheet data.

• Method-1 : Read Google SpreadSheet using SSIS JSON / REST API Source
  • In this approach we directly pull data from SpreadSheet using Core Google SpreadSheet.Values API to read particular Tab from Sheet and you can also specify Range. This is most effective way to read. Make sure you have v2.5.0.10807 or higher version (released after Aug 03 2017). New version added 2D array transformation options to parse multi-dimensional arrays found in SpreadSheet API JSON response.
• Method-2 : Read Google SpreadSheet using Flat File Source (as CSV)
  • Another approach is export SpreadSheet as CSV format. Then use Flat File Source to read that data and load into Target (E.g. SQL Server)

Method-1 : Read Google SpreadSheet using SSIS JSON / REST API Source

Now lets look at real steps to read Google SpreadSheet. To get values from specific tab and specific cell-row range you have to use Google SpreadSheet.Values API

In this example we will load below Google Spreadsheet into SQL Server

Step-By-Step : Loading Google SpreadSheet into SQL Server using SSIS

1. Drag and drop data flow on the control flow designer surface
2. Double click data flow and you will see SSIS Toolbox refreshed with new components
3. Drag ZS JSON Source from SSIS Toolbox onto Data flow designer surface
4. Double click JSON SOurce and configure below setting
   1. Enter your Spreadsheet URL in the following format
      

      https://sheets.googleapis.com/v4/spreadsheets/{your-file-id}/values/{cell-range}

      For example purpose we will use this public SpreadSheet provided by Google. If spreadsheet fileid is 1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms (see previous section how to get fileid) and your tab name is Class Data and you like to extract data from A to F columns starting from 2nd row then your Cell range in the URL can be ‘Class Data’!A2:F
      Here is the actual URL we will use for demo
      

      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/'Class Data'!A2:F

      — OR —
      You can also enter like below (If you omit Tab name then first table is used)
      

      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A1:F
      
      --OR-- Use below URL for to supply Column names manually (Start from A2)
      
      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A2:F

      You can also enter like below, specify first row and last row
      

      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A1:F101

   2. Check Use credentials and select OAuth connection we created in previous section
   3. Click on Select Filter and select values node and click OK  or you can directly enter following filter.
      

      $.values[*]

      

      Configure SSIS JSON/REST API Source – Read data from Google SpreadSheet API

   4. Now click on array Transformation tab. Now we will show you two different ways to configure column names.
   5. Method-1: Auto Detect Column Names (added in version 3.1.2)
      In recent version we added column less array transformation where we have option to detect first record as column name.
      1. Select Transform Type as Column less array
      2. Check option First line has column names like below example (Ignore direct Sample JSON – We provided for demo only. In your case it will be URL)
         

         JSON With 2D Array Pattern – Column names in First Row (Google Sheets API Pattern)

   6. Method-2: Manually Enter Column Names (For old version)
      1. Select Transform simple 2D array option from dropdown
      2. Check Specify columns list manually
      3. Enter column names (Must match the order and count based on range you selected)
         

         Student Name,Gender,Class Level,Home State,Major,Extracurricular Activity

         

         Configure SSIS JSON/REST API Source 2D Array Transformation for Google SpreadSheet API

   7. Now click Preview to see the data
      

      Configure SSIS JSON/REST API Source –
      Preview Google SpreadSheet Data

   8. Click OK to save UI
   9. Attach your JSON Source to target (e.g. OLEDB Destination for SQL Server Table).
   10. Configure Destination
       

       Configure SQL Server destination for Google SpreadSheet to SQL Data load

   11. Execute SSIS Package to load Google SpreadSheet data int SQL Server
       

       SSIS Example -Loading data from Google SpreadSheet into SQL Server Table

Method-2 : Read Google SpreadSheet using Flat File Source (as CSV)

Once we have Google Drive fileId we can perform following actions to export data into CSV file format. After that you can easily read CSV using SSIS FlatFile Source and load into SQL Server or other target.

Export Google SpreadSheet as CSV file using SSIS REST API Task

Here are the steps to export Google Drive SpreadSheet to CSV file

1. Just like previous section configure REST API Task
2. Enter following URL (See we used FileId from SSIS variable. You can hardcode too.
   

   https://www.googleapis.com/drive/v3/files/{{User::varFileId}}/export?mimeType=text/csv

3. Goto response tab. Check Save response and select “Save to file option”.
4. Enter file path.
5. Response character set select utf-8 (This will allow Unicode characters in response)
6. Click Test to see preview
   

   Export Google SpreadSheet to CSV file format using SSIS – Call Google Drive API to Export

7. Click OK to save

Load Google SpreadSheet data into SQL Server using SSIS (REST API Call)

Once you export SpreadSheet to CSV file you can easily consume it to load into SQL Server or any Target (e.g. Oracle, MySQL) using Native SSIS FlatFile Source and OLEDB/ADO.net Destination.

Write data to Google Spread Sheet

Now lets look at how to write to Google SpreadSheet. To write multiple rows in a single request you may use batchUpdate API. Also check this link to learn more.

Below is sample request to write data to columns and some data to rows (set majorDimension = COLUMNS to write vertical and use ROWS to write horizontally. You can mix multiple ways just like below example)

POST 
https://sheets.googleapis.com/v4/spreadsheets/1lkMEgu0zm2Q-cnPeoNZp4hqeIuASxXSpsrxtDDV89tg/values:batchUpdate
Authorization: Bearer ya29.Gl3WBFb3xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: application/json
Accept: */*

{
  "valueInputOption": "USER_ENTERED",
  "data": [
    {
      "range": "Sheet2!A1:A4",
      "majorDimension": "COLUMNS",
      "values": [
        ["Item", "Wheel", "Door", "Engine"]
      ]
    },
    {
      "range": "Sheet2!B1:D2",
      "majorDimension": "ROWS",
      "values": [
        ["Cost", "Stocked", "Ship Date"],
        ["$20.50", "4", "3/1/2016"]
      ]
    }
  ]
}

Write data to Google SpreadSheet using REST API Task (OAuth 2.0 Connection)

Update Google Sheet Cells – Single Range (Set Values for multiple cells)

Now let’s look at the below example of updating multiple cells in google Sheet. For more information, you can see the example here. For parameter detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve Spread Sheet Update scenario. For simple example lets use REST API Task.

1. Drag and Drop ZS REST API Task on the control flow designer
2. Edit Task and select URL from Connection Method
3. Select OAuth Connection from the dropdown (Same connection we created in the previous section)
4. Enter the URL as below. Change Sheet ID, Range designator and API URL Parameters as per your need.
   https://sheets.googleapis.com/v4/spreadsheets/[Sheet-ID-Here]/values/[Range-Here]?[Parameters-Here] Example URL (some part masked with xxxxxxxx):  
   

   https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4?valueInputOption=USER_ENTERED

5. Change Request Method to PUT
6. Change Request Content type to application/json
7. Enter Request Body as below (Assuming we need to update 4 cells)
   

   {
   "majorDimension": "ROWS",
     "values": [
      ["row1_cellA","row1_cellB"],
      ["row2_cellA","row2_cellB"],
     ]
   }

8. Click Test Request see it works.
   

   Calling Google Sheet API to Update Multiple Cell Values

Update Google Sheet Cells – Multiple Ranges

If you like to update values which needs to define multiple ranges then you need to use batchUpdate API. Import changes are you can’t include Range in the URL like previous example of Single Range Update. Also you have to use POST method and body with multiple ranges.

Example of sheet update with multiple range

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHxxxxxxxxxxxxxxxc/values:batchUpdate?valueInputOption=USER_ENTERED
Content-Type: application/json

>>>> BODY <<<<<

{
  "data": [
    {
      "range": "Sheet1!A2:B3",
      "majorDimension": "ROWS",
      "values": [
        [
          "row1_cellA",
          "row1_cellB"
        ],
        [
          "row2_cellA",
          "row2_cellB"
        ]
      ]
    },
    {
      "range": "Sheet1!D2:E3",
      "majorDimension": "ROWS",
      "values": [
        [
          "row1_cellD",
          "row1_cellE"
        ],
        [
          "row2_cellD",
          "row2_cellE"
        ]
      ]
    }
  ]
}

Write SQL Server data to Google Sheet

Now let’s look at real world example. How to write data from SQL Server table or other source and send to Google Sheet.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox
   
2. Drag source (e.g. OLEDB Source) and configure it to read from source like SQL Server Table
   

   OLE DB Source – Drag and Drop

3. Drag ZS JSON Generator.
   1. Connect Source to JSON Generator (blue arrow)
   2. Double click it to configure
   3. Select Output Mode as Single Dataset Array option for Mode. Enter Some Batch value (e.g. 500) this will make sure we send 500 rows at a time to Google API call. This way if we have many rows its not rejected by API call.
   4. Click Add element (Select Static Element) . Name: majorDimension and Value: ROWS
   5. Now click Add Document Array icon, name as values
   6. check Treat as 2D array option and click OK to save
   7. Now under values node click Add elements > Select Multiple Columns and Add columns you like to add.
   8. Click OK to save JSON GeneratorBelow is just an example How you can use Batch Settings to create JSON with N rows in each JSON document.
      
4. Now drag ZS Web API destination
   1. Connect JSON Generator to Web API destination (blue arrow)
   2. Double click it to configure
   3. Select same OAuth connection manager created in earlier section (Assuming you had all scopes with write permission)
   4. Now select Input column for Body (JSON Generator Output)
   5. Enter API call URL as below (Replace ID with your own id)
      

      https://sheets.googleapis.com/v4/spreadsheets/{your-file-id}/values/Sheet1!A2?valueInputOption=USER_ENTERED

   6. Select Request Method as PUT
   7. Select content type as application/json
   8. You can enter sample Body As below.
      

      {
      "majorDimension": "ROWS",
        "values": [
         ["Bob","bob@mycompany.com",55],
         ["Sam","sam@mycompany.com",57]
        ]
      }

   9. Click Test Request / Response button. Check your Sheet in browser after it see it updates?
   10. If you see 200 OK response means we are good to go.
   11. Run package and see your real data updated in Sheet.

Format Cells using Google Sheet API

Now let’s look at an example to format cell using Google Sheet API (e.g. Change Font, Color, Border, Background). For more information review this API documentation.

To format Sheet you first need to get SheetID  (keep in mind this is not same as spreadsheetId which you used in URL). SheetId is basically numeric ID of your Tab in WorkSheet.

Get SheetID

Request

GET https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHtxxxxxxxxxxxxxx

Response

{
  "spreadsheetId": "1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc",
  "properties": {
    "title": "Test.xlsx",
    "locale": "en",
    "autoRecalc": "ON_CHANGE",
    "timeZone": "America/Los_Angeles",
    "defaultFormat": {
      "backgroundColor": {
        "red": 1,
        "green": 1,
        "blue": 1
      },
      "padding": {
        "right": 3,
        "left": 3
      },
      "verticalAlignment": "BOTTOM",
      "wrapStrategy": "OVERFLOW_CELL",
      "textFormat": {
        "foregroundColor": {},
        "fontFamily": "Calibri",
        "fontSize": 11,
        "bold": false,
        "italic": false,
        "strikethrough": false,
        "underline": false
      }
    }
  },
  "sheets": [
    {
      "properties": {
        "sheetId": 1896331083,
        "title": "Sheet1",
        "index": 0,
        "sheetType": "GRID",
        "gridProperties": {
          "rowCount": 1000,
          "columnCount": 26,
          "rowGroupControlAfter": true,
          "columnGroupControlAfter": true
        }
      }
    }
  ],
  "spreadsheetUrl": "https://docs.google.com/spreadsheets/d/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc/edit"
}

Notice that in above response we see “sheetId”: 1896331083. This ID we will use in the Update API to format Cells.

Update Borders Example

Here is the example of Google Sheet Formatting API updateBorders Request

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSxxxxlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "updateBorders": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 0,
          "endRowIndex": 10,
          "startColumnIndex": 0,
          "endColumnIndex": 6
        },
        "top": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
        "bottom": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
        "innerHorizontal": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
      }
    }
  ]
}

Response

{
  "spreadsheetId": "1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc",
  "replies": [
    {}
  ]
}

Update Cell Font Style (Bold, Style, Color, Background)

Now let’s look at how to use repeateCellRequest to apply certain type of formatting to multiple cells. This may be color, font style, background color.

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3xxxxxx4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "repeatCell": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 0,
          "endRowIndex": 1
        },
        "cell": {
          "userEnteredFormat": {
            "backgroundColor": {
              "red": 0,
              "green": 0.5,
              "blue": 0.7,
              "alpha": 1.3
            },
            "textFormat": {
              "foregroundColor": {
                "red": 1
              },
              "bold": true,
              "italic": true,
              "strikethrough": true,
              "underline": true
            }
          }
        },
        "fields": "userEnteredFormat"
      }
    }
  ]
}

In above example we have applied textFormat and backgroundColor but you can use any of below sections.

Set Currency Formatting

Here is an example request to set currency formatting

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "repeatCell": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 1,
          "startColumnIndex": 4,
          "endColumnIndex": 5
        },
        "cell": {
          "userEnteredFormat": {
            "numberFormat": {
              "type": "CURRENCY",
              "pattern": "$#,##0.00"
            }
          }
        },
        "fields": "userEnteredFormat.numberFormat"
      }
    }
  ]
}

Set Cell Validation

You can also set cell validation by calling following API call. This will restrict user’s ability to enter wrong data in certain cells.

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "setDataValidation": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 1,
          "startColumnIndex": 0,
          "endColumnIndex": 6
        },
        "rule": {
          "condition": {
            "type": "ONE_OF_LIST",
            "values": [
              {
                "userEnteredValue": "Red"
              },
              {
                "userEnteredValue": "Green"
              },
              {
                "userEnteredValue": "Yellow"
              }
            ]
          },
          "inputMessage": "Color must be Red, Yellow or Green",
          "strict": true,
          "showCustomUi": true
        }
      }
    }
  ]
}

Dynamically obtain File ID for Google Drive REST API call

Once OAuth connection is defined you can now call any valid API from Google based on Permission you requested in OAuth Connection. OAuth connection takes care of renewing you Token and passing Authorization as per OAuth standard. All these protocol complexity is hidden from you.

First step to get data from Google SpreadSheet using REST API call is get fileId of the Google Drive File you want to read. The easiest way to know file ID is call following API. Below API call returns name, ID and other information about file(s). You can get only fileID and use that in Next Section to export Google SpreadSheet to CSV format.

https://www.googleapis.com/drive/v3/files

OR

https://www.googleapis.com/drive/v3/files/?q=name%3D'Customers'

NOTE: For more information about searching files in Google drive / getting list of files using search expression language check this link. Make sure expression is URL encoded (e.g. name=’Customers’ should be passed as name%3D’Customers’)

Above API returns response as below. Notice the name and fileId. We will use fileId when calling export API in next section.

{
 "kind": "drive#fileList",
 "files": [
  {
   "kind": "drive#file",
   "id": "1lkMEgu0zm2Q-cnPeoNZp4hqeIuASxXSpsrxtDDV89tg",
   "name": "Customers",
   "mimeType": "application/vnd.google-apps.spreadsheet"
  },
  {
   "kind": "drive#file",
   "id": "0B6UyXr6ZlIFTc3RhcnRlcl9maWxl",
   "name": "How to get started with Drive",
   "mimeType": "application/pdf"
  }
 ]
}

If you want to automate retrieval of fileId from document name then perform the following steps.

1. In the SSIS package Control flow. Drag and drop ZS REST API Task from SSIS toolbox
2. In the Request Access mode, select “URL from Connection” option
3. Select OAuth connection manager (Created in Section-2)
4. Enter URL as below (It will return any file name equals to Customers (%3D is encoded value for = sign)
   

   https://www.googleapis.com/drive/v3/files/?q=name%3D'Customers'

5. Click Test (You will see file information)
   

   Get Google Drive File List in SSIS – Call Google Drive API to search file by name or expression

6. If you want to extract only fileID from full JSON then go to response tab
7. Select Format=JSON, Enter following JSONPath expression to extract only fileId from first record
   

   $.files[0].id

8. Check save response content, select SSIS variable where you want to save fileId as below. Now click Test to see preview of response. You will notice now only fileId will be returned from full JSON response.
   

   Get Google Drive fileid from file name using API call in SSIS

9. Click OK to save task

Conclusion

Google Drive API and Google Sheets API provides a great way to automate file related functionality. However, to call Google API  you have to use SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Google API call via simple drag and drop approach without coding. Try out yourself see how long it takes to call virtually any REST API.

The post How to read / write Google SpreadSheet using SSIS appeared first on ZappySys Blog.

Google AdWords is one of the most popular Advertise Platforms. In this article, you will learn how to get data from Google AdWords by calling Google AdWords Web Service method and using drag and drop SSIS XML/SOAP connector (No coding required :)). Google AdWords Web Services allows retrieving various data such as account info, campaigns; as well as writing data, such as creating a campaign, add a keyword. Basically, it allows controlling your AdWords account programmatically instead of using the AdWords dashboard. For demo purposes, we will get data from Google AdWords Account and load it into Microsoft SQL Server (but you can use this technique to save any supported targets in SSIS (e.g. Excel, FlatFile, Oracle, MySQL, IBM DB2, etc.).

If you desire to download a report from AdWords, you must follow a different approach described in an article on how to download AdWords report. It describes how to download a report using ODBC and XML Driver but the concepts introduced there can be applied to SSIS as well.

If you are looking for a similar article for Google DoubleClick for Publisher (Google DFP) then check this article.

You will also learn how to create Google Ads or modify any other objects by calling SOAP API using SSIS REST API Task.

SSIS Tasks/Connectors discussed in this Article

In this article we will use following Connector(s) and Connection Manager to archive full data integration with Google AdWords.

Following SSIS Connectors support OAuth Connectivity. When you use below connectors no coding required to pull data from any OAuth enabled services (e.g. Twitter, Facebook, LinkedIn).

	REST API Task
	OAuth Connection Manager
	XML Source (REST API, SOAP Call or File data)

Two different ways to get Google AdWords Data in SSIS

There are two ways you can extract Google AdWords data in SSIS.

1. Use Google Analytics Source (For more info see this article)  (Consume AdWords data using Google Analytics REST API)
2. Use XML Source along with OAuth connection manager (Consume AdWords data using SOAP API)

In this article we will only discuss Approach #2. For approach #1 refer see this link

Advantage of #2 is you get total control of full API set for AdWords but you have to do lot more manual work. For Approach #1 is lot simpler but you get access to limited reports (Dimensions and Metrics).

Google AdWords API Integration Example Screenshot

Below is screenshot of sample package discussed in this article (Download from here). As you can see we tried to demonstrate Reading data and Managing data (e.g. Create Ad) in the same package. You can extend these technique to perform any type of operations in Google AdWords or similar service such as DoubleClick for Publisher

Google AdWords API Integration Example – Create Ad, Get Accounts, Get Campaigns, Get Ad Groups, Get Ads (Extract AdWords data via SOAP API, Read data using Selector Method or use AWQL query language)

Step-By-Step Tutorial

Here are few high level steps you will have to perform to extract data from Google AdWords API using SSIS. This section assumes you have ClientID and Developer Token to access data. if you don’t have that then read next 2-3 sections.

1. Download and install SSIS PowerPack from here
2. Create new SSIS Project
3. From SSIS toolbox drag DataFlow Task and go to dataflow designer
4. From toolbox of drag ZS XML Source
5. Double click XML Source to configure it
6. Specify API URL from which you want to read data
   Example : https://adwords.google.com/api/adwords/mcm/v201509/ManagedCustomerService
7. Now check Use Credentials check box and select New ZS-OAuth connection manager.
8. On connection manager UI you can select Google from Provider Dropdown.
9. Under Scopes enter following (Scope is basically permission)
   

   https://www.googleapis.com/auth/adwords

    
10. Now click Generate Token and when prompted login and in the last step Accept.
11. If everything went ok then you will see AccessToken and RefreshToken populated.
12. Click Test connection and close the UI by clicking OK
13. Now on XML Source – Select HTTP Request Method as POST
14. In the Request Body click edit and enter request as below (Replace DEV_TOKEN and CLIENT_ID placeholders below)
    

    <?xml version="1.0" encoding="utf-8"?>
    <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    	<soap:Header>
    		<RequestHeader xmlns="https://adwords.google.com/api/adwords/mcm/v201605">
    			<developerToken>ENTER_YOUR_DEV_TOKEN_HERE</developerToken>
    			<clientCustomerId>ENTER_YOUR_CLIENT_ID_HERE</clientCustomerId>
    			<userAgent>ZappySys SSIS PowerPack v1</userAgent>
    		</RequestHeader>
    	</soap:Header>
    	<soap:Body>
    		<get xmlns="https://adwords.google.com/api/adwords/mcm/{{User::ApiVersion}}">
    			<serviceSelector>
    				<fields>CustomerId</fields>
    				<fields>Name</fields>
    			</serviceSelector>
    		</get>
    	</soap:Body>
    </soap:Envelope>

     

    If you are wondering how did we get above XML fragment for Request Body then Check this article on how to use SoapUI (free 3rd party tool) to create SOAP request Bodyfrom WSDL. Google AdWords has many API endpoints for different service. For each service you have different Api URL and different WSDL. In the next section you can learn how to create correct SOAP Body (for POST) using SoapUI tool.

    For example if you are calling ManagedCustomerService API for version v201609 then your help page URL would be like this
    https://developers.google.com/adwords/api/docs/reference/v201609/ManagedCustomerService

    And on the same page you will see WSDL link below
    https://ads.google.com/apis/ads/publisher/v201702/NetworkService?wsdl
    (Just download that WSDL XML and save to local disk then use with SoapUI to generate XML body to submit for any DFP API call.)

    See this link for more information (Its for Google DoubleClick Ad Platform but you will get an idea)

15. Go to Array handling tab at the bottom and enter text : ns2:entries – By doing this we will treat <ns2:entries> node as array eventhough only one entry found in response (By default content with multiple records treated as array).
    

    Handling Array Element Nodes in SOAP Response

16. On the filter expression click edit and highlight entries node and click ok. This will produce something like below in filter textbox
    

    $.soap:Envelope.soap:Body.ns2:getResponse.ns2:rval.ns2:entries[*]

     
17. Now click preview to see actual data.
18. Click OK to save UI. Now you can connect source to target such as SQL Server or Trash Destination.

NOTE: in above example we used v201605 API version but change it to latest if this API is no more valid. Google expires API version pretty often so keep that in mind if you get depreciated API version error.

Register for Developer Token to access AdWords account data using API

First important step to access any Google AdWords Account is request developer token. Click this link for step by step information. Watch short vide to get overview on entire process to access data via API.

Find your Client ID

Next thing you will need to know is your ClientID. Most of API Requests will require your ClientID. Perform following steps to find your ClientID.

Login to your AdWords Account using email address. At the top you will see your client ID as below screenshot.

How to find ClientID of your Google AdWords Account (Useful for making SOAP API Calls)

Connect to Google AdWords API using OAuth

Connecting to any Google API will require OAuth 2.0. SSIS PowerPack has Protocol so you can easily extract data from API without coding complex token extract/renew workflow. Such things usually requires some sort of programming but in our case you will do few clicks as described in Step-By-Step section earlier.

Here is the screenshot of OAuth Connection Manager.

Connecting to Google AdWords API using OAuth 2.0 in SSIS (Consuming XML/SOAP API)

Extract data from Google AdWords using SOAP API Calls

When you define filter expression you can check/uncheck “Include Parent Columns” setting. For AdWords SOAP Api you can uncheck so you only extract columns from rvals node in XML Response.

Extract data from Google AdWords SOAP/XML Response

SOAP Response Paging / Looping

Many time you will get response which is larger than default page size in AdWords API Call. In that case you can specify few pagination attributes on XML Source Paging tab. For more information on paging options you may refer this article . This article is talking about JSON Source but same options are applicable for XML Source too.

REST API Pagination – Passing PageNumber inside POST Body (Example: Google AdWords SOAP Api)

Configure REST API Pagination – using Page Number via POST Body Method

Read Google AdWords data and Load into SQL Server

Using XML Source you can extract data in tabular format and then you can map that to SQL Server Target (Use OLEDB Destination / ADO.net Destination).

Extract data from Google AdWords and load into SQL Server table using SSIS

Write data / Manage AdWords Account via REST API Calls

So far we have seen how to read data from AdWords account. But what if you have to create new records or write/update AdWords data via API Calls in automated manner? No worry – If you have need for that then use SSIS REST API Task which supports adhoc HTTP WebRequests including POST/DELETE/PUT etc. You can also upload files via REST API call using this task. This task supports rich error handling and saving your response into variable or files.

Create Ad under specific AdGroup

Below example shows how to create new ad under specific AdGroup using REST API Task

Notice we have used few placeholders to make things dynamic using SSIS Variables. Once REST API Task is executed you will see new ad created.

Creating Google AdWords Ad using SOAP/XML API in SSIS (REST API Task)

Verify Google Ad created programmatically using SOAP/XML API Call (SSIS REST API Task)

Methods for Creating new records

Google AdWords API supports mutate method for writing data. Many service endpoints supports this method. Below example request will create new Ad.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<developerToken>{{User::varDevToken}}</developerToken>
			<clientCustomerId>{{User::varClientId}}</clientCustomerId>
			<userAgent>My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)</userAgent>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<mutate xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<operations>
				<operator>ADD</operator>
				<operand>
	                <adGroupId>30800928123</adGroupId>
	                <ad xsi:type="TextAd">
	                    <displayUrl>//zappysys.com/ad1</displayUrl>
	                    <finalUrls>//zappysys.com/ad1</finalUrls>
	                    <headline>This is my Ad</headline>
	                    <description1>This is Line 1</description1>
	                    <description2>This is Line 2</description2>
	                </ad>            
				</operand>
			</operations>
		</mutate>
	</soap:Body>
</soap:Envelope>

Methods for Reading AdWords data

Google AdWords API supports two methods for reading data. Most of AdWords service endpoints supports GET method also known as Selector method (Its basically sending query in XML format). Seconds method is Query Method. It uses SQL like query language known as AWQL. (Read more about AWQL query language). If you are SQL Programmer then you may like AWQL Query method better than Selector. But either way you will be able to pass following information.

• Columns you want to return
• Filter (e.g. Where Clause)
• Order By
• Paging information (e.g. PageNumber, PageSize)

Get Method (XML Selector)

For Get Method using XML tags to define columns, filters and order by/paging information. Here is example Get method for below select query

SELECT Id, Heading
WHERE Status = 'ENABLED'
ORDER BY Heading DESC 
LIMIT 0,50

Get Request (Get all Active Ads)

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<developerToken xmlns="https://adwords.google.com/api/adwords/cm/v201605">{{User::varDevToken}}</developerToken>
			<clientCustomerId xmlns="https://adwords.google.com/api/adwords/cm/v201605">{{User::varClientId}}</clientCustomerId>
			<userAgent xmlns="https://adwords.google.com/api/adwords/cm/v201605">My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)</userAgent>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<get xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<serviceSelector>
			    <fields>Id</fields>
			    <fields>Headline</fields>
			    <predicates>
			        <field>Status</field>
			        <operator>EQUALS</operator>
			        <values>ENABLED</values>
			    </predicates>
			    <ordering>
			        <field>Headline</field>
			        <sortOrder>ASCENDING</sortOrder>
			    </ordering>
			    <paging>
			        <startIndex>0</startIndex>
			        <numberResults>50</numberResults>
			    </paging>
			</serviceSelector>
		</get>
	</soap:Body>
</soap:Envelope>

Query Method (AWQL Query – SQL like query language)

Below is sample Soap request for AWQL Method to read Ads listed under from specific AdGroups. To learn more AWQL Query Syntax Click Here.

To learn more about which Columns you can use inside your AWQL Query Click here. Note that each service support different fields for Select Clause, Where Clause and Order By Clause. AWQL language can be also used to query Predefined AdWords Reports. The only difference is .. FROM clause is required when you query Reports.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<developerToken xmlns="https://adwords.google.com/api/adwords/cm/v201605">YOUR_DEV_TOKEN</developerToken>
			<clientCustomerId xmlns="https://adwords.google.com/api/adwords/cm/v201605">YOUR_CLIENT_ID</clientCustomerId>
			<userAgent xmlns="https://adwords.google.com/api/adwords/cm/v201605">My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)</userAgent>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<query xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<query>
SELECT Id, Name
WHERE Status = 'ENABLED'
LIMIT 0,5
			</query>
		</query>
	</soap:Body>
</soap:Envelope>

Call AdWords Reports API – Download Campaign Performance data

There will be a time you like to download Reports from Adwords. For that  read this article. Its showing how to use ZappySys ODBC Driver to call AdWords Reporting API but concept is same in SSIS Product too due to similar UI elements.

Download Example SSIS Package

Download SSIS 2012 / 2014/ 2016 Sample

Conclusion

Google AdWords data integration can be very challenging without right set of tools. SSIS PowerPack certainly makes it easy when you have to extract/manage Google AdWords data. Innovative drag and drop approach gives you various options to cover simple or most advanced scenarios without relying on any Programming language (e.g. C#, Java, Python, Ruby etc).

The post How to get data from Google AdWords using SSIS appeared first on ZappySys Blog.

In our previous article we understood little bit about OAuth 2.0 Protocol. We talked how to use Default App for ease of use. However In this post you will learn how to register Google OAuth Application so you can access various Google Services programmatically (e.g Youtube, Google Analytics, AdWords, DoubleClick etc.). Here is the link to find all services: Google API Services (Using OAuth 2.0)

Recently google started supporting OAuth 2.0 Protocol. There are many articles written explaining OAuth 2.0 protocol so we are not going to cover that in this article.

Following SSIS Connectors support OAuth Connectivity. When you use below connectors no coding required to pull data from OAuth enabled services.

SSIS Tasks / Components which Supports OAuth Connection / Google API

	API Source
	Google Analytics Source
	JSON Source Connector (File, REST API)
	REST API Webservice Task
	XML Source (File, REST, SOAP)
	Web API Destination

ODBC Drivers which Supports OAuth Connection / Google API (Call in Power BI, Excel, Informatica, MS Access)

	API ODBC Driver
	JSON ODBC Driver
	XML ODBC Driver
	CSV ODBC Driver

Create Google Project

Registering OAuth App in Google first requires you to create API Project. API Project is a container which can host many OAuth Apps or other API Keys.

1. To create API Project Visit https://console.developers.google.com/project
   If you already have existing API Project then once you click above link you can pick desired API Project from dropdown (Found in the top bar). If you using existing API Project then you may skip next few steps and directly move to step which explains [how to create OAuth ClientId].
2. To create new project Click on “Create Project” button. Once you click Create it will take few seconds to create project for you.
   

   Create New Google API Project (Google API – Register new OAuth App)

3. Now give some friendly name and change default Project ID (optional) for your project and click Create (See below).
   
   NOTE: Before you click create you can change default Project ID suggested by Google by clicking Edit. This Project ID is used everywhere when calling many APIs like this one so copy this ID somewhere (You can also see this on project screen so dont worry if you miss it).Project ID is unique across google system so you may not get your preferred ID but if you can get one then its easy to remember for sure.
   

   Create new Google Api Project for OAuth App (Set project name / edit default project ID)

Enable Google API for Integration

Once your Project is created, next step is enable APIs you like to call. By default most Google APIs are disabled so make sure you enable each APi category manually using below steps.

1. Once project is created it will redirect you to the API Home page where you will see many common Google API you can use.If you do not see that you can always click below option to enable API manually.Visit : https://console.cloud.google.com/apis/library
   

   Enable Google APIs / Services

2. By default API you like to call might not be enabled. Search for API you wish to enable, click on the API name and click Enable.Some common Google API Examples:
   1. To enable Google YouTube API
      > search for “YouTube“. It has few APIs so pick the one you need under this,
      > For example To manage/view channel use Data API, for Reporting use Analytics API)
   2. To enable Google Drive API
      search for “Drive” and click Enable
   3. To enable Google Sheets API
      search for “Sheet” (Some Sheets API operations need Drive API too see enable that as well)
   4. To enable Google BigQuery API
      > search for “BigQuery“
   5. To enable Google Analytics (GA3)
      > search “Analytics“ and enable Google Analytics API
   6. To enable Google Analytics (GA4)
      > search for Google Analytics Data API and click Enable, Search for Google Analytics Admin API and click enable (needed this to query available properties)
      

      Search Google API for OAuth Application Access

       

      

      Enable Google API for OAuth Application Access

       

Create OAuth Credentials (Client ID / Secret)

Once we create project and enable APIs our next step is to obtain OAuth ClientID / Secret. Before you can obtain that you have to configure few properties of your OAuth Consent Screen. Most fields are optional on this screen but fill out them if you plan to publish this API App for other users (i.e. Public App).

1. Now click on the credentials tab and click “OAuth consent screen”. Fill out necessary information which will be visible to the user (such as App Name). Once you are done, continue on with credentials creation:Google OAuth API – Create OAuth App – Set Application Name on Consent Screen
2. Click on [Create credentials] button and select OAuth Client Id option.
   

   Create OAuth Client ID Credentials for Google API access

3. Next you will be asked to select Application Type. Pick “Desktop app” if your Application is Desktop style application. If your app is web site or running browser then pick Web Application. Click on Create and you will see your ClientID and ClientSecret which you can use during OAuth Authentication. Keep these both information secret. This is used to call OAuth Approval Screen by your Desktop App or Web App… and then user can obtain Token which can be used to call desired Google API.
   NOTE: Make sure you do not choose Web App and only select Desktop App. Desktop App doesnt have Redirect URL making it simple to setup and use with ZappySys software.
   

   Create Google OAuth 2.0 Credentials – Desktop App

   

   Generate ClientID and Client Secret for Google OAuth Application

Publish App

The last step in App creation is to go to the OAuth Consent Screen tab. Under Publishing Status, click PUBLISH APP to ensure your refresh token doesn’t expire often. If you plan to use the App for Private use, you do not have to worry about the Verification Status after Publishing (Which is the case for most users who access their own data).

Publish Google App (This avoids Refresh Token Expiration every 7 days)

Using Client ID and Client Secret in ZappySys OAuth Connection

You can use generated ClientID and ClientSecret to initiate OAuth Authorization process. We are not going to cover how that works, but below is a simple screenshot of how these two pieces of information are used to call the Consent screen with some desired permission requested by calling Application:

ZappySys OAuth Connection can be used by many components suggested in the beginning of this article. So refer them for more information.

Google OAuth Application: Generating Access and Refresh tokens using SSIS and OAuth Connection Manager.

Using System Default Browser to get the token

If embedded browser has some issue to extract your token then you can close the window by pressing [X] in the title bar. Once you do that you will be prompted to use System Default browser to finish the token extraction. If you click Yes then Full browser will be launched.

NOTE: Below feature is added in version published after Dec 31, 2019 (v3.0.0.11231 or later) so make sure you have latest version.

Using System Default Browser to extract OAuth Token

Untrusted App Warning

Sometimes Google might show you below warning. In order to continue the flow you might have to Click on Advanced and then click Go to YourAppName (unsafe) like below

How to ignore Google Unsafe App Warning (OAuth Flow)

OR, in some cases, you may see modern UI like below. In either case, choose continue.

Google App not Verified Warning (New UI)

Select the account

Using System Browser

Copy code and finish the flow

Manually Copy Code and Get OAuth Refresh / Access Token

Why I didn’t get refresh token back?

For some reason if you just see Access token and not refresh token then try following

1. Logout current session in browser and close all browser windows and try again
2. If you use Custom Provider rather than Google on OAuth connection then change Auth URL to include prompt and access_type like below
   

   https://accounts.google.com/o/oauth2/auth?access_type=offline&prompt=consent

What to do if Google Refresh Token expires every 7 days

Technically, once you have a refresh token, you should not worry about renewing as long as the API call is made atleast once every few months (max 6 months idle time). However, in some cases, your refresh token can expire sooner, and you have to click Generate Token manually

One main reason that is not well documented is if the Google Oauth Private App you created to access your own data is still in “testing” mode and not “published,” then as per Google, you may get this error. Perform these steps to publish your app, which is in testing mode, so you do not have to renew the refresh token every 7 days.

1. Go to your Project in Google API console
   Generally here
2. No API call has been made using the generated refresh_token for six months or longer
   SOLUTION:  Make at least some API in six months to keep refresh_token active
3. Your OAuth App is in “testing” status and not published yet
   SOLUTION: Just click the Publish button on your App OAuth Consent screen. Do not worry about getting verified if you plan to use it for internal use; after this step, your refresh_token may not expire every 7 days See this thread.DO NOT WORRY ABOUT GETTING VERIFIED – Just publish it to avoid refresh token expiration every 7 days.
   

   Publish Google App (This avoids Refresh Token Expiration every 7 days)

What is Google OAuth AccessToken

AccessToken is short lived token which you can pass along with each API Service call. once this token is expired you can get new AccessToken as long as you have obtained and saved RefreshToken (Returned to caller after very first time user go through Approval process on Consent screen). If you are using ZappySys OAuth connection manager

What is Google OAuth RefreshToken

May OAuth services support short lived token which expires after few minutes or hours. Google API by default expires token after 3600 seconds … So how to get new token ? Do you always go through that annoying Consent screen… what if your application is Automated Service? Well OAuth has solution for that too… its called RefreshToken.

First time to setup OAuth connection user have to go through Approval screen  (i.e. Consent screen) once they click Approve button they may receive two types of token (AccessToken and RefreshToken).  RefreshToken is used only when you need to get new AccessToken without going through Consent screen. So technically if OAuth supports RefreshToken (This is optional) then you can Save this piece of information in database somewhere and when you get token_expired error or you detect expiration time you can get new AccessToken without involving user consent (because you already did once).

Conclusion

Hope this article helped you to understand how to register Google OAuth Application. Download SSIS PowerPack to try various OAuth data integration scenarios using drag and drop high performance connectors. With SSIS PowerPack OAuth support you can virtually pull data from any API service (XML REST API or JSON REST API). Such as Facbook, Twitter, SurveyMonkey, LinkedIn and many more.

The post How to register Google OAuth Application (Get ClientID and ClientSecret) appeared first on ZappySys Blog.

Meanwhile check following similar articles :

SSIS XML Source (FILE, SOAP, REST API Connector)
SSIS JSON Source (File, REST Connector)

Read Zendesk data in SSIS using REST API

Read Twitter data in SSIS using REST API Task and JSON Source – OAuth2 Protocol

Get data from SurveyMonkey in SSIS using REST API

External links

Google Maps REST API Documentation

The post Get data from Google Maps in SSIS using REST API appeared first on ZappySys Blog.

Google Analytics is the most popular web analytics service that tracks and reports website traffic. In this blog post you will learn how to get data from Google Analytics using SSIS using SSIS Google Analytics Source Connector (Using drag and drop approach without Coding). This article assumes you have basic knowledge about SSIS and you have necessary tools installed – Check this if not)

For demo purpose we will load data from Google Analytics into SQL Server Database but you can use same techniques to load data into Oracle, MySQL, Excel or any other RDBMS.

Prerequisite

This article assumes you have installed following tools

1. Install Client Tools for SSIS (atleast one version from below list) – Check this for detail
   1. BIDS for SSIS 2008 (i.e. SQL Server 2008 or 2008 R2) — OR —
   2. SSDT BI for Visual Studio 2012 (i.e. for SQL Server 2012) — OR —
   3. SSDT BI for Visual Studio 2013 (i.e. for SQL Server 2014) — OR —
   4. SSDT BI for Visual Studio 2015 (i.e. for SQL Server 2012/2014 or 2016)
2. Install SSIS PowerPack (Click here to download) it will add new Connector in SSIS Toolbox to access Google Analytics Connector.
3. You have setup correct permission for user to access google analytics data (See below section or read here)

Video Tutorial – Get data from Google Web Analytics Platform

What is Google Analytics Connector for SSIS?

The SSIS Google Analytics Source connector can be used to read data from your Google Analytics Service without learning complex APIs or coding/scripting in Python, Java, C#, etc. It has an easy to use drag and drop interface which allows you to extract large amounts of data, adjusting parameters such as dimensions, metrics, filters, sortby, date range etc. You can also use in-built reports or specify a direct query URL to get the desired data.

Read more about Google Analytics Source Connector

Features

• The intuitive user interface makes Google Analytics data integration super easy and fast without learning complex APIs or scripting languages
• Support for OAuth 2.0 Authentication and Google service accounts (*.p12 certificate file)
• Support for Proxy Server
• Easy to use interface with three different ways to query Google Analytics data: (1) Build query from UI, (2) Use built-in query templates, and (3) Use direct queries
• Support for dynamic dimensions, metrics, segments, filters and sortby fields (e.g. supply these values at runtime)
• Live preview support at design time
• Support for AdWords, AdSense, Ad Exchange and DoubleClick for Publisher (DFP)

Setup Google analytics data permission for users / service account

If you are not an admin of Google analytics and you need to access google analytics then make sure you ask admin to give you correct permission using below steps

To add/edit user permissions for Google Analytics Data perform following steps.

1. Login to your Google Analytics Portal (Login using Admin account or use google user account who setup Google Analytics) Click here to go to google analytics
2. Click on Admin tab > Click User management > Add/Edit user account permission

For more information you may visit official link : How to setup permission for google analytics

Add/Edit Permissions and User Account for Google Analytics Data Access (Service Account/ Normal Google Login)

SSIS Google Analytics Connector – Select Web Profile

Get Data from Google AdWords using SSIS ( use Google Analytics Source Connector)

Conclusion

Integration with Google Analytics Data can be challenging unless you understand how to write C#, Java, Python code using Google SDKs. If you are not a coder or you are in time crunch then drag and drop integration in SSIS using High quality Connector using ZappySys Google Analytics Source Connector will surely make your life easy. It supports all major SQL Server versions upto SQL Server 2016 32-bit/64-bit. Try SSIS PowerPack yourself for FREE.

Keywords

Fetch data from Google Web Analytics using SSIS | Extract google analytics data using SSIS | Load data from Google Analytics without Coding | Google Web analytics data import using REST API

Google Web Analytics Reporting REST API reference link

The post Get data from Google Analytics in SSIS using REST API Call appeared first on ZappySys Blog.