In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate Google Drive into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful Google Drive Connector that makes downloading files from Google Drive a breeze. In this guide, we’ll walk you through the process of downloading files from Google Drive using SSIS and the ZappySys SSIS PowerPack Google Drive Connector.

Download files from Google Drive with SSIS:

In this section, we will learn how to configure and use Google Drive Connector in the API Source & API Destination to download files from Google Drive.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom

   -`
   

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the Google Drive Connector

1. Use a preinstalled Google Drive Connector from the Popular Connector List or press the Search Online radio button to download Google Drive Connector. Once downloaded simply use it in the configuration:
   
2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to Google Drive using User Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with User Account

   Method2: Connect to Google Drive using Service Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with Service Account

How to download a single file from Google Drive

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to download the single file from Google Drive.

1. Choose the Download a file endpoint from the “Select Table/Endpoint” dropdown menu.
2. To access detailed API documentation for downloading files, you can click on the following link: Help for: Download a File.
3. Please choose a File ID from the drop-down menu to indicate which file you wish to download. This is an essential parameter for the download process.
4. Please indicate the desired filename for the file being downloaded by setting the FileName parameter.
5. If you intend to replace the local file with the one being downloaded, please select the ‘AlwaysOverwrite‘ option from the dropdown menu labeled ‘FileOverwriteMode‘. Alternatively, you can choose ‘FileIfExists’ or ‘SkipIfExists’ based on your specific needs.

Please refer to the sample screenshot below.

SSIS API Source – Google Drive – Download Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for downloading a Single file from Google Drive.

How to download multiple files from Google Drive

In the API Destination Component, we have chosen the API Connection Manager. Now, we must proceed to configure the remaining properties of the component to facilitate the download of multiple files from Google Drive. Additionally, we should make use of an OLE DB Source and Derived Column component to map the files for the download process.

1. Drag and drop the OLE DB Source Component from the SSIS Toolbox. Double-click on the OLE DB Source Component to initiate its configuration.
   

   OLE DB Source – Drag and Drop

2. Create a new OLE DB Connection Manager and configure your SQL Server settings for the connection. Next, select the Data access mode, specify the name of the table or view, or write an SQL Command. Afterward, click on the ‘Preview’ button to view a preview and check the ‘Columns’ tab for the source columns.
   

   SSIS OLE DB Source – Configure SQL Connection and fetch the table data

3. Next, drag and drop the Derived Column Component from the SSIS Toolbox. Double-click on the Derived Column Component to begin its configuration.
   

   Drag and drop the Derived Column

4. Now, add a new Derived Column by setting the expression to concatenate FilePath and FileName. Click ‘OK’ to save the configuration.
   

   SSIS Derived Column – Add New Column & Set Expression

5. Now, drag and drop the API Destination Component from the SSIS Toolbox. Double-click on the API Destination Component to begin its configuration.
   

   Drag and Drop the SSIS API Destination Component

6. Double-click on the API Destination component to configure it for uploading multiple files to Google Drive.
   1. Choose the Download a file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. To access detailed API documentation for downloading files, you can click on the following link: Help for: Download File.
   3. Please specify the ‘FileOverwriteMode’ option as ‘AlwaysOverwrite,’ or you can choose ‘FailIfExists’ or ‘SkipIfExists’ options based on your requirements.
      Please refer to the sample screenshot below.
      

      SSIS API Destination Component – Configure Settings Tab

   4. Now, go to the Mappings tab and specify the source columns with the corresponding target properties.
      Note: Here We’re mapping FileID with $$Id and FileNameWithPath with $$ResponseDataFile. You can create mappings based on your Source columns
      Please refer to the sample screenshot below
      

      SSIS API Destination Component – Configure Mappings Tab

7. Once you have configured all the mentioned properties above, please click on the “Preview Data” button from the Settings tab to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for downloading multiple files from Google Drive.
   Please refer to the sample screenshot below.
   

   Download Multiple files from Google Drive

Conclusion

In this blog, we learned how to connect with Google Drive using SSIS API Connection Manager and download single or multiple files from Google Drive using a combination of SSIS components, including the SSIS API Source component, SSIS API Destination component, OLE DB Source component, and Derived Column component.

The post How to download files from Google Drive using SSIS appeared first on ZappySys Blog.

In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate Google Drive into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful Google Drive Connector that makes uploading files to Google Drive a breeze. In this guide, we’ll walk you through the process of uploading files to Google Drive using SSIS and the ZappySys SSIS PowerPack Google Drive Connector.

Upload files to Google Drive with SSIS:

In this section, we will learn how to configure and use Google Drive Connector in the API Source to upload files on Google Drive.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom
   

   SSIS Toolbox – ZappySys Tasks

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the Google Drive Connector

1. Use a preinstalled Google Drive Connector from the Popular Connector List or press the Search Online radio button to download Google Drive Connector. Once downloaded simply use it in the configuration:
   
2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to Google Drive using User Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with User Account

   Method2: Connect to Google Drive using Service Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with Service Account

How to upload a local file to Google Drive

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to upload a local file to Google Drive.

1. Choose the Upload a file endpoint from the “Select Table/Endpoint” dropdown menu.
2. To access detailed API documentation for uploading files, you can click on the following link: Help for: Upload File.
3. Please specify the desired filename for the uploaded file by setting the FileName parameter.
4. Specify the desired local file path by setting the DiskFilePath parameter.
5. If you wish to upload the local file to a specific folder in Google Drive, please choose the folder ID from the dropdown menu labeled ParentFolderId. Otherwise, you can skip this parameter as it’s optional.

Please refer to the sample screenshot below.

SSIS API Source – Google Drive – Upload Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for uploading a local file to Google Drive.

How to upload multiple files to Google Drive

In the API Source Component, we’ve selected the API Connection Manager. Next, we need to configure the remaining properties of the component to upload multiple files to Google Drive. Additionally, we should utilize a Foreach Loop Container to iterate through each file one by one in a loop.

1. Create 3 SSIS variables as described below.
   

   Variable Name	Datatype	Value
   varFileLists	Object	System.Object
   varFileName	String	fake.extension
   varFullFilePath	String	C:/fake/fake.extension

2. Go to Control Flow, From the SSIS toolbox drag and drop ZS Advanced File System Task on the control flow designer surface, and double click on it to edit it:
   
3. Select the “Get file list as ADO.NET Data Table” action from the Action drop-down menu. Set the file path in the Path textbox. Choose the variable (Object type) where you want to store the file list, and then click the OK button to save the settings.
   

   SSIS Advanced File System Task – Get file list as ADO.net Data Table

4. Now, drag and drop the “Foreach Loop Container” from the SSIS Toolbox and connect it with the “ZS Advanced File System Task.” Double-click on the “Foreach Loop Container” to edit its properties.
   

   SSIS Foreach Loop Container – Drag & Drop

5. In the Foreach Loop Editor’s “Collection” tab, choose the Enumerator as “Foreach ADO Enumerator.” In the “ADO object source variable” field, select your SSIS object-typed variable. Select the “Rows in the first table” radio option.
   

   SSIS Foreach Loop Editor – Collection – Select Variable & Enumeration mode

6. In the Foreach Loop Editor’s “Variable Mappings” tab, select the variables that you want to map to the values from the collection, and then click the “OK” button to save your selections.
   

   SSIS Foreach Loop Editor – Variable Mappings

7. Inside the Foreach Loop Container, drag and drop the Data Flow Task, and then rename it according to your preference.
   

   SSIS Data Flow Task – Drag & Drop inside Foreach Loop Container

8. Double-click on the Data Flow Task to see the Data Flow Designer, From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double click on it to edit it:
   
9. Double-click on the API Source component to configure it for uploading multiple files to Google Drive.
   1. Choose the Upload a file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. To access detailed API documentation for uploading files, you can click on the following link: Help for: Upload File.
   3. Please specify the desired SSIS Variable [varFileName] which contains file name for the uploaded file by setting the FileName parameter.
   4. To specify the desired local SSIS variable [varFullFilePath], which holds the file path, set the DiskFilePath parameter.
      When configuring the SSIS variable directly in the Raw Edit, JSON may become invalid due to the requirement of slashes within the local file path. To address this, we use a JSONENC Placeholder Function for proper escaping.
      

      <<{{User::varUserFullFilePath}},FUN_JSONENC>>

   5. If you wish to upload the local file to a specific folder in Google Drive, please choose the folder ID from the dropdown menu labeled ParentFolderId. Otherwise, you can skip this parameter as it’s optional.

   Please refer to the sample screenshot below.

   

   SSIS API Source – Google Drive – Upload Multiple files configuration

   To configure dynamic filenames and file paths, you can set variables. Click on “Raw Edit,” and in the “Edit Parameters” popup, you can use SSIS variables, as shown in the image below.

   Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for uploading multiple files to Google Drive.

Conclusion

In this blog, we learned how to connect with Google Drive using SSIS API Connection Manager and upload single or multiple files to Google Drive using a combination of SSIS components, including the SSIS API Source component, SSIS ZS Advanced File System Task, Foreach Loop Container, and SSIS variables.

The post How to upload files on Google Drive using SSIS appeared first on ZappySys Blog.

In this tutorial, we will use the REST API Task to call some API. We will use the OAuth connection along the REST API Task to write / update data to Google Spreadsheet.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Google API.

Component Mention in this article

	JSON Generator Transform
	Web API Destination
	REST API Task

Create a Google API Project

The first step to access any Google API is to create an API Project in Google Console. If you don’t want to go through this then Skip this Step-1 and in the next section select the Default OAuth App option on OAth Connection Manager (This is the easiest option for now unless you want to use your own OAuth App).

Check this article for step-by-step instructions. When you follow these instructions make sure you enable Google Sheet API.

Create OAuth Connection Manager in SSIS

Once you create a Google API project and obtained a Client ID and Client Secret your next step is to create OAuth Connection Manager in SSIS. ZappySys OAuth connection manager comes with many predefined OAuth Providers (e.g. Facebook, Twitter, Google, etc) but you can also define custom OAuth settings for any OAuth enabled API.

To create SSIS OAuth 2.0 Connection for Google API perform the following steps.

1. Download and Install SSIS PowerPack.
2. Create a new SSIS Package.
3. Right-click in Connection Manager Area and Click “New Connection”.
4. When prompted select ZS-OAUTH connection type, On the OAuth Connection Manager Select Provider=Google.
5. Click here to create credentials. Then select the “Use Custom OAuth App” option and enter your ClientID, ClientSecret.
6. In the Scopes enter below scopes (or click Select Scopes). This will allow read/write access to Drive Files or Sheet (E.g. Reading file content / Export file to CSV). You can use the Scope browser to see many other available permissions. For more information click here.
   

   https://www.googleapis.com/auth/drive
   https://www.googleapis.com/auth/drive.file
   https://www.googleapis.com/auth/drive.readonly
   https://www.googleapis.com/auth/spreadsheets
   https://www.googleapis.com/auth/spreadsheets.readonly

7. Click on the Generate Token button.
   

   SSIS OAuth Connection Manager – Access Google Drive API using OAuth 2.0

8. You will see UI as below… Click Allow (You may have to scroll to see that button sometimes).
9. Click on Test Connection to see the connection is working and click on the OK button to save configure setting UI.

Find Google Sheet ID for API calls

Before we can write or update Sheet data in SSIS we need to know Google Driver File ID. We will use this ID in the next few sections.

The easiest way to find File ID is to look at the URL like below. If you need to know ID at runtime dynamically then refer to the last section.

Get Google Drive File ID for API call (Sheet ID)

Create Variables to make dynamic URL

1. Here, we are going to make a Dynamic URL value.
2. In Visual Studio Project Right Click on Design Panel, Select Variables.
   

   Create Variables

3. Let’s Create Variables with correct DataType and Value. In variable “varRange” set only the starting point(Sheet1!A2) of data if you don’t know the ending point and leave the first-row for giving header row later or click here.
   

Write SQL Server data to Google Spreadsheet

Now let’s look at a real-world example. How to write data from SQL Server table or other source and send to Google Sheet using Web API Destination.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Dragging and dropping Data Flow Task into Control Flow

2. Double click on the Data Flow task to see the Data Flow designer surface.
3. Here, In Visual Studio, drag and drop the OLE DB Source, JSON Generator Transform and Web API Destination in the design surface and join the components with the blue arrow.
   

   Drag and Drop OLE DB Source, JSON Generator Transform, and Web API Destination

4. Here, you can use our Free ZS Dummy Data Source too. Click here for more information.
5. But, we are going through OLE DB Source so we need OLEDB Connection.

Create OLE DB Connection

1. Lets, create the OLE DB Connection.
2. Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
   

   Create OLE DB Connection

3. Now, click on New Button to create Connection.
   

   Create New Connection

4. Let’s Configure Connection Manager, just Follow steps one by one as we have created.
   

   Configure OLE DB Connection

5. Click on Test Connection to see correct configure it.
6. Click on the OK button to save connection configure setting UI.

Configure JSON Generator Transform

1. Double click on JSON Generator Transform to configure it.
2. Set Output Mode to Single Dataset Array. Right-click on Mappings, select Add Static Element then enter Name: “range” and Select variable we have created for value.
   

   Configure JSON Generator Transform – Add Static Value

3. Again, Right-click on Mappings, select Add Static Element then enter Name: “majorDimension” and Value: “ROWS“.
   

   Configure JSON Generator Transform – Add Static Value

4. Now, Right-click on Mappings, select Add Document Array, Check to Treat as 2D array option and enter Element Name as “values“.
   

   Configure JSON Generator Transform – Add Document Array

5. Then, right-click on Document Array, select Add elements > Select Add Multiple Columns and check on columns you like to add.
   

   Configure JSON Generator Transform – Add Element

6. Click on the OK button to save JSON Generator Transform configure setting UI.

Configure Web API Destination

1. Double click on Web API Destination to configure it.
2. Select the OAuth connection manager created in an earlier section (Assuming you had all scopes with write permission).
3. In the Setting tab,
   1. In URL, Generate Dynamic URL like below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4:append?valueInputOption=USER_ENTERED

   2. Change Request Method to POST.
   3. Select JSON Generator variable into Body (Request Data) or you can select Input column for Body (JSON Generator Output).
   4. Set application/JSON to Body Content-type.
      

      Configure Web API Destination

4. Click on the OK button to save Web API Destination configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet. Now update header row values for click here.
   

   Run or Execute Package and verify data into Google Spreadsheet

Append data to Google SpreadSheet using REST API Task

Now let’s look at how to write to Google SpreadSheet. To write multiple rows in a single request. Also, check this link to learn more.

For more information, you can see the example here. For parameter, detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve the Spreadsheet write scenario. For a simple example, let’s use the REST API Task.

1. Drag and Drop ZS REST API Task on the control flow designer.
   

   Drag and Drop REST API Task

2. Double click on ZS REST API Task to configure it.
3. In the Request Settings tab,
   1. set Request URL Access Mode to Url from a connection. Select OAuth Connection from the dropdown (Same connection we created in the previous section).
   2. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A1:append?valueInputOption=USER_ENTERED

   3. Change Request Method to POST.
   4. Enter Request Body as below
      
      Note: For example purpose, we included header line in data cells but in the real-world header may already exist in the sheet.

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["Item", "Cost", "Stocked", "Ship Date"],
          ["Wheel", "$20.50", "4", "3/1/2016"],
          ["Door", "$15", "2", "3/15/2016"],
          ["Engine", "$100", "1", "3/20/2016"],
          ["Totals", "=SUM(B2:B4)", "=SUM(C2:C4)", "=MAX(D2:D4)"]
        ]
      }

   5. Change Request Content type to application/JSON.
      

      Write data to Google SpreadSheet using REST API Task (OAuth 2.0 Connection)

4. Click on the OK button to save REST API Task configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.
   

   Execute REST API Task and verify Google Spreadsheet data

Update Google Sheet Cells

Now let’s look at the below example of updating multiple cells in google Spreadsheet. For more information, you can see the example here. For parameter, detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve the Spreadsheet Update scenario. For a simple example, let’s use the REST API Task.

Update Google Sheet header cells manually

1. Drag and drop ZS REST API Tasks on the control flow designer.
2. Double click on the First REST API Task to configure it.
3. In the Request Settings tab,
   1. Enter the URL below. Change Range designator(including Sheet name: Sheet1!A1) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A1?valueInputOption=USER_ENTERED

   2. Change Request Method to PUT.
   3. Enter Request Body as below
      

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["CustomerID", "CustomerName"]
        ]
      }

4. Click on the OK button to save REST API Task configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.
   

Single Range (Set Values for multiple cells)

1. Follow the previous section(write data to Google Spreadsheet) to update sheet data.
2. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4?valueInputOption=USER_ENTERED

   2. Change Request Method to PUT.
   3. Enter Request Body as below
      

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["Laptop", "$2000.50"],
          ["Mobile", "$1150"]
        ]
      }

3. That’s all, click on the Test Request/Response button and verify data into Google spreadsheet.
   

   Calling Google Sheet API to Update Multiple Cell Values

Clear data to Google SpreadSheet using REST API Task

Clear a sheet of all values while preserving formats

Lets clear a sheet of all values by REST API Task, Before processed we need sheet Id. So let’s get the SheetId by API Call. For more information about click here.

1. Drag and Drop two ZS REST API Tasks on the control flow designer and rename(Get Sheet ID and Clear Google Sheet of all values) it.
2. Double click on the First REST API(Get Sheet ID) Task to configure it.
3. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc

   2. Change Request Method to GET.
      REST API Task – Get Google Sheet ID
4. Now go to Response Settings Tab, Select Response content Type Json and Write Response Content Filter Expression. Check on Save Response Content. Select Save Mode to Variable and Create a New variable or select it.
   Example of Response Content Filter Expression for the first Sheet : 
   

   $.sheets.[0].properties.sheetId

   

   REST API Task – Save Google Sheet ID into Variable

5. Now, click on the Test Request/Response to check configures are ok.
6. Click on the OK button to save REST API Task configure setting UI.
7. Double click on the second REST API(Clear Google Sheet of all values) Task to configure it.
8. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}:batchUpdate

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc:batchUpdate

   2. Change Request Method to POST. Enter Request Body as below and change a variable parameter as per your need.
      

      {
        "requests": [
          {
            "updateCells": {
              "range": {
                "sheetId": {{User::varGoogleSheetID}}
              },
              "fields": "userEnteredValue"
            }
          }
        ]
      }

   3. Change Request Content type to application/JSON.
      
      

      REST API Task Configure – Clear sheet of all values

9. Click on the OK button to save REST API Task configure setting UI.
10. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.

Clears range cell values from a spreadsheet

1. Double click on REST API Task to configure it. Then, follow the way just like the previous section configures REST API Task.
2. Here you need to set the value of Variable “varSpreadsheetValues” to “clear” or you can change it directly in URL append to clear and value of variable “varRange” (Change cell range and sheet name) as per your need. For more information about it click here.
   1. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4:clear?valueInputOption=USER_ENTERED

3. Change Request Method to POST.
4. Enter Request Body as below
   

   {}

5. Change Request Content type to application/JSON.
6. That’s all, click on the Test Request/Response button and verify data into Google spreadsheet.

Conclusion

Google Drive API and Google Sheets API provide a great way to automate file-related functionality. However, to call Google API  you have to use the SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Google API call via simple drag and drop approach without coding. Try out yourself see how long it takes to call virtually any REST API.

References

Finally, you can use the following links for more information:

• JSON Generator Transform
• Web API Destination
• REST API Task
• Help File:
  • JSON Generator Transform
  • Web API Destination
  • REST API Task

The post How to write/update Google Sheet in SSIS appeared first on ZappySys Blog.

In our previous article we saw how to read Google Analytics data using SSIS. But what if you don’t use SSIS and you still like to Import Google Analytics data into SQL Server or Read inside other Reporting / ETL Apps (e.g. Excel, Power BI, MS Access … and many more). Not all BI tools comes with native driver / connectivity for Google Analytics. Sometimes they do offer Google Analytics connectivity but don’t offer feature you looking for (e.g. Sort, Segment Filter). In such case you have to look for solution outside.

Well not to worry we will show you how simple it is to connect to Google Analytics using ZappySys JSON Driver (For REST API / Files) in few mins. This driver is one of the best REST API drivers you can find in the market. It is so generic that it can virtually access any REST API you may find internally or on public sites (e.g. Facebook, Salesforce, Azure, AWS). Check these articles to see many more use cases of JSON Driver.

At the end of this article you will learn how to Query Google Analytics data in SSMS like below (Yes that’s correct .. Pretty Awesome!!! ).

Import Google Analytics Data into SQL Server Table (using T-SQL Code)

Testing Google Analytics API

If you are new to Google Analytics API then read little bit about Google Analytics REST API here. Google also offers a really good way to test Analytics API using Query Explorer Tool here. So please check that and understand how to craft correct REST API URL which will be used in the next section.

Using Google Analytics Query Explorer Tool – Create REST API Url (Dimension / Metrics Selection Browser)

Sample Google Analytics API Response

Behind the scene here is how API Request and Response Looks like for Google Analytics REST API. Notice that Authorization Header in below request is automatically added by ZappySys Driver or Query Testing tool above. Also notice nextLink attribute in response, its used to fetch more data. By default each response contains upto 10000 rows.

Request

GET https://www.googleapis.com/analytics/v3/data/ga?ids=ga:185737326&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration&dimensions=ga:date
Authorization: Bearer ya29.GlwUxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Response

{
    "kind": "analytics#gaData",
    "id": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday",
    "query": {
        "start-date": "30daysAgo",
        "end-date": "yesterday",
        "ids": "ga:123456789",
        "dimensions": "ga:date",
        "metrics": ["ga:avgSessionDuration"],
        "start-index": 1,
        "max-results": 1000
    },
    "itemsPerPage": 1000,
    "totalResults": 30,
    "selfLink": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday",
    "nextLink": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday&start-index=10001&max-results=10000",
    "profileInfo": {
        "profileId": "123456789",
        "accountId": "33445566",
        "webPropertyId": "UA-33445566-3",
        "internalWebPropertyId": "454545454",
        "profileName": "All Web Site Data",
        "tableId": "ga:123456789"
    },
    "containsSampledData": false,
    "columnHeaders": [
        {
            "name": "ga:date",
            "columnType": "DIMENSION",
            "dataType": "STRING"
        },
        {
            "name": "ga:avgSessionDuration",
            "columnType": "METRIC",
            "dataType": "TIME"
        }
    ], "totalsForAllResults": {
        "ga:avgSessionDuration": "49.3779488037477"
    },
    "rows": [ 
		["20190425", "401.38912133891213"], 
		["20190426", "512.49723756906077"], 
		["20190427", "11.53488372093024"], 
		["20190428", "432.148648648648646"], 
		["20190429", "234.650735294117645"],
		.......
		.......
		.......
		.......
		["20190525", "74.123435294117645"]
	]
}

Creating Google API Project (Obtain Client ID / Secret)

Very first step to call any Google API including Google Analytics is to create Google API Project and register OAuth App to obtain Client ID and Client Secret. If you dont want to go through this hassle and start easy way then ZappySys offers Inbuilt Default App on OAuth Connection UI but we strongly recommend you create your own app rather than using Default App because in Default App API call limit is shared by many. Your data is never shared with ZappySys in any case even you use Default App.

Here is how to create API Project to call Google API. Once you create Google API Project make sure you enable Google Analytics API.

So once you have Client ID and Client Secret we can move forward to the next step.

Loading Google Analytics data into SQL Server

In our previous article we saw in depth general idea on how to import REST API data in SQL Server (T-SQL Script) without relying on any ETL tools.  Now lets look at step by step on how to load Google Analytics data into SQL Server.

In this example, our goal is to extract a Google Analytics report which shows average session duration and new users count by date (For last 30 days).

Setup ZappySys Data Gateway

Very first step to access any REST API Data inside SQL Server is to configure ZappySys Data Gateway. We covered this in previous article.  But here are high level steps for initial setup of Data Gateway.

1. Search for Gateway in Start menu and  Select ZappySys Data Gateway
   

   Open ZappySys Data Gateway

2. Create new User in Data gateway on Users tab. Enter username and password (we will use this when we create Linked Server) . Check Admin Option
   

   Add Data Gateway User

Setup Google Analytics API Data Source in Gateway / ODBC

Once gateway user is setup, now lets create a new Data Source for Google Analytics API. In this section we will talk how to create data source in gateway but most instructions can be used to create data source in ODBC too. At the end of this article we explained how to Launch ODBC Datasource UI. Step#1 and 2 are only different, all other steps same in ODBC DSN creation.

1. Click Add Data Source option (Select Native – JSON Driver )
   

   Add Gateway Data Source (Native JSON Driver)

2. Click Edit to configure data source
   

   Edit Gateway Data Source Settings

3. Now lets configure Driver settings.  Click on Load Connection String button.
   

   Load ZappySys Driver ConnectionString to configure UI

4. Enter the following ConnectionString to get started with predefined settings.
   Change ids (111223344 to your own Profile ID in DataPath URL). Other URL Parameters explained in next section.
   

   DRIVER={ZappySys JSON Driver};
   DataPath='https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date';
   DataConnectionType=OAuth;
   ScopeSeparator='{space}';
   Scope='https://www.googleapis.com/auth/analytics https://www.googleapis.com/auth/analytics.readonly';
   ServiceProvider=Google;
   Filter='$.rows[*]';
   ArrayTransformType=TransformSimpleTwoDimensionalArray;
   ArrayTransColumnNameFilter='$.columnHeaders[*].name';
   RequestMethod='GET';
   NextUrlAttributeOrExpr='$.nextLink';
   ClientId='xxxxxxxxxxxx';
   ClientSecret='yyyyyyyyyyyyyy';
   UseCustomApp='True'

    
5. URL used in DataPath is most important (parameters explained below) . You can use   Query Explorer Tool here to build same URL.
   

   https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date

   Parameters Explained 
   
   Here is some required parameters used in above REST API URL.
   ids=ga:11223344
   This is your Profile ID which you like to extract.  Read more
   start-date=30daysAgo
   This can be yyyy-MM-dd or some other some predefined date keywords (e.g today, yesterday 10daysAgo). Read more.
   end-date=yesterday
   Same as above
   metrics=ga:avgSessionDuration,ga:newUsers
   This can be comma separated list of metrics you like to extract (max 10). Read more.
   dimensions=ga:date
   This can be comma separated list of metrics you like to extract (max 7). Read more.
6. Now its time to configure OAuth settings. Click on Configure Connection. Enter Client ID and Client Secret Obtained in the previous section (see here) and then click Generate Token as below.
   

   Configure Google API Data Source / DSN (Google Analytics API Example)

7. Thats it now can click on Preview Tab and run sample query like below.
   

   select * from $

   –OR–
   (Select Table name from dropdown and generate default query and then you can add alias , remove unwanted columns)
   

   SELECT 
   "ga:date" date, 
   "ga:avgSessionDuration" avgSessionDuration,
   "ga:newUsers" newUsers
   FROM "rows"

   –OR–
   (Type custom query with your own URL generated from here, Change Profile ID – from 11223344 to your own id)
   

   SELECT * FROM $
   WITH(
   	 Src='https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date'
   	,IncludeParentColumns='False'
   )

   

Create Linked Server in SQL Server using T-SQL Script

Once our Gateway Data Source is configured we can now move to SQL Server Part to define Linked Server.

Here is how you can define linked server to access Google Analytics Data inside SQL Server. There are two ways to create Linked Server (1)  via T-SQL script (2) via SSMS UI)

Here is how to create using Script.

USE [master]
GO

--drop existing linked server for same name if already found
EXEC master.dbo.sp_dropserver @server=N'GOOGLE_ANALYTICS_LS', @droplogins='droplogins'
GO

EXEC master.dbo.sp_addlinkedserver 
	@server = N'GOOGLE_ANALYTICS_LS', 
	@srvproduct=N'', 
	@provider=N'SQLNCLI', 
	@datasrc=N'localhost,5000', --this is host name and port number where  ZappySys Data Gateway is running
	@catalog=N'ZS-GA' --this must be same name as defined on Data Source tab Grid on ZappySys Data Gateway

EXEC master.dbo.sp_addlinkedsrvlogin 
	@rmtsrvname=N'GOOGLE_ANALYTICS_LS',
	@useself=N'False',
	@locallogin=NULL,
	@rmtuser=N'SOME-GATEWAY-USER', -- user name created on ZappySys Data Gateway
	@rmtpassword='SOME-GATEWAY-PASSWORD' -- password for ZappySys Data Gateway user
GO

Create Microsoft SQL Server Linked Server using SSMS UI

Import Google Analytics Data into SQL Server Table (T-SQL Code)

Now lets see how to run sample query to import Google Analytics Data into SQL Server. Notice that how we have escaped quotes in OPENQUERY.
NOTE: Change id 11223344 with your own Id (use this tool to create URL)

select * into GaApiLoad from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date''
  ,IncludeParentColumns=''False''
 )
')
select * from GaApiLoad

Import Google Analytics Data into SQL Server Table (using T-SQL Code)

Expose Google Analytics Data as View

To make it simple you can expose above query as SQL Server views so you can access them from any Reporting / ETL Tools.

CREATE VIEW vw_Google_Analytics_BounceRate_PageViews_Last30Days
AS
--change ids to your own Profile ID
select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=2019-04-01&end-date=2019-04-30&metrics=ga:bounceRate,ga:uniquePageviews&dimensions=ga:date''
  ,IncludeParentColumns=''False''
 )
')
GO

Making things Dynamic using Stored Procedure with Parameters

Now let’s make few things dynamic. We will change above query so we can pass Id, Dimensions and Metrics as parameters.

Run below script to Create a stored proc and fetch google analytics data in SQL Server.

use MyDatabase
go
create proc usp_GoogleAnalyticsData
	@ids varchar(1000),
	@startDate varchar(50)='30daysAgo',
	@endDate varchar(50)='yesterday',
	@metrics varchar(200)='ga:avgSessionDuration,ga:newUsers',
	@dimensions varchar(200)='ga:date'	
as

declare @sql varchar(4000)

set @sql='select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
''SELECT * FROM "rows"
WITH 
 (
   SRC=''''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:' + @ids 
   +'&start-date='+ @startDate 
   +'&end-date='+ @endDate 
   +'&metrics='+ @metrics 
   +'&dimensions='+ @dimensions 
   +'''''
  ,IncludeParentColumns=''''False''''
 )
'')'

print @sql
execute(@sql)

go

--Example: query google analytics data
exec usp_GoogleAnalyticsData 
	@ids='11223344',  --Change this ID - Obtain from here https://ga-dev-tools.appspot.com/query-explorer/
	@startDate ='30daysAgo',  --or like this 2019-04-01
	@endDate   ='yesterday', --or like this 2019-04-30
	@dimensions='ga:date', 
	@metrics   ='ga:bounceRate,ga:uniquePageviews'

Insert Google Analytics Data into table (Save Stored Procedure Output)

Now what if you like to save Stored Proc output into a table? Well for that you must create table with same structure as stored proc output and then use INSERT INTO sql like below.

--import google analytics data in a table
if(object_id('GaDataLoad') is not null)
	drop table GaDataLoad
create table GaDataLoad([Date] date, BounceRate numeric(10,5), PageViews numeric(10,5))  

insert into GaDataLoad
exec usp_GoogleAnalyticsData 
	@ids='11223344',  --Change this ID - Obtain from here https://ga-dev-tools.appspot.com/query-explorer/
	@startDate ='30daysAgo',  --or like this 2019-04-01
	@endDate   ='yesterday', --or like this 2019-04-30
	@dimensions='ga:date', 
	@metrics='ga:bounceRate,ga:uniquePageviews'

select * from GaDataLoad

Performance Tips / Handling data errors due to missing rows

By default ZappySys API Drivers sends minimum 2 API requests. First one to fetch metadata and second, third… for Data. There will be a time when you wont have any data and it may throw error about no records found because it fails to parse metadata. You can avoid such issue by supplying metadata before hand so you can avoid expensive API calls. Check this article see how you can supply cached metadata in your SQL Query as below. This will avoid Metadata call and speedup significantly. It will also avoid errors due to no rows found for specified criteria.

select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:bounceRate,ga:uniquePageviews&dimensions=ga:date''
  ,IncludeParentColumns=''False''
  ,Meta=''[ 
     {"Name": "ga:date","Type": "String", "Length": 16 },
     {"Name": "ga:bounceRate", "Type": "String", "Length": 36 },
     {"Name": "ga:uniquePageviews", "Type": "String", "Length": 10 }
    ]''
 )
')

ODBC Connection – Google Analytics Data in Other Apps (e.g. SSRS / Power BI / Excel / Tableau)

So far we have talked accessing data inside SQL Server using Data Gateway Approach but what if you like to access in other apps ? Well you have three options.

1. Use Microsoft SQL Server Driver to call Linked Server Queries (OPENQUERY approach we saw earlier)
2. Access Google Analytics Data using ODBC connectivity
3. Use Microsoft SQL Server Driver to send direct SQL queries to Gateway (Bypass Linked Server – Use inner SQL query without OPENQUERY in this case)

Method-1 : Linked Server Approach

First approach most likely works in all cases because most apps will support connecting to SQL Server using OLEDB / ADO.net / ODBC / JDBC Drivers. So calling SQL Queries which uses Linked Server (i.e. OPENQUERY statement)  is advisable.

Method-2 : ODBC Driver Approach

However in some cases this may not be possible (e.g. You don’t have SQL Server inhouse or you don’t want to rely on SQL Server to access API). In such case you can use ODBC connectivity in your app. Since ODBC is widely adopted standard most app out there (Except JAVA apps) should support ODBC Drivers. If you like to use this approach then create ODBC DSN rather than Data Gateway Data source and use it in your Reporting / ETL / Custom Apps.

Open ODBC Data Sources (Create DSN)

ODBC User DSN Tab: Add new Driver Screen

Method-3 : Direct connection to Data Gateway (By pass Linked Server)

Last option we suggest for cases like JAVA Apps / Linux / Mac machines where you cannot install ZappySys ODBC Drivers and you dont have option to use SQL Server Linked Server either. In such case you can try to send SQL Queries to Gateway directly using Microsoft SQL Server Compatible Drivers (i.e. ODBC /JDBC). See this example how we called API Queries inside JAVA Apps.

Google Analytics / REST API Integration in various apps

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

In this article we explored many ideas of JSON / API integration in SQL Server.  We saw how to create OAuth App for Google API and import Google Analytics data into SQL Server Table without doing any ETL using pure T-SQL code (Query / Views /Stored Procs). You can Download  FREE Trial of ODBC PowerPack and try yourself see how easy it is to query any API inside SQL Server and avoid expensive ETL processes.

The post Import Google Analytics data into SQL Server / Reporting / ETL appeared first on ZappySys Blog.

Connect to YouTube API in SSIS (OAuth Connection)

Very first step to call any Google API is to create an API project for Google Service and then enable APIs you like to call in Console. In our case we need to enable YouTube API.

Check this article to learn how to create API Project and obtain Client ID and Client Secret to connect to YouTube API. For ease of use use Default Application provided by ZappySys but we still recommend using your own Custom Application  on OAuth Connection.

Before we try our first YouTube API example, we recommend you to get familiar with YouTube API here. Once you done with above steps you can perform the following steps to call YouTube API in SSIS.

1. Open existing SSIS Project or Create new One in Visual Studio (i.e. SSDT)
2. Create a new SSIS Package. Right click in the Connection Manager Panel and click “New Connection…”.
   
3. When Dialog box opens select ZS-OAUTH connection type.
   

   SSIS Connection Managers List: Select O-AUTH Connection Manager

4. Once OAuth UI opens, Configure Google connection like below.
   1. Select Service Type as Google
   2. You can select Default App for ease of use or Select Custom Application (Enter Client Id and Secret Obtained using these steps )
   3. In the Scope Enter below – One scope per line for calling correct YouTube API. Refer to YouTube API documentation incase you need extra permissions. For read only operation we just need below scopes. Last scope is only needed if you need to use YouTube Analytics Reports
      

      https://www.googleapis.com/auth/youtube
      https://www.googleapis.com/auth/youtube.readonly
      https://www.googleapis.com/auth/yt-analytics.readonly

5. Click Generate Token to obtain Refresh Token (You may see Login Prompt, and then Grant Permission Confirmation Screen)
   

   Connect to Google API using OAuth 2.0 (YouTube API Example)

6. Click OK to save UI.

Call YouTube REST API Example

Now lets look at how to call YouTube API REST API Task this task is more suitable to call GET, POST, DELETE, PUT requests without parsing data in Rows and Columns. If you need to parse data and load into Database table then see next section (Use JSON Source)

1. From SSIS Toolbox look for items starting with “ZS”. Drag and Drop [ZS Rest API Task] to Designer Surface.
   
2. Double click it to configure like below.
   1. Select URL From Connection option
   2. Select OAuth connection created in previous section.
   3. Enter URL like below (For example get all playlists). Replace xxxxxxxxxxx with your own channel ID (Its usually found in URL when you visit Youtube Channel Homepage)
      

      https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&channelId=xxxxxxxxxxxxx

   4. Click Test Request Button
3. Here is how it looks like
   

   Call Youtube API in SSIS Example (Dynamic URL)

That’s it we have successfully configured Connection for YouTube API in SSIS. In the next section we will see how to use this connection and read various data from YouTube API.

Read YouTube Playlists in SSIS

Once we done creating OAuth Connection Manager we can move forward to read YouTube data inside Data Flow.

Configure SSIS JSON / REST API Source

1. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
2. Double click on the Data Flow task to see Data Flow designer surface.
3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   
4. Double click JSON Source and enter URL as below (Use variable or Hardcode Channel Id). maxResults controls how many rows you want to get in each response. Rep
   

   https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&maxResults=50&channelId={{User::varChannelId}}
   --OR-- 
   https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&maxResults=50&channelId=xxxxxxxxxxxxxxxxxxxxxx

5. Check Use Credentials and select existing OAuth connection or Create New one
6. Select Array Filter (Items node) or type

   $.items[*]

     as below.
   

   Get Playlists for YouTube Channel

7. That’s it you can now Click Preview to see sample data. In the next section we will configure Pagination to read all records if you have more than 100 rows.

Configure YouTube REST API Pagination

Most REST APIs dont return data in one big response. So you have to make sure you implement looping / pagination to read more records. Luckily ZappySys supports many different Pagination Settings. Here is how to configure YouTube REST API Pagination.

1. Go to Pagination Tab found on JSON Source and select following 3 settings
2. Select Pagination Mode as

   Response Attribute Mode

3. Enter Next Link as

   $.nextPageToken

4. Enter Suffix for Next URL as

   &pageToken=<%nextlink%>

YouTube REST API Pagination Settings

Read YouTube Playlist Videos in SSIS

So in previous section we saw simple API call to read all playlists from YouTube. Now lets see how to get information about Playlist Videos.

For that everything should remain same as previous section except change URL as below. Where xxxxxxxxxxxxxxxxxxxxx is your Playlist ID obtained from Previous Call.

https://www.googleapis.com/youtube/v3/playlistItems?part=snippet,id&maxResults=50&playlistId=xxxxxxxxxxxxxxxxxxxxx

Get all Videos for YouTube Channel

If you want to get all video name and ID for specific channel then use below URL. Click here to read more. Replace xxxxxxxxxxxxxxxxxxx with your Channel ID.

https://www.googleapis.com/youtube/v3/search?channelId=xxxxxxxxxxxxxxxxxxx&part=snippet,id&order=date&maxResults=50

Read YouTube Video Views, Likes, Dislikes (Extended Videos)

Another common request from YouTube API is get Video Information such as Views, Likes, Dislikes, Comment Count etc. For that you can use this api . However the issue is this endpoint requires you to pass Comma separated Ids of Video. You can only pass 25 Ids in one requests so if you have 200 videos for which you need to get information then you have to make 8 requests (25 Ids in each Request). In below screenshot you can see we used Fiddler to debug web requests.

So here is how high level process to fetch extended information for all Videos in your channel. Full SSIS Package is attached here (SSIS 2012 Format).

1. Get all videos for your channel by calling (Assuming you will paginate in below request.
   

   https://www.googleapis.com/youtube/v3/search?channelId=xxxxxxxxxxxxxxxxxxx&part=snippet,id&order=date&maxResults=50

2. Build Video Id groups for 25 Ids from above result and submit requests. For example if above request returns 200 Videos then your request to fetch extended information may look like below.
   

   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID1,ID2,ID3........ID25
   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID26,ID27,ID28........ID50
   ........
   ........
   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID176,ID177,ID178........ID200

3. Parse response coming from each Request Above…. Each Response will give you details about 25 videos you requesting.

Here is the flow of our complete data flow to read extended information for all videos. You can read extended information one by one too but it wont be good idea if you have many videos.

We have used Following Components to achieve Bulk Mode

• JSON Source (REST API Source)
• CSV Parser Transform
• Web API Destination
• JSON Parser Transform

Get YouTube Video View Clicks, Likes, Dislikes (Bulk Mode for all videos)

Read data from YouTube Analytics / Reporting API (Metrics, Dimensions Report)

So far we have seen basic YouTube APIs to extract your Channel information. However there is another very powerful API endpoint to query custom reports from YouTube. Check this YouTube Analytics API endpoint. In this section we will learn how to extract useful information from YouTube using Analytics API.

Step-By-Step

Let’s look at an example to read total views, estimated watch time, average view duration and some other metrics by date for channel you have access.

1. Go to Data Flow, Drag ZS JSON Source from SSIS Toolbox
2. Double click to configure.
3. Check Use Credentials. Select OAuth connection we created in previous section.
4. In the URL field, Enter Below Sample URL. Change Parameters as per your need or leave it default.
   For example:
   ids=channel==MINE   (where MINE can be replaced by other channel id or keep it as MINE to get data for your own channel)
   metrics (can be changed from any valid list here)
   dimensions (can be changed to any valid list from here)
   startDate (Must be YYYY-MM-DD format)
   endDate (Must be YYYY-MM-DD format)
   

   https://youtubeanalytics.googleapis.com/v2/reports?ids=channel==MINE&metrics=views,estimatedMinutesWatched,averageViewDuration,averageViewPercentage,subscribersGained&dimensions=day&startDate=2019-03-01&endDate=2019-04-01

5. Change Filter to

   $.rows[*]

6. Go to 2D Array Transform tab.
   Select Method as

   Simple 2-dimensional array

   EnterColumn Filter as 

   $.columnHeaders[*].name

   

   YouTube Analytics API Advanced Transformation

7. That’s it now Preview your data. See below over all steps.
   

   Read data from YouTube Analytics API (Custom Reports using Dimensions, Metrics Query)

Build YouTube Analytics / Reporting API Query (API Playground)

With Analytics API You can extract many interesting data. You can use Test Tool found on this page to build some interesting queries. Scroll to Common Use Cases section on that link. On the right side you can select Credentials Type as OAuth and click Execute.

https://developers.google.com/youtube/analytics/reference/reports/query

YouTube Analytics / Reporting API Explorer (Testing API Calls)

Loading YouTube API Data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Download Sample SSIS Package

SSIS Package is attached here (SSIS 2012 Format).

Conclusion

In this article we saw how to extract information from REST API such as YouTube REST API using OAuth 2.0. We also learned techniques like Pagination, JSON Parsing, Request Batching using multiple components to achieve full scale API integration. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

The post How to read YouTube API data in SSIS (Videos, Channels, Playlists) appeared first on ZappySys Blog.

In this tutorial, we will cover the topics of how to perform geocoding on the addresses and reverse geocoding on the location coordinates using SSIS and Google Maps API. So what is geocoding, exactly? Geocoding is the process of translating an address (e.g. a street address) or a place to coordinates on the Earth’s surface. Simply put, geocoding takes a description of a location and gives back latitude and longitude values as a result. For example, 

1600 Amphitheatre Parkway, Mountain View, CA 94043, USA

(37.4224764, -122.0842499)

(37.4224764, -122.0842499)

1600 Amphitheatre Parkway, Mountain View, CA 94043, USA

To accomplish our goal we will use an add-on for SSIS – ZappySys SSIS PowerPack. It contains many additional SSIS Tasks and Data Flow Components that lets you accomplish what you cannot do with standard SSIS connectors. We will use these ZappySys SSIS PowerPack connectors:

Web API Destination

JSON Parser Transform

Let’s start!

Step-by-Step – SSIS Geocoding with Google Maps API

Before we start, let’s consider the use-case for our tutorial. Suppose, you have a CRM system filled with user-entered addresses of your leads. Since they are user-entered, it means that they can be incomplete, malformed, or just plain rubbish. Also, you have a marketing campaign and you want to send letters with offers to your leads so that they become your clients. Before sending them any letters, you have to make sure the addresses are valid and well-formed. You are also interested in the geographical location of your leads that you could do some business intelligence analysis on the coordinates of your leads and maybe future clientele. Here is where Geocoding API kicks in.

Thus in this example, we will take various street addresses – sometimes invalid, incomplete or unformatted – and submit them to Google Geocoding API for processing. As a result, we will get nicely formatted, cleansed, full addresses and their coordinates. We will take a look on how to determine if lookup worked successfully and to what extent you should trust Geocoding API’s response. Finally, we will save retrieved addresses and their individual components (e.g. street number, street name, city, country, etc.) to a SQL Server database table.

Load addresses from a SQL Server database table

We prepared the Google Geocoding API part, now we are ready to get to the place where the rubber meets the road – we will create an SSIS package and load addresses from a database table:

1. Create a new SSIS package.
2. Drag and drop Data Flow from SSIS Toolbox onto the Control Flow.
   

   Dragging and dropping Data Flow Task into Control Flow

3. Drag and drop OLE DB Source onto the Data Flow.
4. Configure OLE DB Source to retrieve addresses from your table. We will use a table from Microsoft’s Northwind’s modified database “Marketing” (download it):

Loading user-entered addresses to be looked up in Google Geocoding API using SSIS.

Construct lookup URL

1. Drag and drop Derived Column component onto the Data Flow.
2. Add a new column and construct Geocoding lookup URL in this format (just like it says in Geocoding API documentation):

https://maps.googleapis.com/maps/api/geocode/json?address=ADDRESS_TO_LOOKUP&key=YOUR_API_KEY

Adding Google Geocoding API lookup URL as Derived Column to be used for geocoding in SSIS.

Configure the Geocoding API request

1. Drag and drop Web API Destination onto the Data Flow and connect it with Derived Column. We will use Web API Destination to make HTTP requests to Geocoding API.
2. Then double-click on Web API Destination and create a new HTTP connection:
   

   Setting up HTTP Connection Manager for Web API Destination to make HTTP requests to Google Geocoding API using SSIS.

3. Finally, configure Input Column for URL, HTTP Request Method and input into URL field a sample lookup URL:
   

   https://maps.googleapis.com/maps/api/geocode/json?address=One Microsoft Way Redmond 98052-6399 USA&key=YOUR_API_KEY

   

   Configuring Web API Destination to make geocoding lookups in Google Geocoding API using SSIS.

   Once you are done, click Test Request/Response button. This will actually make a geocoding lookup for Microsoft headquarters:

4. In the Response (Raw) tab you see the actual response from Geocoding API. Within it –  cleansed, formatted and full Microsoft headquarters address together with geographical coordinates (even we looked it up using just the part of it). Now copy-paste the response body somewhere, so that we can use it in the next step while parsing:
   

   Checking Google Geocoding API HTTP response in Web API Destination using SSIS.

Parsing Geocoding API JSON response into columns

1. Drag and drop JSON Parser Transform onto the Data Flow and connect it to Web API Destination.
2. Then open it and configure Input JSON Column.
3. After that, set the Filter to $.results[*].
4. Continue by setting Sample JSON string to the one you saved in the previous step. Actually, before doing that let’s modify it and add partial_match node next to place_id so that exact and partial matches could be told apart (otherwise this property won’t be parsed):
   

   {
       "results": [{
           "address_components": [{
               "long_name": "One",
               "short_name": "One",
               "types": ["street_number"]
           }, {
               "long_name": "Microsoft Way",
               "short_name": "Microsoft Way",
               "types": ["route"]
           }, {
               "long_name": "Overlake",
               "short_name": "Overlake",
               "types": ["neighborhood", "political"]
           }, {
               "long_name": "Redmond",
               "short_name": "Redmond",
               "types": ["locality", "political"]
           }, {
               "long_name": "King County",
               "short_name": "King County",
               "types": ["administrative_area_level_2", "political"]
           }, {
               "long_name": "Washington",
               "short_name": "WA",
               "types": ["administrative_area_level_1", "political"]
           }, {
               "long_name": "United States",
               "short_name": "US",
               "types": ["country", "political"]
           }, {
               "long_name": "98052",
               "short_name": "98052",
               "types": ["postal_code"]
           }],
           "formatted_address": "One Microsoft Way, Redmond, WA 98052, USA",
           "geometry": {
               "location": {
                   "lat": 47.6423318,
                   "lng": -122.1369302
               },
               "location_type": "ROOFTOP",
               "viewport": {
                   "northeast": {
                       "lat": 47.6436807802915,
                       "lng": -122.1355812197085
                   },
                   "southwest": {
                       "lat": 47.6409828197085,
                       "lng": -122.1382791802915
                   }
               }
           },
           "partial_match": false,
           "place_id": "ChIJGxUEuHFtkFQRnokD7mkppQE",
           "plus_code": {
               "compound_code": "JVR7+W6 Redmond, Washington, United States",
               "global_code": "84VVJVR7+W6"
           },
           "types": ["establishment", "point_of_interest"]
       }],
       "status": "OK"
   }

   

   Configuring JSON Parser Transform to parse looked up addresses in Google Geocoding API HTTP JSON response using SSIS.

5. To proceed and parse individual address components from JSON response (e.g. number, street, city, country, etc.), you will need to specify columns for each address component manually, just like portrayed in Parse multi-dimensional JSON array in SSIS or Drivers article. Just open 2D Array Transform tab and select Multiple columns using expressions as Transform Type and configure them:
   

   Configuring JSON Parser Transform to parse individual address components from Google Geocoding API HTTP JSON response using SSIS.

   We specify these columns in our example:

   Number = $.address_components[?(@.types[0]=='street_number')].long_name
   Street = $.address_components[?(@.types[0]=='route')].long_name
   City = $.address_components[?(@.types[0]=='locality')].long_name
   State = $.address_components[?(@.types[0]=='administrative_area_level_1')].long_name
   Country = $.address_components[?(@.types[0]=='country')].long_name
   PostalCode = $.address_components[?(@.types[0]=='postal_code')].long_name

6. Finally, be sure to set Include all upstream columns to downstream (quite important), so that CustomerID is passed downstream and which we can save into a destination table. Later it can be a useful column to join the source table and the destination table on. In that way, you will be able to map user-entered addresses with looked up, fully formatted addresses (won’t be demonstrated in this tutorial).

Save geocoding results into a SQL Server database table

1. We are now ready to add OLE DB Destination, execute the package, make the lookups, and save the results into the database:
   

   Creating a database table for the Google Geocoding API looked up addresses in SSIS.

2. Execute the package!
   

   SSIS geocoding package execution and its results.

The results of SSIS geocoding

Viewing the results of SSIS geocoding in SQL Server destination table.

Once you query the table of looked up results, you will notice several things:

1. Perhaps there are more rows in the results table than in the source table
2. partial_match column is either 1 or NULL
3. geometry.location_type column has several values, e.g. ROOFTOP, RANGE_INTERPOLATED, APPROXIMATE, etc.

So to what extent should you trust the results? The answer is to take a look at the documentation and see what these values mean.

Basically, when partial_match is equal to 1, according to Google:

• Partial matches most often occur for street addresses that do not exist within the locality you pass in the request.
• Partial matches may also be returned when a request matches two or more locations in the same locality. For example, “21 Henr St, Bristol, UK” will return a partial match for both Henry Street and Henrietta Street. Note that if a request includes a misspelled address component, the geocoding service may suggest an alternative address. Suggestions triggered in this way will also be marked as a partial match.

While geometry.location_type column tells how accurate the lookup was. Again, according to Google:

• “ROOFTOP” indicates that the returned result is a precise geocode for which we have location information accurate down to street address precision.
• “RANGE_INTERPOLATED” indicates that the returned result reflects an approximation (usually on a road) interpolated between two precise points (such as intersections). Interpolated results are generally returned when rooftop geocodes are unavailable for a street address.
• “GEOMETRIC_CENTER” indicates that the returned result is the geometric center of a result such as a polyline (for example, a street) or polygon (region).
• “APPROXIMATE” indicates that the returned result is approximate.

Simply put, “ROOFTOP” is the most accurate lookup, while “APPROXIMATE” is the least accurate.

Another valuable piece of information is how many individual address components are filled. The more NULLs are there, the less accurate is the address. So if you have NULL in the Number column and in the Street column, most likely this is not a very accurate lookup:

Individual address components of looked up addresses in Google Geocoding API using SSIS.

Concluding, in the figure above, you can see how user-entered address can be validated, cleansed and formatted.

SSIS reverse geocoding with Google Maps API

The concepts presented above about “forward” geocoding will help you to make reverse geocoding requests too. The only difference is the request URL:

https://maps.googleapis.com/maps/api/geocode/json?latlng=40.714224,-73.961452&key=YOUR_API_KEY

For reference and help visit official Google Geocoding API documentation.

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Common errors

Parameterization and deployment to production 

Download a sample package

Be sure to download a sample SQL Server 2012 SSIS package, in case you want to try it right away (you can upgrade it to a higher version). If you want to use the database that was used in this tutorial, don’t forget to create the Marketing database from the script (a modified Microsoft’s Northwind database).

Conclusion

In this tutorial, we step-by-step covered how to make geocoding lookups using Geocoding API, part of Google Maps API. We learned how to cleanse, validate and format address just by submitting it to Geocoding API and parsing the results. We moved forward and looked at how to interpret the results and what geocoding results to trust. Then we touched the point that the same process could be applied to reverse geocoding. Concluding, we took a look at how to handle common errors and how to parameterize an SSIS package and deploy it to production.

The post SSIS Geocoding with Google Maps API appeared first on ZappySys Blog.

Google BigQuery is a fully managed Big Data platform to run queries against large scale data. In this article you will learn how to integrate Google BigQuery data into Microsoft SQL Server using SSIS. We will leverage highly flexible JSON based REST API Connector and OAuth Connection to import / export data from Google BigQuery API just in a few clicks.

If you are looking for a similar product inside Amazon AWS Cloud then check an article about Amazon Athena.

Prerequisites

Before we look into a Step-By-Step section to extract and load data from BigQuery to SQL Server let’s make sure you meet the following requirements:

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have basic familiarity with REST API concepts and Google BigQuery API. This post uses free Public Dataset to query BigQuery table so no billing is required to get your hands dirty 🙂 However, if you like to query your own dataset then make sure you have at least one BigQuery dataset and one table with some sample data created (this part does require billing enabled). Read a Google Quickstart article for more information on how to create a new BigQuery dataset and a table.

Understanding Google BigQuery Object Heirarchy

Google BigQuery has 3 main concepts below.

• Project
  • Dataset
    • Table
      • Query requests (each query creates a unique JobID – valid for 24 hours from which you can read data)

So in order to create BigQuery table you always need Project and then Dataset under that project. You can group objects around Project / Datasets. When you query you can supply fully qualified name of your table in FROM clause (e.g.

select count(*) from `bigquery-public-data.usa_names.usa_1910_2013`

So let’s get started.

Read data from Google BigQuery using SSIS

Basically you can query Google BigQuery data in two ways: In this article we will not cover 2nd method.

• Method-1: Query data using jobs/query method in BigQuery API. Use it if you expect to get a result in a fairly short amount of time. This API method generates a temp table which gets deleted after 24 hours. You can read data within that time frame using newly created JobId reference.
  
  NOTE: If you have records that sum up to more than 10MB of data or ~10,000 rows (assuming 1 row uses 1KB of data) then you need to proceed with two-step process as explained below.
• Method-2: Export SQL query result using jobs/insert method in BigQuery API. Use this method if you expect a query to take a long time to finish.

Now let’s see how to query Google BigQuery data using SSIS 🙂

Create Google API Project and Register OAuth App

To use any Google API firstly you have to finish two tasks:

1. Create Google API Project and obtain projectId (see next section).
2. Enable BigQuery API
3. Register your own Google OAuth App and obtain ClientId and Client Secret.

Check step-by-step instructions on how to create Google API Project and create OAuth App for Google. During instructions make sure you enable BigQuery API (screenshots may differ).

Get Google API Project ID

Once you have Google API Project created you can grab Project ID (see screenshot below). You will need Project ID in the next step when building API URL.

How to find Google API Project ID?

Configure OAuth Connection for Google BigQuery API

In order to call most of Google APIs you will need an OAuth App. If you want to use your own app then refer to how to register Google OAuth Application (Get ClientID and ClientSecret) article. For simplicity we will use Default OAuth App:

1. Right click inside Connection Managers area and click “New Connection…”
2. From the connection type list select “ZS-OAUTH” connection type.
   

   Create new SSIS OAuth API Connection Manager

3. For OAuth Provider select “Google”.
4. For OAuth App type options leave “Use Default OAuth App” selected. If you created custom OAuth App then you would need to select “Use Custom OAuth App” option.
5. For Scopes enter or select the following URLs (URL is just a permission name). Each URL must be in a separate line. For demo purposes we use only the first 3 scopes but we included a few more in case you like to test API in depth with a different permission set:
   

   https://www.googleapis.com/auth/bigquery
   https://www.googleapis.com/auth/cloud-platform
   https://www.googleapis.com/auth/cloud-platform.read-only
   https://www.googleapis.com/auth/bigquery.insertdata
   https://www.googleapis.com/auth/devstorage.full_control
   https://www.googleapis.com/auth/devstorage.read_only
   https://www.googleapis.com/auth/devstorage.read_write

6. Click Generate Token. Login using correct account if needed and then you will be prompted to click “Approve” OAuth App.
7. Once you click OK on the login screen you will see new tokens populated on the connection screen.
8. Click Test to make sure connection / tokens are valid and then click OK to save the connection.
   NOTE: Below screenshot uses Default App but we recommend you to to use Custom OAuth App (Your own Clientid / secret – Obtained in previous section)
   

   Create SSIS OAuth API Connection for Google BigQuery API

Start BigQuery Job and get JobId (Submit Query)

Once you have SSIS OAuth connection created for BigQuery API it’s time to read data from BigQuery. So, basically, there are two ways you can read BigQuery data: using query or insert method. For demo purposes we will use jobs/query method. If you want fire complex queries which can run for many minutes then refer to jobs/insert method.

If you are expecting more than 10MB of data (~10K records) then you have to split data extract in two steps:

1. Step-1 : Call jobs/query method using REST API Task to submit SQL query. This steps returns jobComplete: true if supplied query execution is done within timeoutMs parameter you included. By default its 10 seconds so if your query is going to take longer than 10 seconds then its good idea to add bigger timeout that way you dont need step#2. Once API call is done you will Get JobId in the response. Save that JobId in a variable for later use. This job result is valid for 24-hours only.
2. Step-2 (Optional) – You can add another optional REST API Task after previous step to Wait until Job is completed. For simplicity, this article will Omit setting up Status Check but its very simple….Use below settings
   1. Drag new REST API Task from toolbox. Connect it to Step-1 and double click to configure as below.
   2. Select URL From Connection mode,
   3. Select OAuth connection
   4. Enter URL as https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/jobs/{{User::JobId}} , Method: GET
   5. On Response Settings Tab, Select Response Content Type JSON and for content filter enter $.status.state
   6. On Status Check Tab, check Enable Status Check Loop and enter DONE in SuccessValue field
   7. Doing this setup will make sure we do not query data until Job Status is DONE (System keeps checking every 5 seconds)
3. Step-2 : Read Job result using JSON Source in a Data Flow Task.

Now lets see how to configure REST API Task to submit query and extract JobId (which will be used in next section).

1. In the Control Flow designer drag and drop ZS REST API Task from the SSIS toolbox.
2. Double click the task to configure it.
3. Change Request URL Access Mode to [Url from connection].
4. From the connection dropdown select OAuth connection created in the previous section.
5. Enter URL as below (replace YOUR-PROJECT-ID with a valid Project ID obtained in the previous section):
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-PROJECT-ID/queries

6. Select Method as POST.
7. Enter Body (Request Data) as below (change your query if needed; we used Public dataset for the demo): If you do not specify timeout then default is 10 seconds only. Also in this call we only care about JobID from response so we just added maxResults=10 because in 2nd step we will get all rows by doing pagination.
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000"
   }

   To use a Standard SQL query instead of Legacy SQL, add useLegacySql property and set it to false:
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000",
      "useLegacySql": false
   }

   For all possible parameters refer to jobs/query method documentation.
   NOTE: You can also supply location Parameter in the Body JSON to indicate where job should run. For non EU / US datacenters we suggest you to supply this parameter. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location.
   Example Use Of Location
   

   { "location":"us-east1",  "maxResults": 10, "query": "SELECT title FROM [bigquery-public-data:samples.wikipedia] LIMIT 10" }

8. Select Body Content Type as application/json.
9. Click Test Request/Response. If things go well you will see JSON content. Then just copy JobId from the response and save for later use:
   
10. Now go to Response Settings tab.
11. Select ContentType=Json.
12. Enter below expression to extract JobId from the response. You will need this Id to read SQL query output (this JobId is valid for 24 hrs):
    

    $.jobReference.jobId

13. Choose save response content to save to a variable. Select <New Variable…> option. When prompted give a name to your variable (i.e. vJobId) and for the value field paste JobId copied from the above step:
    
    
    NOTE: This JobId expires after 24-hrs so while you are designing a package and not running it then design time value is used for testing/previewing. So make sure your JobId is valid. If needed click Test Request/Response to grab a new JobId and update the variable with a new value so you can preview data without running a full package.
14. Click OK to save the UI.

Read BigQuery SQL result (for specified JobID)

Now let’s look at how to read data from BigQuery

Configure Google BigQuery Web Request (URL, Method, ContentType, Body etc.)

1. In the Control Flow designer drag and drop Data Flow Task from SSIS toolbox.
   
2. Double click Data Flow Task and drag and drop ZS JSON Source (For API/File) from SSIS toolbox.
   
3. Double click JSON Source to edit and configure as below
4. Enter URL as below. Replace YOUR-API-PROJECT-ID with the API Project ID obtained in the previous section.
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?maxResults=10000

   –OR– (Use below if your Job ran outside US / EU data center)
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?location=YOUR_REGION_ID&maxResults=10000

   NOTE: location Parameter indicates the geographic location of the job. This is Required parameter except for US and EU. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location. You can supply same location parameter for first step when you submit the query

   Failure to supply location parameter for non US /EU users may result in 404 NotFound Error.
   

5. Check Use Credentials option and select OAuth Connection Manager created in the previous section.
6. For HTTP Request Method select GET.
   

Read data from Google BigQuery from Temp Job result

Configure Filter

We need to transform a single JSON response into multiple rows so we need to apply a correct filter.

1. On JSON Source go to Filter Options tab.
2. In the Filter field enter $.rows[*] or click [Select Filter] button to browse hierarchy you want to extract.

Select Filter for JSON Response

Configure BigQuery API Pagination Settings

Most of modern APIs usually implement some sort of pagination technique so you get a part of data in each request rather than all in one go. Thus if you want more you can paginate through pages until the last page is reached. You can also read Understanding REST API Pagination in SSIS article to learn more about pagination in SSIS.

BigQuery API returns pageToken attribute in response JSON if more data is found for requested query result. You can then pass pageToken in the next URL in this format: http://my-api-url/?pageToken=xxxxxxxxxx

Now lets configure JSON Source to automate this pagination for us. On JSON Source go to Pagination Tab and enter the following two settings:

1. Set Next Link as $.pageToken.
2. Set Suffix for Next URL as &pageToken=<%nextlink%>.

See the screenshot below to get more clarity:

Configure BigQuery API Pagination on SSIS JSON Source

Configure Array Transformation

Now the last thing we have to configure is special 2-dimensional JSON array format used by BigQuery API:

1. On the JSON Source UI go to 2D Array Transformation tab.
2. Enter the following settings:
   1. For Transformation Type select Transform complex 2-dimensional array.
   2. For Column Name filter enter $.schema.fields[*].name.
   3. For Row Values Filter enter $.f[*].v.

JSON Array Transformation Options

Preview Data and Save UI

That’s it click Preview to see some data. If you entered sample JobID for your User::vJobID variable then you will see some data. In the next section we will see how to load Google BigQuery data into SQL Server. You can click Columns Tab to review Metadata.

Click OK to save UI and generate Metadata.

Configure Target – Load Google BigQuery data into SQL Server

Now you can connect your JSON Source to any target such as ZS Trash Destination or a real database destination such as OLEDB Destination.

If you wish to dump data from Google BigQuery to a SQL Server table then just perform the following steps:

1. Drag and drop OLEDB Destination from SSIS Toolbox. Rename it to something like SQL Server Table.
2. Double click on OLEDB Destination.
3. Click New to create a new connection > Configure connection > Click OK.
4. Click on New to create a new table > Rename default table name > Click OK.
5. Click on Mappings tab and click OK to save UI with default mappings.

Configure SSIS OLEDB Destination – Google BigQuery to SQL Server Import

Execute Package – Loading BigQuery data into SQL Server

SSIS Package Execution –
Loading Google BigQuery Data into SQL Server

Create / Delete Google BigQuery Dataset using API call

As we mentioned before, you need dataset before you can create a table. Most common way to create dataset is via User Interface but what if you like to automate from your SSIS Package or other workflow? Here is how you can create or delete. Basically you can use REST API Task to send CREATE or DROP command for Dataset object

Create Google Dataset

To create dataset configure REST API Task using below settings (We will call this API)

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/datasets

4. Request Method as POST
5. Enter Request Body as below (change YOUR_DATASET_NAME – e.g. TestDataset )
   

   {"datasetReference": { "datasetId": "YOUR_DATASET_NAME", "projectId": "{{User::ProjectId}}"} }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Dataset already exists then you can go to Error Handling Tab (See next section for screenshot) and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Delete Google BigQuery Dataset

For deleing dataset you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Create / Delete Google BigQuery Table using API call

Now let’s look at how to create /drop Table in Google BigQuery using API calls.

Create Google BigQuery Table

To send CREATE TABLE  SQL statement (DDL) we have to use same approach as we send normal SQL Query using this API call. So notice we used Standard SQL for this call by supplying  useLegacySql: false . DDL Statement must be all in one line

POST https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

Content-Type: application/json

>>>> BODY <<<<<

{
 "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
 "useLegacySql": false,
 "timeoutMs": 100000 
}

To create dataset configure REST API Task using below settings

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

4. Request Method as POST
5. Enter Request Body as below
   

   {
    "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
    "useLegacySql": false,
    "timeoutMs": 100000
   }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Table exists then you can go to enable Error Handling Tab and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Create Google BigQuery Table – API Call (Continue On Error Setting – Skip CREATE if Table / Dataset Already Exists)

Delete Google BigQuery Table

For deleing table you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Write data to Google BigQuery using SSIS – 1 Million row insert test (FAST)

Now let’s look at how easy it is to import data into Google BigQuery using SSIS. We will use the same OAuth connection we created before. To learn more about inserting data into BigQuery check tabledata/insertAll method documentation.

Make sure billing is enabled

Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

Streaming is not available via the free tier. If you attempt to use streaming without enabling billing, you receive the following error: BigQuery: Streaming insert is not allowed in the free tier.

As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call (Write  demo).

Configure OAuth Connection / Permissions

If you want to perform data insert operation in BigQuery using API calls then include the following scopes in your OAuth Connection Manager and generate a token (see our first section of this article – We already included scopes for Write operation but incase you didnt do then regenerate token with below scopes):

https://www.googleapis.com/auth/bigquery
https://www.googleapis.com/auth/cloud-platform
https://www.googleapis.com/auth/bigquery.insertdata

Create BigQuery Dataset (From UI)

For this demo first create a test dataset and one table under it like shown below (billing must be enabled on your Google API Project). To do that go to https://bigquery.cloud.google.com/welcome and configure them:

Insert data into BigQuery using API call

And here is a REST API example to insert data into a BigQuery Table:

Request URL:

https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

Request Headers:

Authorization: Bearer ya29.Gl0cBxxxxxxxxxxxxxxxxxxuEIhJIEnxE6GsQPHI
Content-Type: application/json

Request Body:

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
     {"json": {"RowId": 2,"CustomerName": "BBB"} }
   ]
}

Here is our data flow setup to achive very high throughput for Google BigQuery Data Load. We will show you how to insert one million rows in Google BigQuery in less than a minute based on below setup. We will use Multi Threading option and New Compression Option (Added in v3.1.4)

1. Dummy Data Source – Generate sample records
2. JSON Generator Transform – Generates JSON documents to send as POST request for above /insertAll API call.
3. Web API destination  – Call /insertAll API call to submit our data to BigQuery
4. (Optional) JSON Parser Transform – Parse Error Message for any response
5. (Optional) Trash Destination – Save any errors to text file for review

Google BigQuery Data Load Demo in SSIS – 1 Million Rows Insert with Multi Threads and Compression ON (Fast Upload)

Configure SSIS JSON Generator – Generate JSON for BigQuery Table Insert

Now let’s see how to build an HTTP request with JSON body and send it to BigQuery:

1. Drag Data Flow Task and double click on it.
   
2. Drag and configure your Source (for this demo we use Dummy Data Source with Customer Template). See previous section for configuration of Dummy Data Source.
   
3. Drag and drop ZS JSON Generator Transform to produce JSON from a database or file records. If your source is already sending a valid JSON then you can skip this step (e.g. SQL query is
   returning JSON). You can also read raw JSON from a very large file (new-line separated JSON) using JSON Source with Output as Raw Data option checked.
4. Connect Source to JSON Generator. Double click JSON Generator Transform to start configuring it like below.
5. Select Output Mode as Single Dataset Array and enter Batch Size 10000 (This is Max limit allowed by Google BigQuery API insertAll)
6. First right click on Mappings node and select Add Static Element
   

   Generate JSON for Google BigQuery InsertAll API request – Batch 10000 rows in a single API call

7. Enter Name as kind and value as bigquery#tableDataInsertAllRequest
   
   

   JSON Generator – Add Static Element

8. Now right click on Mappings node and select Add Document Array option
   

   JSON Generator – Add Document Array

9. Enter rows as array title and click OK
   

   JSON Generator – Name array

10. Select newly added array node and right click > Add unbound nested element  enter Output alias as json and click OK.
    

    JSON Generator – Add unbound nested element

11. Select json node and right click > Select Add Elements below this node and select multiple columns you like to send to BigQuery. Click OK to save.
    

    JSON Generator – Add Multiple Elements

12. Now let’s preview our JSON (Copy preview JSON to try in the next step – Web API destination)
    NOTE: Table name and column names are case-sensitive so make sure your JSON attribute matches exact same way.
    Here is the finished JSON Structure for next Step
    

    Sample JSON Request body for Google BigQuery insertAll API request

13. Click OK to save UI.

Configure SSIS Web API destination – Insert data into BigQuery Table

Once you have Input JSON prepared,  now let’s configure destination for BigQuery.

1. Drag and drop ZS Web API Destination.
2. Connect your JSON Generator Transform to Web API destination.
3. Configure general properties:
   

   SSIS Web API Destination –
   Configure for BigQuery Data load

4. Make sure to enter URL in this format:
   

   https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

   Make sure to replace 3 parts in above URL (MY_PROJECT_ID, MY_DATASET_ID, MY_TABLE_ID) with actual values from your Google Project and BigQuery dataset/table configuration.
5. Now you can enable Compression and Multiple Threads for higher throughput as below.
   NOTE: Compression Property was added in v3.1.4 so you may not see it if you have older version.
   

   Google BigQuery Data Loading Performance Optimization Options – Enable Multiple Threads and Compression Options

6. If you want to try test insert request from UI without running full package then go back to first tab and edit Body (Use Sample JSON generated by previous JSON Transform – You can grab from JSON Preview Panel on Generator Transform).Click Test Request / Response and confirm Success as below. You can go back to your BigQuery Portal and check one row is inserted after our test click. If everything looking good then run full package to insert all records.Sample JSON for Body:
   

   {
     "kind": "bigquery#tableDataInsertAllRequest",
     "rows": [
        {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
        {"json": {"RowId": 2,"CustomerName": "BBB"} }
      ]
   }

   

   Test Google BigQuery Table Insert – SSIS Web API Destination UI

7. Hit OK to save UI.
8. Run the package and verify data in Google BigQuery Console:
   

   Loading data into Google BigQuery using SSIS

Error Handling for BigQuery Data Load (Bulk Insert API Calls)

There will be a time when some records you insert may not go well in Google BigQuery. In such case you can read output from Web API destination and parse further with JSON Parser Transform. Check for certain values in the output. You must use JSON Parser after Web API destination (Connect Blue Arrow from Web API destination – Since its Soft Error it won’t redirect in Red Arrow ).

For example here is the sample JSON in POST Body for testing which produces error due to bad column name. When bad row found in batch all records will be rejected. Notice that error returns index of record in batch so you can identify which row went bad. It also returns column name in location attribute.

NOTE: Bad column name in 2nd record

Test Body (Bad):

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RecordID": 1,"CustomerID": "X1"} }, 
     {"json": {"Bad_Column": 2,"CustomerID": "X2"} }
     {"json": {"RecordID": 3,"CustomerID": "X3"} }, 
   ]
}

Response (For Bad Input):

{
  "kind": "bigquery#tableDataInsertAllResponse",
  "insertErrors": [
    {
      "index": 1,
      "errors": [
        {
          "reason": "invalid",
          "location": "bad_column",
          "debugInfo": "",
          "message": "no such field."
        }
      ]
    },
    {
      "index": 0,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    },
    {
      "index": 2,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    }
  ]
}

To configure error detection perform following steps.

1. Drag and drop ZS JSON Parser Transform after Web API destination
2. Click on Web API destination. Connect Blue arrow  to JSON Parser Transform
3. Configure JSON Parser Transform like below
4. Connect JSON Parser TRansform to some Destination to save error information (e.g. SQL Table or Trans destination)

Handling BigQuery Insert Errors in SSIS

Other Common Errors in BigQuery API calls

In this section we will talk about many common API errors in BigQuery.

Error: The project XXXXXXX has not enabled BigQuery

Sometimes you might get below error. To fix this error make sure you go to your Project and Enable BigQuery API

Status Code: BadRequest

Response Body: {
  "error": {
    "code": 400,
    "message": "The project bigquerytest-281915 has not enabled BigQuery.",
    "errors": [
      {
        "message": "The project bigquerytest-281915 has not enabled BigQuery.",
        "domain": "global",
        "reason": "invalid"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}

Error: 404 – Not Found: Table / Job xxxxxx

• Make sure you are setting GET request and not POST.
• Also make sure your jobId is valid because it expires after 24 hours.
• Make sure project-id supplied in URL is valid ID  (DO NOT Specify Alias, use internal ID for project)
• Make sure you supplied location parameter if your outside EU / US region. Reading data using this API call might fail if you failed to supply location in URL

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "notFound",
    "message": "Not Found: Table xxxxxx"
   }
  ],
  "code": 404,
  "message": "Not Found: Table xxxxxxx
 }
}

Error: 403 – Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier

If you trying call certain APIs on sandbox mode or free tier then you might get below error. To overcome this error enable billing on google bigquery. As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call.

Response Url: https://www.googleapis.com/bigquery/v2/projects/bigquerytest-281915/datasets/TestDataset/tables/Table1/insertAll

Status Code: Forbidden

Response Body: {
  "error": {
    "code": 403,
    "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
    "errors": [
      {
        "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
        "domain": "global",
        "reason": "accessDenied"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}

Debugging Web API Requests

If you need to debug actual requests made to Google server then you can use a tool like Fiddler. It’s a very handy tool to troubleshoot JSON format issues. It will allow to see how a request is made to a server.

Using Fiddler to debug Google BigQuery API requests in SSIS

Common Errors

Deployment to Production

Download Sample Package

Click Here to Download SSIS Sample Package – Google BigQuery API Read Write Create Delete (SSIS 2019, 2017, 2012)

Conclusion. What’s next?

In this article we have learned how to load data from Google BigQuery into SQL Server using SSIS (drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from Google BigQuery REST API using OAuth. JSON Source Connector makes it super simple to parse complex/large JSON files or any Web API response into rows and columns so you can load data into a database, e.g. SQL Server database. Download SSIS PowerPack to try many other automation scenarios that were not discussed in this article.

Keywords:

Google BigQuery Integration with SQL Server | How to extract data from google bigquery in SSIS? | How to read data from Google BigQuery API? | Loading BigQuery Data into SQL Server. | BigQuery to SQL Server | SSIS Google Big Query Integration | SSIS Google BigQuery Import  JSON File | SSIS Google BigQuery Export data

The post How to read / write data in Google BigQuery using SSIS appeared first on ZappySys Blog.

In our previous post, we saw how to get data from Google Analytics using SSIS. Once you get going with the drag and drop SSIS Google Analytics Connector, you may bump up to one interesting challenge. If your request includes sessions above a certain threshold, Google may start sampling your data to provide an average for many Metrics. So, how do you make sure your request includes all sessions in the calculation? The following sections will describe a few ways to achieve this.

Google Analytics Data Sampling Tips

There is no way to guarantee 100% Unsampled data in the Standard Google Analytics API call due to restrictions imposed by Google. However, there are a few workarounds that may help you achieve a very high Data Sampling rate.

The most common solution is that if you set the Sampling option to HIGHER_PRECISION, then Google may honor your request up to a certain threshold, but as the dataset grows (e.g., if you select a larger date range), Google may sample your data even though you set HIGHER_PRECISION.

Extract data from Google Analytics in SSIS: Use the Sample Size option to get Unsampled data.

Other ways to get unsampled data from the Google Analytics API

Now, let’s look at a few other ways if the previous approach doesn’t work.

1. Narrow down your Date range… e.g., Process day by day rather than a full extract. If day by day is also sampling, they try Hour-by-Hour.
2. Use the Google API to generate and download Unsampled data (We don’t have an article on this, but see the reference below).
3. Use a Google Analytics 360 Premium Account with the unsampled report feature. You can get this via the Web Interface or call the API to automate the extraction. https://support.google.com/analytics/answer/2601061?hl=en

https://developers.google.com/analytics/devguides/config/mgmt/v3/unsampled-reports
https://support.google.com/analytics/answer/2637192?hl=en&visit_id=1-636288218914624195-2113255108&rd=1

Downloading Unsampled report using Google Analytics 360 API – ZappySys REST API Task

If you are a Google Analytics 360 user and wish to use the API to generate a file and download it in SSIS, refer to the following article. It shows how to call the Google API. It’s not about the Analytics API, but you will get a high-level idea.

https://zappysys.com/blog/get-data-google-spreadsheet-using-ssis/

The post 3 ways to get Google Analytics unsampled data in SSIS appeared first on ZappySys Blog.

Google DoubleClick is one of the most popular platforms for Advertisers. Many times you have need for custom integration or Automation for many operations. In this article we will learn how to Call Google DFP API (i.e. DoubleClick for Publishers) without coding using SSIS (Microsoft SQL Server Integration Services). In our previous article we discussed how to Integrate Google AdWords API using SSIS (Click here).

If you are new to SSIS then no worry there are plenty of tutorials for ssis

To achieve Custom integration for DoubleClick (DFP) API we will use SSIS XML Source. XML Source is part of SSIS PowerPack which has 45+ connectors and Tasks.

Prerequisite

Before you can finish tasks explained in this article you have to finish below steps. This article assumes you have basic knowledge of SSIS (SQL Server Integration Services)

1. Make sure you SSIS designer installed. Sometimes its referred as BIDS or SSDT (Get from here)
2. Download SSIS PowerPack
3. Download SoapUI (Its free third party tool to test SOAP API). SoapUI can help you to generate XML Request BODY easily from WSDL file provided by API Vendor. You can also test your service by supplying parameters.
4. Optional – Another very useful Free tool is Fiddler. You can use it to see raw request/response (Check this article)

Download DFP API Example SSIS Package

Click here to DFP API download Sample for SSIS SSIS Package (SSIS 2012, 2014, 2016)

Screenshot of Sample Package:

Example SSIS Package – Google DFP API Integration (Read / Write data in Google Double Click for Publisher) – No Coding required

Making your first DFP API Call using SSIS

Once you install PowerPack you are ready to execute your very first Google DFP API Call.

NOTE: If you don’t have any DFP network for test then see next section (Explains how to create test network for testing).

Lets look at how to call Google DFP API step-by-step.

1. Create new SSIS Project and open package designer
2. Drag ZS REST API Task from SSIS control flow toolbox and drop it on designer.
   
   REST API Task is useful when you want to call SOAP/REST API but not necessarily parse response into rows and columns. We will look at JSON Source later in this article which can actually parse response into rows and columns (Useful to load Google DFP data into SQL Server or other RDBMS / flatfile )
3. Double click REST API Task to configure. Select Access mode to [Url from Connection]
4. Enter following URL in the URL Textbox
   

    https://ads.google.com/apis/ads/publisher/v201702/NetworkService

5. From the connection dialogbox select ZS-OAUTH to create new OAuth Connection for DFP API
6. On OAuth Connection dialogbox select Google from Provider dropdown. In this demo we will use Default OAuth App but you can register your own google OAuth app if you wish to use Custom OAuth option from UI.
7. Enter following Scopes in the Scopes textbox (or select manually by clicking Select Scopes button). Scope is nothing but permission for API (In our case View/Manage DFP data and Read/Write Report Files to Cloud Storage).
   

   https://www.googleapis.com/auth/dfp
   https://www.googleapis.com/auth/devstorage.read_only
   https://www.googleapis.com/auth/devstorage.write_only
   https://www.googleapis.com/auth/cloud-platform.read-only
   https://www.googleapis.com/auth/devstorage.full_control
   https://www.googleapis.com/auth/cloud-platform
   https://www.googleapis.com/auth/devstorage.read_write

8. If you are not going call ReportService API to generate Reports in CSV format then you will need only one scope
   

   https://www.googleapis.com/auth/dfp

9. Click Generate Token button. You will see browser popup for login and then Accept option to confirm permissions like below. Once you don’t it will populate tokens and prompt to save tokens to backup file. Secure token backup to safe place.
   

   SSIS OAuth Connection – Connect to Google DFP API (DoubleClick SOAP Web Service)

10. Once done Click Test Connection and if its green then Click OK to save connection
11. Once you are back to REST API Task UI then select Method to POST
12. Select Request Body Content Type to XML (i.e. text/xml)
13. Click edit button next to Request Body and enter following XML.
    

    <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
       <soapenv:Header>
          <v20:RequestHeader/>
       </soapenv:Header>
       <soapenv:Body>
          <v20:getAllNetworks/>
       </soapenv:Body>
    </soapenv:Envelope>

    If you are wondering how did we get above XML fragment then read next section about using 3rd party tool called SoapUI. Also we have detail article for calling SOAP request. Check this article on how to use SoapUI (free 3rd party tool) to create SOAP request Bodyfrom WSDL. DFP has many API endpoints for different actions (e.g. NetworkService , InventoryService , …). For each service you have different Api URL and different WSDL. In the next section you can learn how to create correct SOAP Body (for POST) using SoapUI tool.

    For example if you using DFP NetworkService API for version v201702 then your help page URL would be like this
    https://developers.google.com/doubleclick-publishers/docs/reference/v201702/NetworkService
    And on the same page you will see WSDL link below
    https://adwords.google.com/api/adwords/mcm/v201609/ManagedCustomerService?wsdl
    (Just download that WSDL XML and save to local disk then use with SoapUI to generate XML body to submit for any DFP API call.)

14. Once you done with above steps, your Task Configuration will look like below.
    

    SSIS Rest API Task – Call Google DoubleClick API – Get all DFP Networks

15. Now click Test Request button. You will see Response popup with below content if its successful (Scroll at the bottom on response form and you will see XML data) .. see below
    

    ------------------------------------
    Request
    ------------------------------------
    POST https 1.1 ==> /apis/ads/publisher/v201702/NetworkService
    Host: ads.google.com
    
    >>>> HEADERS <<<<<
    Authorization: Bearer ya29.Glz-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    Content-Type: text/xml;charset=UTF-8
    User-Agent: ZappySysApp
    Host: ads.google.com
    Content-Length: 291
    Expect: 100-continue
    Accept-Encoding: gzip, deflate
    Connection: Keep-Alive
    
    ------------------------------------
    Response
    ------------------------------------
    POST 1.1 200 OK==> OK
    
    >>>> HEADERS <<<<<
    Content-Encoding: 
    X-Content-Type-Options: nosniff
    X-Frame-Options: SAMEORIGIN
    X-XSS-Protection: 1; mode=block
    Alt-Svc: quic=":443"; ma=2592000; v="35,34"
    Transfer-Encoding: chunked
    Cache-Control: private, max-age=0
    Content-Type: text/xml; charset=UTF-8
    Date: Sun, 26 Feb 2017 01:18:32 GMT
    Expires: Sun, 26 Feb 2017 01:18:32 GMT
    Server: GSE
    
    >>>> Cookies <<<<<
    
    >>>> CONTENT <<<<<
    
    <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    	<soap:Header>
    		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
    			<requestId>27ab3d2e4e1fcb8f444b317d0b0f871d</requestId>
    			<responseTime>50</responseTime>
    		</ResponseHeader>
    	</soap:Header>
    	<soap:Body>
    		<getAllNetworksResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
    			<rval>
    				<id>551196</id>
    				<displayName>XFP sandbox property</displayName>
    				<networkCode>238897396</networkCode>
    				<propertyCode>ca-pub-2799179143725683</propertyCode>
    				<timeZone>America/New_York</timeZone>
    				<currencyCode>USD</currencyCode>
    				<effectiveRootAdUnitId>237897516</effectiveRootAdUnitId>
    				<isTest>true</isTest>
    			</rval>
    		</getAllNetworksResponse>
    	</soap:Body>
    </soap:Envelope>

     

How to generate SOAP Request Body using SoapUI tool

This section describes how to create DoubleClick SOAP API Request Body (XML fragment) for any DFP SOAP API call described anywhere in this article.

1. Download SoapUI (Its free third party tool to test SOAP API). SoapUI can help you to generate XML Request BODY easily from WSDL file provided by API Vendor. You can also test your service by supplying parameters.
2. Now download WDSL xml for appropriate API calls you want to make. Assume that you want to call  getAllNetworksResponse API found under NetworkService. For that first navigate to help file page and open help page. Make sure you select correct version from help navigation. Copy WSDL link and open in new browser window. Save XML to local disk (e.g. c:\api\dfp_networkservice_wsdl.xml (See below screenshot how to find DFP API WSDL link)
   

   How to get Google DFP API WSDL URL (Used to generate POST Body)

3. Once SoapUI is downloaded and installed click File menu > Create New SOAP Project option
4. Name your project (e.g. DFP API) and specify WSDL URL or File Path if it was saved locally (e.g. c:\api\dfp_networkservice_wsdl.xml) and click OK
5. Now navigate to API Action for which you would like to get Body. Click Request node (If missing create new) and edit request. You may see XML like below (If some optional parameters not visible in XML then click Re-create request with optional parameters button from toolbar)
   

   Call Google DFP API – Generate SOAP Request Body XML using SoapUI (Google DoubleClick Web Service)

6. Edit necessary parameters from above XML code and copy to Body of SSIS REST API Task.

Creating test network – DFP Sandbox

Very first step we recommend before testing DFP API call is create test network. You can call DFP API against LIVE network but not recommended if you are calling create/update API.

To create test network you can follow exact same steps described in the previous section except the Body step (Step#12). Use following Request body to call makeTestNetwork command.

API makeTestNetwork – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/NetworkService
Body (see below):

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
   <soapenv:Header>
      <v20:RequestHeader/>
   </soapenv:Header>
   <soapenv:Body>
      <v20:makeTestNetwork/>
   </soapenv:Body>
</soapenv:Envelope>

Once you click Test you may receive following response. Note your network code (e.g. 1122334455 from below). This network code is used in pretty much all DFP API to view or manage ad network related items or properties.

API makeTestNetwork – Response

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<requestId>ba61d79efee325bf4b5aa45c46836021</requestId>
			<responseTime>1021</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<makeTestNetworkResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<rval>
				<id>551196</id>
				<displayName>XFP sandbox property</displayName>
				<networkCode>1122334455</networkCode>
				<propertyCode>ca-pub-2799179143725683</propertyCode>
				<timeZone>America/Los_Angeles</timeZone>
				<currencyCode>USD</currencyCode>
				<effectiveRootAdUnitId>237897516</effectiveRootAdUnitId>
				<isTest>true</isTest>
			</rval>
		</makeTestNetworkResponse>
	</soap:Body>
</soap:Envelope>

You can also get network code using two different ways.

• Visit your DFP console homepage by visiting https://www.google.com/dfp/ and you will see network name and code in the top potion. Also its listed in the URL (e.g. https://www.google.com/dfp/1234567)
• Another way to get available networks for your login is to visit DFP API Play ground here https://dfp-playground.appspot.com/ and you will see network list in the dropdown ( name and network code)

How to Create new Ad Units

Now lets look at an example which will create few Ad Units by calling createAdUnits API. If you are creating Ad Unit at the root level then you will need to specify correct Parent ID (i.e  effectiveRootAdUnitId). To obtain that Parent ID you may have to call  getCurrentNetwork API. In the response you will see effectiveRootAdUnitId.

There is another way to know effectiveRootAdUnitId or Id of any Ad Units which can be Parent for new AdUnit. Goto google.com/dfp > Click Inventory Tab > Click Ad Units Side menu > Click Download ad units hyper link. You can also download AdUnits as CSV file from DFP Portal and look for Ad Units which are created at the root. Parent ID column for Root level Ad Units is basically called effectiveRootAdUnitId.

Now lets look at how to get EffectiveRootID by calling getCurrentNetwork.

API getCurrentNetwork – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/NetworkService
Body (see below):

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Header>
	<RequestHeader xmlns="https://www.google.com/apis/ads/publisher/v201608">
		<networkCode xmlns="https://www.google.com/apis/ads/publisher/v201608">1122334455</networkCode>
		<applicationName xmlns="https://www.google.com/apis/ads/publisher/v201608">ZappySysApp</applicationName>
	</RequestHeader>
	</soap:Header>
	<soap:Body>
		<getCurrentNetwork xmlns="https://www.google.com/apis/ads/publisher/v201608" />
	</soap:Body>
</soap:Envelope>

API getCurrentNetwork – Response

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<requestId>81d1b00e8fbfb862129eb2dd485cf7bb</requestId>
			<responseTime>4507</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<getCurrentNetworkResponse xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<rval>
				<id>551196</id>
				<displayName>XFP sandbox property</displayName>
				<networkCode>1122334455</networkCode>
				<propertyCode>ca-pub-2799179143725683</propertyCode>
				<timeZone>America/New_York</timeZone>
				<currencyCode>USD</currencyCode>
				<effectiveRootAdUnitId>237897516</effectiveRootAdUnitId>
				<isTest>true</isTest>
			</rval>
		</getCurrentNetworkResponse>
	</soap:Body>
</soap:Envelope>

If you want to save single Element Value from Response into SSIS Variable then perform following steps (example of how to get just effectiveRootAdUnitId from above response)

1. On REST API Task > Go to Response Tab > Select Response Content Type = XML
2. Enter following expression in the XPath filter
   

   //*[local-name() = 'effectiveRootAdUnitId']

3. Check Save response content option
4. Select Variable from Dropdown – Click New Variable > Name it RootAdUnitId.
5. Click Test Request Button to test…. In the content textbox you will now see only Numeric value extracted by Filter

SSIS REST Api Task – How to extract single vale from response and save to SSIS variable

Once you know Parent ID for your AdUnit now lets look at how to create Ad Units by calling createAdUnits

API createAdUnits – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/InventoryService
Body (see below):

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<networkCode xmlns="https://www.google.com/apis/ads/publisher/v201608">1122334455</networkCode>
			<applicationName xmlns="https://www.google.com/apis/ads/publisher/v201608">ZappySysApp</applicationName>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<createAdUnits xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<adUnits>
				<parentId>237897516</parentId>
				<name>Ad_Unit_0</name>
				<description>Ad unit description #0.</description>
				<targetWindow>BLANK</targetWindow>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
					</size>
					<environmentType>BROWSER</environmentType>
				</adUnitSizes>
			</adUnits>
			<adUnits>
				<parentId>237897516</parentId>
				<name>Ad_Unit_1</name>
				<description>Ad unit description #1.</description>
				<targetWindow>BLANK</targetWindow>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
					</size>
					<environmentType>BROWSER</environmentType>
				</adUnitSizes>
			</adUnits>			
		</createAdUnits>
	</soap:Body>
</soap:Envelope>

API createAdUnits – Response

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<requestId>b244d7ba9b65e96fc42b849a7bfb16e4</requestId>
			<responseTime>331</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<createAdUnitsResponse xmlns="https://www.google.com/apis/ads/publisher/v201608">
			<rval>
				<id>252312396</id>
				<parentId>237897516</parentId>
				<hasChildren>false</hasChildren>
				<parentPath>
					<id>237897516</id>
					<name>ca-pub-2799179143725683</name>
					<adUnitCode>ca-pub-2799179143725683</adUnitCode>
				</parentPath>
				<name>Ad_Unit_0</name>
				<description>Ad unit description#0.</description>
				<targetWindow>BLANK</targetWindow>
				<status>ACTIVE</status>
				<adUnitCode>253312396</adUnitCode>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
						<isAspectRatio>false</isAspectRatio>
					</size>
					<environmentType>BROWSER</environmentType>
					<fullDisplayString>300x250</fullDisplayString>
				</adUnitSizes>
				<mobilePlatform>SITE</mobilePlatform>
				<explicitlyTargeted>false</explicitlyTargeted>
				<inheritedAdSenseSettings>
					<value>
						<adSenseEnabled>true</adSenseEnabled>
						<borderColor>FFFFFF</borderColor>
						<titleColor>0000FF</titleColor>
						<backgroundColor>FFFFFF</backgroundColor>
						<textColor>000000</textColor>
						<urlColor>008000</urlColor>
						<adType>TEXT_AND_IMAGE</adType>
						<borderStyle>DEFAULT</borderStyle>
						<fontFamily>DEFAULT</fontFamily>
						<fontSize>DEFAULT</fontSize>
					</value>
				</inheritedAdSenseSettings>
				<lastModifiedDateTime>
					<date>
						<year>2017</year>
						<month>2</month>
						<day>27</day>
					</date>
					<hour>7</hour>
					<minute>23</minute>
					<second>32</second>
					<timeZoneID>PST8PDT</timeZoneID>
				</lastModifiedDateTime>
				<smartSizeMode>NONE</smartSizeMode>
				<isSharedByDistributor>false</isSharedByDistributor>
				<isSetTopBoxEnabled>false</isSetTopBoxEnabled>
			</rval>
			<rval>
				<id>252312516</id>
				<parentId>237897516</parentId>
				<hasChildren>false</hasChildren>
				<parentPath>
					<id>237897516</id>
					<name>ca-pub-2799179143725683</name>
					<adUnitCode>ca-pub-2799179143725683</adUnitCode>
				</parentPath>
				<name>Ad_Unit_1</name>
				<description>Ad unit description#2.</description>
				<targetWindow>BLANK</targetWindow>
				<status>ACTIVE</status>
				<adUnitCode>253312516</adUnitCode>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
						<isAspectRatio>false</isAspectRatio>
					</size>
					<environmentType>BROWSER</environmentType>
					<fullDisplayString>300x250</fullDisplayString>
				</adUnitSizes>
				<mobilePlatform>SITE</mobilePlatform>
				<explicitlyTargeted>false</explicitlyTargeted>
				<inheritedAdSenseSettings>
					<value>
						<adSenseEnabled>true</adSenseEnabled>
						<borderColor>FFFFFF</borderColor>
						<titleColor>0000FF</titleColor>
						<backgroundColor>FFFFFF</backgroundColor>
						<textColor>000000</textColor>
						<urlColor>008000</urlColor>
						<adType>TEXT_AND_IMAGE</adType>
						<borderStyle>DEFAULT</borderStyle>
						<fontFamily>DEFAULT</fontFamily>
						<fontSize>DEFAULT</fontSize>
					</value>
				</inheritedAdSenseSettings>
				<lastModifiedDateTime>
					<date>
						<year>2017</year>
						<month>2</month>
						<day>27</day>
					</date>
					<hour>7</hour>
					<minute>23</minute>
					<second>32</second>
					<timeZoneID>PST8PDT</timeZoneID>
				</lastModifiedDateTime>
				<smartSizeMode>NONE</smartSizeMode>
				<isSharedByDistributor>false</isSharedByDistributor>
				<isSetTopBoxEnabled>false</isSetTopBoxEnabled>
			</rval>
				<id>252312876</id>
				<parentId>237897516</parentId>
				<hasChildren>false</hasChildren>
				<parentPath>
					<id>237897516</id>
					<name>ca-pub-2799179143725683</name>
					<adUnitCode>ca-pub-2799179143725683</adUnitCode>
				</parentPath>
				<name>Ad_Unit_4</name>
				<description>Ad unit description.</description>
				<targetWindow>BLANK</targetWindow>
				<status>ACTIVE</status>
				<adUnitCode>253312876</adUnitCode>
				<adUnitSizes>
					<size>
						<width>300</width>
						<height>250</height>
						<isAspectRatio>false</isAspectRatio>
					</size>
					<environmentType>BROWSER</environmentType>
					<fullDisplayString>300x250</fullDisplayString>
				</adUnitSizes>
				<mobilePlatform>SITE</mobilePlatform>
				<explicitlyTargeted>false</explicitlyTargeted>
				<inheritedAdSenseSettings>
					<value>
						<adSenseEnabled>true</adSenseEnabled>
						<borderColor>FFFFFF</borderColor>
						<titleColor>0000FF</titleColor>
						<backgroundColor>FFFFFF</backgroundColor>
						<textColor>000000</textColor>
						<urlColor>008000</urlColor>
						<adType>TEXT_AND_IMAGE</adType>
						<borderStyle>DEFAULT</borderStyle>
						<fontFamily>DEFAULT</fontFamily>
						<fontSize>DEFAULT</fontSize>
					</value>
				</inheritedAdSenseSettings>
				<lastModifiedDateTime>
					<date>
						<year>2017</year>
						<month>2</month>
						<day>27</day>
					</date>
					<hour>7</hour>
					<minute>23</minute>
					<second>32</second>
					<timeZoneID>PST8PDT</timeZoneID>
				</lastModifiedDateTime>
				<smartSizeMode>NONE</smartSizeMode>
				<isSharedByDistributor>false</isSharedByDistributor>
				<isSetTopBoxEnabled>false</isSetTopBoxEnabled>
			</rval>
		</createAdUnitsResponse>
	</soap:Body>
</soap:Envelope>

Loading Google DoubleClick data into SQL Server Table

So far we saw how to make simple DFP API calls without doing any parsing. Now lets look at how to use SSIS XML Source which not only make API calls but it will parse XML Response into rows and columns which can be loaded into target database such as SQL Server, Oracle, MySQL or even Flat File. XML Source also supports pagination so if you have many records it will automatically loop through all response untill all records are fetched.

Step-By-Step – Using XML Source to read Google DFP data (Parse into rows and columns)

1. Assuming you have tested your first DFP API Call (Explianed in the beginning of this article). You must have OAuth connection tested for API call.
2. Drag new Data flow task from SSIS Toolbox
3. Inside Data flow designer drag and drop ZS XML Source from toolbox
4. Double click XML Source to configure.
5. Set URL as below
   https://ads.google.com/apis/ads/publisher/v201702/InventoryService
6. Check Use credentials option and from Drop down select OAuth connection manager (created in previous section)
7. Select Method = POST,
8. Select Request Content Type = XML (text/xml)
9. Enter Body as below (Click edit button).
   

   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
      <soapenv:Header>
         <v20:RequestHeader>
            <v20:networkCode>238897396</v20:networkCode>
            <v20:applicationName>ZappySysApp</v20:applicationName>
         </v20:RequestHeader>
      </soapenv:Header>
      <soapenv:Body>
         <v20:getAdUnitsByStatement>
            <v20:filterStatement>
               <v20:query>WHERE Status='ACTIVE'</v20:query>
            </v20:filterStatement>
         </v20:getAdUnitsByStatement>
      </soapenv:Body>
   </soapenv:Envelope>

   Notice that in above we calling getAdUnitsByStatement API to fetch all Ad Units which are Active. Here is more information about Query Syntax for any API call getXXXXXXByStaement
10. Now click on Select Filter button and select results node and click OK. If prompted add array to list.
    

    SSIS XML Source – Get data from Google DFP API using OAuth (Configure URL, Body and Filter)

11. Click Preview to see data. Click Columns tab to review data types (Auto detected). If you wish to change length or datatype then edit there and check Lock option (Last column)
    

    SSIS XML Source – Preview Response for Google DoubleClick API Call – Parse into Rows/Columns

12. Click OK to save UI
13. Connect XML Source to same target (e.g. OLEDB Destination – SQL Server Connection)
14. Run data flow
    

    SSIS Execution – Read from Google DFP and load into SQL Server Table

Configure Pagination for Google DFP API / PQL Query result

If you have large dataset to read from DoubleClick API ( when calling API such as getxxxxxxByStatement) then we recommend to supply LIMIT and OFFSET clause in your PQL query.

For example if you have 2000 ad units and you want to limit response size by maximum 300 rows then your query can be like below

--For first request (returns results 1-300)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 0

--For second request (returns results 301-600)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 300

--For third request  (returns results 601-900)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 600

........

This looping logic can be complex to implement if you do manually. But no worry if you are using SSIS XML Source. It comes with many pagination options (Click here to read more about pagination)

See below screenshot how to configure Pagination options for Google DoubleClick API. Basically two places you have to change. Inside body you have to set placeholder and change few settings on Pagination Tab.

SSIS XML Source – Configure Pagination for Google DoubleClick API data fetch – PQL LIMIT and OFFSET clause for Google DFP API

SSIS XML Source – Configure Pagination for Google DoubleClick API data fetch – Set Pagination Mode

Create CSV Report File and Download in GZip format (*.gz)

Sometime you have to download large amount of data (Possibly millions of rows) in that case Bulk approach would be better. DFP ReportService API allows to call following APIs which can be used to produce CSV report file in *.gz format and then you can download it using REST API Task.

Below are high level steps you have to perform to produce report file and download it.

• Create the ReportJob by invoking runReportJob command .
• Poll the ReportJob object using ReportService.getReportJob (This is required because file is not available right away) .
• Continue to poll the ReportJob object until the reportJobStatus field is equal to COMPLETED or FAILED.
• If successful, fetch the URL for downloading the report by invoking getReportDownloadURL.

Here is the sample SSIS Package to Perform this Action.

Click here to DFP API download Sample for SSIS SSIS Package (SSIS 2012, 2014, 2016)

See below screenshot how to generate Google DFP Report File and download / extract (Unzip) using Drag and Drop SSIS workflow.

Create Google DFP Report File (CSV / Gzip) and Download Using SSIS (Example of ReportService functions runReportJob, getReportJob and .getReportDownloadURL)

Call runReportJob (Start DFP Report JOB)

Very first step to produce DFP Report file is call ReportService >> runReportJob API. See below Command.

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract id of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for <id>111111</id> node anywhere in SOAP response XML and extract value (i.e. 111111 )  .. For screenshot see next section

//*[local-name() = 'id']

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxxxxAKqG89gM-TpReRXO
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 1018
Expect: 100-continue
Accept-Encoding: gzip, deflate

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
	<soapenv:Header>
		<v20:RequestHeader>
			<v20:networkCode>238897396</v20:networkCode>
			<v20:applicationName>ZappySysApp</v20:applicationName>
		</v20:RequestHeader>
	</soapenv:Header>
	<soapenv:Body>
		<v20:runReportJob>
			<v20:reportJob>
				<v20:reportQuery>
					<v20:dimensions>DATE</v20:dimensions>
					<v20:dimensions>LINE_ITEM_ID</v20:dimensions>
					<v20:dimensions>LINE_ITEM_NAME</v20:dimensions>
					<v20:dimensions>AD_UNIT_NAME</v20:dimensions>
					<v20:columns>AD_SERVER_IMPRESSIONS</v20:columns>
					<v20:columns>AD_SERVER_CLICKS</v20:columns>
					<v20:dimensionAttributes>LINE_ITEM_COST_TYPE</v20:dimensionAttributes>
					<v20:dimensionAttributes>LINE_ITEM_GOAL_QUANTITY</v20:dimensionAttributes>
					<v20:dateRangeType>LAST_MONTH</v20:dateRangeType>
				</v20:reportQuery>
			</v20:reportJob>
		</v20:runReportJob>
	</soapenv:Body>
</soapenv:Envelope>

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:41 GMT
Expires: Fri, 02 Jun 2017 21:23:41 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 999

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<requestId>f03e7e474fde2a5f9c5d7c9d7eb83923</requestId>
			<responseTime>846</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<runReportJobResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<rval>
				<id>10000297829</id>
				<reportQuery>
					<dimensions>DATE</dimensions>
					<dimensions>LINE_ITEM_ID</dimensions>
					<dimensions>LINE_ITEM_NAME</dimensions>
					<dimensions>AD_UNIT_NAME</dimensions>
					<adUnitView>TOP_LEVEL</adUnitView>
					<columns>AD_SERVER_IMPRESSIONS</columns>
					<columns>AD_SERVER_CLICKS</columns>
					<dimensionAttributes>LINE_ITEM_COST_TYPE</dimensionAttributes>
					<dimensionAttributes>LINE_ITEM_GOAL_QUANTITY</dimensionAttributes>
					<dateRangeType>LAST_MONTH</dateRangeType>
					<useSalesLocalTimeZone>false</useSalesLocalTimeZone>
					<includeZeroSalesRows>false</includeZeroSalesRows>
				</reportQuery>
			</rval>
		</runReportJobResponse>
	</soap:Body>
</soap:Envelope>

Call getReportJob (Poll JOB Status)

Second step is to make sure is Report Status check. For large report it make take several seconds or minutes before you can get Download URL (See Next Step). If you try to call Next step before Report is ready then you may get error. So better to add Polling logic.

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract status of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for <rval>IN_PROGRESS</rval> node anywhere in SOAP response XML and extract value (i.e. IN_PROGRESS )

//*[local-name() = 'rval']

Extract Single XML Node Value from SOAP API Response – Use XPATH in SSIS

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxXmWqmq
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 524
Expect: 100-continue
Accept-Encoding: gzip, deflate

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
   <soapenv:Header>
      <v20:RequestHeader>
         <v20:networkCode>238897396</v20:networkCode>
         <v20:applicationName>ZappySysApp</v20:applicationName>
      </v20:RequestHeader>
   </soapenv:Header>
   <soapenv:Body>
      <v20:getReportJobStatus>
       <v20:reportJobId>10000297829</v20:reportJobId>
      </v20:getReportJobStatus>
   </soapenv:Body>
</soapenv:Envelope>

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:47 GMT
Expires: Fri, 02 Jun 2017 21:23:47 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 451

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<requestId>d52cb3a3d68af24aec2f9b0284379700</requestId>
			<responseTime>803</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<getReportJobStatusResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<rval>COMPLETED</rval>
		</getReportJobStatusResponse>
	</soap:Body>
</soap:Envelope>

Call getReportDownloadURL (Get Download URL)

Once you get status = COMPLETED in above API Call (i.e.  getReportJobStatus) then you ready to call getReportDownloadURL  to fetch URL which you can download.

Here is the API sample Request and Response

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxBp84ltTNyVRb9hlUEWx
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 581
Expect: 100-continue
Accept-Encoding: gzip, deflate

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v20="https://www.google.com/apis/ads/publisher/v201702">
   <soapenv:Header>
      <v20:RequestHeader>
         <v20:networkCode>238897396</v20:networkCode>
         <v20:applicationName>ZappySysApp</v20:applicationName>
      </v20:RequestHeader>
   </soapenv:Header>
   <soapenv:Body>
      <v20:getReportDownloadURL>
       <v20:reportJobId>10000297829</v20:reportJobId>
       <v20:exportFormat>CSV_DUMP</v20:exportFormat>
      </v20:getReportDownloadURL>
   </soapenv:Body>
</soapenv:Envelope>

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:49 GMT
Expires: Fri, 02 Jun 2017 21:23:49 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 1039

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
	<soap:Header>
		<ResponseHeader xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<requestId>8e5ac0bbeb8e88c180ea442fc5378d99</requestId>
			<responseTime>1957</responseTime>
		</ResponseHeader>
	</soap:Header>
	<soap:Body>
		<getReportDownloadURLResponse xmlns="https://www.google.com/apis/ads/publisher/v201702">
			<rval>https://storage.googleapis.com/dfp-report-export/caxxxxxxxx-xxxxxxxxxxxxxxb?GoogleAccessId=3769xxxxxxxxxxx-qxxxxxxxxxxxxxxxxsqdb5dvi8g@developer.gserviceaccount.com&amp;Expires=1496438929&amp;Signature=vnhCiAIqd%2BFgWCtAHo9roFOI3W%2BveUTYuSvyBQ9gtMI2HDTXOCVDh7wzUV%2BPBJZZghaL6k8wKYq33GA4NpjZVNYuaAgPLWgGwAUPdtioHRi%2Bt3Eymp%2FAPRGIoos0ekP3O7l%2FPNPxenC6UTP087HV3j06V%2Fuui9ixpKZWGCK8r56oUPj6S8q1hG95Ns3d3sEcuz3%2BkShu6GYG5mqir5vjtZPuzwsr%2F96G5Kk123mke%2B%2FIVRJGoqnCMghNWtIG1yR1IdZx%2FIpit%2FnaJ9HP6IvAr3oMIXkQ3cHv%2FtOhXGy9dTG34q%2FKDbtkSJAayHgjlSLPGKucygcXBhUrbywBu8Y77A%3D%3D</rval>
		</getReportDownloadURLResponse>
	</soap:Body>
</soap:Envelope>

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract status of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for <rval>https://storage.googleapis.com/dfp-report-export/caxx……….</rval> node anywhere in SOAP response XML and extract value (i.e. URL)

Download File Using REST API Task

Once you get URL its time to download it. You can use REST API Task to download the file. On Response Setting Tab specify full file path (e.g. c:\report_csv.gz) and make sure Binary option is checked (Found next to the Save Path)

Making things dynamic

If you want to supply certain parameters at runtime (e.g. from SSIS Variable) rather than hard code then you can use Variable Placeholders any where in Request Body, URL or Headers. Click Insert Variable option found in Edit screen

Here is an example of dynamic URL (API Version stored in SSIS Variable)

https://ads.google.com/apis/ads/publisher/{{User::ApiVer}}/NetworkService

Deployment to Production

When you ready to deploy your SSIS Package to production make sure to change Test NetworkCode supplied inside Request Body to your Production Ad Network. Also Set RefreshToken Property of OAuth Connection Manager (Get it from Backup file saved when you created OAuth connection first time). Once you set RefreshToken of connection manager you won’t have to Login again to get a new token.

Conclusion

Traditionally if you wanted to integrate Google DFP API (DoubleClick API) calls inside your ETL workflow then it required some sort of coding effort (e.g. Use C# SDK, JAVA SDK or Python SDK). Coding effort is not only time consuming and expensive but also hard to maintain for any non-coder (e.g. ETL Developer or DBA).
Thanks to SSIS PowerPack which changed the way developers do API integration. Now you can reduce time to implement and total cost of you API integration project significantly by using drag and drop approach. This approach is not only easy to maintain but also fast and high quality. You can use ZappySys REST/SOAP SSIS Connectors and SSIS Tasks for any API integration project such as Google DFP API. Try SSIS PowerPack for free to explore possibilities. If you have any issue with your integration Contact Support

The post How to call Google DFP API with SSIS – DoubleClick appeared first on ZappySys Blog.

In this tutorial we will use REST API Task to call some ad-hoc API (e.g. get File List from Google Drive) and save output into Variable or File. We will use OAuth connection along with ZappySys JSON Source connector to read data from Google SpreadSheet (Use Drive API) and load into SQL Server (Export Google SpreadSheet to CSV).

Create Google API Project

First step to access any Google API is create an API Project in Google Console. If you don’t want to go through this then Skip this Step-1 and in the next section select Default OAuth App option on OAth Connection Manager (This is the most easiest option for now unless you want to use your own OAuth App).

Check this article for step-by-step instructions. When you follow these instructions make sure you enable Google Drive API (In the article it shows how to enable YouTube API as an example but you will need to enable Drive API for this article).

Create OAuth Connection Manager in SSIS

Once you create Google API project and obtained Client ID and Client Secret your next step is to create OAuth Connection Manager in SSIS. ZappySys OAuth connection manager comes with many predefined OAuth Providers (e.g. Facebook, Twitter, Google etc) but you can also define custom OAuth settings for any OAuth enabled API.

To create SSIS OAuth 2.0 Connection for Google API perform following steps.

1. Download and Install SSIS PowerPack
2. Create new SSIS Package
3. Right click in Connection Manager Area and Click “New Connection”
4. When prompted select ZS-OAUTH connection type
5. On the OAuth Connection Manager Select Provider=Google Sheets + Drive.
6. You can keep “Use Default App” selected or choose Custom App. Click here to learn – how to register Custom Google App
7. If you choose the Custom App option then enter the below scopes (or click Select Scopes). If you choose the default app then it’s not required.

   This will allow read / write access to Drive Files or Sheet (E.g. Reading file content / Export file to CSV). You can use Scope browser to see many other available permissions.

   https://www.googleapis.com/auth/drive
   https://www.googleapis.com/auth/drive.file
   https://www.googleapis.com/auth/drive.readonly
   https://www.googleapis.com/auth/spreadsheets
   https://www.googleapis.com/auth/spreadsheets.readonly

   Click Generate Token button.

   

   SSIS OAuth Connection Manager – Access Google Drive API using OAuth 2.0

8. You will see UI as below… Click Accept (You may have to scroll to see that button sometimes)
9. Click Test to see connection is working.

NOTE: If you don’t want to use Default OAuth App provided by ZappySys then select “Use Custom OAuth App” option (App created in previous section) specify your ClientID, ClientSecret.

Find Google Sheet ID for API call

Before we can read or write Sheet data in SSIS we need to know Google Driver File ID. We will use this ID in next few sections.

Easiest way to find File ID is look at the URL like below. If you need to know ID at runtime dynamically then refer last section.

Get Google Drive File ID for API call (Sheet ID)

Reading Google SpreadSheet Data in SSIS

Basically there are two ways you can export / read Google SpreadSheet data.

• Method-1 : Read Google SpreadSheet using SSIS JSON / REST API Source
  • In this approach we directly pull data from SpreadSheet using Core Google SpreadSheet.Values API to read particular Tab from Sheet and you can also specify Range. This is most effective way to read. Make sure you have v2.5.0.10807 or higher version (released after Aug 03 2017). New version added 2D array transformation options to parse multi-dimensional arrays found in SpreadSheet API JSON response.
• Method-2 : Read Google SpreadSheet using Flat File Source (as CSV)
  • Another approach is export SpreadSheet as CSV format. Then use Flat File Source to read that data and load into Target (E.g. SQL Server)

Method-1 : Read Google SpreadSheet using SSIS JSON / REST API Source

Now lets look at real steps to read Google SpreadSheet. To get values from specific tab and specific cell-row range you have to use Google SpreadSheet.Values API

In this example we will load below Google Spreadsheet into SQL Server

Step-By-Step : Loading Google SpreadSheet into SQL Server using SSIS

1. Drag and drop data flow on the control flow designer surface
2. Double click data flow and you will see SSIS Toolbox refreshed with new components
3. Drag ZS JSON Source from SSIS Toolbox onto Data flow designer surface
4. Double click JSON SOurce and configure below setting
   1. Enter your Spreadsheet URL in the following format
      

      https://sheets.googleapis.com/v4/spreadsheets/{your-file-id}/values/{cell-range}

      For example purpose we will use this public SpreadSheet provided by Google. If spreadsheet fileid is 1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms (see previous section how to get fileid) and your tab name is Class Data and you like to extract data from A to F columns starting from 2nd row then your Cell range in the URL can be ‘Class Data’!A2:F
      Here is the actual URL we will use for demo
      

      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/'Class Data'!A2:F

      — OR —
      You can also enter like below (If you omit Tab name then first table is used)
      

      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A1:F
      
      --OR-- Use below URL for to supply Column names manually (Start from A2)
      
      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A2:F

      You can also enter like below, specify first row and last row
      

      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A1:F101

   2. Check Use credentials and select OAuth connection we created in previous section
   3. Click on Select Filter and select values node and click OK  or you can directly enter following filter.
      

      $.values[*]

      

      Configure SSIS JSON/REST API Source – Read data from Google SpreadSheet API

   4. Now click on array Transformation tab. Now we will show you two different ways to configure column names.
   5. Method-1: Auto Detect Column Names (added in version 3.1.2)
      In recent version we added column less array transformation where we have option to detect first record as column name.
      1. Select Transform Type as Column less array
      2. Check option First line has column names like below example (Ignore direct Sample JSON – We provided for demo only. In your case it will be URL)
         

         JSON With 2D Array Pattern – Column names in First Row (Google Sheets API Pattern)

   6. Method-2: Manually Enter Column Names (For old version)
      1. Select Transform simple 2D array option from dropdown
      2. Check Specify columns list manually
      3. Enter column names (Must match the order and count based on range you selected)
         

         Student Name,Gender,Class Level,Home State,Major,Extracurricular Activity

         

         Configure SSIS JSON/REST API Source 2D Array Transformation for Google SpreadSheet API

   7. Now click Preview to see the data
      

      Configure SSIS JSON/REST API Source –
      Preview Google SpreadSheet Data

   8. Click OK to save UI
   9. Attach your JSON Source to target (e.g. OLEDB Destination for SQL Server Table).
   10. Configure Destination
       

       Configure SQL Server destination for Google SpreadSheet to SQL Data load

   11. Execute SSIS Package to load Google SpreadSheet data int SQL Server
       

       SSIS Example -Loading data from Google SpreadSheet into SQL Server Table

Method-2 : Read Google SpreadSheet using Flat File Source (as CSV)

Once we have Google Drive fileId we can perform following actions to export data into CSV file format. After that you can easily read CSV using SSIS FlatFile Source and load into SQL Server or other target.

Export Google SpreadSheet as CSV file using SSIS REST API Task

Here are the steps to export Google Drive SpreadSheet to CSV file

1. Just like previous section configure REST API Task
2. Enter following URL (See we used FileId from SSIS variable. You can hardcode too.
   

   https://www.googleapis.com/drive/v3/files/{{User::varFileId}}/export?mimeType=text/csv

3. Goto response tab. Check Save response and select “Save to file option”.
4. Enter file path.
5. Response character set select utf-8 (This will allow Unicode characters in response)
6. Click Test to see preview
   

   Export Google SpreadSheet to CSV file format using SSIS – Call Google Drive API to Export

7. Click OK to save

Load Google SpreadSheet data into SQL Server using SSIS (REST API Call)

Once you export SpreadSheet to CSV file you can easily consume it to load into SQL Server or any Target (e.g. Oracle, MySQL) using Native SSIS FlatFile Source and OLEDB/ADO.net Destination.

Write data to Google Spread Sheet

Now lets look at how to write to Google SpreadSheet. To write multiple rows in a single request you may use batchUpdate API. Also check this link to learn more.

Below is sample request to write data to columns and some data to rows (set majorDimension = COLUMNS to write vertical and use ROWS to write horizontally. You can mix multiple ways just like below example)

POST 
https://sheets.googleapis.com/v4/spreadsheets/1lkMEgu0zm2Q-cnPeoNZp4hqeIuASxXSpsrxtDDV89tg/values:batchUpdate
Authorization: Bearer ya29.Gl3WBFb3xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: application/json
Accept: */*

{
  "valueInputOption": "USER_ENTERED",
  "data": [
    {
      "range": "Sheet2!A1:A4",
      "majorDimension": "COLUMNS",
      "values": [
        ["Item", "Wheel", "Door", "Engine"]
      ]
    },
    {
      "range": "Sheet2!B1:D2",
      "majorDimension": "ROWS",
      "values": [
        ["Cost", "Stocked", "Ship Date"],
        ["$20.50", "4", "3/1/2016"]
      ]
    }
  ]
}

Write data to Google SpreadSheet using REST API Task (OAuth 2.0 Connection)

Update Google Sheet Cells – Single Range (Set Values for multiple cells)

Now let’s look at the below example of updating multiple cells in google Sheet. For more information, you can see the example here. For parameter detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve Spread Sheet Update scenario. For simple example lets use REST API Task.

1. Drag and Drop ZS REST API Task on the control flow designer
2. Edit Task and select URL from Connection Method
3. Select OAuth Connection from the dropdown (Same connection we created in the previous section)
4. Enter the URL as below. Change Sheet ID, Range designator and API URL Parameters as per your need.
   https://sheets.googleapis.com/v4/spreadsheets/[Sheet-ID-Here]/values/[Range-Here]?[Parameters-Here] Example URL (some part masked with xxxxxxxx):  
   

   https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4?valueInputOption=USER_ENTERED

5. Change Request Method to PUT
6. Change Request Content type to application/json
7. Enter Request Body as below (Assuming we need to update 4 cells)
   

   {
   "majorDimension": "ROWS",
     "values": [
      ["row1_cellA","row1_cellB"],
      ["row2_cellA","row2_cellB"],
     ]
   }

8. Click Test Request see it works.
   

   Calling Google Sheet API to Update Multiple Cell Values

Update Google Sheet Cells – Multiple Ranges

If you like to update values which needs to define multiple ranges then you need to use batchUpdate API. Import changes are you can’t include Range in the URL like previous example of Single Range Update. Also you have to use POST method and body with multiple ranges.

Example of sheet update with multiple range

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHxxxxxxxxxxxxxxxc/values:batchUpdate?valueInputOption=USER_ENTERED
Content-Type: application/json

>>>> BODY <<<<<

{
  "data": [
    {
      "range": "Sheet1!A2:B3",
      "majorDimension": "ROWS",
      "values": [
        [
          "row1_cellA",
          "row1_cellB"
        ],
        [
          "row2_cellA",
          "row2_cellB"
        ]
      ]
    },
    {
      "range": "Sheet1!D2:E3",
      "majorDimension": "ROWS",
      "values": [
        [
          "row1_cellD",
          "row1_cellE"
        ],
        [
          "row2_cellD",
          "row2_cellE"
        ]
      ]
    }
  ]
}

Write SQL Server data to Google Sheet

Now let’s look at real world example. How to write data from SQL Server table or other source and send to Google Sheet.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox
   
2. Drag source (e.g. OLEDB Source) and configure it to read from source like SQL Server Table
   

   OLE DB Source – Drag and Drop

3. Drag ZS JSON Generator.
   1. Connect Source to JSON Generator (blue arrow)
   2. Double click it to configure
   3. Select Output Mode as Single Dataset Array option for Mode. Enter Some Batch value (e.g. 500) this will make sure we send 500 rows at a time to Google API call. This way if we have many rows its not rejected by API call.
   4. Click Add element (Select Static Element) . Name: majorDimension and Value: ROWS
   5. Now click Add Document Array icon, name as values
   6. check Treat as 2D array option and click OK to save
   7. Now under values node click Add elements > Select Multiple Columns and Add columns you like to add.
   8. Click OK to save JSON GeneratorBelow is just an example How you can use Batch Settings to create JSON with N rows in each JSON document.
      
4. Now drag ZS Web API destination
   1. Connect JSON Generator to Web API destination (blue arrow)
   2. Double click it to configure
   3. Select same OAuth connection manager created in earlier section (Assuming you had all scopes with write permission)
   4. Now select Input column for Body (JSON Generator Output)
   5. Enter API call URL as below (Replace ID with your own id)
      

      https://sheets.googleapis.com/v4/spreadsheets/{your-file-id}/values/Sheet1!A2?valueInputOption=USER_ENTERED

   6. Select Request Method as PUT
   7. Select content type as application/json
   8. You can enter sample Body As below.
      

      {
      "majorDimension": "ROWS",
        "values": [
         ["Bob","bob@mycompany.com",55],
         ["Sam","sam@mycompany.com",57]
        ]
      }

   9. Click Test Request / Response button. Check your Sheet in browser after it see it updates?
   10. If you see 200 OK response means we are good to go.
   11. Run package and see your real data updated in Sheet.

Format Cells using Google Sheet API

Now let’s look at an example to format cell using Google Sheet API (e.g. Change Font, Color, Border, Background). For more information review this API documentation.

To format Sheet you first need to get SheetID  (keep in mind this is not same as spreadsheetId which you used in URL). SheetId is basically numeric ID of your Tab in WorkSheet.

Get SheetID

Request

GET https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHtxxxxxxxxxxxxxx

Response

{
  "spreadsheetId": "1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc",
  "properties": {
    "title": "Test.xlsx",
    "locale": "en",
    "autoRecalc": "ON_CHANGE",
    "timeZone": "America/Los_Angeles",
    "defaultFormat": {
      "backgroundColor": {
        "red": 1,
        "green": 1,
        "blue": 1
      },
      "padding": {
        "right": 3,
        "left": 3
      },
      "verticalAlignment": "BOTTOM",
      "wrapStrategy": "OVERFLOW_CELL",
      "textFormat": {
        "foregroundColor": {},
        "fontFamily": "Calibri",
        "fontSize": 11,
        "bold": false,
        "italic": false,
        "strikethrough": false,
        "underline": false
      }
    }
  },
  "sheets": [
    {
      "properties": {
        "sheetId": 1896331083,
        "title": "Sheet1",
        "index": 0,
        "sheetType": "GRID",
        "gridProperties": {
          "rowCount": 1000,
          "columnCount": 26,
          "rowGroupControlAfter": true,
          "columnGroupControlAfter": true
        }
      }
    }
  ],
  "spreadsheetUrl": "https://docs.google.com/spreadsheets/d/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc/edit"
}

Notice that in above response we see “sheetId”: 1896331083. This ID we will use in the Update API to format Cells.

Update Borders Example

Here is the example of Google Sheet Formatting API updateBorders Request

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSxxxxlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "updateBorders": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 0,
          "endRowIndex": 10,
          "startColumnIndex": 0,
          "endColumnIndex": 6
        },
        "top": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
        "bottom": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
        "innerHorizontal": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
      }
    }
  ]
}

Response

{
  "spreadsheetId": "1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc",
  "replies": [
    {}
  ]
}

Update Cell Font Style (Bold, Style, Color, Background)

Now let’s look at how to use repeateCellRequest to apply certain type of formatting to multiple cells. This may be color, font style, background color.

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3xxxxxx4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "repeatCell": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 0,
          "endRowIndex": 1
        },
        "cell": {
          "userEnteredFormat": {
            "backgroundColor": {
              "red": 0,
              "green": 0.5,
              "blue": 0.7,
              "alpha": 1.3
            },
            "textFormat": {
              "foregroundColor": {
                "red": 1
              },
              "bold": true,
              "italic": true,
              "strikethrough": true,
              "underline": true
            }
          }
        },
        "fields": "userEnteredFormat"
      }
    }
  ]
}

In above example we have applied textFormat and backgroundColor but you can use any of below sections.

Set Currency Formatting

Here is an example request to set currency formatting

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "repeatCell": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 1,
          "startColumnIndex": 4,
          "endColumnIndex": 5
        },
        "cell": {
          "userEnteredFormat": {
            "numberFormat": {
              "type": "CURRENCY",
              "pattern": "$#,##0.00"
            }
          }
        },
        "fields": "userEnteredFormat.numberFormat"
      }
    }
  ]
}

Set Cell Validation

You can also set cell validation by calling following API call. This will restrict user’s ability to enter wrong data in certain cells.

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "setDataValidation": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 1,
          "startColumnIndex": 0,
          "endColumnIndex": 6
        },
        "rule": {
          "condition": {
            "type": "ONE_OF_LIST",
            "values": [
              {
                "userEnteredValue": "Red"
              },
              {
                "userEnteredValue": "Green"
              },
              {
                "userEnteredValue": "Yellow"
              }
            ]
          },
          "inputMessage": "Color must be Red, Yellow or Green",
          "strict": true,
          "showCustomUi": true
        }
      }
    }
  ]
}

Dynamically obtain File ID for Google Drive REST API call

Once OAuth connection is defined you can now call any valid API from Google based on Permission you requested in OAuth Connection. OAuth connection takes care of renewing you Token and passing Authorization as per OAuth standard. All these protocol complexity is hidden from you.

First step to get data from Google SpreadSheet using REST API call is get fileId of the Google Drive File you want to read. The easiest way to know file ID is call following API. Below API call returns name, ID and other information about file(s). You can get only fileID and use that in Next Section to export Google SpreadSheet to CSV format.

https://www.googleapis.com/drive/v3/files

OR

https://www.googleapis.com/drive/v3/files/?q=name%3D'Customers'

NOTE: For more information about searching files in Google drive / getting list of files using search expression language check this link. Make sure expression is URL encoded (e.g. name=’Customers’ should be passed as name%3D’Customers’)

Above API returns response as below. Notice the name and fileId. We will use fileId when calling export API in next section.

{
 "kind": "drive#fileList",
 "files": [
  {
   "kind": "drive#file",
   "id": "1lkMEgu0zm2Q-cnPeoNZp4hqeIuASxXSpsrxtDDV89tg",
   "name": "Customers",
   "mimeType": "application/vnd.google-apps.spreadsheet"
  },
  {
   "kind": "drive#file",
   "id": "0B6UyXr6ZlIFTc3RhcnRlcl9maWxl",
   "name": "How to get started with Drive",
   "mimeType": "application/pdf"
  }
 ]
}

If you want to automate retrieval of fileId from document name then perform the following steps.

1. In the SSIS package Control flow. Drag and drop ZS REST API Task from SSIS toolbox
2. In the Request Access mode, select “URL from Connection” option
3. Select OAuth connection manager (Created in Section-2)
4. Enter URL as below (It will return any file name equals to Customers (%3D is encoded value for = sign)
   

   https://www.googleapis.com/drive/v3/files/?q=name%3D'Customers'

5. Click Test (You will see file information)
   

   Get Google Drive File List in SSIS – Call Google Drive API to search file by name or expression

6. If you want to extract only fileID from full JSON then go to response tab
7. Select Format=JSON, Enter following JSONPath expression to extract only fileId from first record
   

   $.files[0].id

8. Check save response content, select SSIS variable where you want to save fileId as below. Now click Test to see preview of response. You will notice now only fileId will be returned from full JSON response.
   

   Get Google Drive fileid from file name using API call in SSIS

9. Click OK to save task

Conclusion

Google Drive API and Google Sheets API provides a great way to automate file related functionality. However, to call Google API  you have to use SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Google API call via simple drag and drop approach without coding. Try out yourself see how long it takes to call virtually any REST API.

The post How to read / write Google SpreadSheet using SSIS appeared first on ZappySys Blog.