In this tutorial, we will use the REST API Task to call some API. We will use the OAuth connection along the REST API Task to write / update data to Google Spreadsheet.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Google API.

Component Mention in this article

	JSON Generator Transform
	Web API Destination
	REST API Task

Create a Google API Project

The first step to access any Google API is to create an API Project in Google Console. If you don’t want to go through this then Skip this Step-1 and in the next section select the Default OAuth App option on OAth Connection Manager (This is the easiest option for now unless you want to use your own OAuth App).

Check this article for step-by-step instructions. When you follow these instructions make sure you enable Google Sheet API.

Create OAuth Connection Manager in SSIS

Once you create a Google API project and obtained a Client ID and Client Secret your next step is to create OAuth Connection Manager in SSIS. ZappySys OAuth connection manager comes with many predefined OAuth Providers (e.g. Facebook, Twitter, Google, etc) but you can also define custom OAuth settings for any OAuth enabled API.

To create SSIS OAuth 2.0 Connection for Google API perform the following steps.

1. Download and Install SSIS PowerPack.
2. Create a new SSIS Package.
3. Right-click in Connection Manager Area and Click “New Connection”.
4. When prompted select ZS-OAUTH connection type, On the OAuth Connection Manager Select Provider=Google.
5. Click here to create credentials. Then select the “Use Custom OAuth App” option and enter your ClientID, ClientSecret.
6. In the Scopes enter below scopes (or click Select Scopes). This will allow read/write access to Drive Files or Sheet (E.g. Reading file content / Export file to CSV). You can use the Scope browser to see many other available permissions. For more information click here.
   

   https://www.googleapis.com/auth/drive
   https://www.googleapis.com/auth/drive.file
   https://www.googleapis.com/auth/drive.readonly
   https://www.googleapis.com/auth/spreadsheets
   https://www.googleapis.com/auth/spreadsheets.readonly

7. Click on the Generate Token button.
   

   SSIS OAuth Connection Manager – Access Google Drive API using OAuth 2.0

8. You will see UI as below… Click Allow (You may have to scroll to see that button sometimes).
9. Click on Test Connection to see the connection is working and click on the OK button to save configure setting UI.

Find Google Sheet ID for API calls

Before we can write or update Sheet data in SSIS we need to know Google Driver File ID. We will use this ID in the next few sections.

The easiest way to find File ID is to look at the URL like below. If you need to know ID at runtime dynamically then refer to the last section.

Get Google Drive File ID for API call (Sheet ID)

Create Variables to make dynamic URL

1. Here, we are going to make a Dynamic URL value.
2. In Visual Studio Project Right Click on Design Panel, Select Variables.
   

   Create Variables

3. Let’s Create Variables with correct DataType and Value. In variable “varRange” set only the starting point(Sheet1!A2) of data if you don’t know the ending point and leave the first-row for giving header row later or click here.
   

Write SQL Server data to Google Spreadsheet

Now let’s look at a real-world example. How to write data from SQL Server table or other source and send to Google Sheet using Web API Destination.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Dragging and dropping Data Flow Task into Control Flow

2. Double click on the Data Flow task to see the Data Flow designer surface.
3. Here, In Visual Studio, drag and drop the OLE DB Source, JSON Generator Transform and Web API Destination in the design surface and join the components with the blue arrow.
   

   Drag and Drop OLE DB Source, JSON Generator Transform, and Web API Destination

4. Here, you can use our Free ZS Dummy Data Source too. Click here for more information.
5. But, we are going through OLE DB Source so we need OLEDB Connection.

Create OLE DB Connection

1. Lets, create the OLE DB Connection.
2. Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
   

   Create OLE DB Connection

3. Now, click on New Button to create Connection.
   

   Create New Connection

4. Let’s Configure Connection Manager, just Follow steps one by one as we have created.
   

   Configure OLE DB Connection

5. Click on Test Connection to see correct configure it.
6. Click on the OK button to save connection configure setting UI.

Configure JSON Generator Transform

1. Double click on JSON Generator Transform to configure it.
2. Set Output Mode to Single Dataset Array. Right-click on Mappings, select Add Static Element then enter Name: “range” and Select variable we have created for value.
   

   Configure JSON Generator Transform – Add Static Value

3. Again, Right-click on Mappings, select Add Static Element then enter Name: “majorDimension” and Value: “ROWS“.
   

   Configure JSON Generator Transform – Add Static Value

4. Now, Right-click on Mappings, select Add Document Array, Check to Treat as 2D array option and enter Element Name as “values“.
   

   Configure JSON Generator Transform – Add Document Array

5. Then, right-click on Document Array, select Add elements > Select Add Multiple Columns and check on columns you like to add.
   

   Configure JSON Generator Transform – Add Element

6. Click on the OK button to save JSON Generator Transform configure setting UI.

Configure Web API Destination

1. Double click on Web API Destination to configure it.
2. Select the OAuth connection manager created in an earlier section (Assuming you had all scopes with write permission).
3. In the Setting tab,
   1. In URL, Generate Dynamic URL like below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4:append?valueInputOption=USER_ENTERED

   2. Change Request Method to POST.
   3. Select JSON Generator variable into Body (Request Data) or you can select Input column for Body (JSON Generator Output).
   4. Set application/JSON to Body Content-type.
      

      Configure Web API Destination

4. Click on the OK button to save Web API Destination configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet. Now update header row values for click here.
   

   Run or Execute Package and verify data into Google Spreadsheet

Append data to Google SpreadSheet using REST API Task

Now let’s look at how to write to Google SpreadSheet. To write multiple rows in a single request. Also, check this link to learn more.

For more information, you can see the example here. For parameter, detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve the Spreadsheet write scenario. For a simple example, let’s use the REST API Task.

1. Drag and Drop ZS REST API Task on the control flow designer.
   

   Drag and Drop REST API Task

2. Double click on ZS REST API Task to configure it.
3. In the Request Settings tab,
   1. set Request URL Access Mode to Url from a connection. Select OAuth Connection from the dropdown (Same connection we created in the previous section).
   2. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A1:append?valueInputOption=USER_ENTERED

   3. Change Request Method to POST.
   4. Enter Request Body as below
      
      Note: For example purpose, we included header line in data cells but in the real-world header may already exist in the sheet.

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["Item", "Cost", "Stocked", "Ship Date"],
          ["Wheel", "$20.50", "4", "3/1/2016"],
          ["Door", "$15", "2", "3/15/2016"],
          ["Engine", "$100", "1", "3/20/2016"],
          ["Totals", "=SUM(B2:B4)", "=SUM(C2:C4)", "=MAX(D2:D4)"]
        ]
      }

   5. Change Request Content type to application/JSON.
      

      Write data to Google SpreadSheet using REST API Task (OAuth 2.0 Connection)

4. Click on the OK button to save REST API Task configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.
   

   Execute REST API Task and verify Google Spreadsheet data

Update Google Sheet Cells

Now let’s look at the below example of updating multiple cells in google Spreadsheet. For more information, you can see the example here. For parameter, detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve the Spreadsheet Update scenario. For a simple example, let’s use the REST API Task.

Update Google Sheet header cells manually

1. Drag and drop ZS REST API Tasks on the control flow designer.
2. Double click on the First REST API Task to configure it.
3. In the Request Settings tab,
   1. Enter the URL below. Change Range designator(including Sheet name: Sheet1!A1) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A1?valueInputOption=USER_ENTERED

   2. Change Request Method to PUT.
   3. Enter Request Body as below
      

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["CustomerID", "CustomerName"]
        ]
      }

4. Click on the OK button to save REST API Task configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.
   

Single Range (Set Values for multiple cells)

1. Follow the previous section(write data to Google Spreadsheet) to update sheet data.
2. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4?valueInputOption=USER_ENTERED

   2. Change Request Method to PUT.
   3. Enter Request Body as below
      

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["Laptop", "$2000.50"],
          ["Mobile", "$1150"]
        ]
      }

3. That’s all, click on the Test Request/Response button and verify data into Google spreadsheet.
   

   Calling Google Sheet API to Update Multiple Cell Values

Clear data to Google SpreadSheet using REST API Task

Clear a sheet of all values while preserving formats

Lets clear a sheet of all values by REST API Task, Before processed we need sheet Id. So let’s get the SheetId by API Call. For more information about click here.

1. Drag and Drop two ZS REST API Tasks on the control flow designer and rename(Get Sheet ID and Clear Google Sheet of all values) it.
2. Double click on the First REST API(Get Sheet ID) Task to configure it.
3. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc

   2. Change Request Method to GET.
      REST API Task – Get Google Sheet ID
4. Now go to Response Settings Tab, Select Response content Type Json and Write Response Content Filter Expression. Check on Save Response Content. Select Save Mode to Variable and Create a New variable or select it.
   Example of Response Content Filter Expression for the first Sheet : 
   

   $.sheets.[0].properties.sheetId

   

   REST API Task – Save Google Sheet ID into Variable

5. Now, click on the Test Request/Response to check configures are ok.
6. Click on the OK button to save REST API Task configure setting UI.
7. Double click on the second REST API(Clear Google Sheet of all values) Task to configure it.
8. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}:batchUpdate

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc:batchUpdate

   2. Change Request Method to POST. Enter Request Body as below and change a variable parameter as per your need.
      

      {
        "requests": [
          {
            "updateCells": {
              "range": {
                "sheetId": {{User::varGoogleSheetID}}
              },
              "fields": "userEnteredValue"
            }
          }
        ]
      }

   3. Change Request Content type to application/JSON.
      
      

      REST API Task Configure – Clear sheet of all values

9. Click on the OK button to save REST API Task configure setting UI.
10. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.

Clears range cell values from a spreadsheet

1. Double click on REST API Task to configure it. Then, follow the way just like the previous section configures REST API Task.
2. Here you need to set the value of Variable “varSpreadsheetValues” to “clear” or you can change it directly in URL append to clear and value of variable “varRange” (Change cell range and sheet name) as per your need. For more information about it click here.
   1. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4:clear?valueInputOption=USER_ENTERED

3. Change Request Method to POST.
4. Enter Request Body as below
   

   {}

5. Change Request Content type to application/JSON.
6. That’s all, click on the Test Request/Response button and verify data into Google spreadsheet.

Conclusion

Google Drive API and Google Sheets API provide a great way to automate file-related functionality. However, to call Google API  you have to use the SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Google API call via simple drag and drop approach without coding. Try out yourself see how long it takes to call virtually any REST API.

References

Finally, you can use the following links for more information:

• JSON Generator Transform
• Web API Destination
• REST API Task
• Help File:
  • JSON Generator Transform
  • Web API Destination
  • REST API Task

The post How to write/update Google Sheet in SSIS appeared first on ZappySys Blog.

In the previous article, we learned how to get data from AdWords in SSIS and basically control your AdWords account programmatically, outside the dashboard. In this one, we will continue on how to download AdWords report by using Google Ads API, ODBC, and ZappySys XML Driver. Let’s not waste our time and cut to the chase!

Prerequisites

1. Google Ads account up and running.
2. ZappySys ODBC PowerPack installed.

Step-by-Step – Download AdWords report

Get Developer Token

The first important step to access any Google Ads Account is to request a developer token. Click this link for step by step information. Watch a short video to get an overview of the entire process to access data via API.

Get Client Customer Id

Login into a non-manager account at https://ads.google.com and get your Client Customer Id:

Getting Google Ads (formerly AdWords) Client Customer Id

Copy it to a notepad, we will use it later.

Open ODBC Data Sources Manager

Search for "odbc" in your start menu and click on ODBC (64 bits). If you can't find this then you can also go to Start Menu > ZappySys > ODBC PowerPack > Click on ODBC Data Sources (64-Bit).

Open ODBC Data Source

Create ODBC Data Source based on ZappySys XML Driver

Select tab User DSN if you want to create a data source only for your user, or select System DSN to create a data source for all users or for OS services (e.g. SQL Server Agent):

Creating ZappySys XML Driver based data source

Configure authentication in the ODBC data source

Let’s authenticate first:

1. Select OAuth in Connection Type property.
2. Click Click to Configure link.
3. In OAuth Provider property select Google.
4. Then either select Use Default OAuth App or Use Custom OAuth App. For the latter, you will need to create a Google OAuth App yourself. It’s perfectly safe to use the default application, but if your company policy does not allow it, you may consider creating it yourself.
5. In Scopes enter:
   https://www.googleapis.com/auth/adwords
6. Then hit Generate Token and hit allow once asked.

Download AdWords report – Configuring authentication in ODBC data source

Proceed with HTTP request configuration

Then go on with configuring the data source:

Configuring HTTP request in ODBC data source to download an AdWords report

1. Give your data source a name
2. Then input this URL to download the report:
   https://adwords.google.com/api/adwords/reportdownload/v201809
   If the above URL does not work, get the working one from Google’s documentation.
3. Input POST as HTTP Request Method.
4. In Body enter this text:
   __rdxml=<reportDefinition xmlns="https://adwords.google.com/api/adwords/cm/v201809">
<selector>
<fields>CampaignId</fields>
<fields>Impressions</fields>
<fields>Clicks</fields>
<fields>Cost</fields>
<predicates>
<field>CampaignStatus</field>
<operator>IN</operator>
<values>ENABLED</values>
<values>PAUSED</values>
</predicates>
</selector>
<reportName>My Campaign Performance Report</reportName>
<reportType>CAMPAIGN_PERFORMANCE_REPORT</reportType>
<dateRangeType>LAST_7_DAYS</dateRangeType>
<downloadFormat>XML</downloadFormat>
</reportDefinition>
   This will display the data of CAMPAIGN_PERFORMANCE_REPORT report (campaign Id, impressions, clicks, and cost) for the last 7 days, for all campaigns that are enabled or paused. Make sure to visit the Google documentation page, which contains the information about how to download a different kind of report, how to configure predicates, etc.
5. In Body Content Type property select Form (application/x-www-form-urlencoded) option.
6. In HTTP Headers enter your developerToken and clientCustomerId retrieved in previous steps.

Select data filter in ODBC data source

Then hit Select Filter button and select row node, so that Array Filter is set to $.report.table.row[*]

Filtering on AdWords report data

If asked whether you want to treat the row element as an array, click “yes”. You can also go to the Xml Array Handling tab and see if the element row is specified there.

The Results

To view the results press Preview tab and then click Preview Data:

Previewing AdWords report data

Now you can use this data source and get the data into an Excel sheet, Power BI report, SQL Server (would need to create a data source in ZappySys Data Gateway instead) or elsewhere.

Reusing the data source

If your client application supports, you can override data source settings, such as HTTP body, in SQL query using WITH clause. In that case, you won’t need to create multiple data sources for different kind of reports; e.g.:

SELECT * FROM $ (RequestData='@c:\files\report-definition.txt')

SELECT * FROM $ (RequestData='__rdxml=<reportDefinition xmlns="https://adwords.google.com/api/adwords/cm/v201809">
<selector>
<fields>CampaignId</fields>
<fields>Cost</fields>
</selector>
<reportName>My Campaign Performance Report</reportName>
<reportType>CAMPAIGN_PERFORMANCE_REPORT</reportType>
<dateRangeType>LAST_7_DAYS</dateRangeType>
<downloadFormat>XML</downloadFormat>
</reportDefinition>')

Build a SQL query using the Query Builder:

Use Query Builder to construct SQL query and override data source properties

Conclusion

In this article, we learned how to download AdWords report using ODBC PowerPack and Google API. We created an ODBC data source based on XML Driver, then configured the OAuth authentication, HTTP request and data filtering without a single line of code. Then we previewed AdWords “Campaign Performance Report” data and finally, considered how to reuse the data source we created. Visit https://zappysys.com/blog/category/odbc-powerpack/odbc-app-integration/ for examples of how to integrate data sources based on ZappySys Drivers into applications.

References

https://developers.google.com/adwords/api/docs/guides/reporting

https://developers.google.com/adwords/api/docs/appendix/reports

https://developers.google.com/adwords/api/docs/appendix/reports/campaign-performance-report

The post Download AdWords report using Google Ads API, ODBC, and XML Driver appeared first on ZappySys Blog.

Connect to YouTube API in SSIS (OAuth Connection)

Very first step to call any Google API is to create an API project for Google Service and then enable APIs you like to call in Console. In our case we need to enable YouTube API.

Check this article to learn how to create API Project and obtain Client ID and Client Secret to connect to YouTube API. For ease of use use Default Application provided by ZappySys but we still recommend using your own Custom Application  on OAuth Connection.

Before we try our first YouTube API example, we recommend you to get familiar with YouTube API here. Once you done with above steps you can perform the following steps to call YouTube API in SSIS.

1. Open existing SSIS Project or Create new One in Visual Studio (i.e. SSDT)
2. Create a new SSIS Package. Right click in the Connection Manager Panel and click “New Connection…”.
   
3. When Dialog box opens select ZS-OAUTH connection type.
   

   SSIS Connection Managers List: Select O-AUTH Connection Manager

4. Once OAuth UI opens, Configure Google connection like below.
   1. Select Service Type as Google
   2. You can select Default App for ease of use or Select Custom Application (Enter Client Id and Secret Obtained using these steps )
   3. In the Scope Enter below – One scope per line for calling correct YouTube API. Refer to YouTube API documentation incase you need extra permissions. For read only operation we just need below scopes. Last scope is only needed if you need to use YouTube Analytics Reports
      

      https://www.googleapis.com/auth/youtube
      https://www.googleapis.com/auth/youtube.readonly
      https://www.googleapis.com/auth/yt-analytics.readonly

5. Click Generate Token to obtain Refresh Token (You may see Login Prompt, and then Grant Permission Confirmation Screen)
   

   Connect to Google API using OAuth 2.0 (YouTube API Example)

6. Click OK to save UI.

Call YouTube REST API Example

Now lets look at how to call YouTube API REST API Task this task is more suitable to call GET, POST, DELETE, PUT requests without parsing data in Rows and Columns. If you need to parse data and load into Database table then see next section (Use JSON Source)

1. From SSIS Toolbox look for items starting with “ZS”. Drag and Drop [ZS Rest API Task] to Designer Surface.
   
2. Double click it to configure like below.
   1. Select URL From Connection option
   2. Select OAuth connection created in previous section.
   3. Enter URL like below (For example get all playlists). Replace xxxxxxxxxxx with your own channel ID (Its usually found in URL when you visit Youtube Channel Homepage)
      

      https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&channelId=xxxxxxxxxxxxx

   4. Click Test Request Button
3. Here is how it looks like
   

   Call Youtube API in SSIS Example (Dynamic URL)

That’s it we have successfully configured Connection for YouTube API in SSIS. In the next section we will see how to use this connection and read various data from YouTube API.

Read YouTube Playlists in SSIS

Once we done creating OAuth Connection Manager we can move forward to read YouTube data inside Data Flow.

Configure SSIS JSON / REST API Source

1. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
2. Double click on the Data Flow task to see Data Flow designer surface.
3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   
4. Double click JSON Source and enter URL as below (Use variable or Hardcode Channel Id). maxResults controls how many rows you want to get in each response. Rep
   

   https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&maxResults=50&channelId={{User::varChannelId}}
   --OR-- 
   https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&maxResults=50&channelId=xxxxxxxxxxxxxxxxxxxxxx

5. Check Use Credentials and select existing OAuth connection or Create New one
6. Select Array Filter (Items node) or type

   $.items[*]

     as below.
   

   Get Playlists for YouTube Channel

7. That’s it you can now Click Preview to see sample data. In the next section we will configure Pagination to read all records if you have more than 100 rows.

Configure YouTube REST API Pagination

Most REST APIs dont return data in one big response. So you have to make sure you implement looping / pagination to read more records. Luckily ZappySys supports many different Pagination Settings. Here is how to configure YouTube REST API Pagination.

1. Go to Pagination Tab found on JSON Source and select following 3 settings
2. Select Pagination Mode as

   Response Attribute Mode

3. Enter Next Link as

   $.nextPageToken

4. Enter Suffix for Next URL as

   &pageToken=<%nextlink%>

YouTube REST API Pagination Settings

Read YouTube Playlist Videos in SSIS

So in previous section we saw simple API call to read all playlists from YouTube. Now lets see how to get information about Playlist Videos.

For that everything should remain same as previous section except change URL as below. Where xxxxxxxxxxxxxxxxxxxxx is your Playlist ID obtained from Previous Call.

https://www.googleapis.com/youtube/v3/playlistItems?part=snippet,id&maxResults=50&playlistId=xxxxxxxxxxxxxxxxxxxxx

Get all Videos for YouTube Channel

If you want to get all video name and ID for specific channel then use below URL. Click here to read more. Replace xxxxxxxxxxxxxxxxxxx with your Channel ID.

https://www.googleapis.com/youtube/v3/search?channelId=xxxxxxxxxxxxxxxxxxx&part=snippet,id&order=date&maxResults=50

Read YouTube Video Views, Likes, Dislikes (Extended Videos)

Another common request from YouTube API is get Video Information such as Views, Likes, Dislikes, Comment Count etc. For that you can use this api . However the issue is this endpoint requires you to pass Comma separated Ids of Video. You can only pass 25 Ids in one requests so if you have 200 videos for which you need to get information then you have to make 8 requests (25 Ids in each Request). In below screenshot you can see we used Fiddler to debug web requests.

So here is how high level process to fetch extended information for all Videos in your channel. Full SSIS Package is attached here (SSIS 2012 Format).

1. Get all videos for your channel by calling (Assuming you will paginate in below request.
   

   https://www.googleapis.com/youtube/v3/search?channelId=xxxxxxxxxxxxxxxxxxx&part=snippet,id&order=date&maxResults=50

2. Build Video Id groups for 25 Ids from above result and submit requests. For example if above request returns 200 Videos then your request to fetch extended information may look like below.
   

   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID1,ID2,ID3........ID25
   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID26,ID27,ID28........ID50
   ........
   ........
   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID176,ID177,ID178........ID200

3. Parse response coming from each Request Above…. Each Response will give you details about 25 videos you requesting.

Here is the flow of our complete data flow to read extended information for all videos. You can read extended information one by one too but it wont be good idea if you have many videos.

We have used Following Components to achieve Bulk Mode

• JSON Source (REST API Source)
• CSV Parser Transform
• Web API Destination
• JSON Parser Transform

Get YouTube Video View Clicks, Likes, Dislikes (Bulk Mode for all videos)

Read data from YouTube Analytics / Reporting API (Metrics, Dimensions Report)

So far we have seen basic YouTube APIs to extract your Channel information. However there is another very powerful API endpoint to query custom reports from YouTube. Check this YouTube Analytics API endpoint. In this section we will learn how to extract useful information from YouTube using Analytics API.

Step-By-Step

Let’s look at an example to read total views, estimated watch time, average view duration and some other metrics by date for channel you have access.

1. Go to Data Flow, Drag ZS JSON Source from SSIS Toolbox
2. Double click to configure.
3. Check Use Credentials. Select OAuth connection we created in previous section.
4. In the URL field, Enter Below Sample URL. Change Parameters as per your need or leave it default.
   For example:
   ids=channel==MINE   (where MINE can be replaced by other channel id or keep it as MINE to get data for your own channel)
   metrics (can be changed from any valid list here)
   dimensions (can be changed to any valid list from here)
   startDate (Must be YYYY-MM-DD format)
   endDate (Must be YYYY-MM-DD format)
   

   https://youtubeanalytics.googleapis.com/v2/reports?ids=channel==MINE&metrics=views,estimatedMinutesWatched,averageViewDuration,averageViewPercentage,subscribersGained&dimensions=day&startDate=2019-03-01&endDate=2019-04-01

5. Change Filter to

   $.rows[*]

6. Go to 2D Array Transform tab.
   Select Method as

   Simple 2-dimensional array

   EnterColumn Filter as 

   $.columnHeaders[*].name

   

   YouTube Analytics API Advanced Transformation

7. That’s it now Preview your data. See below over all steps.
   

   Read data from YouTube Analytics API (Custom Reports using Dimensions, Metrics Query)

Build YouTube Analytics / Reporting API Query (API Playground)

With Analytics API You can extract many interesting data. You can use Test Tool found on this page to build some interesting queries. Scroll to Common Use Cases section on that link. On the right side you can select Credentials Type as OAuth and click Execute.

https://developers.google.com/youtube/analytics/reference/reports/query

YouTube Analytics / Reporting API Explorer (Testing API Calls)

Loading YouTube API Data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Download Sample SSIS Package

SSIS Package is attached here (SSIS 2012 Format).

Conclusion

In this article we saw how to extract information from REST API such as YouTube REST API using OAuth 2.0. We also learned techniques like Pagination, JSON Parsing, Request Batching using multiple components to achieve full scale API integration. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

The post How to read YouTube API data in SSIS (Videos, Channels, Playlists) appeared first on ZappySys Blog.

In this tutorial, we will cover the topics of how to perform geocoding on the addresses and reverse geocoding on the location coordinates using SSIS and Google Maps API. So what is geocoding, exactly? Geocoding is the process of translating an address (e.g. a street address) or a place to coordinates on the Earth’s surface. Simply put, geocoding takes a description of a location and gives back latitude and longitude values as a result. For example, 

1600 Amphitheatre Parkway, Mountain View, CA 94043, USA

(37.4224764, -122.0842499)

(37.4224764, -122.0842499)

1600 Amphitheatre Parkway, Mountain View, CA 94043, USA

To accomplish our goal we will use an add-on for SSIS – ZappySys SSIS PowerPack. It contains many additional SSIS Tasks and Data Flow Components that lets you accomplish what you cannot do with standard SSIS connectors. We will use these ZappySys SSIS PowerPack connectors:

Web API Destination

JSON Parser Transform

Let’s start!

Step-by-Step – SSIS Geocoding with Google Maps API

Before we start, let’s consider the use-case for our tutorial. Suppose, you have a CRM system filled with user-entered addresses of your leads. Since they are user-entered, it means that they can be incomplete, malformed, or just plain rubbish. Also, you have a marketing campaign and you want to send letters with offers to your leads so that they become your clients. Before sending them any letters, you have to make sure the addresses are valid and well-formed. You are also interested in the geographical location of your leads that you could do some business intelligence analysis on the coordinates of your leads and maybe future clientele. Here is where Geocoding API kicks in.

Thus in this example, we will take various street addresses – sometimes invalid, incomplete or unformatted – and submit them to Google Geocoding API for processing. As a result, we will get nicely formatted, cleansed, full addresses and their coordinates. We will take a look on how to determine if lookup worked successfully and to what extent you should trust Geocoding API’s response. Finally, we will save retrieved addresses and their individual components (e.g. street number, street name, city, country, etc.) to a SQL Server database table.

Load addresses from a SQL Server database table

We prepared the Google Geocoding API part, now we are ready to get to the place where the rubber meets the road – we will create an SSIS package and load addresses from a database table:

1. Create a new SSIS package.
2. Drag and drop Data Flow from SSIS Toolbox onto the Control Flow.
   

   Dragging and dropping Data Flow Task into Control Flow

3. Drag and drop OLE DB Source onto the Data Flow.
4. Configure OLE DB Source to retrieve addresses from your table. We will use a table from Microsoft’s Northwind’s modified database “Marketing” (download it):

Loading user-entered addresses to be looked up in Google Geocoding API using SSIS.

Construct lookup URL

1. Drag and drop Derived Column component onto the Data Flow.
2. Add a new column and construct Geocoding lookup URL in this format (just like it says in Geocoding API documentation):

https://maps.googleapis.com/maps/api/geocode/json?address=ADDRESS_TO_LOOKUP&key=YOUR_API_KEY

Adding Google Geocoding API lookup URL as Derived Column to be used for geocoding in SSIS.

Configure the Geocoding API request

1. Drag and drop Web API Destination onto the Data Flow and connect it with Derived Column. We will use Web API Destination to make HTTP requests to Geocoding API.
2. Then double-click on Web API Destination and create a new HTTP connection:
   

   Setting up HTTP Connection Manager for Web API Destination to make HTTP requests to Google Geocoding API using SSIS.

3. Finally, configure Input Column for URL, HTTP Request Method and input into URL field a sample lookup URL:
   

   https://maps.googleapis.com/maps/api/geocode/json?address=One Microsoft Way Redmond 98052-6399 USA&key=YOUR_API_KEY

   

   Configuring Web API Destination to make geocoding lookups in Google Geocoding API using SSIS.

   Once you are done, click Test Request/Response button. This will actually make a geocoding lookup for Microsoft headquarters:

4. In the Response (Raw) tab you see the actual response from Geocoding API. Within it –  cleansed, formatted and full Microsoft headquarters address together with geographical coordinates (even we looked it up using just the part of it). Now copy-paste the response body somewhere, so that we can use it in the next step while parsing:
   

   Checking Google Geocoding API HTTP response in Web API Destination using SSIS.

Parsing Geocoding API JSON response into columns

1. Drag and drop JSON Parser Transform onto the Data Flow and connect it to Web API Destination.
2. Then open it and configure Input JSON Column.
3. After that, set the Filter to $.results[*].
4. Continue by setting Sample JSON string to the one you saved in the previous step. Actually, before doing that let’s modify it and add partial_match node next to place_id so that exact and partial matches could be told apart (otherwise this property won’t be parsed):
   

   {
       "results": [{
           "address_components": [{
               "long_name": "One",
               "short_name": "One",
               "types": ["street_number"]
           }, {
               "long_name": "Microsoft Way",
               "short_name": "Microsoft Way",
               "types": ["route"]
           }, {
               "long_name": "Overlake",
               "short_name": "Overlake",
               "types": ["neighborhood", "political"]
           }, {
               "long_name": "Redmond",
               "short_name": "Redmond",
               "types": ["locality", "political"]
           }, {
               "long_name": "King County",
               "short_name": "King County",
               "types": ["administrative_area_level_2", "political"]
           }, {
               "long_name": "Washington",
               "short_name": "WA",
               "types": ["administrative_area_level_1", "political"]
           }, {
               "long_name": "United States",
               "short_name": "US",
               "types": ["country", "political"]
           }, {
               "long_name": "98052",
               "short_name": "98052",
               "types": ["postal_code"]
           }],
           "formatted_address": "One Microsoft Way, Redmond, WA 98052, USA",
           "geometry": {
               "location": {
                   "lat": 47.6423318,
                   "lng": -122.1369302
               },
               "location_type": "ROOFTOP",
               "viewport": {
                   "northeast": {
                       "lat": 47.6436807802915,
                       "lng": -122.1355812197085
                   },
                   "southwest": {
                       "lat": 47.6409828197085,
                       "lng": -122.1382791802915
                   }
               }
           },
           "partial_match": false,
           "place_id": "ChIJGxUEuHFtkFQRnokD7mkppQE",
           "plus_code": {
               "compound_code": "JVR7+W6 Redmond, Washington, United States",
               "global_code": "84VVJVR7+W6"
           },
           "types": ["establishment", "point_of_interest"]
       }],
       "status": "OK"
   }

   

   Configuring JSON Parser Transform to parse looked up addresses in Google Geocoding API HTTP JSON response using SSIS.

5. To proceed and parse individual address components from JSON response (e.g. number, street, city, country, etc.), you will need to specify columns for each address component manually, just like portrayed in Parse multi-dimensional JSON array in SSIS or Drivers article. Just open 2D Array Transform tab and select Multiple columns using expressions as Transform Type and configure them:
   

   Configuring JSON Parser Transform to parse individual address components from Google Geocoding API HTTP JSON response using SSIS.

   We specify these columns in our example:

   Number = $.address_components[?(@.types[0]=='street_number')].long_name
   Street = $.address_components[?(@.types[0]=='route')].long_name
   City = $.address_components[?(@.types[0]=='locality')].long_name
   State = $.address_components[?(@.types[0]=='administrative_area_level_1')].long_name
   Country = $.address_components[?(@.types[0]=='country')].long_name
   PostalCode = $.address_components[?(@.types[0]=='postal_code')].long_name

6. Finally, be sure to set Include all upstream columns to downstream (quite important), so that CustomerID is passed downstream and which we can save into a destination table. Later it can be a useful column to join the source table and the destination table on. In that way, you will be able to map user-entered addresses with looked up, fully formatted addresses (won’t be demonstrated in this tutorial).

Save geocoding results into a SQL Server database table

1. We are now ready to add OLE DB Destination, execute the package, make the lookups, and save the results into the database:
   

   Creating a database table for the Google Geocoding API looked up addresses in SSIS.

2. Execute the package!
   

   SSIS geocoding package execution and its results.

The results of SSIS geocoding

Viewing the results of SSIS geocoding in SQL Server destination table.

Once you query the table of looked up results, you will notice several things:

1. Perhaps there are more rows in the results table than in the source table
2. partial_match column is either 1 or NULL
3. geometry.location_type column has several values, e.g. ROOFTOP, RANGE_INTERPOLATED, APPROXIMATE, etc.

So to what extent should you trust the results? The answer is to take a look at the documentation and see what these values mean.

Basically, when partial_match is equal to 1, according to Google:

• Partial matches most often occur for street addresses that do not exist within the locality you pass in the request.
• Partial matches may also be returned when a request matches two or more locations in the same locality. For example, “21 Henr St, Bristol, UK” will return a partial match for both Henry Street and Henrietta Street. Note that if a request includes a misspelled address component, the geocoding service may suggest an alternative address. Suggestions triggered in this way will also be marked as a partial match.

While geometry.location_type column tells how accurate the lookup was. Again, according to Google:

• “ROOFTOP” indicates that the returned result is a precise geocode for which we have location information accurate down to street address precision.
• “RANGE_INTERPOLATED” indicates that the returned result reflects an approximation (usually on a road) interpolated between two precise points (such as intersections). Interpolated results are generally returned when rooftop geocodes are unavailable for a street address.
• “GEOMETRIC_CENTER” indicates that the returned result is the geometric center of a result such as a polyline (for example, a street) or polygon (region).
• “APPROXIMATE” indicates that the returned result is approximate.

Simply put, “ROOFTOP” is the most accurate lookup, while “APPROXIMATE” is the least accurate.

Another valuable piece of information is how many individual address components are filled. The more NULLs are there, the less accurate is the address. So if you have NULL in the Number column and in the Street column, most likely this is not a very accurate lookup:

Individual address components of looked up addresses in Google Geocoding API using SSIS.

Concluding, in the figure above, you can see how user-entered address can be validated, cleansed and formatted.

SSIS reverse geocoding with Google Maps API

The concepts presented above about “forward” geocoding will help you to make reverse geocoding requests too. The only difference is the request URL:

https://maps.googleapis.com/maps/api/geocode/json?latlng=40.714224,-73.961452&key=YOUR_API_KEY

For reference and help visit official Google Geocoding API documentation.

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Common errors

Parameterization and deployment to production 

Download a sample package

Be sure to download a sample SQL Server 2012 SSIS package, in case you want to try it right away (you can upgrade it to a higher version). If you want to use the database that was used in this tutorial, don’t forget to create the Marketing database from the script (a modified Microsoft’s Northwind database).

Conclusion

In this tutorial, we step-by-step covered how to make geocoding lookups using Geocoding API, part of Google Maps API. We learned how to cleanse, validate and format address just by submitting it to Geocoding API and parsing the results. We moved forward and looked at how to interpret the results and what geocoding results to trust. Then we touched the point that the same process could be applied to reverse geocoding. Concluding, we took a look at how to handle common errors and how to parameterize an SSIS package and deploy it to production.

The post SSIS Geocoding with Google Maps API appeared first on ZappySys Blog.

In this tutorial we will use REST API Task to call some ad-hoc API (e.g. get File List from Google Drive) and save output into Variable or File. We will use OAuth connection along with ZappySys JSON Source connector to read data from Google SpreadSheet (Use Drive API) and load into SQL Server (Export Google SpreadSheet to CSV).

Create Google API Project

First step to access any Google API is create an API Project in Google Console. If you don’t want to go through this then Skip this Step-1 and in the next section select Default OAuth App option on OAth Connection Manager (This is the most easiest option for now unless you want to use your own OAuth App).

Check this article for step-by-step instructions. When you follow these instructions make sure you enable Google Drive API (In the article it shows how to enable YouTube API as an example but you will need to enable Drive API for this article).

Create OAuth Connection Manager in SSIS

Once you create Google API project and obtained Client ID and Client Secret your next step is to create OAuth Connection Manager in SSIS. ZappySys OAuth connection manager comes with many predefined OAuth Providers (e.g. Facebook, Twitter, Google etc) but you can also define custom OAuth settings for any OAuth enabled API.

To create SSIS OAuth 2.0 Connection for Google API perform following steps.

1. Download and Install SSIS PowerPack
2. Create new SSIS Package
3. Right click in Connection Manager Area and Click “New Connection”
4. When prompted select ZS-OAUTH connection type
5. On the OAuth Connection Manager Select Provider=Google Sheets + Drive.
6. You can keep “Use Default App” selected or choose Custom App. Click here to learn – how to register Custom Google App
7. If you choose the Custom App option then enter the below scopes (or click Select Scopes). If you choose the default app then it’s not required.

   This will allow read / write access to Drive Files or Sheet (E.g. Reading file content / Export file to CSV). You can use Scope browser to see many other available permissions.

   https://www.googleapis.com/auth/drive
   https://www.googleapis.com/auth/drive.file
   https://www.googleapis.com/auth/drive.readonly
   https://www.googleapis.com/auth/spreadsheets
   https://www.googleapis.com/auth/spreadsheets.readonly

   Click Generate Token button.

   

   SSIS OAuth Connection Manager – Access Google Drive API using OAuth 2.0

8. You will see UI as below… Click Accept (You may have to scroll to see that button sometimes)
9. Click Test to see connection is working.

NOTE: If you don’t want to use Default OAuth App provided by ZappySys then select “Use Custom OAuth App” option (App created in previous section) specify your ClientID, ClientSecret.

Find Google Sheet ID for API call

Before we can read or write Sheet data in SSIS we need to know Google Driver File ID. We will use this ID in next few sections.

Easiest way to find File ID is look at the URL like below. If you need to know ID at runtime dynamically then refer last section.

Get Google Drive File ID for API call (Sheet ID)

Reading Google SpreadSheet Data in SSIS

Basically there are two ways you can export / read Google SpreadSheet data.

• Method-1 : Read Google SpreadSheet using SSIS JSON / REST API Source
  • In this approach we directly pull data from SpreadSheet using Core Google SpreadSheet.Values API to read particular Tab from Sheet and you can also specify Range. This is most effective way to read. Make sure you have v2.5.0.10807 or higher version (released after Aug 03 2017). New version added 2D array transformation options to parse multi-dimensional arrays found in SpreadSheet API JSON response.
• Method-2 : Read Google SpreadSheet using Flat File Source (as CSV)
  • Another approach is export SpreadSheet as CSV format. Then use Flat File Source to read that data and load into Target (E.g. SQL Server)

Method-1 : Read Google SpreadSheet using SSIS JSON / REST API Source

Now lets look at real steps to read Google SpreadSheet. To get values from specific tab and specific cell-row range you have to use Google SpreadSheet.Values API

In this example we will load below Google Spreadsheet into SQL Server

Step-By-Step : Loading Google SpreadSheet into SQL Server using SSIS

1. Drag and drop data flow on the control flow designer surface
2. Double click data flow and you will see SSIS Toolbox refreshed with new components
3. Drag ZS JSON Source from SSIS Toolbox onto Data flow designer surface
4. Double click JSON SOurce and configure below setting
   1. Enter your Spreadsheet URL in the following format
      

      https://sheets.googleapis.com/v4/spreadsheets/{your-file-id}/values/{cell-range}

      For example purpose we will use this public SpreadSheet provided by Google. If spreadsheet fileid is 1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms (see previous section how to get fileid) and your tab name is Class Data and you like to extract data from A to F columns starting from 2nd row then your Cell range in the URL can be ‘Class Data’!A2:F
      Here is the actual URL we will use for demo
      

      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/'Class Data'!A2:F

      — OR —
      You can also enter like below (If you omit Tab name then first table is used)
      

      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A1:F
      
      --OR-- Use below URL for to supply Column names manually (Start from A2)
      
      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A2:F

      You can also enter like below, specify first row and last row
      

      https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/A1:F101

   2. Check Use credentials and select OAuth connection we created in previous section
   3. Click on Select Filter and select values node and click OK  or you can directly enter following filter.
      

      $.values[*]

      

      Configure SSIS JSON/REST API Source – Read data from Google SpreadSheet API

   4. Now click on array Transformation tab. Now we will show you two different ways to configure column names.
   5. Method-1: Auto Detect Column Names (added in version 3.1.2)
      In recent version we added column less array transformation where we have option to detect first record as column name.
      1. Select Transform Type as Column less array
      2. Check option First line has column names like below example (Ignore direct Sample JSON – We provided for demo only. In your case it will be URL)
         

         JSON With 2D Array Pattern – Column names in First Row (Google Sheets API Pattern)

   6. Method-2: Manually Enter Column Names (For old version)
      1. Select Transform simple 2D array option from dropdown
      2. Check Specify columns list manually
      3. Enter column names (Must match the order and count based on range you selected)
         

         Student Name,Gender,Class Level,Home State,Major,Extracurricular Activity

         

         Configure SSIS JSON/REST API Source 2D Array Transformation for Google SpreadSheet API

   7. Now click Preview to see the data
      

      Configure SSIS JSON/REST API Source –
      Preview Google SpreadSheet Data

   8. Click OK to save UI
   9. Attach your JSON Source to target (e.g. OLEDB Destination for SQL Server Table).
   10. Configure Destination
       

       Configure SQL Server destination for Google SpreadSheet to SQL Data load

   11. Execute SSIS Package to load Google SpreadSheet data int SQL Server
       

       SSIS Example -Loading data from Google SpreadSheet into SQL Server Table

Method-2 : Read Google SpreadSheet using Flat File Source (as CSV)

Once we have Google Drive fileId we can perform following actions to export data into CSV file format. After that you can easily read CSV using SSIS FlatFile Source and load into SQL Server or other target.

Export Google SpreadSheet as CSV file using SSIS REST API Task

Here are the steps to export Google Drive SpreadSheet to CSV file

1. Just like previous section configure REST API Task
2. Enter following URL (See we used FileId from SSIS variable. You can hardcode too.
   

   https://www.googleapis.com/drive/v3/files/{{User::varFileId}}/export?mimeType=text/csv

3. Goto response tab. Check Save response and select “Save to file option”.
4. Enter file path.
5. Response character set select utf-8 (This will allow Unicode characters in response)
6. Click Test to see preview
   

   Export Google SpreadSheet to CSV file format using SSIS – Call Google Drive API to Export

7. Click OK to save

Load Google SpreadSheet data into SQL Server using SSIS (REST API Call)

Once you export SpreadSheet to CSV file you can easily consume it to load into SQL Server or any Target (e.g. Oracle, MySQL) using Native SSIS FlatFile Source and OLEDB/ADO.net Destination.

Write data to Google Spread Sheet

Now lets look at how to write to Google SpreadSheet. To write multiple rows in a single request you may use batchUpdate API. Also check this link to learn more.

Below is sample request to write data to columns and some data to rows (set majorDimension = COLUMNS to write vertical and use ROWS to write horizontally. You can mix multiple ways just like below example)

POST 
https://sheets.googleapis.com/v4/spreadsheets/1lkMEgu0zm2Q-cnPeoNZp4hqeIuASxXSpsrxtDDV89tg/values:batchUpdate
Authorization: Bearer ya29.Gl3WBFb3xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: application/json
Accept: */*

{
  "valueInputOption": "USER_ENTERED",
  "data": [
    {
      "range": "Sheet2!A1:A4",
      "majorDimension": "COLUMNS",
      "values": [
        ["Item", "Wheel", "Door", "Engine"]
      ]
    },
    {
      "range": "Sheet2!B1:D2",
      "majorDimension": "ROWS",
      "values": [
        ["Cost", "Stocked", "Ship Date"],
        ["$20.50", "4", "3/1/2016"]
      ]
    }
  ]
}

Write data to Google SpreadSheet using REST API Task (OAuth 2.0 Connection)

Update Google Sheet Cells – Single Range (Set Values for multiple cells)

Now let’s look at the below example of updating multiple cells in google Sheet. For more information, you can see the example here. For parameter detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve Spread Sheet Update scenario. For simple example lets use REST API Task.

1. Drag and Drop ZS REST API Task on the control flow designer
2. Edit Task and select URL from Connection Method
3. Select OAuth Connection from the dropdown (Same connection we created in the previous section)
4. Enter the URL as below. Change Sheet ID, Range designator and API URL Parameters as per your need.
   https://sheets.googleapis.com/v4/spreadsheets/[Sheet-ID-Here]/values/[Range-Here]?[Parameters-Here] Example URL (some part masked with xxxxxxxx):  
   

   https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4?valueInputOption=USER_ENTERED

5. Change Request Method to PUT
6. Change Request Content type to application/json
7. Enter Request Body as below (Assuming we need to update 4 cells)
   

   {
   "majorDimension": "ROWS",
     "values": [
      ["row1_cellA","row1_cellB"],
      ["row2_cellA","row2_cellB"],
     ]
   }

8. Click Test Request see it works.
   

   Calling Google Sheet API to Update Multiple Cell Values

Update Google Sheet Cells – Multiple Ranges

If you like to update values which needs to define multiple ranges then you need to use batchUpdate API. Import changes are you can’t include Range in the URL like previous example of Single Range Update. Also you have to use POST method and body with multiple ranges.

Example of sheet update with multiple range

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHxxxxxxxxxxxxxxxc/values:batchUpdate?valueInputOption=USER_ENTERED
Content-Type: application/json

>>>> BODY <<<<<

{
  "data": [
    {
      "range": "Sheet1!A2:B3",
      "majorDimension": "ROWS",
      "values": [
        [
          "row1_cellA",
          "row1_cellB"
        ],
        [
          "row2_cellA",
          "row2_cellB"
        ]
      ]
    },
    {
      "range": "Sheet1!D2:E3",
      "majorDimension": "ROWS",
      "values": [
        [
          "row1_cellD",
          "row1_cellE"
        ],
        [
          "row2_cellD",
          "row2_cellE"
        ]
      ]
    }
  ]
}

Write SQL Server data to Google Sheet

Now let’s look at real world example. How to write data from SQL Server table or other source and send to Google Sheet.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox
   
2. Drag source (e.g. OLEDB Source) and configure it to read from source like SQL Server Table
   

   OLE DB Source – Drag and Drop

3. Drag ZS JSON Generator.
   1. Connect Source to JSON Generator (blue arrow)
   2. Double click it to configure
   3. Select Output Mode as Single Dataset Array option for Mode. Enter Some Batch value (e.g. 500) this will make sure we send 500 rows at a time to Google API call. This way if we have many rows its not rejected by API call.
   4. Click Add element (Select Static Element) . Name: majorDimension and Value: ROWS
   5. Now click Add Document Array icon, name as values
   6. check Treat as 2D array option and click OK to save
   7. Now under values node click Add elements > Select Multiple Columns and Add columns you like to add.
   8. Click OK to save JSON GeneratorBelow is just an example How you can use Batch Settings to create JSON with N rows in each JSON document.
      
4. Now drag ZS Web API destination
   1. Connect JSON Generator to Web API destination (blue arrow)
   2. Double click it to configure
   3. Select same OAuth connection manager created in earlier section (Assuming you had all scopes with write permission)
   4. Now select Input column for Body (JSON Generator Output)
   5. Enter API call URL as below (Replace ID with your own id)
      

      https://sheets.googleapis.com/v4/spreadsheets/{your-file-id}/values/Sheet1!A2?valueInputOption=USER_ENTERED

   6. Select Request Method as PUT
   7. Select content type as application/json
   8. You can enter sample Body As below.
      

      {
      "majorDimension": "ROWS",
        "values": [
         ["Bob","bob@mycompany.com",55],
         ["Sam","sam@mycompany.com",57]
        ]
      }

   9. Click Test Request / Response button. Check your Sheet in browser after it see it updates?
   10. If you see 200 OK response means we are good to go.
   11. Run package and see your real data updated in Sheet.

Format Cells using Google Sheet API

Now let’s look at an example to format cell using Google Sheet API (e.g. Change Font, Color, Border, Background). For more information review this API documentation.

To format Sheet you first need to get SheetID  (keep in mind this is not same as spreadsheetId which you used in URL). SheetId is basically numeric ID of your Tab in WorkSheet.

Get SheetID

Request

GET https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHtxxxxxxxxxxxxxx

Response

{
  "spreadsheetId": "1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc",
  "properties": {
    "title": "Test.xlsx",
    "locale": "en",
    "autoRecalc": "ON_CHANGE",
    "timeZone": "America/Los_Angeles",
    "defaultFormat": {
      "backgroundColor": {
        "red": 1,
        "green": 1,
        "blue": 1
      },
      "padding": {
        "right": 3,
        "left": 3
      },
      "verticalAlignment": "BOTTOM",
      "wrapStrategy": "OVERFLOW_CELL",
      "textFormat": {
        "foregroundColor": {},
        "fontFamily": "Calibri",
        "fontSize": 11,
        "bold": false,
        "italic": false,
        "strikethrough": false,
        "underline": false
      }
    }
  },
  "sheets": [
    {
      "properties": {
        "sheetId": 1896331083,
        "title": "Sheet1",
        "index": 0,
        "sheetType": "GRID",
        "gridProperties": {
          "rowCount": 1000,
          "columnCount": 26,
          "rowGroupControlAfter": true,
          "columnGroupControlAfter": true
        }
      }
    }
  ],
  "spreadsheetUrl": "https://docs.google.com/spreadsheets/d/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc/edit"
}

Notice that in above response we see “sheetId”: 1896331083. This ID we will use in the Update API to format Cells.

Update Borders Example

Here is the example of Google Sheet Formatting API updateBorders Request

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSxxxxlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "updateBorders": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 0,
          "endRowIndex": 10,
          "startColumnIndex": 0,
          "endColumnIndex": 6
        },
        "top": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
        "bottom": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
        "innerHorizontal": {
          "style": "DASHED",
          "width": 1,
          "color": {
            "blue": 1.0
          },
        },
      }
    }
  ]
}

Response

{
  "spreadsheetId": "1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc",
  "replies": [
    {}
  ]
}

Update Cell Font Style (Bold, Style, Color, Background)

Now let’s look at how to use repeateCellRequest to apply certain type of formatting to multiple cells. This may be color, font style, background color.

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3xxxxxx4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "repeatCell": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 0,
          "endRowIndex": 1
        },
        "cell": {
          "userEnteredFormat": {
            "backgroundColor": {
              "red": 0,
              "green": 0.5,
              "blue": 0.7,
              "alpha": 1.3
            },
            "textFormat": {
              "foregroundColor": {
                "red": 1
              },
              "bold": true,
              "italic": true,
              "strikethrough": true,
              "underline": true
            }
          }
        },
        "fields": "userEnteredFormat"
      }
    }
  ]
}

In above example we have applied textFormat and backgroundColor but you can use any of below sections.

Set Currency Formatting

Here is an example request to set currency formatting

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "repeatCell": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 1,
          "startColumnIndex": 4,
          "endColumnIndex": 5
        },
        "cell": {
          "userEnteredFormat": {
            "numberFormat": {
              "type": "CURRENCY",
              "pattern": "$#,##0.00"
            }
          }
        },
        "fields": "userEnteredFormat.numberFormat"
      }
    }
  ]
}

Set Cell Validation

You can also set cell validation by calling following API call. This will restrict user’s ability to enter wrong data in certain cells.

POST https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6bRFUWqqrXlt4BRoX8rCHrORJc:batchUpdate
Content-Type: application/json

>>>> BODY <<<<<

{
  "requests": [
    {
      "setDataValidation": {
        "range": {
          "sheetId": 1896331083,
          "startRowIndex": 1,
          "startColumnIndex": 0,
          "endColumnIndex": 6
        },
        "rule": {
          "condition": {
            "type": "ONE_OF_LIST",
            "values": [
              {
                "userEnteredValue": "Red"
              },
              {
                "userEnteredValue": "Green"
              },
              {
                "userEnteredValue": "Yellow"
              }
            ]
          },
          "inputMessage": "Color must be Red, Yellow or Green",
          "strict": true,
          "showCustomUi": true
        }
      }
    }
  ]
}

Dynamically obtain File ID for Google Drive REST API call

Once OAuth connection is defined you can now call any valid API from Google based on Permission you requested in OAuth Connection. OAuth connection takes care of renewing you Token and passing Authorization as per OAuth standard. All these protocol complexity is hidden from you.

First step to get data from Google SpreadSheet using REST API call is get fileId of the Google Drive File you want to read. The easiest way to know file ID is call following API. Below API call returns name, ID and other information about file(s). You can get only fileID and use that in Next Section to export Google SpreadSheet to CSV format.

https://www.googleapis.com/drive/v3/files

OR

https://www.googleapis.com/drive/v3/files/?q=name%3D'Customers'

NOTE: For more information about searching files in Google drive / getting list of files using search expression language check this link. Make sure expression is URL encoded (e.g. name=’Customers’ should be passed as name%3D’Customers’)

Above API returns response as below. Notice the name and fileId. We will use fileId when calling export API in next section.

{
 "kind": "drive#fileList",
 "files": [
  {
   "kind": "drive#file",
   "id": "1lkMEgu0zm2Q-cnPeoNZp4hqeIuASxXSpsrxtDDV89tg",
   "name": "Customers",
   "mimeType": "application/vnd.google-apps.spreadsheet"
  },
  {
   "kind": "drive#file",
   "id": "0B6UyXr6ZlIFTc3RhcnRlcl9maWxl",
   "name": "How to get started with Drive",
   "mimeType": "application/pdf"
  }
 ]
}

If you want to automate retrieval of fileId from document name then perform the following steps.

1. In the SSIS package Control flow. Drag and drop ZS REST API Task from SSIS toolbox
2. In the Request Access mode, select “URL from Connection” option
3. Select OAuth connection manager (Created in Section-2)
4. Enter URL as below (It will return any file name equals to Customers (%3D is encoded value for = sign)
   

   https://www.googleapis.com/drive/v3/files/?q=name%3D'Customers'

5. Click Test (You will see file information)
   

   Get Google Drive File List in SSIS – Call Google Drive API to search file by name or expression

6. If you want to extract only fileID from full JSON then go to response tab
7. Select Format=JSON, Enter following JSONPath expression to extract only fileId from first record
   

   $.files[0].id

8. Check save response content, select SSIS variable where you want to save fileId as below. Now click Test to see preview of response. You will notice now only fileId will be returned from full JSON response.
   

   Get Google Drive fileid from file name using API call in SSIS

9. Click OK to save task

Conclusion

Google Drive API and Google Sheets API provides a great way to automate file related functionality. However, to call Google API  you have to use SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Google API call via simple drag and drop approach without coding. Try out yourself see how long it takes to call virtually any REST API.

The post How to read / write Google SpreadSheet using SSIS appeared first on ZappySys Blog.

Google AdWords is one of the most popular Advertise Platforms. In this article, you will learn how to get data from Google AdWords by calling Google AdWords Web Service method and using drag and drop SSIS XML/SOAP connector (No coding required :)). Google AdWords Web Services allows retrieving various data such as account info, campaigns; as well as writing data, such as creating a campaign, add a keyword. Basically, it allows controlling your AdWords account programmatically instead of using the AdWords dashboard. For demo purposes, we will get data from Google AdWords Account and load it into Microsoft SQL Server (but you can use this technique to save any supported targets in SSIS (e.g. Excel, FlatFile, Oracle, MySQL, IBM DB2, etc.).

If you desire to download a report from AdWords, you must follow a different approach described in an article on how to download AdWords report. It describes how to download a report using ODBC and XML Driver but the concepts introduced there can be applied to SSIS as well.

If you are looking for a similar article for Google DoubleClick for Publisher (Google DFP) then check this article.

You will also learn how to create Google Ads or modify any other objects by calling SOAP API using SSIS REST API Task.

SSIS Tasks/Connectors discussed in this Article

In this article we will use following Connector(s) and Connection Manager to archive full data integration with Google AdWords.

Following SSIS Connectors support OAuth Connectivity. When you use below connectors no coding required to pull data from any OAuth enabled services (e.g. Twitter, Facebook, LinkedIn).

	REST API Task
	OAuth Connection Manager
	XML Source (REST API, SOAP Call or File data)

Two different ways to get Google AdWords Data in SSIS

There are two ways you can extract Google AdWords data in SSIS.

1. Use Google Analytics Source (For more info see this article)  (Consume AdWords data using Google Analytics REST API)
2. Use XML Source along with OAuth connection manager (Consume AdWords data using SOAP API)

In this article we will only discuss Approach #2. For approach #1 refer see this link

Advantage of #2 is you get total control of full API set for AdWords but you have to do lot more manual work. For Approach #1 is lot simpler but you get access to limited reports (Dimensions and Metrics).

Google AdWords API Integration Example Screenshot

Below is screenshot of sample package discussed in this article (Download from here). As you can see we tried to demonstrate Reading data and Managing data (e.g. Create Ad) in the same package. You can extend these technique to perform any type of operations in Google AdWords or similar service such as DoubleClick for Publisher

Google AdWords API Integration Example – Create Ad, Get Accounts, Get Campaigns, Get Ad Groups, Get Ads (Extract AdWords data via SOAP API, Read data using Selector Method or use AWQL query language)

Step-By-Step Tutorial

Here are few high level steps you will have to perform to extract data from Google AdWords API using SSIS. This section assumes you have ClientID and Developer Token to access data. if you don’t have that then read next 2-3 sections.

1. Download and install SSIS PowerPack from here
2. Create new SSIS Project
3. From SSIS toolbox drag DataFlow Task and go to dataflow designer
4. From toolbox of drag ZS XML Source
5. Double click XML Source to configure it
6. Specify API URL from which you want to read data
   Example : https://adwords.google.com/api/adwords/mcm/v201509/ManagedCustomerService
7. Now check Use Credentials check box and select New ZS-OAuth connection manager.
8. On connection manager UI you can select Google from Provider Dropdown.
9. Under Scopes enter following (Scope is basically permission)
   

   https://www.googleapis.com/auth/adwords

    
10. Now click Generate Token and when prompted login and in the last step Accept.
11. If everything went ok then you will see AccessToken and RefreshToken populated.
12. Click Test connection and close the UI by clicking OK
13. Now on XML Source – Select HTTP Request Method as POST
14. In the Request Body click edit and enter request as below (Replace DEV_TOKEN and CLIENT_ID placeholders below)
    

    <?xml version="1.0" encoding="utf-8"?>
    <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    	<soap:Header>
    		<RequestHeader xmlns="https://adwords.google.com/api/adwords/mcm/v201605">
    			<developerToken>ENTER_YOUR_DEV_TOKEN_HERE</developerToken>
    			<clientCustomerId>ENTER_YOUR_CLIENT_ID_HERE</clientCustomerId>
    			<userAgent>ZappySys SSIS PowerPack v1</userAgent>
    		</RequestHeader>
    	</soap:Header>
    	<soap:Body>
    		<get xmlns="https://adwords.google.com/api/adwords/mcm/{{User::ApiVersion}}">
    			<serviceSelector>
    				<fields>CustomerId</fields>
    				<fields>Name</fields>
    			</serviceSelector>
    		</get>
    	</soap:Body>
    </soap:Envelope>

     

    If you are wondering how did we get above XML fragment for Request Body then Check this article on how to use SoapUI (free 3rd party tool) to create SOAP request Bodyfrom WSDL. Google AdWords has many API endpoints for different service. For each service you have different Api URL and different WSDL. In the next section you can learn how to create correct SOAP Body (for POST) using SoapUI tool.

    For example if you are calling ManagedCustomerService API for version v201609 then your help page URL would be like this
    https://developers.google.com/adwords/api/docs/reference/v201609/ManagedCustomerService

    And on the same page you will see WSDL link below
    https://ads.google.com/apis/ads/publisher/v201702/NetworkService?wsdl
    (Just download that WSDL XML and save to local disk then use with SoapUI to generate XML body to submit for any DFP API call.)

    See this link for more information (Its for Google DoubleClick Ad Platform but you will get an idea)

15. Go to Array handling tab at the bottom and enter text : ns2:entries – By doing this we will treat <ns2:entries> node as array eventhough only one entry found in response (By default content with multiple records treated as array).
    

    Handling Array Element Nodes in SOAP Response

16. On the filter expression click edit and highlight entries node and click ok. This will produce something like below in filter textbox
    

    $.soap:Envelope.soap:Body.ns2:getResponse.ns2:rval.ns2:entries[*]

     
17. Now click preview to see actual data.
18. Click OK to save UI. Now you can connect source to target such as SQL Server or Trash Destination.

NOTE: in above example we used v201605 API version but change it to latest if this API is no more valid. Google expires API version pretty often so keep that in mind if you get depreciated API version error.

Register for Developer Token to access AdWords account data using API

First important step to access any Google AdWords Account is request developer token. Click this link for step by step information. Watch short vide to get overview on entire process to access data via API.

Find your Client ID

Next thing you will need to know is your ClientID. Most of API Requests will require your ClientID. Perform following steps to find your ClientID.

Login to your AdWords Account using email address. At the top you will see your client ID as below screenshot.

How to find ClientID of your Google AdWords Account (Useful for making SOAP API Calls)

Connect to Google AdWords API using OAuth

Connecting to any Google API will require OAuth 2.0. SSIS PowerPack has Protocol so you can easily extract data from API without coding complex token extract/renew workflow. Such things usually requires some sort of programming but in our case you will do few clicks as described in Step-By-Step section earlier.

Here is the screenshot of OAuth Connection Manager.

Connecting to Google AdWords API using OAuth 2.0 in SSIS (Consuming XML/SOAP API)

Extract data from Google AdWords using SOAP API Calls

When you define filter expression you can check/uncheck “Include Parent Columns” setting. For AdWords SOAP Api you can uncheck so you only extract columns from rvals node in XML Response.

Extract data from Google AdWords SOAP/XML Response

SOAP Response Paging / Looping

Many time you will get response which is larger than default page size in AdWords API Call. In that case you can specify few pagination attributes on XML Source Paging tab. For more information on paging options you may refer this article . This article is talking about JSON Source but same options are applicable for XML Source too.

REST API Pagination – Passing PageNumber inside POST Body (Example: Google AdWords SOAP Api)

Configure REST API Pagination – using Page Number via POST Body Method

Read Google AdWords data and Load into SQL Server

Using XML Source you can extract data in tabular format and then you can map that to SQL Server Target (Use OLEDB Destination / ADO.net Destination).

Extract data from Google AdWords and load into SQL Server table using SSIS

Write data / Manage AdWords Account via REST API Calls

So far we have seen how to read data from AdWords account. But what if you have to create new records or write/update AdWords data via API Calls in automated manner? No worry – If you have need for that then use SSIS REST API Task which supports adhoc HTTP WebRequests including POST/DELETE/PUT etc. You can also upload files via REST API call using this task. This task supports rich error handling and saving your response into variable or files.

Create Ad under specific AdGroup

Below example shows how to create new ad under specific AdGroup using REST API Task

Notice we have used few placeholders to make things dynamic using SSIS Variables. Once REST API Task is executed you will see new ad created.

Creating Google AdWords Ad using SOAP/XML API in SSIS (REST API Task)

Verify Google Ad created programmatically using SOAP/XML API Call (SSIS REST API Task)

Methods for Creating new records

Google AdWords API supports mutate method for writing data. Many service endpoints supports this method. Below example request will create new Ad.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<developerToken>{{User::varDevToken}}</developerToken>
			<clientCustomerId>{{User::varClientId}}</clientCustomerId>
			<userAgent>My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)</userAgent>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<mutate xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<operations>
				<operator>ADD</operator>
				<operand>
	                <adGroupId>30800928123</adGroupId>
	                <ad xsi:type="TextAd">
	                    <displayUrl>//zappysys.com/ad1</displayUrl>
	                    <finalUrls>//zappysys.com/ad1</finalUrls>
	                    <headline>This is my Ad</headline>
	                    <description1>This is Line 1</description1>
	                    <description2>This is Line 2</description2>
	                </ad>            
				</operand>
			</operations>
		</mutate>
	</soap:Body>
</soap:Envelope>

Methods for Reading AdWords data

Google AdWords API supports two methods for reading data. Most of AdWords service endpoints supports GET method also known as Selector method (Its basically sending query in XML format). Seconds method is Query Method. It uses SQL like query language known as AWQL. (Read more about AWQL query language). If you are SQL Programmer then you may like AWQL Query method better than Selector. But either way you will be able to pass following information.

• Columns you want to return
• Filter (e.g. Where Clause)
• Order By
• Paging information (e.g. PageNumber, PageSize)

Get Method (XML Selector)

For Get Method using XML tags to define columns, filters and order by/paging information. Here is example Get method for below select query

SELECT Id, Heading
WHERE Status = 'ENABLED'
ORDER BY Heading DESC 
LIMIT 0,50

Get Request (Get all Active Ads)

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<developerToken xmlns="https://adwords.google.com/api/adwords/cm/v201605">{{User::varDevToken}}</developerToken>
			<clientCustomerId xmlns="https://adwords.google.com/api/adwords/cm/v201605">{{User::varClientId}}</clientCustomerId>
			<userAgent xmlns="https://adwords.google.com/api/adwords/cm/v201605">My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)</userAgent>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<get xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<serviceSelector>
			    <fields>Id</fields>
			    <fields>Headline</fields>
			    <predicates>
			        <field>Status</field>
			        <operator>EQUALS</operator>
			        <values>ENABLED</values>
			    </predicates>
			    <ordering>
			        <field>Headline</field>
			        <sortOrder>ASCENDING</sortOrder>
			    </ordering>
			    <paging>
			        <startIndex>0</startIndex>
			        <numberResults>50</numberResults>
			    </paging>
			</serviceSelector>
		</get>
	</soap:Body>
</soap:Envelope>

Query Method (AWQL Query – SQL like query language)

Below is sample Soap request for AWQL Method to read Ads listed under from specific AdGroups. To learn more AWQL Query Syntax Click Here.

To learn more about which Columns you can use inside your AWQL Query Click here. Note that each service support different fields for Select Clause, Where Clause and Order By Clause. AWQL language can be also used to query Predefined AdWords Reports. The only difference is .. FROM clause is required when you query Reports.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<developerToken xmlns="https://adwords.google.com/api/adwords/cm/v201605">YOUR_DEV_TOKEN</developerToken>
			<clientCustomerId xmlns="https://adwords.google.com/api/adwords/cm/v201605">YOUR_CLIENT_ID</clientCustomerId>
			<userAgent xmlns="https://adwords.google.com/api/adwords/cm/v201605">My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)</userAgent>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<query xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<query>
SELECT Id, Name
WHERE Status = 'ENABLED'
LIMIT 0,5
			</query>
		</query>
	</soap:Body>
</soap:Envelope>

Call AdWords Reports API – Download Campaign Performance data

There will be a time you like to download Reports from Adwords. For that  read this article. Its showing how to use ZappySys ODBC Driver to call AdWords Reporting API but concept is same in SSIS Product too due to similar UI elements.

Download Example SSIS Package

Download SSIS 2012 / 2014/ 2016 Sample

Conclusion

Google AdWords data integration can be very challenging without right set of tools. SSIS PowerPack certainly makes it easy when you have to extract/manage Google AdWords data. Innovative drag and drop approach gives you various options to cover simple or most advanced scenarios without relying on any Programming language (e.g. C#, Java, Python, Ruby etc).

The post How to get data from Google AdWords using SSIS appeared first on ZappySys Blog.

Google Analytics is the most popular web analytics service that tracks and reports website traffic. In this blog post you will learn how to get data from Google Analytics using SSIS using SSIS Google Analytics Source Connector (Using drag and drop approach without Coding). This article assumes you have basic knowledge about SSIS and you have necessary tools installed – Check this if not)

For demo purpose we will load data from Google Analytics into SQL Server Database but you can use same techniques to load data into Oracle, MySQL, Excel or any other RDBMS.

Prerequisite

This article assumes you have installed following tools

1. Install Client Tools for SSIS (atleast one version from below list) – Check this for detail
   1. BIDS for SSIS 2008 (i.e. SQL Server 2008 or 2008 R2) — OR —
   2. SSDT BI for Visual Studio 2012 (i.e. for SQL Server 2012) — OR —
   3. SSDT BI for Visual Studio 2013 (i.e. for SQL Server 2014) — OR —
   4. SSDT BI for Visual Studio 2015 (i.e. for SQL Server 2012/2014 or 2016)
2. Install SSIS PowerPack (Click here to download) it will add new Connector in SSIS Toolbox to access Google Analytics Connector.
3. You have setup correct permission for user to access google analytics data (See below section or read here)

Video Tutorial – Get data from Google Web Analytics Platform

What is Google Analytics Connector for SSIS?

The SSIS Google Analytics Source connector can be used to read data from your Google Analytics Service without learning complex APIs or coding/scripting in Python, Java, C#, etc. It has an easy to use drag and drop interface which allows you to extract large amounts of data, adjusting parameters such as dimensions, metrics, filters, sortby, date range etc. You can also use in-built reports or specify a direct query URL to get the desired data.

Read more about Google Analytics Source Connector

Features

• The intuitive user interface makes Google Analytics data integration super easy and fast without learning complex APIs or scripting languages
• Support for OAuth 2.0 Authentication and Google service accounts (*.p12 certificate file)
• Support for Proxy Server
• Easy to use interface with three different ways to query Google Analytics data: (1) Build query from UI, (2) Use built-in query templates, and (3) Use direct queries
• Support for dynamic dimensions, metrics, segments, filters and sortby fields (e.g. supply these values at runtime)
• Live preview support at design time
• Support for AdWords, AdSense, Ad Exchange and DoubleClick for Publisher (DFP)

Setup Google analytics data permission for users / service account

If you are not an admin of Google analytics and you need to access google analytics then make sure you ask admin to give you correct permission using below steps

To add/edit user permissions for Google Analytics Data perform following steps.

1. Login to your Google Analytics Portal (Login using Admin account or use google user account who setup Google Analytics) Click here to go to google analytics
2. Click on Admin tab > Click User management > Add/Edit user account permission

For more information you may visit official link : How to setup permission for google analytics

Add/Edit Permissions and User Account for Google Analytics Data Access (Service Account/ Normal Google Login)

SSIS Google Analytics Connector – Select Web Profile

Get Data from Google AdWords using SSIS ( use Google Analytics Source Connector)

Conclusion

Integration with Google Analytics Data can be challenging unless you understand how to write C#, Java, Python code using Google SDKs. If you are not a coder or you are in time crunch then drag and drop integration in SSIS using High quality Connector using ZappySys Google Analytics Source Connector will surely make your life easy. It supports all major SQL Server versions upto SQL Server 2016 32-bit/64-bit. Try SSIS PowerPack yourself for FREE.

Keywords

Fetch data from Google Web Analytics using SSIS | Extract google analytics data using SSIS | Load data from Google Analytics without Coding | Google Web analytics data import using REST API

Google Web Analytics Reporting REST API reference link

The post Get data from Google Analytics in SSIS using REST API Call appeared first on ZappySys Blog.