AdWords Archives | ZappySys Blog

Download AdWords report using Google Ads API, ODBC, and XML Driver

ZappySys — Thu, 24 Oct 2019 13:49:00 +0000

Introduction

UPDATE: ZappySys has released a brand new API Connector for Google AdWords which makes it much simpler to Read/Write Google AdWords Data in SSIS/ODBC compared to the steps listed in this article. You can still use steps from this article but if you are new to API or want to avoid learning curve with API then use newer approach.

Please visit this page to see all Pre-Configured ready to use API connectors which you can use in SSIS API Source / SSIS API Destination OR API ODBC Driver (for non-SSIS Apps such as Excel, Power BI, Informatica).

In the previous article, we learned how to get data from AdWords in SSIS and basically control your AdWords account programmatically, outside the dashboard. In this one, we will continue on how to download AdWords report by using Google Ads API, ODBC, and ZappySys XML Driver. Let’s not waste our time and cut to the chase!

Prerequisites

Google Ads account up and running.
ZappySys ODBC PowerPack installed.

Step-by-Step – Download AdWords report

Get Developer Token

The first important step to access any Google Ads Account is to request a developer token. Click this link for step by step information. Watch a short video to get an overview of the entire process to access data via API.

Get Client Customer Id

Getting Google Ads (formerly AdWords) Client Customer Id

Copy it to a notepad, we will use it later.

Open ODBC Data Sources Manager

Search for "odbc" in your start menu and click on ODBC (64 bits). If you can't find this then you can also go to Start Menu > ZappySys > ODBC PowerPack > Click on ODBC Data Sources (64-Bit).

Open ODBC Data Source

Create ODBC Data Source based on ZappySys XML Driver

Select tab User DSN if you want to create a data source only for your user, or select System DSN to create a data source for all users or for OS services (e.g. SQL Server Agent):

Creating ZappySys XML Driver based data source

Configure authentication in the ODBC data source

Let’s authenticate first:

Select OAuth in Connection Type property.
Click Click to Configure link.
In OAuth Provider property select Google.
Then either select Use Default OAuth App or Use Custom OAuth App. For the latter, you will need to create a Google OAuth App yourself. It’s perfectly safe to use the default application, but if your company policy does not allow it, you may consider creating it yourself.
In Scopes enter:
https://www.googleapis.com/auth/adwords
Then hit Generate Token and hit allow once asked.

Download AdWords report – Configuring authentication in ODBC data source

Proceed with HTTP request configuration

Then go on with configuring the data source:

Configuring HTTP request in ODBC data source to download an AdWords report

Give your data source a name
Then input this URL to download the report:
https://adwords.google.com/api/adwords/reportdownload/v201809
If the above URL does not work, get the working one from Google’s documentation.
Input POST as HTTP Request Method.
In Body enter this text:
__rdxml= CampaignId Impressions Clicks Cost CampaignStatus IN ENABLED PAUSED My Campaign Performance Report CAMPAIGN_PERFORMANCE_REPORT LAST_7_DAYS XML
This will display the data of CAMPAIGN_PERFORMANCE_REPORT report (campaign Id, impressions, clicks, and cost) for the last 7 days, for all campaigns that are enabled or paused. Make sure to visit the Google documentation page, which contains the information about how to download a different kind of report, how to configure predicates, etc.
In Body Content Type property select Form (application/x-www-form-urlencoded) option.
In HTTP Headers enter your developerToken and clientCustomerId retrieved in previous steps.

Select data filter in ODBC data source

Then hit Select Filter button and select row node, so that Array Filter is set to $.report.table.row[*]

Filtering on AdWords report data

If asked whether you want to treat the row element as an array, click “yes”. You can also go to the Xml Array Handling tab and see if the element row is specified there.

The Results

To view the results press Preview tab and then click Preview Data:

Previewing AdWords report data

Now you can use this data source and get the data into an Excel sheet, Power BI report, SQL Server (would need to create a data source in ZappySys Data Gateway instead) or elsewhere.

Reusing the data source

If your client application supports, you can override data source settings, such as HTTP body, in SQL query using WITH clause. In that case, you won’t need to create multiple data sources for different kind of reports; e.g.:

SELECT * FROM $ (RequestData='@c:\files\report-definition.txt')

SELECT * FROM $ (RequestData='__rdxml= CampaignId Cost My Campaign Performance Report CAMPAIGN_PERFORMANCE_REPORT LAST_7_DAYS XML ')

Build a SQL query using the Query Builder:

Use Query Builder to construct SQL query and override data source properties

Conclusion

In this article, we learned how to download AdWords report using ODBC PowerPack and Google API. We created an ODBC data source based on XML Driver, then configured the OAuth authentication, HTTP request and data filtering without a single line of code. Then we previewed AdWords “Campaign Performance Report” data and finally, considered how to reuse the data source we created. Visit https://zappysys.com/blog/category/odbc-powerpack/odbc-app-integration/ for examples of how to integrate data sources based on ZappySys Drivers into applications.

References

https://developers.google.com/adwords/api/docs/guides/reporting

https://developers.google.com/adwords/api/docs/appendix/reports

https://developers.google.com/adwords/api/docs/appendix/reports/campaign-performance-report

The post Download AdWords report using Google Ads API, ODBC, and XML Driver appeared first on ZappySys Blog.

How to call Google DFP API with SSIS – DoubleClick

ZappySys — Tue, 14 Feb 2017 19:28:31 +0000

Introduction

Google DoubleClick is one of the most popular platforms for Advertisers. Many times you have need for custom integration or Automation for many operations. In this article we will learn how to Call Google DFP API (i.e. DoubleClick for Publishers) without coding using SSIS (Microsoft SQL Server Integration Services). In our previous article we discussed how to Integrate Google AdWords API using SSIS (Click here).

If you are new to SSIS then no worry there are plenty of tutorials for ssis

To achieve Custom integration for DoubleClick (DFP) API we will use SSIS XML Source. XML Source is part of SSIS PowerPack which has 45+ connectors and Tasks.

Prerequisite

Before you can finish tasks explained in this article you have to finish below steps. This article assumes you have basic knowledge of SSIS (SQL Server Integration Services)

Make sure you SSIS designer installed. Sometimes its referred as BIDS or SSDT (Get from here)
Download SSIS PowerPack
Download SoapUI (Its free third party tool to test SOAP API). SoapUI can help you to generate XML Request BODY easily from WSDL file provided by API Vendor. You can also test your service by supplying parameters.
Optional – Another very useful Free tool is Fiddler. You can use it to see raw request/response (Check this article)

Download DFP API Example SSIS Package

Click here to DFP API download Sample for SSIS SSIS Package (SSIS 2012, 2014, 2016)

Screenshot of Sample Package:

Example SSIS Package – Google DFP API Integration (Read / Write data in Google Double Click for Publisher) – No Coding required

Making your first DFP API Call using SSIS

Once you install PowerPack you are ready to execute your very first Google DFP API Call.

NOTE: If you don’t have any DFP network for test then see next section (Explains how to create test network for testing).

Lets look at how to call Google DFP API step-by-step.

Create new SSIS Project and open package designer
Drag ZS REST API Task from SSIS control flow toolbox and drop it on designer.

REST API Task is useful when you want to call SOAP/REST API but not necessarily parse response into rows and columns. We will look at JSON Source later in this article which can actually parse response into rows and columns (Useful to load Google DFP data into SQL Server or other RDBMS / flatfile )
Double click REST API Task to configure. Select Access mode to [Url from Connection]

Enter following URL in the URL Textbox

 https://ads.google.com/apis/ads/publisher/v201702/NetworkService

From the connection dialogbox select ZS-OAUTH to create new OAuth Connection for DFP API
On OAuth Connection dialogbox select Google from Provider dropdown. In this demo we will use Default OAuth App but you can register your own google OAuth app if you wish to use Custom OAuth option from UI.

Enter following Scopes in the Scopes textbox (or select manually by clicking Select Scopes button). Scope is nothing but permission for API (In our case View/Manage DFP data and Read/Write Report Files to Cloud Storage).

https://www.googleapis.com/auth/dfp
https://www.googleapis.com/auth/devstorage.read_only
https://www.googleapis.com/auth/devstorage.write_only
https://www.googleapis.com/auth/cloud-platform.read-only
https://www.googleapis.com/auth/devstorage.full_control
https://www.googleapis.com/auth/cloud-platform
https://www.googleapis.com/auth/devstorage.read_write

If you are not going call ReportService API to generate Reports in CSV format then you will need only one scope
```
https://www.googleapis.com/auth/dfp
```
Click Generate Token button. You will see browser popup for login and then Accept option to confirm permissions like below. Once you don’t it will populate tokens and prompt to save tokens to backup file. Secure token backup to safe place.
SSIS OAuth Connection – Connect to Google DFP API (DoubleClick SOAP Web Service)
Once done Click Test Connection and if its green then Click OK to save connection
Once you are back to REST API Task UI then select Method to POST
Select Request Body Content Type to XML (i.e. text/xml)
Click edit button next to Request Body and enter following XML.
```
   
      
   
   
      
   
```
If you are wondering how did we get above XML fragment then read next section about using 3rd party tool called SoapUI. Also we have detail article for calling SOAP request. Check this article on how to use SoapUI (free 3rd party tool) to create SOAP request Bodyfrom WSDL. DFP has many API endpoints for different actions (e.g. NetworkService , InventoryService , …). For each service you have different Api URL and different WSDL. In the next section you can learn how to create correct SOAP Body (for POST) using SoapUI tool.

For example if you using DFP NetworkService API for version v201702 then your help page URL would be like this
https://developers.google.com/doubleclick-publishers/docs/reference/v201702/NetworkService
And on the same page you will see WSDL link below
https://adwords.google.com/api/adwords/mcm/v201609/ManagedCustomerService?wsdl
(Just download that WSDL XML and save to local disk then use with SoapUI to generate XML body to submit for any DFP API call.)
Once you done with above steps, your Task Configuration will look like below.
SSIS Rest API Task – Call Google DoubleClick API – Get all DFP Networks

Now click Test Request button. You will see Response popup with below content if its successful (Scroll at the bottom on response form and you will see XML data) .. see below

------------------------------------
Request
------------------------------------
POST https 1.1 ==> /apis/ads/publisher/v201702/NetworkService
Host: ads.google.com

>>>> HEADERS <<<<<
Authorization: Bearer ya29.Glz-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: text/xml;charset=UTF-8
User-Agent: ZappySysApp
Host: ads.google.com
Content-Length: 291
Expect: 100-continue
Accept-Encoding: gzip, deflate
Connection: Keep-Alive

------------------------------------
Response
------------------------------------
POST 1.1 200 OK==> OK

>>>> HEADERS <<<<<
Content-Encoding: 
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Alt-Svc: quic=":443"; ma=2592000; v="35,34"
Transfer-Encoding: chunked
Cache-Control: private, max-age=0
Content-Type: text/xml; charset=UTF-8
Date: Sun, 26 Feb 2017 01:18:32 GMT
Expires: Sun, 26 Feb 2017 01:18:32 GMT
Server: GSE

>>>> Cookies <<<<<

>>>> CONTENT <<<<<


	
		
			27ab3d2e4e1fcb8f444b317d0b0f871d
			50
		
	
	
		
			
				551196
				XFP sandbox property
				238897396
				ca-pub-2799179143725683
				America/New_York
				USD
				237897516
				true

How to generate SOAP Request Body using SoapUI tool

This section describes how to create DoubleClick SOAP API Request Body (XML fragment) for any DFP SOAP API call described anywhere in this article.

Download SoapUI (Its free third party tool to test SOAP API). SoapUI can help you to generate XML Request BODY easily from WSDL file provided by API Vendor. You can also test your service by supplying parameters.
Now download WDSL xml for appropriate API calls you want to make. Assume that you want to call getAllNetworksResponse API found under NetworkService. For that first navigate to help file page and open help page. Make sure you select correct version from help navigation. Copy WSDL link and open in new browser window. Save XML to local disk (e.g. c:\api\dfp_networkservice_wsdl.xml (See below screenshot how to find DFP API WSDL link)
How to get Google DFP API WSDL URL (Used to generate POST Body)
Once SoapUI is downloaded and installed click File menu > Create New SOAP Project option
Name your project (e.g. DFP API) and specify WSDL URL or File Path if it was saved locally (e.g. c:\api\dfp_networkservice_wsdl.xml) and click OK
Now navigate to API Action for which you would like to get Body. Click Request node (If missing create new) and edit request. You may see XML like below (If some optional parameters not visible in XML then click Re-create request with optional parameters button from toolbar)
Call Google DFP API – Generate SOAP Request Body XML using SoapUI (Google DoubleClick Web Service)
Edit necessary parameters from above XML code and copy to Body of SSIS REST API Task.

Creating test network – DFP Sandbox

Very first step we recommend before testing DFP API call is create test network. You can call DFP API against LIVE network but not recommended if you are calling create/update API.

To create test network you can follow exact same steps described in the previous section except the Body step (Step#12). Use following Request body to call makeTestNetwork command.

API makeTestNetwork – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/NetworkService
Body (see below):

Once you click Test you may receive following response. Note your network code (e.g. 1122334455 from below). This network code is used in pretty much all DFP API to view or manage ad network related items or properties.

API makeTestNetwork – Response


	
		
			ba61d79efee325bf4b5aa45c46836021
			1021
		
	
	
		
			
				551196
				XFP sandbox property
				1122334455
				ca-pub-2799179143725683
				America/Los_Angeles
				USD
				237897516
				true

Above response contains test networkcode copy that code becuase you will need in almost every API call you make later. You can only have one test network. If test network is already created and you try to execute above code then you may receive error AuthenticationError.GOOGLE_ACCOUNT_ALREADY_ASSOCIATED_WITH_NETWORK

You can also get network code using two different ways.

Visit your DFP console homepage by visiting https://www.google.com/dfp/ and you will see network name and code in the top potion. Also its listed in the URL (e.g. https://www.google.com/dfp/1234567)
Another way to get available networks for your login is to visit DFP API Play ground here https://dfp-playground.appspot.com/ and you will see network list in the dropdown ( name and network code)

How to Create new Ad Units

Now lets look at an example which will create few Ad Units by calling createAdUnits API. If you are creating Ad Unit at the root level then you will need to specify correct Parent ID (i.e effectiveRootAdUnitId). To obtain that Parent ID you may have to call getCurrentNetwork API. In the response you will see effectiveRootAdUnitId.

There is another way to know effectiveRootAdUnitId or Id of any Ad Units which can be Parent for new AdUnit. Goto google.com/dfp > Click Inventory Tab > Click Ad Units Side menu > Click Download ad units hyper link. You can also download AdUnits as CSV file from DFP Portal and look for Ad Units which are created at the root. Parent ID column for Root level Ad Units is basically called effectiveRootAdUnitId.

Now lets look at how to get EffectiveRootID by calling getCurrentNetwork.

API getCurrentNetwork – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/NetworkService
Body (see below):




	
		1122334455
		ZappySysApp

API getCurrentNetwork – Response


	
		
			81d1b00e8fbfb862129eb2dd485cf7bb
			4507
		
	
	
		
			
				551196
				XFP sandbox property
				1122334455
				ca-pub-2799179143725683
				America/New_York
				USD
				237897516
				true

If you want to save single Element Value from Response into SSIS Variable then perform following steps (example of how to get just effectiveRootAdUnitId from above response)

On REST API Task > Go to Response Tab > Select Response Content Type = XML
Enter following expression in the XPath filter
```
//*[local-name() = 'effectiveRootAdUnitId']
```
Check Save response content option
Select Variable from Dropdown – Click New Variable > Name it RootAdUnitId.
Click Test Request Button to test…. In the content textbox you will now see only Numeric value extracted by Filter

SSIS REST Api Task – How to extract single vale from response and save to SSIS variable

Once you know Parent ID for your AdUnit now lets look at how to create Ad Units by calling createAdUnits

API createAdUnits – Request

URL: https://ads.google.com/apis/ads/publisher/v201702/InventoryService
Body (see below):



	
		
			1122334455
			ZappySysApp
		
	
	
		
			
				237897516
				Ad_Unit_0
				Ad unit description #0.
				BLANK
				
					
						300
						250
					
					BROWSER
				
			
			
				237897516
				Ad_Unit_1
				Ad unit description #1.
				BLANK
				
					
						300
						250
					
					BROWSER

API createAdUnits – Response


	
		
			b244d7ba9b65e96fc42b849a7bfb16e4
			331
		
	
	
		
			
				252312396
				237897516
				false
				
					237897516
					ca-pub-2799179143725683
					ca-pub-2799179143725683
				
				Ad_Unit_0
				Ad unit description#0.
				BLANK
				ACTIVE
				253312396
				
					
						300
						250
						false
					
					BROWSER
					300x250
				
				SITE
				false
				
					
						true
						FFFFFF
						0000FF
						FFFFFF
						000000
						008000
						TEXT_AND_IMAGE
						DEFAULT
						DEFAULT
						DEFAULT
					
				
				
					
						2017
						2
						27
					
					7
					23
					32
					PST8PDT
				
				NONE
				false
				false
			
			
				252312516
				237897516
				false
				
					237897516
					ca-pub-2799179143725683
					ca-pub-2799179143725683
				
				Ad_Unit_1
				Ad unit description#2.
				BLANK
				ACTIVE
				253312516
				
					
						300
						250
						false
					
					BROWSER
					300x250
				
				SITE
				false
				
					
						true
						FFFFFF
						0000FF
						FFFFFF
						000000
						008000
						TEXT_AND_IMAGE
						DEFAULT
						DEFAULT
						DEFAULT
					
				
				
					
						2017
						2
						27
					
					7
					23
					32
					PST8PDT
				
				NONE
				false
				false
			
				252312876
				237897516
				false
				
					237897516
					ca-pub-2799179143725683
					ca-pub-2799179143725683
				
				Ad_Unit_4
				Ad unit description.
				BLANK
				ACTIVE
				253312876
				
					
						300
						250
						false
					
					BROWSER
					300x250
				
				SITE
				false
				
					
						true
						FFFFFF
						0000FF
						FFFFFF
						000000
						008000
						TEXT_AND_IMAGE
						DEFAULT
						DEFAULT
						DEFAULT
					
				
				
					
						2017
						2
						27
					
					7
					23
					32
					PST8PDT
				
				NONE
				false
				false

Loading Google DoubleClick data into SQL Server Table

So far we saw how to make simple DFP API calls without doing any parsing. Now lets look at how to use SSIS XML Source which not only make API calls but it will parse XML Response into rows and columns which can be loaded into target database such as SQL Server, Oracle, MySQL or even Flat File. XML Source also supports pagination so if you have many records it will automatically loop through all response untill all records are fetched.

Step-By-Step – Using XML Source to read Google DFP data (Parse into rows and columns)

Assuming you have tested your first DFP API Call (Explianed in the beginning of this article). You must have OAuth connection tested for API call.
Drag new Data flow task from SSIS Toolbox
Inside Data flow designer drag and drop ZS XML Source from toolbox
Double click XML Source to configure.
Set URL as below
https://ads.google.com/apis/ads/publisher/v201702/InventoryService
Check Use credentials option and from Drop down select OAuth connection manager (created in previous section)
Select Method = POST,
Select Request Content Type = XML (text/xml)

Enter Body as below (Click edit button).


   
      
         238897396
         ZappySysApp
      
   
   
      
         
            WHERE Status='ACTIVE'

Notice that in above we calling getAdUnitsByStatement API to fetch all Ad Units which are Active. Here is more information about Query Syntax for any API call getXXXXXXByStaement

Now click on Select Filter button and select results node and click OK. If prompted add array to list.
SSIS XML Source – Get data from Google DFP API using OAuth (Configure URL, Body and Filter)
Click Preview to see data. Click Columns tab to review data types (Auto detected). If you wish to change length or datatype then edit there and check Lock option (Last column)
SSIS XML Source – Preview Response for Google DoubleClick API Call – Parse into Rows/Columns
Click OK to save UI
Connect XML Source to same target (e.g. OLEDB Destination – SQL Server Connection)
Run data flow
SSIS Execution – Read from Google DFP and load into SQL Server Table

Configure Pagination for Google DFP API / PQL Query result

If you have large dataset to read from DoubleClick API ( when calling API such as getxxxxxxByStatement) then we recommend to supply LIMIT and OFFSET clause in your PQL query.

For example if you have 2000 ad units and you want to limit response size by maximum 300 rows then your query can be like below

--For first request (returns results 1-300)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 0

--For second request (returns results 301-600)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 300

--For third request  (returns results 601-900)
WHERE Status='ACTIVE' LIMIT 300 OFFSET 600

........

This looping logic can be complex to implement if you do manually. But no worry if you are using SSIS XML Source. It comes with many pagination options (Click here to read more about pagination)

See below screenshot how to configure Pagination options for Google DoubleClick API. Basically two places you have to change. Inside body you have to set placeholder and change few settings on Pagination Tab.

SSIS XML Source – Configure Pagination for Google DoubleClick API data fetch – PQL LIMIT and OFFSET clause for Google DFP API

SSIS XML Source – Configure Pagination for Google DoubleClick API data fetch – Set Pagination Mode

Create CSV Report File and Download in GZip format (*.gz)

Sometime you have to download large amount of data (Possibly millions of rows) in that case Bulk approach would be better. DFP ReportService API allows to call following APIs which can be used to produce CSV report file in *.gz format and then you can download it using REST API Task.

Below are high level steps you have to perform to produce report file and download it.

Create the ReportJob by invoking runReportJob command .
Poll the ReportJob object using ReportService.getReportJob (This is required because file is not available right away) .
Continue to poll the ReportJob object until the reportJobStatus field is equal to COMPLETED or FAILED.
If successful, fetch the URL for downloading the report by invoking getReportDownloadURL.

Here is the sample SSIS Package to Perform this Action.

Click here to DFP API download Sample for SSIS SSIS Package (SSIS 2012, 2014, 2016)

See below screenshot how to generate Google DFP Report File and download / extract (Unzip) using Drag and Drop SSIS workflow.

Create Google DFP Report File (CSV / Gzip) and Download Using SSIS (Example of ReportService functions runReportJob, getReportJob and .getReportDownloadURL)

Call runReportJob (Start DFP Report JOB)

Very first step to produce DFP Report file is call ReportService >> runReportJob API. See below Command.

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract id of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for 111111 node anywhere in SOAP response XML and extract value (i.e. 111111 ) .. For screenshot see next section

//*[local-name() = 'id']

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxxxxAKqG89gM-TpReRXO
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 1018
Expect: 100-continue
Accept-Encoding: gzip, deflate


	
		
			238897396
			ZappySysApp
		
	
	
		
			
				
					DATE
					LINE_ITEM_ID
					LINE_ITEM_NAME
					AD_UNIT_NAME
					AD_SERVER_IMPRESSIONS
					AD_SERVER_CLICKS
					LINE_ITEM_COST_TYPE
					LINE_ITEM_GOAL_QUANTITY
					LAST_MONTH

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:41 GMT
Expires: Fri, 02 Jun 2017 21:23:41 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 999


	
		
			f03e7e474fde2a5f9c5d7c9d7eb83923
			846
		
	
	
		
			
				10000297829
				
					DATE
					LINE_ITEM_ID
					LINE_ITEM_NAME
					AD_UNIT_NAME
					TOP_LEVEL
					AD_SERVER_IMPRESSIONS
					AD_SERVER_CLICKS
					LINE_ITEM_COST_TYPE
					LINE_ITEM_GOAL_QUANTITY
					LAST_MONTH
					false
					false

Call getReportJob (Poll JOB Status)

Second step is to make sure is Report Status check. For large report it make take several seconds or minutes before you can get Download URL (See Next Step). If you try to call Next step before Report is ready then you may get error. So better to add Polling logic.

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract status of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for IN_PROGRESS node anywhere in SOAP response XML and extract value (i.e. IN_PROGRESS )

//*[local-name() = 'rval']

Extract Single XML Node Value from SOAP API Response – Use XPATH in SSIS

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxXmWqmq
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 524
Expect: 100-continue
Accept-Encoding: gzip, deflate


   
      
         238897396
         ZappySysApp
      
   
   
      
       10000297829

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:47 GMT
Expires: Fri, 02 Jun 2017 21:23:47 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 451


	
		
			d52cb3a3d68af24aec2f9b0284379700
			803
		
	
	
		
			COMPLETED

Call getReportDownloadURL (Get Download URL)

Once you get status = COMPLETED in above API Call (i.e. getReportJobStatus) then you ready to call getReportDownloadURL to fetch URL which you can download.

Here is the API sample Request and Response

Request

POST https://ads.google.com/apis/ads/publisher/v201702/ReportService HTTP/1.1
Authorization: Bearer ya29.xxxxxxxxxxxxxxxxxxBp84ltTNyVRb9hlUEWx
User-Agent: ZappySysApp/1.0.2017.10531
Content-Type: text/xml
Accept: */*
Cache-Control: no-cache
Host: ads.google.com
Content-Length: 581
Expect: 100-continue
Accept-Encoding: gzip, deflate


   
      
         238897396
         ZappySysApp
      
   
   
      
       10000297829
       CSV_DUMP

Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Fri, 02 Jun 2017 21:23:49 GMT
Expires: Fri, 02 Jun 2017 21:23:49 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"
Content-Length: 1039


	
		
			8e5ac0bbeb8e88c180ea442fc5378d99
			1957
		
	
	
		
			https://storage.googleapis.com/dfp-report-export/caxxxxxxxx-xxxxxxxxxxxxxxb?GoogleAccessId=3769xxxxxxxxxxx-qxxxxxxxxxxxxxxxxsqdb5dvi8g@developer.gserviceaccount.com&Expires=1496438929&Signature=vnhCiAIqd%2BFgWCtAHo9roFOI3W%2BveUTYuSvyBQ9gtMI2HDTXOCVDh7wzUV%2BPBJZZghaL6k8wKYq33GA4NpjZVNYuaAgPLWgGwAUPdtioHRi%2Bt3Eymp%2FAPRGIoos0ekP3O7l%2FPNPxenC6UTP087HV3j06V%2Fuui9ixpKZWGCK8r56oUPj6S8q1hG95Ns3d3sEcuz3%2BkShu6GYG5mqir5vjtZPuzwsr%2F96G5Kk123mke%2B%2FIVRJGoqnCMghNWtIG1yR1IdZx%2FIpit%2FnaJ9HP6IvAr3oMIXkQ3cHv%2FtOhXGy9dTG34q%2FKDbtkSJAayHgjlSLPGKucygcXBhUrbywBu8Y77A%3D%3D

Extract Single XML Node Value from SOAP API Response using XPATH

Once you get response you can extract status of your JOB using XPAth Expression (See REST API Task response settings tab. Change Format to XML and type following XPATH expression. This will look for https://storage.googleapis.com/dfp-report-export/caxx………. node anywhere in SOAP response XML and extract value (i.e. URL)

Download File Using REST API Task

Once you get URL its time to download it. You can use REST API Task to download the file. On Response Setting Tab specify full file path (e.g. c:\report_csv.gz) and make sure Binary option is checked (Found next to the Save Path)

Making things dynamic

If you want to supply certain parameters at runtime (e.g. from SSIS Variable) rather than hard code then you can use Variable Placeholders any where in Request Body, URL or Headers. Click Insert Variable option found in Edit screen

Here is an example of dynamic URL (API Version stored in SSIS Variable)

https://ads.google.com/apis/ads/publisher/{{User::ApiVer}}/NetworkService

Deployment to Production

When you ready to deploy your SSIS Package to production make sure to change Test NetworkCode supplied inside Request Body to your Production Ad Network. Also Set RefreshToken Property of OAuth Connection Manager (Get it from Backup file saved when you created OAuth connection first time). Once you set RefreshToken of connection manager you won’t have to Login again to get a new token.

Conclusion

Traditionally if you wanted to integrate Google DFP API (DoubleClick API) calls inside your ETL workflow then it required some sort of coding effort (e.g. Use C# SDK, JAVA SDK or Python SDK). Coding effort is not only time consuming and expensive but also hard to maintain for any non-coder (e.g. ETL Developer or DBA).
Thanks to SSIS PowerPack which changed the way developers do API integration. Now you can reduce time to implement and total cost of you API integration project significantly by using drag and drop approach. This approach is not only easy to maintain but also fast and high quality. You can use ZappySys REST/SOAP SSIS Connectors and SSIS Tasks for any API integration project such as Google DFP API. Try SSIS PowerPack for free to explore possibilities. If you have any issue with your integration Contact Support

The post How to call Google DFP API with SSIS – DoubleClick appeared first on ZappySys Blog.

How to get data from Google AdWords using SSIS

ZappySys — Mon, 13 Jun 2016 16:32:55 +0000

Introduction

Google AdWords is one of the most popular Advertise Platforms. In this article, you will learn how to get data from Google AdWords by calling Google AdWords Web Service method and using drag and drop SSIS XML/SOAP connector (No coding required :)). Google AdWords Web Services allows retrieving various data such as account info, campaigns; as well as writing data, such as creating a campaign, add a keyword. Basically, it allows controlling your AdWords account programmatically instead of using the AdWords dashboard. For demo purposes, we will get data from Google AdWords Account and load it into Microsoft SQL Server (but you can use this technique to save any supported targets in SSIS (e.g. Excel, FlatFile, Oracle, MySQL, IBM DB2, etc.).

If you desire to download a report from AdWords, you must follow a different approach described in an article on how to download AdWords report. It describes how to download a report using ODBC and XML Driver but the concepts introduced there can be applied to SSIS as well.

If you are looking for a similar article for Google DoubleClick for Publisher (Google DFP) then check this article.

You will also learn how to create Google Ads or modify any other objects by calling SOAP API using SSIS REST API Task.

SSIS Tasks/Connectors discussed in this Article

In this article we will use following Connector(s) and Connection Manager to archive full data integration with Google AdWords.

Following SSIS Connectors support OAuth Connectivity. When you use below connectors no coding required to pull data from any OAuth enabled services (e.g. Twitter, Facebook, LinkedIn).

	REST API Task
	OAuth Connection Manager
	XML Source (REST API, SOAP Call or File data)

Two different ways to get Google AdWords Data in SSIS

There are two ways you can extract Google AdWords data in SSIS.

Use Google Analytics Source (For more info see this article) (Consume AdWords data using Google Analytics REST API)
Use XML Source along with OAuth connection manager (Consume AdWords data using SOAP API)

In this article we will only discuss Approach #2. For approach #1 refer see this link

Advantage of #2 is you get total control of full API set for AdWords but you have to do lot more manual work. For Approach #1 is lot simpler but you get access to limited reports (Dimensions and Metrics).

Google AdWords API Integration Example Screenshot

Below is screenshot of sample package discussed in this article (Download from here). As you can see we tried to demonstrate Reading data and Managing data (e.g. Create Ad) in the same package. You can extend these technique to perform any type of operations in Google AdWords or similar service such as DoubleClick for Publisher

Google AdWords API Integration Example – Create Ad, Get Accounts, Get Campaigns, Get Ad Groups, Get Ads (Extract AdWords data via SOAP API, Read data using Selector Method or use AWQL query language)

Step-By-Step Tutorial

Here are few high level steps you will have to perform to extract data from Google AdWords API using SSIS. This section assumes you have ClientID and Developer Token to access data. if you don’t have that then read next 2-3 sections.

Download and install SSIS PowerPack from here
Create new SSIS Project
From SSIS toolbox drag DataFlow Task and go to dataflow designer
From toolbox of drag ZS XML Source
Double click XML Source to configure it
Specify API URL from which you want to read data
Example : https://adwords.google.com/api/adwords/mcm/v201509/ManagedCustomerService
Now check Use Credentials check box and select New ZS-OAuth connection manager.
On connection manager UI you can select Google from Provider Dropdown.
Under Scopes enter following (Scope is basically permission)
```
https://www.googleapis.com/auth/adwords
```
Now click Generate Token and when prompted login and in the last step Accept.
If everything went ok then you will see AccessToken and RefreshToken populated.
Click Test connection and close the UI by clicking OK
Now on XML Source – Select HTTP Request Method as POST
In the Request Body click edit and enter request as below (Replace DEV_TOKEN and CLIENT_ID placeholders below)
```
	
		
			ENTER_YOUR_DEV_TOKEN_HERE
			ENTER_YOUR_CLIENT_ID_HERE
			ZappySys SSIS PowerPack v1
		
	
	
		
			
				CustomerId
				Name
			
		
	
```
If you are wondering how did we get above XML fragment for Request Body then Check this article on how to use SoapUI (free 3rd party tool) to create SOAP request Bodyfrom WSDL. Google AdWords has many API endpoints for different service. For each service you have different Api URL and different WSDL. In the next section you can learn how to create correct SOAP Body (for POST) using SoapUI tool.

For example if you are calling ManagedCustomerService API for version v201609 then your help page URL would be like this
https://developers.google.com/adwords/api/docs/reference/v201609/ManagedCustomerService

And on the same page you will see WSDL link below
https://ads.google.com/apis/ads/publisher/v201702/NetworkService?wsdl
(Just download that WSDL XML and save to local disk then use with SoapUI to generate XML body to submit for any DFP API call.)

See this link for more information (Its for Google DoubleClick Ad Platform but you will get an idea)
Go to Array handling tab at the bottom and enter text : ns2:entries – By doing this we will treat node as array eventhough only one entry found in response (By default content with multiple records treated as array).
Handling Array Element Nodes in SOAP Response
On the filter expression click edit and highlight entries node and click ok. This will produce something like below in filter textbox
```
$.soap:Envelope.soap:Body.ns2:getResponse.ns2:rval.ns2:entries[*]
```
Now click preview to see actual data.
Click OK to save UI. Now you can connect source to target such as SQL Server or Trash Destination.

NOTE: in above example we used v201605 API version but change it to latest if this API is no more valid. Google expires API version pretty often so keep that in mind if you get depreciated API version error.

Register for Developer Token to access AdWords account data using API

First important step to access any Google AdWords Account is request developer token. Click this link for step by step information. Watch short vide to get overview on entire process to access data via API.

Find your Client ID

Next thing you will need to know is your ClientID. Most of API Requests will require your ClientID. Perform following steps to find your ClientID.

How to find ClientID of your Google AdWords Account (Useful for making SOAP API Calls)

Connect to Google AdWords API using OAuth

Connecting to any Google API will require OAuth 2.0. SSIS PowerPack has Protocol so you can easily extract data from API without coding complex token extract/renew workflow. Such things usually requires some sort of programming but in our case you will do few clicks as described in Step-By-Step section earlier.

Here is the screenshot of OAuth Connection Manager.

Connecting to Google AdWords API using OAuth 2.0 in SSIS (Consuming XML/SOAP API)

Extract data from Google AdWords using SOAP API Calls

When you define filter expression you can check/uncheck “Include Parent Columns” setting. For AdWords SOAP Api you can uncheck so you only extract columns from rvals node in XML Response.

Extract data from Google AdWords SOAP/XML Response

SOAP Response Paging / Looping

Many time you will get response which is larger than default page size in AdWords API Call. In that case you can specify few pagination attributes on XML Source Paging tab. For more information on paging options you may refer this article . This article is talking about JSON Source but same options are applicable for XML Source too.

REST API Pagination – Passing PageNumber inside POST Body (Example: Google AdWords SOAP Api)

Configure REST API Pagination – using Page Number via POST Body Method

Read Google AdWords data and Load into SQL Server

Using XML Source you can extract data in tabular format and then you can map that to SQL Server Target (Use OLEDB Destination / ADO.net Destination).

Extract data from Google AdWords and load into SQL Server table using SSIS

Write data / Manage AdWords Account via REST API Calls

So far we have seen how to read data from AdWords account. But what if you have to create new records or write/update AdWords data via API Calls in automated manner? No worry – If you have need for that then use SSIS REST API Task which supports adhoc HTTP WebRequests including POST/DELETE/PUT etc. You can also upload files via REST API call using this task. This task supports rich error handling and saving your response into variable or files.

Create Ad under specific AdGroup

Below example shows how to create new ad under specific AdGroup using REST API Task

Notice we have used few placeholders to make things dynamic using SSIS Variables. Once REST API Task is executed you will see new ad created.

Creating Google AdWords Ad using SOAP/XML API in SSIS (REST API Task)

Verify Google Ad created programmatically using SOAP/XML API Call (SSIS REST API Task)

Methods for Creating new records

Google AdWords API supports mutate method for writing data. Many service endpoints supports this method. Below example request will create new Ad.



	
		
			{{User::varDevToken}}
			{{User::varClientId}}
			My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)
		
	
	
		
			
				ADD
				
	                30800928123
	                
	                    //zappysys.com/ad1
	                    //zappysys.com/ad1
	                    This is my Ad
	                    This is Line 1
	                    This is Line 2

Methods for Reading AdWords data

Google AdWords API supports two methods for reading data. Most of AdWords service endpoints supports GET method also known as Selector method (Its basically sending query in XML format). Seconds method is Query Method. It uses SQL like query language known as AWQL. (Read more about AWQL query language). If you are SQL Programmer then you may like AWQL Query method better than Selector. But either way you will be able to pass following information.

Columns you want to return
Filter (e.g. Where Clause)
Order By
Paging information (e.g. PageNumber, PageSize)

Get Method (XML Selector)

For Get Method using XML tags to define columns, filters and order by/paging information. Here is example Get method for below select query

SELECT Id, Heading
WHERE Status = 'ENABLED'
ORDER BY Heading DESC 
LIMIT 0,50

Get Request (Get all Active Ads)



	
		
			{{User::varDevToken}}
			{{User::varClientId}}
			My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)
		
	
	
		
			
			    Id
			    Headline
			    
			        Status
			        EQUALS
			        ENABLED
			    
			    
			        Headline
			        ASCENDING
			    
			    
			        0
			        50

Query Method (AWQL Query – SQL like query language)

Below is sample Soap request for AWQL Method to read Ads listed under from specific AdGroups. To learn more AWQL Query Syntax Click Here.

To learn more about which Columns you can use inside your AWQL Query Click here. Note that each service support different fields for Select Clause, Where Clause and Order By Clause. AWQL language can be also used to query Predefined AdWords Reports. The only difference is .. FROM clause is required when you query Reports.



	
		
			YOUR_DEV_TOKEN
			YOUR_CLIENT_ID
			My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)
		
	
	
		
			
SELECT Id, Name
WHERE Status = 'ENABLED'
LIMIT 0,5

Call AdWords Reports API – Download Campaign Performance data

There will be a time you like to download Reports from Adwords. For that read this article. Its showing how to use ZappySys ODBC Driver to call AdWords Reporting API but concept is same in SSIS Product too due to similar UI elements.

Download Example SSIS Package

Download SSIS 2012 / 2014/ 2016 Sample

Conclusion

Google AdWords data integration can be very challenging without right set of tools. SSIS PowerPack certainly makes it easy when you have to extract/manage Google AdWords data. Innovative drag and drop approach gives you various options to cover simple or most advanced scenarios without relying on any Programming language (e.g. C#, Java, Python, Ruby etc).

The post How to get data from Google AdWords using SSIS appeared first on ZappySys Blog.

Get data from Google Analytics in SSIS using REST API Call

ZappySys — Fri, 08 Jan 2016 14:54:01 +0000

Introduction

Google Analytics is the most popular web analytics service that tracks and reports website traffic. In this blog post you will learn how to get data from Google Analytics using SSIS using SSIS Google Analytics Source Connector (Using drag and drop approach without Coding). This article assumes you have basic knowledge about SSIS and you have necessary tools installed – Check this if not)

For demo purpose we will load data from Google Analytics into SQL Server Database but you can use same techniques to load data into Oracle, MySQL, Excel or any other RDBMS.

Prerequisite

This article assumes you have installed following tools

Install Client Tools for SSIS (atleast one version from below list) – Check this for detail
1. BIDS for SSIS 2008 (i.e. SQL Server 2008 or 2008 R2) — OR —
2. SSDT BI for Visual Studio 2012 (i.e. for SQL Server 2012) — OR —
3. SSDT BI for Visual Studio 2013 (i.e. for SQL Server 2014) — OR —
4. SSDT BI for Visual Studio 2015 (i.e. for SQL Server 2012/2014 or 2016)
Install SSIS PowerPack (Click here to download) it will add new Connector in SSIS Toolbox to access Google Analytics Connector.
You have setup correct permission for user to access google analytics data (See below section or read here)

Video Tutorial – Get data from Google Web Analytics Platform

What is Google Analytics Connector for SSIS?

The SSIS Google Analytics Source connector can be used to read data from your Google Analytics Service without learning complex APIs or coding/scripting in Python, Java, C#, etc. It has an easy to use drag and drop interface which allows you to extract large amounts of data, adjusting parameters such as dimensions, metrics, filters, sortby, date range etc. You can also use in-built reports or specify a direct query URL to get the desired data.

Read more about Google Analytics Source Connector

Features

The intuitive user interface makes Google Analytics data integration super easy and fast without learning complex APIs or scripting languages
Support for OAuth 2.0 Authentication and Google service accounts (*.p12 certificate file)
Support for Proxy Server
Easy to use interface with three different ways to query Google Analytics data: (1) Build query from UI, (2) Use built-in query templates, and (3) Use direct queries
Support for dynamic dimensions, metrics, segments, filters and sortby fields (e.g. supply these values at runtime)
Live preview support at design time
Support for AdWords, AdSense, Ad Exchange and DoubleClick for Publisher (DFP)

Setup Google analytics data permission for users / service account

If you are not an admin of Google analytics and you need to access google analytics then make sure you ask admin to give you correct permission using below steps

To add/edit user permissions for Google Analytics Data perform following steps.

Login to your Google Analytics Portal (Login using Admin account or use google user account who setup Google Analytics) Click here to go to google analytics
Click on Admin tab > Click User management > Add/Edit user account permission

For more information you may visit official link : How to setup permission for google analytics

Add/Edit Permissions and User Account for Google Analytics Data Access (Service Account/ Normal Google Login)

How to setup permissions for Service Account (*.p12 certificate)

Service account is another method to access google data. This is useful when you don’t want to use google login to access data (Example: Some consultant working to setup automated jobs for your company and they don’t really have login for google or you don’t want to share your personal google login with them).

Service Account method (*.p12 certificate file) doesn’t require complex flow like OAuth so no more authorization prompts but make sure certificate generated for Service Account is kept secret.

To grant permission to service account so it can access google analytics data, First make sure Service Account is created and find account email (It may look like this account-1234@some-project-id-xyz.gserviceaccount.com ) . Click here to know more how to create google service account. Once service account created you may follow same steps described in above section to setup permission for Google Analytics Data.

Access Google Analytics data using OAuth 2.0 authentication protocol

Access Google Analytics Data using service account method (*.P12 Certificate file)

SSIS Google Analytics Connector – Select Web Profile

SSIS Google Analytics Connector – Using build your own query mode

SSIS Google Analytics Connector – Using direct query mode

SSIS Google Analytics Connector – Predefined Report Templates

SSIS Google Analytics Connector – Report Preview

SSIS Google Analytics Connector – Select Dimensions

SSIS Google Analytics Connector – Select Metrics

SSIS Google Analytics Connector – Select Filters

SSIS Google Analytics Connector – Select Segment

How to read google Adwords Data using Google Analytics Source

If you wish to access Google AdWords data using Google Analytics source then refer instructions listed on AdWords Tab on the UI. Once you enable integration you can view AdWords data easily without any other connector (see below)

Get Data from Google AdWords using SSIS ( use Google Analytics Source Connector)

SSIS Package – Get data from Google Web Analytics Platform and load into SQL Server

Conclusion

Integration with Google Analytics Data can be challenging unless you understand how to write C#, Java, Python code using Google SDKs. If you are not a coder or you are in time crunch then drag and drop integration in SSIS using High quality Connector using ZappySys Google Analytics Source Connector will surely make your life easy. It supports all major SQL Server versions upto SQL Server 2016 32-bit/64-bit. Try SSIS PowerPack yourself for FREE.

Keywords

Fetch data from Google Web Analytics using SSIS | Extract google analytics data using SSIS | Load data from Google Analytics without Coding | Google Web analytics data import using REST API

Google Web Analytics Reporting REST API reference link

The post Get data from Google Analytics in SSIS using REST API Call appeared first on ZappySys Blog.