In our previous article we saw how to read Google Analytics data using SSIS. But what if you don’t use SSIS and you still like to Import Google Analytics data into SQL Server or Read inside other Reporting / ETL Apps (e.g. Excel, Power BI, MS Access … and many more). Not all BI tools comes with native driver / connectivity for Google Analytics. Sometimes they do offer Google Analytics connectivity but don’t offer feature you looking for (e.g. Sort, Segment Filter). In such case you have to look for solution outside.

Well not to worry we will show you how simple it is to connect to Google Analytics using ZappySys JSON Driver (For REST API / Files) in few mins. This driver is one of the best REST API drivers you can find in the market. It is so generic that it can virtually access any REST API you may find internally or on public sites (e.g. Facebook, Salesforce, Azure, AWS). Check these articles to see many more use cases of JSON Driver.

At the end of this article you will learn how to Query Google Analytics data in SSMS like below (Yes that’s correct .. Pretty Awesome!!! ).

Import Google Analytics Data into SQL Server Table (using T-SQL Code)

Testing Google Analytics API

If you are new to Google Analytics API then read little bit about Google Analytics REST API here. Google also offers a really good way to test Analytics API using Query Explorer Tool here. So please check that and understand how to craft correct REST API URL which will be used in the next section.

Using Google Analytics Query Explorer Tool – Create REST API Url (Dimension / Metrics Selection Browser)

Sample Google Analytics API Response

Behind the scene here is how API Request and Response Looks like for Google Analytics REST API. Notice that Authorization Header in below request is automatically added by ZappySys Driver or Query Testing tool above. Also notice nextLink attribute in response, its used to fetch more data. By default each response contains upto 10000 rows.

Request

GET https://www.googleapis.com/analytics/v3/data/ga?ids=ga:185737326&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration&dimensions=ga:date
Authorization: Bearer ya29.GlwUxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Response

{
    "kind": "analytics#gaData",
    "id": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday",
    "query": {
        "start-date": "30daysAgo",
        "end-date": "yesterday",
        "ids": "ga:123456789",
        "dimensions": "ga:date",
        "metrics": ["ga:avgSessionDuration"],
        "start-index": 1,
        "max-results": 1000
    },
    "itemsPerPage": 1000,
    "totalResults": 30,
    "selfLink": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday",
    "nextLink": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday&start-index=10001&max-results=10000",
    "profileInfo": {
        "profileId": "123456789",
        "accountId": "33445566",
        "webPropertyId": "UA-33445566-3",
        "internalWebPropertyId": "454545454",
        "profileName": "All Web Site Data",
        "tableId": "ga:123456789"
    },
    "containsSampledData": false,
    "columnHeaders": [
        {
            "name": "ga:date",
            "columnType": "DIMENSION",
            "dataType": "STRING"
        },
        {
            "name": "ga:avgSessionDuration",
            "columnType": "METRIC",
            "dataType": "TIME"
        }
    ], "totalsForAllResults": {
        "ga:avgSessionDuration": "49.3779488037477"
    },
    "rows": [ 
		["20190425", "401.38912133891213"], 
		["20190426", "512.49723756906077"], 
		["20190427", "11.53488372093024"], 
		["20190428", "432.148648648648646"], 
		["20190429", "234.650735294117645"],
		.......
		.......
		.......
		.......
		["20190525", "74.123435294117645"]
	]
}

Creating Google API Project (Obtain Client ID / Secret)

Very first step to call any Google API including Google Analytics is to create Google API Project and register OAuth App to obtain Client ID and Client Secret. If you dont want to go through this hassle and start easy way then ZappySys offers Inbuilt Default App on OAuth Connection UI but we strongly recommend you create your own app rather than using Default App because in Default App API call limit is shared by many. Your data is never shared with ZappySys in any case even you use Default App.

Here is how to create API Project to call Google API. Once you create Google API Project make sure you enable Google Analytics API.

So once you have Client ID and Client Secret we can move forward to the next step.

Loading Google Analytics data into SQL Server

In our previous article we saw in depth general idea on how to import REST API data in SQL Server (T-SQL Script) without relying on any ETL tools.  Now lets look at step by step on how to load Google Analytics data into SQL Server.

In this example, our goal is to extract a Google Analytics report which shows average session duration and new users count by date (For last 30 days).

Setup ZappySys Data Gateway

Very first step to access any REST API Data inside SQL Server is to configure ZappySys Data Gateway. We covered this in previous article.  But here are high level steps for initial setup of Data Gateway.

1. Search for Gateway in Start menu and  Select ZappySys Data Gateway
   

   Open ZappySys Data Gateway

2. Create new User in Data gateway on Users tab. Enter username and password (we will use this when we create Linked Server) . Check Admin Option
   

   Add Data Gateway User

Setup Google Analytics API Data Source in Gateway / ODBC

Once gateway user is setup, now lets create a new Data Source for Google Analytics API. In this section we will talk how to create data source in gateway but most instructions can be used to create data source in ODBC too. At the end of this article we explained how to Launch ODBC Datasource UI. Step#1 and 2 are only different, all other steps same in ODBC DSN creation.

1. Click Add Data Source option (Select Native – JSON Driver )
   

   Add Gateway Data Source (Native JSON Driver)

2. Click Edit to configure data source
   

   Edit Gateway Data Source Settings

3. Now lets configure Driver settings.  Click on Load Connection String button.
   

   Load ZappySys Driver ConnectionString to configure UI

4. Enter the following ConnectionString to get started with predefined settings.
   Change ids (111223344 to your own Profile ID in DataPath URL). Other URL Parameters explained in next section.
   

   DRIVER={ZappySys JSON Driver};
   DataPath='https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date';
   DataConnectionType=OAuth;
   ScopeSeparator='{space}';
   Scope='https://www.googleapis.com/auth/analytics https://www.googleapis.com/auth/analytics.readonly';
   ServiceProvider=Google;
   Filter='$.rows[*]';
   ArrayTransformType=TransformSimpleTwoDimensionalArray;
   ArrayTransColumnNameFilter='$.columnHeaders[*].name';
   RequestMethod='GET';
   NextUrlAttributeOrExpr='$.nextLink';
   ClientId='xxxxxxxxxxxx';
   ClientSecret='yyyyyyyyyyyyyy';
   UseCustomApp='True'

    
5. URL used in DataPath is most important (parameters explained below) . You can use   Query Explorer Tool here to build same URL.
   

   https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date

   Parameters Explained 
   
   Here is some required parameters used in above REST API URL.
   ids=ga:11223344
   This is your Profile ID which you like to extract.  Read more
   start-date=30daysAgo
   This can be yyyy-MM-dd or some other some predefined date keywords (e.g today, yesterday 10daysAgo). Read more.
   end-date=yesterday
   Same as above
   metrics=ga:avgSessionDuration,ga:newUsers
   This can be comma separated list of metrics you like to extract (max 10). Read more.
   dimensions=ga:date
   This can be comma separated list of metrics you like to extract (max 7). Read more.
6. Now its time to configure OAuth settings. Click on Configure Connection. Enter Client ID and Client Secret Obtained in the previous section (see here) and then click Generate Token as below.
   

   Configure Google API Data Source / DSN (Google Analytics API Example)

7. Thats it now can click on Preview Tab and run sample query like below.
   

   select * from $

   –OR–
   (Select Table name from dropdown and generate default query and then you can add alias , remove unwanted columns)
   

   SELECT 
   "ga:date" date, 
   "ga:avgSessionDuration" avgSessionDuration,
   "ga:newUsers" newUsers
   FROM "rows"

   –OR–
   (Type custom query with your own URL generated from here, Change Profile ID – from 11223344 to your own id)
   

   SELECT * FROM $
   WITH(
   	 Src='https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date'
   	,IncludeParentColumns='False'
   )

   

Create Linked Server in SQL Server using T-SQL Script

Once our Gateway Data Source is configured we can now move to SQL Server Part to define Linked Server.

Here is how you can define linked server to access Google Analytics Data inside SQL Server. There are two ways to create Linked Server (1)  via T-SQL script (2) via SSMS UI)

Here is how to create using Script.

USE [master]
GO

--drop existing linked server for same name if already found
EXEC master.dbo.sp_dropserver @server=N'GOOGLE_ANALYTICS_LS', @droplogins='droplogins'
GO

EXEC master.dbo.sp_addlinkedserver 
	@server = N'GOOGLE_ANALYTICS_LS', 
	@srvproduct=N'', 
	@provider=N'SQLNCLI', 
	@datasrc=N'localhost,5000', --this is host name and port number where  ZappySys Data Gateway is running
	@catalog=N'ZS-GA' --this must be same name as defined on Data Source tab Grid on ZappySys Data Gateway

EXEC master.dbo.sp_addlinkedsrvlogin 
	@rmtsrvname=N'GOOGLE_ANALYTICS_LS',
	@useself=N'False',
	@locallogin=NULL,
	@rmtuser=N'SOME-GATEWAY-USER', -- user name created on ZappySys Data Gateway
	@rmtpassword='SOME-GATEWAY-PASSWORD' -- password for ZappySys Data Gateway user
GO

Create Microsoft SQL Server Linked Server using SSMS UI

Import Google Analytics Data into SQL Server Table (T-SQL Code)

Now lets see how to run sample query to import Google Analytics Data into SQL Server. Notice that how we have escaped quotes in OPENQUERY.
NOTE: Change id 11223344 with your own Id (use this tool to create URL)

select * into GaApiLoad from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date''
  ,IncludeParentColumns=''False''
 )
')
select * from GaApiLoad

Import Google Analytics Data into SQL Server Table (using T-SQL Code)

Expose Google Analytics Data as View

To make it simple you can expose above query as SQL Server views so you can access them from any Reporting / ETL Tools.

CREATE VIEW vw_Google_Analytics_BounceRate_PageViews_Last30Days
AS
--change ids to your own Profile ID
select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=2019-04-01&end-date=2019-04-30&metrics=ga:bounceRate,ga:uniquePageviews&dimensions=ga:date''
  ,IncludeParentColumns=''False''
 )
')
GO

Making things Dynamic using Stored Procedure with Parameters

Now let’s make few things dynamic. We will change above query so we can pass Id, Dimensions and Metrics as parameters.

Run below script to Create a stored proc and fetch google analytics data in SQL Server.

use MyDatabase
go
create proc usp_GoogleAnalyticsData
	@ids varchar(1000),
	@startDate varchar(50)='30daysAgo',
	@endDate varchar(50)='yesterday',
	@metrics varchar(200)='ga:avgSessionDuration,ga:newUsers',
	@dimensions varchar(200)='ga:date'	
as

declare @sql varchar(4000)

set @sql='select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
''SELECT * FROM "rows"
WITH 
 (
   SRC=''''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:' + @ids 
   +'&start-date='+ @startDate 
   +'&end-date='+ @endDate 
   +'&metrics='+ @metrics 
   +'&dimensions='+ @dimensions 
   +'''''
  ,IncludeParentColumns=''''False''''
 )
'')'

print @sql
execute(@sql)

go

--Example: query google analytics data
exec usp_GoogleAnalyticsData 
	@ids='11223344',  --Change this ID - Obtain from here https://ga-dev-tools.appspot.com/query-explorer/
	@startDate ='30daysAgo',  --or like this 2019-04-01
	@endDate   ='yesterday', --or like this 2019-04-30
	@dimensions='ga:date', 
	@metrics   ='ga:bounceRate,ga:uniquePageviews'

Insert Google Analytics Data into table (Save Stored Procedure Output)

Now what if you like to save Stored Proc output into a table? Well for that you must create table with same structure as stored proc output and then use INSERT INTO sql like below.

--import google analytics data in a table
if(object_id('GaDataLoad') is not null)
	drop table GaDataLoad
create table GaDataLoad([Date] date, BounceRate numeric(10,5), PageViews numeric(10,5))  

insert into GaDataLoad
exec usp_GoogleAnalyticsData 
	@ids='11223344',  --Change this ID - Obtain from here https://ga-dev-tools.appspot.com/query-explorer/
	@startDate ='30daysAgo',  --or like this 2019-04-01
	@endDate   ='yesterday', --or like this 2019-04-30
	@dimensions='ga:date', 
	@metrics='ga:bounceRate,ga:uniquePageviews'

select * from GaDataLoad

Performance Tips / Handling data errors due to missing rows

By default ZappySys API Drivers sends minimum 2 API requests. First one to fetch metadata and second, third… for Data. There will be a time when you wont have any data and it may throw error about no records found because it fails to parse metadata. You can avoid such issue by supplying metadata before hand so you can avoid expensive API calls. Check this article see how you can supply cached metadata in your SQL Query as below. This will avoid Metadata call and speedup significantly. It will also avoid errors due to no rows found for specified criteria.

select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:bounceRate,ga:uniquePageviews&dimensions=ga:date''
  ,IncludeParentColumns=''False''
  ,Meta=''[ 
     {"Name": "ga:date","Type": "String", "Length": 16 },
     {"Name": "ga:bounceRate", "Type": "String", "Length": 36 },
     {"Name": "ga:uniquePageviews", "Type": "String", "Length": 10 }
    ]''
 )
')

ODBC Connection – Google Analytics Data in Other Apps (e.g. SSRS / Power BI / Excel / Tableau)

So far we have talked accessing data inside SQL Server using Data Gateway Approach but what if you like to access in other apps ? Well you have three options.

1. Use Microsoft SQL Server Driver to call Linked Server Queries (OPENQUERY approach we saw earlier)
2. Access Google Analytics Data using ODBC connectivity
3. Use Microsoft SQL Server Driver to send direct SQL queries to Gateway (Bypass Linked Server – Use inner SQL query without OPENQUERY in this case)

Method-1 : Linked Server Approach

First approach most likely works in all cases because most apps will support connecting to SQL Server using OLEDB / ADO.net / ODBC / JDBC Drivers. So calling SQL Queries which uses Linked Server (i.e. OPENQUERY statement)  is advisable.

Method-2 : ODBC Driver Approach

However in some cases this may not be possible (e.g. You don’t have SQL Server inhouse or you don’t want to rely on SQL Server to access API). In such case you can use ODBC connectivity in your app. Since ODBC is widely adopted standard most app out there (Except JAVA apps) should support ODBC Drivers. If you like to use this approach then create ODBC DSN rather than Data Gateway Data source and use it in your Reporting / ETL / Custom Apps.

Open ODBC Data Sources (Create DSN)

ODBC User DSN Tab: Add new Driver Screen

Method-3 : Direct connection to Data Gateway (By pass Linked Server)

Last option we suggest for cases like JAVA Apps / Linux / Mac machines where you cannot install ZappySys ODBC Drivers and you dont have option to use SQL Server Linked Server either. In such case you can try to send SQL Queries to Gateway directly using Microsoft SQL Server Compatible Drivers (i.e. ODBC /JDBC). See this example how we called API Queries inside JAVA Apps.

Google Analytics / REST API Integration in various apps

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

In this article we explored many ideas of JSON / API integration in SQL Server.  We saw how to create OAuth App for Google API and import Google Analytics data into SQL Server Table without doing any ETL using pure T-SQL code (Query / Views /Stored Procs). You can Download  FREE Trial of ODBC PowerPack and try yourself see how easy it is to query any API inside SQL Server and avoid expensive ETL processes.

The post Import Google Analytics data into SQL Server / Reporting / ETL appeared first on ZappySys Blog.

Google AdWords is one of the most popular Advertise Platforms. In this article, you will learn how to get data from Google AdWords by calling Google AdWords Web Service method and using drag and drop SSIS XML/SOAP connector (No coding required :)). Google AdWords Web Services allows retrieving various data such as account info, campaigns; as well as writing data, such as creating a campaign, add a keyword. Basically, it allows controlling your AdWords account programmatically instead of using the AdWords dashboard. For demo purposes, we will get data from Google AdWords Account and load it into Microsoft SQL Server (but you can use this technique to save any supported targets in SSIS (e.g. Excel, FlatFile, Oracle, MySQL, IBM DB2, etc.).

If you desire to download a report from AdWords, you must follow a different approach described in an article on how to download AdWords report. It describes how to download a report using ODBC and XML Driver but the concepts introduced there can be applied to SSIS as well.

If you are looking for a similar article for Google DoubleClick for Publisher (Google DFP) then check this article.

You will also learn how to create Google Ads or modify any other objects by calling SOAP API using SSIS REST API Task.

SSIS Tasks/Connectors discussed in this Article

In this article we will use following Connector(s) and Connection Manager to archive full data integration with Google AdWords.

Following SSIS Connectors support OAuth Connectivity. When you use below connectors no coding required to pull data from any OAuth enabled services (e.g. Twitter, Facebook, LinkedIn).

	REST API Task
	OAuth Connection Manager
	XML Source (REST API, SOAP Call or File data)

Two different ways to get Google AdWords Data in SSIS

There are two ways you can extract Google AdWords data in SSIS.

1. Use Google Analytics Source (For more info see this article)  (Consume AdWords data using Google Analytics REST API)
2. Use XML Source along with OAuth connection manager (Consume AdWords data using SOAP API)

In this article we will only discuss Approach #2. For approach #1 refer see this link

Advantage of #2 is you get total control of full API set for AdWords but you have to do lot more manual work. For Approach #1 is lot simpler but you get access to limited reports (Dimensions and Metrics).

Google AdWords API Integration Example Screenshot

Below is screenshot of sample package discussed in this article (Download from here). As you can see we tried to demonstrate Reading data and Managing data (e.g. Create Ad) in the same package. You can extend these technique to perform any type of operations in Google AdWords or similar service such as DoubleClick for Publisher

Google AdWords API Integration Example – Create Ad, Get Accounts, Get Campaigns, Get Ad Groups, Get Ads (Extract AdWords data via SOAP API, Read data using Selector Method or use AWQL query language)

Step-By-Step Tutorial

Here are few high level steps you will have to perform to extract data from Google AdWords API using SSIS. This section assumes you have ClientID and Developer Token to access data. if you don’t have that then read next 2-3 sections.

1. Download and install SSIS PowerPack from here
2. Create new SSIS Project
3. From SSIS toolbox drag DataFlow Task and go to dataflow designer
4. From toolbox of drag ZS XML Source
5. Double click XML Source to configure it
6. Specify API URL from which you want to read data
   Example : https://adwords.google.com/api/adwords/mcm/v201509/ManagedCustomerService
7. Now check Use Credentials check box and select New ZS-OAuth connection manager.
8. On connection manager UI you can select Google from Provider Dropdown.
9. Under Scopes enter following (Scope is basically permission)
   

   https://www.googleapis.com/auth/adwords

    
10. Now click Generate Token and when prompted login and in the last step Accept.
11. If everything went ok then you will see AccessToken and RefreshToken populated.
12. Click Test connection and close the UI by clicking OK
13. Now on XML Source – Select HTTP Request Method as POST
14. In the Request Body click edit and enter request as below (Replace DEV_TOKEN and CLIENT_ID placeholders below)
    

    <?xml version="1.0" encoding="utf-8"?>
    <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    	<soap:Header>
    		<RequestHeader xmlns="https://adwords.google.com/api/adwords/mcm/v201605">
    			<developerToken>ENTER_YOUR_DEV_TOKEN_HERE</developerToken>
    			<clientCustomerId>ENTER_YOUR_CLIENT_ID_HERE</clientCustomerId>
    			<userAgent>ZappySys SSIS PowerPack v1</userAgent>
    		</RequestHeader>
    	</soap:Header>
    	<soap:Body>
    		<get xmlns="https://adwords.google.com/api/adwords/mcm/{{User::ApiVersion}}">
    			<serviceSelector>
    				<fields>CustomerId</fields>
    				<fields>Name</fields>
    			</serviceSelector>
    		</get>
    	</soap:Body>
    </soap:Envelope>

     

    If you are wondering how did we get above XML fragment for Request Body then Check this article on how to use SoapUI (free 3rd party tool) to create SOAP request Bodyfrom WSDL. Google AdWords has many API endpoints for different service. For each service you have different Api URL and different WSDL. In the next section you can learn how to create correct SOAP Body (for POST) using SoapUI tool.

    For example if you are calling ManagedCustomerService API for version v201609 then your help page URL would be like this
    https://developers.google.com/adwords/api/docs/reference/v201609/ManagedCustomerService

    And on the same page you will see WSDL link below
    https://ads.google.com/apis/ads/publisher/v201702/NetworkService?wsdl
    (Just download that WSDL XML and save to local disk then use with SoapUI to generate XML body to submit for any DFP API call.)

    See this link for more information (Its for Google DoubleClick Ad Platform but you will get an idea)

15. Go to Array handling tab at the bottom and enter text : ns2:entries – By doing this we will treat <ns2:entries> node as array eventhough only one entry found in response (By default content with multiple records treated as array).
    

    Handling Array Element Nodes in SOAP Response

16. On the filter expression click edit and highlight entries node and click ok. This will produce something like below in filter textbox
    

    $.soap:Envelope.soap:Body.ns2:getResponse.ns2:rval.ns2:entries[*]

     
17. Now click preview to see actual data.
18. Click OK to save UI. Now you can connect source to target such as SQL Server or Trash Destination.

NOTE: in above example we used v201605 API version but change it to latest if this API is no more valid. Google expires API version pretty often so keep that in mind if you get depreciated API version error.

Register for Developer Token to access AdWords account data using API

First important step to access any Google AdWords Account is request developer token. Click this link for step by step information. Watch short vide to get overview on entire process to access data via API.

Find your Client ID

Next thing you will need to know is your ClientID. Most of API Requests will require your ClientID. Perform following steps to find your ClientID.

Login to your AdWords Account using email address. At the top you will see your client ID as below screenshot.

How to find ClientID of your Google AdWords Account (Useful for making SOAP API Calls)

Connect to Google AdWords API using OAuth

Connecting to any Google API will require OAuth 2.0. SSIS PowerPack has Protocol so you can easily extract data from API without coding complex token extract/renew workflow. Such things usually requires some sort of programming but in our case you will do few clicks as described in Step-By-Step section earlier.

Here is the screenshot of OAuth Connection Manager.

Connecting to Google AdWords API using OAuth 2.0 in SSIS (Consuming XML/SOAP API)

Extract data from Google AdWords using SOAP API Calls

When you define filter expression you can check/uncheck “Include Parent Columns” setting. For AdWords SOAP Api you can uncheck so you only extract columns from rvals node in XML Response.

Extract data from Google AdWords SOAP/XML Response

SOAP Response Paging / Looping

Many time you will get response which is larger than default page size in AdWords API Call. In that case you can specify few pagination attributes on XML Source Paging tab. For more information on paging options you may refer this article . This article is talking about JSON Source but same options are applicable for XML Source too.

REST API Pagination – Passing PageNumber inside POST Body (Example: Google AdWords SOAP Api)

Configure REST API Pagination – using Page Number via POST Body Method

Read Google AdWords data and Load into SQL Server

Using XML Source you can extract data in tabular format and then you can map that to SQL Server Target (Use OLEDB Destination / ADO.net Destination).

Extract data from Google AdWords and load into SQL Server table using SSIS

Write data / Manage AdWords Account via REST API Calls

So far we have seen how to read data from AdWords account. But what if you have to create new records or write/update AdWords data via API Calls in automated manner? No worry – If you have need for that then use SSIS REST API Task which supports adhoc HTTP WebRequests including POST/DELETE/PUT etc. You can also upload files via REST API call using this task. This task supports rich error handling and saving your response into variable or files.

Create Ad under specific AdGroup

Below example shows how to create new ad under specific AdGroup using REST API Task

Notice we have used few placeholders to make things dynamic using SSIS Variables. Once REST API Task is executed you will see new ad created.

Creating Google AdWords Ad using SOAP/XML API in SSIS (REST API Task)

Verify Google Ad created programmatically using SOAP/XML API Call (SSIS REST API Task)

Methods for Creating new records

Google AdWords API supports mutate method for writing data. Many service endpoints supports this method. Below example request will create new Ad.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<developerToken>{{User::varDevToken}}</developerToken>
			<clientCustomerId>{{User::varClientId}}</clientCustomerId>
			<userAgent>My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)</userAgent>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<mutate xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<operations>
				<operator>ADD</operator>
				<operand>
	                <adGroupId>30800928123</adGroupId>
	                <ad xsi:type="TextAd">
	                    <displayUrl>//zappysys.com/ad1</displayUrl>
	                    <finalUrls>//zappysys.com/ad1</finalUrls>
	                    <headline>This is my Ad</headline>
	                    <description1>This is Line 1</description1>
	                    <description2>This is Line 2</description2>
	                </ad>            
				</operand>
			</operations>
		</mutate>
	</soap:Body>
</soap:Envelope>

Methods for Reading AdWords data

Google AdWords API supports two methods for reading data. Most of AdWords service endpoints supports GET method also known as Selector method (Its basically sending query in XML format). Seconds method is Query Method. It uses SQL like query language known as AWQL. (Read more about AWQL query language). If you are SQL Programmer then you may like AWQL Query method better than Selector. But either way you will be able to pass following information.

• Columns you want to return
• Filter (e.g. Where Clause)
• Order By
• Paging information (e.g. PageNumber, PageSize)

Get Method (XML Selector)

For Get Method using XML tags to define columns, filters and order by/paging information. Here is example Get method for below select query

SELECT Id, Heading
WHERE Status = 'ENABLED'
ORDER BY Heading DESC 
LIMIT 0,50

Get Request (Get all Active Ads)

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<developerToken xmlns="https://adwords.google.com/api/adwords/cm/v201605">{{User::varDevToken}}</developerToken>
			<clientCustomerId xmlns="https://adwords.google.com/api/adwords/cm/v201605">{{User::varClientId}}</clientCustomerId>
			<userAgent xmlns="https://adwords.google.com/api/adwords/cm/v201605">My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)</userAgent>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<get xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<serviceSelector>
			    <fields>Id</fields>
			    <fields>Headline</fields>
			    <predicates>
			        <field>Status</field>
			        <operator>EQUALS</operator>
			        <values>ENABLED</values>
			    </predicates>
			    <ordering>
			        <field>Headline</field>
			        <sortOrder>ASCENDING</sortOrder>
			    </ordering>
			    <paging>
			        <startIndex>0</startIndex>
			        <numberResults>50</numberResults>
			    </paging>
			</serviceSelector>
		</get>
	</soap:Body>
</soap:Envelope>

Query Method (AWQL Query – SQL like query language)

Below is sample Soap request for AWQL Method to read Ads listed under from specific AdGroups. To learn more AWQL Query Syntax Click Here.

To learn more about which Columns you can use inside your AWQL Query Click here. Note that each service support different fields for Select Clause, Where Clause and Order By Clause. AWQL language can be also used to query Predefined AdWords Reports. The only difference is .. FROM clause is required when you query Reports.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<soap:Header>
		<RequestHeader xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<developerToken xmlns="https://adwords.google.com/api/adwords/cm/v201605">YOUR_DEV_TOKEN</developerToken>
			<clientCustomerId xmlns="https://adwords.google.com/api/adwords/cm/v201605">YOUR_CLIENT_ID</clientCustomerId>
			<userAgent xmlns="https://adwords.google.com/api/adwords/cm/v201605">My Test AdWords App (AwApi-DotNet/18.20.0, Common-Dotnet/3.11.0, .NET CLR/4.0.30319.18444, gzip)</userAgent>
		</RequestHeader>
	</soap:Header>
	<soap:Body>
		<query xmlns="https://adwords.google.com/api/adwords/cm/v201605">
			<query>
SELECT Id, Name
WHERE Status = 'ENABLED'
LIMIT 0,5
			</query>
		</query>
	</soap:Body>
</soap:Envelope>

Call AdWords Reports API – Download Campaign Performance data

There will be a time you like to download Reports from Adwords. For that  read this article. Its showing how to use ZappySys ODBC Driver to call AdWords Reporting API but concept is same in SSIS Product too due to similar UI elements.

Download Example SSIS Package

Download SSIS 2012 / 2014/ 2016 Sample

Conclusion

Google AdWords data integration can be very challenging without right set of tools. SSIS PowerPack certainly makes it easy when you have to extract/manage Google AdWords data. Innovative drag and drop approach gives you various options to cover simple or most advanced scenarios without relying on any Programming language (e.g. C#, Java, Python, Ruby etc).

The post How to get data from Google AdWords using SSIS appeared first on ZappySys Blog.

Google Analytics is the most popular web analytics service that tracks and reports website traffic. In this blog post you will learn how to get data from Google Analytics using SSIS using SSIS Google Analytics Source Connector (Using drag and drop approach without Coding). This article assumes you have basic knowledge about SSIS and you have necessary tools installed – Check this if not)

For demo purpose we will load data from Google Analytics into SQL Server Database but you can use same techniques to load data into Oracle, MySQL, Excel or any other RDBMS.

Prerequisite

This article assumes you have installed following tools

1. Install Client Tools for SSIS (atleast one version from below list) – Check this for detail
   1. BIDS for SSIS 2008 (i.e. SQL Server 2008 or 2008 R2) — OR —
   2. SSDT BI for Visual Studio 2012 (i.e. for SQL Server 2012) — OR —
   3. SSDT BI for Visual Studio 2013 (i.e. for SQL Server 2014) — OR —
   4. SSDT BI for Visual Studio 2015 (i.e. for SQL Server 2012/2014 or 2016)
2. Install SSIS PowerPack (Click here to download) it will add new Connector in SSIS Toolbox to access Google Analytics Connector.
3. You have setup correct permission for user to access google analytics data (See below section or read here)

Video Tutorial – Get data from Google Web Analytics Platform

What is Google Analytics Connector for SSIS?

The SSIS Google Analytics Source connector can be used to read data from your Google Analytics Service without learning complex APIs or coding/scripting in Python, Java, C#, etc. It has an easy to use drag and drop interface which allows you to extract large amounts of data, adjusting parameters such as dimensions, metrics, filters, sortby, date range etc. You can also use in-built reports or specify a direct query URL to get the desired data.

Read more about Google Analytics Source Connector

Features

• The intuitive user interface makes Google Analytics data integration super easy and fast without learning complex APIs or scripting languages
• Support for OAuth 2.0 Authentication and Google service accounts (*.p12 certificate file)
• Support for Proxy Server
• Easy to use interface with three different ways to query Google Analytics data: (1) Build query from UI, (2) Use built-in query templates, and (3) Use direct queries
• Support for dynamic dimensions, metrics, segments, filters and sortby fields (e.g. supply these values at runtime)
• Live preview support at design time
• Support for AdWords, AdSense, Ad Exchange and DoubleClick for Publisher (DFP)

Setup Google analytics data permission for users / service account

If you are not an admin of Google analytics and you need to access google analytics then make sure you ask admin to give you correct permission using below steps

To add/edit user permissions for Google Analytics Data perform following steps.

1. Login to your Google Analytics Portal (Login using Admin account or use google user account who setup Google Analytics) Click here to go to google analytics
2. Click on Admin tab > Click User management > Add/Edit user account permission

For more information you may visit official link : How to setup permission for google analytics

Add/Edit Permissions and User Account for Google Analytics Data Access (Service Account/ Normal Google Login)

SSIS Google Analytics Connector – Select Web Profile

Get Data from Google AdWords using SSIS ( use Google Analytics Source Connector)

Conclusion

Integration with Google Analytics Data can be challenging unless you understand how to write C#, Java, Python code using Google SDKs. If you are not a coder or you are in time crunch then drag and drop integration in SSIS using High quality Connector using ZappySys Google Analytics Source Connector will surely make your life easy. It supports all major SQL Server versions upto SQL Server 2016 32-bit/64-bit. Try SSIS PowerPack yourself for FREE.

Keywords

Fetch data from Google Web Analytics using SSIS | Extract google analytics data using SSIS | Load data from Google Analytics without Coding | Google Web analytics data import using REST API

Google Web Analytics Reporting REST API reference link

The post Get data from Google Analytics in SSIS using REST API Call appeared first on ZappySys Blog.