In our previous article we saw how to read Google Analytics data using SSIS. But what if you don’t use SSIS and you still like to Import Google Analytics data into SQL Server or Read inside other Reporting / ETL Apps (e.g. Excel, Power BI, MS Access … and many more). Not all BI tools comes with native driver / connectivity for Google Analytics. Sometimes they do offer Google Analytics connectivity but don’t offer feature you looking for (e.g. Sort, Segment Filter). In such case you have to look for solution outside.

Well not to worry we will show you how simple it is to connect to Google Analytics using ZappySys JSON Driver (For REST API / Files) in few mins. This driver is one of the best REST API drivers you can find in the market. It is so generic that it can virtually access any REST API you may find internally or on public sites (e.g. Facebook, Salesforce, Azure, AWS). Check these articles to see many more use cases of JSON Driver.

At the end of this article you will learn how to Query Google Analytics data in SSMS like below (Yes that’s correct .. Pretty Awesome!!! ).

Import Google Analytics Data into SQL Server Table (using T-SQL Code)

Testing Google Analytics API

If you are new to Google Analytics API then read little bit about Google Analytics REST API here. Google also offers a really good way to test Analytics API using Query Explorer Tool here. So please check that and understand how to craft correct REST API URL which will be used in the next section.

Using Google Analytics Query Explorer Tool – Create REST API Url (Dimension / Metrics Selection Browser)

Sample Google Analytics API Response

Behind the scene here is how API Request and Response Looks like for Google Analytics REST API. Notice that Authorization Header in below request is automatically added by ZappySys Driver or Query Testing tool above. Also notice nextLink attribute in response, its used to fetch more data. By default each response contains upto 10000 rows.

Request

GET https://www.googleapis.com/analytics/v3/data/ga?ids=ga:185737326&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration&dimensions=ga:date
Authorization: Bearer ya29.GlwUxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Response

{
    "kind": "analytics#gaData",
    "id": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday",
    "query": {
        "start-date": "30daysAgo",
        "end-date": "yesterday",
        "ids": "ga:123456789",
        "dimensions": "ga:date",
        "metrics": ["ga:avgSessionDuration"],
        "start-index": 1,
        "max-results": 1000
    },
    "itemsPerPage": 1000,
    "totalResults": 30,
    "selfLink": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday",
    "nextLink": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday&start-index=10001&max-results=10000",
    "profileInfo": {
        "profileId": "123456789",
        "accountId": "33445566",
        "webPropertyId": "UA-33445566-3",
        "internalWebPropertyId": "454545454",
        "profileName": "All Web Site Data",
        "tableId": "ga:123456789"
    },
    "containsSampledData": false,
    "columnHeaders": [
        {
            "name": "ga:date",
            "columnType": "DIMENSION",
            "dataType": "STRING"
        },
        {
            "name": "ga:avgSessionDuration",
            "columnType": "METRIC",
            "dataType": "TIME"
        }
    ], "totalsForAllResults": {
        "ga:avgSessionDuration": "49.3779488037477"
    },
    "rows": [ 
		["20190425", "401.38912133891213"], 
		["20190426", "512.49723756906077"], 
		["20190427", "11.53488372093024"], 
		["20190428", "432.148648648648646"], 
		["20190429", "234.650735294117645"],
		.......
		.......
		.......
		.......
		["20190525", "74.123435294117645"]
	]
}

Creating Google API Project (Obtain Client ID / Secret)

Very first step to call any Google API including Google Analytics is to create Google API Project and register OAuth App to obtain Client ID and Client Secret. If you dont want to go through this hassle and start easy way then ZappySys offers Inbuilt Default App on OAuth Connection UI but we strongly recommend you create your own app rather than using Default App because in Default App API call limit is shared by many. Your data is never shared with ZappySys in any case even you use Default App.

Here is how to create API Project to call Google API. Once you create Google API Project make sure you enable Google Analytics API.

So once you have Client ID and Client Secret we can move forward to the next step.

Loading Google Analytics data into SQL Server

In our previous article we saw in depth general idea on how to import REST API data in SQL Server (T-SQL Script) without relying on any ETL tools.  Now lets look at step by step on how to load Google Analytics data into SQL Server.

In this example, our goal is to extract a Google Analytics report which shows average session duration and new users count by date (For last 30 days).

Setup ZappySys Data Gateway

Very first step to access any REST API Data inside SQL Server is to configure ZappySys Data Gateway. We covered this in previous article.  But here are high level steps for initial setup of Data Gateway.

1. Search for Gateway in Start menu and  Select ZappySys Data Gateway
   

   Open ZappySys Data Gateway

2. Create new User in Data gateway on Users tab. Enter username and password (we will use this when we create Linked Server) . Check Admin Option
   

   Add Data Gateway User

Setup Google Analytics API Data Source in Gateway / ODBC

Once gateway user is setup, now lets create a new Data Source for Google Analytics API. In this section we will talk how to create data source in gateway but most instructions can be used to create data source in ODBC too. At the end of this article we explained how to Launch ODBC Datasource UI. Step#1 and 2 are only different, all other steps same in ODBC DSN creation.

1. Click Add Data Source option (Select Native – JSON Driver )
   

   Add Gateway Data Source (Native JSON Driver)

2. Click Edit to configure data source
   

   Edit Gateway Data Source Settings

3. Now lets configure Driver settings.  Click on Load Connection String button.
   

   Load ZappySys Driver ConnectionString to configure UI

4. Enter the following ConnectionString to get started with predefined settings.
   Change ids (111223344 to your own Profile ID in DataPath URL). Other URL Parameters explained in next section.
   

   DRIVER={ZappySys JSON Driver};
   DataPath='https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date';
   DataConnectionType=OAuth;
   ScopeSeparator='{space}';
   Scope='https://www.googleapis.com/auth/analytics https://www.googleapis.com/auth/analytics.readonly';
   ServiceProvider=Google;
   Filter='$.rows[*]';
   ArrayTransformType=TransformSimpleTwoDimensionalArray;
   ArrayTransColumnNameFilter='$.columnHeaders[*].name';
   RequestMethod='GET';
   NextUrlAttributeOrExpr='$.nextLink';
   ClientId='xxxxxxxxxxxx';
   ClientSecret='yyyyyyyyyyyyyy';
   UseCustomApp='True'

    
5. URL used in DataPath is most important (parameters explained below) . You can use   Query Explorer Tool here to build same URL.
   

   https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date

   Parameters Explained 
   
   Here is some required parameters used in above REST API URL.
   ids=ga:11223344
   This is your Profile ID which you like to extract.  Read more
   start-date=30daysAgo
   This can be yyyy-MM-dd or some other some predefined date keywords (e.g today, yesterday 10daysAgo). Read more.
   end-date=yesterday
   Same as above
   metrics=ga:avgSessionDuration,ga:newUsers
   This can be comma separated list of metrics you like to extract (max 10). Read more.
   dimensions=ga:date
   This can be comma separated list of metrics you like to extract (max 7). Read more.
6. Now its time to configure OAuth settings. Click on Configure Connection. Enter Client ID and Client Secret Obtained in the previous section (see here) and then click Generate Token as below.
   

   Configure Google API Data Source / DSN (Google Analytics API Example)

7. Thats it now can click on Preview Tab and run sample query like below.
   

   select * from $

   –OR–
   (Select Table name from dropdown and generate default query and then you can add alias , remove unwanted columns)
   

   SELECT 
   "ga:date" date, 
   "ga:avgSessionDuration" avgSessionDuration,
   "ga:newUsers" newUsers
   FROM "rows"

   –OR–
   (Type custom query with your own URL generated from here, Change Profile ID – from 11223344 to your own id)
   

   SELECT * FROM $
   WITH(
   	 Src='https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date'
   	,IncludeParentColumns='False'
   )

   

Create Linked Server in SQL Server using T-SQL Script

Once our Gateway Data Source is configured we can now move to SQL Server Part to define Linked Server.

Here is how you can define linked server to access Google Analytics Data inside SQL Server. There are two ways to create Linked Server (1)  via T-SQL script (2) via SSMS UI)

Here is how to create using Script.

USE [master]
GO

--drop existing linked server for same name if already found
EXEC master.dbo.sp_dropserver @server=N'GOOGLE_ANALYTICS_LS', @droplogins='droplogins'
GO

EXEC master.dbo.sp_addlinkedserver 
	@server = N'GOOGLE_ANALYTICS_LS', 
	@srvproduct=N'', 
	@provider=N'SQLNCLI', 
	@datasrc=N'localhost,5000', --this is host name and port number where  ZappySys Data Gateway is running
	@catalog=N'ZS-GA' --this must be same name as defined on Data Source tab Grid on ZappySys Data Gateway

EXEC master.dbo.sp_addlinkedsrvlogin 
	@rmtsrvname=N'GOOGLE_ANALYTICS_LS',
	@useself=N'False',
	@locallogin=NULL,
	@rmtuser=N'SOME-GATEWAY-USER', -- user name created on ZappySys Data Gateway
	@rmtpassword='SOME-GATEWAY-PASSWORD' -- password for ZappySys Data Gateway user
GO

Create Microsoft SQL Server Linked Server using SSMS UI

Import Google Analytics Data into SQL Server Table (T-SQL Code)

Now lets see how to run sample query to import Google Analytics Data into SQL Server. Notice that how we have escaped quotes in OPENQUERY.
NOTE: Change id 11223344 with your own Id (use this tool to create URL)

select * into GaApiLoad from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date''
  ,IncludeParentColumns=''False''
 )
')
select * from GaApiLoad

Import Google Analytics Data into SQL Server Table (using T-SQL Code)

Expose Google Analytics Data as View

To make it simple you can expose above query as SQL Server views so you can access them from any Reporting / ETL Tools.

CREATE VIEW vw_Google_Analytics_BounceRate_PageViews_Last30Days
AS
--change ids to your own Profile ID
select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=2019-04-01&end-date=2019-04-30&metrics=ga:bounceRate,ga:uniquePageviews&dimensions=ga:date''
  ,IncludeParentColumns=''False''
 )
')
GO

Making things Dynamic using Stored Procedure with Parameters

Now let’s make few things dynamic. We will change above query so we can pass Id, Dimensions and Metrics as parameters.

Run below script to Create a stored proc and fetch google analytics data in SQL Server.

use MyDatabase
go
create proc usp_GoogleAnalyticsData
	@ids varchar(1000),
	@startDate varchar(50)='30daysAgo',
	@endDate varchar(50)='yesterday',
	@metrics varchar(200)='ga:avgSessionDuration,ga:newUsers',
	@dimensions varchar(200)='ga:date'	
as

declare @sql varchar(4000)

set @sql='select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
''SELECT * FROM "rows"
WITH 
 (
   SRC=''''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:' + @ids 
   +'&start-date='+ @startDate 
   +'&end-date='+ @endDate 
   +'&metrics='+ @metrics 
   +'&dimensions='+ @dimensions 
   +'''''
  ,IncludeParentColumns=''''False''''
 )
'')'

print @sql
execute(@sql)

go

--Example: query google analytics data
exec usp_GoogleAnalyticsData 
	@ids='11223344',  --Change this ID - Obtain from here https://ga-dev-tools.appspot.com/query-explorer/
	@startDate ='30daysAgo',  --or like this 2019-04-01
	@endDate   ='yesterday', --or like this 2019-04-30
	@dimensions='ga:date', 
	@metrics   ='ga:bounceRate,ga:uniquePageviews'

Insert Google Analytics Data into table (Save Stored Procedure Output)

Now what if you like to save Stored Proc output into a table? Well for that you must create table with same structure as stored proc output and then use INSERT INTO sql like below.

--import google analytics data in a table
if(object_id('GaDataLoad') is not null)
	drop table GaDataLoad
create table GaDataLoad([Date] date, BounceRate numeric(10,5), PageViews numeric(10,5))  

insert into GaDataLoad
exec usp_GoogleAnalyticsData 
	@ids='11223344',  --Change this ID - Obtain from here https://ga-dev-tools.appspot.com/query-explorer/
	@startDate ='30daysAgo',  --or like this 2019-04-01
	@endDate   ='yesterday', --or like this 2019-04-30
	@dimensions='ga:date', 
	@metrics='ga:bounceRate,ga:uniquePageviews'

select * from GaDataLoad

Performance Tips / Handling data errors due to missing rows

By default ZappySys API Drivers sends minimum 2 API requests. First one to fetch metadata and second, third… for Data. There will be a time when you wont have any data and it may throw error about no records found because it fails to parse metadata. You can avoid such issue by supplying metadata before hand so you can avoid expensive API calls. Check this article see how you can supply cached metadata in your SQL Query as below. This will avoid Metadata call and speedup significantly. It will also avoid errors due to no rows found for specified criteria.

select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:bounceRate,ga:uniquePageviews&dimensions=ga:date''
  ,IncludeParentColumns=''False''
  ,Meta=''[ 
     {"Name": "ga:date","Type": "String", "Length": 16 },
     {"Name": "ga:bounceRate", "Type": "String", "Length": 36 },
     {"Name": "ga:uniquePageviews", "Type": "String", "Length": 10 }
    ]''
 )
')

ODBC Connection – Google Analytics Data in Other Apps (e.g. SSRS / Power BI / Excel / Tableau)

So far we have talked accessing data inside SQL Server using Data Gateway Approach but what if you like to access in other apps ? Well you have three options.

1. Use Microsoft SQL Server Driver to call Linked Server Queries (OPENQUERY approach we saw earlier)
2. Access Google Analytics Data using ODBC connectivity
3. Use Microsoft SQL Server Driver to send direct SQL queries to Gateway (Bypass Linked Server – Use inner SQL query without OPENQUERY in this case)

Method-1 : Linked Server Approach

First approach most likely works in all cases because most apps will support connecting to SQL Server using OLEDB / ADO.net / ODBC / JDBC Drivers. So calling SQL Queries which uses Linked Server (i.e. OPENQUERY statement)  is advisable.

Method-2 : ODBC Driver Approach

However in some cases this may not be possible (e.g. You don’t have SQL Server inhouse or you don’t want to rely on SQL Server to access API). In such case you can use ODBC connectivity in your app. Since ODBC is widely adopted standard most app out there (Except JAVA apps) should support ODBC Drivers. If you like to use this approach then create ODBC DSN rather than Data Gateway Data source and use it in your Reporting / ETL / Custom Apps.

Open ODBC Data Sources (Create DSN)

ODBC User DSN Tab: Add new Driver Screen

Method-3 : Direct connection to Data Gateway (By pass Linked Server)

Last option we suggest for cases like JAVA Apps / Linux / Mac machines where you cannot install ZappySys ODBC Drivers and you dont have option to use SQL Server Linked Server either. In such case you can try to send SQL Queries to Gateway directly using Microsoft SQL Server Compatible Drivers (i.e. ODBC /JDBC). See this example how we called API Queries inside JAVA Apps.

Google Analytics / REST API Integration in various apps

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

In this article we explored many ideas of JSON / API integration in SQL Server.  We saw how to create OAuth App for Google API and import Google Analytics data into SQL Server Table without doing any ETL using pure T-SQL code (Query / Views /Stored Procs). You can Download  FREE Trial of ODBC PowerPack and try yourself see how easy it is to query any API inside SQL Server and avoid expensive ETL processes.

The post Import Google Analytics data into SQL Server / Reporting / ETL appeared first on ZappySys Blog.

Tableau is one of the most popular Reporting / Visualization tool for BI / Data analytics. It comes with many out-of the box connectors to pull data from some popular data sources but still it seriously lakes capability to consume data from millions of other REST / SOAP data sources out there for which Tableau not going to create native connectors.

In this article we will cover how to import REST API in Tableau, We will also look at various examples to read from Files or any JSON / XML SOAP / CSV API in few clicks. We will use ZappySys API Drivers to read data from Web API or Local XML / JSON / CSV Files without doing any coding or ETL.

ZappySys API drivers are most advanced API drivers available in the market with many options to connect to virtually any REST / SOAP API data source. For demo purpose we will load ODATA API (JSON Format) using ZappySys JSON Driver. Concepts / steps listed in this articles are mostly same for other type of API formats too (e.g. XML API or CSV API).

Requirements

Before we get started for step by step tutorial, make sure following requirements are met.

1. You have installed Tableau Desktop
2. Make sure SQL Server Instance is installed somewhere. We will use Linked Server to access REST API. You can download FREE SQL Express Version (DB Server) and SSMS from here (Client).
3. Download and Install ZappySys ODBC PowerPack (API Drivers) – These drivers must be installed on Windows Machine (Can be different than Tableau Desktop / Server)

Step By Step – Loading REST API data in Tableau

Now let’s get started. To load REST API data in Tableau using ZappySys drivers basically there are two approaches.

Approach#1 You can use direct ODBC Connection to call ZappySys Drivers
Approach#2 Use SQL Server Connection in Tableau to send ZappySys Driver queries to ZappySys Data Gateway (using Linked Server OPENQUERY).

We recommend Approach#2 for any ZappySys driver use case in Tableau.  Now you must be wondering why use approach#2 and not #1 ? Tableau doesn’t play well when it comes to 3rd party ODBC drivers because behind the scene Tableau generate complex SQL statements which may not be supported by ODBC driver its trying to call (e.g. JOIN Syntax, CAST functions, Sub queries). See some issues like this and this one.  So if we use SQL Server to wrap those complex queries then it becomes much simpler because Tableau has very good support for SQL Server compared to 3rd party ODBC Drivers.

To access REST API in tableau using SQL Server Connection, you have to perform the following high level steps.

1. Make sure you have access to Microsoft SQL Server instance. If you don’t have one you can always use lightweight FREE SQL Express Edition (Download). This is free for Production use too.
2. Install ZappySys ODBC PowerPack (API Drivers)
3. Configure ZappySys Data Gateway (Create API connection) – We will see in depth, later in this article.
4. Create Linked Server in Microsoft SQL Server and connect to ZappySys Data Gateway
5. Import API data in Tableau using Microsoft SQL Server Connector by calling Linked Server SQL Queries

Assuming you have already done first step lets get started with remaining steps. You can also read this article for other example of calling SQL Server Queries via ZappySys Data Gateway.

Configure ZappySys Data Gateway

Create Linked Server in SQL Server

Once REST API Data source is configured in ZappySys Data Gateway, we can move forward to creating Linked Server part. At this point, you must be wondering why I have to use SQL Server to call ZappySys ODBC Driver? Why can’t I directly use ZappySys ODBC Driver in Tableau?

Technically, you can use Generic ODBC Connector in Tableau to call ZappySys ODBC Drivers for JSON API / XML API or CSV API but there are many problems with it.

Tableau Generic ODBC Connection option generates many unsupported SQL Syntax (e.g. Temp tables, Subqueries, date functions) which may not be supported by most non-relational ODBC drivers (e.g. ZappySys API drivers). Using SQL Server Linked Server we can bridge between ZappySys Drivers and Tableau to access any REST API. So let’s see how to do this.

Import REST API data in Tableau using Microsoft SQL Server Connector

Now we are ready to call REST API in Tableau. Perform following steps to Connect REST API to Tableau.

1. Open Tableau Desktop and click File > New
2. To create new Connection click More > Microsoft SQL Server > Enter your credentials to connect to SQL Server.
   

   Connect to REST API in Tableau using Microsoft SQL Server (Linked Server OPENQUERY approach)

3. Once connection is created for SQL Server we can read REST API data 3 different ways
   1. Query View which contains OPENQUERY to Linked Server for REST API data
   2. Use direct SQL Query using OPENQUERY
   3. Use Stored Procedure (Mostly useful to parameterize calls
4. See below example to pull data from REST API in Tableau using SQL View approach
   

   Create new Data Source in Tableau to Query REST API or file data (JSON / XML / CSV )

5. Here is how to Create Data Source using direct SQL Query. You can enter following Example Query.
   

   SELECT * FROM OPENQUERY( YOUR_LINKED_SERVER , 
   'SELECT * FROM $
   WITH (Src=''https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json''
   	 ,Filter=''$.value[*]''
   )');

    
   

   Create Tableau REST API Datasource using direct SQL Query

6. Once your data sources are created you can click on Sheet1 and drag fields to create visualizations for Tableau Dashboard.
   

   Create Tableau Dashboard from REST API data (JSON / XML / CSV)

Passing Parameters to REST API calls in Tableau (Dynamic SQL)

Now let’s look at scenario where you have to pass parameters to build Dynamic Dashboard. You can try to insert Parameters in your Direct SQL when you build Dynamic SQL but we found some issues with that so we are going to suggest Stored Procedure approach. For more information on Known issue on Dynamic Metadata Check this post.

1. First lets create a stored procedure in SQL Server for Parameter Example. Notice how we added WITH RESULT SETS in the code to describe metadata.
   

   --DROP PROC dbo.usp_GetInvoicesByCountry
   --GO
   /*
   Purpose: Parameterize REST API call via SQL. Call ZappySys Drivers inside SQL Server.
   */
   CREATE PROC dbo.usp_GetInvoicesByCountry
   	@country varchar(100) 
   AS 
   
   DECLARE @sql varchar(max)
   --//Escape single ticks carefully
   SET @sql =  'SELECT OrderID,CustomerID,Country,Quantity FROM $
   WITH (Src=''https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json&filter=Country eq '+ @country +'''
   	 ,Filter=''$.value[*]''
   	 ,DataFormat=''OData''
   )'
   
   DECLARE @sqlFull varchar(max)
   SET @sqlFull='SELECT * FROM OPENQUERY( LS , ''' + REPLACE( @sql, '''', '''''' ) + ''' )'
   PRINT @sqlFull --//For DEBUG purpose
   
   EXECUTE (@sqlFull) 
   WITH RESULT SETS ( 
    (OrderID int,CustomerID varchar(100),Country varchar(100),Quantity int) --//describe first result. If you don't do this then wont work in Tableau 
   )
   GO
   -- Example call
   EXEC dbo.usp_GetInvoicesByCountry @country='Germany'

2. Once you create a stored procedure go to Tableau datasource and select Database which contains the stored procedure we just created.
3. Now find your stored proc and drag it on the datasource pane. You will see parameters UI as below. You can create new parameter – Select New Parameter under Value Column.
   

   Parameterize Tableau REST API datasource (Stored Procedure Parameters)

4. Thats it now you can reuse your parameterized datasource anywhere in Dashboard.
5. If you have need to select Parameters from predefined values rather than free text then edit your parameter and select List option. Define values you like to select from as below.
   

   Allow Tableau Parameter selection from multiple values

6. When you create Tableau Dashboard you will see Parameter dropdown (If you selected List) elase you may see Textbox to enter custom value.
   

   Tableau Dashboard Example – REST API Source with Parameterized Datasource

Read data from JSON Files in Tableau (Single or Multiple)

So far we have seen how to read data from REST API but there will be a time to read data from JSON Files (Single or Multiple). Also files may be compressed (GZip or Zip). Regardless no worries if you are using ZappySys drivers. See below sample query you can submit via SQL Linked Server to read JSON files stored on network share.

Single File Example

SELECT * FROM OPENQUERY([MY_LINKED_SERVER]
, 'SELECT * FROM $
WITH(
   Src=''c:\ssis\large_2018.json''
  ,Filter=''$.Branches[*]''
)')

Multiple Files Example

SELECT * FROM OPENQUERY([MY_LINKED_SERVER]
, 'SELECT * FROM $
WITH(
   Src=''c:\ssis\large_*.json''
  ,Filter=''$.Branches[*]''
)')

Read Compressed Files (GZip / Zip Format)

SELECT * FROM OPENQUERY([MY_LINKED_SERVER]
, 'SELECT * FROM $
WITH(
   Src=''c:\ssis\large_*.json.gz''
  --Src=''https://zappysys.com/downloads/files/test/large_file_10k.json.gz''
  ,Filter=''$.Branches[*]''
  ,FileCompressionType=''GZip''  --OR-- Zip --OR-- None
)')

Read CSV Files or URL in Tableau

If you like to read data from CSV files or URL then you can use ZappySys CSV Driver same way.

Here is sample query for CSV data.

Reading Compressed CSV Files in Tableau

SELECT * FROM OPENQUERY([MY_API_SERVICE]
, 'SELECT * FROM $
WITH(
  Src=''https://zappysys.com/downloads/files/test/customerdata.csv.gz''
  ,FileCompressionType=''GZip''  --OR-- Zip --OR-- None
  ,ColumnDelimiter=''|''
  ,HasColumnHeaderRow=''False''
)')

Read multiple files (TAB delimited) in Tableau

SELECT * FROM OPENQUERY([MY_API_SERVICE]
, 'SELECT * FROM $
WITH(
  Src=''''c:\ssis\customerdata_*.csv''''
  ,ColumnDelimiter=''{tab}'' --or use \t for tab delimited files
  ,HasColumnHeaderRow=''True''
)')

Read XML Files or API in Tableau

If you like to read data from XML files or URL then you can use ZappySys XML Driver same way. See below example to read some data from XML Web API. We have also passed credentials along with it for demo purpose.

SELECT * FROM OPENQUERY([MY_API_SERVICE]
, 'SELECT * FROM $
WITH(
	 ElementsToTreatAsArray=''item,slide''
	,Src=''http://httpbin.org/xml''
	,DataConnectionType=''HTTP''
	,UserName=''user1''
	,CredentialType=''Basic''
	,Password=''P@11s#''
	,Filter=''$.slideshow.slide[*]''
)')

Read local XML files

SELECT * FROM OPENQUERY([MY_API_SERVICE]
, 'SELECT * FROM $ WITH (
 SRC=''C:\Data\CustomerData.xml''
, Filter=''$.Root.Row[*]''
, ElementsToTreatAsArray=''Row''
)')

Call POST REST API in Tableau

There will be a time when you like to submit API request as POST rather than GET. See below example how to write POST API Request to send Body along with Custom Headers and Method.

POST Body, Set Method, Headers

SELECT * FROM OPENQUERY([MY_API_SERVICE]
, 'SELECT * FROM $
WITH 
(METHOD=''POST'', HEADER=''Content-Type: text/plain || x-hdr1: AAA''
,SRC=''http://httpbin.org/post''
,BODY=''
{
    id:1,
    notes:"Line1\r\nLine2"
}''
)')

Call SOAP API using XML Driver (POST data) – Token based Authentication

Here is another interesting example to call XML SOAP API. It first authenticates and get Token from SOAP Response. Then use that token for API call. To read more about Dynamic Token Method check this article.

SELECT * FROM OPENQUERY([MY_API_SERVICE]
, 'SELECT * FROM $
WITH(
	 ElementsToTreatAsArray=''urn:Row''
	,Src=''https://zappysys.com/downloads/files/test/soap-getdata.aspx''
	,DataConnectionType=''HTTP''
	,AuthScheme=''{none}''
	,TokenUrl=''https://zappysys.com/downloads/files/test/soap-login.aspx''
	,TokenRequestData=''
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:partner.soap.sforce.com">
   <soapenv:Body>
      <urn:login>
         <urn:username>[$userid$]</urn:username>
         <urn:password>[$password$]</urn:password>
      </urn:login>
   </soapenv:Body>
</soapenv:Envelope>''

	,TokenRequestMethod=''POST''
	,TokenResponseContentFilter=''//*[local-name() = ''''sessionid'''']''
	,TokenRequestHeaders=''Content-Type:text/xml|Accept:*/*|Cache-Control:no-cache''
	,TokenResponseContentType=''Xml''
	,UserName=''MyUser001''
    ,Password=''P@$$w0rdAAc12''
	,CredentialType=''TokenDynamic''
	,Filter=''$.soapenv:Envelope.soapenv:Body.urn:Row[*]''
	,RequestData=''
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:partner.soap.sforce.com">
   <soapenv:Body>
      <urn:sessionid>[$token$]</urn:sessionid>
   </soapenv:Body>
</soapenv:Envelope>''

	,Header=''Content-Type: text/xml;charset=UTF-8 || SOAPAction: "https://zappysys.com/soap-getdata" || Accept: */* || Cache-Control: no-cache''
	,RequestMethod=''POST''
)')

REST API / XML SOAP Pagination Settings for Tableau

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

REST API / SOAP Web Service Connection Settings for Tableau

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Other settings for REST API / SOAP XML Call in Tableau

REST API / XML SOAP Performance Tips for Tableau

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Calling SOAP Web Service in Tableau

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

Conclusion

Tableau has no out of the box features to consume many Web API / JSON / XML data sources out there. ZappySys has developed most innovative drivers which can connect to virtually any REST API / SOAP data sources in Tableau. These drivers also capable reading local files in JSON / CSV or XML format. Feel free to download  ZappySys Drivers here and try yourself . You can always contact ZappySys Support Team here if you need any API integration help.

The post Import REST API in Tableau – Read JSON, SOAP XML, CSV appeared first on ZappySys Blog.