In our previous article we saw how to read Google Analytics data using SSIS. But what if you don’t use SSIS and you still like to Import Google Analytics data into SQL Server or Read inside other Reporting / ETL Apps (e.g. Excel, Power BI, MS Access … and many more). Not all BI tools comes with native driver / connectivity for Google Analytics. Sometimes they do offer Google Analytics connectivity but don’t offer feature you looking for (e.g. Sort, Segment Filter). In such case you have to look for solution outside.

Well not to worry we will show you how simple it is to connect to Google Analytics using ZappySys JSON Driver (For REST API / Files) in few mins. This driver is one of the best REST API drivers you can find in the market. It is so generic that it can virtually access any REST API you may find internally or on public sites (e.g. Facebook, Salesforce, Azure, AWS). Check these articles to see many more use cases of JSON Driver.

At the end of this article you will learn how to Query Google Analytics data in SSMS like below (Yes that’s correct .. Pretty Awesome!!! ).

Import Google Analytics Data into SQL Server Table (using T-SQL Code)

Testing Google Analytics API

If you are new to Google Analytics API then read little bit about Google Analytics REST API here. Google also offers a really good way to test Analytics API using Query Explorer Tool here. So please check that and understand how to craft correct REST API URL which will be used in the next section.

Using Google Analytics Query Explorer Tool – Create REST API Url (Dimension / Metrics Selection Browser)

Sample Google Analytics API Response

Behind the scene here is how API Request and Response Looks like for Google Analytics REST API. Notice that Authorization Header in below request is automatically added by ZappySys Driver or Query Testing tool above. Also notice nextLink attribute in response, its used to fetch more data. By default each response contains upto 10000 rows.

Request

GET https://www.googleapis.com/analytics/v3/data/ga?ids=ga:185737326&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration&dimensions=ga:date
Authorization: Bearer ya29.GlwUxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Response

{
    "kind": "analytics#gaData",
    "id": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday",
    "query": {
        "start-date": "30daysAgo",
        "end-date": "yesterday",
        "ids": "ga:123456789",
        "dimensions": "ga:date",
        "metrics": ["ga:avgSessionDuration"],
        "start-index": 1,
        "max-results": 1000
    },
    "itemsPerPage": 1000,
    "totalResults": 30,
    "selfLink": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday",
    "nextLink": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday&start-index=10001&max-results=10000",
    "profileInfo": {
        "profileId": "123456789",
        "accountId": "33445566",
        "webPropertyId": "UA-33445566-3",
        "internalWebPropertyId": "454545454",
        "profileName": "All Web Site Data",
        "tableId": "ga:123456789"
    },
    "containsSampledData": false,
    "columnHeaders": [
        {
            "name": "ga:date",
            "columnType": "DIMENSION",
            "dataType": "STRING"
        },
        {
            "name": "ga:avgSessionDuration",
            "columnType": "METRIC",
            "dataType": "TIME"
        }
    ], "totalsForAllResults": {
        "ga:avgSessionDuration": "49.3779488037477"
    },
    "rows": [ 
		["20190425", "401.38912133891213"], 
		["20190426", "512.49723756906077"], 
		["20190427", "11.53488372093024"], 
		["20190428", "432.148648648648646"], 
		["20190429", "234.650735294117645"],
		.......
		.......
		.......
		.......
		["20190525", "74.123435294117645"]
	]
}

Creating Google API Project (Obtain Client ID / Secret)

Very first step to call any Google API including Google Analytics is to create Google API Project and register OAuth App to obtain Client ID and Client Secret. If you dont want to go through this hassle and start easy way then ZappySys offers Inbuilt Default App on OAuth Connection UI but we strongly recommend you create your own app rather than using Default App because in Default App API call limit is shared by many. Your data is never shared with ZappySys in any case even you use Default App.

Here is how to create API Project to call Google API. Once you create Google API Project make sure you enable Google Analytics API.

So once you have Client ID and Client Secret we can move forward to the next step.

Loading Google Analytics data into SQL Server

In our previous article we saw in depth general idea on how to import REST API data in SQL Server (T-SQL Script) without relying on any ETL tools.  Now lets look at step by step on how to load Google Analytics data into SQL Server.

In this example, our goal is to extract a Google Analytics report which shows average session duration and new users count by date (For last 30 days).

Setup ZappySys Data Gateway

Very first step to access any REST API Data inside SQL Server is to configure ZappySys Data Gateway. We covered this in previous article.  But here are high level steps for initial setup of Data Gateway.

1. Search for Gateway in Start menu and  Select ZappySys Data Gateway
   

   Open ZappySys Data Gateway

2. Create new User in Data gateway on Users tab. Enter username and password (we will use this when we create Linked Server) . Check Admin Option
   

   Add Data Gateway User

Setup Google Analytics API Data Source in Gateway / ODBC

Once gateway user is setup, now lets create a new Data Source for Google Analytics API. In this section we will talk how to create data source in gateway but most instructions can be used to create data source in ODBC too. At the end of this article we explained how to Launch ODBC Datasource UI. Step#1 and 2 are only different, all other steps same in ODBC DSN creation.

1. Click Add Data Source option (Select Native – JSON Driver )
   

   Add Gateway Data Source (Native JSON Driver)

2. Click Edit to configure data source
   

   Edit Gateway Data Source Settings

3. Now lets configure Driver settings.  Click on Load Connection String button.
   

   Load ZappySys Driver ConnectionString to configure UI

4. Enter the following ConnectionString to get started with predefined settings.
   Change ids (111223344 to your own Profile ID in DataPath URL). Other URL Parameters explained in next section.
   

   DRIVER={ZappySys JSON Driver};
   DataPath='https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date';
   DataConnectionType=OAuth;
   ScopeSeparator='{space}';
   Scope='https://www.googleapis.com/auth/analytics https://www.googleapis.com/auth/analytics.readonly';
   ServiceProvider=Google;
   Filter='$.rows[*]';
   ArrayTransformType=TransformSimpleTwoDimensionalArray;
   ArrayTransColumnNameFilter='$.columnHeaders[*].name';
   RequestMethod='GET';
   NextUrlAttributeOrExpr='$.nextLink';
   ClientId='xxxxxxxxxxxx';
   ClientSecret='yyyyyyyyyyyyyy';
   UseCustomApp='True'

    
5. URL used in DataPath is most important (parameters explained below) . You can use   Query Explorer Tool here to build same URL.
   

   https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date

   Parameters Explained 
   
   Here is some required parameters used in above REST API URL.
   ids=ga:11223344
   This is your Profile ID which you like to extract.  Read more
   start-date=30daysAgo
   This can be yyyy-MM-dd or some other some predefined date keywords (e.g today, yesterday 10daysAgo). Read more.
   end-date=yesterday
   Same as above
   metrics=ga:avgSessionDuration,ga:newUsers
   This can be comma separated list of metrics you like to extract (max 10). Read more.
   dimensions=ga:date
   This can be comma separated list of metrics you like to extract (max 7). Read more.
6. Now its time to configure OAuth settings. Click on Configure Connection. Enter Client ID and Client Secret Obtained in the previous section (see here) and then click Generate Token as below.
   

   Configure Google API Data Source / DSN (Google Analytics API Example)

7. Thats it now can click on Preview Tab and run sample query like below.
   

   select * from $

   –OR–
   (Select Table name from dropdown and generate default query and then you can add alias , remove unwanted columns)
   

   SELECT 
   "ga:date" date, 
   "ga:avgSessionDuration" avgSessionDuration,
   "ga:newUsers" newUsers
   FROM "rows"

   –OR–
   (Type custom query with your own URL generated from here, Change Profile ID – from 11223344 to your own id)
   

   SELECT * FROM $
   WITH(
   	 Src='https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date'
   	,IncludeParentColumns='False'
   )

   

Create Linked Server in SQL Server using T-SQL Script

Once our Gateway Data Source is configured we can now move to SQL Server Part to define Linked Server.

Here is how you can define linked server to access Google Analytics Data inside SQL Server. There are two ways to create Linked Server (1)  via T-SQL script (2) via SSMS UI)

Here is how to create using Script.

USE [master]
GO

--drop existing linked server for same name if already found
EXEC master.dbo.sp_dropserver @server=N'GOOGLE_ANALYTICS_LS', @droplogins='droplogins'
GO

EXEC master.dbo.sp_addlinkedserver 
	@server = N'GOOGLE_ANALYTICS_LS', 
	@srvproduct=N'', 
	@provider=N'SQLNCLI', 
	@datasrc=N'localhost,5000', --this is host name and port number where  ZappySys Data Gateway is running
	@catalog=N'ZS-GA' --this must be same name as defined on Data Source tab Grid on ZappySys Data Gateway

EXEC master.dbo.sp_addlinkedsrvlogin 
	@rmtsrvname=N'GOOGLE_ANALYTICS_LS',
	@useself=N'False',
	@locallogin=NULL,
	@rmtuser=N'SOME-GATEWAY-USER', -- user name created on ZappySys Data Gateway
	@rmtpassword='SOME-GATEWAY-PASSWORD' -- password for ZappySys Data Gateway user
GO

Create Microsoft SQL Server Linked Server using SSMS UI

Import Google Analytics Data into SQL Server Table (T-SQL Code)

Now lets see how to run sample query to import Google Analytics Data into SQL Server. Notice that how we have escaped quotes in OPENQUERY.
NOTE: Change id 11223344 with your own Id (use this tool to create URL)

select * into GaApiLoad from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date''
  ,IncludeParentColumns=''False''
 )
')
select * from GaApiLoad

Import Google Analytics Data into SQL Server Table (using T-SQL Code)

Expose Google Analytics Data as View

To make it simple you can expose above query as SQL Server views so you can access them from any Reporting / ETL Tools.

CREATE VIEW vw_Google_Analytics_BounceRate_PageViews_Last30Days
AS
--change ids to your own Profile ID
select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=2019-04-01&end-date=2019-04-30&metrics=ga:bounceRate,ga:uniquePageviews&dimensions=ga:date''
  ,IncludeParentColumns=''False''
 )
')
GO

Making things Dynamic using Stored Procedure with Parameters

Now let’s make few things dynamic. We will change above query so we can pass Id, Dimensions and Metrics as parameters.

Run below script to Create a stored proc and fetch google analytics data in SQL Server.

use MyDatabase
go
create proc usp_GoogleAnalyticsData
	@ids varchar(1000),
	@startDate varchar(50)='30daysAgo',
	@endDate varchar(50)='yesterday',
	@metrics varchar(200)='ga:avgSessionDuration,ga:newUsers',
	@dimensions varchar(200)='ga:date'	
as

declare @sql varchar(4000)

set @sql='select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
''SELECT * FROM "rows"
WITH 
 (
   SRC=''''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:' + @ids 
   +'&start-date='+ @startDate 
   +'&end-date='+ @endDate 
   +'&metrics='+ @metrics 
   +'&dimensions='+ @dimensions 
   +'''''
  ,IncludeParentColumns=''''False''''
 )
'')'

print @sql
execute(@sql)

go

--Example: query google analytics data
exec usp_GoogleAnalyticsData 
	@ids='11223344',  --Change this ID - Obtain from here https://ga-dev-tools.appspot.com/query-explorer/
	@startDate ='30daysAgo',  --or like this 2019-04-01
	@endDate   ='yesterday', --or like this 2019-04-30
	@dimensions='ga:date', 
	@metrics   ='ga:bounceRate,ga:uniquePageviews'

Insert Google Analytics Data into table (Save Stored Procedure Output)

Now what if you like to save Stored Proc output into a table? Well for that you must create table with same structure as stored proc output and then use INSERT INTO sql like below.

--import google analytics data in a table
if(object_id('GaDataLoad') is not null)
	drop table GaDataLoad
create table GaDataLoad([Date] date, BounceRate numeric(10,5), PageViews numeric(10,5))  

insert into GaDataLoad
exec usp_GoogleAnalyticsData 
	@ids='11223344',  --Change this ID - Obtain from here https://ga-dev-tools.appspot.com/query-explorer/
	@startDate ='30daysAgo',  --or like this 2019-04-01
	@endDate   ='yesterday', --or like this 2019-04-30
	@dimensions='ga:date', 
	@metrics='ga:bounceRate,ga:uniquePageviews'

select * from GaDataLoad

Performance Tips / Handling data errors due to missing rows

By default ZappySys API Drivers sends minimum 2 API requests. First one to fetch metadata and second, third… for Data. There will be a time when you wont have any data and it may throw error about no records found because it fails to parse metadata. You can avoid such issue by supplying metadata before hand so you can avoid expensive API calls. Check this article see how you can supply cached metadata in your SQL Query as below. This will avoid Metadata call and speedup significantly. It will also avoid errors due to no rows found for specified criteria.

select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:bounceRate,ga:uniquePageviews&dimensions=ga:date''
  ,IncludeParentColumns=''False''
  ,Meta=''[ 
     {"Name": "ga:date","Type": "String", "Length": 16 },
     {"Name": "ga:bounceRate", "Type": "String", "Length": 36 },
     {"Name": "ga:uniquePageviews", "Type": "String", "Length": 10 }
    ]''
 )
')

ODBC Connection – Google Analytics Data in Other Apps (e.g. SSRS / Power BI / Excel / Tableau)

So far we have talked accessing data inside SQL Server using Data Gateway Approach but what if you like to access in other apps ? Well you have three options.

1. Use Microsoft SQL Server Driver to call Linked Server Queries (OPENQUERY approach we saw earlier)
2. Access Google Analytics Data using ODBC connectivity
3. Use Microsoft SQL Server Driver to send direct SQL queries to Gateway (Bypass Linked Server – Use inner SQL query without OPENQUERY in this case)

Method-1 : Linked Server Approach

First approach most likely works in all cases because most apps will support connecting to SQL Server using OLEDB / ADO.net / ODBC / JDBC Drivers. So calling SQL Queries which uses Linked Server (i.e. OPENQUERY statement)  is advisable.

Method-2 : ODBC Driver Approach

However in some cases this may not be possible (e.g. You don’t have SQL Server inhouse or you don’t want to rely on SQL Server to access API). In such case you can use ODBC connectivity in your app. Since ODBC is widely adopted standard most app out there (Except JAVA apps) should support ODBC Drivers. If you like to use this approach then create ODBC DSN rather than Data Gateway Data source and use it in your Reporting / ETL / Custom Apps.

Open ODBC Data Sources (Create DSN)

ODBC User DSN Tab: Add new Driver Screen

Method-3 : Direct connection to Data Gateway (By pass Linked Server)

Last option we suggest for cases like JAVA Apps / Linux / Mac machines where you cannot install ZappySys ODBC Drivers and you dont have option to use SQL Server Linked Server either. In such case you can try to send SQL Queries to Gateway directly using Microsoft SQL Server Compatible Drivers (i.e. ODBC /JDBC). See this example how we called API Queries inside JAVA Apps.

Google Analytics / REST API Integration in various apps

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

In this article we explored many ideas of JSON / API integration in SQL Server.  We saw how to create OAuth App for Google API and import Google Analytics data into SQL Server Table without doing any ETL using pure T-SQL code (Query / Views /Stored Procs). You can Download  FREE Trial of ODBC PowerPack and try yourself see how easy it is to query any API inside SQL Server and avoid expensive ETL processes.

The post Import Google Analytics data into SQL Server / Reporting / ETL appeared first on ZappySys Blog.

In this article, you will learn how to refresh a Power BI dataset with Power BI REST API using SSIS and ZappySys SSIS PowerPack. Power BI is a Microsoft reporting product that can get data from virtually any source and display it nicely in a report or a dashboard. Each Power BI report contains a dataset, which can be refreshed to show relevant data in the report. Power BI also has a Scheduled refresh capability, in case you want to refresh a dataset periodically. However, you are limited to only 8 specific times per day you can refresh a dataset (as of May 23, 2019). In the Power BI Premium license you can do it 48 times per day. But what if you need to refresh it more frequently? Or if you desire to refresh it at a non-predefined time; which is not known in advance? E.g. after an ETL process has finished? Here Power BI REST API comes into rescue and lets you refresh a dataset on demand.

Here are a few use cases when you may want to refresh a dataset from a SSIS package, using Power BI REST API:

• The need to refresh a dataset more often than Power BI allows.
• The requirement to refresh a dataset just after some process is finished; e.g. after an ETL process.
• The ability to have absolute control when to refresh multiple datasets. Sometimes, refreshing multiple huge datasets at once could cause memory problems in the Power BI service. So, having an orchestrated refresh would solve this problem.

We will use these ZappySys SSIS PowerPack connectors/connections in this article:

OAuth Connection Manager

REST API Task

Let’s begin!

Prerequisites

1. You have a Power BI account and you have basic Power BI skills.
2. An uploaded Power BI report to Power BI service.
3. A Microsoft Azure subscription (there is a free option with a 200 USD credit).
4. Azure Active Directory set up.
5. SSIS designer installed. Sometimes it is referred to as SSDT or BIDS (download it from the Microsoft site).
6. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
7. ZappySys SSIS PowerPack is installed (download it).

Getting started

Suppose you have a simple report and you want to refresh its dataset using Power BI REST API and SSIS. So far, the dataset has not been refreshed at all:

Preparing to refresh the Power BI dataset using SSIS and Power BI REST API.

To start, create an SSIS package and follow the steps below to accomplish the goal.

Step By Step – How to refresh a Power BI dataset with REST API using SSIS

Create an OAuth application

Create OAuth Connection Manager in SSIS package

1. Drag and drop REST API Task from SSIS Toolbox, double click, select Url from connection, and then select <New ZS-OAUTH Connection> item:
   

   Creating OAuth Connection Manager to connect to Power BI OAuth Application from SSIS.

2. Input Client ID from Step 2 when we were creating an Azure Active Directory OAuth Application.
   

   Configuring OAuth Connection Manager to authenticate to Power BI OAuth Application using SSIS.

3. Then proceed and input the Client Secret from Step 3, Create an Azure Active Directory OAuth application section.
4. Please go on, and fill in Authorization Url and Access Token Url properties with values we copied in Step 4, Create an Azure Active Directory OAuth application section.
5. In Scopes / Permissions field input these scopes:
   https://analysis.windows.net/powerbi/api/Dataset.ReadWrite.All
offline_access
6. Then go to the Advanced tab and in the Callback/Return Url field input this value:
   https://login.microsoftonline.com/common/oauth2/nativeclient
   

   Setting Callback/Return Url (Redirect URI) in OAuth Connection Manager to authenticate to Power BI OAuth Application using SSIS.

7. Then go back to the tab General and click the Generate Token button.
8. Once asked for credentials, log in with your organization user; this will generate Refresh and Access tokens.

Retrieve the ID of the Power BI dataset you want to refresh

1. Go to Power BI datasets.
2. Then click on Datasets.
3. Proceed and click the dataset you want to refresh.
4. Finally, copy the ID of the dataset from the URL, we will use it in the next section.

Getting Power BI dataset ID to refresh it in an HTTP request to Power BI REST API.

Configure REST API Task to refresh a Power BI dataset

We configured OAuth Connection Manager and retrieved the ID of the dataset we want to refresh, now it’s time to configure REST API Task and refresh the dataset. So do the following:

1. Go to the SSIS package and open the REST API Task for configuration.
2. Input URL in such a format:
   https://api.powerbi.com/v1.0/myorg/datasets/DATASET-ID/refreshes Replace DATASET-ID with an actual ID of the dataset. E.g.:
   https://api.powerbi.com/v1.0/myorg/datasets/a38c78fc-8c87-45e3-b961-6dcd3ac1eada/refreshes
   NOTE: If your dataset resides in a Workspace, use this format instead:
   https://api.powerbi.com/v1.0/WORKSPACE-ID/datasets/DATASET-ID/refreshes
3. Then set POST as HTTP Request Method.
4. In the Body field, input this JSON:
   { "notifyOption": "MailOnFailure" }
5. Press the Test Request/Response button to test the request and refresh the dataset at the same time.

Configuring a REST API Task to refresh a Power BI dataset using Power BI REST API and SSIS.

Run the package

Hit F5 and execute the package:

Executing SSIS package and refreshing Power BI dataset.

The Results

Now it’s time to check the results and see if the dataset was refreshed successfully:

1. Again, go to Power BI datasets.
2. Then click on Datasets.
3. Click the dataset you refreshed.
4. Then click “Refresh history” to check the refresh history. We expect two manual refreshes to be there.

Results: Successful Power BI dataset refresh using Power BI REST API and SSIS.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Deployment to Production

Conclusion

You learned how to refresh a Power BI dataset with REST API using SSIS. To accomplish the goal, firstly, we created the Azure Active Directory application and configured OAuth Connection Manager based on it. Then we retrieved the Power BI dataset ID and used it in the ZappySys SSIS PowerPack connector REST API Task. Finally, using the same REST API Task we made the HTTP request to Power BI REST API “refreshes” method and successfully refreshed the dataset. If you need to make a call to another Power BI REST API method, then just configure the REST API Task according to the documentation (for getting data use JSON Source or Web API Destination instead). Finally, if you make many requests to the Power BI REST API service, make sure you check its limitations.

References

https://powerbi.microsoft.com/en-us/

https://docs.microsoft.com/en-us/rest/api/power-bi/datasets/refreshdataset

https://docs.microsoft.com/en-us/rest/api/power-bi/

https://docs.microsoft.com/en-us/power-bi/developer/api-rest-api-limitations

The post How to refresh Power BI dataset with REST API using SSIS appeared first on ZappySys Blog.

Power BI is a very popular Business Analytic tool used to get business information. It is very popular because it is easy to install, simple to learn and very intuitive. Also, REST API is very popular these days and we wanted to show you a way to integrate them in Power BI with the help of the ZappySys ODBC PowerPack. This software includes powerful drivers to query REST API URL, Local XML / JSON files and XML SOAP Web Service with simple SQL queries.

In nutshell, this post will focus on how to call JIRA API using Zappysys ODBC PowerPack. So let’s get started

Prerequisite

1. First, you will need to have our ZappySys ODBC Power Pack.
2. In the second place, you will need to have internet access.
3. On third place, Power BI Desktop (FREE) installed.
4. JIRA Atlassian account.

What is JIRA

JIRA is Atlassian’s popular project- and issue-tracking platform used by development and technical support teams to get more work done and faster.

It combines:

issue tracking – a software application that allows to record and follow the progress of every problem or “issue” that user identifies until the problem is resolved.
agile project management – an iterative approach to planning and guiding project processes. customizable workflow, and a pluggable integration – integrates with Freshdesk, GitHub, Zendesk, Zapbook, Asana and so on.

Getting Started

In order to start, we will show several examples. ZappySys ODBC PowerPack includes the ZappySys JSON Driver that will help you to call JIRA API, retrieve various data from JIRA and load that data in Power BI. To learn more about JIRA API check this help file.

https://jira.atlassian.com/rest/api/2/search?jql=project=JRASERVER

https://YOUR-DOMAIN.atlassian.net/rest/api/2/search?startAt=0&maxResults=100&jql=project=MYPROJECT

Load JIRA data into Power BI (Connect JIRA REST API)

Conclusion

In this blog, we have learned how to retrieve the data from JIRA using ODBC JSON Driver and then Load those data in Power BI with ODBC DSN in a very simple way. You can achieve many more functionalities with the use of ODBC PowerPack.

References

Finally, you can use the following links for more information about the use of JIRA API with our tools:

• JIRA Documentation for API
• Landing Page for ODBC JSON / REST API Driver, you can also find Tutorial Video here.
• Help File: Documentation of JSON Driver.
• Blog/articles link: https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/json-rest-api-driver/

The post How to Import JIRA data in Power BI appeared first on ZappySys Blog.

In this article, we will get Office 365  data in Power BI using Microsoft Graph API and ODBC drivers. Specifically, we will get Excel file data in OneDrive, events from a Calendar and finally, a list in a note in OneNote and then load that data into a Power BI report. We will achieve that by creating an ODBC data source using ZappySys ODBC PowerPack and then querying corresponding Microsoft Graph API methods. ODBC PowerPack allows to query REST/SOAP APIs in an easy way and lets to view responses in a tabular format, which will be perfect to accomplish our goal.

Prerequisites

1. A Power BI account.
2. Wield basic Power BI skills.
3. Power BI Desktop installed.
4. ZappySys ODBC PowerPack installed.

Step By Step – Get Office 365 data in Power BI using Microsoft Graph API and ODBC

Set up an ODBC data source based on ZappySys JSON Driver

Add a data source

Configure the data source

Set up an ODBC data source based on ZappySys XML Driver

1. Before adding another data source based on ZappySys XML Driver, copy the connection string of the data source we just created:
2. Close the window and then add a new data source, but this time based on ZappySys XML Driver.
3. Once a window is opened, give it a name, e.g. System - XML.
4. Then press Load connection string button to load the connection string.
5. Paste the copied connection string and replace JSON text to XML, so that you have { ZappySys XML Driver } in DRIVER property.
6. Close the window.

Create a Power BI report

Reading files from OneDrive (getting Excel data)

Let’s say you have Stock.xlsx stored in Sample Documents folder, OneDrive with this kind of data and you want to show it in a Power BI report:

To have that data in a Power BI report, perform these steps:

1. Open Power BI Desktop and create a new Power BI report.
2. Create a data source based on System – JSON DSN (since we get a JSON response):
   
3. Then in SQL statement area enter this query:
   SELECT * FROM $
WITH(Src='https://graph.microsoft.com/v1.0/me/drive/root:/YOUR FOLDER/Stock.xlsx:/workbook/worksheets/SHEET NAME/usedRange'
,Filter='$.values[1:]'
,ArrayTransformType='TransformSimpleTwoDimensionalArray'
,ArrayTransColumnNameFilter='$.values[0][*]'
)
Using WITH clause lets us override the properties. You can generate the query above using the Query Builder:
   In this step we are overriding Url, Filter and array transformation fields you configured in JSON DSN. The Src value overrides the URL. The Filter property enables us to select the data we want to see in the report. We use [1:] in $.values[1:] to skip the first row because the first row contains the headers and we want only the data. You can configure this option in the Filter Options tab (read more about JSON path).We also have to make the transformation on the JSON and turn JSON arrays into rows. To accomplish that, we override ArrayTransformType. With ArrayTransColumnNameFilter property and [0][*] text, we specify that the first array of parent array will be the headers of the columns. You can read more about transformations in Parse multi-dimensional JSON array article.
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other OneDrive endpoints.
4. Click OK.
5. Drag a table into the report, select the columns and you should see a similar view:
   

Reading calendar entries

Now, let’s say you use a calendar in Office 365 and you want your appointments to be visible in a Power BI report:

1. To start, create a new Power BI report.
2. Then add a data source based on System – JSON DSN (again, we get the response in JSON):
   
3. In SQL statement section enter this query:
   SELECT * FROM $
WITH(Src='https://graph.microsoft.com/v1.0/me/events?$select=subject,start,end,location'
,Filter='$.value[*]'
,Header='Prefer: outlook.timezone="FLE Standard Time"'
)Here we also are overriding HTTP Header so that we get the event times not in UTC/GMT but for the specified timezone. It must be specified in Prefer: outlook.timezone=”{time-zone-string}” format (check an article on how to get a list of supported time zones).
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other Calendar endpoints.
4. Click OK.
5. Drag a table control into the report, select several columns and you should see a similar view:
   

Reading notes from OneNote

Suppose, you have these notes in OneNote and you want “Groceries list” to be shown in a Power BI report:

1. First, we will need to find NOTE ID we are trying to get. To do that, open ODBC Administrator, find System – JSON data source and hit Configure.
2. Then go to the Preview tab, and input this SQL query and hit Preview Data:
   SELECT * FROM $
WITH (Src='https://graph.microsoft.com/v1.0/me/onenote/pages'
,Filter='$.value[*]'
)
3. Once you get a result set, find your note name and copy its ID:
   
4. Then create a new Power BI report.
5. Create a new data source based on System – XML DSN (we are using XML-based DSN because the response will be in HTML):
   
6. In SQL statement section enter this query (use the NOTE ID you retrieved in the previous step):
   SELECT * FROM $
WITH(ElementsToTreatAsArray='div,p'
,Src='https://graph.microsoft.com/v1.0/me/onenote/pages/NOTE ID/content'
,Filter='$.html.body.div[0].p[1:]'
)

   
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other OneNote endpoints.
7. Click OK.
8. Drag and drop Multi-row card visual, select #text column and you should see a similar view:
   
9. We finished! Now you can publish your Power BI reports to Power BI service.

Conclusion

Today we learned how to call Microsoft Graph API methods using ZappySys ODBC PowerPack and load response data into Power BI without any ETL procedures. Particularly, we used OneDrive, Calendar and OneNote Graph API endpoints to achieve that. We created two ODBC data sources based on JSON and XML ODBC Driver to query those endpoints, which let us read data directly in a Power BI report. Finally, we examined how to refresh data source on Power BI using on-premises Power BI gateway.

References

https://powerbi.microsoft.com/en-us/

https://apps.dev.microsoft.com/

https://developer.microsoft.com/en-us/graph/graph-explorer/

https://docs.microsoft.com/en-us/graph/overview

https://docs.microsoft.com/en-us/graph/onedrive-concept-overview

https://docs.microsoft.com/en-us/graph/outlook-calendar-concept-overview

https://docs.microsoft.com/en-us/graph/integrate-with-onenote

The post Get Office 365 data in Power BI using Microsoft Graph API and ODBC appeared first on ZappySys Blog.

In this article, you will learn how to push data into a Power BI dataset from SQL Server. Power BI is a Microsoft reporting product that can get data from virtually any source and display it nicely in a report or a dashboard. Among its dataset types, it has a streaming dataset into which you can push data from a Power BI REST API. Once pushed, data will be immediately refreshed in a visual, pinned to a dashboard. Thus this approach of sending data to a streaming dataset is a perfect way for displaying real-time data. Yet it is not possible to push data to a dataset straight from SQL Server out of the box. However, with ZappySys ODBC PowerPack it is possible to achieve that quite easily. We will employ one of its apps – ZappySys Data Gateway – to make HTTP requests and push data to a streaming dataset.

The use cases when you may want to push data to a streaming dataset from a SQL Server are:

• Displaying real-time data from many SQL Server instances at once (and you cannot use the DirectQuery option in a report).
• Displaying real-time data when many sources write to a dataset, and a SQL Server is one of them.
• Any other case.

Let’s begin and write some data to a Power BI streaming dataset!

Prerequisites

1. You have a Power BI account and you have basic Power BI skills.
2. SQL Server instance installed (can be a SQL Server Express LocalDB instance).
3. SQL Server Management Studio (SSMS) installed.
4. ZappySys ODBC PowerPack installed (must be installed on a Windows machine; can be a different machine than SQL Server is installed on).

Step By Step – Push data into a Power BI dataset from a SQL Server

Add a data source in the Gateway

It’s time to add a data source, which we will use to POST data to Power BI REST API:

1. In the Gateway, click on Data sources tab, click Add, give data source a name and set Connector Type to Native – ZappySys JSON Driver:
2. Then click Edit to modify the data source:
   
3. Finally, paste into Url textbox Push URL of the dataset which we saved in a previous step. Also, set HTTP Request Method to POST:
   

   Configure REST API Connection for Power BI dataset insert (Using ZappySys JSON / REST Driver)

Add a Linked Server to the Gateway

Writing data into Power BI dataset from SQL Server

Once you add a Linked Server you can execute a query to push data into a Power BI streaming dataset:

1. In the same SQL Server Management Studio create a new query and paste the code below. In Body attribute, we pass the JSON in the same format which was shown when we created a Power BI dataset:
   

   SELECT * FROM
   OPENQUERY([GATEWAY], 'SELECT * FROM $
   WITH (Body=''[ 
   	{ "Timestamp" :"2018-11-15T02:57:04.547Z", "Temperature" : 50.5 },
   	{ "Timestamp" :"2018-11-16T02:57:04.547Z", "Temperature" : 55.5 },
   	{ "Timestamp" :"2018-11-17T02:57:04.547Z", "Temperature" : 60.5 },
   	{ "Timestamp" :"2018-11-18T02:57:04.547Z", "Temperature" : 57.5 },
   	{ "Timestamp" :"2018-11-19T02:57:04.547Z", "Temperature" : 62.3 }
   ]''
   ,meta=''[ { "Name": "Dummy", "Type": "String" }]'' )')

   Note: Use the same Linked Server name you created in a previous step. In our case, it is “GATEWAY”.
2. Execute the query.

The results: creating a Power BI report

Now it’s time to check the results and see if data went through:

1. Go to Power BI portal.
2. Select Datasets and click on the add new report icon:
   

   Create new Power BI Report for real time dataset

3. Add a Line Chart, a Table and link them to the dataset.
4. You should see the data we pushed to the dataset:
   

   Showing real time data in Power BI dashboard

Conclusion

You learned how to push data into a Power BI streaming dataset directly from SQL Server. It was possible to achieve that with ZappySys Data Gateway and SQL Server Linked Server option. At first, we created a streaming dataset in Power BI, then configured a datasource in the Gateway, followed with adding a Linked Server to the Gateway, executed T-SQL query in SSMS and pushed some data to the dataset and finally created a Power BI report to check the received data.

When we set Historic data analysis option when creating a dataset, we, in turn, created a dataset of type Push (if we left it off, it would be of type Streaming). It allowed us to create a report easily on top of it, but it has certain limitations. Just be aware when using this option and check Power BI service documentation for more info.

Finally, in SSMS, we executed a query with predefined data. Therefore in real life, if you want to push data from an existing table, you will have to build JSON from it. Make sure you construct a valid JSON and escape the symbols that need to be escaped (such as a double-quote or a backslash).

References

https://powerbi.microsoft.com/en-us/

https://docs.microsoft.com/en-us/rest/api/power-bi/

https://docs.microsoft.com/en-us/power-bi/service-real-time-streaming

https://docs.microsoft.com/en-us/power-bi/developer/api-rest-api-limitations

The post Push data into a Power BI dataset from SQL Server appeared first on ZappySys Blog.

In our previous blog post we saw how to import SOAP API data in Power BI. Now let’s continue to explore more use cases. In this article we will check how to Connect WorkDay in Power BI. We will show you step by step approach on how to call Workday SOAP API in Power BI and create Power BI reports / dashboards using imported datasets.

We will use ZappySys XML / SOAP Driver for this demonstration.

Requirements

1. First, you will need to have Power BI Desktop installed
2. Make sure you have ZappySys ODBC Power Pack installed
3. Download and have install SoapUI

Get Started

Once above requirements are fulfilled, let’s learn how to use Workday SOAP API and import workday data in Power BI.

Here are high level steps to import Workday data in Power BI using ZappySys XML / SOAP Driver.

1. Craft Workday SOAP request XML using tool like SoapUI for desired operation (e.g Get_Employee )
2. Configure ODBC DSN for Workday SOAP API Connection using ZappySys XML / SOAP Driver. (Select HTTP Connection with SOAP WSS mode using workday Userid / password).
3. Test SQL Query for desired SOAP API call
4. Finally, Import Data in Power BI using ODBC Connection, Supply Custom SQL Query during Import Process (Crafted in previous step).

Now let’s look at each step in detail in the following sections

Craft Workday SOAP request XML using SoapUI

Now let’s look at how to use tool like SoapUI to craft SOAP API requests. It would be helpful if you can refer this article to learn more about using SoapUI tool (Check SoapUI section in that article).

Obtain Workday SOAP WSDL URL (API Metadata URL)

First step to consume workday api using SSIS is download SOAP WSDL file. WSDL is an XML file which describes available API operations in Workday. It also describes the structure of XML request / response. Here is the list of available WSDL for various Workday API Services. You can right click on WSDL icon and save to local disk or use Direct WSDL URL in next section for importing in SoapUI tool.

You can find more information about Other Workday API here.

Obtain Workday API URL

Once you have WSDL file, next step is craft correct URL for API service you like to call. Service name can be obtained from here (Check service column). Notice my-instance and my-tenantid placeholders (Its specific to your company).

Syntax:  

https://<workday host name>.workday.com/ccx/service/<tenant name>/<service-name>

https://MY-INSTANCE.workday.com/ccx/service/MY-TenantID/Human_Resources

Load WSDL in SoapUI

Now it’s time to load WSDL and craft some SOAP Request and test. For detailed steps check this (watch Video to learn SoapUI tool) .

1. Open SoapUI tool
2. Click new File > New Soap Project
3. Enter WSDL Path and Name of the project
4. You will see many different operations in the imported project. For example below screenshot shows Human_Resources service and Get Employee API call. You can fill out necessary parameters in your request / remove optional parameters from XML as needed.
   

   Create SOAP Request Body from WSDL (Using SoapUI tool)

Configure Soap Request Body, URL and Credentials in SoapUI

Now it’s time to configure SOAP Request service URL and Credentials. To generate correct Credentials Header in SOAP Request use tool like SoapUI. Check this article to learn more how to generate WSS Security header.

1. Go to the API method you like to call and double click on the default Request node. Check this article to learn more.
2. Change default request parameters (Add /Remove as per your need). You can click on Re-generate icon in the toolbar to reset the request and create optional parameters.
3. Enter the Service URL
   Example:  https://myinstance0001.workday.com/ccx/service/AB11111111/Human_Resources
4. In the properties window select WSS-Password Type as PasswordDigest, Enter workday username and password
5. Once you done click Play button to execute the request. If things go well then you should see response on your right panel as below.
   

   SoapUI – Call Workday SOAP API (Configure WSS Security, Service URL, Body, Test Request / Response)

Creating ODBC DSN for Workday Connection

Once you test correct parameters in your XML Request we can create ODBC DSN for Workday.

1. Open start menu and search for “ODBC 64”. Launch ODBC Data sources for 64 bit
   

   Open ODBC Data Sources from Startup menu

2. Click Add and select ZappySys XML Driver
   

   ZappySys ODBC Driver for XML / SOAP API

3. Once driver UI is visible change default DSN name (e.g. change something to  WorkdayDSN)
4. Enter API EndPoint URL (Get it from SoapUI) and Select connection type as HTTP.
   Example URL:  https://YOUR-INSTANCE.workday.com/ccx/service/YOUR-TENANT/Human_Resources
5. Configure HTTP connection as below.
   

   Configure Workday Connection in ODBC DSN – XML / SOAP Driver

6. Click OK to save connection.
7. Now Configure other settings as below.
   1. Select Method as POST
   2. Select Content Type as text/xml
   3. Enter one required Header for SOAP API as below
      

      SOAPAction :  ""

   4. Enter Body (Get it from SoapUI ). Here is example body for Get_Employee request.
      

      <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:bsvc="urn:com.workday/bsvc">
         <soapenv:Header/>
         <soapenv:Body>
            <bsvc:Employee_Get>
               <bsvc:Employee_Reference>
                  <bsvc:Integration_ID_Reference>
                     <bsvc:ID>XXXX-1234-CHANGE-THIS-XXXXXXX</bsvc:ID>
                  </bsvc:Integration_ID_Reference>
               </bsvc:Employee_Reference>
            </bsvc:Employee_Get>
         </soapenv:Body>
      </soapenv:Envelope>

      NOTE: Change ID to your own record id.
   5. ( Optional ) If your API returns more than one record (e.g. if you call API like Get_Workers) then select Filter as per below screenshot (Click Yes if asked to treat selection as array). For example in below screenshot we called Get_Workers request which returns multiple records under wd:Worker node. Look for Array Icon in the Filter selection. Skip this step if no array extraction needed.
8. After Configuration your screen may look like below.
   

   Configure Workday SOAP Request – URL, Body, Filter

9. Now go to Preview Tab and you will see default SQL query generated for you. Here is example query (Replace Body part as per your need)
   

   SELECT * FROM $
   /*Below WITH clause is optional, uncomment parameter(s) values you like to set 
    which are different than DSN properties.
   */
   WITH(
   	Src='https://xxxxxxxxxx.workday.com/ccx/service/yyyyyyyyy/Human_Resources'
   	--,ElementsToTreatAsArray='wd:Worker'
   	--,DataConnectionType='HTTP'
   	--,SoapWssPasswordType='PasswordDigest'
   	--,UserName='testuser001'
   	--,CredentialType='SoapWss'
   	--,Filter='$.env:Envelope.env:Body.wd:Get_Workers_Response.wd:Response_Data.wd:Worker[*]'
   	--,RequestContentTypeCode='TextXmlUtf8'
   	--,RequestMethod='POST'
   	--,Header='SOAPAction: ""'	
   	,RequestData='
   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:bsvc="urn:com.workday/bsvc">
      <soapenv:Header/>
      <soapenv:Body>
           .......... .......... .......... 
           .......... FROM SOAPUI .........  
           .......... .......... ..........        
      </soapenv:Body>
   </soapenv:Envelope>'
   )

10. Click Preview button to see data. Copy your SQL Query for later step.
    

    Preview Workday Data (Test SQL query for ODBC XML / SOAP Driver)

11. Click OK to save DSN

Import Workday data in Power BI

Now let’s move to the final step. We will use SQL query and DSN created in previous step for data load in Power BI.

1. Open Power BI Desktop
2. Click on Get Data > More Data… > Other > ODBC
   

   Import SOAP API in Power BI (Using ZappySys XML ODBC Driver)

3. Select your ODBC DSN Source from the dropdown (Assuming you have created a ODBC Data Source using ZappySys XML Driver – See the previous Section)
4. Expand Advanced Options to enter custom Query.
5. Enter your SOAP API call SQL query here. For example purpose, you can use below query to get all JOB categories.
   

   SELECT * FROM $
   WITH(
   	 Src='https://xxxxxxxxxx.workday.com/ccx/service/yyyyyyyyyy/Human_Resources'	
   	--,DataConnectionType='HTTP'
   	--,SoapWssPasswordType='PasswordText'
   	--,UserName='testuser001'
   	--,Password='###########'
   	--,CredentialType='SoapWss'
   	,RequestMethod='POST'
   	,RequestContentTypeCode='TextXmlUtf8'
   	,Filter='$.soapenv:Envelope.soapenv:Body.bsvc:Get_Job_Categories_Response.bsvc:Response_Data.bsvc:Job_Category[*]'
   	,ElementsToTreatAsArray='bsvc:Job_Category'
   	,RequestData='
   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:bsvc="urn:com.workday/bsvc">
      <soapenv:Body>
         <bsvc:Get_Job_Categories_Request/>
      </soapenv:Body>
   </soapenv:Envelope>'
   )

    
   

   Import Workday SOAP Web Service Data in Power BI (ZappySys XML Driver)

6. Once you see data preview click OK to import.
7. After data import is done you can edit your dataset (e.g. remove unwanted columns)
8. You can also edit Source query after data is imported (See below)
   

   Edit Power BI Data Source SQL after REST / SOAP Web Service Import

9. That’s it. You can now create your dashboards using data we just imported. See below example.
   

   Import Workday data in Power BI dashboard

Workday API Pagination – Retrieve more than 100 rows

By default Workday API will return only 100 records unless you specify Page / Count in Response_Filter . Regardless it will only send you 999 maximum rows. To fetch more records after first page you have to setup pagination in XML Driver. Below screenshot

1. Select Paging Mode= POST Data Mode
2. Enter Page Number Indicator (e.g. <%page%> )
3. Click browse button for Max Pages Expression   (select node which indicates how many pages in response. e.g. Total_Pages)
   

   Workday SOAP API Pagination Settings in ZappySys XML / SOAP Driver

4. Once this is done go to Settings Tab. Edit your Body and make sure you supply Response_Filter. Pass Response_Filter parameter as below. See how we used <%page%> placeholder. You can use tool like fiddler to see multiple response at runtime. Run package and check fiddler. In the below example we are fetching maximum 300 rows per response. It will keep reading until all pages are done. For example if you have 1000 rows to read then it will take 4 response behind the scene.
   

   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:bsvc="urn:com.workday/bsvc">
      <soapenv:Body>
         <bsvc:Get_Job_Categories_Request bsvc:version="">
           <bsvc:Response_Filter>
               <bsvc:Page><%page%></bsvc:Page>
               <bsvc:Count>300</bsvc:Count>
            </bsvc:Response_Filter>
         </bsvc:Get_Job_Categories_Request>
      </soapenv:Body>
   </soapenv:Envelope>

   

   Workday API Pagination – Supply Response_Filter (Page Size and Current Page Number)

That’s it. Now if you run your query it will fetch all pages and you can now read more than 1000 rows. Make sure you configure Count at reasonable size (Rows per Page) in above screenshot we use 5 just to demo small sample. We suggest you to to set around 300.

How to pass parameters to SQL query in Power BI

let
    vKey=paraAPIKey,
    Source = Odbc.Query(
"dsn=ZS - OData Customers", 
"SELECT * FROM $ WITH (SRC='http://httpbin.org/post'," 
& "METHOD='POST'," 
& "HEADER='Content-Type:application/json'," 
& "BODY=@'{""CallerId"":1111, ""ApiKey"":""" & vKey & """}')")
in
    Source

How to debug Web Requests using Fiddler

There will be a time you like to see RAW HTTP Requests generated by driver for NetSuite SOAP API.  If you like to do that check this article.

Conclusion

Consuming data from Workday API or any XML / SOAP based API in your Reporting / ETL apps can be tricky. But we have seen how to solve this challenging task using ZappySys XML Driver in few clicks. Download ODBC PowerPack and explore many other API integration scenarios not discussed in this article.

The post Connect Workday in Power BI – Import SOAP Data appeared first on ZappySys Blog.

In our previous blog we discussed how to import REST / SOAP API data in Power BI. Now we will use that knowledge to read NetSuite data in Power BI (NetSuite CRM – SuiteTalk). We will use ZappySys XML Driver to call Search function in NetSuite SOAP API. If you are not familiar with SOAP API concept then we highly recommend you read previous blog post. In this post we will use Power BI as an example App to consume NetSuite data but you can use steps listed in this article to load data in any other ODBC compliant ETL / reporting app (e.g. Excel, Qlik, Informatica, SSRS) or programming language (e.g. C#, JAVA, Python) or SQL Server (Linked Server).

We will look at each step in depth, how to enable SOAP Api access in NetSuite, How to register Application, how to configure NetSuite ODBC DSN, and finally how to load NetSuite data into Power BI dashboard.

So let’s get started. The approach listed in this article can be used to call other SOAP based API such as Salesforce, Workday, SAP, Sage etc.

Prerequisite

Before you get started make sure following requirements are met.

1. NetSuite CRM Login (UserID and Password)
2. Enable Web Service Feature in NetSuite
3. Obtain NetSuite Account ID
4. Obtain Application ID if you have created an App. Else Register new Netsuite Application and get Application ID
5. Download ZappySys ODBC PowerPack (for SOAP / XML Driver)
6. Download Power BI Desktop (FREE) if you have not already done it.

What is NetSuite CRM?

NetSuite is a CRM / ERP product. It gives you scalable cloud CRM / ERP solution targeted at high-growing, mid-sized businesses and large enterprises. It automates front- and back-office processes including: financial management, revenue management, fixed assets, order management, billing, and inventory management.

Preparing for NetSuite WebService Call in Power BI

For NetSuite API call we have to make sure certain settings and obtain Application ID and AccountID. So let’s get started.

How to enable NetSuite Web Service Feature

Before you call any NetSuite SOAP API (i.e. SuiteTask Web Service), First thing to get started with NetSuite SOAP API (i.e. ) is to enable Web Service Feature. Once you enable Web Service, you can programmatically access NetSuite data in your App (e.g. Power BI, Tableau, Informatica). Click this link to see steps to enable Web Service Feature

Enable NetSuite Web Service Feature (SuiteTalk API)

Obtain NetSuite Account ID

NetSuite Account ID is needed by Login API call. So perform following steps if you dont know account id.

1. Login to your NetSuite Portal
2. Click the Setup Tab
3. In the drop-down click Integration >  Web services preferences
4. On that page you will find your Account ID (example account ID = TSTDRV1234567 )

Obtain NetSuite Account ID (Needed for Web Service Login)

Obtain NetSuite Application ID

Once you have account ID now we need to get Application ID. For that follow these steps.

1. Login to your NetSuite Portal
2. Click the Setup Tab
3. In the drop-down click Integration >  Manage Integrations
4. On that page you will find your existing Applications along with Account ID / Status (example Application ID = AE2EACFD-1234-1222-1111-12345544555 )
5. If you don’t have any application created yet then just click on New.

Create new App / Get NetSuite Application ID

Understanding how NetSuite SOAP API Call works

In this section we will briefly understand how NetSuite SOAP API call works. When you use ZappySys Driver / Component it will take care many details for you but still you have to configure correct XML Body. There are two steps involved before you can call any SOAP API in NetSuite.

1. Call Login method and extract JSESSIONID from response Cookie (This token expires after some time)
2. Call any other API (Pass obtained JSESSIONID token in Cookie header)

NetSuite Login Call

Request

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml;charset=UTF-8
SOAPAction: "login"
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
  </soap:Header>
  <soap:Body>
    <login xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <passport>
        <email xmlns="urn:core_2017_1.platform.webservices.netsuite.com">xxxxxxxxxxxxx</email>
        <password xmlns="urn:core_2017_1.platform.webservices.netsuite.com">xxxxxxxxxxxx</password>
        <account xmlns="urn:core_2017_1.platform.webservices.netsuite.com">TSTxxxxxxxxxxxxxxx</account>
        <role internalId="3" xmlns="urn:core_2017_1.platform.webservices.netsuite.com" />
      </passport>
    </login>
  </soap:Body>
</soap:Envelope>

Response Notice JSESSIONID=6xI6xxxxxxxxx in the login call response below. This Session ID can be used to call next call. ZappySys HTTP Connection Manager configured with Dynamic Token option can extract Cookie and reuse it until its expired (1 hour by default).

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 16:50:41 GMT
Content-Length: 1676
Content-Type: text/xml; charset=utf-8
X-N-OperationId: 777274f6-49c6-4883-bcd1-1d70df90d78e
Strict-Transport-Security: max-age=31536000
NS_RTIMER_COMPOSITE: 13633xxxxxxxxx:70617xxxxxxxxxxxxx36F6D:80
Set-Cookie: JSESSIONID=6xI61rdFt26pxxxxxxxxxxxxxxxqvJIMlu!-17981123445; path=/; secure; HttpOnly
Set-Cookie: NS_VER=2018.1.0; domain=webservices.netsuite.com; path=/
Set-Cookie: NS_ROUTING_VERSION=LAGGING; path=/
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent
Keep-Alive: timeout=10, max=990
Connection: Keep-Alive

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_xxxxxxxxxxxxxxxxx_9fc719</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <loginResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <sessionResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
        <ns1:status isSuccess="true" xmlns:ns1="urn:core_2017_1.platform.webservices.netsuite.com"/>
        <ns2:wsRoleList xmlns:ns2="urn:core_2017_1.platform.webservices.netsuite.com">
          <ns2:wsRole>
            <ns2:role internalId="3">
              <ns2:name>Administrator</ns2:name>
            </ns2:role>
            <ns2:isDefault>false</ns2:isDefault>
            <ns2:isInactive>false</ns2:isInactive>
          </ns2:wsRole>
          <ns2:wsRole>
            <ns2:role internalId="15">
              <ns2:name>Employee Center</ns2:name>
            </ns2:role>
            <ns2:isDefault>false</ns2:isDefault>
            <ns2:isInactive>false</ns2:isInactive>
          </ns2:wsRole>
      </sessionResponse>
    </loginResponse>
  </soapenv:Body>
</soapenv:Envelope>

 

NetSuite API Call ( getAll method )

Now once you have JSESSIONID extracted you can pass it to call next API call as below. For example here is how to call getAll method to obtain currency list and exchange rates. If you are not sure how to craft correct Request XML then use tool like SoapUI. You can use NetSuite WSDL file from here and import it in SoapUI.   Request See how Cookie is passed for authentication purpose. Again you can pass this manually or use HTTP connection with Dynamic Token to pass it automatically along with each API call.

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml;charset=UTF-8
SOAPAction: "getAll"
Cookie: JSESSIONID=6xI61rxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxMlu!-1798141234
Accept-Encoding: gzip, deflate

<soapenv:Envelope 
   xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
   xmlns:urn="urn:messages_2018_1.platform.webservices.netsuite.com" 
   xmlns:urn1="urn:core_2018_1.platform.webservices.netsuite.com">
   <soapenv:Body>
      <urn:getAll>
         <record recordType="currency"/> 
      </urn:getAll>
   </soapenv:Body>
</soapenv:Envelope>

Response

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 16:50:42 GMT
Content-Length: 9648
Content-Type: text/xml; charset=utf-8
X-N-OperationId: 1b2989e4-b930-4a72-b471-00a8e131dfd4
NS_RTIMER_COMPOSITE: 21403xxxxxx:706172746E65xxxxxxxxxxxx6F6D:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_TSTDRV1439151_100320189871507121000128715_6ae8e8</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <getAllResponse xmlns="urn:messages_2018_1.platform.webservices.netsuite.com">
      <platformCore:getAllResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>11</platformCore:totalRecords>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>USD</listAcct:name>
            <listAcct:symbol>USD</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>$</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_unitedStatesEnglish</listAcct:locale>
            <listAcct:formatSample>$1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>1.0</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>GBP</listAcct:name>
            <listAcct:symbol>GBP</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>£</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_unitedKingdomEnglish</listAcct:locale>
            <listAcct:formatSample>£1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>1.33165497</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
          <platformCore:record internalId="3" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>CAD</listAcct:name>
            <listAcct:symbol>CAD</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>$</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_canadaEnglish</listAcct:locale>
            <listAcct:formatSample>$1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>0.76457273</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
        </platformCore:recordList>
      </platformCore:getAllResult>
    </getAllResponse>
  </soapenv:Body>
</soapenv:Envelope>

NetSuite Search API (Pagination Example)

Now let's look at more complex API calls where Pagination is needed. For example when you call search API to pull data from specific table (e.g. Account) you need to loop through multiple responses until all records are consumed. Request (Page-1)

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml; charset=utf-8
SOAPAction: "search"
Cookie: JSESSIONID=BKQ7xxxxxxxxxxxx4Ug!-995xxxxxx
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
    <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <pageSize>50</pageSize>
    </searchPreferences>
  </soap:Header>
  <soap:Body>
    <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:AccountSearch">
        <q1:basic>
          <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
            <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">x</searchValue>
          </description>
          <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
            <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">a</searchValue>
          </name>
        </q1:basic>
      </searchRecord>
    </search>
  </soap:Body>
</soap:Envelope>

  Response (Page-1) In below response notice 2 things. SearchId is like a cursor id which you have to pass for each page you request after first page. And you can stop looping after 4th page based on totalPages count. platformCore:searchId = WEBSERVICES_ABCD_12345-xxxxxxxxx platformCore:totalPages = 4

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 18:40:33 GMT
Content-Length: 49266
Content-Type: text/xml; charset=utf-8
X-N-OperationId: c1e79336-63b7-4eb7-989e-5c3d24c56891
NS_RTIMER_COMPOSITE: 118xxxxxxxx:70617xxxxxxxxxxxxxxxxxxxxxD:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="utf-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <searchResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformCore:searchResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>193</platformCore:totalRecords>
        <platformCore:pageSize>50</platformCore:pageSize>
        <platformCore:totalPages>4</platformCore:totalPages>
        <platformCore:pageIndex>1</platformCore:pageIndex>
        <platformCore:searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformCore:searchId>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>1111</listAcct:acctNumber>
            <listAcct:acctName>Checking</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>xxxxxxxxxxxxxxxxxxx</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Checking</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>22222</listAcct:acctNumber>
            <listAcct:acctName>Savings</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>yyyyyyyyy</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Savings</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          .........
          .........
          .........
          .........		  
        </platformCore:recordList>
      </platformCore:searchResult>
    </searchResponse>
  </soapenv:Body>
</soapenv:Envelope>

  Request (Page-2 and onwards) Here is the second page request. Notice few things here. In heraders we are now sending SOAPAction: "searchMoreWithId" rather than "search".  Also our SOAP Body is also different compared to first page request. Now we only sending pageIndex we want to fetch and searchId. We have also change <search> tag with <searchMoreWithId> under Body.

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml; charset=utf-8
SOAPAction: "searchMoreWithId"
Cookie: JSESSIONID=BKQ7xxxxxxxxxxxx4Ug!-995xxxxxx
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
    <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <pageSize>50</pageSize>
    </searchPreferences>
  </soap:Header>
  <soap:Body>
    <searchMoreWithId xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</searchId>
      <pageIndex>2</pageIndex>
    </searchMoreWithId>
  </soap:Body>
</soap:Envelope>

Response (Page-2 and onwards)

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 18:40:33 GMT
Content-Length: 49266
Content-Type: text/xml; charset=utf-8
X-N-OperationId: c1e79336-63b7-4eb7-989e-5c3d24c56891
NS_RTIMER_COMPOSITE: 118xxxxxxxx:70617xxxxxxxxxxxxxxxxxxxxxD:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="utf-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <searchMoreWithIdResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformCore:searchResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>193</platformCore:totalRecords>
        <platformCore:pageSize>50</platformCore:pageSize>
        <platformCore:totalPages>4</platformCore:totalPages>
        <platformCore:pageIndex>2</platformCore:pageIndex>
        <platformCore:searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformCore:searchId>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>8000</listAcct:acctNumber>
            <listAcct:acctName>Checking</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>xxxxxxxxxxxxxxxxxxx</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Checking</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>8001</listAcct:acctNumber>
            <listAcct:acctName>Savings</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>yyyyyyyyy</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Savings</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          .........
          .........
          .........
          .........		  
        </platformCore:recordList>
      </platformCore:searchResult>
    </searchMoreWithIdResponse>
  </soapenv:Body>
</soapenv:Envelope>

   

Getting Started

Now let’s check step by step instructions on how to load NetSuite data in Power BI. In this exercise we will focus on calling search API but you can call virtually any API using the same technique.

For detailed help on NetSuite SOAP API and other useful information check this documentation. To browse schema you can use this link. From schema browser we will use Search Type name later on in this article. For example later on in this article to search accounts we will use AccountSearch as below example. This name can be found from here(click search tab) You can change it to something else. For now dont worry too much until we get to that point.

Creating NetSuite ODBC Connection (Using ZappySys XML / SOAP Driver)

Once you have all necessary information handy we can create ODBC DSN for NetSuite which can be used by any ODBC compliant app such as Power BI. Perform the following steps to set NetSuite Connection which we will use to query data (e.g. Accounts, Budget, Campaign etc).

1. Open start menu and search for “ODBC 64”. Launch ODBC Data sources for 64 bit
2. Click Add and select ZappySys XML Driver
   

   ZappySys ODBC Driver for XML / SOAP API

3. Once driver UI is visible change default DSN name (e.g. change to ZS – NetSuite)
4. Click on the Load Connection String button and enter the following string after replacing below placeholders to configure the NetSuite connection. If you are unsure how to get Account ID and Application ID then check previous sections. For now dont worry about various options in the connection string. We will cover them later.
   Replace $$$YOUR-APPLICATION-ID$$$  (2 times) with your own Application ID
   Replace $$$YOUR-ACCOUNT-ID$$$ with your own Account ID
   Replace $$$YOUR-USER-ID$$$  with your own Login ID (e.g myuser@mycompany.com)
   Replace $$$YOUR-PASSWORD$$$  with your own Password
   Replace <pageSize>50</pageSize>  with your own value. We suggest not to go more than 200 rows per page for better performance.
   Replace c:\temp\netsuite.cache.db with your own path for Cache File. Caching must be ON in order to Import data in Power BI correct way. We also use CacheTtl=120 seconds (Cache Expires in 120 seconds) but adjust it longer if your data import takes long time.
   

   DRIVER={ZappySys XML Driver};ElementsToTreatAsArray='platformCore:record';DataPath='https://webservices.netsuite.com/services/NetSuitePort_2017_1';DataConnectionType=HTTP;AuthScheme='JSESSIONID=';TokenUrl='https://webservices.netsuite.com/services/NetSuitePort_2017_1';TokenRequestData='<?xml version="1.0" encoding="utf-8"?>
   <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
     <soap:Header>
       <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <applicationId>$$$YOUR-APPLICATION-ID$$$</applicationId>
       </applicationInfo>
     </soap:Header>
     <soap:Body>
       <login xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <passport>
           <email xmlns="urn:core_2017_1.platform.webservices.netsuite.com">[$userid$]</email>
           <password xmlns="urn:core_2017_1.platform.webservices.netsuite.com">[$password$]</password>
           <account xmlns="urn:core_2017_1.platform.webservices.netsuite.com">$$$YOUR-ACCOUNT-ID$$$</account>
           <role internalId="3" xmlns="urn:core_2017_1.platform.webservices.netsuite.com" />
         </passport>
       </login>
     </soap:Body>
   </soap:Envelope>';TokenRequestMethod='POST';TokenRequestContentType=TextXmlUtf8;TokenResponseValueExtractMode=Cookie;TokenRequestHeaders='SOAPAction:"login"';TokenResponseCookie='JSESSIONID';TokenAuthHeader='Cookie';UserName='$$$YOUR-USER-ID$$$';CredentialType=TokenDynamic;Password='$$$YOUR-PASSWORD$$$';Filter='$.soapenv:Envelope.soapenv:Body.search[$tag$]Response.platformCore:searchResult.platformCore:recordList.platformCore:record[*]';RequestData='<?xml version="1.0" encoding="utf-8"?>
   <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
     <soap:Header>
       <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <applicationId>$$$YOUR-APPLICATION-ID$$$</applicationId>
       </applicationInfo>
       <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <pageSize>50</pageSize>
       </searchPreferences>
     </soap:Header>
     <soap:Body>
   [$tag$]
     </soap:Body>
   </soap:Envelope>';RequestMethod='POST';NextUrlAttributeOrExpr='$.soapenv:Envelope.soapenv:Body.search[$tag$]Response.platformCore:searchResult.platformCore:searchId';FirstPageBodyPart='    <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:AccountSearch">
           <q1:basic>
   		<!--
             <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">x</searchValue>
             </description>
             <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">a</searchValue>
             </name>
   		 --> 
           </q1:basic>
         </searchRecord>
       </search>';NextPageBodyPart='    <searchMoreWithId xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchId>[$pagetoken$]</searchId>
         <pageIndex>[$pagenumber$]</pageIndex>
       </searchMoreWithId>';PagingMaxPagesExpr='$.soapenv:Envelope.soapenv:Body.searchResponse.platformCore:searchResult.platformCore:totalPages';HasDifferentNextPageInfo=True;EnablePageTokenForBody=True;PagePlaceholders='header=|MoreWithId;filter=|MoreWithId';RequestHeaders='Content-Type: text/xml; charset=utf-8 || SOAPAction: "search[$tag$]"';CachingMode=All;CacheStorage=File;CacheEntryTtl=120;CacheFileLocation='c:\temp\netsuite.cache.db'

    
   

   ZappySys ODBC Driver Load Connection String – NetSuite Connection Example

5. After configuration your screen may look like below. You can also click on Advanced View (Radio button) to see some Advanced properties (e.g. Caching Properties such as CachingMode, CacheFileLocation and Advanced Pagination Properties such as PagingMaxPagesExpr, HasDifferentNextPageInfo, NextPageBodyPart, FirstPageBodyPart, EnablePageTokenForBody, PagePlaceholders )
   
   Notice [$tag$] in Body, Header and Filter. This gets replaced at runtime based on some settings we did for pagination (see next 2 screenshots). NetSuite API has different Header, Body and Filter need for First page vs Next Pages so it needs special configuration. [$tag$] in Body gets replacement based on FirstPageBodyPart and NextPageBodyPart properties found on Advanced View under Pagination. [$tag$] used in Filter, Header or any other place gets its replacements from PagePlaceHolders Property found on Advanced View (see filter=firstPartString|nextPartString;header=firstPartString|nextPartString). Use pipe (vertical bar to separate first part placeholder and next part placeholder) In our case we use blank for first part.
   

   NetSuite ODBC Connection Settings (ZappySys XML Driver)

6. Here is what it looks like on Advanced Tab.
   

   ODBC XML Driver – Ádvanced View (Pagination, Caching, Logging and other properties not found in Simple Mode)

7. Now go to preview tab and click Preview Data button. You will see upto 100 sample records. Using below query it will use all options from UI (No override in query).SELECT * from $
   

   Preview NetSuite data (Query Accounts) using ODBC Driver

8. Now let’s change our default query a bit  to get list of Expense Category (i.e. ExpenseCategorySearch). As you see we change FirstRequestBody a little bit (change q1:ExpenseCategorySearch). We also removed some unwanted columns from preview by setting IncludeParentColumns=’false’
   

   SELECT * FROM $
   WITH(
   	FirstPageBodyPart='    <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:ExpenseCategorySearch">
           <q1:basic>
             <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">Some-String-Goes-Here</searchValue>
             </description>
             <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">Some-String-Goes-Here</searchValue>
             </name>
           </q1:basic>
         </searchRecord>
       </search>',
       IncludeParentColumns='False'
   )

   We have commented SearchQuery portion to make it simple but you can learn more about how to write search query from here

   Here is one more example of query which combines Basic Search and Advanced Search

   Pay close attention to namespace prefix in SOAP code you copy from Help center. It may be using different prefix than our original connection string (e.g. we use q1:xxxxx at some places but online sample may use ns1, ns2, ns3 ….)

   SELECT * FROM $
   WITH(
   	FirstPageBodyPart='
       <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchRecord xsi:type="ns4:ContactSearch" xmlns:ns4="urn:relationships_2017_1.lists.webservices.netsuite.com">
           <ns4:basic xsi:type="ns5:ContactSearchBasic" xmlns:ns5="urn:common_2017_1.platform.webservices.netsuite.com">
             <ns5:email operator="contains" xsi:type="ns6:SearchStringField" xmlns:ns6="urn:core_2017_1.platform.webservices.netsuite.com">
               <ns6:searchValue xsi:type="xsd:string">.com</ns6:searchValue>
             </ns5:email>
           </ns4:basic>
           <ns4:customerJoin xsi:type="ns7:CustomerSearchBasic" xmlns:ns7="urn:common_2017_1.platform.webservices.netsuite.com">
             <ns7:entityId operator="contains" xsi:type="ns8:SearchStringField" xmlns:ns8="urn:core_2017_1.platform.webservices.netsuite.com">
               <ns8:searchValue xsi:type="xsd:string">C</ns8:searchValue>
             </ns7:entityId>
           </ns4:customerJoin>
         </searchRecord>
       </search>',
       IncludeParentColumns='False'
   )

    
   

   NetSuite Search API Example (Mixing BasicSearch / AdvancedSearch with JOIN)

9. Copy your Final Query you like to use to import NetSuite data in Power BI
10. Click OK to save DSN

Import NetSuite data in Power BI

Now lets perform the final step to import our NetSuite Query to Power BI. Assume that we want to import Contacts in Power BI to report them by demographics or some other criteria.

1. Open Power BI Desktop
2. Click on Get Data > More Data… > Other > ODBC
   

   Import SOAP API in Power BI (Using ZappySys XML ODBC Driver)

3. Select your ODBC DSN Source from the dropdown (Assuming you have created a ODBC Data Source using ZappySys XML Driver – See the previous Section)
4. Expand Advanced Options to enter custom Query.
5. Enter your SOAP API call SQL query here. For example purpose, we will use below query to get all account using NetSuite Search Web service.
   

   SELECT * FROM $
   WITH(
   	FirstPageBodyPart='    <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:AccountSearch">
           <q1:basic>
           <!--
             <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">x</searchValue>
             </description>
             <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">a</searchValue>
             </name>
             -->
           </q1:basic>
         </searchRecord>
       </search>',
       IncludeParentColumns='False'
   )

    
   

   Import NetSuite SOAP Web Service Data in Power BI (ZappySys XML Driver)

6. Once you see data preview click OK to import.
7. After data import is done you can edit your dataset (e.g. remove unwanted columns)
8. You can also edit Source query after data is imported (See below)
   

   Edit Power BI Data Source SQL after REST / SOAP Web Service Import

9. That’s it. You can now create your dashboards using NetSuite data. See below example.
   

   Import NetSuite data in Power BI dashboard

How to debug Web Requests using Fiddler

There will be a time you like to see RAW HTTP Requests generated by driver for NetSuite SOAP API.  If you like to do that check this article.

How to call other NetSuite SOAP API

So far we have seen only search API to query data. But there are many other SOAP APIs which can be consumed using approach listed in this article. Check this article to learn more how to call any SOAP API using ZappySys XML Driver. It shows techniques to use tool called SoapUI. You can use NetSuite WSDL file from here copy link listed on that page and use it to import in SoapUI.

Conclusion

Consuming data from NetSuite API or any XML / SOAP based API in your Reporting / ETL apps can be tricky. But we have seen how to solve this challenging task using ZappySys XML Driver in few clicks. Download ODBC PowerPack and explore many other API integration scenarios not discussed in this article.

The post How to read NetSuite data in Power BI appeared first on ZappySys Blog.

In this article, we will learn how to import REST API in Power BI.

Power BI is a very popular Business Analytic tool used to get business information. It is very popular because it is easy to install, simple to learn and very intuitive. Also, REST API is very popular these days and we wanted to show you a way to integrate them in Power BI with the help of the ZappySys ODBC PowerPack. This software includes powerful drivers to query REST API URL, Local XML / JSON files and XML SOAP Web Service with simple SQL queries.

The following video will show how to import REST API in Power BI (JSON or XML Data):
[youtube https://www.youtube.com/watch?v=-3OPfhMY1aI&w=720&h=405]

Requirements

1. First, you will need to have Power BI Desktop (FREE) installed
2. In the second place, you will need to have internet access
3. On third place, our ZappySys ODBC Power Pack

Step-by-Step: Import REST API into Power BI

We’ll walk through the steps to import a REST API into Power BI. The steps outlined below focus on working with JSON APIs, but the same principles can be applied when importing SOAP XML Web Services or local JSON/XML files.

For demonstration purposes, we will utilize a REST API to retrieve data in JSON format. The example URL we’ll use is:

https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json

This URL leverages OData and presents information in JSON format. Our objective is to use the ZappySys ODBC Power Pack to establish a connection to this URL and retrieve the information seamlessly into Power BI.

Create ODBC DSN – JSON Driver

DRIVER={DRIVER-NAME-GOES-HERE}[;Prop1=xxxxxx][;Prop2=xxxxxx] .... [;PropN=xxxxxx]

DRIVER={ZappySys JSON Driver}
DRIVER={ZappySys JSON Driver};DataPath='c:\mydata.json'
DRIVER={ZappySys XML Driver};DataPath='http://myserver/api/xml/getOrders'
DRIVER={ZappySys XML Driver};DataPath='c:\mydata.xml'

DRIVER={DRIVER-NAME-GOES-HERE}[;Prop1=xxxxxx][;Prop2=xxxxxx] .... [;PropN=xxxxxx]

DSN=MyJsonAPI_DSN
DSN=MyJsonAPI_DSN;DataPath='http://myserver/api/json/getOrders'

Connect to REST API data source in Power BI (Connect JSON / XML data)

Publish Power BI dashboard

1. In order to publish the report, press Publish
2. Press Save
   

   Save changes in Power BI:

    

3. Select a workspace for the report
   

   Select Power BI workplace.

4. And that is all. You have now your report ready.

Import data using ODBC ConnectionString (DSN-less)

Let’s explore the process of importing data using the ODBC Driver without the need for configuring an ODBC DSN.

When importing data, you have the option to either use an ODBC DSN or a Direct ConnectionString. This section will guide you through the steps of utilizing the ODBC ConnectionString.

After opting for the ODBC option to retrieve data, you will encounter the “Select ODBC Data Source” screen. To proceed without a DSN, follow these steps:

1. Change the DSN dropdown to “(none).”
2. Enter the full ODBC ConnectionString, adhering to the syntax below. You can construct your own connection string using the ODBC Driver UI and utilize the “Copy ConnectionString” option.
   Syntax: 

   Driver={Your Driver Name}[;Property1=value][;Property2=value]......

3. Here is a screenshot with ConnectionString Setting
   

   Import Data from ODBC Driver – DSN less mode (Use Connection String)

4. Click “Next” to select the authentication method. Choose “Windows” and input the password value if necessary. If your connection string already includes the Password attribute, there’s no need to worry.
   

   Import Data from ODBC Driver – using Connection String – With Password Attribute

POST data to REST API URL

Up until now, we’ve explored the process of fetching data from URLs and importing it into Power BI. However, in certain scenarios, APIs may necessitate the use of the PUT or POST method. These methods involve submitting parameters in the body and receiving a response.

For a more in-depth understanding of how to seamlessly POST data to a URL in Power BI, watch the informative video below.

Calling XML SOAP Web Service in Power BI

So far, we have looked at examples to consume data using JSON driver. Now let’s look at an example, to call XML SOAP Web Service in Power BI.

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

SOAP / REST API pagination in Power BI

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

SOAP / REST API Error Handling in Power BI

Sometimes errors occur... they just do and there is nothing you can do! Or can you? Actually, in ODBC PowerPack you can handle them in two ways.

METHOD 1 - Using Error Handling Options

When to use?

You may want to use them when your source is a resource located on the Internet; e.g. a file on a website, a file on an FTP server or just a plain API HTTP response. By default, when a remote server returns an error, data retrieval is stopped, an error is raised and no data is given back to you. This might not be always desirable.

Scenario 1

Imagine a scenario, that there is a web server which each day at 12 AM releases a new JSON file with that day's date as filename, e.g. http://www.some-server.com/data/2018-06-20.json. And, of course, you want to download it and use it daily in your Power BI report. But you have a problem: Power BI report data sources are refreshed each hour and you may get HTTP 404 status code (no file was found) when a file is not released yet. Which consequentially means other data sources won't be updated as well and you will see old and cached data on the report. That's where you could use Continue on any error or Continue when Url is invalid or missing (404 Errors) to avoid an error being raised and let other data sources to be updated.

Scenario 2

Another scenario is when you expect a web server to raise some kind of HTTP error when accessing a URL. You don't want ODBC Data Source to raise an error but instead, you want to get response data. That's where you can use Continue on any error or alike together with  Get response data on error to continue on an error and get the data:

METHOD 2 - Using Connection [Retry Settings]

Another scenario you may run into is a buggy web server. You ask it to give you some file or data and it, like a snotty kid, just doesn't give it to you! You have to ask twice or thrice before it does its job. If that's the case, you have to retry HTTP requests using Connection:

Security / Connection Types (Options for HTTP, OAuth, SOAP) in Power BI

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Import data into Power BI from SQL Query

If you prefer to import data from an SQL query instead of selecting a table name, you can leverage advanced options during the import steps. Here’s a step-by-step guide:

1. After selecting the DSN, proceed to the import steps.
2. Click on the “Advanced Options” to access the SQL Query editor.

By utilizing these advanced options, you gain the flexibility to tailor your import process by specifying custom SQL queries to retrieve the exact data you need.

Import data into Power BI using SQL Query (ODBC Data source)

Edit Query / Using Parameters in Power BI (Dynamic Query)

let
    vKey=paraAPIKey,
    Source = Odbc.Query(
"dsn=ZS - OData Customers", 
"SELECT * FROM $ WITH (SRC='http://httpbin.org/post'," 
& "METHOD='POST'," 
& "HEADER='Content-Type:application/json'," 
& "BODY=@'{""CallerId"":1111, ""ApiKey"":""" & vKey & """}')")
in
    Source

Edit Query Settings after Import

There will be a time you need to change initial Query after dataset import in Power BI. Not to worry, just follow these steps to edit your SQL.

Edit Power BI Data Source SQL after REST / SOAP Web Service Import

Performance Tips for REST API / XML SOAP Calls

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Conclusion

In this article, we guide you through the process of configuring the ZappySys Power Pack, focusing on the ZappySys driver for JSON. This driver is instrumental in extracting data from REST APIs or JSON files. Follow these steps to seamlessly add the extracted data to Power BI and generate insightful reports.

Additionally, we’ll walk you through the configuration of the Power BI gateway, ensuring that the data is refreshed on a specified schedule for up-to-date and accurate reporting. To try this configuration yourself, download ODBC PowerPack and embark on a journey of efficient data extraction and reporting.

References

• What is Power BI?
• Keep your dashboards and reports up-to-date with your on-premises data sources
• Power BI – Read REST API / JSON File / XML File / SOAP (Pagination, OAuth, OData)

The post How to import REST API in Power BI (Load JSON / SOAP XML) appeared first on ZappySys Blog.