Power BI native ODBC connectors do not support DirectQuery. This significant platform limitation forces users into Import mode, preventing real-time data access and causing refresh delays for critical business dashboards. If you have been searching for a “Power BI ODBC DirectQuery workaround” or a way to “enable DirectQuery for custom ODBC drivers,” this guide provides the exact steps to bridge that gap.

The Industry Challenge: Standard Power BI ODBC settings lack the DirectQuery option.

The Problem: Why “Import Mode” Fails Modern Data Teams

While the native Microsoft connector is useful for simple tasks, its lack of DirectQuery support creates several enterprise-level issues:

• Stale Data: There is no true real-time reporting, as data is only as fresh as your last refresh.
• Memory Constraints: Large datasets quickly bloat the Power BI model and increase memory usage.
• Refresh Overhead: Frequent changes in underlying data require complex and constant refresh schedules.
• API Lag: Data from cloud apps like Jira, SharePoint, or OneDrive becomes outdated between scheduled updates.

The Solution: ZappySys DirectQuery ODBC Connector

To fix this widespread industry problem, you can use the ZappySys DirectQuery ODBC Connector (a specialized custom Power BI connector). This tool allows you to treat any ODBC source as a live connection, unlocking real-time analytics for databases, APIs, and cloud apps.

Advanced Capabilities:

• Enable DirectQuery mode for virtually any 64-bit ODBC source.
• Query live data directly without importing massive datasets into Power BI.
• Seamlessly integrate with the ZappySys ODBC PowerPack for API and JSON connectivity.

Step 1: Install the ZappySys Connector

1. First, download and install the ZappySys ODBC PowerPack.
   This is essential for those using ZappySys drivers (e.g. REST API, JSON, JDBC Bridge (e.g. Trino, Infor, Athena) and more).
2. Download the Power BI Connector file: ZappySys-DirectQuery-ODBC-Connector.zip
3. Copy the extracted .mez file to <your-profile>\Documents\Power BI Desktop\Custom Connectors: (See next section to find exact path)
   

   Examples:
   
   C:\Users\<YourUsername>\Documents\Power BI Desktop\Custom Connectors
   C:\Users\<YourUsername>\OneDrive - <your-company>\Documents\Power BI Desktop\Custom Connectors

   
   Note: If the Custom Connectors folder does not exist, you must create it manually.For finding exact connector folder path see Microsoft Connector Extensibility documentation.

How to find the Custom Connector folder Path for Power BI

1. Type PowerShell on command line.
2. Run the lines below to print the base path and the full path. You can run one after another or paste both lines.
   

   [Environment]::GetFolderPath('MyDocuments')
   Join-Path ([Environment]::GetFolderPath('MyDocuments')) 'Microsoft Power BI Desktop\Custom Connectors'

Step 2: Update Power BI Security for Custom Connectors

1. Open Power BI Desktop.
2. Navigate to File > Options and settings > Options > Security.
3. Under Data Extensions, enable the option “Allow any extension to load without validation”.
4. Save your changes and restart Power BI Desktop to apply the new security policy.

Power BI options security settings to enable custom connectors

Step 3: Set Up a 64-bit ODBC DSN

1. Open the ODBC Data Sources (64-bit) administrator.
2. Click the System DSN tab and select Add.
3. Choose your driver (for example, the ZappySys ODBC Driver for API/Jira/SharePoint).
4. Save the DSN with a clear name, such as MyZappyDSN.

Important Architecture Note: Power BI Desktop and the Gateway require a 64-bit DSN; 32-bit DSNs are not supported for this workflow.

Windows 64-bit ODBC DSN Configuration.

Step 4: Connecting for Live Data / DirectQuery Support

1. In Power BI Desktop, go to Get Data > More...
2. Search for “zappysys” in the list.
3. Select ZappySys ODBC Connector (For DirectQuery Support).
4. Provide your DSN name (MyZappyDSN) or a full connection string.
5. Select DirectQuery mode and click connect.
   

   ZappySys ODBC Connector – To enable DirectQuery option for ODBC Datasource (Custom Install required)

Note for PowerPack Users: You can use a full connection string (e.g., Driver={ZappySys ODBC Driver};Host=...;) by using the “Copy Settings” feature within the ZappySys Driver UI. Read more how to copy full connection string

Power BI Get Data showing ZappySys DirectQuery ODBC Connector selected

Choosing Your Selection Mode

When the navigator opens, you must choose how to fetch your data:

• Table Mode: Best for simple datasets where you want to select a table from a list.
• Query Mode (SQL): Recommended for performance. Writing a custom SQL query allows you to filter and join data at the source, which is much faster for DirectQuery.

Step 5: Verify Your DirectQuery Connection (Final Result)

Once connected, it is crucial to verify that the report is truly “Live.” A successful connection will change the behavior of Power BI Desktop.

Power BI Test Dashboard showing Storage Mode: DirectQuery

Step 6: Configure the Gateway for Power BI Service

Once your report is ready, you must configure the gateway so Power BI Service can maintain the live DirectQuery connection to your data source. Follow the steps below to deploy and enable your custom connector for the On-premises Data Gateway.

6.1 Create a Dedicated Connector Folder

Do not place the connector in a user-specific folder. The gateway service account needs reliable access, so use a shared folder on a local drive instead.

• Recommended paths: C:\CustomConnectors or C:\ZappySysConnectors

6.2 Copy the Connector File

Copy your custom connector file (.mez) into the folder you created.

C:\CustomConnectors\YourConnector.mez

6.3 Grant Folder Permissions to the Gateway Service

If the connector does not appear in Power BI Service, first make sure the gateway service account can access the connector folder.

1. Right-click the folder (for example, C:\CustomConnectors) and select Properties > Security.
2. Click Edit > Add.
3. Enter NT SERVICE\PBIEgwService and click Check Names.
   NOTE: If you are using a different service account, then use that instead. To verify Services account name try below steps.
   Press Windows Key + R > open services.msc > locate On-premises data gateway service > look at Log On As
   > If it says NT SERVICE\PBIEgwService then use exact same user name
   > If it says Local System then use SYSTEM user for folder permission
   > If it says DOMAIN\Svc-PBI-Gateway then use exact same user name
4. If Windows cannot find the account, make sure the Location is set to the local machine, not the domain.
5. Grant at least Read and List folder contents permissions.

6.4 Enable Custom Connectors in the Gateway Config File

You must explicitly enable custom connectors and point the gateway to your connector folder.

1. Open this file in Notepad as Administrator:

C:\Program Files\On-premises data gateway\Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.dll.config

2. Find the <appSettings> section and add these entries:

<add key="EnableCustomConnectors" value="true" />
<add key="CustomConnectorsPath" value="C:\CustomConnectors" />

3. Save the file.
4. Restart the gateway service from services.msc or from the gateway application.

6.5 Verify the Connector in Power BI Service

After restarting the gateway, sign in to Power BI Service and try adding or mapping the data source. Your custom connector should now be available through the gateway.

FAQ: Frequently Asked Questions

Does the native Microsoft ODBC connector support DirectQuery?

No. The native Microsoft ODBC connector is limited to Import mode only.

How can I enable DirectQuery for ODBC sources in Power BI?

By using the ZappySys DirectQuery ODBC Connector and following the custom extension setup outlined in this guide.

Where is the ZappySys DirectQuery connector download?

You can find the latest version here: ZappySys DirectQuery Connector ZIP.

Troubleshooting Checklist

• Connector Missing: Ensure “Allow any extension” is checked in Power BI security settings.
• Gateway Errors: Verify the DSN name is identical on both the local PC and the gateway server.
• 32-bit vs 64-bit: Ensure you are using a 64-bit System DSN; 32-bit will cause connection failures.
• Driver Installation: Ensure the ZappySys ODBC driver is installed on all machines where the report or gateway runs.

The post Enable Power BI DirectQuery for ODBC: Solving the Live Data Problem appeared first on ZappySys Blog.

Power BI is a very popular Business Analytic tool used to get business information. It is very popular because it is easy to install, simple to learn and very intuitive. Also, REST API is very popular these days and we wanted to show you a way to integrate them in Power BI with the help of the ZappySys ODBC PowerPack. This software includes powerful drivers to query REST API URL, Local XML / JSON files and XML SOAP Web Service with simple SQL queries.

In nutshell, this post will focus on how to call JIRA API using Zappysys ODBC PowerPack. So let’s get started

Prerequisite

1. First, you will need to have our ZappySys ODBC Power Pack.
2. In the second place, you will need to have internet access.
3. On third place, Power BI Desktop (FREE) installed.
4. JIRA Atlassian account.

What is JIRA

JIRA is Atlassian’s popular project- and issue-tracking platform used by development and technical support teams to get more work done and faster.

It combines:

issue tracking – a software application that allows to record and follow the progress of every problem or “issue” that user identifies until the problem is resolved.
agile project management – an iterative approach to planning and guiding project processes. customizable workflow, and a pluggable integration – integrates with Freshdesk, GitHub, Zendesk, Zapbook, Asana and so on.

Getting Started

In order to start, we will show several examples. ZappySys ODBC PowerPack includes the ZappySys JSON Driver that will help you to call JIRA API, retrieve various data from JIRA and load that data in Power BI. To learn more about JIRA API check this help file.

https://jira.atlassian.com/rest/api/2/search?jql=project=JRASERVER

https://YOUR-DOMAIN.atlassian.net/rest/api/2/search?startAt=0&maxResults=100&jql=project=MYPROJECT

Load JIRA data into Power BI (Connect JIRA REST API)

Conclusion

In this blog, we have learned how to retrieve the data from JIRA using ODBC JSON Driver and then Load those data in Power BI with ODBC DSN in a very simple way. You can achieve many more functionalities with the use of ODBC PowerPack.

References

Finally, you can use the following links for more information about the use of JIRA API with our tools:

• JIRA Documentation for API
• Landing Page for ODBC JSON / REST API Driver, you can also find Tutorial Video here.
• Help File: Documentation of JSON Driver.
• Blog/articles link: https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/json-rest-api-driver/

The post How to Import JIRA data in Power BI appeared first on ZappySys Blog.

In this article, we will get Office 365  data in Power BI using Microsoft Graph API and ODBC drivers. Specifically, we will get Excel file data in OneDrive, events from a Calendar and finally, a list in a note in OneNote and then load that data into a Power BI report. We will achieve that by creating an ODBC data source using ZappySys ODBC PowerPack and then querying corresponding Microsoft Graph API methods. ODBC PowerPack allows to query REST/SOAP APIs in an easy way and lets to view responses in a tabular format, which will be perfect to accomplish our goal.

Prerequisites

1. A Power BI account.
2. Wield basic Power BI skills.
3. Power BI Desktop installed.
4. ZappySys ODBC PowerPack installed.

Step By Step – Get Office 365 data in Power BI using Microsoft Graph API and ODBC

Set up an ODBC data source based on ZappySys JSON Driver

Add a data source

Configure the data source

Set up an ODBC data source based on ZappySys XML Driver

1. Before adding another data source based on ZappySys XML Driver, copy the connection string of the data source we just created:
2. Close the window and then add a new data source, but this time based on ZappySys XML Driver.
3. Once a window is opened, give it a name, e.g. System - XML.
4. Then press Load connection string button to load the connection string.
5. Paste the copied connection string and replace JSON text to XML, so that you have { ZappySys XML Driver } in DRIVER property.
6. Close the window.

Create a Power BI report

Reading files from OneDrive (getting Excel data)

Let’s say you have Stock.xlsx stored in Sample Documents folder, OneDrive with this kind of data and you want to show it in a Power BI report:

To have that data in a Power BI report, perform these steps:

1. Open Power BI Desktop and create a new Power BI report.
2. Create a data source based on System – JSON DSN (since we get a JSON response):
   
3. Then in SQL statement area enter this query:
   SELECT * FROM $
WITH(Src='https://graph.microsoft.com/v1.0/me/drive/root:/YOUR FOLDER/Stock.xlsx:/workbook/worksheets/SHEET NAME/usedRange'
,Filter='$.values[1:]'
,ArrayTransformType='TransformSimpleTwoDimensionalArray'
,ArrayTransColumnNameFilter='$.values[0][*]'
)
Using WITH clause lets us override the properties. You can generate the query above using the Query Builder:
   In this step we are overriding Url, Filter and array transformation fields you configured in JSON DSN. The Src value overrides the URL. The Filter property enables us to select the data we want to see in the report. We use [1:] in $.values[1:] to skip the first row because the first row contains the headers and we want only the data. You can configure this option in the Filter Options tab (read more about JSON path).We also have to make the transformation on the JSON and turn JSON arrays into rows. To accomplish that, we override ArrayTransformType. With ArrayTransColumnNameFilter property and [0][*] text, we specify that the first array of parent array will be the headers of the columns. You can read more about transformations in Parse multi-dimensional JSON array article.
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other OneDrive endpoints.
4. Click OK.
5. Drag a table into the report, select the columns and you should see a similar view:
   

Reading calendar entries

Now, let’s say you use a calendar in Office 365 and you want your appointments to be visible in a Power BI report:

1. To start, create a new Power BI report.
2. Then add a data source based on System – JSON DSN (again, we get the response in JSON):
   
3. In SQL statement section enter this query:
   SELECT * FROM $
WITH(Src='https://graph.microsoft.com/v1.0/me/events?$select=subject,start,end,location'
,Filter='$.value[*]'
,Header='Prefer: outlook.timezone="FLE Standard Time"'
)Here we also are overriding HTTP Header so that we get the event times not in UTC/GMT but for the specified timezone. It must be specified in Prefer: outlook.timezone=”{time-zone-string}” format (check an article on how to get a list of supported time zones).
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other Calendar endpoints.
4. Click OK.
5. Drag a table control into the report, select several columns and you should see a similar view:
   

Reading notes from OneNote

Suppose, you have these notes in OneNote and you want “Groceries list” to be shown in a Power BI report:

1. First, we will need to find NOTE ID we are trying to get. To do that, open ODBC Administrator, find System – JSON data source and hit Configure.
2. Then go to the Preview tab, and input this SQL query and hit Preview Data:
   SELECT * FROM $
WITH (Src='https://graph.microsoft.com/v1.0/me/onenote/pages'
,Filter='$.value[*]'
)
3. Once you get a result set, find your note name and copy its ID:
   
4. Then create a new Power BI report.
5. Create a new data source based on System – XML DSN (we are using XML-based DSN because the response will be in HTML):
   
6. In SQL statement section enter this query (use the NOTE ID you retrieved in the previous step):
   SELECT * FROM $
WITH(ElementsToTreatAsArray='div,p'
,Src='https://graph.microsoft.com/v1.0/me/onenote/pages/NOTE ID/content'
,Filter='$.html.body.div[0].p[1:]'
)

   
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other OneNote endpoints.
7. Click OK.
8. Drag and drop Multi-row card visual, select #text column and you should see a similar view:
   
9. We finished! Now you can publish your Power BI reports to Power BI service.

Conclusion

Today we learned how to call Microsoft Graph API methods using ZappySys ODBC PowerPack and load response data into Power BI without any ETL procedures. Particularly, we used OneDrive, Calendar and OneNote Graph API endpoints to achieve that. We created two ODBC data sources based on JSON and XML ODBC Driver to query those endpoints, which let us read data directly in a Power BI report. Finally, we examined how to refresh data source on Power BI using on-premises Power BI gateway.

References

https://powerbi.microsoft.com/en-us/

https://apps.dev.microsoft.com/

https://developer.microsoft.com/en-us/graph/graph-explorer/

https://docs.microsoft.com/en-us/graph/overview

https://docs.microsoft.com/en-us/graph/onedrive-concept-overview

https://docs.microsoft.com/en-us/graph/outlook-calendar-concept-overview

https://docs.microsoft.com/en-us/graph/integrate-with-onenote

The post Get Office 365 data in Power BI using Microsoft Graph API and ODBC appeared first on ZappySys Blog.

In our previous blog we saw how to call REST API in Power BI to import data from JSON based REST API services. Now let’s continue on that and learn how to import Amazon MWS data in Power BI using XML driver. To learn more about how to consume Amazon MWS data in ETL using SSIS check this article.

We will read data from Amazon MWS API using ZappySys XML Driver which is part of ODBC PowerPack.

Requirements

1. Active Amazon Seller Account needed (Make sure you have Professional Plan or higher (around $39/month) to call Amazon MWS API)
2. You are already registered as a Developer and you obtained AWS Access Key / Secret Key and Seller ID (i.e. Merchant ID) (see next section)
3. Download and Install ODBC PowerPack
4. You will also need Power BI Desktop (FREE) installed

About Amazon MWS / API

If you want to sell something on Amazon you can use their e-commerce platform with some monthly fee. You can setup your entire online store / inventory using admin interface.

Amazon Marketplace Web Service (Amazon MWS) is an integrated web service API that helps Amazon sellers to programmatically exchange data on listings, orders, payments, reports, and more. Using these API you can read or write data from your Seller account and integrate it inside your own Systems (e.g. Reporting / ETL / BI tools).

Obtain MWS API Access Key / Secret and Seller ID

Very first thing to call any MWS API is to obtain Access Key, Secret Key and Seller ID (i.e. Merchant ID). To obtain this you first have to register a developer.

Here is where to look for your Merchant ID and Key / Secret.  Here is direct link to User Permissions page.

How to get Amazon MWS developer keys and know your Merchant ID (i.e. Seller ID)

Getting Started

Now let’s look at step by step instructions on how to call Amazon MWS API and then import into Power BI. In below steps we will use  ZappySys ODBC PowerPack so make sure it’s installed first.

Using ScratchPad to Test Amazon MWS API

Each MWS API has set of required and optional parameters. You can refer each API page for details. For example ListMatchingProducts API which we will use as an example. To make MWS API testing / learning easy, Amazon provides online testing tool called MWS ScratchPad. We will use this tool to craft API requests and use that information in ODBC driver SQL queries and later import data in Power BI.

1. Open MWS ScratchPad  by vising https://mws.amazonservices.com/scratchpad/index.html
2. Select Products API category and pick ListMatchingProducts API
3. Enter your SellerID (i.e. Merchant ID), Developer AccessKey (i.e. AWSAccessKeyId) and Secret Key we obtained in the previous section. Enter MarketId (e.g. ATVPDKIKX0DER for USA Market) and other parameters as below and Click Submit to see response.
   

   Using Amazon MWS ScratchPad (API Testing Tool – Created by Amazon)

4. Click on the Response Details Tab to extract some important information we will use for ODBC configuration. Notice attributes in red rectangles. They are the ones which we will need in the POST request Body. ZappySys Drivers supply many common parameters automatically but any API endpoint specific parameter must be supplied in the Request Body. For example you dont have to supply SignatureVersion, Timestamp, Signature or SignatureMethod
   

   Calling Amazon MWS API in Scratchpad Testing Tool (Response Details Tab) – ListMatchingProducts Example

This API requires Market ID parameter so enter it as per your Market where you selling. For example USA marketplaceid = ATVPDKIKX0DER. You can lookup correct MarketplaceId by checking this help (endpoints / marketplaces).

Create ODBC DSN Connection for MWS API / Test Query

Once you have necessary details to call your Amazon MWS API, we can create ODBC DSN which will be used as a reusable connection for MWS API data access in apps like Power BI.

So let’s get started.

1. In your start menu search for ODBC and select Data Source (64 bit):  If you need to access DSN by Service or scheduled Job then make sure you choose 64-bit DSN if caller App is 64 bit App.
   

   Open odbc data source

2. Use the User DSN page and press Add. For DSN access by Service account or automated jobs which runs under different user, choose System DSN Tab, Make sure You create DSN for correct Bitness (i.e. for 64bit Apps create 64 Bit DSN)
   

   Add new data source

3. Select ZappySys XML Driver and click Finish. Amazon MWS API is XML format so we selected XML driver however you can choose JSON or CSV Driver instead of XML if you dealing with JSON format.
   

   Add new ZappySys XML Driver (For XML / SOAP API)

4. Once you see Driver Configuration UI window open, enter the following settings
5. ODBC DSN name (e.g. ZS – Amazon MWS API Connection)
6. Enter MWS API Service URL as below. Notice that we added /Products/2011-10-01 (This was extracted from previous section). You can check full list of Amazon MWS Endpoints here. Because we are connecting to USA region we will use below URL.
   

   https://mws.amazonservices.com/Products/2011-10-01

7. Select OAuth from the Connection dropdown and Click Configure Settings
8. On the Connection UI, Select OAuth Provider Amazon Market Web Service (MWS)
9. Enter your API Access Key and Secret Key (obtained from here)
10. Your connection will look like below.
    

    ODBC Connection Credentials for Amazon MWS API (Access Key, Secret Key, URL Endpoint)

11. Also configure Retry options to handle API limit reached error when you call API too often. For Amazon MWS we get Status code 503 when request is throttled so lets narrow down on which error we want to retry.
    

    Retry Options

12. Click OK to save OAuth Connection UI to go back to main UI
13. Now let’s use some information we gathered from the previous section (Using Scraptchpad Test).
14. Select POST for HTTP Request Method
15. Enter Body as below (Change last parameter with your own SellerID and if needed change your MarketPlaceId too if its not USA)
    

    Action=ListMatchingProducts&MarketplaceId=ATVPDKIKX0DER&Query=harry%20potter%20dvd&SellerId=ZZZZZ-YOUR-SELLER-ID-ZZZZZ

16. Click on Select Filter to select correct node (in our case it will be Product node as below.
    

    $.ListMatchingProductsResponse.ListMatchingProductsResult.Products.Product[*]

17. Now Go to Throttling Tab and enter some delay (e.g. enter 20000 for 20sec delay after each request). This will slow down API calls so we don’t face errors about API Limit reached.
18. You can also consider enabling Data Caching for MWS API as below to avoid API throttling / rate exceeded error. Change Cache Mode to Cache All (Use Cache for Metadata + Data Requests)
    
    NOTE: MWS may allow only certain number of requests per minute so you might want to use each call wisely. You can find Throttling information on API help page itself (like this one see throttling section. Also this one).
    

    Enable Caching Options for Metadata / Data for ZappySys ODBC Drivers

19. Now goto Preview Tab and click on Green Play button to test query
    

    Query Amazon MWS API using ZappySys XML Driver – Preview UI

20. That’s it. We have successfully tested first query to read data from Amazon MWS API

Caching Metadata and Using in SQL Query

We talked briefly about Data Caching but still there will be a time you will face such errors. So lets do another safeguard to reduce total API requests for MWS. Check this article to learn about Caching and META Clause in SQL.

Here are high level steps to Use predefined Schema File so Driver doesn’t call API request to guess metadata.

1. Go to ODBC Preview window and write desired query you like to use. Click Preview Button to see sample data
2. Now click Save Metadata of query which we can use later. Save to File in our case (name as mws-query-meta.txt ).
   

   Create Metadata for ODBC Driver Query – ZappySys API Drivers

3. Now change our query as below
   

   SELECT 
    ......
    ......
   FROM xxxxxxxxx
   WITH (META='c:\somefolder\mws-query-meta.txt')

4. That’s it. Use query like above when we import Amazon MWS data in Power BI in the next section.

Import Amazon MWS data in Power BI (List Orders Example)

Now it’s time to load our data in Power BI and create reports.

1. Open Power BI Desktop and select the option Get data
   

   Get data in power bi

2. Get Data will allow adding the ZappySys ODBC driver. Go to Other and select ODBC.
   

   select odbc in power bi

3. Now it’s time to import data. Basically, there are two modes to import data. Table Mode and Query Mode. Query mode is the most common way. Select your DSN and click Advanced Option to enter custom SQL Query to Import your MWS API data. You can use ODBC DSN Data sources Preview tool to generate SQL Query. For example you can enter query to import Orders for specified date range. We have added some pagination attributes which we will cover in the next section. Click OK to continue.
   

   SELECT * FROM $
   WITH(
   	 ElementsToTreatAsArray='Order'
   	,Src='https://mws.amazonservices.com/Orders/2013-09-01'
   	,Filter='$.ListOrders[$tag$]Response.ListOrders[$tag$]Result.Orders.Order[*]'
   	,RequestData='Action=ListOrders[$tag$]&SellerId=A1xxxxxxxxxxxxx&CreatedAfter=2018-09-01T04%3A00%3A00Z&MarketplaceId.Id.1=ATVPDKIKX0DER'
   	,RequestMethod='POST'
   	,NextUrlAttributeOrExpr='$.ListOrders[$tag$]Response.ListOrders[$tag$]Result.NextToken'
   	,NextUrlSuffix='&NextToken=<%nextlink_encoded%>'
   	,HasDifferentNextPageInfo='True'
   	,EnablePageTokenForBody='True'
   	,PagePlaceholders='body=|ByNextToken;filter=|ByNextToken'
   	,WaitTimeMs=300 -- API throttling for pagination
   	--Uncomment Caching Line to Speedup API calls and Avoid API Limit. Power BI issues many API calls behind the scene
   	--,CachingMode='All',CacheStorage='File',CacheEntryTtl=120,CacheFileLocation='c:\temp\mws-orders.cache'
   	--,DataConnectionType='OAuth'
   	--,ScopeSeparator='{space}'
   	--,ServiceProvider='AmazonMWS'
   	--,ClientId='AKIAIWxxxxxxxxxxxxx' --Skip this to use DSN credentials
   	--,ClientSecret='AKIAIWxxxxxxxxxxxxx' --Skip this to use DSN credentials
   	--,UseCustomApp='True'	
   )
   --Uncomment below if you are using Schema file for cached metadata
   --WITH (META='c:\somefolder\mws-query-meta.txt')
4. Use above query for example and paste in Query editor while you Import Amazon MWS data.
   

   Enter custom SQL to load SOAP Web Service Data in Power BI (ZappySys XML Driver)

5. On the next screen select Windows Authentication
   
   

   Select Authentication for Amazon MWS connection in Power BI data load

6. Once data is imported you can use dataset to build your dashboard
   

   Load Amazon MWS data in Power BI – Create Amazon Sales Dashboard

Using Pagination in Amazon MWS API calls

Now let’s look at how to query certain API which allows paginated response. Pagination is fully explained here. Certain APIs like ListOrders , ListOrderItems, ListInboundShipments, ListInventorySupply, GetFullfillmentOrder and few more supports Pagination.

When you call Paginated response next request changes Action with ByNextToken suffix. For example to get orders call ListOrders and next request will be ListOrdersByNextToken

MWS API has multiple pagination patterns so based on which action you calling you have to use correct SQL parameters.

Using NextToken for Pagination (ListOrders Example)

Many MWS API returns NextToken attribute in the response which you can pass in the body of next request until no more NextToken found in the response.

Check below example query for ListOrders API. We have enabled pagination by supplying few extra parameters in our original SQL Query.

SELECT * FROM $
WITH(
	 ElementsToTreatAsArray='Order'
	,Src='https://mws.amazonservices.com/Orders/2013-09-01'
	,Filter='$.ListOrders[$tag$]Response.ListOrders[$tag$]Result.Orders.Order[*]'
	,RequestData='Action=ListOrders[$tag$]&SellerId=A10JBWAX563WMC&CreatedAfter=2018-09-01T04%3A00%3A00Z&MarketplaceId.Id.1=ATVPDKIKX0DER'
	,RequestMethod='POST'
	,NextUrlAttributeOrExpr='$.ListOrders[$tag$]Response.ListOrders[$tag$]Result.NextToken'
	,NextUrlSuffix='&NextToken=<%nextlink_encoded%>'
	,WaitTimeMs=300 -- slow down to avoid throttling
	,HasDifferentNextPageInfo='True'
	,EnablePageTokenForBody='True'
	,PagePlaceholders='body=|ByNextToken;filter=|ByNextToken'
	--,DataConnectionType='OAuth'
	--,ScopeSeparator='{space}'
	--,ServiceProvider='AmazonMWS'
	--,ClientId='AKIxxxxxx'
	--,ClientSecret='AKIxxxxxx'
	--,UseCustomApp='True'	
)
--Uncomment below if you are using Schema file for cached metadata 
--WITH (META='c:\somefolder\mws-query-meta.txt')

Using NextToken with operations that return HasNext / MoreResultsAvailable

Some API has attribute which tells you to stop pagination (e.g. HasNext). Check below example query for GetReportRequestList. We have enabled pagination by supplying few extra parameters in SQL Query.

SELECT * FROM $
WITH(
	 ElementsToTreatAsArray='Product,ReportRequestInfo'
	,Src='https://mws.amazonservices.com'
	--,DataConnectionType='OAuth'
	--,ScopeSeparator='{space}'
	--,ServiceProvider='AmazonMWS'
	--,ClientId='Axxxxxxxxxxxxxx' --This will come from connection if you comment
	--,ClientSecret='Bxxxxxxxxxxxxxxxxx' --This will come from connection you comment
	--,UseCustomApp='True'
	,Filter='$.GetReportRequestList[$tag$]Response.GetReportRequestList[$tag$]Result.ReportRequestInfo[*]'
	,RequestData='Action=GetReportRequestList[$tag$]&Merchant=A10xxxxxxxxxxxxxxxx'
	,RequestMethod='POST'
	,NextUrlAttributeOrExpr='$.GetReportRequestList[$tag$]Response.GetReportRequestList[$tag$]Result.NextToken'
	,NextUrlEndIndicator='false'
	,StopIndicatorAttributeOrExpr='$.GetReportRequestList[$tag$]Response.GetReportRequestList[$tag$]Result.HasNext'
	,NextUrlSuffix='&NextToken=<%nextlink_encoded%>'
	,HasDifferentNextPageInfo='True'
	,EnablePageTokenForBody='True'
	,PagePlaceholders='body=|ByNextToken;filter=|ByNextToken'
)
--Uncomment below if you are using Schema file for cached metadata 
--WITH (META='c:\somefolder\mws-query-meta.txt')

Some API calls return slightly different response so rather than HasNext attribute you will see MoreResultsAvailable. In that case change StopIndicatorAttributeOrExpr property to use MoreResultsAvailable property.

Properties you need to change for pagination

Now let’s look at properties which need to be changed to support pagination.

POST /Orders/2013-09-01?AWSAccessKeyId=AKxxxxxxxxxx
  &Action=ListOrders
  &SellerId=A10zzzzzzz
  &SignatureVersion=2
  &Timestamp=2018-09-17T21%3A35%3A57Z
  &Version=2013-09-01
  &Signature=3GigipfRsQgzzzzzzzzzzzzzs%3D
  &SignatureMethod=HmacSHA256

Action=ListOrders[$tag$]&SellerId=A10zzzzzzz&CreatedAfter=2018-09-01T04%3A00%3A00Z

<?xml version="1.0"?>
<ListOrdersResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

<?xml version="1.0"?>
<ListOrdersByNextTokenResponse xmlns="https://mws.amazonservices.com/
    Orders/2013-09-01">
    <ListOrdersByNextTokenResult>
        <NextToken>2YgYW55IxxxxxxxxxxxxVyZS4=</NextToken>
        <LastUpdatedBefore>2017-02-25T18%3A10%3A21.687Z</LastUpdatedBefore>
        <Orders>
            <Order> ....... </Order>
            <Order> ....... </Order>
            <Order> ....... </Order>
        </Orders>
    </ListOrdersResult>
</ListOrdersResponse>

Here is the preview of MWS API query output with Pagination Attributes.

Amazon MWS API Pagination Example – Preview SQL Query Result

Import data from MWS Custom Reports (Inventory Report Example)

Now lets look at how to extract data from Custom Report API (You must need ODBC PowerPack v1.1.1 or higher). However reading data from Reports not single step process because report generation is Job style API. Which means you send report request and wait  until its done. Once Report ready you have to read in CSV format or XML. Some Reports only available in CSV format. Check this post to understand how complex it can be to get data. We will make it simple for you to understand this in 3 steps. Basically calling reports requires minimum 3 API calls (see below)

1. Create a new report request – Cal RequestReport API . It returns ReportRequestId (You can use it in the next step)
2. Check Report Status see its done – Call GetReportRequestList API (Pass ReportRequestId got in the previous step to get data for only one Report request we care). Keep checking Report Status every few seconds until you get _DONE_ status in response.
3. Read Report Data – Call GetReport. This API call returns CSV or XML data. So use correct Driver for this step. If CSV data is returned then we must use CSV Driver rather than XML Driver.

Now lets see how to get this 3 steps done in Power BI. In this example, We will get Inventory by calling _GET_MERCHANT_LISTINGS_DATA_ report type.

1. Create a directory c:\temp
2. Create new ZappySys XML Driver DSN (User DSN) using same steps we described before. Lets name this DSN as ZAPPY-MWS-API-XML
3. Create new ZappySys CSV Driver DSN  (User DSN) using same steps we described before (Except the Driver Type should be ZappySys CSV Driver). Lets name this DSN as ZAPPY-MWS-API-CSV
4. Once you done with ODBC DSN Setup open or create new Power BI Project.
5. Click on Create Source > from Blank Query
   

   Power BI – Create New Data source from Query (M Script)

6. When Query Editor Shows up click on Advanced Editor and paste below code (Its Power BI M Language). If you are unsure how to write Power BI check this tutorial.
   Replace MarketPlaceId and Seller ID in the script (i.e. A1xxxxxx and A2xxxxxx ). See we are using ReportType=_GET_MERCHANT_LISTINGS_DATA_  but you can use any valid report ID from here. You can also pass other optional parameter available for each report Type (e.g. StartDate and EndDate)
   

   Edit Power BI Query Script – Use custom M Language Script to import data from ODBC or other Relational

   Power BI M Script (New Driver) – Import MWS Custom Report in Power BI
   

   //////////////////////////////////////////////////////////////////////////
   // Change _GET_MERCHANT_LISTINGS_DATA_ to something else if you need different report type (all 3 steps) -- https://docs.developer.amazonservices.com/en_US/reports/Reports_ReportType.html
   // Replace MarketplaceId in all 3 steps (i.e for USA use ATVPDKIKX0DER) --https://docs.developer.amazonservices.com/en_US/dev_guide/DG_Endpoints.html
   // Replace SellerId in all 3 steps
   // Remove or Change CreatedAfter in first step (or use CreatedBefore parameter instread of CreatedAfter)
   //////////////////////////////////////////////////////////////////////////
   
   let
       sql="SELECT s3.* FROM $
   WITH(
   Src='https://mws.amazonservices.com/Reports/2009-01-01',RequestMethod='POST'
   ,RequestData='Action=RequestReport&ReportType=_GET_MERCHANT_LISTINGS_DATA_&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxx&CreatedAfter=2017-12-31T04%3A00%3A00Z'
   ,IncludeParentColumns='False'
   ,EnableRawOutputModeSingleRow='True'
   ,RawOutputFilterExpr='//*[local-name()=""ReportRequestId""]' 
   ,RawOutputDataRowTemplate='{ReportRequestId:""[$1]""}' 
   ,RawOutputExtractMode='Xml' 
   ,WaitTimeMs=5000
   ,Meta='[{""Name"": ""ReportRequestId"",""Type"": ""Int64""}]'
   ,Alias='s1'
   
   ,Join1_alias='s2'	
   ,Join1_Src='https://mws.amazonservices.com/Reports/2009-01-01',Join1_RequestMethod='POST'
   ,Join1_RequestData='Action=GetReportRequestList&ReportRequestIdList.Id.1=[$s1.ReportRequestId$]&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxx'
   ,Join1_EnableRawOutputModeSingleRow='True'
   ,Join1_RawOutputFilterExpr='//*[local-name()=""ReportProcessingStatus""]||//*[local-name()=""GeneratedReportId""]' 
   ,Join1_RawOutputDataRowTemplate='{ReportProcessingStatus:""[$1]"",GeneratedReportId:""[$2]""}' 
   ,Join1_RawOutputExtractMode='Xml' 
   ,Join1_EnableStatusCheck='True', 
   ,Join1_StatucCheckMaxWaitSeconds=300,
   ,Join1_StatucCheckIterationWaitSeconds=25,
   ,Join1_StatusSuccessValue='_DONE_'
   ,Join1_StatusFailedValue='_DONE_NO_DATA_|_CANCELLED_'
   
   ,Join2_Alias='s3'
   ,Join2_Src='https://mws.amazonservices.com/Reports/2009-01-01',Join2_RequestMethod='POST'
   ,Join2_RequestData='Action=GetReport&ReportId=[$s2.GeneratedReportId$]&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxx'	
   ,Join2_ColumnDelimiter='{TAB}'
   ,Join2_ResponseCharset='Windows-1252'	 
   ,Join2_EnableStatusCheck='False' 
   ,Join2_EnableRawOutputModeSingleRow='False'
   )",
       Source = Odbc.Query("dsn=ZS-AMAZON-MWS-CSV", sql)
   in
       Source

    

   Power BI M Script (Old Driver) – v1.1.2 or older

   If you are using older version – v1.1.2 or older then use below script.

   /*
    All Copy Rights Reserved - ZappySys LLC. Feel free to share this snippet as long as you retain copyrights notice. https://zappysys.com
    Description:  This snippet will call Amazon MWS API to generate Inventory Report (Its multi step process - Job Style API call. First 2 calls gets XML data and Last call CSV data)
   */
   
   let
       //NOTE: In Power BI M Language... each declare is separated by comma... 
       //variable can be any type.. string, number, table, list , record, function reference. More info: https://docs.microsoft.com/en-us/powerquery-m/power-query-m-reference
       /* 
       let 
           var1=expression, var2=expression,  ... varN=expression
       in
           varN   //return some variable
       */
   
       //define some common variables
   
       MarketplaceId="A1xxxxxx",
       SellerId="A2xxxxxxxxxx",
   
       //////////////////////////////////////////////////
       /*Step-1 : Send new report request  - Must use XML Driver */
       //////////////////////////////////////////////////
   
       Step1Sql="SELECT ReportRequestId FROM $
   WITH(
       --Src='c:\temp\reportrequest.txt'
       Src='https://mws.amazonservices.com/Reports/2009-01-01'
   	--,ElementsToTreatAsArray='Product'
   	,Filter='$.RequestReportResponse.RequestReportResult.ReportRequestInfo'
   	,RequestData='Action=RequestReport&ReportType=_GET_MERCHANT_LISTINGS_DATA_&MarketplaceId=" & MarketplaceId & "&SellerId=" & SellerId & "'
   	,RequestMethod='POST'
   	,WaitTimeMs='300'
   	,IncludeParentColumns='False'
   	,CachingMode='All'
   	,CacheStorage='File'
   	,CacheFileLocation='c:\temp\mws-step1.cache'
   	,CacheEntryTtl=300
   	,Meta='[{""Name"": ""ReportRequestId"",""Type"": ""Int64""}]'
   )",
       Source1 = Odbc.Query("dsn=ZAPPY-MWS-API-XML", Step1Sql),
       ReportRequestId= Source1{0}[ReportRequestId],
       
       //NOTE: Technically we need to implement status check loop Wait for few seconds before 
       //Increase this if you get ERROR in 3rd step
       BlankTextForDelay = Function.InvokeAfter(()=>"", #duration(0,0,0,15)), //define delay - You must use it somewhere inorder to invoke delay
   
       //////////////////////////////////////////////////
       /*Step-2 : Get report id - once report is done  - Must use XML Driver */
       //////////////////////////////////////////////////
       Step2Sql=BlankTextForDelay & "SELECT *  FROM $
   WITH(
       --Src='c:\temp\report-status.xml'
       Src='https://mws.amazonservices.com/Reports/2009-01-01'
   	,ElementsToTreatAsArray='ReportRequestInfo'
   	,Filter='$.GetReportRequestListResponse.GetReportRequestListResult.ReportRequestInfo'
   	,RequestData='Action=GetReportRequestList&ReportRequestIdList.Id.1=" &  ReportRequestId & "&MarketplaceId=" & MarketplaceId & "&SellerId=" & SellerId & "'
   	,RequestMethod='POST'
   	,WaitTimeMs='300'
   	,IncludeParentColumns='False'
   	,CachingMode='All'
   	,CacheStorage='File'
   	,CacheFileLocation='c:\temp\mws-step2.cache'
   	,CacheEntryTtl=15
   	,Meta='[{""Name"": ""ReportProcessingStatus"",""Type"": ""String"", Length: 100},{""Name"": ""GeneratedReportId"",""Type"": ""Int64""}]'
   )",    
       Source2 = Odbc.Query("dsn=ZAPPY-MWS-API-XML", Step2Sql),
       ReportId= Source2{0}[GeneratedReportId],
   
   
       //////////////////////////////////////////////////
       /*Step-3 : Get report data in CSV format - Must use CSV Driver */
       //////////////////////////////////////////////////
       Step3Sql="SELECT * FROM $
   WITH(
          --Src='c:\temp\inventory.csv.txt'
   	 Src='https://mws.amazonservices.com/Reports/2009-01-01'
   	,RequestData='Action=GetReport&ReportId=" &  ReportId & "&MarketplaceId=" & MarketplaceId & "&SellerId=" & SellerId & "'
   	,RequestMethod='POST'
   	,WaitTimeMs='300'
   	,ColumnDelimiter='{TAB}'
   	,CachingMode='All'
   	,CacheStorage='File'
   	,CacheFileLocation='c:\temp\mws-step3.cache'    
   	,ResponseCharset='Windows-1252'	    
   	)",    
       Source3 = Odbc.Query("dsn=AMAZON-MWS-CSV", Step3Sql)
   
   in
       Source3  //return Source3 (i.e. Table type variable)

    

7. Once done click OK.
8. You may be prompted few times to trust each Query we defined in M Script above. Also you may be prompted to select DSN Credentials (Just Select Windows Credentials)
   

   Power BI – Run Native Database Query Prompt

9. If you get prompted to select DSN Credentials then select as below (Click Windows).
   

   Select Authentication for Amazon MWS connection in Power BI data load

10. After few seconds (Depending on Delay we define in Script – default 15 sec) you will see data loaded.
11. Once data is loaded click Close & Apply and it will close the query editor and we will see main designer where you can see Dataset and Fields we just created.
    

    Loading Amazon MWS Reports data in Power BI (List Inventory Example using M Script)

Conclusion

In this article, we saw how to build sales report by reading data from your Amazon Seller Account. We used simple SQL queries with ZappySys XML Driver to pull data from Amazon MWS API. ZappySys ODBC PowerPack makes it super easy to consume data from virtually any API (JSON / XML or CSV Web Services), no coding required. Download here to get started with ZappySys ODBC drivers.

The post How to Import Amazon MWS data in Power BI appeared first on ZappySys Blog.

In our previous blog we discussed how to import REST / SOAP API data in Power BI. Now we will use that knowledge to read NetSuite data in Power BI (NetSuite CRM – SuiteTalk). We will use ZappySys XML Driver to call Search function in NetSuite SOAP API. If you are not familiar with SOAP API concept then we highly recommend you read previous blog post. In this post we will use Power BI as an example App to consume NetSuite data but you can use steps listed in this article to load data in any other ODBC compliant ETL / reporting app (e.g. Excel, Qlik, Informatica, SSRS) or programming language (e.g. C#, JAVA, Python) or SQL Server (Linked Server).

We will look at each step in depth, how to enable SOAP Api access in NetSuite, How to register Application, how to configure NetSuite ODBC DSN, and finally how to load NetSuite data into Power BI dashboard.

So let’s get started. The approach listed in this article can be used to call other SOAP based API such as Salesforce, Workday, SAP, Sage etc.

Prerequisite

Before you get started make sure following requirements are met.

1. NetSuite CRM Login (UserID and Password)
2. Enable Web Service Feature in NetSuite
3. Obtain NetSuite Account ID
4. Obtain Application ID if you have created an App. Else Register new Netsuite Application and get Application ID
5. Download ZappySys ODBC PowerPack (for SOAP / XML Driver)
6. Download Power BI Desktop (FREE) if you have not already done it.

What is NetSuite CRM?

NetSuite is a CRM / ERP product. It gives you scalable cloud CRM / ERP solution targeted at high-growing, mid-sized businesses and large enterprises. It automates front- and back-office processes including: financial management, revenue management, fixed assets, order management, billing, and inventory management.

Preparing for NetSuite WebService Call in Power BI

For NetSuite API call we have to make sure certain settings and obtain Application ID and AccountID. So let’s get started.

How to enable NetSuite Web Service Feature

Before you call any NetSuite SOAP API (i.e. SuiteTask Web Service), First thing to get started with NetSuite SOAP API (i.e. ) is to enable Web Service Feature. Once you enable Web Service, you can programmatically access NetSuite data in your App (e.g. Power BI, Tableau, Informatica). Click this link to see steps to enable Web Service Feature

Enable NetSuite Web Service Feature (SuiteTalk API)

Obtain NetSuite Account ID

NetSuite Account ID is needed by Login API call. So perform following steps if you dont know account id.

1. Login to your NetSuite Portal
2. Click the Setup Tab
3. In the drop-down click Integration >  Web services preferences
4. On that page you will find your Account ID (example account ID = TSTDRV1234567 )

Obtain NetSuite Account ID (Needed for Web Service Login)

Obtain NetSuite Application ID

Once you have account ID now we need to get Application ID. For that follow these steps.

1. Login to your NetSuite Portal
2. Click the Setup Tab
3. In the drop-down click Integration >  Manage Integrations
4. On that page you will find your existing Applications along with Account ID / Status (example Application ID = AE2EACFD-1234-1222-1111-12345544555 )
5. If you don’t have any application created yet then just click on New.

Create new App / Get NetSuite Application ID

Understanding how NetSuite SOAP API Call works

In this section we will briefly understand how NetSuite SOAP API call works. When you use ZappySys Driver / Component it will take care many details for you but still you have to configure correct XML Body. There are two steps involved before you can call any SOAP API in NetSuite.

1. Call Login method and extract JSESSIONID from response Cookie (This token expires after some time)
2. Call any other API (Pass obtained JSESSIONID token in Cookie header)

NetSuite Login Call

Request

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml;charset=UTF-8
SOAPAction: "login"
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
  </soap:Header>
  <soap:Body>
    <login xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <passport>
        <email xmlns="urn:core_2017_1.platform.webservices.netsuite.com">xxxxxxxxxxxxx</email>
        <password xmlns="urn:core_2017_1.platform.webservices.netsuite.com">xxxxxxxxxxxx</password>
        <account xmlns="urn:core_2017_1.platform.webservices.netsuite.com">TSTxxxxxxxxxxxxxxx</account>
        <role internalId="3" xmlns="urn:core_2017_1.platform.webservices.netsuite.com" />
      </passport>
    </login>
  </soap:Body>
</soap:Envelope>

Response Notice JSESSIONID=6xI6xxxxxxxxx in the login call response below. This Session ID can be used to call next call. ZappySys HTTP Connection Manager configured with Dynamic Token option can extract Cookie and reuse it until its expired (1 hour by default).

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 16:50:41 GMT
Content-Length: 1676
Content-Type: text/xml; charset=utf-8
X-N-OperationId: 777274f6-49c6-4883-bcd1-1d70df90d78e
Strict-Transport-Security: max-age=31536000
NS_RTIMER_COMPOSITE: 13633xxxxxxxxx:70617xxxxxxxxxxxxx36F6D:80
Set-Cookie: JSESSIONID=6xI61rdFt26pxxxxxxxxxxxxxxxqvJIMlu!-17981123445; path=/; secure; HttpOnly
Set-Cookie: NS_VER=2018.1.0; domain=webservices.netsuite.com; path=/
Set-Cookie: NS_ROUTING_VERSION=LAGGING; path=/
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent
Keep-Alive: timeout=10, max=990
Connection: Keep-Alive

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_xxxxxxxxxxxxxxxxx_9fc719</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <loginResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <sessionResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
        <ns1:status isSuccess="true" xmlns:ns1="urn:core_2017_1.platform.webservices.netsuite.com"/>
        <ns2:wsRoleList xmlns:ns2="urn:core_2017_1.platform.webservices.netsuite.com">
          <ns2:wsRole>
            <ns2:role internalId="3">
              <ns2:name>Administrator</ns2:name>
            </ns2:role>
            <ns2:isDefault>false</ns2:isDefault>
            <ns2:isInactive>false</ns2:isInactive>
          </ns2:wsRole>
          <ns2:wsRole>
            <ns2:role internalId="15">
              <ns2:name>Employee Center</ns2:name>
            </ns2:role>
            <ns2:isDefault>false</ns2:isDefault>
            <ns2:isInactive>false</ns2:isInactive>
          </ns2:wsRole>
      </sessionResponse>
    </loginResponse>
  </soapenv:Body>
</soapenv:Envelope>

 

NetSuite API Call ( getAll method )

Now once you have JSESSIONID extracted you can pass it to call next API call as below. For example here is how to call getAll method to obtain currency list and exchange rates. If you are not sure how to craft correct Request XML then use tool like SoapUI. You can use NetSuite WSDL file from here and import it in SoapUI.   Request See how Cookie is passed for authentication purpose. Again you can pass this manually or use HTTP connection with Dynamic Token to pass it automatically along with each API call.

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml;charset=UTF-8
SOAPAction: "getAll"
Cookie: JSESSIONID=6xI61rxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxMlu!-1798141234
Accept-Encoding: gzip, deflate

<soapenv:Envelope 
   xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
   xmlns:urn="urn:messages_2018_1.platform.webservices.netsuite.com" 
   xmlns:urn1="urn:core_2018_1.platform.webservices.netsuite.com">
   <soapenv:Body>
      <urn:getAll>
         <record recordType="currency"/> 
      </urn:getAll>
   </soapenv:Body>
</soapenv:Envelope>

Response

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 16:50:42 GMT
Content-Length: 9648
Content-Type: text/xml; charset=utf-8
X-N-OperationId: 1b2989e4-b930-4a72-b471-00a8e131dfd4
NS_RTIMER_COMPOSITE: 21403xxxxxx:706172746E65xxxxxxxxxxxx6F6D:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_TSTDRV1439151_100320189871507121000128715_6ae8e8</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <getAllResponse xmlns="urn:messages_2018_1.platform.webservices.netsuite.com">
      <platformCore:getAllResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>11</platformCore:totalRecords>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>USD</listAcct:name>
            <listAcct:symbol>USD</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>$</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_unitedStatesEnglish</listAcct:locale>
            <listAcct:formatSample>$1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>1.0</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>GBP</listAcct:name>
            <listAcct:symbol>GBP</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>£</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_unitedKingdomEnglish</listAcct:locale>
            <listAcct:formatSample>£1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>1.33165497</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
          <platformCore:record internalId="3" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>CAD</listAcct:name>
            <listAcct:symbol>CAD</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>$</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_canadaEnglish</listAcct:locale>
            <listAcct:formatSample>$1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>0.76457273</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
        </platformCore:recordList>
      </platformCore:getAllResult>
    </getAllResponse>
  </soapenv:Body>
</soapenv:Envelope>

NetSuite Search API (Pagination Example)

Now let's look at more complex API calls where Pagination is needed. For example when you call search API to pull data from specific table (e.g. Account) you need to loop through multiple responses until all records are consumed. Request (Page-1)

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml; charset=utf-8
SOAPAction: "search"
Cookie: JSESSIONID=BKQ7xxxxxxxxxxxx4Ug!-995xxxxxx
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
    <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <pageSize>50</pageSize>
    </searchPreferences>
  </soap:Header>
  <soap:Body>
    <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:AccountSearch">
        <q1:basic>
          <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
            <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">x</searchValue>
          </description>
          <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
            <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">a</searchValue>
          </name>
        </q1:basic>
      </searchRecord>
    </search>
  </soap:Body>
</soap:Envelope>

  Response (Page-1) In below response notice 2 things. SearchId is like a cursor id which you have to pass for each page you request after first page. And you can stop looping after 4th page based on totalPages count. platformCore:searchId = WEBSERVICES_ABCD_12345-xxxxxxxxx platformCore:totalPages = 4

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 18:40:33 GMT
Content-Length: 49266
Content-Type: text/xml; charset=utf-8
X-N-OperationId: c1e79336-63b7-4eb7-989e-5c3d24c56891
NS_RTIMER_COMPOSITE: 118xxxxxxxx:70617xxxxxxxxxxxxxxxxxxxxxD:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="utf-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <searchResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformCore:searchResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>193</platformCore:totalRecords>
        <platformCore:pageSize>50</platformCore:pageSize>
        <platformCore:totalPages>4</platformCore:totalPages>
        <platformCore:pageIndex>1</platformCore:pageIndex>
        <platformCore:searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformCore:searchId>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>1111</listAcct:acctNumber>
            <listAcct:acctName>Checking</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>xxxxxxxxxxxxxxxxxxx</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Checking</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>22222</listAcct:acctNumber>
            <listAcct:acctName>Savings</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>yyyyyyyyy</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Savings</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          .........
          .........
          .........
          .........		  
        </platformCore:recordList>
      </platformCore:searchResult>
    </searchResponse>
  </soapenv:Body>
</soapenv:Envelope>

  Request (Page-2 and onwards) Here is the second page request. Notice few things here. In heraders we are now sending SOAPAction: "searchMoreWithId" rather than "search".  Also our SOAP Body is also different compared to first page request. Now we only sending pageIndex we want to fetch and searchId. We have also change <search> tag with <searchMoreWithId> under Body.

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml; charset=utf-8
SOAPAction: "searchMoreWithId"
Cookie: JSESSIONID=BKQ7xxxxxxxxxxxx4Ug!-995xxxxxx
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
    <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <pageSize>50</pageSize>
    </searchPreferences>
  </soap:Header>
  <soap:Body>
    <searchMoreWithId xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</searchId>
      <pageIndex>2</pageIndex>
    </searchMoreWithId>
  </soap:Body>
</soap:Envelope>

Response (Page-2 and onwards)

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 18:40:33 GMT
Content-Length: 49266
Content-Type: text/xml; charset=utf-8
X-N-OperationId: c1e79336-63b7-4eb7-989e-5c3d24c56891
NS_RTIMER_COMPOSITE: 118xxxxxxxx:70617xxxxxxxxxxxxxxxxxxxxxD:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="utf-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <searchMoreWithIdResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformCore:searchResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>193</platformCore:totalRecords>
        <platformCore:pageSize>50</platformCore:pageSize>
        <platformCore:totalPages>4</platformCore:totalPages>
        <platformCore:pageIndex>2</platformCore:pageIndex>
        <platformCore:searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformCore:searchId>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>8000</listAcct:acctNumber>
            <listAcct:acctName>Checking</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>xxxxxxxxxxxxxxxxxxx</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Checking</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>8001</listAcct:acctNumber>
            <listAcct:acctName>Savings</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>yyyyyyyyy</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Savings</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          .........
          .........
          .........
          .........		  
        </platformCore:recordList>
      </platformCore:searchResult>
    </searchMoreWithIdResponse>
  </soapenv:Body>
</soapenv:Envelope>

   

Getting Started

Now let’s check step by step instructions on how to load NetSuite data in Power BI. In this exercise we will focus on calling search API but you can call virtually any API using the same technique.

For detailed help on NetSuite SOAP API and other useful information check this documentation. To browse schema you can use this link. From schema browser we will use Search Type name later on in this article. For example later on in this article to search accounts we will use AccountSearch as below example. This name can be found from here(click search tab) You can change it to something else. For now dont worry too much until we get to that point.

Creating NetSuite ODBC Connection (Using ZappySys XML / SOAP Driver)

Once you have all necessary information handy we can create ODBC DSN for NetSuite which can be used by any ODBC compliant app such as Power BI. Perform the following steps to set NetSuite Connection which we will use to query data (e.g. Accounts, Budget, Campaign etc).

1. Open start menu and search for “ODBC 64”. Launch ODBC Data sources for 64 bit
2. Click Add and select ZappySys XML Driver
   

   ZappySys ODBC Driver for XML / SOAP API

3. Once driver UI is visible change default DSN name (e.g. change to ZS – NetSuite)
4. Click on the Load Connection String button and enter the following string after replacing below placeholders to configure the NetSuite connection. If you are unsure how to get Account ID and Application ID then check previous sections. For now dont worry about various options in the connection string. We will cover them later.
   Replace $$$YOUR-APPLICATION-ID$$$  (2 times) with your own Application ID
   Replace $$$YOUR-ACCOUNT-ID$$$ with your own Account ID
   Replace $$$YOUR-USER-ID$$$  with your own Login ID (e.g myuser@mycompany.com)
   Replace $$$YOUR-PASSWORD$$$  with your own Password
   Replace <pageSize>50</pageSize>  with your own value. We suggest not to go more than 200 rows per page for better performance.
   Replace c:\temp\netsuite.cache.db with your own path for Cache File. Caching must be ON in order to Import data in Power BI correct way. We also use CacheTtl=120 seconds (Cache Expires in 120 seconds) but adjust it longer if your data import takes long time.
   

   DRIVER={ZappySys XML Driver};ElementsToTreatAsArray='platformCore:record';DataPath='https://webservices.netsuite.com/services/NetSuitePort_2017_1';DataConnectionType=HTTP;AuthScheme='JSESSIONID=';TokenUrl='https://webservices.netsuite.com/services/NetSuitePort_2017_1';TokenRequestData='<?xml version="1.0" encoding="utf-8"?>
   <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
     <soap:Header>
       <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <applicationId>$$$YOUR-APPLICATION-ID$$$</applicationId>
       </applicationInfo>
     </soap:Header>
     <soap:Body>
       <login xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <passport>
           <email xmlns="urn:core_2017_1.platform.webservices.netsuite.com">[$userid$]</email>
           <password xmlns="urn:core_2017_1.platform.webservices.netsuite.com">[$password$]</password>
           <account xmlns="urn:core_2017_1.platform.webservices.netsuite.com">$$$YOUR-ACCOUNT-ID$$$</account>
           <role internalId="3" xmlns="urn:core_2017_1.platform.webservices.netsuite.com" />
         </passport>
       </login>
     </soap:Body>
   </soap:Envelope>';TokenRequestMethod='POST';TokenRequestContentType=TextXmlUtf8;TokenResponseValueExtractMode=Cookie;TokenRequestHeaders='SOAPAction:"login"';TokenResponseCookie='JSESSIONID';TokenAuthHeader='Cookie';UserName='$$$YOUR-USER-ID$$$';CredentialType=TokenDynamic;Password='$$$YOUR-PASSWORD$$$';Filter='$.soapenv:Envelope.soapenv:Body.search[$tag$]Response.platformCore:searchResult.platformCore:recordList.platformCore:record[*]';RequestData='<?xml version="1.0" encoding="utf-8"?>
   <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
     <soap:Header>
       <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <applicationId>$$$YOUR-APPLICATION-ID$$$</applicationId>
       </applicationInfo>
       <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <pageSize>50</pageSize>
       </searchPreferences>
     </soap:Header>
     <soap:Body>
   [$tag$]
     </soap:Body>
   </soap:Envelope>';RequestMethod='POST';NextUrlAttributeOrExpr='$.soapenv:Envelope.soapenv:Body.search[$tag$]Response.platformCore:searchResult.platformCore:searchId';FirstPageBodyPart='    <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:AccountSearch">
           <q1:basic>
   		<!--
             <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">x</searchValue>
             </description>
             <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">a</searchValue>
             </name>
   		 --> 
           </q1:basic>
         </searchRecord>
       </search>';NextPageBodyPart='    <searchMoreWithId xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchId>[$pagetoken$]</searchId>
         <pageIndex>[$pagenumber$]</pageIndex>
       </searchMoreWithId>';PagingMaxPagesExpr='$.soapenv:Envelope.soapenv:Body.searchResponse.platformCore:searchResult.platformCore:totalPages';HasDifferentNextPageInfo=True;EnablePageTokenForBody=True;PagePlaceholders='header=|MoreWithId;filter=|MoreWithId';RequestHeaders='Content-Type: text/xml; charset=utf-8 || SOAPAction: "search[$tag$]"';CachingMode=All;CacheStorage=File;CacheEntryTtl=120;CacheFileLocation='c:\temp\netsuite.cache.db'

    
   

   ZappySys ODBC Driver Load Connection String – NetSuite Connection Example

5. After configuration your screen may look like below. You can also click on Advanced View (Radio button) to see some Advanced properties (e.g. Caching Properties such as CachingMode, CacheFileLocation and Advanced Pagination Properties such as PagingMaxPagesExpr, HasDifferentNextPageInfo, NextPageBodyPart, FirstPageBodyPart, EnablePageTokenForBody, PagePlaceholders )
   
   Notice [$tag$] in Body, Header and Filter. This gets replaced at runtime based on some settings we did for pagination (see next 2 screenshots). NetSuite API has different Header, Body and Filter need for First page vs Next Pages so it needs special configuration. [$tag$] in Body gets replacement based on FirstPageBodyPart and NextPageBodyPart properties found on Advanced View under Pagination. [$tag$] used in Filter, Header or any other place gets its replacements from PagePlaceHolders Property found on Advanced View (see filter=firstPartString|nextPartString;header=firstPartString|nextPartString). Use pipe (vertical bar to separate first part placeholder and next part placeholder) In our case we use blank for first part.
   

   NetSuite ODBC Connection Settings (ZappySys XML Driver)

6. Here is what it looks like on Advanced Tab.
   

   ODBC XML Driver – Ádvanced View (Pagination, Caching, Logging and other properties not found in Simple Mode)

7. Now go to preview tab and click Preview Data button. You will see upto 100 sample records. Using below query it will use all options from UI (No override in query).SELECT * from $
   

   Preview NetSuite data (Query Accounts) using ODBC Driver

8. Now let’s change our default query a bit  to get list of Expense Category (i.e. ExpenseCategorySearch). As you see we change FirstRequestBody a little bit (change q1:ExpenseCategorySearch). We also removed some unwanted columns from preview by setting IncludeParentColumns=’false’
   

   SELECT * FROM $
   WITH(
   	FirstPageBodyPart='    <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:ExpenseCategorySearch">
           <q1:basic>
             <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">Some-String-Goes-Here</searchValue>
             </description>
             <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">Some-String-Goes-Here</searchValue>
             </name>
           </q1:basic>
         </searchRecord>
       </search>',
       IncludeParentColumns='False'
   )

   We have commented SearchQuery portion to make it simple but you can learn more about how to write search query from here

   Here is one more example of query which combines Basic Search and Advanced Search

   Pay close attention to namespace prefix in SOAP code you copy from Help center. It may be using different prefix than our original connection string (e.g. we use q1:xxxxx at some places but online sample may use ns1, ns2, ns3 ….)

   SELECT * FROM $
   WITH(
   	FirstPageBodyPart='
       <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchRecord xsi:type="ns4:ContactSearch" xmlns:ns4="urn:relationships_2017_1.lists.webservices.netsuite.com">
           <ns4:basic xsi:type="ns5:ContactSearchBasic" xmlns:ns5="urn:common_2017_1.platform.webservices.netsuite.com">
             <ns5:email operator="contains" xsi:type="ns6:SearchStringField" xmlns:ns6="urn:core_2017_1.platform.webservices.netsuite.com">
               <ns6:searchValue xsi:type="xsd:string">.com</ns6:searchValue>
             </ns5:email>
           </ns4:basic>
           <ns4:customerJoin xsi:type="ns7:CustomerSearchBasic" xmlns:ns7="urn:common_2017_1.platform.webservices.netsuite.com">
             <ns7:entityId operator="contains" xsi:type="ns8:SearchStringField" xmlns:ns8="urn:core_2017_1.platform.webservices.netsuite.com">
               <ns8:searchValue xsi:type="xsd:string">C</ns8:searchValue>
             </ns7:entityId>
           </ns4:customerJoin>
         </searchRecord>
       </search>',
       IncludeParentColumns='False'
   )

    
   

   NetSuite Search API Example (Mixing BasicSearch / AdvancedSearch with JOIN)

9. Copy your Final Query you like to use to import NetSuite data in Power BI
10. Click OK to save DSN

Import NetSuite data in Power BI

Now lets perform the final step to import our NetSuite Query to Power BI. Assume that we want to import Contacts in Power BI to report them by demographics or some other criteria.

1. Open Power BI Desktop
2. Click on Get Data > More Data… > Other > ODBC
   

   Import SOAP API in Power BI (Using ZappySys XML ODBC Driver)

3. Select your ODBC DSN Source from the dropdown (Assuming you have created a ODBC Data Source using ZappySys XML Driver – See the previous Section)
4. Expand Advanced Options to enter custom Query.
5. Enter your SOAP API call SQL query here. For example purpose, we will use below query to get all account using NetSuite Search Web service.
   

   SELECT * FROM $
   WITH(
   	FirstPageBodyPart='    <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:AccountSearch">
           <q1:basic>
           <!--
             <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">x</searchValue>
             </description>
             <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">a</searchValue>
             </name>
             -->
           </q1:basic>
         </searchRecord>
       </search>',
       IncludeParentColumns='False'
   )

    
   

   Import NetSuite SOAP Web Service Data in Power BI (ZappySys XML Driver)

6. Once you see data preview click OK to import.
7. After data import is done you can edit your dataset (e.g. remove unwanted columns)
8. You can also edit Source query after data is imported (See below)
   

   Edit Power BI Data Source SQL after REST / SOAP Web Service Import

9. That’s it. You can now create your dashboards using NetSuite data. See below example.
   

   Import NetSuite data in Power BI dashboard

How to debug Web Requests using Fiddler

There will be a time you like to see RAW HTTP Requests generated by driver for NetSuite SOAP API.  If you like to do that check this article.

How to call other NetSuite SOAP API

So far we have seen only search API to query data. But there are many other SOAP APIs which can be consumed using approach listed in this article. Check this article to learn more how to call any SOAP API using ZappySys XML Driver. It shows techniques to use tool called SoapUI. You can use NetSuite WSDL file from here copy link listed on that page and use it to import in SoapUI.

Conclusion

Consuming data from NetSuite API or any XML / SOAP based API in your Reporting / ETL apps can be tricky. But we have seen how to solve this challenging task using ZappySys XML Driver in few clicks. Download ODBC PowerPack and explore many other API integration scenarios not discussed in this article.

The post How to read NetSuite data in Power BI appeared first on ZappySys Blog.

In this article, we will learn how to import LinkedIn data in Power BI. For that, we will create a successful OAuth Connection then request for reading LinkedIn Profile data and loading into Power BI

Power BI is a very popular Business Analytic tool used to get business information. It is very popular because it is easy to install, simple to learn and very intuitive. Also, REST API is very popular these days and we wanted to show you a way to integrate them in Power BI with the help of the ZappySys ODBC PowerPack. This software includes powerful drivers to query REST API URL, Local XML / JSON files and XML SOAP Web Service with simple SQL queries.

In nutshell, this post will focus on how to call LinkedIn API using Zappysys ODBC PowerPack. So let’s get started

Prerequisite

1. First, you will need to have our ZappySys ODBC Power Pack
2. In the second place, you will need to have internet access
3. On third place, Power BI Desktop (FREE) installed

Getting Started

In order to start, we will show several examples. ZappySys ODBC PowerPack includes the ZappySys JSON Driver that will help you to call LinkedIn API, retrieve basic profile data from LinkedIn and load profile data in Power BI. To learn more about LinkedIn API check this help file.

Call LinkedIn API using ZappySys JSON Driver

To access LinkedIn data using REST API call you need to create a LinkedIn app. See this article, it has mentioned the steps for How to register an App in LinkedIn. once you registered OAuth app in the LinkedIn website we can move to ODBC DataSources(DSN) Part.

1. Double-click on the ODBC Data Sources App.
   

   Open ODBC Data Sources from Startup menu

2. Click on Add button in User DSN Tab and then select JSON Driver from the driver list and click on Finish button to Create a new JSON Driver.
   

   ODBC User DSN Tab: Add new Driver Screen

3. Enter API URL in Properties Tab. Here is the sample Example:
   

   https://api.linkedin.com/v1/people/~:(id,firstName,lastName,headline,num-connections)?format=json

   Select a type of Connection from the Connection drop-down. here for this call, we have to select OAuth Connection.
   

   ZappySys ODBC JSON Driver: Configurations settings for LinkedIn API

4. Configure new OAuth Connection for LinkedIn as below.
   Basically, you need to enter 3 pieces on the General tab (i.e. ClientId, ClientSecret, Scopes) and one more info on the Advanced tab (i.e. callback URL ). If you are not sure what is Scope then check this LinkedIn API documentation. For each API page, you will see which permission needed. Just include the desired scope on ZappySys OAuth Connection (see below).
   

   OAuth Connection Manager: Configure the settings for LinkedIn

   also, Enter your Callback URL in Advanced Tab.

   

   OAuth Connection Manager: Configurations settings for LinkedIn in Advanced Tab

5. After successfully establishing the Connection with LinkedIn, you can Click on Test Connection button.
   

   ODBC JSON Driver: Test Connection for LinkedIn API Request

6. now, let’s go to Preview tab and click on the Preview data
   

   ODBC JSON Driver: Preview Screen of LinkedIn API Request

7. And that’s it, Now we will move forward to the next Phase i.e To Load data in Power BI

Load data into Power BI

In the preview section, we configured and added the ZappySys drivers in the ODBC Driver Administrator with information to connect to LinkedIn REST API. We queried the REST API data in JSON format. Now we will open Power BI and import the information from ODBC.

1. Open Power BI Desktop and select the option Get data
   
   

   Power BI: Click on Get Data for Loading from ODBC DSN

2. Get Data will allow adding the ZappySys ODBC driver. Go to Other and select ODBC.
   
   

   Select ODBC from Get Data Popup in Power BI

3. Select ODBC DSN name from the DSN dropdown, and click on OK button for Import data
   

   Get data from ODBC Driver

4. Select data to import and click OK (Below screenshot is for the Table mode import when no SQL specified. You can pick the desired table to import)
   

   see the Preview of Linked In data

5. See the Imported data in table visualization mode. you can select any visualizations as per your need.
   

   Power BI Imported LinkedIn data in Table Visualization

6. That’s All 🙂 you have successfully imported the LinkedIn data in Power BI dashboard.

Conclusion

In this blog, we have learned how to retrieve the basic profile data from LinkedIn using ODBC JSON Driver and then Load those data in Power BI with ODBC DSN in a very simple way. You can achieve many more functionalities with the use of ODBC PowerPack. Check our blogs/articles on How to import REST API in Power BI (Load JSON / SOAP XML) to find out what these ODBC Drivers are capable of more.

How to import REST API in Power BI (Load JSON / SOAP XML)

References

Finally, you can use the following links for more information about the use of LinkedIn API with our ODBC PowerPack:

• LinkedIn Documentation for API
• https://zappysys.com/blog/howto-import-json-rest-api-power-bi/

The post How to Import LinkedIn data in Power BI appeared first on ZappySys Blog.

If you are looking around to find connectivity options to get Amazon AWS data in Power BI (e.g. Read from Amazon S3 files (CSV, JSON, XML) or get AWS API data such as Billing Data by calling REST API) then unfortunately as of now Power BI doesn’t support it natively.

In this article, you will learn  How to read Amazon S3 data in Power BI (Using S3 CSV Driver) and in later section we will also learn How to import data from AWS API Calls (e.g. Billing, EC2, S3, DynamoDB). We will use ZappySys REST / JSON Driver to access AWS data without any ETL or Programming.

In our previous article, we discussed how to access REST API data in Power BI (Read JSON / XML Web API). Now let’s learn how to connect to AWS in Power BI.

In this article, We will explore interesting scenarios such as below. It will help you to connect Amazon AWS in Power BI and import data from AWS without doing any ETL or programming.

• Import JSON file from S3 bucket in Power BI (Using Amazon S3 Driver for JSON Files).
• Read CSV file from S3 bucket in Power BI (Using Amazon S3 Driver for CSV Files).
• Read XML file from S3 bucket in Power BI (Using Amazon S3 Driver for XML Files).
• Call Amazon AWS REST API (JSON or XML) and get data in Power BI. Some examples of API calls.
  • Get EC2 VM count and their Status in Power BI Dashboard
  • Read data from AWS Athena Service
  • Use AWS Cost / Billing API to display monthly Cost by Service
  • Display summary of Redshift Clusters
  • Extract response from Lambda function and display in Power BI

So let’s get started…

Requirements

1. First, you will need to have Power BI Desktop (FREE) installed
2. Make sure you have AWS Access Key and Secret Key to access AWS data or call API for the desired service
3. On third place, our ZappySys ODBC Power Pack

Import Amazon S3 data in Power BI (CSV, JSON or XML Files)

In this section we will look at step by step approach to load Amazon S3 data in Power BI. Your file can be compressed  (GZip, Zip) or un-compressed. We will use ZappySys Amazon S3 driver for CSV Files for demo but you can also use Amazon S3 JSON Driver or Amazon S3 Driver for XML Files same way for reading different file formats.

Configure ODBC DSN for Amazon S3 Files ( JSON / XML / CSV )

1. Type odbcad32.exe in your search box and launch the DSN Config utility.
2. you want access for yourself then stay of User DSN Tab. If you want grant access other users then go to System DSN tab. For SQL Server Integration go to System Tab and add new System DSN rather than User DSN. Click New button.
   
3. From the Driver list Select ZappySys ODBC Driver. For this example select [ZappySys Amazon S3 CSV Driver].
   
4. Now, we need Amazon S3 Storage Connection. Lets create it.
   
5. Now, When you see DSN Config Editor with zappysys logo first thing you need to do is change default DSN Name at the top and Select your bucket and file from it.
   
6. Here, in the Compression (Zip/GZip) tab set suitable file format.
   
7. Click on Preview Tab, Select Table from Tables Dropdown and select [value] and click Preview.
   
8. If you are using DSN method then Click OK to save our DSN Configuration.

Load Amazon S3 Data in Power BI – using ODBC Driver Connection

In the previous section, we configured ODBC DSN for AWS API call and added the ZappySys drivers in the ODBC Driver Administrator with information to connect to AWS S3 Files. We queried S3 File using ZappySys Amazon S3 Driver for CSV Files. Now we will open Power BI and import the information from ODBC  DSN we defined earlier.

1. Open Power BI Desktop and select the option Get data
   

   Get data in power bi

2. Get Data will allow adding the ZappySys ODBC driver. Go to Other and select ODBC.
   

   Select odbc in Power BI

3. Select ODBC DSN name from the DSN dropdown
4. Now its time to import data. Basically, there are two modes to import data. Table Mode and Query Mode. We will Import using Power BI Query Mode
5. Select your DSN and click Advanced Option to enter custom SQL Query to Import your REST API data. You can use ODBC DSN Data sources Preview tool to generate SQL Query. For example, you can enter a query like below. WITH clause is optional but you can supply many options to override DSN UI settings (e.g. Path, Credentials ).
   

   /*--------- Amazon S3 Driver - Read CSV File Example - Override UI credentials in WITH clause ----------*/  
   SELECT * FROM $  
   WITH (  
     Src='Northwind-Bucket/Orders*.csv' --use wildcard * for pattern search
     -- ,AccessKey='AKIAxxxx' --not needed if you set in DSN connection
     -- ,SecretKey='a1wxxxxxxxxxxxxxx' --not needed if you set in DSN connection
     -- ,RegionName='us-west-1'
     -- ,ColumnDelimiter='|'
     -- ,HasColumnHeaderRow='False'
   )

   
6. On the next screen select Windows Authentication to continue (No userid and password).
   

   Select Windows Credentials to connect to zappysys driver

7. Click OK to import data in Power BI
8. Now you can create custom Dashboard from imported Dataset.

AWS API Example –  Call Amazon S3 API in Power BI

In our previous section we saw how to read Amazon S3 data using native ZappySys S3 Drivers (For CSV , JSON and XML Files), this approach is preferred way to read S3 Files for sure because it gives you UI to browse files, it gives you ability to read from multiple files and many more S3 specific features.

However in some cases you have to call REST API in a manual way using REST API driver rather than S3 Driver. In next section we will see more interesting use case of calling AWS API (e.g. Billing / Cost API).

Now lets look at steps – How to import Amazon S3 data Power BI by calling REST API manually. Steps listed below are almost identical for XML or JSON data format except for selecting Driver Type (i.e. JSON, XML, CSV).

Configure ODBC DSN for REST API ( JSON / XML / CSV )

To consume S3 data in Power BI, first we have to create ODBC DSN using ZappySys Driver (XML or JSON). Perform the following steps.

1. To do this, we will first open the ODBC Data Source (32 bit):
   

   Open odbc data source

2. Use the User DSN page and press Add
   

   Add new data source

3. Add the ZappySys XML Driver if you are accessing XML files from S3 Bucket or calling any AWS API which returns data in XML format. Select JSON driver for JSON data (or API Access in JSON Format)
   

   Create DSN using ZappySys ODBC Driver for XML / SOAP API

4. It is time to connect with Amazon S3 File and read data. In Data Source (URL or File Path), we will use XML file URL as below. Notice that S3 URL has 3 parts (zs-dump1 is bucket name, s3.us-east-1.amazonaws.com is service endpoint for S3 (some service doesn’t require region) and store_001.xml is our file name. After you specify URL select Connection as per the screenshot.Syntax for URL :  BucketName + ServiceURL + FilePath.If you are not sure what can be your service endpoint then check this table.
   

   https://zs-dump1.s3.amazonaws.com/store_001.xml
   --OR--
   https://zs-dump1.s3.amazonaws.com/mysubfolder/store_001.xml
   --OR--
   https://zs-dump1.s3.us-east-1.amazonaws.com/mysubfolder/store_001.xml

   

   Configure Amazon S3 Connection – Read from XML File or JSON File

   NOTE: If a filename contains a space, e.g. “store abc.xml“, make sure to replace space with “%20” (“+” will not work). So “store abc.xml” should become “store%20abc.xml“.
5. Once you specify the URL and Credentials, Next select Filter (This helps to flatten the nested hierarchy of your Data). If you don’t have nested Hierarchy then skip this step.
   

   Select Filter (Flatten the hierarchy to extract data from nested structure)

    

6. Now go to Preview Tab and click Preview button in the toolbar. You can remove attributes in the query to use default attributes from Previous Tab. You can always override DSN level setting inside WITH clause of SQL.
   

   Preview data from AWS S3 File (XML Data) – Use ZappySys XML Driver to Connect to AWS API

7. To learn more about query language click on View Examples
8. Click OK to save your ODBC DSN settings.

Import AWS S3 File or AWS API data in Power BI dataset

Now let’s look at how to import Amazon S3 data in Power BI using ZappySys XML driver.

In the previous section, we configured ODBC DSN for AWS API call and added the ZappySys drivers in the ODBC Driver Administrator with information to connect to AWS REST API. We queried S3 File using ZappySys XML Driver. Now we will open Power BI and import the information from ZappySys JSON Driver via ODBC connectivity option in Power BI.

1. Open Power BI Desktop and select the option Get data
   

   Get data in power bi

2. Get Data will allow adding the ZappySys ODBC driver. Go to Other and select ODBC.
   

   Select odbc in Power BI

3. Select ODBC DSN name from the DSN dropdown
4. Now its time to import data. Basically, there are two modes to import data. Table Mode and Query Mode. We will Import using Power BI Query Mode
5. Select your DSN and click Advanced Option to enter custom SQL Query to Import your REST API data. You can use ODBC DSN Data sources Preview tool to generate SQL Query. For example, you can enter a query like below.
   

   SELECT "author", "title",  "genre", "price",  "publish_date", "description", "tag"
   FROM $
   WITH(
   	 Src='https://zs-dump1.s3.amazonaws.com/store_001.xml'
   	,Filter='$.store.book[*]'
   	,ElementsToTreatAsArray='book'
   )

   

   Import JSON / REST API data in Power BI using SQL Query Mode

    

6. On the next screen select Windows Authentication to continue (No userid and password).
   

   Select Windows Credentials to connect to zappysys driver

7. Click OK to import data in Power BI
8. Now you can create custom Dashboard from imported Dataset.

AWS API Example – Import Amazon AWS Cost / Billing data in Power BI

So far you have seen how to connect to S3 File. Now let’s look at another interesting example to call AWS API and display information in Power BI dashboard. For example purpose, we are going to call this AWS Billing and Cost Analysis API. Perform following steps.

1. Create new DSN using Zappysys JSON Driver
2. Configure ODBC DSN as below
3. Enter URL as we discussed in previous section. For example to call Billing API Service for East-1 Region we need to enter following way.
   

   https://ce.us-east-1.amazonaws.com

4. Select Connection Type as OAuth and click Configure Link.
5. Select Provider as AWS API v4 and enter AWS Key and Secret. Click OK to save the connection.
6. Select Method as POST
7. Enter HTTP Request Headers as below (Note each Billing Cost API action will have different Header value for X-Amz-Target so refer API documentation
   

   Configure AWS API Connection for Cost / Billing API Data – Use ZappySys JSON Driver

8. Enter Request Body (Modify your report parameters). As you see in below request we are requesting billing data for 3 services for Jan-1-2018 till Jun-15-2018. If you dont know service name then simply refer to your Bill and copy Service Name in Values array.
   

   {
     "TimePeriod": {
       "Start":"2018-01-01",
       "End": "2018-06-15"
     },
     "Granularity": "MONTHLY",
     "Filter": {      
       "Dimensions": {
         "Key": "SERVICE",
         "Values": [
           "Amazon Simple Storage Service",
           "Amazon DynamoDB",
           "Amazon Redshift",
           "AWS Key Management Service"
         ]
       }
     },
     "GroupBy":[
       {
         "Type":"DIMENSION",
         "Key":"SERVICE"
       },
       {
         "Type":"TAG",
         "Key":"Environment"
       }
     ],
      "Metrics":["BlendedCost", "UnblendedCost", "UsageQuantity"]
   }

    
9. Select Filter as $.ResultsByTime[*].Groups[*]
10. Click Test Connection button see it shows OK.
11. If the connection is OK then switch to Preview Tab to test query.
12. You can select Table name from the drop-down or type query as below
    Simple-Query to Extract AWS Billing / Cost Information
    

    SELECT  
     "p_ResultsByTime_TimePeriod_Start" as BillStartDate,
     "p_ResultsByTime_TimePeriod_End" as BillEndDate,
     json_array_first("Keys") as ServiceName,
     "Metrics_BlendedCost_Amount" as BillAmount,
     "Metrics_UsageQuantity_Amount" as Quantity
    FROM [ResultsByTime_x_Groups] 
    ORDER BY p_ResultsByTime_TimePeriod_Start

13. You can also override attributes in your query using below syntax
    Custom-Query to Extract AWS Billing / Cost Information (WITH clause)
    

    SELECT 
     "p_ResultsByTime_TimePeriod_Start" as BillStartDate,
     "p_ResultsByTime_TimePeriod_End" as BillEndDate,
     json_array_first("Keys") as ServiceName,
     "Metrics_BlendedCost_Amount" as BillAmount,
     "Metrics_UsageQuantity_Amount" as Quantity
    FROM $
    ORDER BY p_ResultsByTime_TimePeriod_Start
    WITH(
    	 Src='https://ce.us-east-1.amazonaws.com'
    	,DataConnectionType='OAuth'
    	,ScopeSeparator='{space}'
    	,ServiceProvider='AmazonAWS'
    	--,ClientId='AKIAxxxxxxxxxxxxxxx'
    	--,ClientSecret='lPi+XQxxxxxxxxxxxxxxxx'
    	,UseCustomApp='True'
    	,Filter='$.ResultsByTime[*].Groups[*]'
    	,RequestData='{
      "TimePeriod": {
        "Start":"2018-01-01",
        "End": "2018-06-15"
      },
      "Granularity": "MONTHLY",
      "Filter": {      
        "Dimensions": {
          "Key": "SERVICE",
          "Values": [
            "Amazon Simple Storage Service",
            "Amazon DynamoDB",
            "Amazon Redshift",
            "AWS Key Management Service"
          ]
        }
      },
      "GroupBy":[
        {
          "Type":"DIMENSION",
          "Key":"SERVICE"
        },
        {
          "Type":"TAG",
          "Key":"Environment"
        }
      ],
       "Metrics":["BlendedCost", "UnblendedCost", "UsageQuantity"]
    }'
    	,Header='X-Amz-Target: AWSInsightsIndexService.GetCostAndUsage || Content-Type: application/x-amz-json-1.1'
    	,RequestMethod='POST'
    )

14. Click Preview button to see the result.
    

    Query AWS Cost / Billing Data using API in Power BI (Use ZappySys JSON Driver)

15. Click OK to save Your DSN
16. Now you can Import Billing / Cost data in Power BI using same steps as previous section (Except use custom Query and DSN we just created for AWS API call)
    

    Import AWS Billing / Cost Data in Power BI (Call AWS REST API)

17. Repeat steps to import more datasets using different queries

AWS API Example – Import Data From AWS Lambda in Power BI

Now let’s look at another scenario. Assume that someone from other team has written a Lambda Function which exposes certain data in JSON format which you want to display that in your Dashboard. Refer to this Lambda API documentation to learn more about API syntax.

Here is an example query to Call Lambda Function (submit input data in JSON and receive data in JSON format)

SELECT * FROM $
WITH(
	 Src='https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/HelloWorld/invocations'
	,RequestMethod='POST'
	,RequestData='{"id":1}'
	,RequestContentTypeCode='ApplicationJson'
	
	/*** Uncomment to override DSN settings	  
	,DataConnectionType='OAuth'
	,ScopeSeparator='{space}'
	,ServiceProvider='AmazonAWS'
	--,ClientId='AKxxxxxxxxxxx'
	--,ClientSecret='lPi+Xxxxxxxxxxxxxxxxxxxxxxx'
	,UseCustomApp='True'
	***/
)

AWS API Example – Get EC2 VM information in Power BI

Now let’s look at AWS EC2 Service API (Virtual Machine Service). This API is in XML format so you have to use ZappySys XML Driver.  Below query calls this EC2 API to get list of all instances.

SELECT * FROM $
WITH(
	 ElementsToTreatAsArray='item'
	,Src='https://ec2.us-east-1.amazonaws.com/'
	,Filter='$.DescribeInstancesResponse.reservationSet.item[*].instancesSet.item[*]'
	,RequestData='Action=DescribeInstances&Version=2016-11-15'
	,RequestMethod='POST'
	/*
	,DataConnectionType='OAuth'
	,ScopeSeparator='{space}'
	,ServiceProvider='AmazonAWS'
	,ClientId='AKIAxxxxxxxxxxxxxxxxxxx'
	,ClientSecret='lPi+XQvxxxxxxxxxxxxxxxxxxxxxxxxx'
	,UseCustomApp='True'
	*/
)

Get EC2 Instance List in Power BI (Use ZappySys XML Driver)

AWS API Example – Get AWS S3 Buckets and Files Count / Size in Power BI

Now let’s look at how to Query S3 File System in Power BI. For example if you have to prepare dashboard to show Total Data Size and File count in a given Bucket then you can write following Query. For more information check Bucket List API Help

Get File Size and Count for a given S3 Bucket

SELECT count(*) as TotalFiles,sum(Size)/1024/1024 as TotalMB  FROM $
WITH(
	 ElementsToTreatAsArray='Contents'
	,Src='https://zs-dump1.s3.amazonaws.com/?list-type=2&prefix=&max-keys=1000'
	,Filter='$.ListBucketResult.Contents[*]'
	,NextUrlAttributeOrExpr='$.ListBucketResult.NextContinuationToken'
	,NextUrlSuffix='&continuation-token=<%nextlink_encoded%>'
	
	/**** Uncomment below to use override DSN values
	,DataConnectionType='OAuth'
	,ScopeSeparator='{space}'
	,ServiceProvider='AmazonAWS'
	,ClientId='AKIAxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
	,ClientSecret='lPi+Xcxxxxxxxxxxxxxxxxxxxxxxxx'
	,UseCustomApp='True'	
	,RequestMethod='GET'
	***/
)

To list buckets you can use following query

List S3 Buckets

SELECT *  FROM $
WITH(	 
	 Src='https://s3.amazonaws.com'
	,Filter='$.ListAllMyBucketsResult.Buckets.Bucket[*]'
	,ElementsToTreatAsArray='Buckets'

	/**** Uncomment below to use override DSN values
	,DataConnectionType='OAuth'
	,ScopeSeparator='{space}'
	,ServiceProvider='AmazonAWS'
	,ClientId='AKIAxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
	,ClientSecret='lPi+Xcxxxxxxxxxxxxxxxxxxxxxxxx'
	,UseCustomApp='True'	
	,RequestMethod='GET'
	***/
)

Debugging / Crafting AWS API Call Request

Check below article to learn various techniques to debug and craft AWS API Request Body / Headers and URL.

How to call Amazon AWS API using SSIS (EC2, Lambda, API Gateway, SQS)

Understanding AWS REST API Pagination in Power BI

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

Handling AWS Web API Error in Power BI

Sometimes errors occur... they just do and there is nothing you can do! Or can you? Actually, in ODBC PowerPack you can handle them in two ways.

METHOD 1 - Using Error Handling Options

When to use?

You may want to use them when your source is a resource located on the Internet; e.g. a file on a website, a file on an FTP server or just a plain API HTTP response. By default, when a remote server returns an error, data retrieval is stopped, an error is raised and no data is given back to you. This might not be always desirable.

Scenario 1

Imagine a scenario, that there is a web server which each day at 12 AM releases a new JSON file with that day's date as filename, e.g. http://www.some-server.com/data/2018-06-20.json. And, of course, you want to download it and use it daily in your Power BI report. But you have a problem: Power BI report data sources are refreshed each hour and you may get HTTP 404 status code (no file was found) when a file is not released yet. Which consequentially means other data sources won't be updated as well and you will see old and cached data on the report. That's where you could use Continue on any error or Continue when Url is invalid or missing (404 Errors) to avoid an error being raised and let other data sources to be updated.

Scenario 2

Another scenario is when you expect a web server to raise some kind of HTTP error when accessing a URL. You don't want ODBC Data Source to raise an error but instead, you want to get response data. That's where you can use Continue on any error or alike together with  Get response data on error to continue on an error and get the data:

METHOD 2 - Using Connection [Retry Settings]

Another scenario you may run into is a buggy web server. You ask it to give you some file or data and it, like a snotty kid, just doesn't give it to you! You have to ask twice or thrice before it does its job. If that's the case, you have to retry HTTP requests using Connection:

Other Considerations for Calling AWS API in Power BI

Conclusion

In this article, we show how to connect to AWS API and S3 File data in Power BI. We configured ODBC DSN for AWS S3 / REST API connection and finally imported data in Power BI.  We used ZappySys driver for JSON that can be used to extract data from any REST API or a JSON file.  Download ODBC PowerPack to try yourself see how easy it is to consume XML / JSON / REST API and AWS data in Power BI and Say goodbye to ETL or Coding.  If you need any help with your API integration feel free to contact zappysys support you will be amazed for sure how quickly your issue is resolved.

The post Read Amazon S3 data in Power BI or Call AWS REST API (JSON / XML) appeared first on ZappySys Blog.

In our previous post, we saw in general how to call REST API in Power BI. Now let’s look at how to call SOAP API in Power BI (e.g. WorkDay, Google AdWords services). SOAP API is in XML format so we will be using ZappySys XML Driver for a demonstration in this article.

To learn how to automate dataset refresh on a scheduled basis you can again refer to the previous post because that won’t be covered in this article.

So let’s get started !!!

Requirements

1. First, you will need to have Power BI Desktop (FREE) installed
2. Make sure you have ZappySys ODBC Power Pack installed

Creating SQL Queries to call SOAP API in Power BI (Using XML Driver)

Very first step to call SOAP API in Power BI is to understand how to craft SOAP Requests using ZappySys XML Driver. Once you play with sample SOAP queries you can copy/paste that into Power BI.

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

Import XML SOAP Web Service in Power BI (Using XML Driver)

Once you know how to craft SQL Queries for SOAP API calls, and you save created DSN as per the previous section we are ready to load data into Power BI from XML Web Service.

So let’s get started.

1. First launch Power BI and Click on Get Data >> More Data >> Other >> ODBC
   

   Import SOAP API in Power BI (Using ZappySys XML ODBC Driver)

2. Select your ODBC DSN Source from the dropdown (Assuming you have created a ODBC Data Source using ZappySys XML Driver – See the previous Section)
3. Expand Advanced Options to enter custom Query.
4. Enter your SOAP API call SQL query here. For example purpose, we will use below query to try demo SOAP web service.
   

   SELECT * FROM $
   WITH(
   	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
   	,DataConnectionType='HTTP'
   	,CredentialType='Basic' --OR SoapWss
   	,SoapWssPasswordType='PasswordText'
   	,UserName='myuser'
   	,Password='pass$$w123'
   	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
   	,ElementsToTreatAsArray='HolidayCode'	
   	,RequestMethod='POST'	
   	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
   	,RequestData='
   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
      <soapenv:Header/>
      <soapenv:Body>
         <hol:GetHolidaysAvailable>
            <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
            <hol:countryCode>UnitedStates</hol:countryCode>
         </hol:GetHolidaysAvailable>
      </soapenv:Body>
   </soapenv:Envelope>'
   )

   

   Enter custom SQL to load SOAP Web Service Data in Power BI (ZappySys XML Driver)

5. Once you see data preview click OK to import.
6. After data import is done you can edit your dataset (e.g. remove unwanted columns)
7. You can also edit Source query after data is imported (See below)
   

   Edit Power BI Data Source SQL after SOAP Web Service Import

8. That’s it. Use your dataset to build Power BI dashboard and when you ready to publish just hit Publish Button.

Using Parameters in Power BI (Dynamic Query)

let
    vKey=paraAPIKey,
    Source = Odbc.Query(
"dsn=ZS - OData Customers", 
"SELECT * FROM $ WITH (SRC='http://httpbin.org/post'," 
& "METHOD='POST'," 
& "HEADER='Content-Type:application/json'," 
& "BODY=@'{""CallerId"":1111, ""ApiKey"":""" & vKey & """}')")
in
    Source

SOAP XML / REST API pagination in Power BI

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

SOAP XML / REST API Error Handling in Power BI

Sometimes errors occur... they just do and there is nothing you can do! Or can you? Actually, in ODBC PowerPack you can handle them in two ways.

METHOD 1 - Using Error Handling Options

When to use?

You may want to use them when your source is a resource located on the Internet; e.g. a file on a website, a file on an FTP server or just a plain API HTTP response. By default, when a remote server returns an error, data retrieval is stopped, an error is raised and no data is given back to you. This might not be always desirable.

Scenario 1

Imagine a scenario, that there is a web server which each day at 12 AM releases a new JSON file with that day's date as filename, e.g. http://www.some-server.com/data/2018-06-20.json. And, of course, you want to download it and use it daily in your Power BI report. But you have a problem: Power BI report data sources are refreshed each hour and you may get HTTP 404 status code (no file was found) when a file is not released yet. Which consequentially means other data sources won't be updated as well and you will see old and cached data on the report. That's where you could use Continue on any error or Continue when Url is invalid or missing (404 Errors) to avoid an error being raised and let other data sources to be updated.

Scenario 2

Another scenario is when you expect a web server to raise some kind of HTTP error when accessing a URL. You don't want ODBC Data Source to raise an error but instead, you want to get response data. That's where you can use Continue on any error or alike together with  Get response data on error to continue on an error and get the data:

METHOD 2 - Using Connection [Retry Settings]

Another scenario you may run into is a buggy web server. You ask it to give you some file or data and it, like a snotty kid, just doesn't give it to you! You have to ask twice or thrice before it does its job. If that's the case, you have to retry HTTP requests using Connection:

Security / Connection Types (Options for HTTP, OAuth, SOAP) in Power BI

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Other Considerations for Calling Web API in Power BI

Conclusion

In this article, we explored possibilities to load SOAP API data in Power BI without any coding or ETL approach. You can build powerful dashboards and reports by consuming data from any REST / XML SOAP API services using ZappySys Drivers. You can download ZappySys ODBC Drivers and try yourself to learn many other features not covered in this article.

The post Calling SOAP API in Power BI (Read XML Web Service data) appeared first on ZappySys Blog.

In this article, we will learn how to import REST API in Power BI.

Power BI is a very popular Business Analytic tool used to get business information. It is very popular because it is easy to install, simple to learn and very intuitive. Also, REST API is very popular these days and we wanted to show you a way to integrate them in Power BI with the help of the ZappySys ODBC PowerPack. This software includes powerful drivers to query REST API URL, Local XML / JSON files and XML SOAP Web Service with simple SQL queries.

The following video will show how to import REST API in Power BI (JSON or XML Data):
[youtube https://www.youtube.com/watch?v=-3OPfhMY1aI&w=720&h=405]

Requirements

1. First, you will need to have Power BI Desktop (FREE) installed
2. In the second place, you will need to have internet access
3. On third place, our ZappySys ODBC Power Pack

Step-by-Step: Import REST API into Power BI

We’ll walk through the steps to import a REST API into Power BI. The steps outlined below focus on working with JSON APIs, but the same principles can be applied when importing SOAP XML Web Services or local JSON/XML files.

For demonstration purposes, we will utilize a REST API to retrieve data in JSON format. The example URL we’ll use is:

https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json

This URL leverages OData and presents information in JSON format. Our objective is to use the ZappySys ODBC Power Pack to establish a connection to this URL and retrieve the information seamlessly into Power BI.

Create ODBC DSN – JSON Driver

DRIVER={DRIVER-NAME-GOES-HERE}[;Prop1=xxxxxx][;Prop2=xxxxxx] .... [;PropN=xxxxxx]

DRIVER={ZappySys JSON Driver}
DRIVER={ZappySys JSON Driver};DataPath='c:\mydata.json'
DRIVER={ZappySys XML Driver};DataPath='http://myserver/api/xml/getOrders'
DRIVER={ZappySys XML Driver};DataPath='c:\mydata.xml'

DRIVER={DRIVER-NAME-GOES-HERE}[;Prop1=xxxxxx][;Prop2=xxxxxx] .... [;PropN=xxxxxx]

DSN=MyJsonAPI_DSN
DSN=MyJsonAPI_DSN;DataPath='http://myserver/api/json/getOrders'

Connect to REST API data source in Power BI (Connect JSON / XML data)

Publish Power BI dashboard

1. In order to publish the report, press Publish
2. Press Save
   

   Save changes in Power BI:

    

3. Select a workspace for the report
   

   Select Power BI workplace.

4. And that is all. You have now your report ready.

Import data using ODBC ConnectionString (DSN-less)

Let’s explore the process of importing data using the ODBC Driver without the need for configuring an ODBC DSN.

When importing data, you have the option to either use an ODBC DSN or a Direct ConnectionString. This section will guide you through the steps of utilizing the ODBC ConnectionString.

After opting for the ODBC option to retrieve data, you will encounter the “Select ODBC Data Source” screen. To proceed without a DSN, follow these steps:

1. Change the DSN dropdown to “(none).”
2. Enter the full ODBC ConnectionString, adhering to the syntax below. You can construct your own connection string using the ODBC Driver UI and utilize the “Copy ConnectionString” option.
   Syntax: 

   Driver={Your Driver Name}[;Property1=value][;Property2=value]......

3. Here is a screenshot with ConnectionString Setting
   

   Import Data from ODBC Driver – DSN less mode (Use Connection String)

4. Click “Next” to select the authentication method. Choose “Windows” and input the password value if necessary. If your connection string already includes the Password attribute, there’s no need to worry.
   

   Import Data from ODBC Driver – using Connection String – With Password Attribute

POST data to REST API URL

Up until now, we’ve explored the process of fetching data from URLs and importing it into Power BI. However, in certain scenarios, APIs may necessitate the use of the PUT or POST method. These methods involve submitting parameters in the body and receiving a response.

For a more in-depth understanding of how to seamlessly POST data to a URL in Power BI, watch the informative video below.

Calling XML SOAP Web Service in Power BI

So far, we have looked at examples to consume data using JSON driver. Now let’s look at an example, to call XML SOAP Web Service in Power BI.

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

SOAP / REST API pagination in Power BI

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

SOAP / REST API Error Handling in Power BI

Sometimes errors occur... they just do and there is nothing you can do! Or can you? Actually, in ODBC PowerPack you can handle them in two ways.

METHOD 1 - Using Error Handling Options

When to use?

You may want to use them when your source is a resource located on the Internet; e.g. a file on a website, a file on an FTP server or just a plain API HTTP response. By default, when a remote server returns an error, data retrieval is stopped, an error is raised and no data is given back to you. This might not be always desirable.

Scenario 1

Imagine a scenario, that there is a web server which each day at 12 AM releases a new JSON file with that day's date as filename, e.g. http://www.some-server.com/data/2018-06-20.json. And, of course, you want to download it and use it daily in your Power BI report. But you have a problem: Power BI report data sources are refreshed each hour and you may get HTTP 404 status code (no file was found) when a file is not released yet. Which consequentially means other data sources won't be updated as well and you will see old and cached data on the report. That's where you could use Continue on any error or Continue when Url is invalid or missing (404 Errors) to avoid an error being raised and let other data sources to be updated.

Scenario 2

Another scenario is when you expect a web server to raise some kind of HTTP error when accessing a URL. You don't want ODBC Data Source to raise an error but instead, you want to get response data. That's where you can use Continue on any error or alike together with  Get response data on error to continue on an error and get the data:

METHOD 2 - Using Connection [Retry Settings]

Another scenario you may run into is a buggy web server. You ask it to give you some file or data and it, like a snotty kid, just doesn't give it to you! You have to ask twice or thrice before it does its job. If that's the case, you have to retry HTTP requests using Connection:

Security / Connection Types (Options for HTTP, OAuth, SOAP) in Power BI

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Import data into Power BI from SQL Query

If you prefer to import data from an SQL query instead of selecting a table name, you can leverage advanced options during the import steps. Here’s a step-by-step guide:

1. After selecting the DSN, proceed to the import steps.
2. Click on the “Advanced Options” to access the SQL Query editor.

By utilizing these advanced options, you gain the flexibility to tailor your import process by specifying custom SQL queries to retrieve the exact data you need.

Import data into Power BI using SQL Query (ODBC Data source)

Edit Query / Using Parameters in Power BI (Dynamic Query)

let
    vKey=paraAPIKey,
    Source = Odbc.Query(
"dsn=ZS - OData Customers", 
"SELECT * FROM $ WITH (SRC='http://httpbin.org/post'," 
& "METHOD='POST'," 
& "HEADER='Content-Type:application/json'," 
& "BODY=@'{""CallerId"":1111, ""ApiKey"":""" & vKey & """}')")
in
    Source

Edit Query Settings after Import

There will be a time you need to change initial Query after dataset import in Power BI. Not to worry, just follow these steps to edit your SQL.

Edit Power BI Data Source SQL after REST / SOAP Web Service Import

Performance Tips for REST API / XML SOAP Calls

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Conclusion

In this article, we guide you through the process of configuring the ZappySys Power Pack, focusing on the ZappySys driver for JSON. This driver is instrumental in extracting data from REST APIs or JSON files. Follow these steps to seamlessly add the extracted data to Power BI and generate insightful reports.

Additionally, we’ll walk you through the configuration of the Power BI gateway, ensuring that the data is refreshed on a specified schedule for up-to-date and accurate reporting. To try this configuration yourself, download ODBC PowerPack and embark on a journey of efficient data extraction and reporting.

References

• What is Power BI?
• Keep your dashboards and reports up-to-date with your on-premises data sources
• Power BI – Read REST API / JSON File / XML File / SOAP (Pagination, OAuth, OData)

The post How to import REST API in Power BI (Load JSON / SOAP XML) appeared first on ZappySys Blog.