In the previous article, we learned how to get data from AdWords in SSIS and basically control your AdWords account programmatically, outside the dashboard. In this one, we will continue on how to download AdWords report by using Google Ads API, ODBC, and ZappySys XML Driver. Let’s not waste our time and cut to the chase!

Prerequisites

1. Google Ads account up and running.
2. ZappySys ODBC PowerPack installed.

Step-by-Step – Download AdWords report

Get Developer Token

The first important step to access any Google Ads Account is to request a developer token. Click this link for step by step information. Watch a short video to get an overview of the entire process to access data via API.

Get Client Customer Id

Login into a non-manager account at https://ads.google.com and get your Client Customer Id:

Getting Google Ads (formerly AdWords) Client Customer Id

Copy it to a notepad, we will use it later.

Open ODBC Data Sources Manager

Search for "odbc" in your start menu and click on ODBC (64 bits). If you can't find this then you can also go to Start Menu > ZappySys > ODBC PowerPack > Click on ODBC Data Sources (64-Bit).

Open ODBC Data Source

Create ODBC Data Source based on ZappySys XML Driver

Select tab User DSN if you want to create a data source only for your user, or select System DSN to create a data source for all users or for OS services (e.g. SQL Server Agent):

Creating ZappySys XML Driver based data source

Configure authentication in the ODBC data source

Let’s authenticate first:

1. Select OAuth in Connection Type property.
2. Click Click to Configure link.
3. In OAuth Provider property select Google.
4. Then either select Use Default OAuth App or Use Custom OAuth App. For the latter, you will need to create a Google OAuth App yourself. It’s perfectly safe to use the default application, but if your company policy does not allow it, you may consider creating it yourself.
5. In Scopes enter:
   https://www.googleapis.com/auth/adwords
6. Then hit Generate Token and hit allow once asked.

Download AdWords report – Configuring authentication in ODBC data source

Proceed with HTTP request configuration

Then go on with configuring the data source:

Configuring HTTP request in ODBC data source to download an AdWords report

1. Give your data source a name
2. Then input this URL to download the report:
   https://adwords.google.com/api/adwords/reportdownload/v201809
   If the above URL does not work, get the working one from Google’s documentation.
3. Input POST as HTTP Request Method.
4. In Body enter this text:
   __rdxml=<reportDefinition xmlns="https://adwords.google.com/api/adwords/cm/v201809">
<selector>
<fields>CampaignId</fields>
<fields>Impressions</fields>
<fields>Clicks</fields>
<fields>Cost</fields>
<predicates>
<field>CampaignStatus</field>
<operator>IN</operator>
<values>ENABLED</values>
<values>PAUSED</values>
</predicates>
</selector>
<reportName>My Campaign Performance Report</reportName>
<reportType>CAMPAIGN_PERFORMANCE_REPORT</reportType>
<dateRangeType>LAST_7_DAYS</dateRangeType>
<downloadFormat>XML</downloadFormat>
</reportDefinition>
   This will display the data of CAMPAIGN_PERFORMANCE_REPORT report (campaign Id, impressions, clicks, and cost) for the last 7 days, for all campaigns that are enabled or paused. Make sure to visit the Google documentation page, which contains the information about how to download a different kind of report, how to configure predicates, etc.
5. In Body Content Type property select Form (application/x-www-form-urlencoded) option.
6. In HTTP Headers enter your developerToken and clientCustomerId retrieved in previous steps.

Select data filter in ODBC data source

Then hit Select Filter button and select row node, so that Array Filter is set to $.report.table.row[*]

Filtering on AdWords report data

If asked whether you want to treat the row element as an array, click “yes”. You can also go to the Xml Array Handling tab and see if the element row is specified there.

The Results

To view the results press Preview tab and then click Preview Data:

Previewing AdWords report data

Now you can use this data source and get the data into an Excel sheet, Power BI report, SQL Server (would need to create a data source in ZappySys Data Gateway instead) or elsewhere.

Reusing the data source

If your client application supports, you can override data source settings, such as HTTP body, in SQL query using WITH clause. In that case, you won’t need to create multiple data sources for different kind of reports; e.g.:

SELECT * FROM $ (RequestData='@c:\files\report-definition.txt')

SELECT * FROM $ (RequestData='__rdxml=<reportDefinition xmlns="https://adwords.google.com/api/adwords/cm/v201809">
<selector>
<fields>CampaignId</fields>
<fields>Cost</fields>
</selector>
<reportName>My Campaign Performance Report</reportName>
<reportType>CAMPAIGN_PERFORMANCE_REPORT</reportType>
<dateRangeType>LAST_7_DAYS</dateRangeType>
<downloadFormat>XML</downloadFormat>
</reportDefinition>')

Build a SQL query using the Query Builder:

Use Query Builder to construct SQL query and override data source properties

Conclusion

In this article, we learned how to download AdWords report using ODBC PowerPack and Google API. We created an ODBC data source based on XML Driver, then configured the OAuth authentication, HTTP request and data filtering without a single line of code. Then we previewed AdWords “Campaign Performance Report” data and finally, considered how to reuse the data source we created. Visit https://zappysys.com/blog/category/odbc-powerpack/odbc-app-integration/ for examples of how to integrate data sources based on ZappySys Drivers into applications.

References

https://developers.google.com/adwords/api/docs/guides/reporting

https://developers.google.com/adwords/api/docs/appendix/reports

https://developers.google.com/adwords/api/docs/appendix/reports/campaign-performance-report

The post Download AdWords report using Google Ads API, ODBC, and XML Driver appeared first on ZappySys Blog.

In our previous post we saw various ways to transform JSON arrays. However there will be a time when your JSON / XML file wont have Array and you need to Pivot JSON Data.

Sample JSON data file

Here is a sample JSON file which we like to parse into rows and columns. Notice how it is using values inside in property name. This pattern needs Pivot option usage described later in this article.

{
	"version": 1.0,
	"products": {
		"P001": {
			"name": "Product 1",
			"price": 10
		},
		"P002": {
			"name": "Product 2",
			"price": 11
		},
		"P003": {
			"name": "Product 3",
			"price": 12
		}
	}
}

If you like to play with real data then use below URL. This JSON has 60MB worth of data for all Products offered on AWS.  And It has similar structure as above except many more attributes under products node.

https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/us-east-1/index.json

Using SSIS PowerPack to Pivot JSON data

ZappySys offers powerful JSON and XML Connectors. Below section will describe how to Parse and Pivot JSON data or Pivot XML data. For demo purpose we will use JSON Source but steps are almost same for XML too.

Step-By-Step JSON Pivot

1. First drag data flow from SSIS Toolbox and then double click data flow
   

   Drag SSIS Data Flow Task from Toolbox

2. Once Data flow designer is open, drag ZS JSON Source from SSIS Toolbox.
   

   Drag ZappySys JSON Source

3. Now for demo we will use Hard coded JSON using Direct Value option as below. You can also enter URL or File path if you select Direct Path option.
4. Go to Pivot Columns to Rows Tab and check Enable Pivoting Option
   

   Pivot JSON / XML Data option for ZappySys API Connectors / Drivers

5. Now come back to Filter Options Tab here you can either browse Filter or enter by hand. If file is too large then enter by hand. For example in our case we will enter

   $.products

     because we like to to parse structure below that node. If you have nested hierarchy (i.e. products node under orders node and so on) then use dot to separate them e.g. 

   $.customer.orders.products

6. For very Large file you need to Un-check Include Parent Columns option else you will get OutOfMemory Exception
7. Now click Preview to see our data.
   

   Pivot JSON Data using SSIS JSON Source

    

8. Now you can connect your JSON / XML source to destination. See below example.
   

   SSIS Package Execution –
   Loading Google BigQuery Data into SQL Server

Using ODBC PowerPack to Pivot JSON data

So far we talked how to use SSIS to read your JSON data and Pivot it but what if you want to consume JSON / XML / REST API data in some other App without doing ETL via SSIS?  Thats where ZappySys ODBC PowerPack comes in picture. Its a collection of many drivers including JSON Driver and XML Driver.  You can also use Data Gateway Approach, which allows T-SQL code to fetch JSON / XML data directly into SQL Server Table without any ETL / Programming.

Here is sample query you can run using JSON Driver or XML Driver

SELECT * FROM $
WITH(
	 Src='https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/us-east-1/index.json'
	,Filter='$.products'
	,IncludeParentColumns='False'
	,EnablePivot='True'
)

See below screenshot of JSON Driver Configuration to get idea.
Note: Use above query rather than what is displayed in the Screenshot.

Using ODBC DSN in C# code to call REST API

Preview / Generate Query in JSON Driver / XML Driver

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Advanced Pivot with Path / Search and Replace Option

Now let’s look at an advanced scenario where you like to extract Pivot_Path (know where Pivoted Property came from). For that, you can download the latest version which providers IncludePivotPath and EnablePivotPathSearchReplace options as below.

As you can see in the below example let’s say you like to extract table names and view names if defined under “views” node.

{
  "table-1": {
      "views": {
        "view-tbl1-1": {},
        "view-tbl1-2": {}
    }
  },
  "table-2": {
      "views": {}
  },
  "table-3": {
      "views": {
        "view-tbl3-1": {}
    }
  }
}

We set following

• On the Filter Options Tab
  Filter = 

  $..views

• on the Pivot Columns To Rows Tab
  Check Enable Pivoting
  Check Include_PivotPath
  Check Enable Pivot_Path search/replace
  Set Search for

  (.*).views.(.*)--regex

  Set Replace With

  $1

Advanced Pivot Option – Search and Replace Path

Conclusion

In this particle we saw how to use some advanced options for XML/ JSON Source and ODBC Drivers. You can download respective product depending your usecase. For SSIS usecase Download SSIS PowerPack and for other ETL / Reporting app integration scenarios (i.e. Power BI, Informatica, SSRS , Excel, MS Access, SQL Server, JAVA, C# ….) Download ODBC PowerPack.

The post Pivot JSON and XML data using SSIS or ODBC Drivers appeared first on ZappySys Blog.

In our previous blog we saw how to read JIRA data in SQL Server. Now let’s learn how to Import SAP S/4HANA OData Service Data Into MS Access. SAP S/4HANA provides OData REST API interface to access data in your application using HTTP Protocol. We will use ODBC XML Driver to read SAP data and load into MS Access.

So, let’s get started.

About SAP HANA / OData REST API Service

You can expose your SAP Data using ODATA REST API Service. Here is a good article which shows how to expose data as OData Service.   For more information on SAP OData Service feature check this SAP help page. If you are new to OData Standard then read here to know more how OData can facilitate data extraction using HTTP REST API. If you need Sample XML based OData Service then use below test URLs.

https://services.odata.org/Northwind/Northwind.svc/
https://services.odata.org/Northwind/Northwind.svc/Customers
https://services.odata.org/Northwind/Northwind.svc/Orders
https://services.odata.org/Northwind/Northwind.svc/Invoices
https://services.odata.org/Northwind/Northwind.svc/Products

Now let’s look at how to read SAP Data using ODBC Driver. At this point we assume you have exposed your data as OData Endpoint.

Read SAP S/4HANA data using XML Driver

First let’s look at steps to configure XML Driver. We will read data from SAP S/4HANA OData Service and then in next section we will look at how to load data into MS Access or other target.

1. First, Windows search, write ODBC and select the ODBC Data sources (32 bits)
   

   Open ODBC Data Source

2. As a second step, in ODBC Data source Administrator press the Add button.
   

   Add ZappySys

3. In this step, create the new data source, select ZappySys XML Driver.
   

   Add new zappysys xml driver

4. Now edit that XML data source to configure it. Enter your OData Service URL its typically like below. Replace 3 parts with your own value (i.e. replace {MY-INSTANCE},  {MY-PROJECT}, {MY-TABLE})
   

   https://{MY-INSTANCE}/sap/opu/odata/sap/{MY-PROJECT}/{MY-TABLE}

   For example if you are hosting SAP HANA in Cloud Instance then your URL may look like below
   

   https://myXXXXXX-api.s4hana.ondemand.com/sap/opu/odata/sap/MyTestProject/PurchaseOrders

   And create New ZS-HTTP connection in it select Basic Authentication and enter your SAP HANA UserID / Password to call OData Service and select the desire filter.
   

   XML Driver : SAP S/4HANA OData Service Configuration

5. Now go to Data Format / Compression (Zip/GZip) tab and select Data Format as OData to get all the records.
   

   Data Format : Odata

6. Finally, now using Query Builder and Code Generator we will generate the query and click on Preview the data.
   

   Generate Query

7. That’s it we are ready to load SAP S/4HANA OData Service data to MS Access.

Load SAP S/4HANA OData Service in MS Access

1. In MS Access, go to External Data Ribbon and select New Data Source and select From Other Sources and ODBC Database.
   

   A configuration in MS Access: Import Data From REST API

2. Select the Source and destination of the data, select Import the source data into a new table in the current database and press OK.
   

   Get External Data – ODBC Database

3. Select your newly created JSON Driver Data Source and click OK.
   
   

   Get External Data – Select your newly created ODBC Data Source (JSON Driver)

    

4. Select tables from the list of tables in the Import Objects screen and click OK.
   
   

   Get External Data – Select tables from Import Objects Screen

5. Select Save Import Steps from the screen and click on Save Import button.
   

   Get External Data – Select Save Import Steps

6. That’s it If everything is OK, you will be able to see the data.
   

   Data Imported Successfully in MS Access DB Table

How to link REST API data to Access

The linking process is very similar to importing data into the Access table. Follow steps of the previous paragraph – How to import REST API data to Access – but on step 2 select the second option to create a linked table instead. Then at last it’s asked to select a unique record identifier, don’t select anything:

Linked table approach – Don’t select a unique Record identifier

SAP S/4HANA Integration with Other BI Tools

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

So in this blog, we learned how to Import SAP S/4HANA OData Service Data in Access using ODBC XML / SOAP API Driver in a very simple way. You can achieve many more functionalities with this tool. Check our blogs/articles on XML File / REST API Driver to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the use of SAP S/4HANA OData Service with our tools:

• SAP API Business Hub
• Landing Page for ODBC XML / SOAP API Driver, you can also find Tutorial Video here.
• Help File: Documentation of XML Driver.
• Blog/articles link: https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/xml-soap-api-driver/

The post Import SAP S/4HANA OData Service Data Into Ms Access via ODBC Driver appeared first on ZappySys Blog.

In our previous blog we saw how to read JIRA data in SQL Server. Now let’s learn how to Import SAP S/4HANA OData Service Data Into Sql Server. SAP S/4HANA provides OData REST API interface to access data in your application using HTTP Protocol. We will use ODBC XML Driver to read SAP data and load into SQL Server.

So, let’s get started.

About SAP HANA / OData REST API Service

You can expose your SAP Data using ODATA REST API Service. Here is a good article which shows how to expose data as OData Service.   For more information on SAP OData Service feature check this SAP help page. If you are new to OData Standard then read here to know more how OData can facilitate data extraction using HTTP REST API. If you need Sample XML based OData Service then use below test URLs.

https://services.odata.org/Northwind/Northwind.svc/
https://services.odata.org/Northwind/Northwind.svc/Customers
https://services.odata.org/Northwind/Northwind.svc/Orders
https://services.odata.org/Northwind/Northwind.svc/Invoices
https://services.odata.org/Northwind/Northwind.svc/Products

Now let’s look at how to read SAP Data using ODBC Driver. At this point we assume you have exposed your data as OData Endpoint.

Read SAP S/4HANA data using XML Driver

First let’s look at steps to configure XML Driver. We will read data from SAP S/4HANA OData Service and then in next section we will look at how to load data into SQL Server or other target.

1. To do this, first of all, we will open the ZappySys Data Gateway Configuration:
   

   Open ZappySys Data Gateway

2. Add the Native – ZappySys XML Driver Data source.
   

   Add Gateway Data Source

3. Now edit that XML data source to configure it. Enter your OData Service URL its typically like below. Replace 3 parts with your own value (i.e. replace {MY-INSTANCE},  {MY-PROJECT}, {MY-TABLE})
   

   https://{MY-INSTANCE}/sap/opu/odata/sap/{MY-PROJECT}/{MY-TABLE}

   For example if you are hosting SAP HANA in Cloud Instance then your URL may look like below
   

   https://myXXXXXX-api.s4hana.ondemand.com/sap/opu/odata/sap/MyTestProject/PurchaseOrders

   And create New ZS-HTTP connection in it select Basic Authentication and enter your SAP HANA UserID / Password to call OData Service and select the desire filter.
   

   XML Driver : SAP S/4HANA OData Service Configuration

4. Now go to Data Format / Compression (Zip/GZip) tab and select Data Format as OData to get all the records.
   

   Data Format : Odata

5. Finally, now using Query Builder and Code Generator we will generate the query.
   

   Generate Query

6. That’s it we are ready to load SAP S/4HANA OData Service data to SQL Server.

Load SAP S/4HANA OData Service in MS SQL Server

SAP S/4HANA Integration with Other BI Tools (Power BI, Excel, SSRS, MS Access…)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

So in this blog, we learned how to Import SAP S/4HANA OData Service Data in MS SQL Server using ODBC XML / SOAP API Driver in a very simple way. You can achieve many more functionalities with this tool. Check our blogs/articles on XML File / REST API Driver https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/xml-soap-api-driver/ to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the use of SAP S/4HANA OData Service with our tools:

• SAP API Business Hub
• Landing Page for ODBC XML / SOAP API Driver, you can also find Tutorial Video here.
• Help File: Documentation of XML Driver.
• Blog/articles link: https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/xml-soap-api-driver/

The post Import SAP S/4HANA OData Service Data Into Sql Server via ODBC Driver appeared first on ZappySys Blog.

In this article, we will see how to read Walmart API data in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call Walmart APIs can be useful for any developer regardless of which programming language or toolset you use.

Are you a Walmart.com Supplier or Marketplace Seller? Then you may need to click here to visit the site that is focused on these profiles.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Walmart API. Click here to create API.

Components Mentioned in this article

	ZS Logging Task
	JSON Source(REST API or File)

What is Walmart?

Walmart is the world’s largest retailer, and the Walmart Open API provides access to our extensive product catalog, thus enabling digital distribution partners to earn substantial affiliate revenues from customer referrals.

Read Walmart API Data using SSIS JSON Source

Let’s start with an example. We use SSIS JSON Source component to make the call to Walmart API, we will read Walmart data and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

Get Dynamic Token (Configure ZS-HTTP Connection)

To get API response data using SOAP / REST API call, you need to get an API Token from your login request URL. Follow the steps mentioned below to get API Token:

1. Now, Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
   

   Create a New HTTP Connection.

2. Select ZS HTTP Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
   

   Select ZS-HTTP Connection from SSIS Connection Manager.

3. In the SSIS ZS HTTP Connection Manager, Go to General Tab and Configure HTTP Connection Manager. Enter API URL, Set Credentials Type to Dynamic Token – Two-Step Auth, and configure some basic settings for the same. We used Login URL as just example but you can enter any valid API URL endpoint here. Actual Login URL goes on Dynamic Token Tab (see next step).
   

   https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20

   WM_SEC.ACCESS_TOKEN

   

   ZS HTTP Connection – Configure General Tab

4. Now, In the Dynamic Token Tab do configure like below image. Please click here to refer about Walmart API Token.
   

   https://marketplace.walmartapis.com/v3/token

   grant_type=client_credentials

   Accept: application/json
   Cache-Control: no-cache
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>
   Authorization: Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>

   

   ZS HTTP Connection – Dynamic Token Tab

5. On the Response Settings tab configure as below
   

   Walmart API – Extract Token – Response Settings

   1. Extract Mode = Extract from Body
   2. Expression Type = Json
   3. Response Content Filter =

      $.access_token

6. Click on OK button to save HTTP Connection configure settings.

That’s it we have successfully configured Connection for Walmart API in SSIS. In the next section, we will see how to use this connection and read various data from Walmart API.

Read Walmart API Data in SSIS

Once we have done creating HTTP Connection Manager we can move forward to read Walmart API data inside Data Flow. So lets Configure SSIS JSON / REST API Source.

1. Now, Drag and drop Data Flow Task from SSIS Toolbox in the currently open project and double click it to edit.
   
   

   Drag and Drop Data Flow Task.

2. Double click on the Data Flow task to see the Data Flow designer surface.
3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   

   Drag and Drop JSON Source(REST API or File).

4. Double click JSON Source and enter the following URL as below.
   

   https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20

5. Check Use Credentials and select existing HTTP connection we have created already it.
6. Enter Following HTTP Headers. If you prefer Raw edit .. Just click on Raw Edit (#6 in screenshot) and paste below headers. Make sure to replace YourClientIdGoesHere and YourSecretGoesHere
   

   Content-Type: application/xml
   Accept: application/xml
   Authorization: Basic <<YourClientIdGoesHere:YourSecretGoesHere,FUN_BASE64ENC>>
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>

    
7. Select Array Filter (ItemResponse node) or type $.ItemResponse[*]  as below.
   

   ZS JSON Source – Configure

8. That’s it, you are ready to load Wallmart API Data into SQL Server and you can use more ZappySys Destination components.
9. For pagination call please click here for reference to this link.

Load Walmart API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Walmart API integration using ODBC Driver (Power BI, Excel, MS Access)

So far we have seen how to access Walmart API using SSIS. However there will be a time when you like to access API data inside other Apps using ODBC Driver. ZappySys Provides JSON/XML/CSV drivers for virtually any API including Walmart API. Here is how to access Walmart API using XML ODBC Driver.

1. Type “ODBC” in Start menu. Open ODBC Data sources (64 bit)
   
2. Click Add on User DSN tab (if you need to use DSN under different Account …i.e. Service Account then click on System DSN Tab)
3. Select ZappySys XML Driver
4. When New DSN UI shows up, Change DSN name to anything you like (e.g. WalmartAPI)
5. Click on Load ConnectionString button and enter below connection. Make sure to replace few things
   First ######YourClientId####### with your own clientid
   Second ######YourSecret####### with your own secret
   Third ######Base64OfClientIdAndSecret####### with base64 encoded string of

   clientId:Secret

       (both separated with colon). You can use any site like this one to generate base64 string of your clientid and secret (seperated by “:”)
   

   DRIVER={ZappySys XML Driver};
   ElementsToTreatAsArray='ns2:ItemResponse';
   DataPath='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20';
   DataConnectionType=HTTP;
   AuthScheme='{none}';
   Url='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20';
   TokenUrl='https://marketplace.walmartapis.com/v3/token';
   TokenRequestData='grant_type=client_credentials';
   TokenRequestMethod='POST';
   TokenResponseContentFilter='$.access_token';
   TokenRequestHeaders='Accept:application/json||Cache-Control:no-cache||WM_SVC.NAME:Walmart Marketplace||WM_QOS.CORRELATION_ID:TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>||Authorization:Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>';
   TokenAuthHeader='WM_SEC.ACCESS_TOKEN';
   TokenResponseContentType=Json;
   UserName='######YourClientId#######';
   CredentialType=TokenDynamic;
   Password='######YourSecret#######';
   Filter='$.ns2:ItemResponses.ns2:ItemResponse[*]';
   RequestMethod='GET';
   RequestHeaders='Content-Type: application/xml || Accept: application/xml || Authorization: Basic ######Base64OfClientIdAndSecret####### || WM_SVC.NAME: Walmart Marketplace || WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>'

   

   ZappySys Driver – Load Connection String

    

6. Now go to preview tab and enter below query and click Run below SQL.

   select * from $

   –OR–
   NOTE: Change YourClientId and YourSecret in below sql before running

   SELECT * FROM $
   WITH(
   Src='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20'
   ,Header='Content-Type: application/xml
   Accept: application/xml
   Authorization: Basic <<YourClientid:YourSecret,FUN_BASE64ENC>>
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>'
   )

    

Walmart API Integration with Other BI Tools (Power BI, Excel, SSRS, MS Access…)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

How to encrypt Authorization Token as Base64Encode

1. If you want to directly pass Authorization header Token as Base64Encode(clientId:clientSecret) encrypted in the get token call, you can encrypted it using ZS Logging task and used it. Please refer to this link for the same.
2. In visual studio just Drag and Drop ZS Logging Task in the design panel.
   

   Drag and Drop ZS Logging Task

3. Double click on ZS Logging Task to configure it.
4. In the ZS Logging Task configure like the following image.
   

   Logging Task : Base64Enocde

5. That’s all, Save Base64Encoded token for use in HTTP Connection.

Conclusion

After all, we saw you how to extract information from REST API such as Walmart REST API using HTTP Connection and load into SQL Server. We also learned techniques like How to get Dynamic Token using HTTP Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File), ZS Logging Task
• Walmart API: HTTP Connection

The post How to read Walmart API data in SSIS / ODBC appeared first on ZappySys Blog.

In our previous blog post we saw how to import REST / SOAP API in SQL Server. Using same concepts let’s look at how to import Bing Ads data into SQL Server. We will explore many techniques to call Bing Ads API and learn how to automate data extraction without doing any ETL. You can call Bing Ads API just using T-SQL code (Yes you heard it right). At the end of this article you will learn how to Download Performance Reports from Bing Ads Account without any coding (See sample screenshot below). Please go through full article carefully.

NOTE: Bing Ads now renamed as Microsoft Advertising but this article will use popular name for time being.

Import Bing Ads data into SQL Server (Using SSMS T-SQL Code)

About Bing Ads API / SOAP Web Service

If you are new to Bing Ads API then start from here. Bing Ads APIs are SOAP XML APIs. You can call it to automate many scenarios for Bing Ads. In this article we will mainly focus on read scenario (e.g. Read Performance Data) but you can use these techniques to write / update (e.g. Create New Campaign).

If you are new to SOAP WebService then Check this article to learn more. It will explain you how to use tools like SoapUI tool to craft SOAP Requests for Bing API or any other SOAP API.

For Bing API you can use Service WSDL files found here. Use URL ending ?wsdl to import in SoapUI.

Example Bing API Request

Here is raw API request for Bing API. This sample assumes you have completed 3-Legged OAuth Authorization to obtain AccessToken and RefreshToken (Explained later in this article). Below request uses AccessToken you might have already obtained via process like this. If you are using ZappySys tools then this will be generated automatically for you.

Request Method / URL:

POST https://clientcenter.api.bingads.microsoft.com/Api/CustomerManagement/v13/CustomerManagementService.svc

Headers:
Notice that for each Api you may need correct SOAPAction in Header. If you are not sure then use SoapUI Tool. Execute request in SoapUI and then go to Raw tab like this one.

Content-Type: text/xml; charset=utf-8
SOAPAction: "GetUser"

Body:
Notice that you have to replace AuthenticationToken and DeveloperToken in below request. AccessToken is placed inside AuthenticationToken tag. This Auth token is short lived and extracted via OAuth Process and DeveloperToken you can extract as mentioned in the previous section or get it from here.

  
    GetUser
    <%access_token%>
    1052Bxxxxxxxxxxxxxx
  
  
    
      
    
  

Response:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Header>
    <h:TrackingId xmlns:h="https://bingads.microsoft.com/Customer/v13">fddf3cb9-e4c2-4e75-9546-f3bc20ea91a8</h:TrackingId>
  </s:Header>
  <s:Body>
    <GetUserResponse xmlns="https://bingads.microsoft.com/Customer/v13">
      <User xmlns:a="https://bingads.microsoft.com/Customer/v13/Entities" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
        <a:ContactInfo>
          <a:Address>
            <a:City i:nil="true" />
            <a:CountryCode>US</a:CountryCode>
            <a:Id i:nil="true" />
            <a:Line1 i:nil="true" />
            <a:Line2 i:nil="true" />
            <a:Line3 i:nil="true" />
            <a:Line4 i:nil="true" />
            <a:PostalCode i:nil="true" />
            <a:StateOrProvince i:nil="true" />
            <a:TimeStamp i:nil="true" />
            <a:BusinessName i:nil="true" />
          </a:Address>
          <a:ContactByPhone>false</a:ContactByPhone>
          <a:ContactByPostalMail>false</a:ContactByPostalMail>
          <a:Email>someone@zappysys.com</a:Email>
          <a:EmailFormat i:nil="true" />
          <a:Fax i:nil="true" />
          <a:HomePhone i:nil="true" />
          <a:Id>48045678</a:Id>
          <a:Mobile i:nil="true" />
          <a:Phone1>111-222-3333</a:Phone1>
          <a:Phone2 i:nil="true" />
        </a:ContactInfo>
        <a:CustomerId>19112345</a:CustomerId>
        <a:Id>48012345</a:Id>
        <a:JobTitle i:nil="true" />
        <a:LastModifiedByUserId>48012345</a:LastModifiedByUserId>
        <a:LastModifiedTime>2019-06-04T14:22:35.38</a:LastModifiedTime>
        <a:Lcid>EnglishUS</a:Lcid>
        <a:Name>
          <a:FirstName>Someone</a:FirstName>
          <a:LastName>Good</a:LastName>
          <a:MiddleInitial i:nil="true" />
        </a:Name>
        <a:Password i:nil="true" />
        <a:SecretAnswer i:nil="true" />
        <a:SecretQuestion>None</a:SecretQuestion>
        <a:UserLifeCycleStatus>Active</a:UserLifeCycleStatus>
        <a:TimeStamp>AAAAAJiX/cU=</a:TimeStamp>
        <a:UserName>someone@zappysys.com</a:UserName>
        <a:ForwardCompatibilityMap i:nil="true" xmlns:b="http://schemas.datacontract.org/2004/07/System.Collections.Generic" />
      </User>
      <CustomerRoles xmlns:a="https://bingads.microsoft.com/Customer/v13/Entities" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
        <a:CustomerRole>
          <a:RoleId>41</a:RoleId>
          <a:CustomerId>19160000</a:CustomerId>
          <a:AccountIds i:nil="true" xmlns:b="http://schemas.microsoft.com/2003/10/Serialization/Arrays" />
          <a:LinkedAccountIds xmlns:b="http://schemas.microsoft.com/2003/10/Serialization/Arrays" />
          <a:CustomerLinkPermission i:nil="true" />
        </a:CustomerRole>
      </CustomerRoles>
    </GetUserResponse>
  </s:Body>
</s:Envelope>

POST http://myhost/api/v2/customers/new
X-SecretKey: Abcdxxxxxxxxxxxxxx
Content-Type: application/json

{ firstname: "Someone", lastname: "Good" }

{
 request_status: "CreatedOK",
 new_customer_id: "1234566"
}

Step-By-Step: How to Import Bing Ads Data into SQL Server

Now let’s look at how to import Bing Ads Data into SQL Server Table. This tutorial will require 10-15 mins of your time. At the end of this tutorial you will be able to Read Bing Ads Performance Report and Import data into SQL Server. Step listed in this tutorial may help you to call Bing API in other tools like SSIS, Power BI, SSRS, Tableau and so on.

So let’s get started.

Register OAuth App for Bing Ads API (Azure Active Directory App)

Very first thing to call any Bing API is to register OAuth App (Under Azure Active Directory Tab).

Here is how to register OAuth App which can be used to call Bing Ads API later on.

1. Login to Azure Portal:
   • Navigate to the Azure Portal and log in using your credentials.
2. Access Azure Active Directory:
   • In the left-hand menu, click on Azure Active Directory.
3. Register a New Application:
   • Go to App registrations and click on New registration.
     
   • Application Name: Enter a name for your application.
     
   • Supported Account Types: Choose the account types your app will support. For example, select Accounts in this organizational directory only if you need access to data in your organization only.
   • Redirect URI:
     • Set the type to Web.
     • In the textbox enter https://login.microsoftonline.com/common/oauth2/nativeclient as the Redirect URI or any other valid redirect URL, e.g., https://zappysys.com/oauth.
     • Use this Redirect URI in the Redirect URL grid row.
4. Save Client ID:
   • After registering the app, copy the Application (client) ID and paste it into the Client ID field in the API Connection Manager configuration.
5. Set Authorization & Token URLs:
   • Click on the Endpoints link in the App registration overview.
   • Authorization URL: Copy the OAuth 2.0 authorization endpoint (v2) URL (e.g., https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/authorize) and paste it into the Authorization URL field in the configuration grid.
   • Token URL: Copy the OAuth 2.0 token endpoint (v2) URL (e.g., https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/token) and paste it into the Token URL field.
6. Create a Client Secret:
   • In the Certificates & secrets tab, click New client secret.
   • Set an expiration period for the secret.
   • Copy the generated client secret and paste it into the Client Secret field in the API Connection Manager configuration.
7. Configure API Permissions:
   • Go to the API Permissions section.
   • Click on Add a permission, select Microsoft Graph, and choose Delegated Permissions.
   • Add the required permissions:
     • offline_access
     • openid
     • profile
     • Sites.Read.All
     • Sites.ReadWrite.All
     • User.Read
     • email
     

     Azure AD App – Required API Permissions

   • Grant Admin Consent for the permissions that require it.
8. Generate Tokens:
   • Use the Generate Token feature in the API Connection Manager to generate authentication tokens.
9. Use a Generic Account for Automation:
   NOTE: If you are planning to use your current data connection/token for automated processes, we recommend that you use a generic account for token generation when the login box appears (e.g. sales_automation@mycompany.com instead of bob_smith@mycompany.com). When you use a personal account which is tied to a specific employee profile and that employee leaves the company, the token may become invalid and any automated processes using that token will fail. Another potentially unwanted effect of using a personal token is incorrect logging; the API calls (e.g. Read, Edit, Delete, Upload) made with that token will record the specific user as performing the calls instead of an automated process.
10. That’s it!

Get Bing API Developer Token

Once we have OAuth App registered next thing is to obtain Developer Token for Bing Ads API. Click here to request new token or view existing Developer token.

Create new Bing Ads API Data Source in Gateway

Once you setup Data Gateway user we can move to next step. To call Bing API we need to setup Connection for XML Driver in ZappySys Data Gateway.

1. Click on Add New Data source. Name as BingAds and select Native – ZappySys XML Driver option.
   

   Add Gateway Data Source

2. Now click Edit to Configure Settings. For now we will only care OAuth Connection settings. All other options are overwritten in SQL Query.
   

   Edit Gateway Data Source Settings

3. When UI opens Click on Load ConnectionString and Paste below connection string after changing some items (Screenshot may be differ). Make sure to replace following attributes.RefreshTokenFilePath – Enter some valid file path which will hold refresh token
   ClientId – get Application Id from Azure Portal under App we created earlier)
   DeveloperToken – Replace 1052Bxxxxxxxxxxxx with your own devloper token.
   

   Load ZappySys Driver ConnectionString to configure UI

   Use below string and replace parameters 

   DRIVER={ZappySys XML Driver};
   DataPath='https://clientcenter.api.bingads.microsoft.com/Api/CustomerManagement/v13/CustomerManagementService.svc';
   DataConnectionType=OAuth;
   AuthUrl='https://login.microsoftonline.com/common/oauth2/v2.0/authorize';
   TokenUrl='https://login.microsoftonline.com/common/oauth2/v2.0/token';
   ReturnUrl='https://login.microsoftonline.com/common/oauth2/nativeclient';
   RefreshTokenFilePath='c:\xxxxxx_some_folder_xxxxxxx\BingAds_RefreshToken.txt';
   ScopeSeparator='{space}';
   Scope='https://ads.microsoft.com/ads.manage offline_access';
   ClientId='xxxxxxxxxxxxxx';
   UseCustomApp=True;
   RequestData='<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
     <s:Header>
       <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Customer/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Customer/v13"><%access_token%></h:AuthenticationToken>
       <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Customer/v13">1052Bxxxxxxxxxxxx</h:DeveloperToken>
       <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Customer/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Customer/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
     </s:Header>
     <s:Body>
       <GetUserRequest xmlns="https://bingads.microsoft.com/Customer/v13">
         <UserId i:nil="true" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       </GetUserRequest>
     </s:Body>
   </s:Envelope>';
   RequestContentTypeCode=TextXml;
   RequestMethod='POST';
   RequestHeaders='SOAPAction: "GetUser"'

4. Next to OAuth Click on Configure Settings
5. By default most of settings on OAuth connections should be pre-populated for you but still lets Notice few things. This is how to setup redirect URI
6. 

   Configure Redirect URI for Bing Ads API OAuth Connection

7. Here is where you setup Changing Refresh Token File path. if you dont supply this path then your refresh token extracted first time will expire in 60 days.
   

   Refresh Token File path (Only needed if API issues new Refresh Token when you request AccessToken)

8. Once you verify all settings click Generate Token and Finish the process. It will populate AccessToken and RefreshToken Fields if everything goes well.
   

   Configure Bing Ads API Connection (OAuth 2.0)

9. Click OK to save OAuth Connection UI to go back to main UI
10. You can now Click Test Connection to confirm everything
11. If you need to access Gateway from Other machine then go to Firewall tab and Click Add Rule
12. Click Save button on Gateway UI to save and restart the service.

Create a new Linked Server for Bing Ads API

Once we setup setup Bing Ads Data Source in Gateway we can move to next step which is create a Linked Server which points to Data gateway Data source we created. This will create a bridge between ZappySys Driver and SQL Server so you can call API via T-SQL.

1. If you are using Latest ODBC PowerPack then on Data gateway UI you will See App Integration Tab. Copy Code and run it in SSMS to create a new linked server.
   

   ZappySys Data Gateway code generator for Linked Server

2. If things go well when you execute code you may see some result in grid.

Run sample Queries to call Bing API in SQL Server / Import Data

Once Linked Server is created we can run various queries like below.

Example-1

--use all default settings from gateway data source UI
--Read
Select * from OPENQUERY( [YourLinkedServer] , 'select * from $') 

/*
--import
Select * into #tempTable from OPENQUERY( [YourLinkedServer] , 'select * from $') 
Select * from #tempTable
*/

Example-2

--use Filter to extract specific Node from response XML
Select * from OPENQUERY( [YourLinkedServer] , 
'select * from $
WITH(Filter=''$.s:Envelope.s:Body.GetUserRequest[*]'', 
  ElementsToTreatAsArray=''GetUserResponse'' 
   )'
)

Example-3

--Override URL, Headers and Body Settings. Change DeveloperToken in below request
--Header, Filter, URL, Request data is different for each Endpoint

SELECT * FROM OPENQUERY([YourLinkedServer]
, 'SELECT * FROM $
WITH(
	 ElementsToTreatAsArray=''GetUserResponse''
	,Src=''https://clientcenter.api.bingads.microsoft.com/Api/CustomerManagement/v13/CustomerManagementService.svc''
	,Filter=''$.s:Envelope.s:Body.GetUserResponse[*]''	
	,Header=''SOAPAction: "GetUser"''	
	,RequestData=''<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v13="https://bingads.microsoft.com/Customer/v13">
<soapenv:Header>
   <v13:DeveloperToken>1052xxxxxxxxxxxxx</v13:DeveloperToken>
   <v13:AuthenticationToken><%access_token%></v13:AuthenticationToken>
</soapenv:Header>
<soapenv:Body>
   <v13:GetUserRequest>
      <v13:UserId xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/>
   </v13:GetUserRequest>
</soapenv:Body>
</soapenv:Envelope>''
)')

Download Bing Ads Performance Data in SQL Server Table

Now lets look at more complex example to read data from Bing Performance Report. Calling This API is tricky due to many steps involved in data extraction. But not to worry if you use ZappySys Drivers. This is API is somewhat similar as Amazon MWS Report Request (See last section).

Here are high level steps needs to be performed to call API for Bing Ads Report (e.g. Performance Report)

1. Call SubmitGenerateReportResponse API to request new Report. This returns ReportRequestId
2. Now we have to keep checking every few seconds see report is ready or not. Call PollGenerateReport until Status field is Success from Pending
3. Once previous step returns Success extract Report Download URL (Its Zip File which may contain data in CSV or XML)

So lets get started to implement above login in a single T-SQL Query and load data into Table. There is a known limitation with Query Size we can send to data gateway from SSMS so we can use some tricks to read Body from File rather than hard coding in SQL. We can use IsMultiPart=True and then supply Filepath in RequestBody (Path must start with @ symbol)

1. Bing Report API needs 2 additional values along with DeveloperToken we got earlier. You need to supply AccountId and CustomerId part of Body so obtain it by visiting below page in your bing account.
   Visit https://ads.microsoft.com/cc/accounts
   

   Obtain Bing Ads AccountId, ClientId for APi calls

2. Create new XML file for first step in a Notepad. Enter below content and save as C:\Requests\BingAds\sql_body1.xml
   Replace Following Values as per your settingsCustomerAccountId (Found 2 places)
   CustomerId
   DeveloperToken
   

   <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
     <s:Header>
       <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Reporting/v13"><%access_token%></h:AuthenticationToken>
       <h:CustomerAccountId xmlns:h="https://bingads.microsoft.com/Reporting/v13">481xxxxxxxxxxx</h:CustomerAccountId>
       <h:CustomerId xmlns:h="https://bingads.microsoft.com/Reporting/v13">191xxxxxxxxxxx</h:CustomerId>
       <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Reporting/v13">1052Bxxxxxxxxxxx</h:DeveloperToken>
       <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
     </s:Header>
     <s:Body>
       <SubmitGenerateReportRequest xmlns="https://bingads.microsoft.com/Reporting/v13">
         <ReportRequest i:type="CampaignPerformanceReportRequest" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
           <ExcludeColumnHeaders>false</ExcludeColumnHeaders>
           <ExcludeReportFooter>false</ExcludeReportFooter>
           <ExcludeReportHeader>false</ExcludeReportHeader>
           <Format>Xml</Format>
           <ReportName>My Campaign Performance Report</ReportName>
           <ReturnOnlyCompleteData>false</ReturnOnlyCompleteData>
           <Aggregation>Hourly</Aggregation>
           <Columns>
             <CampaignPerformanceReportColumn>TimePeriod</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AccountId</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>CampaignId</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>CampaignName</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>CampaignStatus</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>DeviceType</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>BidMatchType</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>QualityScore</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AdRelevance</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>LandingPageExperience</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Revenue</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Assists</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>ExpectedCtr</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>DeliveredMatchType</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AveragePosition</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Conversions</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AdDistribution</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Network</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Clicks</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Impressions</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Ctr</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AverageCpc</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Spend</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>LowQualityClicks</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>LowQualityConversionRate</CampaignPerformanceReportColumn>
           </Columns>
           <Filter>
             <AccountStatus i:nil="true" />
             <AdDistribution i:nil="true" />
             <DeviceOS i:nil="true" />
             <DeviceType i:nil="true" />
             <Status i:nil="true" />
           </Filter>
           <Scope>
             <AccountIds xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
               <a:long>481xxxxxxxxxxx</a:long>
             </AccountIds>
             <Campaigns i:nil="true" />
           </Scope>
           <Time>
             <CustomDateRangeEnd i:nil="true" />
             <CustomDateRangeStart i:nil="true" />
             <PredefinedTime>Yesterday</PredefinedTime>
             <ReportTimeZone>PacificTimeUSCanadaTijuana</ReportTimeZone>
           </Time>
         </ReportRequest>
       </SubmitGenerateReportRequest>
     </s:Body>
   </s:Envelope>

    
3. Create another XML file for second step in a Notepad. Enter below content and save as C:\Requests\BingAds\sql_body2.xml
   Replace following values as per your need.
   

   <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
     <s:Header>
       <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Reporting/v13"><%access_token%></h:AuthenticationToken>
       <h:CustomerAccountId xmlns:h="https://bingads.microsoft.com/Reporting/v13">480xxxxxxxxxxx</h:CustomerAccountId>
       <h:CustomerId xmlns:h="https://bingads.microsoft.com/Reporting/v13">191xxxxxxxxxxx</h:CustomerId>
       <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Reporting/v13">1052Bxxxxxxxxxxx</h:DeveloperToken>
       <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
     </s:Header>
     <s:Body>
       <PollGenerateReportRequest xmlns="https://bingads.microsoft.com/Reporting/v13">
         <ReportRequestId>[$a.ReportRequestId$]</ReportRequestId>
       </PollGenerateReportRequest>
     </s:Body>
   </s:Envelope>

    
4. Now try to execute below query. This should generate Performance report and download it in a SQL table for you.
   

   select * into tmpBingAdsReport from OPENQUERY(Zs_BingAds,
   'SELECT c.* FROM $
   WITH(
   	--submit report request for XML format
   
   	 alias=''a'' 
   	,Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
   	,Filter=''$.s:Envelope.s:Body.SubmitGenerateReportResponse''
   	,RequestData=''@C:\Requests\BingAds\sql_body1.xml''
   	,IsMultiPart=''True''
   	,RequestContentTypeCode=''TextXml''
   	,Header=''SOAPAction: "SubmitGenerateReport"''
   	,RequestMethod=''POST''
   	,EnableBodyPlaceholderForMultiPart=''True''
   	,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''
   
   	--keep checking status until report is ready - returns final url when done
   
   	,join1_alias=''b''
   	,join1_Filter=''$.s:Envelope.s:Body.PollGenerateReportResponse.ReportRequestStatus''
   	,join1_RequestData=''@C:\Requests\BingAds\sql_body2.xml''
   	,join1_IsMultiPart=''True''
   	,join1_Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
   	,join1_RequestContentTypeCode=''TextXml''
   	,join1_Header=''SOAPAction: "PollGenerateReport"''
   	,join1_RequestMethod=''POST''
   	,join1_EnableBodyPlaceholderForMultiPart=''True''
   	,join1_EnableStatusCheck=''True'', 
   	,join1_StatucCheckMaxWaitSeconds=300, --wait max 5 mins?
   	,join1_StatucCheckIterationWaitSeconds=5, --check every 5 sec?
   	,join1_StatusSuccessValue=''Success'', --look for success word in response
   
   	--Download report (Zip file) and Parse rows
   
   	,join2_alias=''c''
   	,join2_DataConnectionType=''Default''
   	,join2_Src=''[$b.ReportDownloadUrl$]''
   	,join2_Filter=''$.Report.Table.Row[*]''
   	,join2_ElementsToTreatAsArray=''Row''	
   	,join2_RequestMethod=''GET''
   	,join2_IsMultiPart=''False''
   	,join2_FileCompressionType=''Zip''	
   )')
   
   select * from tmpBingAdsReport

   

   Import Bing Ads data into SQL Server

Multi Step Query to download Report

So in previous section we saw how to write a single query to get your Performance Report from Bing Ads. However there will be time you want to split 3 steps into seperate queries.

Here is how to do.

/*
RPC and RPC OUT options must be turned on to run this proc

EXEC master.dbo.sp_serveroption @server=N'Zs_BingAds', @optname=N'rpc', @optvalue=N'true'
EXEC master.dbo.sp_serveroption @server=N'Zs_BingAds', @optname=N'rpc out', @optvalue=N'true'
--Below needed to support EXEC + INSERT (dynamic query)
EXEC master.dbo.sp_serveroption @server=N'Zs_BingAds, @optname=N'remote proc transaction promotion', @optvalue=N'false'
Also 
1. Change Body file path
2. Change CustomerAccountId, CustomerId, DeveloperToken
*/

ALTER proc usp_Api_BingAds_Reports
as
--///////////////////////////////////////////////////////////////////////////////
--Step-1 - submit report request for XML format
--///////////////////////////////////////////////////////////////////////////////
create table #tmpBingAdsReport_Step1(ReportRequestId bigint)
INSERT into #tmpBingAdsReport_Step1
EXEC(
'SELECT * FROM $
WITH(
 Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,Filter=''$.s:Envelope.s:Body.SubmitGenerateReportResponse''
,RequestData=''@c:\BWProjects\Requests\BingAds\sql_body1.xml''
,IsMultiPart=''True''
,RequestContentTypeCode=''TextXml''
,Header=''SOAPAction: "SubmitGenerateReport"''
,RequestMethod=''POST''
,EnableBodyPlaceholderForMultiPart=''True''
,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''
)') AT zs_BingAds


--///////////////////////////////////////////////////////////////////////////////
--Step2 - keep checking status until report is ready - returns final url when done
--///////////////////////////////////////////////////////////////////////////////

declare @reportid varchar(100)
select top 1 @reportid=cast(ReportRequestId as varchar(100)) from #tmpBingAdsReport_Step1

create table #tmpBingAdsReport_Step2(ReportDownloadUrl varchar(500))
INSERT into #tmpBingAdsReport_Step2 
EXEC(
'SELECT * FROM $
WITH(
 Filter=''$.s:Envelope.s:Body.PollGenerateReportResponse.ReportRequestStatus''
,RequestData=''<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Header>
    <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
    <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Reporting/v13"><%access_token%></h:AuthenticationToken>
    <h:CustomerAccountId xmlns:h="https://bingads.microsoft.com/Reporting/v13">480xxxxxxxxx</h:CustomerAccountId>
    <h:CustomerId xmlns:h="https://bingads.microsoft.com/Reporting/v13">191xxxxxxxxx</h:CustomerId>
    <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Reporting/v13">105xxxxxxxxxxx</h:DeveloperToken>
    <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
    <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
  </s:Header>
  <s:Body>
    <PollGenerateReportRequest xmlns="https://bingads.microsoft.com/Reporting/v13">
      <ReportRequestId>'+ @reportid +'</ReportRequestId>
    </PollGenerateReportRequest>
  </s:Body>
</s:Envelope>''
--,IsMultiPart=''True''
,Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,RequestContentTypeCode=''TextXml''
,Header=''SOAPAction: "PollGenerateReport"''
,RequestMethod=''POST''
--,EnableBodyPlaceholderForMultiPart=''True''
,EnableStatusCheck=''True'', 
,StatucCheckMaxWaitSeconds=300, --wait max 5 mins?
,StatucCheckIterationWaitSeconds=5, --check every 5 sec?
,StatusSuccessValue=''Success'', --look for success word in response
,Meta=''[{"Name": "ReportDownloadUrl","Type": "String",Length:"300"}]''
)') AT zs_BingAds


declare @reportpath varchar(500)
select top 1 @reportpath=ReportDownloadUrl from #tmpBingAdsReport_Step2

select * from #tmpBingAdsReport_Step1
select * from #tmpBingAdsReport_Step2

--///////////////////////////////////////////////////////////////////////////////
--Step-3 - Get report file
--///////////////////////////////////////////////////////////////////////////////
--Static table
--//INSERT INTO BingAdsReport
EXEC(
'SELECT * FROM $
WITH(
 DataConnectionType=''Default''
,Src='''+ @reportpath +'''
,Filter=''$.Report.Table.Row[*]''
,ElementsToTreatAsArray=''Row''	
,RequestMethod=''GET''
,IsMultiPart=''False''
,FileCompressionType=''Zip''	
)') AT zs_BingAds


--select * from tmpBingAdsReport_Step3
--select * from openquery(Bing, 'select * from $')


/*
select * into tmpBingAdsReport from OPENQUERY(zs_BingAds,
'SELECT c.* FROM $
WITH(
--submit report request for XML format

alias=''a'' 
,Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,Filter=''$.s:Envelope.s:Body.SubmitGenerateReportResponse''
,RequestData=''@c:\BWProjects\Requests\BingAds\sql_body1.xml''
,IsMultiPart=''True''
,RequestContentTypeCode=''TextXml''
,Header=''SOAPAction: "SubmitGenerateReport"''
,RequestMethod=''POST''
,EnableBodyPlaceholderForMultiPart=''True''
,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''

--keep checking status until report is ready - returns final url when done

,join1_alias=''b''
,join1_Filter=''$.s:Envelope.s:Body.PollGenerateReportResponse.ReportRequestStatus''
,join1_RequestData=''@c:\BWProjects\Requests\BingAds\sql_body2.xml''
,join1_IsMultiPart=''True''
,join1_Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,join1_RequestContentTypeCode=''TextXml''
,join1_Header=''SOAPAction: "PollGenerateReport"''
,join1_RequestMethod=''POST''
,join1_EnableBodyPlaceholderForMultiPart=''True''
,join1_EnableStatusCheck=''True'', 
,join1_StatucCheckMaxWaitSeconds=300, --wait max 5 mins?
,join1_StatucCheckIterationWaitSeconds=5, --check every 5 sec?
,join1_StatusSuccessValue=''Success'', --look for success word in response

--Download report (Zip file) and Parse rows

,join2_alias=''c''
,join2_DataConnectionType=''Default''
,join2_Src=''[$b.ReportDownloadUrl$]''
,join2_Filter=''$.Report.Table.Row[*]''
,join2_ElementsToTreatAsArray=''Row''	
,join2_RequestMethod=''GET''
,join2_IsMultiPart=''False''
,join2_FileCompressionType=''Zip''	
)')

select * from tmpBingAdsReport

select * from openquery(Bing, 'select * from $')
*/
go

Conclusion

In this article we saw how to import Bing Ads API data in SQL Server without any coding using ZappySys ODBC PowerPack (SOAP / XML Driver). Download it it and try it for FREE to explore many API integration features not discussed in this article.

The post Import Bing Ads data into SQL Server (Performance Reports) appeared first on ZappySys Blog.

In our previous blog post we saw how to import rest API in SQL Server. Using same concepts lets look at how to import Amazon MWS Data into SQL Server. We will explore many techniques to call Amazon MWS API and learn how to automate Amazon MWS data extraction without doing any ETL. We will call MWS XML or CSV API just using T-SQL code. We will also learn how to Download Reports from by calling MWS API.

You can also refer other articles here (Power BI) and here (SSIS)

About Amazon MWS  API

If you want to sell something on Amazon you can use their e-commerce platform with some monthly fee. You can setup your entire online store / inventory using admin interface.

Amazon Marketplace Web Service (Amazon MWS) is an integrated web service API that helps Amazon sellers to programmatically exchange data on listings, orders, payments, reports, and more. Using these API you can read or write data from your Seller account and integrate it inside your own Systems (e.g. Reporting / ETL / BI tools).

Obtain MWS API Access Key / Secret and Seller ID

Very first thing to call any MWS API is to obtain Access Key, Secret Key and Seller ID (i.e. Merchant ID). To obtain this you first have to register a developer.

Here is where to look for your Merchant ID and Key / Secret.  Here is direct link to User Permissions page.

How to get Amazon MWS developer keys and know your Merchant ID (i.e. Seller ID)

Getting Started

Now let’s look at step by step instructions on how to call Amazon MWS API and then import into Power BI. In below steps we will use  ZappySys ODBC PowerPack so make sure it’s installed first.

Using ScratchPad to Test Amazon MWS API

Each MWS API has set of required and optional parameters. You can refer each API page for details. For example ListMatchingProducts API which we will use as an example. To make MWS API testing / learning easy, Amazon provides online testing tool called MWS ScratchPad. We will use this tool to craft API requests and use that information in ODBC driver SQL queries and later import data in Power BI.

1. Open MWS ScratchPad  by vising https://mws.amazonservices.com/scratchpad/index.html
2. Select Products API category and pick ListMatchingProducts API
3. Enter your SellerID (i.e. Merchant ID), Developer AccessKey (i.e. AWSAccessKeyId) and Secret Key we obtained in the previous section. Enter MarketId (e.g. ATVPDKIKX0DER for USA Market) and other parameters as below and Click Submit to see response.
   

   Using Amazon MWS ScratchPad (API Testing Tool – Created by Amazon)

4. Click on the Response Details Tab to extract some important information we will use for ODBC configuration. Notice attributes in red rectangles. They are the ones which we will need in the POST request Body. ZappySys Drivers supply many common parameters automatically but any API endpoint specific parameter must be supplied in the Request Body. For example you dont have to supply SignatureVersion, Timestamp, Signature or SignatureMethod
   

   Calling Amazon MWS API in Scratchpad Testing Tool (Response Details Tab) – ListMatchingProducts Example

This API requires Market ID parameter so enter it as per your Market where you selling. For example USA marketplaceid = ATVPDKIKX0DER. You can lookup correct MarketplaceId by checking this help (endpoints / marketplaces).

Setup Amazon MWS API Connections

Create XML Driver Connection for Amazon MWS API (e.g. RequestReport )

Some APIs we need to call in this article is XML format and some API (e.g. GetReport) is CSV format – Tab delimited so we need to create 2 different connections using CSV driver and XML driver. So lets get started with Amazon MWS Connection for XML API.

1. Click on Add New Data source. Name as AMAZON-MWS-XML and select Native – ZappySys XML Driver option.
   

   Add Gateway Data Source

2. Now click Edit to Configure Settings. For now we will only care OAuth Connection settings. All other options are overwritten in SQL Query.
   

   Edit Gateway Data Source Settings

3. When UI opens for Driver Enter MWS API Service URL as below. Notice that we added /Products/2011-10-01 (This was extracted from previous section). You can check full list of Amazon MWS Endpoints here. Because we are connecting to USA region we will use below URL.
   

   https://mws.amazonservices.com/Products/2011-10-01

4. Select OAuth from the Connection dropdown and Click Configure Settings
5. On the Connection UI, Select OAuth Provider Amazon Market Web Service (MWS)
6. Enter your API Access Key and Secret Key (obtained from here)
7. Your connection will look like below.
   

   ODBC Connection Credentials for Amazon MWS API (Access Key, Secret Key, URL Endpoint)

8. Also configure Retry options to handle API limit reached error when you call API too often. For Amazon MWS we get Status code 503 when request is throttled so lets narrow down on which error we want to retry.
   

   Retry Options

9. Click OK to save OAuth Connection UI to go back to main UI
10. Now Go to Throttling Tab and enter some delay (e.g. enter 20000 for 20sec delay after each request). This will slow down API calls so we don’t face errors about API Limit reached.
11. Click OK to Save UI
12. Click Save button on Gateway UI to save and restart

Create CSV Driver Connection for Amazon MWS API (e.g. GetReport )

Now lets create a connection CSV API (e.g. GetReport ).

Follow almost same steps as previous section except use CSV Driver when you click on Add new Data source select Native – ZappySys CSV Driver and name data source as AMAZON-MWS-CSV . As we mentioned for now we mostly want to setup Connection and not worry about Request Body, Pagination etc.

Create Linked Server for SQL Server

Run following Script in SSMS to create Linked servers. Change ######## with Your gateway password. Change localhost with machine name or IP where ZappySys Gateway Is running. Use localhost if SQL Server and Gateway both on the same machine.

USE [master]
GO

EXEC master.dbo.sp_addlinkedserver @server = N'AMAZON-MWS-XML', @srvproduct=N'', @provider=N'SQLNCLI', @datasrc=N'localhost,5000', @catalog=N'AMAZON-MWS-XML'
EXEC master.dbo.sp_addlinkedsrvlogin @rmtsrvname=N'AMAZON-MWS-XML',@useself=N'False',@locallogin=NULL,@rmtuser=N'tdsuser',@rmtpassword='########'
GO

EXEC master.dbo.sp_addlinkedserver @server = N'AMAZON-MWS-CSV', @srvproduct=N'', @provider=N'SQLNCLI', @datasrc=N'localhost,5000', @catalog=N'AMAZON-MWS-CSV'
EXEC master.dbo.sp_addlinkedsrvlogin @rmtsrvname=N'AMAZON-MWS-CSV',@useself=N'False',@locallogin=NULL,@rmtuser=N'tdsuser',@rmtpassword='########'
GO

If you want to know how to configure Linked Server using UI then check this article.

Amazon MWS ListOrders Example

Now lets look at how to call ListOrders request with pagination in SQL Server. Create a following Stored Procedure to List MWS Orders. If you want to extract many orders then call Report rather than this proc (See next section for generating reports)

USE MyDatabase
GO
/*
Example: 
 exec usp_MWS_ListOrders @SellerId='Axxxxxxx', @MarketplaceId ='Bxxxxxxx',@CreatedAfter='2019-01-01'  
*/
Create proc [dbo].[usp_MWS_ListOrders]
	@SellerId varchar(100),
	@MarketplaceId varchar(100),
	@CreatedAfter varchar(100),
	@CreatedBefore varchar(100)=null,
	@MaxRows int=0
as
declare @sql varchar(max) 
declare @limitClase varchar(100)=''
if(@MaxRows>0)
	set @limitClase	=' LIMIT ' + cast(@MaxRows as varchar(20))

--//Confirm ISO Date format 2019-06-01T04%3A00%3A00Z  rather 2019-06-01T04:00:00Z
if(isnull(@CreatedAfter,'')<>'')
begin
	set @CreatedAfter='&CreatedAfter=' + @CreatedAfter
	set @CreatedAfter=replace(@CreatedAfter,':','%3A')
	if(@CreatedAfter NOT LIKE @CreatedAfter + 'T%')
		set @CreatedAfter=@CreatedAfter + 'T00%3A00%3A00Z'
	if(@CreatedAfter NOT LIKE @CreatedAfter + '%Z')
		set @CreatedAfter=@CreatedAfter + 'Z'
	--//T04%3A00%3A00Z
end
if(isnull(@CreatedBefore,'')<>'')
begin
	set @CreatedBefore='&CreatedBefore=' + @CreatedBefore
	set @CreatedBefore=replace(@CreatedBefore,':','%3A')
	if(@CreatedBefore NOT LIKE @CreatedBefore + 'T%')
		set @CreatedBefore=@CreatedBefore + 'T00%3A00%3A00Z'
	if(@CreatedBefore NOT LIKE @CreatedBefore + '%Z')
		set @CreatedBefore=@CreatedBefore + 'Z'
end

set @CreatedAfter = isnull(@CreatedAfter,'')
set @CreatedBefore = isnull(@CreatedBefore,'')


set @sql = 'SELECT * FROM OPENQUERY([AMAZON-MWS-XML]
, ''
SELECT ReportRequestId FROM $
' + @limitClase +' 
WITH(
     Src=''''https://mws.amazonservices.com/Orders/2013-09-01''''
	,Filter=''''$.RequestReportResponse.RequestReportResult.ReportRequestInfo''''
	,ElementsToTreatAsArray=''''Order''''
	,RequestData=''''Action=ListOrders[$tag$]' + @CreatedAfter + @CreatedBefore + '&MarketplaceId.Id.1='+ @MarketplaceId +'&SellerId='+ @SellerId +'''''
	,RequestMethod=''''POST''''
	,NextUrlAttributeOrExpr=''''$.ListOrders[$tag$]Response.ListOrders[$tag$]Result.NextToken''''
	,NextUrlSuffix=''''&NextToken=<%nextlink_encoded%>''''
	,WaitTimeMs=20000 -- slow down to avoid throttling. 6 requests per min
	,HasDifferentNextPageInfo=''''True''''
	,EnablePageTokenForBody=''''True''''
	,PagePlaceholders=''''body=|ByNextToken;filter=|ByNextToken''''
	--,DataConnectionType=''''OAuth''''
	--,ScopeSeparator=''''{space}''''
	--,ServiceProvider=''''AmazonMWS''''
	--,ClientId=''''AKIxxxxxx''''
	--,ClientSecret=''''AKIxxxxxx''''
	--,UseCustomApp=''''True''''
	,CacheFileLocation=''''c:\temp\mws-orders.cache''''
	,CacheEntryTtl=300 --send every 5 mins else use from cache
	,IncludeParentColumns=''''False''''
)
'')'
print @sql

EXECUTE( @sql)

Here is how to call this proc. Change SellerId, MarketplaceId and Date.

exec usp_MWS_ListOrders @SellerId='Axxxxxxx', @MarketplaceId ='Bxxxxxxx',@CreatedAfter='2019-01-01'

Download Amazon MWS Report data in SQL Server (CSV / XML Format)

Now lets look at how to extract data from Custom Report API (You must need ODBC PowerPack v1.1.1 or higher). However reading data from Reports not single step process because report generation is Job style API. Which means you send report request and wait  until its done. Once Report ready you have to read in CSV format or XML. Some Reports only available in CSV format. Check this post to understand how complex it can be to get data. We will make it simple for you to understand this in 3 steps. Basically calling reports requires minimum 3 API calls (see below)

1. Create a new report request – Cal RequestReport API . It returns ReportRequestId (You can use it in the next step)
2. Check Report Status see its done – Call GetReportRequestList API (Pass ReportRequestId got in the previous step to get data for only one Report request we care). Keep checking Report Status every few seconds until you get _DONE_  or _DONE_NO_DATA_ status in response.
3. Read Report Data – Call GetReport. This API call returns CSV or XML data. So use correct Driver for this step. If CSV data is returned then we must use CSV Driver rather than XML Driver.

Method 1 – Single query approach (New Driver)

In new driver ZappySys added few new features which makes it possible to call above 3 steps process in a single query. New query syntax offers StatusCheck and Mixing Content Formats using same driver. Because Most MWS Reports are in CSV Format we have to use CSV Driver but oddly first 2 steps listed in above sections are in XML Format so we have to use XML driver. Technically we need atleast 2 steps (i.e 2 queries) but we will show you how to achieve using single query as below.

1. Assuming you have followed steps to create a linked server AMAZON-MWS-CSV (see previous section)
2. Run below query. Replace necessary parameters as per your need. (i.e MarketplaceId , SellerId, CreatedAfter date)
   

   --- //////////////////////////////////////////////////////////////////////////
   -- Change _GET_MERCHANT_LISTINGS_DATA_ to something else if you need different report type (all 3 steps) -- https://docs.developer.amazonservices.com/en_US/reports/Reports_ReportType.html
   -- Replace MarketplaceId in all 3 steps (i.e for USA use ATVPDKIKX0DER) --https://docs.developer.amazonservices.com/en_US/dev_guide/DG_Endpoints.html
   -- Replace SellerId in all 3 steps
   -- Remove or Change CreatedAfter in first step (or use CreatedBefore parameter instread of CreatedAfter)
   --- //////////////////////////////////////////////////////////////////////////
   select * into tmpMWS_Report from OPENQUERY([AMAZON-MWS-CSV],
   '
   SELECT s3.* FROM $
   WITH(
   Src=''https://mws.amazonservices.com/Reports/2009-01-01'',RequestMethod=''POST''
   ,RequestData=''Action=RequestReport&ReportType=_GET_MERCHANT_LISTINGS_DATA_&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxx&CreatedAfter=2017-12-31T04%3A00%3A00Z''
   ,IncludeParentColumns=''False''
   ,EnableRawOutputModeSingleRow=''True''
   ,RawOutputFilterExpr=''//*[local-name()="ReportRequestId"]'' 
   ,RawOutputDataRowTemplate=''{ReportRequestId:"[$1]"}'' 
   ,RawOutputExtractMode=''Xml'' 
   ,WaitTimeMs=5000
   ,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''
   ,Alias=''s1''
   
   ,Join1_alias=''s2''	
   ,Join1_Src=''https://mws.amazonservices.com/Reports/2009-01-01'',Join1_RequestMethod=''POST''
   ,Join1_RequestData=''Action=GetReportRequestList&ReportRequestIdList.Id.1=[$s1.ReportRequestId$]&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxxxxx''
   ,Join1_EnableRawOutputModeSingleRow=''True''
   ,Join1_RawOutputFilterExpr=''//*[local-name()="ReportProcessingStatus"]||//*[local-name()="GeneratedReportId"]'' 
   ,Join1_RawOutputDataRowTemplate=''{ReportProcessingStatus:"[$1]",GeneratedReportId:"[$2]"}'' 
   ,Join1_RawOutputExtractMode=''Xml'' 
   ,Join1_EnableStatusCheck=''True'', 
   ,Join1_StatucCheckMaxWaitSeconds=300,
   ,Join1_StatucCheckIterationWaitSeconds=25,
   ,Join1_StatusSuccessValue=''_DONE_''
   ,Join1_StatusFailedValue=''_DONE_NO_DATA_|_CANCELLED_''
   
   ,Join2_Alias=''s3''
   ,Join2_Src=''https://mws.amazonservices.com/Reports/2009-01-01'',Join2_RequestMethod=''POST''
   ,Join2_RequestData=''Action=GetReport&ReportId=[$s2.GeneratedReportId$]&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxxxxxx''	
   ,Join2_ColumnDelimiter=''{TAB}''
   ,Join2_ResponseCharset=''Windows-1252''	 
   ,Join2_EnableStatusCheck=''False'' 
   ,Join2_EnableRawOutputModeSingleRow=''False''
   ,Join2_WaitTimeMs=10000
   )') 
   go
   select * from tmpMWS_Report

    

Performance Tip

If you like to improve performance and dont want to use SELECT INTO approach to load into temp table as above then you can use EXEC (….) AT YourLinkedServerName  as below.

Advantage of this method is your Query speed will increase because system calls API only once when you call EXEC AT. In OPENROWSET it needs to call above query twice (Once to obtain metadata and once to get data).

INSERT INTO tmpMWS_Report
EXEC('select s3.* .................. ') AT [AMAZON-MWS-CSV]

Method 2 – Multiple query approach (For old driver)

Now lets see how to load Amazon MWS data in SQL Server using old driver (v1.1.2 or older) which didnt support single query approach like previous section. If you are using older driver or for some reason you like to split steps into multiple procedures then use below approach.

usp_MWS_Report  Stored Proc ( Generate ReportRequest, Wait until done, Read data )

USE MyDatabase
GO

/*
Examples: 
Get ReportType from here https://docs.developer.amazonservices.com/en_UK/reports/Reports_ReportType.html

exec usp_MWS_Report @MarketplaceId ='ATVPDKIKX0DER', @SellerId='AZxxxxxxxx', @ReportType = '_GET_MERCHANT_LISTINGS_DATA_'
exec usp_MWS_Report @MarketplaceId ='ATVPDKIKX0DER', @SellerId='AZxxxxxxxx', @ReportType = '_GET_FLAT_FILE_ORDER_REPORT_DATA_'

--Do not Throw Error if no data found (return empty resultset or run custom sql)
exec usp_MWS_Report @MarketplaceId ='ATVPDKIKX0DER', @SellerId='AZxxxxxxxx', @ReportType = '_GET_FLAT_FILE_ORDER_REPORT_DATA_',@ThrowErrIfNoData=0,@EmptyRowSql='select top 0 * from Orders'
--Supply date range
exec usp_MWS_Report @MarketplaceId ='ATVPDKIKX0DER', @SellerId='AZxxxxxxxx', @ReportType = '_GET_MERCHANT_LISTINGS_DATA_', @StartDate='2019-01-01T00:00:00Z', @EndDate='2019-01-31T00:00:00Z'

--loading into table 

--//////////////////////////////////
— Step-1 : Create Local Linked server to point to self
--//////////////////////////////////
USE [master]
GO
--change default catalog
--For MSSQL 2012, 2014, 2016, 2017, and 2019 use @provider=N'MSOLEDBSQL' below (SQL Server Native Client 11.0)---
EXEC master.dbo.sp_addlinkedserver @server = N'ME', @srvproduct=N'', @provider=N'SQLNCLI11', @datasrc=N'localhost', @catalog=N'Northwind'
--For MSSQL 2022 or higher use @provider=N'MSOLEDBSQL' below (Microsoft OLE DB Driver for SQL Server)---
--EXEC master.dbo.sp_addlinkedserver @server = N'ME', @srvproduct=N'', @provider=N'MSOLEDBSQL', @datasrc=N'localhost', @catalog=N'Northwind'

EXEC master.dbo.sp_addlinkedsrvlogin @rmtsrvname=N'ME',@useself=N'True',@locallogin=NULL,@rmtuser=NULL,@rmtpassword=NULL
GO
--//////////////////////////////////
— Step-2 : Enable RPC OUT and Disable MSDTC
--//////////////////////////////////
EXEC sp_serveroption 'ME', 'rpc out', true;  
GO
--Below needed to support EXEC + INSERT (dynamic query)
EXEC sp_serveroption 'ME', 'remote proc transaction promotion', false;
GO
--test simple query
select * from OPENQUERY(ME,'select 1')
GO

--//////////////////////////////////
— Step-3 : Load data into table from MWS Report
--//////////////////////////////////
SELECT * INTO #amazon_mws_report
FROM OPENQUERY(me,'
SET NOCOUNT ON;
EXEC usp_MWS_Report 
   @Columns=''[item-name], [item-description], [listing-id]'',
   @MarketplaceId =''ATVPDKIKX0DER'', 
   @SellerId=''Axxxxxxxxxxxxxxxxx'', 
   @ReportType = ''_GET_MERCHANT_LISTINGS_DATA_''
WITH RESULT SETS 
(
	(itmname varchar(1000),itmdesc varchar(1000),itmid varchar(1000))
)
')

select * from #amazon_mws_report


*/
CREATE proc [dbo].[usp_MWS_Report]
	@MarketplaceId varchar(100),
	@SellerId varchar(100),
	@ReportType varchar(100)='_GET_MERCHANT_LISTINGS_DATA_',
	@StartDate varchar(100)=null,
	@EndDate varchar(100)=null,
	@ReportId bigint=null,  ---i.e. 14783134522018025
	@ThrowErrIfNoData bit=1,
	@EmptyRowSql varchar(1000)=null, --SQL to run if no data found
	@Columns varchar(8000)='*'  --report columns which goes in final SELECT  e.g. [item-name], [item-description], [listing-id] ....
as


declare @retGeneratedReportId bigint
set @retGeneratedReportId=@ReportId
---////// STEP 1 /////////
if(@ReportId is null)
begin
	create table #step1(ReportRequestId bigint)
	insert into #step1
	exec usp_MWS_RequestReport @ReportType,@MarketplaceId,@SellerId,@StartDate,@EndDate

	--select * from #step1 --debug

	declare @retReportRequestId bigint
	select @retReportRequestId=ReportRequestId from #step1

	---////// STEP 2 /////////
	declare @status varchar(100)
	declare @cnt int
	create table #step2(ReportProcessingStatus varchar(100),GeneratedReportId bigint)
	
	--Keep looping until status is done or we waited too long
	while(isnull(@status,'') IN ('','_SUBMITTED_','_IN_PROGRESS_') )
	begin
		truncate table #step2
		insert into #step2
		exec usp_MWS_GetReportIdAndStatus @retReportRequestId,@MarketplaceId,@SellerId 
	
		select @status=ReportProcessingStatus, @retGeneratedReportId=GeneratedReportId from #step2
		if(isnull(@status,'') IN ('','_SUBMITTED_','_IN_PROGRESS_') )			
		begin
			--amazon says dont check often https://docs.developer.amazonservices.com/en_UK/reports/Reports_GetReportRequestList.html
			Print 'Status='+  isnull(@status,'') +'. Checking after 45 sec... #' + cast(@cnt as varchar(10))
			WAITFOR DELAY '00:00:45'			
		end
		else
			print @status

		set @cnt=@cnt+1

		--if(@cnt>50)
			--//throw err?
	end
	--select * from #step2 --debug
end

if(@retGeneratedReportId is null and @status='_DONE_NO_DATA_')
begin
	if(@ThrowErrIfNoData=1)
		RAISERROR('Cannot generate report - No data found for specified criteria',16,1); 
	else if (isnull(@EmptyRowSql,'')<>'')
		execute(@EmptyRowSql)
end
else if(@retGeneratedReportId is null)
	RAISERROR('Cannot generate report - Unknown error (either report got cancelled or other error occurred)',16,1); 

	
---////// STEP 3 /////////
--If xml format of report data then use XML driver else CSV (most are Tab delimited reports in CSV format)
--exec usp_MWS_GetReportDataXml @retGeneratedReportId,@MarketplaceId,@SellerId
exec usp_MWS_GetReportDataCsv @retGeneratedReportId,@MarketplaceId,@SellerId,@Columns

drop table #step1
drop table #step2

GO

usp_MWS_RequestReport  Stored Proc ( Step1 – Create Report Request )

USE [MyDatabase]
GO

CREATE proc [dbo].[usp_MWS_RequestReport]
	@ReportType varchar(100),
	@MarketplaceId varchar(100),
	@SellerId varchar(100),
	@StartDate varchar(100)=null,
	@EndDate varchar(100)=null
as

/*Learn more : https://docs.developer.amazonservices.com/en_UK/reports/Reports_RequestReport.html */

--//Confirm ISO Date format 2019-06-01T04%3A00%3A00Z  rather 2019-06-01T04:00:00Z
if(isnull(@StartDate,'')<>'')
begin
	set @StartDate='&StartDate=' + @StartDate
	set @StartDate=replace(@StartDate,':','%3A')
	if(@StartDate NOT LIKE @StartDate + 'T%')
		set @StartDate=@StartDate + 'T00%3A00%3A00Z'
	if(@StartDate NOT LIKE @StartDate + '%Z')
		set @StartDate=@StartDate + 'Z'
	--//T04%3A00%3A00Z
end
if(isnull(@EndDate,'')<>'')
begin
	set @EndDate='&EndDate=' + @EndDate
	set @EndDate=replace(@EndDate,':','%3A')
	if(@EndDate NOT LIKE @EndDate + 'T%')
		set @EndDate=@EndDate + 'T00%3A00%3A00Z'
	if(@EndDate NOT LIKE @EndDate + '%Z')
		set @EndDate=@EndDate + 'Z'
end

set @StartDate = isnull(@StartDate,'')
set @EndDate = isnull(@EndDate,'')

declare @sql varchar(max) 
set @sql = 'SELECT * FROM OPENQUERY([AMAZON-MWS-XML]
, ''
SELECT ReportRequestId FROM $
WITH(
	Src=''''https://mws.amazonservices.com/Reports/2009-01-01'''',RequestMethod=''''POST''''
	--,ElementsToTreatAsArray=''''Product''''
	,Filter=''''$.RequestReportResponse.RequestReportResult.ReportRequestInfo''''
	,RequestData=''''Action=RequestReport&ReportType=' + @ReportType  + '&MarketplaceId='+ @MarketplaceId +'&SellerId='+ @SellerId + @StartDate + @EndDate +'''''
	,CacheFileLocation=''''c:\temp\mws-cache-step-1.cache''''
	,CacheEntryTtl=300 --send every 5 mins else use from cache
	,WaitTimeMs=''''300''''
	,IncludeParentColumns=''''False''''
	,Meta=''''[{"Name": "ReportRequestId","Type": "Int64"}]''''
)'')'
--print @sql

EXECUTE( @sql)
--sleep 25 seconds after this - hope report is ready by that time else We have to loop
WAITFOR DELAY '00:00:25'  --lets wait 30 sec... incase we get luck and report is done?
GO

usp_MWS_GetReportIdAndStatus Stored Proc ( Step2 – Get Report Id and Report Status )

USE MyDatabase
GO


CREATE proc [dbo].[usp_MWS_GetReportIdAndStatus]
	@ReportRequestId bigint,
	@MarketplaceId varchar(100),
	@SellerId varchar(100)
as

declare @sql varchar(max) 
set @sql = 'SELECT * FROM OPENQUERY([AMAZON-MWS-XML]
, ''SELECT *  FROM $
WITH(
	 Src=''''https://mws.amazonservices.com/Reports/2009-01-01'''',RequestMethod=''''POST''''
	,ElementsToTreatAsArray=''''ReportRequestInfo''''
	,Filter=''''$.GetReportRequestListResponse.GetReportRequestListResult.ReportRequestInfo''''
	,RequestData=''''Action=GetReportRequestList&ReportRequestIdList.Id.1=' + cast(@ReportRequestId as varchar(20)) + '&MarketplaceId='+ @MarketplaceId +'&SellerId='+ @SellerId +'''''
	,WaitTimeMs=''''300''''
	,IncludeParentColumns=''''False''''
	--,CacheFileLocation=''''c:\temp\mws-cache-step-2.cache''''
	--,CacheEntryTtl=10  --do not use from cache too old.. but cache still prevents too many requests
	,Meta=''''[{"Name": "ReportProcessingStatus","Type": "String", Length: 100},{"Name": "GeneratedReportId","Type": "Int64"}]''''	
)'')'

--print @sql

EXECUTE( @sql)

GO

usp_MWS_GetReportDataCsv Stored Proc (Step3 – Get Report Data in CSV format )

Last step in report generation is download the report data in CSV format.

USE MyDatabase
GO

CREATE proc [dbo].[usp_MWS_GetReportDataCsv]
	@ReportId bigint,
	@MarketplaceId varchar(100),
	@SellerId varchar(100),
	@Columns varchar(8000)='*'

as

declare @sql varchar(max) 
set @sql = 'SELECT '+ @Columns +' FROM OPENQUERY([AMAZON-MWS-CSV], 
''SELECT * FROM $
WITH(
    Src=''''https://mws.amazonservices.com/Reports/2009-01-01'''',RequestMethod=''''POST''''
	,RequestData=''''Action=GetReport&ReportId=' + cast(@ReportId as varchar(20)) + '&MarketplaceId='+ @MarketplaceId +'&SellerId='+ @SellerId +'''''
	,WaitTimeMs=''''300''''
	,ColumnDelimiter=''''{TAB}''''
	,ResponseCharset=''''Windows-1252''''	
	,CacheFileLocation=''''c:\temp\mws-cache-step-3.cache''''
	,CacheEntryTtl=300 --once report is generated lets try to read from cache 
	)''
)'

print @sql

EXECUTE( @sql)
GO

Example – Import data from Amazon MWS into SQL Server Table

Finally when we have all stored procs created we can load into target table as below. In our previous article we saw how to use SELECT INTO along with OPENQUERY to load data into new temp table or insert into some existing table. However when you want to load Stored Proc output into table its not straight forward specially when we calling dynamic SQL.

Three tricks worth mentioning here which makes it possible to load data from stored proc into temp table dynamically.

1. Create a linked server to Self so we can use OPENQUERY against to call stored proc so its easy to use SELECT INTO.
2. Use WITH RESULT SETS to describe metadata of output. This is needed because we are using dynamic SQL inside proc and WITH RESULT SETS describes the metadata.
3. Use of SET NO COUNT ON to avoid error like below
   

   "SOMESERVERNAME" indicates that either the object has no columns 
   or the current user does not have permissions on that object

4. If you dont do above tricks then you may see errors like this (INSERT is not allowed when we use DYNAMIC SQL inside nested Stored Procs)

To load output into table you must know column names from the output.  If you dont know you can try to run stored proc without @Columns like below.

EXEC usp_MWS_Report @MarketplaceId =''ATVPDKIKX0DER'', @SellerId=''Axxxxxxxxxxxxx'', @ReportType = ''_GET_MERCHANT_LISTINGS_DATA_''

Once we know columns from report, you can run following script to insert into table.

Full Script – Load MWS Report Output into Table

--//////////////////////////////////
— Step-1 : Create Local Linked server to point to self
--//////////////////////////////////
USE [master]
GO
--change default catalog
--For MSSQL 2012, 2014, 2016, 2017, and 2019 use @provider=N'MSOLEDBSQL' below (SQL Server Native Client 11.0)---
EXEC master.dbo.sp_addlinkedserver @server = N'ME', @srvproduct=N'', @provider=N'SQLNCLI11', @datasrc=N'localhost', @catalog=N'Northwind'
--For MSSQL 2022 or higher use @provider=N'MSOLEDBSQL' below (Microsoft OLE DB Driver for SQL Server)---
--EXEC master.dbo.sp_addlinkedserver @server = N'ME', @srvproduct=N'', @provider=N'MSOLEDBSQL', @datasrc=N'localhost', @catalog=N'Northwind'

EXEC master.dbo.sp_addlinkedsrvlogin @rmtsrvname=N'ME',@useself=N'True',@locallogin=NULL,@rmtuser=NULL,@rmtpassword=NULL
GO
--//////////////////////////////////
— Step-2 : Enable RPC OUT and Disable MSDTC
--//////////////////////////////////
EXEC sp_serveroption 'ME', 'rpc out', true;  
GO
--Below needed to support EXEC + INSERT (dynamic query)
EXEC sp_serveroption 'ME', 'remote proc transaction promotion', false;
GO
--test simple query
select * from OPENQUERY(ME,'select 1')
GO

--//////////////////////////////////
— Step-3 : Load data into temp table from Amazon MWS Report
--//////////////////////////////////
SELECT * INTO #amazon_mws_report
FROM OPENQUERY(me,'
SET NOCOUNT ON;
EXEC usp_MWS_Report 
   @Columns=''[item-name], [item-description], [listing-id]'',
   @MarketplaceId =''ATVPDKIKX0DER'', 
   @SellerId=''Axxxxxxxxxxxxx'', 
   @ReportType = ''_GET_MERCHANT_LISTINGS_DATA_''
WITH RESULT SETS 
(
	(itmname varchar(1000),itmdesc varchar(1000),itmid varchar(1000))
)
')

select * from #amazon_mws_report

Import Amazon MWS Data into SQL Table (Load Report output into temp table)

Conclusion

In this article, we saw how to load Amazon MWS Data into SQL Server. We used simple SQL queries with ZappySys XML / CSV Driver to pull data from Amazon MWS API. ZappySys ODBC PowerPack makes it super easy to consume data from virtually any API (JSON / XML or CSV Web Services), no coding required. Download here to get started with ZappySys ODBC drivers.

The post How to import Amazon MWS data into SQL Server (T-SQL) appeared first on ZappySys Blog.

Read RSS feed in SSIS can be challenging. RSS named first RDF Site Summary and later named Rich Site Summary and Really Simple Syndication allows customer applications to be updated with the news of a site. For example, Microsoft RSS feeds, Apple RSS feeds, Samsung RSS feeds, etc.  With RSS the information comes to you directly and you do not need to waste time checking the web site. In this article, we will show how to read RSS feed in SSIS and later using ODBC.

Requirements

1. First of all, ZappySys SSIS Powerpack installed.
2. Secondly, SSDT for Business Intelligence (SSIS) installed.
3. Finally, ODBC Powerpack installed.

Getting Started

Understanding RSS data

1. In order to start, we will work with the WordPress RSS example:
   

   https://wordpress.org/news/feed/

2. When you go to the link, you see the feed like this:
   

   RSS information

3. Basically, it is an XML file with the Last Build date (lastBuildDate), the title, link, description. It shows the WordPress news per version and date.

Read RSS feed in SSIS

1. First of all, you need an SSIS Project and a Data Flow with the ZS XML Source (if you are not familiar with SSIS and the ZS XML Source, refer to this link.
   

   Drag and drop Data Flow

2. Secondly, In the ZS XML Source, add the following RSS WordPress URL:
   

   https://wordpress.org/news/feed/

   

   SSIS read RSS

3. After that, you will be able to see the following tags when you press the array filter:
4. This time, we will focus on the information under channel which is the most relevant, make sure that the array filter is this one:
   

   $.rss.channel[*].item[*]

   

   Using RSS filters in SSIS

5. In addition, press the preview data button and you will see the following information:
   

   SSIS preview RSS

Read RSS feed in SSIS and copy to SQL Server

1. In the previous section, we got data from RSS. Now we will copy to SQL Server.
2. First of all, drag and drop the OLEDB Destination and join the XML Source with the OLEDB Destination:
   

   SSIS RSS to OLEDB

3. Also, in the OLEDB Destination, press New button and select a SQL Server connection or create a new one:
   

   RSS to Oledb

4. In addition, in the table or view textbox press new to create a new destination table:
   

   Create a new SQL table to store RSS data

5. Next, modify the table name:
   

   Change SSIS destination table name

6. Also, go to the mappings page to generate the mappings:
   

   RSS Source to SQL Destination in SSIS

7. Finally, run the package and you will export the data from RSS to SQL Server.

How to read RSS feed in SSIS using pagination

If you have multiple rows, pagination helps a lot to handle multiple items. You can handle the information in pages. Instead of loading all the information, you can get the data from page 1, 2, 3 and more.

1. First of all, in the URL, you can include the following to specify the page:
   

   https://wordpress.org/news/feed/?paged=2

2. Also, note that it used the word paged and not page. Many URLs use page and not paged, but WordPress is different:
3. In addition, if you want to use variables, you can use a variable like this:
   
   RSS page as a variable
4. Also, we have a pagination tab with different pagination options. You can check the configuration here:
   

   RSS pagination configuration

5. Finally, for more information about pagination, refer to this link.

How to read RSS feed in ZappySys ODBC PowerPack

Zappysys ODBC PowerPack is a powerful tool to access to REST API, SOAP, XML, JSON, CSV, Odata. In this article, we will show how to access to RSS using ODBC PowerPack.

1. First of all, open your ODBC Data Source Administrator in Windows.
2. Secondly, in the user DSN, press Add.
3. Also, add the Zappysys XML Driver:
   

   odbc XML driver

4.  In addition, add the following URL:
   

   https://wordpress.org/news/feed/

   

   ODBC to RSS connection

5. Finally, you can use the filters and pagination as we did with the SSIS PowerPack in the previous sections.
   

   ODBC pagination RSS

Conclusion

To conclude, we can say that with SSIS PowerPack and ODBC PowerPack, it is easy to access, filter and paginate RSS information or any XML / SOAP API. We strongly recommend you to try the product by yourself and check the results.

References

For more information about RSS, refer to these links:

• RSS
• What Is RSS? RSS Explained

The post How to read RSS feed in SSIS and ODBC (with pagination) appeared first on ZappySys Blog.

In this post we will learn how to read / write REST API data in Talend Open Studio. We will create a simple Talend Job using ZappySys JSON  Driver to read from REST API / JSON Files and load into Target (e.g. File / DB). Techniques listed in this article can be also used to read from SOAP API / XML Files or CSV Files / API using XML Driver / CSV Driver.

These drivers support familiar SQL query language. Using SQL you can query virtually any API services just like relational database table. It can flatten nested hierarchy and provide output in rows / columns. Many complex REST API / SOAP API complexity is taken care automatically (e.g. Authentication, Pagination, Security, Error Handling).

So let’s get started.

Requirements

1. Download and install Talend Open Studio (FREE) from here. Skip this step if you already installed.
2. Download ZappySys ODBC PowerPack (JSON / XML Drivers)
3. Get Microsoft JDBC driver for SQL Server from here (Download sqljdbc_6.0.8112.200_enu.exe which is self extracting file you can run and extract to some folder)
   After you extract jdbc files, go to sqljdbc_6.0\enu\jre8\ folder rename sqljdbc42.jar to  mssql-jdbc.jar (name must be this). We will load this file in Talend later in this article.
4. Basic knowledge about REST API and JSON / XML format.

Configure Data Gateway

Register MS SQL JDBC driver in Talend

Now lets register Microsoft JDBC Driver in Talend. This is very important step because MSSQL JDBC driver is used to communicate with ZappySys Data Gateway we configured in previous step.

If you missed steps mentioned in the Requirements section then make sure you first download JDBC driver using below steps.

Get Microsoft JDBC driver for SQL Server from here (Download sqljdbc_6.0.8112.200_enu.exe which is self extracting file you can run and extract to some folder). After you extract jdbc files, go to sqljdbc_6.0\enu\jre8\ folder rename sqljdbc42.jar to  mssql-jdbc.jar (name must be this).

Now lets go through the steps to register MSSQL jdbc driver in Talend.

1. Open Talend Open Studio
2. Go to Windows > Click Show View
   

   Talend – Show View

3. When you see Popup selection under Talend > Select Modules
   

   Talend – Select Modules Window

4. When Module window is visible click on Little Jar Icon (Bottle icon) in the toolbar.
5. Select mssql-jdbc.jar file we renamed earlier and load this file.
   

   Talend – Import JAR file / module (Register MS SQL JDBC Driver Example)

6. That’s it. Now we ready to make API calls / read from JSON / XML in the next section.

Setup Talend REST API Connection (JSON / XML / CSV)

Now let’s configure REST API Connection in Talend. To read from JSON / XML Files you can use same steps too. We will use MSSQL JDBC Driver to connect to ZappySys Data Gateway.

1. In Talend Go to Metadata > Db Connections (Right click) > Create Connection
   

   Talend – Create new DB connection (JSON / REST API Example)

2. On the connection Wizard specify following attributes.
   DB Version : Microsoft
   Login :  username you setup in zappysys data gateway
   Password :  password of data gateway user
   Server : machine name or IP where zappysys data gateway is running
   Port : <default is 5000> Port on which zappysys data gateway is listening
   DataBase :  Data source name you setup in zappysys data gateway (case-sensitive)
   

   Talend – Connect to JSON / REST API using ZappySys Gateway (Use MS SQL JDBC Driver)

3. That’s all we need to do to setup a connection which can be used to read / write REST API data in Talend. In the next section we will see how to create a job to read data from REST API Service using this connection.

Read from REST API in Talend

Now let’s look at how to read data from REST API source or JSON / XML File using the connection we configured in the previous section.

Configure REST API Source

1. Create a Talend JOB and double click to open designer
2. Now drag and drop MSSQL Connection we created for ZappySys Data gateway, drop it on the designer surface. It will popup UI like below.
3. Select tDBInput (Microsoft SQL Server). Remember that we are using MSSQL JDBC Driver to connect to ZappySys Data Gateway for REST API Call. This gateway uses Microsoft TDS Protocol so MSSQL JDBC driver is used to communicate.
4. Now rename Source to something meaningful (e.g. Read from JSON REST API)
5. Double click REST Source to configure.
6. Enter Query like below (Make sure to enter between double quotes). See below examples to read from URL or File. If you have double quote in SQL then escape using \” character (e.g. select \”my col\” from $ )Read From REST API Url
   

   "SELECT * FROM value 
   WITH (SRC='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json')"

   Read from Local File (Wildcard Pattern Search allowed)
   

   "SELECT * FROM value 
   WITH (SRC='c:\data\Customers_*.json')"

7. Click on Guess Schema button and Click OK to accept detected schema.
   

   Talend – Configure REST API / JSON Source (Enter SQL Query / Guess Schema)

8. Now we will configure target in the next section.

Configure Target (Delimited File)

1. Now search for “FileOut” in the toolbox (Hit Enter). You will see tFileOutputDelimited so just select that for now and drag on the surface.
   

   Talend – Add File Output (tFileOutputDelimited)

2. Double click it to configure.
3. Enter correct file path (e.g. “C:/Talend/workspace/rest-api-output.csv” )
4. On Advanced Tab you can configure some additional settings (e.g. Throw an error if file already exists)

Connect and Run

1. Once you have configured Source and Target its time to connect them
2. Drag Source Port to Target to connect like below.
   

   Talend – Connect REST / JSON Source to File Target

3. Run the job
   

   Run Talend Job – Loading REST API data into File (Read JSON / XML / CSV)

4. That’s it. So in few clicks you loaded data from REST API to File in Talend Open Studio.

Write / Send data to REST API (POST Example)

There will be a time when you want to POST data to REST API service. Let’s check how to write POST query to submit data to REST API.

Just like how we did Read query in previous example, we can set POST Body in the SQL Query to send data. Use query like below and click Guess Schema button. If Blank Filter gives you no data error then make sure you remove Filter on Data Gateway Data source. (Notice we used \” to escape double quote inside query )

"SELECT * FROM _root_
WITH (
 METHOD='POST'
,HEADER='Content-Type: text/plain || first-header: AAA || second-header: BBB'
,SRC='http://httpbin.org/post'
,BODY='{id:1,notes:\"Some notes\"}'
,FILTER=''
)
"

Talend – POST data to REST API URL

SQL Query Examples

Click on below link to learn more writing SQL Query using ZappySys Drivers.

JSON / REST Driver – SQL Query Examples

XML / SOAP Driver – SQL Query Examples

CSV / REST Driver – SQL Query Examples

REST API / XML SOAP Pagination Settings for Talend

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

REST API / SOAP Web Service Connection Settings for Talend

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Other settings for REST API / SOAP XML Call in Talend

REST API / XML SOAP Performance Tips for Talend

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Calling SOAP Web Service in Talend

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

Conclusion

In this article, we used ZappySys Drivers to read data from JSON REST API / File. You can use same technique to consume SOAP / XML API or File. Try ODBC PowerPack for FREE and check out how easy it is to consume virtually any REST / SOAP API or Read from JSON / XML / CSV Files in  Talend Open Studio.

The post Read / Write REST API data in Talend (JSON / XML / SOAP) appeared first on ZappySys Blog.

In this article, we will get Office 365  data in Power BI using Microsoft Graph API and ODBC drivers. Specifically, we will get Excel file data in OneDrive, events from a Calendar and finally, a list in a note in OneNote and then load that data into a Power BI report. We will achieve that by creating an ODBC data source using ZappySys ODBC PowerPack and then querying corresponding Microsoft Graph API methods. ODBC PowerPack allows to query REST/SOAP APIs in an easy way and lets to view responses in a tabular format, which will be perfect to accomplish our goal.

Prerequisites

1. A Power BI account.
2. Wield basic Power BI skills.
3. Power BI Desktop installed.
4. ZappySys ODBC PowerPack installed.

Step By Step – Get Office 365 data in Power BI using Microsoft Graph API and ODBC

Set up an ODBC data source based on ZappySys JSON Driver

Add a data source

Configure the data source

Set up an ODBC data source based on ZappySys XML Driver

1. Before adding another data source based on ZappySys XML Driver, copy the connection string of the data source we just created:
2. Close the window and then add a new data source, but this time based on ZappySys XML Driver.
3. Once a window is opened, give it a name, e.g. System - XML.
4. Then press Load connection string button to load the connection string.
5. Paste the copied connection string and replace JSON text to XML, so that you have { ZappySys XML Driver } in DRIVER property.
6. Close the window.

Create a Power BI report

Reading files from OneDrive (getting Excel data)

Let’s say you have Stock.xlsx stored in Sample Documents folder, OneDrive with this kind of data and you want to show it in a Power BI report:

To have that data in a Power BI report, perform these steps:

1. Open Power BI Desktop and create a new Power BI report.
2. Create a data source based on System – JSON DSN (since we get a JSON response):
   
3. Then in SQL statement area enter this query:
   SELECT * FROM $
WITH(Src='https://graph.microsoft.com/v1.0/me/drive/root:/YOUR FOLDER/Stock.xlsx:/workbook/worksheets/SHEET NAME/usedRange'
,Filter='$.values[1:]'
,ArrayTransformType='TransformSimpleTwoDimensionalArray'
,ArrayTransColumnNameFilter='$.values[0][*]'
)
Using WITH clause lets us override the properties. You can generate the query above using the Query Builder:
   In this step we are overriding Url, Filter and array transformation fields you configured in JSON DSN. The Src value overrides the URL. The Filter property enables us to select the data we want to see in the report. We use [1:] in $.values[1:] to skip the first row because the first row contains the headers and we want only the data. You can configure this option in the Filter Options tab (read more about JSON path).We also have to make the transformation on the JSON and turn JSON arrays into rows. To accomplish that, we override ArrayTransformType. With ArrayTransColumnNameFilter property and [0][*] text, we specify that the first array of parent array will be the headers of the columns. You can read more about transformations in Parse multi-dimensional JSON array article.
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other OneDrive endpoints.
4. Click OK.
5. Drag a table into the report, select the columns and you should see a similar view:
   

Reading calendar entries

Now, let’s say you use a calendar in Office 365 and you want your appointments to be visible in a Power BI report:

1. To start, create a new Power BI report.
2. Then add a data source based on System – JSON DSN (again, we get the response in JSON):
   
3. In SQL statement section enter this query:
   SELECT * FROM $
WITH(Src='https://graph.microsoft.com/v1.0/me/events?$select=subject,start,end,location'
,Filter='$.value[*]'
,Header='Prefer: outlook.timezone="FLE Standard Time"'
)Here we also are overriding HTTP Header so that we get the event times not in UTC/GMT but for the specified timezone. It must be specified in Prefer: outlook.timezone=”{time-zone-string}” format (check an article on how to get a list of supported time zones).
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other Calendar endpoints.
4. Click OK.
5. Drag a table control into the report, select several columns and you should see a similar view:
   

Reading notes from OneNote

Suppose, you have these notes in OneNote and you want “Groceries list” to be shown in a Power BI report:

1. First, we will need to find NOTE ID we are trying to get. To do that, open ODBC Administrator, find System – JSON data source and hit Configure.
2. Then go to the Preview tab, and input this SQL query and hit Preview Data:
   SELECT * FROM $
WITH (Src='https://graph.microsoft.com/v1.0/me/onenote/pages'
,Filter='$.value[*]'
)
3. Once you get a result set, find your note name and copy its ID:
   
4. Then create a new Power BI report.
5. Create a new data source based on System – XML DSN (we are using XML-based DSN because the response will be in HTML):
   
6. In SQL statement section enter this query (use the NOTE ID you retrieved in the previous step):
   SELECT * FROM $
WITH(ElementsToTreatAsArray='div,p'
,Src='https://graph.microsoft.com/v1.0/me/onenote/pages/NOTE ID/content'
,Filter='$.html.body.div[0].p[1:]'
)

   
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other OneNote endpoints.
7. Click OK.
8. Drag and drop Multi-row card visual, select #text column and you should see a similar view:
   
9. We finished! Now you can publish your Power BI reports to Power BI service.

Conclusion

Today we learned how to call Microsoft Graph API methods using ZappySys ODBC PowerPack and load response data into Power BI without any ETL procedures. Particularly, we used OneDrive, Calendar and OneNote Graph API endpoints to achieve that. We created two ODBC data sources based on JSON and XML ODBC Driver to query those endpoints, which let us read data directly in a Power BI report. Finally, we examined how to refresh data source on Power BI using on-premises Power BI gateway.

References

https://powerbi.microsoft.com/en-us/

https://apps.dev.microsoft.com/

https://developer.microsoft.com/en-us/graph/graph-explorer/

https://docs.microsoft.com/en-us/graph/overview

https://docs.microsoft.com/en-us/graph/onedrive-concept-overview

https://docs.microsoft.com/en-us/graph/outlook-calendar-concept-overview

https://docs.microsoft.com/en-us/graph/integrate-with-onenote

The post Get Office 365 data in Power BI using Microsoft Graph API and ODBC appeared first on ZappySys Blog.