In our previous article we saw how to call PayPal API in SSIS. Now in this article, we will see How to Make PayPal API Call in ODBC from the SQL Server. This blog mainly focuses on ODBC approach but steps mentioned to call PayPal APIs can be useful for any developer regardless of which programming language or tool set you use. We will also see How to send basic Authorization header along with body in OAuth2.

We will go through the steps to Create PayPal Order from MS SQL Server.

In nutshell, this post will focus on how to make PayPal API Call using ODBC Driver.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. A first requirement, make sure that Microsoft Access installed
2. Make sure to have ZappySys ODBC PowerPack installed.
3. Credentials of PayPal API. Click here to create API.

What is PayPal?

PayPal is an American company operating a worldwide online payments system that supports online money transfers and serves as an electronic alternative to traditional paper methods like checks and money orders. The company operates as a payment processor for online vendors, auction sites, and many other commercial users, for which it charges a fee in exchange for benefits such as one-click transactions and password memory. PayPal’s payment system, also called PayPal, is considered a type of payment rail.

Get Started with PayPal API

If you have need to automate PayPal operations (e.g. read / write / update / delete ) then you can use PayPal REST API. Before you can integrate a PayPal product or solution, you must set up your development environment to get OAuth 2.0 client ID and secret credentials for the sandbox and live environments. You exchange these credentials for an access token that authorizes your REST API calls. To test your web and mobile apps, you create sandbox accounts.

Get PayPal credentials

To generate OAuth 2.0 credentials for the sandbox and live environments:

1. Log into Dashboard and type your PayPal personal or business account email and password.
2. In the REST API apps section, click Create App. The purpose of this app is to generate your credentials.
3. Type a name for your app and click Create App. The page shows your sandbox app information, which includes your credentials.
   

   Note: To show your live app information, toggle to Live.

4. Copy and save the client ID and secret for your sandbox app.
   

   PayPal Client ID and Secret

5. Review your app details and save your app.

Call PayPal Create Order API in SQL Server (T-SQL) using Data Gateway

Let’s start with an example. We use JSON Driver to make the call to PayPal API, we will Create the PayPal Order from SQL Server.

In REST API calls, include the URL to the API service for the environment:

• Sandbox: https://api.sandbox.paypal.com
• Live: https://api.paypal.com

1. To do this, first of all, we will open the ZappySys Data Gateway Configuration:
   

   Open ZappySys Data Gateway

2. Add the Native – ZappySys JSON Driver Data source.
   

   Add Gateway Data Source (Native JSON Driver)

3. Now it’s time to connect with PayPal. Let’s use JSON Driver Data source with the following URL:
   

   https://api.sandbox.paypal.com/v2/checkout/orders

   Body:
   

   {
     "intent": "CAPTURE",
     "purchase_units": [
       {
         "amount": {
           "currency_code": "USD",
           "value": "100.00"
         }
       }
     ]
   }

   

   PayPal : Create Order

4. Create Basic OAuth Client Credentials Grant (Configure ZS-OAuth Connection).
   In the SSIS ZS OAuth Connection Manager, Select Provider as Custom, select version as OAuth2 and select Grant Type as  Client Credentials Grant. Enter the credentials and access token url and click OK.
   

   https://api.sandbox.paypal.com/v1/oauth2/token

   

   ZS OAuth Connection – Client Credentials Grant

5. Furthermore, Now click on Test Connection button to test PayPal Connection.
   

   PayPal : Test Connection

6. Write your Query and Preview Data.
7. (Optional Step) If your Query use POST method (e.g. Create Invoice) to submit data then you have Save Metadata (Read More). We will use this meta file in later step when we call POST request.
   

   Generate Metadata File for REST API POST Operation using ZappySys API Driver (JSON / XML)

8. Now using Code Generator we will generate the query.
   

   Generate Query

Setup Linked Server in MS SQL Server for PayPal

Once you configured the data source in Gateway, we can now setup Linked Server in SQL Server to query API Calls.

1. Assuming you have installed SQL Server and SSMS. If not then get both for FREE from here: Get SQL Server Express and  Get SSMS
2. Open SSMS and connect to SQL Server.
3. Go to Root > Server Objects > Linked Servers node. Right click and click New Linked Server…

   

   Add Linked Server in SQL Server

4.  Now enter the linked server name, select Provider as SQL Native Client
5. Enter data source as 

   GatewayServerName, PORT_NUMBER

    where server name is where ZappySys Gateway is running (Can be same as SQL Server machine or remote machine). Default PORT_NUMBER is 5000 but confirm on Data gateway > General tab in case its different.
6. Enter Catalog Name. This must match name from Data gateway Data sources grid > Name column

   

   Configure Linked Server Provider, Catalog, Server, Port for ZappySys Data Gateway Connection

7. Click on Security Tab and select last option “Be made using this security context“. Enter your gateway user account here.
8. Click OK to save Linked Server

Create Order in PayPal from SQL Server (Call POST API Method)

1. To Create Order in PayPal we need to execute the query like this. Please refer to this link for more information: Create Order example.
   
   Important Note*: Below query uses META property (fully explained here). This is needed if you creating anew record using POST, PUT. If you don’t pass META='{xxxx}’ in query driver calls REST API two times (first time to parse metadata and second time to get actual data). So that you find some odd thing in PayPal if you execute the query without Meta then you will find two same Orders created in your PayPal as the driver makes two calls as mentioned above.

   -- In below query rather than WITH(META=''[{"Name": "Success","Type": "Int32"}]'' ....
   -- You can also use WITH(META=''c:\my-meta-file.txt'' .... We created this file in previous section
   
   SELECT * FROM OPENQUERY([MY_LINKED_SERVER_NAME]
   , 'SELECT * FROM $
   WITH(META=''[{"Name": "Success","Type": "Int32"}]''
   	 ,Src=''https://api.sandbox.paypal.com/v2/checkout/orders''
   	,RequestData=''{
     "intent": "CAPTURE",
     "purchase_units": [
       {
         "amount": {
           "currency_code": "USD",
           "value": "100.00"
         }
       }
     ]
   }''
   	,RequestContentTypeCode=''ApplicationJson''
   	,RequestMethod=''POST''
   )')

2. Here is the preview after you run REST API query in SQL Server. It will return created Order Id, Links and status. Notice that you can override default configuration by supplying many parameters in WITH clause.
   

   Create Order In PayPal Using SQL Query

3. You can also find this created Order by opening that URL in the browser.
   

   Created Order In PayPal

PayPal API integration in ODBC apps (e.g. Power BI, Excel, Informatica, Access, C#…)

So far we have looked at SQL Server integration only, but what if you like to use same powerful ZappySys API drivers part of ODBC PowerPack in other Apps like Power BI, Excel, MS Access, Informatica and many more… ?? Well not to worry if you know that App supports ODBC you can use ZappySys ODBC Driver same way described earlier… only difference is you dont need Linked Server and you dont have to use OPENQUERY statement when you use ODBC Drivers.

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

So in this blog, we learned how to generate a PayPal API Token, Load PayPal data in MS SQL Server table using ODBC JSON / REST API Driver in a very simple way. You can achieve many more functionalities with this tool. Check our blogs/articles on JSON / REST API Driver to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the use of PayPal API with our tools:

• Landing Page for ODBC JSON / REST API Driver, you can also find Tutorial Video here.
• Help File: Documentation of JSON Driver.
• PayPal API

The post How to Make PayPal API Call in ODBC / SQL Server (T-SQL) appeared first on ZappySys Blog.

In our previous blog we saw how to read JIRA data in SQL Server. Now let’s learn how to Import SAP S/4HANA OData Service Data Into Sql Server. SAP S/4HANA provides OData REST API interface to access data in your application using HTTP Protocol. We will use ODBC XML Driver to read SAP data and load into SQL Server.

So, let’s get started.

About SAP HANA / OData REST API Service

You can expose your SAP Data using ODATA REST API Service. Here is a good article which shows how to expose data as OData Service.   For more information on SAP OData Service feature check this SAP help page. If you are new to OData Standard then read here to know more how OData can facilitate data extraction using HTTP REST API. If you need Sample XML based OData Service then use below test URLs.

https://services.odata.org/Northwind/Northwind.svc/
https://services.odata.org/Northwind/Northwind.svc/Customers
https://services.odata.org/Northwind/Northwind.svc/Orders
https://services.odata.org/Northwind/Northwind.svc/Invoices
https://services.odata.org/Northwind/Northwind.svc/Products

Now let’s look at how to read SAP Data using ODBC Driver. At this point we assume you have exposed your data as OData Endpoint.

Read SAP S/4HANA data using XML Driver

First let’s look at steps to configure XML Driver. We will read data from SAP S/4HANA OData Service and then in next section we will look at how to load data into SQL Server or other target.

1. To do this, first of all, we will open the ZappySys Data Gateway Configuration:
   

   Open ZappySys Data Gateway

2. Add the Native – ZappySys XML Driver Data source.
   

   Add Gateway Data Source

3. Now edit that XML data source to configure it. Enter your OData Service URL its typically like below. Replace 3 parts with your own value (i.e. replace {MY-INSTANCE},  {MY-PROJECT}, {MY-TABLE})
   

   https://{MY-INSTANCE}/sap/opu/odata/sap/{MY-PROJECT}/{MY-TABLE}

   For example if you are hosting SAP HANA in Cloud Instance then your URL may look like below
   

   https://myXXXXXX-api.s4hana.ondemand.com/sap/opu/odata/sap/MyTestProject/PurchaseOrders

   And create New ZS-HTTP connection in it select Basic Authentication and enter your SAP HANA UserID / Password to call OData Service and select the desire filter.
   

   XML Driver : SAP S/4HANA OData Service Configuration

4. Now go to Data Format / Compression (Zip/GZip) tab and select Data Format as OData to get all the records.
   

   Data Format : Odata

5. Finally, now using Query Builder and Code Generator we will generate the query.
   

   Generate Query

6. That’s it we are ready to load SAP S/4HANA OData Service data to SQL Server.

Load SAP S/4HANA OData Service in MS SQL Server

SAP S/4HANA Integration with Other BI Tools (Power BI, Excel, SSRS, MS Access…)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

So in this blog, we learned how to Import SAP S/4HANA OData Service Data in MS SQL Server using ODBC XML / SOAP API Driver in a very simple way. You can achieve many more functionalities with this tool. Check our blogs/articles on XML File / REST API Driver https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/xml-soap-api-driver/ to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the use of SAP S/4HANA OData Service with our tools:

• SAP API Business Hub
• Landing Page for ODBC XML / SOAP API Driver, you can also find Tutorial Video here.
• Help File: Documentation of XML Driver.
• Blog/articles link: https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/xml-soap-api-driver/

The post Import SAP S/4HANA OData Service Data Into Sql Server via ODBC Driver appeared first on ZappySys Blog.

In our previous blog post we saw how to import REST / SOAP API in SQL Server. Using same concepts let’s look at how to import Bing Ads data into SQL Server. We will explore many techniques to call Bing Ads API and learn how to automate data extraction without doing any ETL. You can call Bing Ads API just using T-SQL code (Yes you heard it right). At the end of this article you will learn how to Download Performance Reports from Bing Ads Account without any coding (See sample screenshot below). Please go through full article carefully.

NOTE: Bing Ads now renamed as Microsoft Advertising but this article will use popular name for time being.

Import Bing Ads data into SQL Server (Using SSMS T-SQL Code)

About Bing Ads API / SOAP Web Service

If you are new to Bing Ads API then start from here. Bing Ads APIs are SOAP XML APIs. You can call it to automate many scenarios for Bing Ads. In this article we will mainly focus on read scenario (e.g. Read Performance Data) but you can use these techniques to write / update (e.g. Create New Campaign).

If you are new to SOAP WebService then Check this article to learn more. It will explain you how to use tools like SoapUI tool to craft SOAP Requests for Bing API or any other SOAP API.

For Bing API you can use Service WSDL files found here. Use URL ending ?wsdl to import in SoapUI.

Example Bing API Request

Here is raw API request for Bing API. This sample assumes you have completed 3-Legged OAuth Authorization to obtain AccessToken and RefreshToken (Explained later in this article). Below request uses AccessToken you might have already obtained via process like this. If you are using ZappySys tools then this will be generated automatically for you.

Request Method / URL:

POST https://clientcenter.api.bingads.microsoft.com/Api/CustomerManagement/v13/CustomerManagementService.svc

Headers:
Notice that for each Api you may need correct SOAPAction in Header. If you are not sure then use SoapUI Tool. Execute request in SoapUI and then go to Raw tab like this one.

Content-Type: text/xml; charset=utf-8
SOAPAction: "GetUser"

Body:
Notice that you have to replace AuthenticationToken and DeveloperToken in below request. AccessToken is placed inside AuthenticationToken tag. This Auth token is short lived and extracted via OAuth Process and DeveloperToken you can extract as mentioned in the previous section or get it from here.

  
    GetUser
    <%access_token%>
    1052Bxxxxxxxxxxxxxx
  
  
    
      
    
  

Response:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Header>
    <h:TrackingId xmlns:h="https://bingads.microsoft.com/Customer/v13">fddf3cb9-e4c2-4e75-9546-f3bc20ea91a8</h:TrackingId>
  </s:Header>
  <s:Body>
    <GetUserResponse xmlns="https://bingads.microsoft.com/Customer/v13">
      <User xmlns:a="https://bingads.microsoft.com/Customer/v13/Entities" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
        <a:ContactInfo>
          <a:Address>
            <a:City i:nil="true" />
            <a:CountryCode>US</a:CountryCode>
            <a:Id i:nil="true" />
            <a:Line1 i:nil="true" />
            <a:Line2 i:nil="true" />
            <a:Line3 i:nil="true" />
            <a:Line4 i:nil="true" />
            <a:PostalCode i:nil="true" />
            <a:StateOrProvince i:nil="true" />
            <a:TimeStamp i:nil="true" />
            <a:BusinessName i:nil="true" />
          </a:Address>
          <a:ContactByPhone>false</a:ContactByPhone>
          <a:ContactByPostalMail>false</a:ContactByPostalMail>
          <a:Email>someone@zappysys.com</a:Email>
          <a:EmailFormat i:nil="true" />
          <a:Fax i:nil="true" />
          <a:HomePhone i:nil="true" />
          <a:Id>48045678</a:Id>
          <a:Mobile i:nil="true" />
          <a:Phone1>111-222-3333</a:Phone1>
          <a:Phone2 i:nil="true" />
        </a:ContactInfo>
        <a:CustomerId>19112345</a:CustomerId>
        <a:Id>48012345</a:Id>
        <a:JobTitle i:nil="true" />
        <a:LastModifiedByUserId>48012345</a:LastModifiedByUserId>
        <a:LastModifiedTime>2019-06-04T14:22:35.38</a:LastModifiedTime>
        <a:Lcid>EnglishUS</a:Lcid>
        <a:Name>
          <a:FirstName>Someone</a:FirstName>
          <a:LastName>Good</a:LastName>
          <a:MiddleInitial i:nil="true" />
        </a:Name>
        <a:Password i:nil="true" />
        <a:SecretAnswer i:nil="true" />
        <a:SecretQuestion>None</a:SecretQuestion>
        <a:UserLifeCycleStatus>Active</a:UserLifeCycleStatus>
        <a:TimeStamp>AAAAAJiX/cU=</a:TimeStamp>
        <a:UserName>someone@zappysys.com</a:UserName>
        <a:ForwardCompatibilityMap i:nil="true" xmlns:b="http://schemas.datacontract.org/2004/07/System.Collections.Generic" />
      </User>
      <CustomerRoles xmlns:a="https://bingads.microsoft.com/Customer/v13/Entities" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
        <a:CustomerRole>
          <a:RoleId>41</a:RoleId>
          <a:CustomerId>19160000</a:CustomerId>
          <a:AccountIds i:nil="true" xmlns:b="http://schemas.microsoft.com/2003/10/Serialization/Arrays" />
          <a:LinkedAccountIds xmlns:b="http://schemas.microsoft.com/2003/10/Serialization/Arrays" />
          <a:CustomerLinkPermission i:nil="true" />
        </a:CustomerRole>
      </CustomerRoles>
    </GetUserResponse>
  </s:Body>
</s:Envelope>

POST http://myhost/api/v2/customers/new
X-SecretKey: Abcdxxxxxxxxxxxxxx
Content-Type: application/json

{ firstname: "Someone", lastname: "Good" }

{
 request_status: "CreatedOK",
 new_customer_id: "1234566"
}

Step-By-Step: How to Import Bing Ads Data into SQL Server

Now let’s look at how to import Bing Ads Data into SQL Server Table. This tutorial will require 10-15 mins of your time. At the end of this tutorial you will be able to Read Bing Ads Performance Report and Import data into SQL Server. Step listed in this tutorial may help you to call Bing API in other tools like SSIS, Power BI, SSRS, Tableau and so on.

So let’s get started.

Register OAuth App for Bing Ads API (Azure Active Directory App)

Very first thing to call any Bing API is to register OAuth App (Under Azure Active Directory Tab).

Here is how to register OAuth App which can be used to call Bing Ads API later on.

1. Login to Azure Portal:
   • Navigate to the Azure Portal and log in using your credentials.
2. Access Azure Active Directory:
   • In the left-hand menu, click on Azure Active Directory.
3. Register a New Application:
   • Go to App registrations and click on New registration.
     
   • Application Name: Enter a name for your application.
     
   • Supported Account Types: Choose the account types your app will support. For example, select Accounts in this organizational directory only if you need access to data in your organization only.
   • Redirect URI:
     • Set the type to Web.
     • In the textbox enter https://login.microsoftonline.com/common/oauth2/nativeclient as the Redirect URI or any other valid redirect URL, e.g., https://zappysys.com/oauth.
     • Use this Redirect URI in the Redirect URL grid row.
4. Save Client ID:
   • After registering the app, copy the Application (client) ID and paste it into the Client ID field in the API Connection Manager configuration.
5. Set Authorization & Token URLs:
   • Click on the Endpoints link in the App registration overview.
   • Authorization URL: Copy the OAuth 2.0 authorization endpoint (v2) URL (e.g., https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/authorize) and paste it into the Authorization URL field in the configuration grid.
   • Token URL: Copy the OAuth 2.0 token endpoint (v2) URL (e.g., https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/token) and paste it into the Token URL field.
6. Create a Client Secret:
   • In the Certificates & secrets tab, click New client secret.
   • Set an expiration period for the secret.
   • Copy the generated client secret and paste it into the Client Secret field in the API Connection Manager configuration.
7. Configure API Permissions:
   • Go to the API Permissions section.
   • Click on Add a permission, select Microsoft Graph, and choose Delegated Permissions.
   • Add the required permissions:
     • offline_access
     • openid
     • profile
     • Sites.Read.All
     • Sites.ReadWrite.All
     • User.Read
     • email
     

     Azure AD App – Required API Permissions

   • Grant Admin Consent for the permissions that require it.
8. Generate Tokens:
   • Use the Generate Token feature in the API Connection Manager to generate authentication tokens.
9. Use a Generic Account for Automation:
   NOTE: If you are planning to use your current data connection/token for automated processes, we recommend that you use a generic account for token generation when the login box appears (e.g. sales_automation@mycompany.com instead of bob_smith@mycompany.com). When you use a personal account which is tied to a specific employee profile and that employee leaves the company, the token may become invalid and any automated processes using that token will fail. Another potentially unwanted effect of using a personal token is incorrect logging; the API calls (e.g. Read, Edit, Delete, Upload) made with that token will record the specific user as performing the calls instead of an automated process.
10. That’s it!

Get Bing API Developer Token

Once we have OAuth App registered next thing is to obtain Developer Token for Bing Ads API. Click here to request new token or view existing Developer token.

Create new Bing Ads API Data Source in Gateway

Once you setup Data Gateway user we can move to next step. To call Bing API we need to setup Connection for XML Driver in ZappySys Data Gateway.

1. Click on Add New Data source. Name as BingAds and select Native – ZappySys XML Driver option.
   

   Add Gateway Data Source

2. Now click Edit to Configure Settings. For now we will only care OAuth Connection settings. All other options are overwritten in SQL Query.
   

   Edit Gateway Data Source Settings

3. When UI opens Click on Load ConnectionString and Paste below connection string after changing some items (Screenshot may be differ). Make sure to replace following attributes.RefreshTokenFilePath – Enter some valid file path which will hold refresh token
   ClientId – get Application Id from Azure Portal under App we created earlier)
   DeveloperToken – Replace 1052Bxxxxxxxxxxxx with your own devloper token.
   

   Load ZappySys Driver ConnectionString to configure UI

   Use below string and replace parameters 

   DRIVER={ZappySys XML Driver};
   DataPath='https://clientcenter.api.bingads.microsoft.com/Api/CustomerManagement/v13/CustomerManagementService.svc';
   DataConnectionType=OAuth;
   AuthUrl='https://login.microsoftonline.com/common/oauth2/v2.0/authorize';
   TokenUrl='https://login.microsoftonline.com/common/oauth2/v2.0/token';
   ReturnUrl='https://login.microsoftonline.com/common/oauth2/nativeclient';
   RefreshTokenFilePath='c:\xxxxxx_some_folder_xxxxxxx\BingAds_RefreshToken.txt';
   ScopeSeparator='{space}';
   Scope='https://ads.microsoft.com/ads.manage offline_access';
   ClientId='xxxxxxxxxxxxxx';
   UseCustomApp=True;
   RequestData='<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
     <s:Header>
       <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Customer/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Customer/v13"><%access_token%></h:AuthenticationToken>
       <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Customer/v13">1052Bxxxxxxxxxxxx</h:DeveloperToken>
       <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Customer/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Customer/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
     </s:Header>
     <s:Body>
       <GetUserRequest xmlns="https://bingads.microsoft.com/Customer/v13">
         <UserId i:nil="true" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       </GetUserRequest>
     </s:Body>
   </s:Envelope>';
   RequestContentTypeCode=TextXml;
   RequestMethod='POST';
   RequestHeaders='SOAPAction: "GetUser"'

4. Next to OAuth Click on Configure Settings
5. By default most of settings on OAuth connections should be pre-populated for you but still lets Notice few things. This is how to setup redirect URI
6. 

   Configure Redirect URI for Bing Ads API OAuth Connection

7. Here is where you setup Changing Refresh Token File path. if you dont supply this path then your refresh token extracted first time will expire in 60 days.
   

   Refresh Token File path (Only needed if API issues new Refresh Token when you request AccessToken)

8. Once you verify all settings click Generate Token and Finish the process. It will populate AccessToken and RefreshToken Fields if everything goes well.
   

   Configure Bing Ads API Connection (OAuth 2.0)

9. Click OK to save OAuth Connection UI to go back to main UI
10. You can now Click Test Connection to confirm everything
11. If you need to access Gateway from Other machine then go to Firewall tab and Click Add Rule
12. Click Save button on Gateway UI to save and restart the service.

Create a new Linked Server for Bing Ads API

Once we setup setup Bing Ads Data Source in Gateway we can move to next step which is create a Linked Server which points to Data gateway Data source we created. This will create a bridge between ZappySys Driver and SQL Server so you can call API via T-SQL.

1. If you are using Latest ODBC PowerPack then on Data gateway UI you will See App Integration Tab. Copy Code and run it in SSMS to create a new linked server.
   

   ZappySys Data Gateway code generator for Linked Server

2. If things go well when you execute code you may see some result in grid.

Run sample Queries to call Bing API in SQL Server / Import Data

Once Linked Server is created we can run various queries like below.

Example-1

--use all default settings from gateway data source UI
--Read
Select * from OPENQUERY( [YourLinkedServer] , 'select * from $') 

/*
--import
Select * into #tempTable from OPENQUERY( [YourLinkedServer] , 'select * from $') 
Select * from #tempTable
*/

Example-2

--use Filter to extract specific Node from response XML
Select * from OPENQUERY( [YourLinkedServer] , 
'select * from $
WITH(Filter=''$.s:Envelope.s:Body.GetUserRequest[*]'', 
  ElementsToTreatAsArray=''GetUserResponse'' 
   )'
)

Example-3

--Override URL, Headers and Body Settings. Change DeveloperToken in below request
--Header, Filter, URL, Request data is different for each Endpoint

SELECT * FROM OPENQUERY([YourLinkedServer]
, 'SELECT * FROM $
WITH(
	 ElementsToTreatAsArray=''GetUserResponse''
	,Src=''https://clientcenter.api.bingads.microsoft.com/Api/CustomerManagement/v13/CustomerManagementService.svc''
	,Filter=''$.s:Envelope.s:Body.GetUserResponse[*]''	
	,Header=''SOAPAction: "GetUser"''	
	,RequestData=''<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v13="https://bingads.microsoft.com/Customer/v13">
<soapenv:Header>
   <v13:DeveloperToken>1052xxxxxxxxxxxxx</v13:DeveloperToken>
   <v13:AuthenticationToken><%access_token%></v13:AuthenticationToken>
</soapenv:Header>
<soapenv:Body>
   <v13:GetUserRequest>
      <v13:UserId xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/>
   </v13:GetUserRequest>
</soapenv:Body>
</soapenv:Envelope>''
)')

Download Bing Ads Performance Data in SQL Server Table

Now lets look at more complex example to read data from Bing Performance Report. Calling This API is tricky due to many steps involved in data extraction. But not to worry if you use ZappySys Drivers. This is API is somewhat similar as Amazon MWS Report Request (See last section).

Here are high level steps needs to be performed to call API for Bing Ads Report (e.g. Performance Report)

1. Call SubmitGenerateReportResponse API to request new Report. This returns ReportRequestId
2. Now we have to keep checking every few seconds see report is ready or not. Call PollGenerateReport until Status field is Success from Pending
3. Once previous step returns Success extract Report Download URL (Its Zip File which may contain data in CSV or XML)

So lets get started to implement above login in a single T-SQL Query and load data into Table. There is a known limitation with Query Size we can send to data gateway from SSMS so we can use some tricks to read Body from File rather than hard coding in SQL. We can use IsMultiPart=True and then supply Filepath in RequestBody (Path must start with @ symbol)

1. Bing Report API needs 2 additional values along with DeveloperToken we got earlier. You need to supply AccountId and CustomerId part of Body so obtain it by visiting below page in your bing account.
   Visit https://ads.microsoft.com/cc/accounts
   

   Obtain Bing Ads AccountId, ClientId for APi calls

2. Create new XML file for first step in a Notepad. Enter below content and save as C:\Requests\BingAds\sql_body1.xml
   Replace Following Values as per your settingsCustomerAccountId (Found 2 places)
   CustomerId
   DeveloperToken
   

   <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
     <s:Header>
       <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Reporting/v13"><%access_token%></h:AuthenticationToken>
       <h:CustomerAccountId xmlns:h="https://bingads.microsoft.com/Reporting/v13">481xxxxxxxxxxx</h:CustomerAccountId>
       <h:CustomerId xmlns:h="https://bingads.microsoft.com/Reporting/v13">191xxxxxxxxxxx</h:CustomerId>
       <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Reporting/v13">1052Bxxxxxxxxxxx</h:DeveloperToken>
       <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
     </s:Header>
     <s:Body>
       <SubmitGenerateReportRequest xmlns="https://bingads.microsoft.com/Reporting/v13">
         <ReportRequest i:type="CampaignPerformanceReportRequest" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
           <ExcludeColumnHeaders>false</ExcludeColumnHeaders>
           <ExcludeReportFooter>false</ExcludeReportFooter>
           <ExcludeReportHeader>false</ExcludeReportHeader>
           <Format>Xml</Format>
           <ReportName>My Campaign Performance Report</ReportName>
           <ReturnOnlyCompleteData>false</ReturnOnlyCompleteData>
           <Aggregation>Hourly</Aggregation>
           <Columns>
             <CampaignPerformanceReportColumn>TimePeriod</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AccountId</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>CampaignId</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>CampaignName</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>CampaignStatus</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>DeviceType</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>BidMatchType</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>QualityScore</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AdRelevance</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>LandingPageExperience</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Revenue</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Assists</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>ExpectedCtr</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>DeliveredMatchType</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AveragePosition</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Conversions</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AdDistribution</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Network</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Clicks</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Impressions</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Ctr</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AverageCpc</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Spend</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>LowQualityClicks</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>LowQualityConversionRate</CampaignPerformanceReportColumn>
           </Columns>
           <Filter>
             <AccountStatus i:nil="true" />
             <AdDistribution i:nil="true" />
             <DeviceOS i:nil="true" />
             <DeviceType i:nil="true" />
             <Status i:nil="true" />
           </Filter>
           <Scope>
             <AccountIds xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
               <a:long>481xxxxxxxxxxx</a:long>
             </AccountIds>
             <Campaigns i:nil="true" />
           </Scope>
           <Time>
             <CustomDateRangeEnd i:nil="true" />
             <CustomDateRangeStart i:nil="true" />
             <PredefinedTime>Yesterday</PredefinedTime>
             <ReportTimeZone>PacificTimeUSCanadaTijuana</ReportTimeZone>
           </Time>
         </ReportRequest>
       </SubmitGenerateReportRequest>
     </s:Body>
   </s:Envelope>

    
3. Create another XML file for second step in a Notepad. Enter below content and save as C:\Requests\BingAds\sql_body2.xml
   Replace following values as per your need.
   

   <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
     <s:Header>
       <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Reporting/v13"><%access_token%></h:AuthenticationToken>
       <h:CustomerAccountId xmlns:h="https://bingads.microsoft.com/Reporting/v13">480xxxxxxxxxxx</h:CustomerAccountId>
       <h:CustomerId xmlns:h="https://bingads.microsoft.com/Reporting/v13">191xxxxxxxxxxx</h:CustomerId>
       <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Reporting/v13">1052Bxxxxxxxxxxx</h:DeveloperToken>
       <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
     </s:Header>
     <s:Body>
       <PollGenerateReportRequest xmlns="https://bingads.microsoft.com/Reporting/v13">
         <ReportRequestId>[$a.ReportRequestId$]</ReportRequestId>
       </PollGenerateReportRequest>
     </s:Body>
   </s:Envelope>

    
4. Now try to execute below query. This should generate Performance report and download it in a SQL table for you.
   

   select * into tmpBingAdsReport from OPENQUERY(Zs_BingAds,
   'SELECT c.* FROM $
   WITH(
   	--submit report request for XML format
   
   	 alias=''a'' 
   	,Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
   	,Filter=''$.s:Envelope.s:Body.SubmitGenerateReportResponse''
   	,RequestData=''@C:\Requests\BingAds\sql_body1.xml''
   	,IsMultiPart=''True''
   	,RequestContentTypeCode=''TextXml''
   	,Header=''SOAPAction: "SubmitGenerateReport"''
   	,RequestMethod=''POST''
   	,EnableBodyPlaceholderForMultiPart=''True''
   	,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''
   
   	--keep checking status until report is ready - returns final url when done
   
   	,join1_alias=''b''
   	,join1_Filter=''$.s:Envelope.s:Body.PollGenerateReportResponse.ReportRequestStatus''
   	,join1_RequestData=''@C:\Requests\BingAds\sql_body2.xml''
   	,join1_IsMultiPart=''True''
   	,join1_Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
   	,join1_RequestContentTypeCode=''TextXml''
   	,join1_Header=''SOAPAction: "PollGenerateReport"''
   	,join1_RequestMethod=''POST''
   	,join1_EnableBodyPlaceholderForMultiPart=''True''
   	,join1_EnableStatusCheck=''True'', 
   	,join1_StatucCheckMaxWaitSeconds=300, --wait max 5 mins?
   	,join1_StatucCheckIterationWaitSeconds=5, --check every 5 sec?
   	,join1_StatusSuccessValue=''Success'', --look for success word in response
   
   	--Download report (Zip file) and Parse rows
   
   	,join2_alias=''c''
   	,join2_DataConnectionType=''Default''
   	,join2_Src=''[$b.ReportDownloadUrl$]''
   	,join2_Filter=''$.Report.Table.Row[*]''
   	,join2_ElementsToTreatAsArray=''Row''	
   	,join2_RequestMethod=''GET''
   	,join2_IsMultiPart=''False''
   	,join2_FileCompressionType=''Zip''	
   )')
   
   select * from tmpBingAdsReport

   

   Import Bing Ads data into SQL Server

Multi Step Query to download Report

So in previous section we saw how to write a single query to get your Performance Report from Bing Ads. However there will be time you want to split 3 steps into seperate queries.

Here is how to do.

/*
RPC and RPC OUT options must be turned on to run this proc

EXEC master.dbo.sp_serveroption @server=N'Zs_BingAds', @optname=N'rpc', @optvalue=N'true'
EXEC master.dbo.sp_serveroption @server=N'Zs_BingAds', @optname=N'rpc out', @optvalue=N'true'
--Below needed to support EXEC + INSERT (dynamic query)
EXEC master.dbo.sp_serveroption @server=N'Zs_BingAds, @optname=N'remote proc transaction promotion', @optvalue=N'false'
Also 
1. Change Body file path
2. Change CustomerAccountId, CustomerId, DeveloperToken
*/

ALTER proc usp_Api_BingAds_Reports
as
--///////////////////////////////////////////////////////////////////////////////
--Step-1 - submit report request for XML format
--///////////////////////////////////////////////////////////////////////////////
create table #tmpBingAdsReport_Step1(ReportRequestId bigint)
INSERT into #tmpBingAdsReport_Step1
EXEC(
'SELECT * FROM $
WITH(
 Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,Filter=''$.s:Envelope.s:Body.SubmitGenerateReportResponse''
,RequestData=''@c:\BWProjects\Requests\BingAds\sql_body1.xml''
,IsMultiPart=''True''
,RequestContentTypeCode=''TextXml''
,Header=''SOAPAction: "SubmitGenerateReport"''
,RequestMethod=''POST''
,EnableBodyPlaceholderForMultiPart=''True''
,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''
)') AT zs_BingAds


--///////////////////////////////////////////////////////////////////////////////
--Step2 - keep checking status until report is ready - returns final url when done
--///////////////////////////////////////////////////////////////////////////////

declare @reportid varchar(100)
select top 1 @reportid=cast(ReportRequestId as varchar(100)) from #tmpBingAdsReport_Step1

create table #tmpBingAdsReport_Step2(ReportDownloadUrl varchar(500))
INSERT into #tmpBingAdsReport_Step2 
EXEC(
'SELECT * FROM $
WITH(
 Filter=''$.s:Envelope.s:Body.PollGenerateReportResponse.ReportRequestStatus''
,RequestData=''<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Header>
    <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
    <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Reporting/v13"><%access_token%></h:AuthenticationToken>
    <h:CustomerAccountId xmlns:h="https://bingads.microsoft.com/Reporting/v13">480xxxxxxxxx</h:CustomerAccountId>
    <h:CustomerId xmlns:h="https://bingads.microsoft.com/Reporting/v13">191xxxxxxxxx</h:CustomerId>
    <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Reporting/v13">105xxxxxxxxxxx</h:DeveloperToken>
    <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
    <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
  </s:Header>
  <s:Body>
    <PollGenerateReportRequest xmlns="https://bingads.microsoft.com/Reporting/v13">
      <ReportRequestId>'+ @reportid +'</ReportRequestId>
    </PollGenerateReportRequest>
  </s:Body>
</s:Envelope>''
--,IsMultiPart=''True''
,Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,RequestContentTypeCode=''TextXml''
,Header=''SOAPAction: "PollGenerateReport"''
,RequestMethod=''POST''
--,EnableBodyPlaceholderForMultiPart=''True''
,EnableStatusCheck=''True'', 
,StatucCheckMaxWaitSeconds=300, --wait max 5 mins?
,StatucCheckIterationWaitSeconds=5, --check every 5 sec?
,StatusSuccessValue=''Success'', --look for success word in response
,Meta=''[{"Name": "ReportDownloadUrl","Type": "String",Length:"300"}]''
)') AT zs_BingAds


declare @reportpath varchar(500)
select top 1 @reportpath=ReportDownloadUrl from #tmpBingAdsReport_Step2

select * from #tmpBingAdsReport_Step1
select * from #tmpBingAdsReport_Step2

--///////////////////////////////////////////////////////////////////////////////
--Step-3 - Get report file
--///////////////////////////////////////////////////////////////////////////////
--Static table
--//INSERT INTO BingAdsReport
EXEC(
'SELECT * FROM $
WITH(
 DataConnectionType=''Default''
,Src='''+ @reportpath +'''
,Filter=''$.Report.Table.Row[*]''
,ElementsToTreatAsArray=''Row''	
,RequestMethod=''GET''
,IsMultiPart=''False''
,FileCompressionType=''Zip''	
)') AT zs_BingAds


--select * from tmpBingAdsReport_Step3
--select * from openquery(Bing, 'select * from $')


/*
select * into tmpBingAdsReport from OPENQUERY(zs_BingAds,
'SELECT c.* FROM $
WITH(
--submit report request for XML format

alias=''a'' 
,Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,Filter=''$.s:Envelope.s:Body.SubmitGenerateReportResponse''
,RequestData=''@c:\BWProjects\Requests\BingAds\sql_body1.xml''
,IsMultiPart=''True''
,RequestContentTypeCode=''TextXml''
,Header=''SOAPAction: "SubmitGenerateReport"''
,RequestMethod=''POST''
,EnableBodyPlaceholderForMultiPart=''True''
,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''

--keep checking status until report is ready - returns final url when done

,join1_alias=''b''
,join1_Filter=''$.s:Envelope.s:Body.PollGenerateReportResponse.ReportRequestStatus''
,join1_RequestData=''@c:\BWProjects\Requests\BingAds\sql_body2.xml''
,join1_IsMultiPart=''True''
,join1_Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,join1_RequestContentTypeCode=''TextXml''
,join1_Header=''SOAPAction: "PollGenerateReport"''
,join1_RequestMethod=''POST''
,join1_EnableBodyPlaceholderForMultiPart=''True''
,join1_EnableStatusCheck=''True'', 
,join1_StatucCheckMaxWaitSeconds=300, --wait max 5 mins?
,join1_StatucCheckIterationWaitSeconds=5, --check every 5 sec?
,join1_StatusSuccessValue=''Success'', --look for success word in response

--Download report (Zip file) and Parse rows

,join2_alias=''c''
,join2_DataConnectionType=''Default''
,join2_Src=''[$b.ReportDownloadUrl$]''
,join2_Filter=''$.Report.Table.Row[*]''
,join2_ElementsToTreatAsArray=''Row''	
,join2_RequestMethod=''GET''
,join2_IsMultiPart=''False''
,join2_FileCompressionType=''Zip''	
)')

select * from tmpBingAdsReport

select * from openquery(Bing, 'select * from $')
*/
go

Conclusion

In this article we saw how to import Bing Ads API data in SQL Server without any coding using ZappySys ODBC PowerPack (SOAP / XML Driver). Download it it and try it for FREE to explore many API integration features not discussed in this article.

The post Import Bing Ads data into SQL Server (Performance Reports) appeared first on ZappySys Blog.

In our previous blog post we saw how to import rest API in SQL Server. Using same concepts lets look at how to import Amazon MWS Data into SQL Server. We will explore many techniques to call Amazon MWS API and learn how to automate Amazon MWS data extraction without doing any ETL. We will call MWS XML or CSV API just using T-SQL code. We will also learn how to Download Reports from by calling MWS API.

You can also refer other articles here (Power BI) and here (SSIS)

About Amazon MWS  API

If you want to sell something on Amazon you can use their e-commerce platform with some monthly fee. You can setup your entire online store / inventory using admin interface.

Amazon Marketplace Web Service (Amazon MWS) is an integrated web service API that helps Amazon sellers to programmatically exchange data on listings, orders, payments, reports, and more. Using these API you can read or write data from your Seller account and integrate it inside your own Systems (e.g. Reporting / ETL / BI tools).

Obtain MWS API Access Key / Secret and Seller ID

Very first thing to call any MWS API is to obtain Access Key, Secret Key and Seller ID (i.e. Merchant ID). To obtain this you first have to register a developer.

Here is where to look for your Merchant ID and Key / Secret.  Here is direct link to User Permissions page.

How to get Amazon MWS developer keys and know your Merchant ID (i.e. Seller ID)

Getting Started

Now let’s look at step by step instructions on how to call Amazon MWS API and then import into Power BI. In below steps we will use  ZappySys ODBC PowerPack so make sure it’s installed first.

Using ScratchPad to Test Amazon MWS API

Each MWS API has set of required and optional parameters. You can refer each API page for details. For example ListMatchingProducts API which we will use as an example. To make MWS API testing / learning easy, Amazon provides online testing tool called MWS ScratchPad. We will use this tool to craft API requests and use that information in ODBC driver SQL queries and later import data in Power BI.

1. Open MWS ScratchPad  by vising https://mws.amazonservices.com/scratchpad/index.html
2. Select Products API category and pick ListMatchingProducts API
3. Enter your SellerID (i.e. Merchant ID), Developer AccessKey (i.e. AWSAccessKeyId) and Secret Key we obtained in the previous section. Enter MarketId (e.g. ATVPDKIKX0DER for USA Market) and other parameters as below and Click Submit to see response.
   

   Using Amazon MWS ScratchPad (API Testing Tool – Created by Amazon)

4. Click on the Response Details Tab to extract some important information we will use for ODBC configuration. Notice attributes in red rectangles. They are the ones which we will need in the POST request Body. ZappySys Drivers supply many common parameters automatically but any API endpoint specific parameter must be supplied in the Request Body. For example you dont have to supply SignatureVersion, Timestamp, Signature or SignatureMethod
   

   Calling Amazon MWS API in Scratchpad Testing Tool (Response Details Tab) – ListMatchingProducts Example

This API requires Market ID parameter so enter it as per your Market where you selling. For example USA marketplaceid = ATVPDKIKX0DER. You can lookup correct MarketplaceId by checking this help (endpoints / marketplaces).

Setup Amazon MWS API Connections

Create XML Driver Connection for Amazon MWS API (e.g. RequestReport )

Some APIs we need to call in this article is XML format and some API (e.g. GetReport) is CSV format – Tab delimited so we need to create 2 different connections using CSV driver and XML driver. So lets get started with Amazon MWS Connection for XML API.

1. Click on Add New Data source. Name as AMAZON-MWS-XML and select Native – ZappySys XML Driver option.
   

   Add Gateway Data Source

2. Now click Edit to Configure Settings. For now we will only care OAuth Connection settings. All other options are overwritten in SQL Query.
   

   Edit Gateway Data Source Settings

3. When UI opens for Driver Enter MWS API Service URL as below. Notice that we added /Products/2011-10-01 (This was extracted from previous section). You can check full list of Amazon MWS Endpoints here. Because we are connecting to USA region we will use below URL.
   

   https://mws.amazonservices.com/Products/2011-10-01

4. Select OAuth from the Connection dropdown and Click Configure Settings
5. On the Connection UI, Select OAuth Provider Amazon Market Web Service (MWS)
6. Enter your API Access Key and Secret Key (obtained from here)
7. Your connection will look like below.
   

   ODBC Connection Credentials for Amazon MWS API (Access Key, Secret Key, URL Endpoint)

8. Also configure Retry options to handle API limit reached error when you call API too often. For Amazon MWS we get Status code 503 when request is throttled so lets narrow down on which error we want to retry.
   

   Retry Options

9. Click OK to save OAuth Connection UI to go back to main UI
10. Now Go to Throttling Tab and enter some delay (e.g. enter 20000 for 20sec delay after each request). This will slow down API calls so we don’t face errors about API Limit reached.
11. Click OK to Save UI
12. Click Save button on Gateway UI to save and restart

Create CSV Driver Connection for Amazon MWS API (e.g. GetReport )

Now lets create a connection CSV API (e.g. GetReport ).

Follow almost same steps as previous section except use CSV Driver when you click on Add new Data source select Native – ZappySys CSV Driver and name data source as AMAZON-MWS-CSV . As we mentioned for now we mostly want to setup Connection and not worry about Request Body, Pagination etc.

Create Linked Server for SQL Server

Run following Script in SSMS to create Linked servers. Change ######## with Your gateway password. Change localhost with machine name or IP where ZappySys Gateway Is running. Use localhost if SQL Server and Gateway both on the same machine.

USE [master]
GO

EXEC master.dbo.sp_addlinkedserver @server = N'AMAZON-MWS-XML', @srvproduct=N'', @provider=N'SQLNCLI', @datasrc=N'localhost,5000', @catalog=N'AMAZON-MWS-XML'
EXEC master.dbo.sp_addlinkedsrvlogin @rmtsrvname=N'AMAZON-MWS-XML',@useself=N'False',@locallogin=NULL,@rmtuser=N'tdsuser',@rmtpassword='########'
GO

EXEC master.dbo.sp_addlinkedserver @server = N'AMAZON-MWS-CSV', @srvproduct=N'', @provider=N'SQLNCLI', @datasrc=N'localhost,5000', @catalog=N'AMAZON-MWS-CSV'
EXEC master.dbo.sp_addlinkedsrvlogin @rmtsrvname=N'AMAZON-MWS-CSV',@useself=N'False',@locallogin=NULL,@rmtuser=N'tdsuser',@rmtpassword='########'
GO

If you want to know how to configure Linked Server using UI then check this article.

Amazon MWS ListOrders Example

Now lets look at how to call ListOrders request with pagination in SQL Server. Create a following Stored Procedure to List MWS Orders. If you want to extract many orders then call Report rather than this proc (See next section for generating reports)

USE MyDatabase
GO
/*
Example: 
 exec usp_MWS_ListOrders @SellerId='Axxxxxxx', @MarketplaceId ='Bxxxxxxx',@CreatedAfter='2019-01-01'  
*/
Create proc [dbo].[usp_MWS_ListOrders]
	@SellerId varchar(100),
	@MarketplaceId varchar(100),
	@CreatedAfter varchar(100),
	@CreatedBefore varchar(100)=null,
	@MaxRows int=0
as
declare @sql varchar(max) 
declare @limitClase varchar(100)=''
if(@MaxRows>0)
	set @limitClase	=' LIMIT ' + cast(@MaxRows as varchar(20))

--//Confirm ISO Date format 2019-06-01T04%3A00%3A00Z  rather 2019-06-01T04:00:00Z
if(isnull(@CreatedAfter,'')<>'')
begin
	set @CreatedAfter='&CreatedAfter=' + @CreatedAfter
	set @CreatedAfter=replace(@CreatedAfter,':','%3A')
	if(@CreatedAfter NOT LIKE @CreatedAfter + 'T%')
		set @CreatedAfter=@CreatedAfter + 'T00%3A00%3A00Z'
	if(@CreatedAfter NOT LIKE @CreatedAfter + '%Z')
		set @CreatedAfter=@CreatedAfter + 'Z'
	--//T04%3A00%3A00Z
end
if(isnull(@CreatedBefore,'')<>'')
begin
	set @CreatedBefore='&CreatedBefore=' + @CreatedBefore
	set @CreatedBefore=replace(@CreatedBefore,':','%3A')
	if(@CreatedBefore NOT LIKE @CreatedBefore + 'T%')
		set @CreatedBefore=@CreatedBefore + 'T00%3A00%3A00Z'
	if(@CreatedBefore NOT LIKE @CreatedBefore + '%Z')
		set @CreatedBefore=@CreatedBefore + 'Z'
end

set @CreatedAfter = isnull(@CreatedAfter,'')
set @CreatedBefore = isnull(@CreatedBefore,'')


set @sql = 'SELECT * FROM OPENQUERY([AMAZON-MWS-XML]
, ''
SELECT ReportRequestId FROM $
' + @limitClase +' 
WITH(
     Src=''''https://mws.amazonservices.com/Orders/2013-09-01''''
	,Filter=''''$.RequestReportResponse.RequestReportResult.ReportRequestInfo''''
	,ElementsToTreatAsArray=''''Order''''
	,RequestData=''''Action=ListOrders[$tag$]' + @CreatedAfter + @CreatedBefore + '&MarketplaceId.Id.1='+ @MarketplaceId +'&SellerId='+ @SellerId +'''''
	,RequestMethod=''''POST''''
	,NextUrlAttributeOrExpr=''''$.ListOrders[$tag$]Response.ListOrders[$tag$]Result.NextToken''''
	,NextUrlSuffix=''''&NextToken=<%nextlink_encoded%>''''
	,WaitTimeMs=20000 -- slow down to avoid throttling. 6 requests per min
	,HasDifferentNextPageInfo=''''True''''
	,EnablePageTokenForBody=''''True''''
	,PagePlaceholders=''''body=|ByNextToken;filter=|ByNextToken''''
	--,DataConnectionType=''''OAuth''''
	--,ScopeSeparator=''''{space}''''
	--,ServiceProvider=''''AmazonMWS''''
	--,ClientId=''''AKIxxxxxx''''
	--,ClientSecret=''''AKIxxxxxx''''
	--,UseCustomApp=''''True''''
	,CacheFileLocation=''''c:\temp\mws-orders.cache''''
	,CacheEntryTtl=300 --send every 5 mins else use from cache
	,IncludeParentColumns=''''False''''
)
'')'
print @sql

EXECUTE( @sql)

Here is how to call this proc. Change SellerId, MarketplaceId and Date.

exec usp_MWS_ListOrders @SellerId='Axxxxxxx', @MarketplaceId ='Bxxxxxxx',@CreatedAfter='2019-01-01'

Download Amazon MWS Report data in SQL Server (CSV / XML Format)

Now lets look at how to extract data from Custom Report API (You must need ODBC PowerPack v1.1.1 or higher). However reading data from Reports not single step process because report generation is Job style API. Which means you send report request and wait  until its done. Once Report ready you have to read in CSV format or XML. Some Reports only available in CSV format. Check this post to understand how complex it can be to get data. We will make it simple for you to understand this in 3 steps. Basically calling reports requires minimum 3 API calls (see below)

1. Create a new report request – Cal RequestReport API . It returns ReportRequestId (You can use it in the next step)
2. Check Report Status see its done – Call GetReportRequestList API (Pass ReportRequestId got in the previous step to get data for only one Report request we care). Keep checking Report Status every few seconds until you get _DONE_  or _DONE_NO_DATA_ status in response.
3. Read Report Data – Call GetReport. This API call returns CSV or XML data. So use correct Driver for this step. If CSV data is returned then we must use CSV Driver rather than XML Driver.

Method 1 – Single query approach (New Driver)

In new driver ZappySys added few new features which makes it possible to call above 3 steps process in a single query. New query syntax offers StatusCheck and Mixing Content Formats using same driver. Because Most MWS Reports are in CSV Format we have to use CSV Driver but oddly first 2 steps listed in above sections are in XML Format so we have to use XML driver. Technically we need atleast 2 steps (i.e 2 queries) but we will show you how to achieve using single query as below.

1. Assuming you have followed steps to create a linked server AMAZON-MWS-CSV (see previous section)
2. Run below query. Replace necessary parameters as per your need. (i.e MarketplaceId , SellerId, CreatedAfter date)
   

   --- //////////////////////////////////////////////////////////////////////////
   -- Change _GET_MERCHANT_LISTINGS_DATA_ to something else if you need different report type (all 3 steps) -- https://docs.developer.amazonservices.com/en_US/reports/Reports_ReportType.html
   -- Replace MarketplaceId in all 3 steps (i.e for USA use ATVPDKIKX0DER) --https://docs.developer.amazonservices.com/en_US/dev_guide/DG_Endpoints.html
   -- Replace SellerId in all 3 steps
   -- Remove or Change CreatedAfter in first step (or use CreatedBefore parameter instread of CreatedAfter)
   --- //////////////////////////////////////////////////////////////////////////
   select * into tmpMWS_Report from OPENQUERY([AMAZON-MWS-CSV],
   '
   SELECT s3.* FROM $
   WITH(
   Src=''https://mws.amazonservices.com/Reports/2009-01-01'',RequestMethod=''POST''
   ,RequestData=''Action=RequestReport&ReportType=_GET_MERCHANT_LISTINGS_DATA_&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxx&CreatedAfter=2017-12-31T04%3A00%3A00Z''
   ,IncludeParentColumns=''False''
   ,EnableRawOutputModeSingleRow=''True''
   ,RawOutputFilterExpr=''//*[local-name()="ReportRequestId"]'' 
   ,RawOutputDataRowTemplate=''{ReportRequestId:"[$1]"}'' 
   ,RawOutputExtractMode=''Xml'' 
   ,WaitTimeMs=5000
   ,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''
   ,Alias=''s1''
   
   ,Join1_alias=''s2''	
   ,Join1_Src=''https://mws.amazonservices.com/Reports/2009-01-01'',Join1_RequestMethod=''POST''
   ,Join1_RequestData=''Action=GetReportRequestList&ReportRequestIdList.Id.1=[$s1.ReportRequestId$]&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxxxxx''
   ,Join1_EnableRawOutputModeSingleRow=''True''
   ,Join1_RawOutputFilterExpr=''//*[local-name()="ReportProcessingStatus"]||//*[local-name()="GeneratedReportId"]'' 
   ,Join1_RawOutputDataRowTemplate=''{ReportProcessingStatus:"[$1]",GeneratedReportId:"[$2]"}'' 
   ,Join1_RawOutputExtractMode=''Xml'' 
   ,Join1_EnableStatusCheck=''True'', 
   ,Join1_StatucCheckMaxWaitSeconds=300,
   ,Join1_StatucCheckIterationWaitSeconds=25,
   ,Join1_StatusSuccessValue=''_DONE_''
   ,Join1_StatusFailedValue=''_DONE_NO_DATA_|_CANCELLED_''
   
   ,Join2_Alias=''s3''
   ,Join2_Src=''https://mws.amazonservices.com/Reports/2009-01-01'',Join2_RequestMethod=''POST''
   ,Join2_RequestData=''Action=GetReport&ReportId=[$s2.GeneratedReportId$]&MarketplaceId=ATVPDKIKX0DER&SellerId=Axxxxxxxxxxxxxx''	
   ,Join2_ColumnDelimiter=''{TAB}''
   ,Join2_ResponseCharset=''Windows-1252''	 
   ,Join2_EnableStatusCheck=''False'' 
   ,Join2_EnableRawOutputModeSingleRow=''False''
   ,Join2_WaitTimeMs=10000
   )') 
   go
   select * from tmpMWS_Report

    

Performance Tip

If you like to improve performance and dont want to use SELECT INTO approach to load into temp table as above then you can use EXEC (….) AT YourLinkedServerName  as below.

Advantage of this method is your Query speed will increase because system calls API only once when you call EXEC AT. In OPENROWSET it needs to call above query twice (Once to obtain metadata and once to get data).

INSERT INTO tmpMWS_Report
EXEC('select s3.* .................. ') AT [AMAZON-MWS-CSV]

Method 2 – Multiple query approach (For old driver)

Now lets see how to load Amazon MWS data in SQL Server using old driver (v1.1.2 or older) which didnt support single query approach like previous section. If you are using older driver or for some reason you like to split steps into multiple procedures then use below approach.

usp_MWS_Report  Stored Proc ( Generate ReportRequest, Wait until done, Read data )

USE MyDatabase
GO

/*
Examples: 
Get ReportType from here https://docs.developer.amazonservices.com/en_UK/reports/Reports_ReportType.html

exec usp_MWS_Report @MarketplaceId ='ATVPDKIKX0DER', @SellerId='AZxxxxxxxx', @ReportType = '_GET_MERCHANT_LISTINGS_DATA_'
exec usp_MWS_Report @MarketplaceId ='ATVPDKIKX0DER', @SellerId='AZxxxxxxxx', @ReportType = '_GET_FLAT_FILE_ORDER_REPORT_DATA_'

--Do not Throw Error if no data found (return empty resultset or run custom sql)
exec usp_MWS_Report @MarketplaceId ='ATVPDKIKX0DER', @SellerId='AZxxxxxxxx', @ReportType = '_GET_FLAT_FILE_ORDER_REPORT_DATA_',@ThrowErrIfNoData=0,@EmptyRowSql='select top 0 * from Orders'
--Supply date range
exec usp_MWS_Report @MarketplaceId ='ATVPDKIKX0DER', @SellerId='AZxxxxxxxx', @ReportType = '_GET_MERCHANT_LISTINGS_DATA_', @StartDate='2019-01-01T00:00:00Z', @EndDate='2019-01-31T00:00:00Z'

--loading into table 

--//////////////////////////////////
— Step-1 : Create Local Linked server to point to self
--//////////////////////////////////
USE [master]
GO
--change default catalog
--For MSSQL 2012, 2014, 2016, 2017, and 2019 use @provider=N'MSOLEDBSQL' below (SQL Server Native Client 11.0)---
EXEC master.dbo.sp_addlinkedserver @server = N'ME', @srvproduct=N'', @provider=N'SQLNCLI11', @datasrc=N'localhost', @catalog=N'Northwind'
--For MSSQL 2022 or higher use @provider=N'MSOLEDBSQL' below (Microsoft OLE DB Driver for SQL Server)---
--EXEC master.dbo.sp_addlinkedserver @server = N'ME', @srvproduct=N'', @provider=N'MSOLEDBSQL', @datasrc=N'localhost', @catalog=N'Northwind'

EXEC master.dbo.sp_addlinkedsrvlogin @rmtsrvname=N'ME',@useself=N'True',@locallogin=NULL,@rmtuser=NULL,@rmtpassword=NULL
GO
--//////////////////////////////////
— Step-2 : Enable RPC OUT and Disable MSDTC
--//////////////////////////////////
EXEC sp_serveroption 'ME', 'rpc out', true;  
GO
--Below needed to support EXEC + INSERT (dynamic query)
EXEC sp_serveroption 'ME', 'remote proc transaction promotion', false;
GO
--test simple query
select * from OPENQUERY(ME,'select 1')
GO

--//////////////////////////////////
— Step-3 : Load data into table from MWS Report
--//////////////////////////////////
SELECT * INTO #amazon_mws_report
FROM OPENQUERY(me,'
SET NOCOUNT ON;
EXEC usp_MWS_Report 
   @Columns=''[item-name], [item-description], [listing-id]'',
   @MarketplaceId =''ATVPDKIKX0DER'', 
   @SellerId=''Axxxxxxxxxxxxxxxxx'', 
   @ReportType = ''_GET_MERCHANT_LISTINGS_DATA_''
WITH RESULT SETS 
(
	(itmname varchar(1000),itmdesc varchar(1000),itmid varchar(1000))
)
')

select * from #amazon_mws_report


*/
CREATE proc [dbo].[usp_MWS_Report]
	@MarketplaceId varchar(100),
	@SellerId varchar(100),
	@ReportType varchar(100)='_GET_MERCHANT_LISTINGS_DATA_',
	@StartDate varchar(100)=null,
	@EndDate varchar(100)=null,
	@ReportId bigint=null,  ---i.e. 14783134522018025
	@ThrowErrIfNoData bit=1,
	@EmptyRowSql varchar(1000)=null, --SQL to run if no data found
	@Columns varchar(8000)='*'  --report columns which goes in final SELECT  e.g. [item-name], [item-description], [listing-id] ....
as


declare @retGeneratedReportId bigint
set @retGeneratedReportId=@ReportId
---////// STEP 1 /////////
if(@ReportId is null)
begin
	create table #step1(ReportRequestId bigint)
	insert into #step1
	exec usp_MWS_RequestReport @ReportType,@MarketplaceId,@SellerId,@StartDate,@EndDate

	--select * from #step1 --debug

	declare @retReportRequestId bigint
	select @retReportRequestId=ReportRequestId from #step1

	---////// STEP 2 /////////
	declare @status varchar(100)
	declare @cnt int
	create table #step2(ReportProcessingStatus varchar(100),GeneratedReportId bigint)
	
	--Keep looping until status is done or we waited too long
	while(isnull(@status,'') IN ('','_SUBMITTED_','_IN_PROGRESS_') )
	begin
		truncate table #step2
		insert into #step2
		exec usp_MWS_GetReportIdAndStatus @retReportRequestId,@MarketplaceId,@SellerId 
	
		select @status=ReportProcessingStatus, @retGeneratedReportId=GeneratedReportId from #step2
		if(isnull(@status,'') IN ('','_SUBMITTED_','_IN_PROGRESS_') )			
		begin
			--amazon says dont check often https://docs.developer.amazonservices.com/en_UK/reports/Reports_GetReportRequestList.html
			Print 'Status='+  isnull(@status,'') +'. Checking after 45 sec... #' + cast(@cnt as varchar(10))
			WAITFOR DELAY '00:00:45'			
		end
		else
			print @status

		set @cnt=@cnt+1

		--if(@cnt>50)
			--//throw err?
	end
	--select * from #step2 --debug
end

if(@retGeneratedReportId is null and @status='_DONE_NO_DATA_')
begin
	if(@ThrowErrIfNoData=1)
		RAISERROR('Cannot generate report - No data found for specified criteria',16,1); 
	else if (isnull(@EmptyRowSql,'')<>'')
		execute(@EmptyRowSql)
end
else if(@retGeneratedReportId is null)
	RAISERROR('Cannot generate report - Unknown error (either report got cancelled or other error occurred)',16,1); 

	
---////// STEP 3 /////////
--If xml format of report data then use XML driver else CSV (most are Tab delimited reports in CSV format)
--exec usp_MWS_GetReportDataXml @retGeneratedReportId,@MarketplaceId,@SellerId
exec usp_MWS_GetReportDataCsv @retGeneratedReportId,@MarketplaceId,@SellerId,@Columns

drop table #step1
drop table #step2

GO

usp_MWS_RequestReport  Stored Proc ( Step1 – Create Report Request )

USE [MyDatabase]
GO

CREATE proc [dbo].[usp_MWS_RequestReport]
	@ReportType varchar(100),
	@MarketplaceId varchar(100),
	@SellerId varchar(100),
	@StartDate varchar(100)=null,
	@EndDate varchar(100)=null
as

/*Learn more : https://docs.developer.amazonservices.com/en_UK/reports/Reports_RequestReport.html */

--//Confirm ISO Date format 2019-06-01T04%3A00%3A00Z  rather 2019-06-01T04:00:00Z
if(isnull(@StartDate,'')<>'')
begin
	set @StartDate='&StartDate=' + @StartDate
	set @StartDate=replace(@StartDate,':','%3A')
	if(@StartDate NOT LIKE @StartDate + 'T%')
		set @StartDate=@StartDate + 'T00%3A00%3A00Z'
	if(@StartDate NOT LIKE @StartDate + '%Z')
		set @StartDate=@StartDate + 'Z'
	--//T04%3A00%3A00Z
end
if(isnull(@EndDate,'')<>'')
begin
	set @EndDate='&EndDate=' + @EndDate
	set @EndDate=replace(@EndDate,':','%3A')
	if(@EndDate NOT LIKE @EndDate + 'T%')
		set @EndDate=@EndDate + 'T00%3A00%3A00Z'
	if(@EndDate NOT LIKE @EndDate + '%Z')
		set @EndDate=@EndDate + 'Z'
end

set @StartDate = isnull(@StartDate,'')
set @EndDate = isnull(@EndDate,'')

declare @sql varchar(max) 
set @sql = 'SELECT * FROM OPENQUERY([AMAZON-MWS-XML]
, ''
SELECT ReportRequestId FROM $
WITH(
	Src=''''https://mws.amazonservices.com/Reports/2009-01-01'''',RequestMethod=''''POST''''
	--,ElementsToTreatAsArray=''''Product''''
	,Filter=''''$.RequestReportResponse.RequestReportResult.ReportRequestInfo''''
	,RequestData=''''Action=RequestReport&ReportType=' + @ReportType  + '&MarketplaceId='+ @MarketplaceId +'&SellerId='+ @SellerId + @StartDate + @EndDate +'''''
	,CacheFileLocation=''''c:\temp\mws-cache-step-1.cache''''
	,CacheEntryTtl=300 --send every 5 mins else use from cache
	,WaitTimeMs=''''300''''
	,IncludeParentColumns=''''False''''
	,Meta=''''[{"Name": "ReportRequestId","Type": "Int64"}]''''
)'')'
--print @sql

EXECUTE( @sql)
--sleep 25 seconds after this - hope report is ready by that time else We have to loop
WAITFOR DELAY '00:00:25'  --lets wait 30 sec... incase we get luck and report is done?
GO

usp_MWS_GetReportIdAndStatus Stored Proc ( Step2 – Get Report Id and Report Status )

USE MyDatabase
GO


CREATE proc [dbo].[usp_MWS_GetReportIdAndStatus]
	@ReportRequestId bigint,
	@MarketplaceId varchar(100),
	@SellerId varchar(100)
as

declare @sql varchar(max) 
set @sql = 'SELECT * FROM OPENQUERY([AMAZON-MWS-XML]
, ''SELECT *  FROM $
WITH(
	 Src=''''https://mws.amazonservices.com/Reports/2009-01-01'''',RequestMethod=''''POST''''
	,ElementsToTreatAsArray=''''ReportRequestInfo''''
	,Filter=''''$.GetReportRequestListResponse.GetReportRequestListResult.ReportRequestInfo''''
	,RequestData=''''Action=GetReportRequestList&ReportRequestIdList.Id.1=' + cast(@ReportRequestId as varchar(20)) + '&MarketplaceId='+ @MarketplaceId +'&SellerId='+ @SellerId +'''''
	,WaitTimeMs=''''300''''
	,IncludeParentColumns=''''False''''
	--,CacheFileLocation=''''c:\temp\mws-cache-step-2.cache''''
	--,CacheEntryTtl=10  --do not use from cache too old.. but cache still prevents too many requests
	,Meta=''''[{"Name": "ReportProcessingStatus","Type": "String", Length: 100},{"Name": "GeneratedReportId","Type": "Int64"}]''''	
)'')'

--print @sql

EXECUTE( @sql)

GO

usp_MWS_GetReportDataCsv Stored Proc (Step3 – Get Report Data in CSV format )

Last step in report generation is download the report data in CSV format.

USE MyDatabase
GO

CREATE proc [dbo].[usp_MWS_GetReportDataCsv]
	@ReportId bigint,
	@MarketplaceId varchar(100),
	@SellerId varchar(100),
	@Columns varchar(8000)='*'

as

declare @sql varchar(max) 
set @sql = 'SELECT '+ @Columns +' FROM OPENQUERY([AMAZON-MWS-CSV], 
''SELECT * FROM $
WITH(
    Src=''''https://mws.amazonservices.com/Reports/2009-01-01'''',RequestMethod=''''POST''''
	,RequestData=''''Action=GetReport&ReportId=' + cast(@ReportId as varchar(20)) + '&MarketplaceId='+ @MarketplaceId +'&SellerId='+ @SellerId +'''''
	,WaitTimeMs=''''300''''
	,ColumnDelimiter=''''{TAB}''''
	,ResponseCharset=''''Windows-1252''''	
	,CacheFileLocation=''''c:\temp\mws-cache-step-3.cache''''
	,CacheEntryTtl=300 --once report is generated lets try to read from cache 
	)''
)'

print @sql

EXECUTE( @sql)
GO

Example – Import data from Amazon MWS into SQL Server Table

Finally when we have all stored procs created we can load into target table as below. In our previous article we saw how to use SELECT INTO along with OPENQUERY to load data into new temp table or insert into some existing table. However when you want to load Stored Proc output into table its not straight forward specially when we calling dynamic SQL.

Three tricks worth mentioning here which makes it possible to load data from stored proc into temp table dynamically.

1. Create a linked server to Self so we can use OPENQUERY against to call stored proc so its easy to use SELECT INTO.
2. Use WITH RESULT SETS to describe metadata of output. This is needed because we are using dynamic SQL inside proc and WITH RESULT SETS describes the metadata.
3. Use of SET NO COUNT ON to avoid error like below
   

   "SOMESERVERNAME" indicates that either the object has no columns 
   or the current user does not have permissions on that object

4. If you dont do above tricks then you may see errors like this (INSERT is not allowed when we use DYNAMIC SQL inside nested Stored Procs)

To load output into table you must know column names from the output.  If you dont know you can try to run stored proc without @Columns like below.

EXEC usp_MWS_Report @MarketplaceId =''ATVPDKIKX0DER'', @SellerId=''Axxxxxxxxxxxxx'', @ReportType = ''_GET_MERCHANT_LISTINGS_DATA_''

Once we know columns from report, you can run following script to insert into table.

Full Script – Load MWS Report Output into Table

--//////////////////////////////////
— Step-1 : Create Local Linked server to point to self
--//////////////////////////////////
USE [master]
GO
--change default catalog
--For MSSQL 2012, 2014, 2016, 2017, and 2019 use @provider=N'MSOLEDBSQL' below (SQL Server Native Client 11.0)---
EXEC master.dbo.sp_addlinkedserver @server = N'ME', @srvproduct=N'', @provider=N'SQLNCLI11', @datasrc=N'localhost', @catalog=N'Northwind'
--For MSSQL 2022 or higher use @provider=N'MSOLEDBSQL' below (Microsoft OLE DB Driver for SQL Server)---
--EXEC master.dbo.sp_addlinkedserver @server = N'ME', @srvproduct=N'', @provider=N'MSOLEDBSQL', @datasrc=N'localhost', @catalog=N'Northwind'

EXEC master.dbo.sp_addlinkedsrvlogin @rmtsrvname=N'ME',@useself=N'True',@locallogin=NULL,@rmtuser=NULL,@rmtpassword=NULL
GO
--//////////////////////////////////
— Step-2 : Enable RPC OUT and Disable MSDTC
--//////////////////////////////////
EXEC sp_serveroption 'ME', 'rpc out', true;  
GO
--Below needed to support EXEC + INSERT (dynamic query)
EXEC sp_serveroption 'ME', 'remote proc transaction promotion', false;
GO
--test simple query
select * from OPENQUERY(ME,'select 1')
GO

--//////////////////////////////////
— Step-3 : Load data into temp table from Amazon MWS Report
--//////////////////////////////////
SELECT * INTO #amazon_mws_report
FROM OPENQUERY(me,'
SET NOCOUNT ON;
EXEC usp_MWS_Report 
   @Columns=''[item-name], [item-description], [listing-id]'',
   @MarketplaceId =''ATVPDKIKX0DER'', 
   @SellerId=''Axxxxxxxxxxxxx'', 
   @ReportType = ''_GET_MERCHANT_LISTINGS_DATA_''
WITH RESULT SETS 
(
	(itmname varchar(1000),itmdesc varchar(1000),itmid varchar(1000))
)
')

select * from #amazon_mws_report

Import Amazon MWS Data into SQL Table (Load Report output into temp table)

Conclusion

In this article, we saw how to load Amazon MWS Data into SQL Server. We used simple SQL queries with ZappySys XML / CSV Driver to pull data from Amazon MWS API. ZappySys ODBC PowerPack makes it super easy to consume data from virtually any API (JSON / XML or CSV Web Services), no coding required. Download here to get started with ZappySys ODBC drivers.

The post How to import Amazon MWS data into SQL Server (T-SQL) appeared first on ZappySys Blog.

In our previous article we saw how to read Google Analytics data using SSIS. But what if you don’t use SSIS and you still like to Import Google Analytics data into SQL Server or Read inside other Reporting / ETL Apps (e.g. Excel, Power BI, MS Access … and many more). Not all BI tools comes with native driver / connectivity for Google Analytics. Sometimes they do offer Google Analytics connectivity but don’t offer feature you looking for (e.g. Sort, Segment Filter). In such case you have to look for solution outside.

Well not to worry we will show you how simple it is to connect to Google Analytics using ZappySys JSON Driver (For REST API / Files) in few mins. This driver is one of the best REST API drivers you can find in the market. It is so generic that it can virtually access any REST API you may find internally or on public sites (e.g. Facebook, Salesforce, Azure, AWS). Check these articles to see many more use cases of JSON Driver.

At the end of this article you will learn how to Query Google Analytics data in SSMS like below (Yes that’s correct .. Pretty Awesome!!! ).

Import Google Analytics Data into SQL Server Table (using T-SQL Code)

Testing Google Analytics API

If you are new to Google Analytics API then read little bit about Google Analytics REST API here. Google also offers a really good way to test Analytics API using Query Explorer Tool here. So please check that and understand how to craft correct REST API URL which will be used in the next section.

Using Google Analytics Query Explorer Tool – Create REST API Url (Dimension / Metrics Selection Browser)

Sample Google Analytics API Response

Behind the scene here is how API Request and Response Looks like for Google Analytics REST API. Notice that Authorization Header in below request is automatically added by ZappySys Driver or Query Testing tool above. Also notice nextLink attribute in response, its used to fetch more data. By default each response contains upto 10000 rows.

Request

GET https://www.googleapis.com/analytics/v3/data/ga?ids=ga:185737326&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration&dimensions=ga:date
Authorization: Bearer ya29.GlwUxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Response

{
    "kind": "analytics#gaData",
    "id": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday",
    "query": {
        "start-date": "30daysAgo",
        "end-date": "yesterday",
        "ids": "ga:123456789",
        "dimensions": "ga:date",
        "metrics": ["ga:avgSessionDuration"],
        "start-index": 1,
        "max-results": 1000
    },
    "itemsPerPage": 1000,
    "totalResults": 30,
    "selfLink": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday",
    "nextLink": "https://www.googleapis.com/analytics/v3/data/ga?ids=ga:123456789&dimensions=ga:date&metrics=ga:avgSessionDuration&start-date=30daysAgo&end-date=yesterday&start-index=10001&max-results=10000",
    "profileInfo": {
        "profileId": "123456789",
        "accountId": "33445566",
        "webPropertyId": "UA-33445566-3",
        "internalWebPropertyId": "454545454",
        "profileName": "All Web Site Data",
        "tableId": "ga:123456789"
    },
    "containsSampledData": false,
    "columnHeaders": [
        {
            "name": "ga:date",
            "columnType": "DIMENSION",
            "dataType": "STRING"
        },
        {
            "name": "ga:avgSessionDuration",
            "columnType": "METRIC",
            "dataType": "TIME"
        }
    ], "totalsForAllResults": {
        "ga:avgSessionDuration": "49.3779488037477"
    },
    "rows": [ 
		["20190425", "401.38912133891213"], 
		["20190426", "512.49723756906077"], 
		["20190427", "11.53488372093024"], 
		["20190428", "432.148648648648646"], 
		["20190429", "234.650735294117645"],
		.......
		.......
		.......
		.......
		["20190525", "74.123435294117645"]
	]
}

Creating Google API Project (Obtain Client ID / Secret)

Very first step to call any Google API including Google Analytics is to create Google API Project and register OAuth App to obtain Client ID and Client Secret. If you dont want to go through this hassle and start easy way then ZappySys offers Inbuilt Default App on OAuth Connection UI but we strongly recommend you create your own app rather than using Default App because in Default App API call limit is shared by many. Your data is never shared with ZappySys in any case even you use Default App.

Here is how to create API Project to call Google API. Once you create Google API Project make sure you enable Google Analytics API.

So once you have Client ID and Client Secret we can move forward to the next step.

Loading Google Analytics data into SQL Server

In our previous article we saw in depth general idea on how to import REST API data in SQL Server (T-SQL Script) without relying on any ETL tools.  Now lets look at step by step on how to load Google Analytics data into SQL Server.

In this example, our goal is to extract a Google Analytics report which shows average session duration and new users count by date (For last 30 days).

Setup ZappySys Data Gateway

Very first step to access any REST API Data inside SQL Server is to configure ZappySys Data Gateway. We covered this in previous article.  But here are high level steps for initial setup of Data Gateway.

1. Search for Gateway in Start menu and  Select ZappySys Data Gateway
   

   Open ZappySys Data Gateway

2. Create new User in Data gateway on Users tab. Enter username and password (we will use this when we create Linked Server) . Check Admin Option
   

   Add Data Gateway User

Setup Google Analytics API Data Source in Gateway / ODBC

Once gateway user is setup, now lets create a new Data Source for Google Analytics API. In this section we will talk how to create data source in gateway but most instructions can be used to create data source in ODBC too. At the end of this article we explained how to Launch ODBC Datasource UI. Step#1 and 2 are only different, all other steps same in ODBC DSN creation.

1. Click Add Data Source option (Select Native – JSON Driver )
   

   Add Gateway Data Source (Native JSON Driver)

2. Click Edit to configure data source
   

   Edit Gateway Data Source Settings

3. Now lets configure Driver settings.  Click on Load Connection String button.
   

   Load ZappySys Driver ConnectionString to configure UI

4. Enter the following ConnectionString to get started with predefined settings.
   Change ids (111223344 to your own Profile ID in DataPath URL). Other URL Parameters explained in next section.
   

   DRIVER={ZappySys JSON Driver};
   DataPath='https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date';
   DataConnectionType=OAuth;
   ScopeSeparator='{space}';
   Scope='https://www.googleapis.com/auth/analytics https://www.googleapis.com/auth/analytics.readonly';
   ServiceProvider=Google;
   Filter='$.rows[*]';
   ArrayTransformType=TransformSimpleTwoDimensionalArray;
   ArrayTransColumnNameFilter='$.columnHeaders[*].name';
   RequestMethod='GET';
   NextUrlAttributeOrExpr='$.nextLink';
   ClientId='xxxxxxxxxxxx';
   ClientSecret='yyyyyyyyyyyyyy';
   UseCustomApp='True'

    
5. URL used in DataPath is most important (parameters explained below) . You can use   Query Explorer Tool here to build same URL.
   

   https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date

   Parameters Explained 
   
   Here is some required parameters used in above REST API URL.
   ids=ga:11223344
   This is your Profile ID which you like to extract.  Read more
   start-date=30daysAgo
   This can be yyyy-MM-dd or some other some predefined date keywords (e.g today, yesterday 10daysAgo). Read more.
   end-date=yesterday
   Same as above
   metrics=ga:avgSessionDuration,ga:newUsers
   This can be comma separated list of metrics you like to extract (max 10). Read more.
   dimensions=ga:date
   This can be comma separated list of metrics you like to extract (max 7). Read more.
6. Now its time to configure OAuth settings. Click on Configure Connection. Enter Client ID and Client Secret Obtained in the previous section (see here) and then click Generate Token as below.
   

   Configure Google API Data Source / DSN (Google Analytics API Example)

7. Thats it now can click on Preview Tab and run sample query like below.
   

   select * from $

   –OR–
   (Select Table name from dropdown and generate default query and then you can add alias , remove unwanted columns)
   

   SELECT 
   "ga:date" date, 
   "ga:avgSessionDuration" avgSessionDuration,
   "ga:newUsers" newUsers
   FROM "rows"

   –OR–
   (Type custom query with your own URL generated from here, Change Profile ID – from 11223344 to your own id)
   

   SELECT * FROM $
   WITH(
   	 Src='https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date'
   	,IncludeParentColumns='False'
   )

   

Create Linked Server in SQL Server using T-SQL Script

Once our Gateway Data Source is configured we can now move to SQL Server Part to define Linked Server.

Here is how you can define linked server to access Google Analytics Data inside SQL Server. There are two ways to create Linked Server (1)  via T-SQL script (2) via SSMS UI)

Here is how to create using Script.

USE [master]
GO

--drop existing linked server for same name if already found
EXEC master.dbo.sp_dropserver @server=N'GOOGLE_ANALYTICS_LS', @droplogins='droplogins'
GO

EXEC master.dbo.sp_addlinkedserver 
	@server = N'GOOGLE_ANALYTICS_LS', 
	@srvproduct=N'', 
	@provider=N'SQLNCLI', 
	@datasrc=N'localhost,5000', --this is host name and port number where  ZappySys Data Gateway is running
	@catalog=N'ZS-GA' --this must be same name as defined on Data Source tab Grid on ZappySys Data Gateway

EXEC master.dbo.sp_addlinkedsrvlogin 
	@rmtsrvname=N'GOOGLE_ANALYTICS_LS',
	@useself=N'False',
	@locallogin=NULL,
	@rmtuser=N'SOME-GATEWAY-USER', -- user name created on ZappySys Data Gateway
	@rmtpassword='SOME-GATEWAY-PASSWORD' -- password for ZappySys Data Gateway user
GO

Create Microsoft SQL Server Linked Server using SSMS UI

Import Google Analytics Data into SQL Server Table (T-SQL Code)

Now lets see how to run sample query to import Google Analytics Data into SQL Server. Notice that how we have escaped quotes in OPENQUERY.
NOTE: Change id 11223344 with your own Id (use this tool to create URL)

select * into GaApiLoad from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:avgSessionDuration,ga:newUsers&dimensions=ga:date''
  ,IncludeParentColumns=''False''
 )
')
select * from GaApiLoad

Import Google Analytics Data into SQL Server Table (using T-SQL Code)

Expose Google Analytics Data as View

To make it simple you can expose above query as SQL Server views so you can access them from any Reporting / ETL Tools.

CREATE VIEW vw_Google_Analytics_BounceRate_PageViews_Last30Days
AS
--change ids to your own Profile ID
select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=2019-04-01&end-date=2019-04-30&metrics=ga:bounceRate,ga:uniquePageviews&dimensions=ga:date''
  ,IncludeParentColumns=''False''
 )
')
GO

Making things Dynamic using Stored Procedure with Parameters

Now let’s make few things dynamic. We will change above query so we can pass Id, Dimensions and Metrics as parameters.

Run below script to Create a stored proc and fetch google analytics data in SQL Server.

use MyDatabase
go
create proc usp_GoogleAnalyticsData
	@ids varchar(1000),
	@startDate varchar(50)='30daysAgo',
	@endDate varchar(50)='yesterday',
	@metrics varchar(200)='ga:avgSessionDuration,ga:newUsers',
	@dimensions varchar(200)='ga:date'	
as

declare @sql varchar(4000)

set @sql='select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
''SELECT * FROM "rows"
WITH 
 (
   SRC=''''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:' + @ids 
   +'&start-date='+ @startDate 
   +'&end-date='+ @endDate 
   +'&metrics='+ @metrics 
   +'&dimensions='+ @dimensions 
   +'''''
  ,IncludeParentColumns=''''False''''
 )
'')'

print @sql
execute(@sql)

go

--Example: query google analytics data
exec usp_GoogleAnalyticsData 
	@ids='11223344',  --Change this ID - Obtain from here https://ga-dev-tools.appspot.com/query-explorer/
	@startDate ='30daysAgo',  --or like this 2019-04-01
	@endDate   ='yesterday', --or like this 2019-04-30
	@dimensions='ga:date', 
	@metrics   ='ga:bounceRate,ga:uniquePageviews'

Insert Google Analytics Data into table (Save Stored Procedure Output)

Now what if you like to save Stored Proc output into a table? Well for that you must create table with same structure as stored proc output and then use INSERT INTO sql like below.

--import google analytics data in a table
if(object_id('GaDataLoad') is not null)
	drop table GaDataLoad
create table GaDataLoad([Date] date, BounceRate numeric(10,5), PageViews numeric(10,5))  

insert into GaDataLoad
exec usp_GoogleAnalyticsData 
	@ids='11223344',  --Change this ID - Obtain from here https://ga-dev-tools.appspot.com/query-explorer/
	@startDate ='30daysAgo',  --or like this 2019-04-01
	@endDate   ='yesterday', --or like this 2019-04-30
	@dimensions='ga:date', 
	@metrics='ga:bounceRate,ga:uniquePageviews'

select * from GaDataLoad

Performance Tips / Handling data errors due to missing rows

By default ZappySys API Drivers sends minimum 2 API requests. First one to fetch metadata and second, third… for Data. There will be a time when you wont have any data and it may throw error about no records found because it fails to parse metadata. You can avoid such issue by supplying metadata before hand so you can avoid expensive API calls. Check this article see how you can supply cached metadata in your SQL Query as below. This will avoid Metadata call and speedup significantly. It will also avoid errors due to no rows found for specified criteria.

select * from OPENQUERY([GOOGLE_ANALYTICS_LS] ,
'SELECT * FROM "rows"
WITH 
 (
   SRC=''https://www.googleapis.com/analytics/v3/data/ga?ids=ga:11223344&start-date=30daysAgo&end-date=yesterday&metrics=ga:bounceRate,ga:uniquePageviews&dimensions=ga:date''
  ,IncludeParentColumns=''False''
  ,Meta=''[ 
     {"Name": "ga:date","Type": "String", "Length": 16 },
     {"Name": "ga:bounceRate", "Type": "String", "Length": 36 },
     {"Name": "ga:uniquePageviews", "Type": "String", "Length": 10 }
    ]''
 )
')

ODBC Connection – Google Analytics Data in Other Apps (e.g. SSRS / Power BI / Excel / Tableau)

So far we have talked accessing data inside SQL Server using Data Gateway Approach but what if you like to access in other apps ? Well you have three options.

1. Use Microsoft SQL Server Driver to call Linked Server Queries (OPENQUERY approach we saw earlier)
2. Access Google Analytics Data using ODBC connectivity
3. Use Microsoft SQL Server Driver to send direct SQL queries to Gateway (Bypass Linked Server – Use inner SQL query without OPENQUERY in this case)

Method-1 : Linked Server Approach

First approach most likely works in all cases because most apps will support connecting to SQL Server using OLEDB / ADO.net / ODBC / JDBC Drivers. So calling SQL Queries which uses Linked Server (i.e. OPENQUERY statement)  is advisable.

Method-2 : ODBC Driver Approach

However in some cases this may not be possible (e.g. You don’t have SQL Server inhouse or you don’t want to rely on SQL Server to access API). In such case you can use ODBC connectivity in your app. Since ODBC is widely adopted standard most app out there (Except JAVA apps) should support ODBC Drivers. If you like to use this approach then create ODBC DSN rather than Data Gateway Data source and use it in your Reporting / ETL / Custom Apps.

Open ODBC Data Sources (Create DSN)

ODBC User DSN Tab: Add new Driver Screen

Method-3 : Direct connection to Data Gateway (By pass Linked Server)

Last option we suggest for cases like JAVA Apps / Linux / Mac machines where you cannot install ZappySys ODBC Drivers and you dont have option to use SQL Server Linked Server either. In such case you can try to send SQL Queries to Gateway directly using Microsoft SQL Server Compatible Drivers (i.e. ODBC /JDBC). See this example how we called API Queries inside JAVA Apps.

Google Analytics / REST API Integration in various apps

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

In this article we explored many ideas of JSON / API integration in SQL Server.  We saw how to create OAuth App for Google API and import Google Analytics data into SQL Server Table without doing any ETL using pure T-SQL code (Query / Views /Stored Procs). You can Download  FREE Trial of ODBC PowerPack and try yourself see how easy it is to query any API inside SQL Server and avoid expensive ETL processes.

The post Import Google Analytics data into SQL Server / Reporting / ETL appeared first on ZappySys Blog.

QuickBooks Online is a well-known Cloud-based Accounting Software. In this post, you will learn how to implement QuickBooks Online API Integration with SQL Server or any other RDBMS (e.g. Oracle, MySQL, Postgresql) using ODBC in few clicks. We will use ODBC JSON / REST API Driver to read data from QuickBooks Online and Load into SQL Server / other targets (Using OAuth Connection). We will discuss on How to Create an Intuit Developer Account, How to Create QuickBooks Online App for OAuth.

In nutshell, this post will focus on how to call QuickBooks Online API using ODBC.
So let’s get started.

Prerequisites

Before we look into Step-By-Step section to extract and read data from QuickBooks Online to SQL Server let’s make sure you met the following requirements.

1. A first requirement will be to SQL Server Database Engine Installed
2. The second requirement will be SSMS installed
3. Finally, make sure to have ZappySys ODBC PowerPack installed.

Create ODBC – Native Zappysys JSON Driver Data Source

1. To do this, first of all, we will open the ZappySys Data Gateway Configuration:
   

   Open ZappySys Data Gateway

2. Add the Native – ZappySys JSON Driver Data source.
   

   Add Gateway Data Source (Native JSON Driver)

3. Now it’s time to connect with QuickBooks. Let’s use JSON Driver Data source with the following URL:
   

   https://{RequestURL}/v3/company/{realmID}/query?query={selectStatement}

   here, We have used 3 placeholders.
   * {RequestURL}is either a SandBox base URL or Production base URL.
   

   Sandbox Base URL: https://sandbox-quickbooks.api.intuit.com
   Production Base URL: https://quickbooks.api.intuit.com

   * {realmID} is your QuickBooks Online sandbox/production company ID.
   * {selectStatement} can be any query something like Select * From Invoice (*Note: select statement must be URL encoded first)
   For more information on Query, an Invoice Request API see this
   I am using my sandbox URL like this:
   

   https://sandbox-quickbooks.api.intuit.com/v3/company/193514799372394/query?query=SELECT * FROM Invoice STARTPOSITION <%startposition%> MAXRESULTS 1000

   

   Setup QuickBooks ODBC-based data source configuration

4. Configure the new OAuth Connection for QuickBooks Online as below.
   Basically, you need to configure the configurations of App on the General tab (i.e. OAuth Provider, Client ID, Client Secret, Scopes, etc.)
   

   OAuth Connection Manager: General Tab Configurations

5. Most importantly, you also need to Enter your Callback URL in Advanced Tab.
   

   OAuth Connection Manager: Advanced Tab Configurations

6. Similarly, you need to check “Do not include credentials in header”
   

   OAuth Connection Manager: OAuth2 Grant Options Tab Configurations

7. Click on Generate Token button to Generate token and then click on Test Connection button and click on OK to save the connection manager configurations settings.
   

   Generate Token and Test Connection

8. Let’s continue on, and enable pagination by configuring it like in the image below in the Pagination tab. In the Page Num Indicator field input this value: <%startposition%> Then set Increment By value to a maximum number of results possible: 1000
   
   

   Configuring pagination to get QuickBooks all rows in ODBC JSON-based driver data source

9. Finally, now using Code Generator we will generate the query.
   

   Generate Query

Load QuickBooks Invoices data in MS SQL Server

Conclusion

So in this blog, we learned how to generate a QuickBooks API Token, Load QuickBooks Invoices data in MS SQL Server table using ODBC JSON / REST API Driver in a very simple way. You can achieve many more functionalities with this tool. Check our blogs/articles on JSON / REST API Driver https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/json-rest-api-driver/ to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the use of QuickBooks API with our tools:

• QuickBooks Documentation for API
• Landing Page for ODBC JSON / REST API Driver, you can also find Tutorial Video here.
• Help File: Documentation of JSON Driver.
• Blog/articles link: https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/json-rest-api-driver/

The post Read data from QuickBooks Online into SQL Server via ODBC Driver appeared first on ZappySys Blog.

So, let’s get started.

Requirements

1. A first requirement will be to SQL Server Database Engine Installed
2. The second requirement will be SSMS installed
3. Finally, make sure to have ZappySys ODBC PowerPack installed.

What is JIRA

JIRA is Atlassian’s popular project- and issue-tracking platform used by development and technical support teams to get more work done and faster.

It combines:

issue tracking – a software application that allows to record and follow the progress of every problem or “issue” that user identifies until the problem is resolved.
agile project management – an iterative approach to planning and guiding project processes. customizable workflow, and a pluggable integration – integrates with Freshdesk, GitHub, Zendesk, Zapbook, Asana and so on.

Getting Started

First of all, we will see several examples. ZappySys includes an ODBC JSON Source Connector that will help you to call JIRA API, Insert issue in JIRA using static data from the SQL server using ODBC JSON driver REST API Call. Update and Delete JIRA issue data From the SQL Server. Insert Bulk JIRA issues from the stored procedure. To learn more about JIRA API check this help file.

About JIRA REST API

JIRA On-premises and Cloud Version both offers REST API. Here are some examples of JIRA REST API. For inserting issue in JIRA we will use JIRA REST API create issue method. We can also insert issues in bulk using Create issues bulk method. To update issue we will use this PUT method: Updating an Issue via the JIRA REST APIs. To delete issue we will use this DELETE method. You can find JIRA Rest API example on this link: Jira REST API examples

Step-by-Step: Write in JIRA from SQL Server

Now once you have API token in the JIRA Atlassian Cloud website we can move to ODBC JSON Driver Configuration Part. For this example purpose you can use the following URL:

https://YOUR-DOMAIN.atlassian.net/rest/api/2/issue/

hence, your-domain is your Atlassian Cloud account’s domain name (we are using zappysys2.atlassian.net).

Above all, each API endpoint may have a different limit for the number of items returned, and these limits may change without notice. (For more See this JIRA REST API reference)

Configure ZappySys Gateway data Source for JIRA

1. To do this, first of all, we will open the ZappySys Data Gateway Configuration:
   

   Open ZappySys Data Gateway

2. Add the Native – ZappySys JSON Driver Data source.
   

   Add Gateway Data Source (Native JSON Driver)

3. Now it’s time to connect with JIRA. Let’s use JSON Driver Data source with the following URL:
   

   https://YOUR-DOMAIN.atlassian.net/rest/api/2/issue/

   

   ODBC JSON Driver: Enter Request URL

4. It is time to configure the Other Settings. Select HTTP Connection Type from the Connection Type drop-down and Click on Click to Configure Link to Configure the Connection Manager.
   

   JIRA REST API Connection – Use ZappySys JSON Driver HTTP Connection

5. Now, Let’s Add request body input data of issue.
   

   {
       "fields": {
          "project":
          {
             "id": "<YOUR-PROJECT-ID>"
          },
          "summary": "No REST for the Wicked.",
          "description": "Creating of an issue using IDs for projects and issue types using the REST API",
          "issuetype": {
             "id": "<ISSUE-TYPE-ID>"
          }
      }
   }

   

   Issue Input Data

6. Now using Query Builder we will generate the query.
   

   Generate JIRA Rest API Post Method Query

7. Now Let’s Test Connection and it will prompt the message “Connection is OK!”
8. That’s it we have successfully configured JIRA Rest API in ODBC JSON Driver.

Setup Linked Server in MS SQL Server for JIRA

Once you configured the data source in Gateway, we can now setup Linked Server in SQL Server to query API Calls.

1. Assuming you have installed SQL Server and SSMS. If not then get both for FREE from here: Get SQL Server Express and  Get SSMS
2. Open SSMS and connect to SQL Server.
3. Go to Root > Server Objects > Linked Servers node. Right click and click New Linked Server…

   

   Add Linked Server in SQL Server

4.  Now enter the linked server name, select Provider as SQL Native Client
5. Enter data source as 

   GatewayServerName, PORT_NUMBER

    where server name is where ZappySys Gateway is running (Can be same as SQL Server machine or remote machine). Default PORT_NUMBER is 5000 but confirm on Data gateway > General tab in case its different.
6. Enter Catalog Name. This must match name from Data gateway Data sources grid > Name column

   

   Configure Linked Server Provider, Catalog, Server, Port for ZappySys Data Gateway Connection

7. Click on Security Tab and select last option “Be made using this security context“. Enter your gateway user account here.
8. Click OK to save Linked Server

Insert data in JIRA from SQL Server (Call POST API Method)

1. To Insert the issue in JIRA we need to execute the query like this. Please refer to this link for more information: Creating an issue example.
   
   Important Note*: If you don’t pass META='{xxxx}’ in query driver call REST API 2 times first time to parse metadata and second time to get actual data. So that you find some odd thing in JIRA project if you execute the query without Meta then you will find two same issues created in your JIRA project as the driver makes two calls as mentioned above.

   SELECT * FROM OPENQUERY([MY_LINKED_SERVER_NAME], 
   'SELECT * FROM $
   WITH( META=''[{"Name": "Success","Type": "Int32"}]''
        ,Src=''https://zappysys2.atlassian.net/rest/api/2/issue/''
        ,RequestData=''{
          "fields": {
           "project":
           {
             "id": "10110"
           },
           "summary": "No REST for the Wicked.",
           "description": "Creating of an issue using IDs for projects and issue types using the REST API",
           "issuetype": {
             "id": "1"
          }
        }
   }''
   ,Header=''Content-Type: application/json || Accept: */*''
   ,RequestMethod=''POST''
   )')

2. Here is the preview after you run REST API query in SQL Server. It will return created issue id, key, and Link. Now let’s note down this response to update and delete the same issue if you want to. Notice that you can override default configuration by supplying many parameters in WITH clause.
   

   Insert Issue In JIRA Using SQL Query

3. You can also find this created issue in your JIRA project.
   

   Created Issue in JIRA

Update data in JIRA from SQL Server (Call PUT API Method)

1. Let’s update the issue which we created in the previous step using id. I Added Extra Meta details in Query to prevent driver to make two calls and also to prevent error as JIRA Rest API Response is a blank

   Status code of "204 No Content"

   . Please refer to this link for more information: Updating an Issue via the JIRA REST APIs
   

   SELECT * FROM OPENQUERY([MY_LINKED_SERVER_NAME]
   , 'SELECT * FROM $
      WITH(META=''[{"Name": "Success","Type": "Int32"}]'' ,
   	 Src=''https://zappysys2.atlassian.net/rest/api/2/issue/10391''
   	,DataConnectionType=''HTTP''
   	,RequestData=''{
       "fields": {
          "summary": "Updated One!...Ye merry gentlemen."
      }
   }''
   	,Header=''Content-Type: application/json || Accept: */*''
   	,RequestMethod=''PUT''
   )')

2. Let’s is the preview after you running update query in SQL Server.
   

   Update Jira Issue From SQL Server

3. Now, Let’s see the updated issue in JIRA project.
   

   Updated Issue In JIRA Project

4. One Import note if you don’t add that metadata then you will get an error like below as JIRA Rest API Response is a blank “Status code of ‘204 No Content'”. If you don’t set META the REST API Make Two calls First to parse metadata and for the second time to get actual data.
   
   Query execution error: No data returned for specified table name or filter expression. Make sure table name or filter is typed correctly (it is case sensitive) and input data contains a corresponding element(s).

Delete data in JIRA from SQL Server (Call DELETE API Method)

1. Now let’s delete the issue which we created by executing following query in SQL Server. We also need to add extra meta details in query same as update one as JIRA Rest API Response is a blank

   Status code of "204 No Content"

   . Please refer to this link for more information: Delete an Issue via the JIRA REST API
   

   SELECT * FROM OPENQUERY([MY_LINKED_SERVER_NAME]
   , 'SELECT * FROM $
      WITH(META=''[{"Name": "Success","Type": "Int32"}]'' ,
   	 Src=''https://zappysys2.atlassian.net/rest/api/2/issue/10391''
   	,DataConnectionType=''HTTP''
   	,Header=''Content-Type: application/json || Accept: */*''
   	,RequestMethod=''DELETE''
   )')

2. Now, Let’s check the deleted issue in JIRA project.
   

   Issue Deleted in JIRA Project

Create multiple JIRA issues using SQL Server Stored Proc

Now lets put all previous things to more practical use. We will create a stored procedure which will read Issues stored in a SQL Table and Submit it to JIRA to create new Issue via REST API call. This will make it totally dynamic so SQL Table can act as your Source of information for API call.

1. Let’s create stored procedure to read data from “tblIssues” data table and create bulk issues in JIRA.
   

   alter PROCEDURE usp_AddMultipleJiraIssues
   AS
   BEGIN
   
   Declare @issueTemp nvarchar(2000)
   Declare @PostQuery nvarchar(MAX)
   Declare @strIssues nvarchar(MAX)
   set @PostQuery='';
   set @strIssues='';
   
   ---- Template 
   set @issueTemp=	'{
   			"fields": {
   				"project": {
   					"key": "<YOUR-JIRA-PROJECT-KEY>"
   				},
   				"summary": "[ISSUE-SUMMARY]",
   				"description": "[ISSUE-DESCRIPTION]",
   			        "issuetype": {
   					  "id": "<YOUR-JIRA-ISSUE-TYPE>"
   				}
   			}
   		}';
   
   -----Generate issues JSON String from the data table 
   Select @strIssues = @strIssues
                       + REPLACE(REPLACE(@issueTemp,'[ISSUE-SUMMARY]',isnull(i.summary,''))
                                                ,'[ISSUE-DESCRIPTION]',isnull(i.description,''))
                       + ',' 
   from tblissues i
   
   -----Trim last , comma from the string otherwise you will get an error 
   SET @strIssues = LEFT(@strIssues,(LEN(@strIssues)-1))
   
   -----JIRA Post Query
   SET @PostQuery= 'SELECT * FROM OPENQUERY([UPDATE_JIRA_DATA_JSON_DRIVER]
   					, ''SELECT * FROM $
   					WITH(
   						META=''''[{"Name": "Success","Type": "Int32"}]''''
                                                   ,Src=''''https://zappysys2.atlassian.net/rest/api/2/issue/bulk''''
   						,DataConnectionType=''''HTTP''''
   						,RequestData=''''{
   						   "issueUpdates": [' + @strIssues + ']
   					        }''''
   						,Header=''''Content-Type: application/json || Accept: */*''''
   						,RequestMethod=''''POST''''
   					)'')'
   
   --Print(@PostQuery)
   
   ----Posting issues data in JIRA
   Execute(@PostQuery)
   
   END
   GO

2. That’s it now executes the stored procedure and you will find bulk issues created in your JIRA project.
   

   Bulk Issues Created In Jira

Conclusion

In this blog, we learned how to generate a JIRA API Token and how to configure it,  Insert, Update and delete Issue in JIRA using ODBC JSON / REST API Driver from the SQL server. You can achieve many more functionalities with this tool. Check our blogs/articles on JSON / REST API Driver to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the use of JIRA API with our tools:

• JIRA Documentation for API
• Landing Page for ODBC JSON / REST API Driver, you can also find Tutorial Video here.
• Help File: Documentation of JSON Driver.
• Blog/articles link: https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/json-rest-api-driver/

The post Insert, Update and Delete JIRA Issue in SQL Server appeared first on ZappySys Blog.

In this article, you will learn how to push data into a Power BI dataset from SQL Server. Power BI is a Microsoft reporting product that can get data from virtually any source and display it nicely in a report or a dashboard. Among its dataset types, it has a streaming dataset into which you can push data from a Power BI REST API. Once pushed, data will be immediately refreshed in a visual, pinned to a dashboard. Thus this approach of sending data to a streaming dataset is a perfect way for displaying real-time data. Yet it is not possible to push data to a dataset straight from SQL Server out of the box. However, with ZappySys ODBC PowerPack it is possible to achieve that quite easily. We will employ one of its apps – ZappySys Data Gateway – to make HTTP requests and push data to a streaming dataset.

The use cases when you may want to push data to a streaming dataset from a SQL Server are:

• Displaying real-time data from many SQL Server instances at once (and you cannot use the DirectQuery option in a report).
• Displaying real-time data when many sources write to a dataset, and a SQL Server is one of them.
• Any other case.

Let’s begin and write some data to a Power BI streaming dataset!

Prerequisites

1. You have a Power BI account and you have basic Power BI skills.
2. SQL Server instance installed (can be a SQL Server Express LocalDB instance).
3. SQL Server Management Studio (SSMS) installed.
4. ZappySys ODBC PowerPack installed (must be installed on a Windows machine; can be a different machine than SQL Server is installed on).

Step By Step – Push data into a Power BI dataset from a SQL Server

Add a data source in the Gateway

It’s time to add a data source, which we will use to POST data to Power BI REST API:

1. In the Gateway, click on Data sources tab, click Add, give data source a name and set Connector Type to Native – ZappySys JSON Driver:
2. Then click Edit to modify the data source:
   
3. Finally, paste into Url textbox Push URL of the dataset which we saved in a previous step. Also, set HTTP Request Method to POST:
   

   Configure REST API Connection for Power BI dataset insert (Using ZappySys JSON / REST Driver)

Add a Linked Server to the Gateway

Writing data into Power BI dataset from SQL Server

Once you add a Linked Server you can execute a query to push data into a Power BI streaming dataset:

1. In the same SQL Server Management Studio create a new query and paste the code below. In Body attribute, we pass the JSON in the same format which was shown when we created a Power BI dataset:
   

   SELECT * FROM
   OPENQUERY([GATEWAY], 'SELECT * FROM $
   WITH (Body=''[ 
   	{ "Timestamp" :"2018-11-15T02:57:04.547Z", "Temperature" : 50.5 },
   	{ "Timestamp" :"2018-11-16T02:57:04.547Z", "Temperature" : 55.5 },
   	{ "Timestamp" :"2018-11-17T02:57:04.547Z", "Temperature" : 60.5 },
   	{ "Timestamp" :"2018-11-18T02:57:04.547Z", "Temperature" : 57.5 },
   	{ "Timestamp" :"2018-11-19T02:57:04.547Z", "Temperature" : 62.3 }
   ]''
   ,meta=''[ { "Name": "Dummy", "Type": "String" }]'' )')

   Note: Use the same Linked Server name you created in a previous step. In our case, it is “GATEWAY”.
2. Execute the query.

The results: creating a Power BI report

Now it’s time to check the results and see if data went through:

1. Go to Power BI portal.
2. Select Datasets and click on the add new report icon:
   

   Create new Power BI Report for real time dataset

3. Add a Line Chart, a Table and link them to the dataset.
4. You should see the data we pushed to the dataset:
   

   Showing real time data in Power BI dashboard

Conclusion

You learned how to push data into a Power BI streaming dataset directly from SQL Server. It was possible to achieve that with ZappySys Data Gateway and SQL Server Linked Server option. At first, we created a streaming dataset in Power BI, then configured a datasource in the Gateway, followed with adding a Linked Server to the Gateway, executed T-SQL query in SSMS and pushed some data to the dataset and finally created a Power BI report to check the received data.

When we set Historic data analysis option when creating a dataset, we, in turn, created a dataset of type Push (if we left it off, it would be of type Streaming). It allowed us to create a report easily on top of it, but it has certain limitations. Just be aware when using this option and check Power BI service documentation for more info.

Finally, in SSMS, we executed a query with predefined data. Therefore in real life, if you want to push data from an existing table, you will have to build JSON from it. Make sure you construct a valid JSON and escape the symbols that need to be escaped (such as a double-quote or a backslash).

References

https://powerbi.microsoft.com/en-us/

https://docs.microsoft.com/en-us/rest/api/power-bi/

https://docs.microsoft.com/en-us/power-bi/service-real-time-streaming

https://docs.microsoft.com/en-us/power-bi/developer/api-rest-api-limitations

The post Push data into a Power BI dataset from SQL Server appeared first on ZappySys Blog.

In this article we will cover how to call REST API in PowerShell and export result to CSV file. If you want to export JSON based API result to file then you have to consider few things. Such as API Authentication, API Pagination, Parsing JSON, Error Handling and many more. Luckily we can use ODBC Driver like this one to call API services and get data from RESTful services using familiar SQL query language.

ZappySys ODBC driver can take care most of details about your API calls and data parsing so you can focus on business problem.

Requirements

Before we start step by step tutorial let’s make sure we have necessary tools installed.

1. You have PowerShell ISE – Editor for PowerShell (Search in your Start menu to confirm)
2. Make sure you have installed ZappySys ODBC PowerPack

Getting Started

Once you have installed ODBC PowerPack we are ready to look at few examples on how to call REST API in PowerShell.

Example 1 – Call REST API and Export JSON data to CSV in PowerShell

Open PowerShell ISE (Editor) and Paste below code. Click

$conn = New-Object System.Data.Odbc.OdbcConnection
$conn.ConnectionString = "DRIVER={ZappySys JSON Driver}"

#--OR-- Use DSN name
#$conn.connectionstring = "DSN=MyDSNName"

$conn.Open()

# -------------------------------------------------------------------------------
# In powershell $ is special char so we used `$ in below string to escape it. 
# Also We used multi string start with "@<new line> and ends with <new line>"@
# -------------------------------------------------------------------------------
$sql = 
@"
SELECT * FROM value 
WITH (SRC='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?`$format=json')
"@

$cmd = $conn.CreateCommand()
$cmd.CommandText = $sql

$dataset = New-Object System.Data.DataSet
#Load data in DataSet
(New-Object System.Data.Odbc.OdbcDataAdapter($cmd)).Fill($dataSet)

#Export datatable to file in CSV format
$dataset.Tables[0] | ConvertTo-csv -NoTypeInformation -Delimiter "`t" | Out-File "c:\temp\dump.csv" -fo

Write-Host "Total rows $($dataSet.Tables[0].Rows.Count)"
$conn.Close()

Here is what it will look like once you execute.

Example : Call REST API in PowerShell and Export JSON data to CSV File

Example 2 – Reading REST API data using ODBC DSN (User Interface Mode) in PowerShell

In previous section we saw simple script which was using DSN less connection string (i.e. DRIVER={xxxxxxx} ).

Now let’s look at another way to call API (i.e. Use DSN=xxxxx in your connection string). We will use ODBC Driver’s User Interface to configure various properties. We will also use query preview tool to test your API query. Finally we will use DSN name in Our PowerShell Script.

Advantage of using ODBC DSN is you don’t have to expose credentials in your PowerShell Script. Also when you have multiple scripts using same connection to query multiple API endpoints it becomes very easy to reuse just DSN name rather than many connection string settings.

So let’s get started

1. To do this, we will first open the ODBC Data Source (32 bit):
   

   Open odbc data source

2. Use the User DSN page and press Add
   

   Add new data source

3. Add the ZappySys JSON Driver
   

   Select zappysys JSON Driver (needed for JSON / REST API Connection)

4. Name your data source (e.g. in below screenshot its ZappySys JSON PowerBI). This will be used in our Script to refer in DSN=xxxxxxx for connectionstring.
5. It is time to connect with OData. In Data Source (URL or File Path), we will use the following URL.
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json

   After URL select Filter (Basically its a setting to flatten your data from selected hierarchy). In our case Filter = $.value[*]

   UI Mode (Simple View) – New Version

   

   Simple View – Configure ZappySys JSON / REST API Driver Properties

   Grid Mode (Advanced View) – Old Version

   

   Advanced View – ZappySys JSON / REST API Driver Properties

    

6. We can now generate queries using a format similar to SQL. Click Select Table and select value. This option will generate a query to retrieve all the data automatically. Next, press Preview Data:
7. When Preview is pressed, you can visualize the data.
   

   Display rest api information

8. You should also review several other examples to learn many other features of Driver Query language.
   

   ZappySys JSON / REST Driver – Query Examples for PowerShell

9. ZappySys is pretty cool because you can run any type of query like SQL. For example, you can run a query to RESP API using like and or:
   

   /* 
   Object name can be wrapped in double quotes or rectangle brackets
   useful if name has space, dash or dot 
   */
   SELECT 
    CustomerID,
    CompanyName,
    ContactName,
    ContactTitle,
    Address,
    City,
    Region,
    PostalCode,
    Country,
    Phone,
    Fax
   FROM [value]
   WHERE ContactTitle='Owner'

That’s it once you save your DSN you can change your previous script as below to use DSN

$conn = New-Object System.Data.Odbc.OdbcConnection

$conn.connectionstring = "DSN=ZappySys JSON PowerBI"
$conn.Open()
$sql = "SELECT * FROM value"
$cmd = $conn.CreateCommand()
$cmd.CommandText = $sql
$dataset = New-Object System.Data.DataSet

#Load data in DataSet
(New-Object System.Data.Odbc.OdbcDataAdapter($cmd)).Fill($dataSet)

#Export datatable to file in CSV format
$dataset.Tables[0] | ConvertTo-csv -NoTypeInformation -Delimiter "`t" | Out-File "c:\temp\dump.csv" -fo

Write-Host "Total rows $($dataSet.Tables[0].Rows.Count)"
$conn.Close()

Example 3 – Reading data from JSON  files (Single / Multiple Files) in PowerShell

If you like to read JSON data from local file then use File Path rather than URL as below. Notice that we are reading data from multiple files using Wildcard pattern (*.json). For reading from multiple file make sure all files have same structure.

$conn = New-Object System.Data.Odbc.OdbcConnection

$conn.connectionstring = "DRIVER={ZappySys JSON Driver}"
$conn.Open()
$sql = "SELECT * FROM value WITH (SRC='C:\temp\File*.json')"
$cmd = $conn.CreateCommand()
$cmd.CommandText = $sql
$dataset = New-Object System.Data.DataSet

#Load data in DataSet
(New-Object System.Data.Odbc.OdbcDataAdapter($cmd)).Fill($dataSet)

#Export datatable to file in CSV format
$dataset.Tables[0] | ConvertTo-csv -NoTypeInformation -Delimiter "`t" | Out-File "c:\temp\dump_multi_files.csv" -fo

Write-Host "Total rows $($dataSet.Tables[0].Rows.Count)"
$conn.Close()

Example 4 – POST data to REST API URL (Supply Header / Body) in PowerShell

Now let’s look at some example to POST data to API.  In below example we will also see how to pass HTTP headers and Body for POST request. POST requests usually made to Create or Update existing record. However sometimes you will call POST request to read data too (specially you have to supply many parameters in Body rather than URL to call your request).

$conn = New-Object System.Data.Odbc.OdbcConnection

$conn.connectionstring = "DRIVER={ZappySys JSON Driver}"
$conn.Open()
$sql = @"
SELECT * FROM `$
WITH 
( 
  SRC='http://httpbin.org/post' ,
  METHOD='POST' ,
  HEADER='Content-Type: application/json || x-hdr1: SomeValue' ,  
  BODY='{ id:1, notes:"My Line1\r\nMy Line2" }'
)
"@
$cmd = $conn.CreateCommand()
$cmd.CommandText = $sql
$dataset = New-Object System.Data.DataSet

#Load data in DataSet
(New-Object System.Data.Odbc.OdbcDataAdapter($cmd)).Fill($dataSet)

#Export datatable to file in CSV format
$dataset.Tables[0] | ConvertTo-csv -NoTypeInformation -Delimiter "`t" | Out-File "c:\temp\dump_post_response.csv" -fo

Write-Host "Total rows $($dataSet.Tables[0].Rows.Count)"
$conn.Close()

Example 5 – Upload File Content to Web API in PowerShell

Now let’s look at example where you have to Upload File Content to Web API using PowerShell Script.

$conn = New-Object System.Data.Odbc.OdbcConnection

$conn.connectionstring = "DRIVER={ZappySys JSON Driver}"
$conn.Open()
$sql = @"
SELECT * FROM `$
WITH 
( 
  SRC='http://httpbin.org/post' ,
  METHOD='POST' ,
  HEADER='Content-Type: application/json || x-hdr1: SomeValue' ,  
  BODY='@c:\temp\test.json',
  IsMultiPart='True'
)
"@
$cmd = $conn.CreateCommand()
$cmd.CommandText = $sql
$dataset = New-Object System.Data.DataSet

#Load data in DataSet
(New-Object System.Data.Odbc.OdbcDataAdapter($cmd)).Fill($dataSet)

#Export datatable to file in CSV format
$dataset.Tables[0] | ConvertTo-csv -NoTypeInformation -Delimiter "`t" | Out-File "c:\temp\file-upload-response.csv" -fo

Write-Host "Total rows $($dataSet.Tables[0].Rows.Count)"
$conn.Close()

REST API with Pagination in PowerShell

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

REST API Error Handling in PowerShell

Sometimes errors occur... they just do and there is nothing you can do! Or can you? Actually, in ODBC PowerPack you can handle them in two ways.

METHOD 1 - Using Error Handling Options

When to use?

You may want to use them when your source is a resource located on the Internet; e.g. a file on a website, a file on an FTP server or just a plain API HTTP response. By default, when a remote server returns an error, data retrieval is stopped, an error is raised and no data is given back to you. This might not be always desirable.

Scenario 1

Imagine a scenario, that there is a web server which each day at 12 AM releases a new JSON file with that day's date as filename, e.g. http://www.some-server.com/data/2018-06-20.json. And, of course, you want to download it and use it daily in your Power BI report. But you have a problem: Power BI report data sources are refreshed each hour and you may get HTTP 404 status code (no file was found) when a file is not released yet. Which consequentially means other data sources won't be updated as well and you will see old and cached data on the report. That's where you could use Continue on any error or Continue when Url is invalid or missing (404 Errors) to avoid an error being raised and let other data sources to be updated.

Scenario 2

Another scenario is when you expect a web server to raise some kind of HTTP error when accessing a URL. You don't want ODBC Data Source to raise an error but instead, you want to get response data. That's where you can use Continue on any error or alike together with  Get response data on error to continue on an error and get the data:

METHOD 2 - Using Connection [Retry Settings]

Another scenario you may run into is a buggy web server. You ask it to give you some file or data and it, like a snotty kid, just doesn't give it to you! You have to ask twice or thrice before it does its job. If that's the case, you have to retry HTTP requests using Connection:

REST API Authentication in PowerShell (Basic Auth, OAuth, JWT, SOAP WSS)

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Other Consideration for REST API call in PowerShell

Conclusion

In this article we checked how to read REST API and JSON Files in PowerShell with minimum coding effort. Download ZappySys ODBC PowerPack to enhance your overall API integration scenarios in PowerShell and other programming languages such as Python, C# or tools like Power BI, Excel, SSRS, Informatica.

The post Call REST API in PowerShell Script – Export JSON to CSV appeared first on ZappySys Blog.

In this new article, we will see how to connect Java to REST API using a simple connector. This ODBC driver allows connecting to REST API or a local file in Java using SQL queries. If you are familiar with SQL, you can easily access to REST API or JSON using simple SQL queries. This article mainly focus on JSON based REST API but techniques listed in this article is almost same for XML based API (e.g. SOAP Web Services).

Requirements

1. First of all, we will require the JDBC Driver for SQL Server.
2. Secondly, we will require ZappySys ODBC PowerPack.
3. Also, we will require a Java IDE. We will use Java NetBeans for this.

Getting started

First of all, we will create an ODBC connection with a Gateway to the SQL Server connector and then we will connect with Java code.

Add ZappySys ODBC Driver to connect to REST API

The ZappySys JSON Driver allows connecting to different resources like JSON files, REST API, OData, etc.

1. First of all, in the ODBC Data Source Administrator, press add to add a new System Data Source:
   

   Add new ODBC driver

2. Secondly, select the ZappySys JSON Driver:
   

   Select JSON driver in ODBC DS Administrator

3. Also, add this URL to the ZappySys JSON Driver:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json

   

   REST API ODBC URL

ZappySys Gateway to connect Java to REST API

In order to connect Java to REST API, we will use a Gateway that is installed with the ZappySys ODBC PowerPack. This gateway uses a SQL Server connection. Let’s take a look at the steps.

1. First of all, open the ZappysSys Data Gateway Configuration.
2. Secondly, press the add button and create a new data source named customer:
   

   Add new Data Source in ZappySys Gateway

3. Also, enable the custom data source and make sure to check the public option.
4. In addition, press the Edit and add the ODBC Data Source created in the Add ZappySys ODBC Driver to connect to REST API section:
   

   <strong>NOTE:</strong> Whenever possible use native driver option for better performance / security and ease of use.

   

   Zappysys ODBC driver

5. Additionally, we will add a new user:
   

   Add the user to the data source gateway

6. Also, add the user to the data source:
   

   Add user to the data source

7. Finally, check the default port in the general tab. By default this value is 5000:
   

   Default gateway with port 5000

Connect Java to REST API

1. First of all, we will first try to connect to the following URL:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json

2. Secondly, we will get the countries and the sum or quantity orders using the following code:
   

   package padaone;
   
   import java.sql.*;
   
   public class zappy {
   
   	public static void main(String[] args) {
   		
   		// Create a variable for the connection string.
   		String connectionUrl = "jdbc:sqlserver://localhost:5000;databasename=customer;user=test;password=test";
   
   		// Declare the JDBC objects.
   		Connection con = null;
   		Statement stmt = null;
   		ResultSet rs = null;
   		
           	try {
           		// Establish the connection.
           		Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
               		con = DriverManager.getConnection(connectionUrl);
               
               		// Create and execute an SQL statement that returns some data.
               		String SQL = "SELECT Country , SUM(UnitPrice * Quantity) Total " 
   	+ "FROM value " 
   	+ "GROUP BY Country " 
   	+ "WITH (SRC='https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json')";
               		stmt = con.createStatement();
               		rs = stmt.executeQuery(SQL);
               
               		// Iterate through the data in the result set and display it.
               		while (rs.next()) {
               			System.out.println(rs.getString(1) + " " + rs.getString(2));
               		}
           	}
           
   		// Handle any errors that may have occurred.
   		catch (Exception e) {
   			e.printStackTrace();
   		}
   
   		finally {
   			if (rs != null) try { rs.close(); } catch(Exception e) {}
   	    		if (stmt != null) try { stmt.close(); } catch(Exception e) {}
   	    		if (con != null) try { con.close(); } catch(Exception e) {}
   		}
   	}
   }

3. In addition, in the project, go to libraries, right click and select the option to add JAR or folder and select the sqljdbc42.jar. This jar file is used to connect to SQL Server. The ZappySys ODBC PowerPack uses the SQL Server JDBC connector with a gateway to connect to REST API or JSON files:
   

   Add SQL Server jar file

4. Also, we will need to add the sqljdb4.2.jar and the sqljdbc_auth.dll in the C:/Windows/System32 and the path where the JDK is installed (in this example C:/Program Files/Java/jdk1.8.0_171/bin).
5. Finally, run the code and you will visualize the query results of the REST API:
   

   REST API results in Java

Explanation of the code to Connect Java to REST API

Previously, we run the code to query REST API data in Java. Now we will explain the code.

1. First of all, we will import the java.sql to connect to SQL Server using Java:
   

   import java.sql.*;

2. Secondly, we will create a connection to the ZappySys Gateway. Using the port 5000 and we are using the data source and user created in the Add ZappySys ODBC Driver to connect to REST API:
   

   String connectionUrl = "jdbc:sqlserver://localhost:5000;databasename=customer;
   user=test;password=test";

3. Also, we will set the variables to null to initialize them:
   

   Connection con = null;
   Statement stmt = null;
   ResultSet rs = null;

4. In addition, we will connect to the JDBC SQL driver:
   

   Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
   con = DriverManager.getConnection(connectionUrl);

5. Additionally, we will send a query to the URL and get the country and the sum of price multiplied by quantity, group by country:
   

   String SQL = "SELECT Country , SUM(UnitPrice * Quantity) Total " 
   	+ "FROM value " 
   	+ "GROUP BY Country " 
   	+ "WITH (SRC='https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json')";
               		stmt = con.createStatement();
               		rs = stmt.executeQuery(SQL);

6. Next, we will print the results of the query:
   

   while (rs.next()) {
      System.out.println(rs.getString(1) + " " + rs.getString(2));
   }

7. Finally, we will close all the connections:
   

   finally {
   			if (rs != null) try { rs.close(); } catch(Exception e) {}
   	    		if (stmt != null) try { stmt.close(); } catch(Exception e) {}
   	    		if (con != null) try { con.close(); } catch(Exception e) {}
   		}

    

How to connect to multiple JSON files

The ZappySys ODBC PowerPack can be used to easily read local JSON files in Java and query using SQL with SELECT, WHERE, GROUP BY, HAVING and other SQL clauses. For more information, refer to this link.

1. The following code will list the files and the content of the JSON files:
   

   package padaone;
   
   import java.io.File;
   import java.sql.Connection;
   import java.sql.DriverManager;
   import java.sql.ResultSet;
   import java.sql.Statement;
   
   public class localfile {
       public static void main(String[] args) {
           File directory = new File ("C:\\temp");
           File[] list = directory.listFiles();
           
           printNames(list);	
       }
   
       private static void printNames(File[] list) {
           for(int i =0;i<list.length;i++){
               if(list[i].isFile()){
                   System.out.println(list[i].getName());
   
                   
                   showContent(list[i]);
                   
               }
           }
       }
   
       private static void showContent(File file) {
         
           // Create a variable for the connection string.
           String connectionUrl = "jdbc:sqlserver://localhost:5000;databasename=customer;user=test;password=test";
   
           // Declare the JDBC objects.
           Connection con = null;
           Statement stmt = null;
           ResultSet rs = null;
   
           try {
                   // Establish the connection.
                   Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
                   con = DriverManager.getConnection(connectionUrl);
   
                   // Create and execute an SQL statement that returns some data.
                   String SQL = "SELECT * FROM $ WITH (SRC='C:\\temp\\"+file.getName()+"')";
                   stmt = con.createStatement();
                   rs = stmt.executeQuery(SQL);
   
                   // Iterate through the data in the result set and display it.
                   while (rs.next()) {
                           System.out.println(rs.getString(1) + " " + rs.getString(2));//+ " " + rs.getString(3)+ " " + rs.getString(4)+ " " + rs.getString(5)+ " " + rs.getString(6));
                   }
           }
   
           // Handle any errors that may have occurred.
           catch (Exception e) {
                   e.printStackTrace();
           }
   
           finally {
                   if (rs != null) try { rs.close(); } catch(Exception e) {}
                   if (stmt != null) try { stmt.close(); } catch(Exception e) {}
                   if (con != null) try { con.close(); } catch(Exception e) {}
           }
   	
       }
   
   }

2. The JSON files are the following:
   

   example_1.json
   
   [{
     "id": 1,
     "first_name": "Jeanette",
     "last_name": "Penddreth",
     "email": "jpenddreth0@census.gov",
     "gender": "Female",
     "ip_address": "26.58.193.2"
   }, {
     "id": 2,
     "first_name": "Giavani",
     "last_name": "Frediani",
     "email": "gfrediani1@senate.gov",
     "gender": "Male",
     "ip_address": "229.179.4.212"
   }, {
     "id": 3,
     "first_name": "Noell",
     "last_name": "Bea",
     "email": "nbea2@imageshack.us",
     "gender": "Female",
     "ip_address": "180.66.162.255"
   }, {
     "id": 4,
     "first_name": "Willard",
     "last_name": "Valek",
     "email": "wvalek3@vk.com",
     "gender": "Male",
     "ip_address": "67.76.188.26"
   }]
   
   example_2.json
   
   {
       "quiz": {
           "sport": {
               "q1": {
                   "question": "Which one is correct team name in NBA?",
                   "options": [
                       "New York Bulls",
                       "Los Angeles Kings",
                       "Golden State Warriros",
                       "Huston Rocket"
                   ],
                   "answer": "Huston Rocket"
               }
           },
           "maths": {
               "q1": {
                   "question": "5 + 7 = ?",
                   "options": [
                       "10",
                       "11",
                       "12",
                       "13"
                   ],
                   "answer": "12"
               },
               "q2": {
                   "question": "12 - 8 = ?",
                   "options": [
                       "1",
                       "2",
                       "3",
                       "4"
                   ],
                   "answer": "4"
               }
           }
       }
   }

3. Finally, if everything is fine, you will be able to see the file names
   and the content:
   

   Read JSON files in Java

How to connect Java to REST API Gmail

Connect Java to Google, Facebook, OneDrive or other REST API tools is a straightforward process using the ZappySys ODBC Driver. You just need to create the Data Source in ODBC and then create a ZappySys Gateway, the rest is simple in Java. Let’s take a look at an example.

1. First of all, we will create a new Data source in the Windows ODBC Data
   Source Administrator. We will create a new ZappySys JSON Driver and add the following link:
   

   https://www.googleapis.com/gmail/v1/users/me/messages/

2. Secondly, select the OAuth connection type and $.messages[*] in the
   filter and press the Click to Configure link:
   

   Setup ODBC configuration to URL

3. Also, in the OAuth parameters link, select the Google OAuth Provider.
4. In addition in scopes write the scope https://mail.google.com:
   

   Rest API connect to Gmail

5. Additionally, in the preview tab, you can generate your queries and save
   your Data Source:
   

   Connect Java to REST API

6. Next, open your ZappySys Data Gateway. We will create a new Data Source
   named gm:
   

   Add Gmail Data Source

7. Also, we will associate the ODBC Data Source with the Gateway Data Source. Make sure to add the test user to the Gateway data source:
   

   Include the data source for the ODBC

8. Finally, you can run this code. The code in Java will connect to the Gateway data source using the port 5000 and the Gateway user and password. The query will show the Gmail message ids:
   

   package padaone;
   
   import java.sql.Connection;
   import java.sql.DriverManager;
   import java.sql.ResultSet;
   import java.sql.Statement;
   
   
   public class gmail {
       public static void main(String[] args) {
   		
   		// Create a variable for the connection string.
   		String connectionUrl = "jdbc:sqlserver://localhost:5000;databasename=gm;user=test;password=test";
   
   		// Declare the JDBC objects.
   		Connection con = null;
   		Statement stmt = null;
   		ResultSet rs = null;
   		
           	try {
           		// Establish the connection.
           		Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
               		con = DriverManager.getConnection(connectionUrl);
               
               		// Create and execute an SQL statement that returns some data.
               		String SQL = "SELECT id FROM [_root_]";
               		stmt = con.createStatement();
               		rs = stmt.executeQuery(SQL);
               
               		// Iterate through the data in the result set and display it.
               		while (rs.next()) {
               			System.out.println(rs.getString(1));
               		}
           	}
           
   		// Handle any errors that may have occurred.
   		catch (Exception e) {
   			e.printStackTrace();
   		}
   
   		finally {
   			if (rs != null) try { rs.close(); } catch(Exception e) {}
   	    		if (stmt != null) try { stmt.close(); } catch(Exception e) {}
   	    		if (con != null) try { con.close(); } catch(Exception e) {}
   		}
   	}
   }

9. You can find your message id in Google by clicking your messages:
   

   Display Gmail information in Java

ZappySys JSON /REST API Driver Query Examples

Reading from XML files or API can be done using the same way as previous sections except you have to use ZappySys XML Driver. Read help file here to see json query examples.

ZappySys XML / SOAP Driver Query Examples

Reading from XML files or API can be done using the same way as previous sections except you have to use ZappySys XML Driver. Read help file here to see xml query examples.

Calling XML SOAP Web Service in JAVA

So far we have looked at examples to consume data using JSON driver. Now lets look at an example, to call XML SOAP Web Service in JAVA.

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

POST data to REST API URL from file in JAVA

Above example was POST data to API URL but what if your Request Body is large and you have saved that to file? Well here is the way to get your request body from a file (Use @ symbol before path).

SELECT * FROM $
WITH 
(METHOD='POST' 
,HEADER='Content-Type:text/plain || x-hdr1:AAA'
,SRC='http://httpbin.org/post'
,BODY='@c:\files\dump.xml'
,IsMultiPart='True'
)

REST API Pagination in JAVA

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

Error Handling in REST API / SOAP

Sometimes errors occur... they just do and there is nothing you can do! Or can you? Actually, in ODBC PowerPack you can handle them in two ways.

METHOD 1 - Using Error Handling Options

When to use?

You may want to use them when your source is a resource located on the Internet; e.g. a file on a website, a file on an FTP server or just a plain API HTTP response. By default, when a remote server returns an error, data retrieval is stopped, an error is raised and no data is given back to you. This might not be always desirable.

Scenario 1

Imagine a scenario, that there is a web server which each day at 12 AM releases a new JSON file with that day's date as filename, e.g. http://www.some-server.com/data/2018-06-20.json. And, of course, you want to download it and use it daily in your Power BI report. But you have a problem: Power BI report data sources are refreshed each hour and you may get HTTP 404 status code (no file was found) when a file is not released yet. Which consequentially means other data sources won't be updated as well and you will see old and cached data on the report. That's where you could use Continue on any error or Continue when Url is invalid or missing (404 Errors) to avoid an error being raised and let other data sources to be updated.

Scenario 2

Another scenario is when you expect a web server to raise some kind of HTTP error when accessing a URL. You don't want ODBC Data Source to raise an error but instead, you want to get response data. That's where you can use Continue on any error or alike together with  Get response data on error to continue on an error and get the data:

METHOD 2 - Using Connection [Retry Settings]

Another scenario you may run into is a buggy web server. You ask it to give you some file or data and it, like a snotty kid, just doesn't give it to you! You have to ask twice or thrice before it does its job. If that's the case, you have to retry HTTP requests using Connection:

REST API / SOAP Connection Types in JAVA (OAuth / HTTP)

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Performance consideration for Web API Calls

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Other considerations for Web API calls in JAVA

Conclusion

To conclude, in this tutorial, we learned how to access the REST API using Java. We created a ZappySys ODBC connection and then used a ZappySys Gateway to connect to it.

References

• ZappySys ODBC PowerPack
• Java example ZappySys ODBC PowerPack
• Add connection in Netbeans 8 to the SQL Server 2012 database.

The post How to connect Java to REST API (JSON / SOAP XML) appeared first on ZappySys Blog.