Using SOAP API with SharePoint in SSIS is a common requirement to Administer SharePoint. SharePoint on-premises is a nice Microsoft application used to share documents and collaborate with the company. You can have schedules, projects, documents, and more shared using a Web platform.

In addition, it is a very useful tool that your company may need. It is also possible to automate, administer tasks using SOAP API.
In this article, we will learn how to create SOAP API queries to SharePoint on-premises.

Also, we will show our ZappySys SSIS PowerPack that includes very powerful tools to export SharePoint data from REST API to any other source.

	SSIS REST API Web Service Task
	SSIS XML Source (File, REST, SOAP Web Service)

Requirements

1. Firstly, you will need SSDT installed.
2. Secondly, SSIS PowerPack installed.
3. SoapUI installed.

Getting started

We will start with SoapUI application. It will help us to get all the possible methods SharePoint SOAP API allows. Also, it will show the inner workings of how the requests should look, which we will later replicate on XML Source. SoapUI will be our polestar.

How to call SharePoint SOAP API (On-Premises)

To call SOAP API in SSIS you have to perform the following steps at a high level. Detailed steps are described in the next section

1. Import WSDL file or URL for your Service in SoapUI for testing.
   1. File > New SOAP Project.
   2. In the Initial WSDL field enter
      http://your.domain.com/_vti_bin/Lists.asmx?wsdl
      or
      http://your.domain.com/my-site/_vti_bin/Lists.asmx?wsdl
2. Add credentials for SOAP Web service (Basic Auth or WSS)
3. Create SOAP Request XML (Edit default parameters) and Click Play Button to test your request. Associative photo:
   

   Create SOAP Request Body from WSDL (Using SoapUI tool)

4. If your test works in SoapUI then you are ready to move to SSIS part.
5. Copy Service URL, Request Body and Headers to Notepad — we will use it in SSIS.
   For Header see Raw Tab on SoapUI > Copy two lines Content-Type and SOAPAction.
   

   Get ContentType and SOAPAction Headers from SoapUI Raw tab

In the next section, we will see how to call SharePoint SOAP Web Service in SSIS.

How to read data from a SharePoint List via SOAP Web Service

Now let’s move onto SSIS and get some data from SharePoint using XML Source.

1. Open SSIS, create a new Integration project and create a new SSIS package.
2. Drag and drop Data Flow onto the Control Flow from SSIS Toolbox.
3. Then drag and drop XML Source onto Data Flow (you may also use other similar connectors such as REST API Task or Web API Destination).
4. Then configure XML Source in this way:
   
   1. Enter URL:
      https://my.sharepoint.site.com/sites/mysite/_vti_bin/Lists.asmx
      or
      https://my.sharepoint.site.com/_vti_bin/Lists.asmx (depends if you have many sites or just one).
   2. Check Use Credentials checkbox and create ZS-HTTP connection. Use Basic or NTLM authorization scheme.
   3. Then copy configuration from SoapUI to XML Source. E.g. HTTP Request Method, Body, Body Content-Type, HTTP Headers (e.g. SOAPAction header).
   4. Use $.soap:Envelope.soap:Body.GetListItemsResponse.GetListItemsResult.listitems.rs:data.z:row[*] as filter. If it doesn’t work, use Select Filter button to select the path manually and select row node.
5. What’s left is to connect XML Source with OLEDB Destination or a similar destination component:
   

Conclusion

To conclude, we can say that using SSIS PowerPack component XML Source and SoapUI we could achieve our goal of getting data from SharePoint On-Premises. The next step could be writing data to SharePoint via the same SOAP API interface, but this time using Web API Destination component.

The post Read data from SharePoint List in SSIS (On-Premises SOAP API) appeared first on ZappySys Blog.

In our previous blog post we saw how to import REST / SOAP API in SQL Server. Using same concepts let’s look at how to import Bing Ads data into SQL Server. We will explore many techniques to call Bing Ads API and learn how to automate data extraction without doing any ETL. You can call Bing Ads API just using T-SQL code (Yes you heard it right). At the end of this article you will learn how to Download Performance Reports from Bing Ads Account without any coding (See sample screenshot below). Please go through full article carefully.

NOTE: Bing Ads now renamed as Microsoft Advertising but this article will use popular name for time being.

Import Bing Ads data into SQL Server (Using SSMS T-SQL Code)

About Bing Ads API / SOAP Web Service

If you are new to Bing Ads API then start from here. Bing Ads APIs are SOAP XML APIs. You can call it to automate many scenarios for Bing Ads. In this article we will mainly focus on read scenario (e.g. Read Performance Data) but you can use these techniques to write / update (e.g. Create New Campaign).

If you are new to SOAP WebService then Check this article to learn more. It will explain you how to use tools like SoapUI tool to craft SOAP Requests for Bing API or any other SOAP API.

For Bing API you can use Service WSDL files found here. Use URL ending ?wsdl to import in SoapUI.

Example Bing API Request

Here is raw API request for Bing API. This sample assumes you have completed 3-Legged OAuth Authorization to obtain AccessToken and RefreshToken (Explained later in this article). Below request uses AccessToken you might have already obtained via process like this. If you are using ZappySys tools then this will be generated automatically for you.

Request Method / URL:

POST https://clientcenter.api.bingads.microsoft.com/Api/CustomerManagement/v13/CustomerManagementService.svc

Headers:
Notice that for each Api you may need correct SOAPAction in Header. If you are not sure then use SoapUI Tool. Execute request in SoapUI and then go to Raw tab like this one.

Content-Type: text/xml; charset=utf-8
SOAPAction: "GetUser"

Body:
Notice that you have to replace AuthenticationToken and DeveloperToken in below request. AccessToken is placed inside AuthenticationToken tag. This Auth token is short lived and extracted via OAuth Process and DeveloperToken you can extract as mentioned in the previous section or get it from here.

  
    GetUser
    <%access_token%>
    1052Bxxxxxxxxxxxxxx
  
  
    
      
    
  

Response:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Header>
    <h:TrackingId xmlns:h="https://bingads.microsoft.com/Customer/v13">fddf3cb9-e4c2-4e75-9546-f3bc20ea91a8</h:TrackingId>
  </s:Header>
  <s:Body>
    <GetUserResponse xmlns="https://bingads.microsoft.com/Customer/v13">
      <User xmlns:a="https://bingads.microsoft.com/Customer/v13/Entities" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
        <a:ContactInfo>
          <a:Address>
            <a:City i:nil="true" />
            <a:CountryCode>US</a:CountryCode>
            <a:Id i:nil="true" />
            <a:Line1 i:nil="true" />
            <a:Line2 i:nil="true" />
            <a:Line3 i:nil="true" />
            <a:Line4 i:nil="true" />
            <a:PostalCode i:nil="true" />
            <a:StateOrProvince i:nil="true" />
            <a:TimeStamp i:nil="true" />
            <a:BusinessName i:nil="true" />
          </a:Address>
          <a:ContactByPhone>false</a:ContactByPhone>
          <a:ContactByPostalMail>false</a:ContactByPostalMail>
          <a:Email>someone@zappysys.com</a:Email>
          <a:EmailFormat i:nil="true" />
          <a:Fax i:nil="true" />
          <a:HomePhone i:nil="true" />
          <a:Id>48045678</a:Id>
          <a:Mobile i:nil="true" />
          <a:Phone1>111-222-3333</a:Phone1>
          <a:Phone2 i:nil="true" />
        </a:ContactInfo>
        <a:CustomerId>19112345</a:CustomerId>
        <a:Id>48012345</a:Id>
        <a:JobTitle i:nil="true" />
        <a:LastModifiedByUserId>48012345</a:LastModifiedByUserId>
        <a:LastModifiedTime>2019-06-04T14:22:35.38</a:LastModifiedTime>
        <a:Lcid>EnglishUS</a:Lcid>
        <a:Name>
          <a:FirstName>Someone</a:FirstName>
          <a:LastName>Good</a:LastName>
          <a:MiddleInitial i:nil="true" />
        </a:Name>
        <a:Password i:nil="true" />
        <a:SecretAnswer i:nil="true" />
        <a:SecretQuestion>None</a:SecretQuestion>
        <a:UserLifeCycleStatus>Active</a:UserLifeCycleStatus>
        <a:TimeStamp>AAAAAJiX/cU=</a:TimeStamp>
        <a:UserName>someone@zappysys.com</a:UserName>
        <a:ForwardCompatibilityMap i:nil="true" xmlns:b="http://schemas.datacontract.org/2004/07/System.Collections.Generic" />
      </User>
      <CustomerRoles xmlns:a="https://bingads.microsoft.com/Customer/v13/Entities" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
        <a:CustomerRole>
          <a:RoleId>41</a:RoleId>
          <a:CustomerId>19160000</a:CustomerId>
          <a:AccountIds i:nil="true" xmlns:b="http://schemas.microsoft.com/2003/10/Serialization/Arrays" />
          <a:LinkedAccountIds xmlns:b="http://schemas.microsoft.com/2003/10/Serialization/Arrays" />
          <a:CustomerLinkPermission i:nil="true" />
        </a:CustomerRole>
      </CustomerRoles>
    </GetUserResponse>
  </s:Body>
</s:Envelope>

POST http://myhost/api/v2/customers/new
X-SecretKey: Abcdxxxxxxxxxxxxxx
Content-Type: application/json

{ firstname: "Someone", lastname: "Good" }

{
 request_status: "CreatedOK",
 new_customer_id: "1234566"
}

Step-By-Step: How to Import Bing Ads Data into SQL Server

Now let’s look at how to import Bing Ads Data into SQL Server Table. This tutorial will require 10-15 mins of your time. At the end of this tutorial you will be able to Read Bing Ads Performance Report and Import data into SQL Server. Step listed in this tutorial may help you to call Bing API in other tools like SSIS, Power BI, SSRS, Tableau and so on.

So let’s get started.

Register OAuth App for Bing Ads API (Azure Active Directory App)

Very first thing to call any Bing API is to register OAuth App (Under Azure Active Directory Tab).

Here is how to register OAuth App which can be used to call Bing Ads API later on.

1. Login to Azure Portal:
   • Navigate to the Azure Portal and log in using your credentials.
2. Access Azure Active Directory:
   • In the left-hand menu, click on Azure Active Directory.
3. Register a New Application:
   • Go to App registrations and click on New registration.
     
   • Application Name: Enter a name for your application.
     
   • Supported Account Types: Choose the account types your app will support. For example, select Accounts in this organizational directory only if you need access to data in your organization only.
   • Redirect URI:
     • Set the type to Web.
     • In the textbox enter https://login.microsoftonline.com/common/oauth2/nativeclient as the Redirect URI or any other valid redirect URL, e.g., https://zappysys.com/oauth.
     • Use this Redirect URI in the Redirect URL grid row.
4. Save Client ID:
   • After registering the app, copy the Application (client) ID and paste it into the Client ID field in the API Connection Manager configuration.
5. Set Authorization & Token URLs:
   • Click on the Endpoints link in the App registration overview.
   • Authorization URL: Copy the OAuth 2.0 authorization endpoint (v2) URL (e.g., https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/authorize) and paste it into the Authorization URL field in the configuration grid.
   • Token URL: Copy the OAuth 2.0 token endpoint (v2) URL (e.g., https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/token) and paste it into the Token URL field.
6. Create a Client Secret:
   • In the Certificates & secrets tab, click New client secret.
   • Set an expiration period for the secret.
   • Copy the generated client secret and paste it into the Client Secret field in the API Connection Manager configuration.
7. Configure API Permissions:
   • Go to the API Permissions section.
   • Click on Add a permission, select Microsoft Graph, and choose Delegated Permissions.
   • Add the required permissions:
     • offline_access
     • openid
     • profile
     • Sites.Read.All
     • Sites.ReadWrite.All
     • User.Read
     • email
     

     Azure AD App – Required API Permissions

   • Grant Admin Consent for the permissions that require it.
8. Generate Tokens:
   • Use the Generate Token feature in the API Connection Manager to generate authentication tokens.
9. Use a Generic Account for Automation:
   NOTE: If you are planning to use your current data connection/token for automated processes, we recommend that you use a generic account for token generation when the login box appears (e.g. sales_automation@mycompany.com instead of bob_smith@mycompany.com). When you use a personal account which is tied to a specific employee profile and that employee leaves the company, the token may become invalid and any automated processes using that token will fail. Another potentially unwanted effect of using a personal token is incorrect logging; the API calls (e.g. Read, Edit, Delete, Upload) made with that token will record the specific user as performing the calls instead of an automated process.
10. That’s it!

Get Bing API Developer Token

Once we have OAuth App registered next thing is to obtain Developer Token for Bing Ads API. Click here to request new token or view existing Developer token.

Create new Bing Ads API Data Source in Gateway

Once you setup Data Gateway user we can move to next step. To call Bing API we need to setup Connection for XML Driver in ZappySys Data Gateway.

1. Click on Add New Data source. Name as BingAds and select Native – ZappySys XML Driver option.
   

   Add Gateway Data Source

2. Now click Edit to Configure Settings. For now we will only care OAuth Connection settings. All other options are overwritten in SQL Query.
   

   Edit Gateway Data Source Settings

3. When UI opens Click on Load ConnectionString and Paste below connection string after changing some items (Screenshot may be differ). Make sure to replace following attributes.RefreshTokenFilePath – Enter some valid file path which will hold refresh token
   ClientId – get Application Id from Azure Portal under App we created earlier)
   DeveloperToken – Replace 1052Bxxxxxxxxxxxx with your own devloper token.
   

   Load ZappySys Driver ConnectionString to configure UI

   Use below string and replace parameters 

   DRIVER={ZappySys XML Driver};
   DataPath='https://clientcenter.api.bingads.microsoft.com/Api/CustomerManagement/v13/CustomerManagementService.svc';
   DataConnectionType=OAuth;
   AuthUrl='https://login.microsoftonline.com/common/oauth2/v2.0/authorize';
   TokenUrl='https://login.microsoftonline.com/common/oauth2/v2.0/token';
   ReturnUrl='https://login.microsoftonline.com/common/oauth2/nativeclient';
   RefreshTokenFilePath='c:\xxxxxx_some_folder_xxxxxxx\BingAds_RefreshToken.txt';
   ScopeSeparator='{space}';
   Scope='https://ads.microsoft.com/ads.manage offline_access';
   ClientId='xxxxxxxxxxxxxx';
   UseCustomApp=True;
   RequestData='<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
     <s:Header>
       <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Customer/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Customer/v13"><%access_token%></h:AuthenticationToken>
       <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Customer/v13">1052Bxxxxxxxxxxxx</h:DeveloperToken>
       <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Customer/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Customer/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
     </s:Header>
     <s:Body>
       <GetUserRequest xmlns="https://bingads.microsoft.com/Customer/v13">
         <UserId i:nil="true" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       </GetUserRequest>
     </s:Body>
   </s:Envelope>';
   RequestContentTypeCode=TextXml;
   RequestMethod='POST';
   RequestHeaders='SOAPAction: "GetUser"'

4. Next to OAuth Click on Configure Settings
5. By default most of settings on OAuth connections should be pre-populated for you but still lets Notice few things. This is how to setup redirect URI
6. 

   Configure Redirect URI for Bing Ads API OAuth Connection

7. Here is where you setup Changing Refresh Token File path. if you dont supply this path then your refresh token extracted first time will expire in 60 days.
   

   Refresh Token File path (Only needed if API issues new Refresh Token when you request AccessToken)

8. Once you verify all settings click Generate Token and Finish the process. It will populate AccessToken and RefreshToken Fields if everything goes well.
   

   Configure Bing Ads API Connection (OAuth 2.0)

9. Click OK to save OAuth Connection UI to go back to main UI
10. You can now Click Test Connection to confirm everything
11. If you need to access Gateway from Other machine then go to Firewall tab and Click Add Rule
12. Click Save button on Gateway UI to save and restart the service.

Create a new Linked Server for Bing Ads API

Once we setup setup Bing Ads Data Source in Gateway we can move to next step which is create a Linked Server which points to Data gateway Data source we created. This will create a bridge between ZappySys Driver and SQL Server so you can call API via T-SQL.

1. If you are using Latest ODBC PowerPack then on Data gateway UI you will See App Integration Tab. Copy Code and run it in SSMS to create a new linked server.
   

   ZappySys Data Gateway code generator for Linked Server

2. If things go well when you execute code you may see some result in grid.

Run sample Queries to call Bing API in SQL Server / Import Data

Once Linked Server is created we can run various queries like below.

Example-1

--use all default settings from gateway data source UI
--Read
Select * from OPENQUERY( [YourLinkedServer] , 'select * from $') 

/*
--import
Select * into #tempTable from OPENQUERY( [YourLinkedServer] , 'select * from $') 
Select * from #tempTable
*/

Example-2

--use Filter to extract specific Node from response XML
Select * from OPENQUERY( [YourLinkedServer] , 
'select * from $
WITH(Filter=''$.s:Envelope.s:Body.GetUserRequest[*]'', 
  ElementsToTreatAsArray=''GetUserResponse'' 
   )'
)

Example-3

--Override URL, Headers and Body Settings. Change DeveloperToken in below request
--Header, Filter, URL, Request data is different for each Endpoint

SELECT * FROM OPENQUERY([YourLinkedServer]
, 'SELECT * FROM $
WITH(
	 ElementsToTreatAsArray=''GetUserResponse''
	,Src=''https://clientcenter.api.bingads.microsoft.com/Api/CustomerManagement/v13/CustomerManagementService.svc''
	,Filter=''$.s:Envelope.s:Body.GetUserResponse[*]''	
	,Header=''SOAPAction: "GetUser"''	
	,RequestData=''<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v13="https://bingads.microsoft.com/Customer/v13">
<soapenv:Header>
   <v13:DeveloperToken>1052xxxxxxxxxxxxx</v13:DeveloperToken>
   <v13:AuthenticationToken><%access_token%></v13:AuthenticationToken>
</soapenv:Header>
<soapenv:Body>
   <v13:GetUserRequest>
      <v13:UserId xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/>
   </v13:GetUserRequest>
</soapenv:Body>
</soapenv:Envelope>''
)')

Download Bing Ads Performance Data in SQL Server Table

Now lets look at more complex example to read data from Bing Performance Report. Calling This API is tricky due to many steps involved in data extraction. But not to worry if you use ZappySys Drivers. This is API is somewhat similar as Amazon MWS Report Request (See last section).

Here are high level steps needs to be performed to call API for Bing Ads Report (e.g. Performance Report)

1. Call SubmitGenerateReportResponse API to request new Report. This returns ReportRequestId
2. Now we have to keep checking every few seconds see report is ready or not. Call PollGenerateReport until Status field is Success from Pending
3. Once previous step returns Success extract Report Download URL (Its Zip File which may contain data in CSV or XML)

So lets get started to implement above login in a single T-SQL Query and load data into Table. There is a known limitation with Query Size we can send to data gateway from SSMS so we can use some tricks to read Body from File rather than hard coding in SQL. We can use IsMultiPart=True and then supply Filepath in RequestBody (Path must start with @ symbol)

1. Bing Report API needs 2 additional values along with DeveloperToken we got earlier. You need to supply AccountId and CustomerId part of Body so obtain it by visiting below page in your bing account.
   Visit https://ads.microsoft.com/cc/accounts
   

   Obtain Bing Ads AccountId, ClientId for APi calls

2. Create new XML file for first step in a Notepad. Enter below content and save as C:\Requests\BingAds\sql_body1.xml
   Replace Following Values as per your settingsCustomerAccountId (Found 2 places)
   CustomerId
   DeveloperToken
   

   <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
     <s:Header>
       <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Reporting/v13"><%access_token%></h:AuthenticationToken>
       <h:CustomerAccountId xmlns:h="https://bingads.microsoft.com/Reporting/v13">481xxxxxxxxxxx</h:CustomerAccountId>
       <h:CustomerId xmlns:h="https://bingads.microsoft.com/Reporting/v13">191xxxxxxxxxxx</h:CustomerId>
       <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Reporting/v13">1052Bxxxxxxxxxxx</h:DeveloperToken>
       <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
     </s:Header>
     <s:Body>
       <SubmitGenerateReportRequest xmlns="https://bingads.microsoft.com/Reporting/v13">
         <ReportRequest i:type="CampaignPerformanceReportRequest" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
           <ExcludeColumnHeaders>false</ExcludeColumnHeaders>
           <ExcludeReportFooter>false</ExcludeReportFooter>
           <ExcludeReportHeader>false</ExcludeReportHeader>
           <Format>Xml</Format>
           <ReportName>My Campaign Performance Report</ReportName>
           <ReturnOnlyCompleteData>false</ReturnOnlyCompleteData>
           <Aggregation>Hourly</Aggregation>
           <Columns>
             <CampaignPerformanceReportColumn>TimePeriod</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AccountId</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>CampaignId</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>CampaignName</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>CampaignStatus</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>DeviceType</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>BidMatchType</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>QualityScore</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AdRelevance</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>LandingPageExperience</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Revenue</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Assists</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>ExpectedCtr</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>DeliveredMatchType</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AveragePosition</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Conversions</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AdDistribution</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Network</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Clicks</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Impressions</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Ctr</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AverageCpc</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Spend</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>LowQualityClicks</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>LowQualityConversionRate</CampaignPerformanceReportColumn>
           </Columns>
           <Filter>
             <AccountStatus i:nil="true" />
             <AdDistribution i:nil="true" />
             <DeviceOS i:nil="true" />
             <DeviceType i:nil="true" />
             <Status i:nil="true" />
           </Filter>
           <Scope>
             <AccountIds xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
               <a:long>481xxxxxxxxxxx</a:long>
             </AccountIds>
             <Campaigns i:nil="true" />
           </Scope>
           <Time>
             <CustomDateRangeEnd i:nil="true" />
             <CustomDateRangeStart i:nil="true" />
             <PredefinedTime>Yesterday</PredefinedTime>
             <ReportTimeZone>PacificTimeUSCanadaTijuana</ReportTimeZone>
           </Time>
         </ReportRequest>
       </SubmitGenerateReportRequest>
     </s:Body>
   </s:Envelope>

    
3. Create another XML file for second step in a Notepad. Enter below content and save as C:\Requests\BingAds\sql_body2.xml
   Replace following values as per your need.
   

   <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
     <s:Header>
       <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Reporting/v13"><%access_token%></h:AuthenticationToken>
       <h:CustomerAccountId xmlns:h="https://bingads.microsoft.com/Reporting/v13">480xxxxxxxxxxx</h:CustomerAccountId>
       <h:CustomerId xmlns:h="https://bingads.microsoft.com/Reporting/v13">191xxxxxxxxxxx</h:CustomerId>
       <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Reporting/v13">1052Bxxxxxxxxxxx</h:DeveloperToken>
       <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
     </s:Header>
     <s:Body>
       <PollGenerateReportRequest xmlns="https://bingads.microsoft.com/Reporting/v13">
         <ReportRequestId>[$a.ReportRequestId$]</ReportRequestId>
       </PollGenerateReportRequest>
     </s:Body>
   </s:Envelope>

    
4. Now try to execute below query. This should generate Performance report and download it in a SQL table for you.
   

   select * into tmpBingAdsReport from OPENQUERY(Zs_BingAds,
   'SELECT c.* FROM $
   WITH(
   	--submit report request for XML format
   
   	 alias=''a'' 
   	,Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
   	,Filter=''$.s:Envelope.s:Body.SubmitGenerateReportResponse''
   	,RequestData=''@C:\Requests\BingAds\sql_body1.xml''
   	,IsMultiPart=''True''
   	,RequestContentTypeCode=''TextXml''
   	,Header=''SOAPAction: "SubmitGenerateReport"''
   	,RequestMethod=''POST''
   	,EnableBodyPlaceholderForMultiPart=''True''
   	,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''
   
   	--keep checking status until report is ready - returns final url when done
   
   	,join1_alias=''b''
   	,join1_Filter=''$.s:Envelope.s:Body.PollGenerateReportResponse.ReportRequestStatus''
   	,join1_RequestData=''@C:\Requests\BingAds\sql_body2.xml''
   	,join1_IsMultiPart=''True''
   	,join1_Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
   	,join1_RequestContentTypeCode=''TextXml''
   	,join1_Header=''SOAPAction: "PollGenerateReport"''
   	,join1_RequestMethod=''POST''
   	,join1_EnableBodyPlaceholderForMultiPart=''True''
   	,join1_EnableStatusCheck=''True'', 
   	,join1_StatucCheckMaxWaitSeconds=300, --wait max 5 mins?
   	,join1_StatucCheckIterationWaitSeconds=5, --check every 5 sec?
   	,join1_StatusSuccessValue=''Success'', --look for success word in response
   
   	--Download report (Zip file) and Parse rows
   
   	,join2_alias=''c''
   	,join2_DataConnectionType=''Default''
   	,join2_Src=''[$b.ReportDownloadUrl$]''
   	,join2_Filter=''$.Report.Table.Row[*]''
   	,join2_ElementsToTreatAsArray=''Row''	
   	,join2_RequestMethod=''GET''
   	,join2_IsMultiPart=''False''
   	,join2_FileCompressionType=''Zip''	
   )')
   
   select * from tmpBingAdsReport

   

   Import Bing Ads data into SQL Server

Multi Step Query to download Report

So in previous section we saw how to write a single query to get your Performance Report from Bing Ads. However there will be time you want to split 3 steps into seperate queries.

Here is how to do.

/*
RPC and RPC OUT options must be turned on to run this proc

EXEC master.dbo.sp_serveroption @server=N'Zs_BingAds', @optname=N'rpc', @optvalue=N'true'
EXEC master.dbo.sp_serveroption @server=N'Zs_BingAds', @optname=N'rpc out', @optvalue=N'true'
--Below needed to support EXEC + INSERT (dynamic query)
EXEC master.dbo.sp_serveroption @server=N'Zs_BingAds, @optname=N'remote proc transaction promotion', @optvalue=N'false'
Also 
1. Change Body file path
2. Change CustomerAccountId, CustomerId, DeveloperToken
*/

ALTER proc usp_Api_BingAds_Reports
as
--///////////////////////////////////////////////////////////////////////////////
--Step-1 - submit report request for XML format
--///////////////////////////////////////////////////////////////////////////////
create table #tmpBingAdsReport_Step1(ReportRequestId bigint)
INSERT into #tmpBingAdsReport_Step1
EXEC(
'SELECT * FROM $
WITH(
 Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,Filter=''$.s:Envelope.s:Body.SubmitGenerateReportResponse''
,RequestData=''@c:\BWProjects\Requests\BingAds\sql_body1.xml''
,IsMultiPart=''True''
,RequestContentTypeCode=''TextXml''
,Header=''SOAPAction: "SubmitGenerateReport"''
,RequestMethod=''POST''
,EnableBodyPlaceholderForMultiPart=''True''
,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''
)') AT zs_BingAds


--///////////////////////////////////////////////////////////////////////////////
--Step2 - keep checking status until report is ready - returns final url when done
--///////////////////////////////////////////////////////////////////////////////

declare @reportid varchar(100)
select top 1 @reportid=cast(ReportRequestId as varchar(100)) from #tmpBingAdsReport_Step1

create table #tmpBingAdsReport_Step2(ReportDownloadUrl varchar(500))
INSERT into #tmpBingAdsReport_Step2 
EXEC(
'SELECT * FROM $
WITH(
 Filter=''$.s:Envelope.s:Body.PollGenerateReportResponse.ReportRequestStatus''
,RequestData=''<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Header>
    <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
    <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Reporting/v13"><%access_token%></h:AuthenticationToken>
    <h:CustomerAccountId xmlns:h="https://bingads.microsoft.com/Reporting/v13">480xxxxxxxxx</h:CustomerAccountId>
    <h:CustomerId xmlns:h="https://bingads.microsoft.com/Reporting/v13">191xxxxxxxxx</h:CustomerId>
    <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Reporting/v13">105xxxxxxxxxxx</h:DeveloperToken>
    <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
    <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
  </s:Header>
  <s:Body>
    <PollGenerateReportRequest xmlns="https://bingads.microsoft.com/Reporting/v13">
      <ReportRequestId>'+ @reportid +'</ReportRequestId>
    </PollGenerateReportRequest>
  </s:Body>
</s:Envelope>''
--,IsMultiPart=''True''
,Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,RequestContentTypeCode=''TextXml''
,Header=''SOAPAction: "PollGenerateReport"''
,RequestMethod=''POST''
--,EnableBodyPlaceholderForMultiPart=''True''
,EnableStatusCheck=''True'', 
,StatucCheckMaxWaitSeconds=300, --wait max 5 mins?
,StatucCheckIterationWaitSeconds=5, --check every 5 sec?
,StatusSuccessValue=''Success'', --look for success word in response
,Meta=''[{"Name": "ReportDownloadUrl","Type": "String",Length:"300"}]''
)') AT zs_BingAds


declare @reportpath varchar(500)
select top 1 @reportpath=ReportDownloadUrl from #tmpBingAdsReport_Step2

select * from #tmpBingAdsReport_Step1
select * from #tmpBingAdsReport_Step2

--///////////////////////////////////////////////////////////////////////////////
--Step-3 - Get report file
--///////////////////////////////////////////////////////////////////////////////
--Static table
--//INSERT INTO BingAdsReport
EXEC(
'SELECT * FROM $
WITH(
 DataConnectionType=''Default''
,Src='''+ @reportpath +'''
,Filter=''$.Report.Table.Row[*]''
,ElementsToTreatAsArray=''Row''	
,RequestMethod=''GET''
,IsMultiPart=''False''
,FileCompressionType=''Zip''	
)') AT zs_BingAds


--select * from tmpBingAdsReport_Step3
--select * from openquery(Bing, 'select * from $')


/*
select * into tmpBingAdsReport from OPENQUERY(zs_BingAds,
'SELECT c.* FROM $
WITH(
--submit report request for XML format

alias=''a'' 
,Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,Filter=''$.s:Envelope.s:Body.SubmitGenerateReportResponse''
,RequestData=''@c:\BWProjects\Requests\BingAds\sql_body1.xml''
,IsMultiPart=''True''
,RequestContentTypeCode=''TextXml''
,Header=''SOAPAction: "SubmitGenerateReport"''
,RequestMethod=''POST''
,EnableBodyPlaceholderForMultiPart=''True''
,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''

--keep checking status until report is ready - returns final url when done

,join1_alias=''b''
,join1_Filter=''$.s:Envelope.s:Body.PollGenerateReportResponse.ReportRequestStatus''
,join1_RequestData=''@c:\BWProjects\Requests\BingAds\sql_body2.xml''
,join1_IsMultiPart=''True''
,join1_Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,join1_RequestContentTypeCode=''TextXml''
,join1_Header=''SOAPAction: "PollGenerateReport"''
,join1_RequestMethod=''POST''
,join1_EnableBodyPlaceholderForMultiPart=''True''
,join1_EnableStatusCheck=''True'', 
,join1_StatucCheckMaxWaitSeconds=300, --wait max 5 mins?
,join1_StatucCheckIterationWaitSeconds=5, --check every 5 sec?
,join1_StatusSuccessValue=''Success'', --look for success word in response

--Download report (Zip file) and Parse rows

,join2_alias=''c''
,join2_DataConnectionType=''Default''
,join2_Src=''[$b.ReportDownloadUrl$]''
,join2_Filter=''$.Report.Table.Row[*]''
,join2_ElementsToTreatAsArray=''Row''	
,join2_RequestMethod=''GET''
,join2_IsMultiPart=''False''
,join2_FileCompressionType=''Zip''	
)')

select * from tmpBingAdsReport

select * from openquery(Bing, 'select * from $')
*/
go

Conclusion

In this article we saw how to import Bing Ads API data in SQL Server without any coding using ZappySys ODBC PowerPack (SOAP / XML Driver). Download it it and try it for FREE to explore many API integration features not discussed in this article.

The post Import Bing Ads data into SQL Server (Performance Reports) appeared first on ZappySys Blog.

In this post we will learn how to read / write REST API data in Talend Open Studio. We will create a simple Talend Job using ZappySys JSON  Driver to read from REST API / JSON Files and load into Target (e.g. File / DB). Techniques listed in this article can be also used to read from SOAP API / XML Files or CSV Files / API using XML Driver / CSV Driver.

These drivers support familiar SQL query language. Using SQL you can query virtually any API services just like relational database table. It can flatten nested hierarchy and provide output in rows / columns. Many complex REST API / SOAP API complexity is taken care automatically (e.g. Authentication, Pagination, Security, Error Handling).

So let’s get started.

Requirements

1. Download and install Talend Open Studio (FREE) from here. Skip this step if you already installed.
2. Download ZappySys ODBC PowerPack (JSON / XML Drivers)
3. Get Microsoft JDBC driver for SQL Server from here (Download sqljdbc_6.0.8112.200_enu.exe which is self extracting file you can run and extract to some folder)
   After you extract jdbc files, go to sqljdbc_6.0\enu\jre8\ folder rename sqljdbc42.jar to  mssql-jdbc.jar (name must be this). We will load this file in Talend later in this article.
4. Basic knowledge about REST API and JSON / XML format.

Configure Data Gateway

Register MS SQL JDBC driver in Talend

Now lets register Microsoft JDBC Driver in Talend. This is very important step because MSSQL JDBC driver is used to communicate with ZappySys Data Gateway we configured in previous step.

If you missed steps mentioned in the Requirements section then make sure you first download JDBC driver using below steps.

Get Microsoft JDBC driver for SQL Server from here (Download sqljdbc_6.0.8112.200_enu.exe which is self extracting file you can run and extract to some folder). After you extract jdbc files, go to sqljdbc_6.0\enu\jre8\ folder rename sqljdbc42.jar to  mssql-jdbc.jar (name must be this).

Now lets go through the steps to register MSSQL jdbc driver in Talend.

1. Open Talend Open Studio
2. Go to Windows > Click Show View
   

   Talend – Show View

3. When you see Popup selection under Talend > Select Modules
   

   Talend – Select Modules Window

4. When Module window is visible click on Little Jar Icon (Bottle icon) in the toolbar.
5. Select mssql-jdbc.jar file we renamed earlier and load this file.
   

   Talend – Import JAR file / module (Register MS SQL JDBC Driver Example)

6. That’s it. Now we ready to make API calls / read from JSON / XML in the next section.

Setup Talend REST API Connection (JSON / XML / CSV)

Now let’s configure REST API Connection in Talend. To read from JSON / XML Files you can use same steps too. We will use MSSQL JDBC Driver to connect to ZappySys Data Gateway.

1. In Talend Go to Metadata > Db Connections (Right click) > Create Connection
   

   Talend – Create new DB connection (JSON / REST API Example)

2. On the connection Wizard specify following attributes.
   DB Version : Microsoft
   Login :  username you setup in zappysys data gateway
   Password :  password of data gateway user
   Server : machine name or IP where zappysys data gateway is running
   Port : <default is 5000> Port on which zappysys data gateway is listening
   DataBase :  Data source name you setup in zappysys data gateway (case-sensitive)
   

   Talend – Connect to JSON / REST API using ZappySys Gateway (Use MS SQL JDBC Driver)

3. That’s all we need to do to setup a connection which can be used to read / write REST API data in Talend. In the next section we will see how to create a job to read data from REST API Service using this connection.

Read from REST API in Talend

Now let’s look at how to read data from REST API source or JSON / XML File using the connection we configured in the previous section.

Configure REST API Source

1. Create a Talend JOB and double click to open designer
2. Now drag and drop MSSQL Connection we created for ZappySys Data gateway, drop it on the designer surface. It will popup UI like below.
3. Select tDBInput (Microsoft SQL Server). Remember that we are using MSSQL JDBC Driver to connect to ZappySys Data Gateway for REST API Call. This gateway uses Microsoft TDS Protocol so MSSQL JDBC driver is used to communicate.
4. Now rename Source to something meaningful (e.g. Read from JSON REST API)
5. Double click REST Source to configure.
6. Enter Query like below (Make sure to enter between double quotes). See below examples to read from URL or File. If you have double quote in SQL then escape using \” character (e.g. select \”my col\” from $ )Read From REST API Url
   

   "SELECT * FROM value 
   WITH (SRC='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json')"

   Read from Local File (Wildcard Pattern Search allowed)
   

   "SELECT * FROM value 
   WITH (SRC='c:\data\Customers_*.json')"

7. Click on Guess Schema button and Click OK to accept detected schema.
   

   Talend – Configure REST API / JSON Source (Enter SQL Query / Guess Schema)

8. Now we will configure target in the next section.

Configure Target (Delimited File)

1. Now search for “FileOut” in the toolbox (Hit Enter). You will see tFileOutputDelimited so just select that for now and drag on the surface.
   

   Talend – Add File Output (tFileOutputDelimited)

2. Double click it to configure.
3. Enter correct file path (e.g. “C:/Talend/workspace/rest-api-output.csv” )
4. On Advanced Tab you can configure some additional settings (e.g. Throw an error if file already exists)

Connect and Run

1. Once you have configured Source and Target its time to connect them
2. Drag Source Port to Target to connect like below.
   

   Talend – Connect REST / JSON Source to File Target

3. Run the job
   

   Run Talend Job – Loading REST API data into File (Read JSON / XML / CSV)

4. That’s it. So in few clicks you loaded data from REST API to File in Talend Open Studio.

Write / Send data to REST API (POST Example)

There will be a time when you want to POST data to REST API service. Let’s check how to write POST query to submit data to REST API.

Just like how we did Read query in previous example, we can set POST Body in the SQL Query to send data. Use query like below and click Guess Schema button. If Blank Filter gives you no data error then make sure you remove Filter on Data Gateway Data source. (Notice we used \” to escape double quote inside query )

"SELECT * FROM _root_
WITH (
 METHOD='POST'
,HEADER='Content-Type: text/plain || first-header: AAA || second-header: BBB'
,SRC='http://httpbin.org/post'
,BODY='{id:1,notes:\"Some notes\"}'
,FILTER=''
)
"

Talend – POST data to REST API URL

SQL Query Examples

Click on below link to learn more writing SQL Query using ZappySys Drivers.

JSON / REST Driver – SQL Query Examples

XML / SOAP Driver – SQL Query Examples

CSV / REST Driver – SQL Query Examples

REST API / XML SOAP Pagination Settings for Talend

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

REST API / SOAP Web Service Connection Settings for Talend

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Other settings for REST API / SOAP XML Call in Talend

REST API / XML SOAP Performance Tips for Talend

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Calling SOAP Web Service in Talend

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

Conclusion

In this article, we used ZappySys Drivers to read data from JSON REST API / File. You can use same technique to consume SOAP / XML API or File. Try ODBC PowerPack for FREE and check out how easy it is to consume virtually any REST / SOAP API or Read from JSON / XML / CSV Files in  Talend Open Studio.

The post Read / Write REST API data in Talend (JSON / XML / SOAP) appeared first on ZappySys Blog.

In this post we will learn how to access data from Oracle UCM Web Service (Middle layer for WebLogic) and load into SQL Server or any other target. We will use SSIS XML Source to achieve this result.

About Oracle UCM Web Service

If you are not sure what is SOAP Web Service and how to call in SSIS, First read this article to understand general concepts of calling SOAP API in SSIS.

UCM Web Service offers calling various commands. You have to send XML Request and server will send you XML response with requested data. We will use Array Transformation Technique (Key / Value Pattern) to extract complex nested XML data.

Step-By-Step

Now let’s look at how to call Oracle UCM SOAP API in SSIS.

Read from Oracle UCM Web Service using XML Source

1. Drag Data Flow Task in SSIS Designer
2. Inside Data flow drag ZS SSIS XML Source from toolbox
3. First enter URL as below (Replace INSTANCE ID with your own instance)
   

   https://YOUR-INSTANCE-ID.oraclecloud.com/idcws/GenericSoapPort

4. Now Check Use Credentials and select new ZS-HTTP connection
5. Enter your User ID and Password to call Web Service using Basic Authentication. Click OK to save.
   

   Configure HTTP Connection – Call Oracle UCM Web Service (SOAP XML)

6. On XML Source UI Select Content Type = XML (text / xml, charset=UTF-8)
7. Select Request Method as POST
8. Click Enter Raw Edit for HTTP Headers and enter below
   

   SOAPAction: "urn:GenericSoap/GenericSoapOperation"

9. Click on Edit Body and enter your Request Body in XML format.For example to call GET_SEARCH_RESULTS service to retrieve the search results for the passed query text you can submit Request Body as below.
   NOTICE:  String in Search Text was surrounded by backtick (  `  ). Single tick / double quote didn’t work in our case.
   

   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ucm="http://www.oracle.com/UCM">
   <soapenv:Body>
     <ucm:GenericRequest webKey="cs">
       <ucm:Service IdcService="GET_SEARCH_RESULTS">
         <ucm:Document>
           <ucm:Field name="QueryText">dDocTitle &lt;starts&gt; `MY_sites_201`</ucm:Field>
         </ucm:Document>
       </ucm:Service>
     </ucm:GenericRequest>
   </soapenv:Body>
   </soapenv:Envelope>

   
10. Now Select Filter (Browse upto Fields under SearchResult node. Once you click OK try to edit your Expression so it looks like below. As you may notice we changed ResultSet[*] to ResultSet[?(@@name==’SearchResults’)]
    This will ensure that we extract records from only SearchResult Node and not other nodes such as UserAttribInfo or EnterpriseSearchResults
    

    $.env:Envelope.env:Body.ns2:GenericResponse.ns2:Service.ns2:Document.ns2:ResultSet[?(@@name=='SearchResults')].ns2:Row[*]

    

    Select Filter

11. Now go to 2D Array Tab. Select Key / Value Pattern Option.
12. Select Column Names / Value Filters as below. You can Use Browse Option to navigate to that field or just type below directly. Below expression will Transform Rows into Column. Foreach attribute name we will get one column.Column Name Filter =>  $.ns2:Field[*].@name
    Column Value Filter =>  $.ns2:Field[*].#text
13. Now Click Preview to confirm it works
    
14. That’s it, you can now click OK to close UI.

Loading Oracle UCM SOAP API data into SQL Server / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

In this post we saw how easy it is to consume XML SOAP Web Service in SSIS. We called SOAP Service for Oracle UCM Web Service and loaded data into target like SQL Server. You can download SSIS PowerPack and try many other scenarios like this using 70+ other SSIS Tasks /Components not mentioned in this article.

The post Call Oracle UCM Web Service in SSIS (Read XML SOAP API) appeared first on ZappySys Blog.

In this post, we will learn how to import REST API in MS Access (JSON or SOAP XML data).  We will use ODBC PowerPack to connect and query a JSON file / URL.

JSON stands for JavaScript Object Notation and it is an Open and Standard format to read an object with attributes and values.  JSON is replacing XML because it is faster to read data, it is easier to parse data, it does not require end tags, it is shorter and it can use arrays.

We will load the data using the ZappySys JSON / REST API driver and then import the data in Access. You can use similar techniques to load SOAP Data using XML / SOAP Driver.

Requirements

This article assumes the following requirements are met before you can follow the steps listed in this article.

1. Make sure that Microsoft Access installed
2. Also, the driver ODBC PowerPack installed

An introduction to Rest API and OData

In this example, we will use OData (Open Data Protocol) to consume REST API. REST API (Representational State Transfer Application Program Interface) allows handling the interoperability between computers and the internet.

In REST API we can handle web services in different formats. In this example, we will work with the Northwind example. The Northwind example is available in this URL:

https://services.odata.org/V3/Northwind/Northwind.svc

1. By default, the data is displayed in XML format. To show the data in JSON use this URL:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/?$format=json

2. There are collections of data like Categories, CustomerDemographic, Customers, Invoices, etc. For example, the following URL will show the data of the categories collection:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Categories?$format=json

3. In the next steps, we will use ZappySys drivers to connect to this URL and query using OData.

Configure ODBC DSN for ZappySys JSON Driver

ODBC driver can be accessed in two modes.

1. Using DSN
2. Without DSN (Supply direct Connection String e.g. DRIVER={ZappySys JSON Driver}; …….. )

In this article, we will use the DSN approach (User DSN). We will first add the ZappySys JSON Driver in the ODBC Data Source Administrator.

Follow these steps to accomplish the task:

1. First, Windows search, write ODBC and select the ODBC Data sources (32 bits)
   

   Open ODBC Data Source

2. As a second step, in ODBC Data source Administrator press the Add button.
   

   Add ZappySys

3. In this step, create the new data source, select ZappySys ODBC Driver.
   

   Add new zappysys json driver

4. Here we have several properties, write a data source name. In this example, the name will be ZappySys JSON to Excel.
5. The Data Source (URL or file path) can specify the URL of the source or if it is a local file, you can specify the local path. In this example, the URL is:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json

   You can also specify a local file path as Data SourceFor a single file:  c:\data\myfile_1.json
   For multiple files: c:\data\myfile_*.json
6. Configure the ODBC JSON Driver like this:
   

   Configure JSON Driver for REST API Call

How to import REST API data to Access

1. In MS Access, go to External Data Ribbon and select New Data Source and select From Other Sources and ODBC Database.
   

   A configuration in MS Access: Import Data From REST API

2. Select the Source and destination of the data, select Import the source data into a new table in the current database and press OK.
   

   Get External Data – ODBC Database

3. Select your newly created JSON Driver from the Machine Data Source and click OK.
   
   

   Get External Data – Select your newly created ODBC Data Source (JSON Driver)

    

    

4. Select tables from the list of tables in the Import Objects screen and click OK.
   
   

   Get External Data – Select tables from Import Objects Screen

5. Select Save Import Steps from the screen and click on Save Import button.
   

   Get External Data – Select Save Import Steps

6. If everything is OK, you will be able to see the data.
   

   Data Imported Successfully in MS Access DB Table

How to link REST API data to Access

The linking process is very similar to importing data into the Access table. Follow steps of the previous paragraph – How to import REST API data to Access – but on step 2 select the second option to create a linked table instead. Then once asked to select a unique record identifier, don’t select anything:

Linked table approach – Don’t select a unique record identifier

REST API / XML SOAP Pagination Settings for MS Access

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

REST API / SOAP Web Service Connection Settings for MS Access

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Other settings for REST API / SOAP XML Call in MS Access

REST API / XML SOAP Performance Tips for MS Access

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Calling SOAP Web Service in MS Access

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

Conclusion

In this article, we learned how to use the ZappySys ODBC PowerPack to import JSON to Access. We used the OData protocol for Import data to Access.  With ZappySys ODBC Power Pack, you can query REST API information or JSON files using SQL and filter the information or write custom queries according to your needs.

References

• JSON – Introduction
• Learn REST: A RESTful Tutorial

The post Import REST API in MS Access (Load JSON / SOAP XML) appeared first on ZappySys Blog.

In our previous blog post we saw how to import SOAP API data in Power BI. Now let’s continue to explore more use cases. In this article we will check how to Connect WorkDay in Power BI. We will show you step by step approach on how to call Workday SOAP API in Power BI and create Power BI reports / dashboards using imported datasets.

We will use ZappySys XML / SOAP Driver for this demonstration.

Requirements

1. First, you will need to have Power BI Desktop installed
2. Make sure you have ZappySys ODBC Power Pack installed
3. Download and have install SoapUI

Get Started

Once above requirements are fulfilled, let’s learn how to use Workday SOAP API and import workday data in Power BI.

Here are high level steps to import Workday data in Power BI using ZappySys XML / SOAP Driver.

1. Craft Workday SOAP request XML using tool like SoapUI for desired operation (e.g Get_Employee )
2. Configure ODBC DSN for Workday SOAP API Connection using ZappySys XML / SOAP Driver. (Select HTTP Connection with SOAP WSS mode using workday Userid / password).
3. Test SQL Query for desired SOAP API call
4. Finally, Import Data in Power BI using ODBC Connection, Supply Custom SQL Query during Import Process (Crafted in previous step).

Now let’s look at each step in detail in the following sections

Craft Workday SOAP request XML using SoapUI

Now let’s look at how to use tool like SoapUI to craft SOAP API requests. It would be helpful if you can refer this article to learn more about using SoapUI tool (Check SoapUI section in that article).

Obtain Workday SOAP WSDL URL (API Metadata URL)

First step to consume workday api using SSIS is download SOAP WSDL file. WSDL is an XML file which describes available API operations in Workday. It also describes the structure of XML request / response. Here is the list of available WSDL for various Workday API Services. You can right click on WSDL icon and save to local disk or use Direct WSDL URL in next section for importing in SoapUI tool.

You can find more information about Other Workday API here.

Obtain Workday API URL

Once you have WSDL file, next step is craft correct URL for API service you like to call. Service name can be obtained from here (Check service column). Notice my-instance and my-tenantid placeholders (Its specific to your company).

Syntax:  

https://<workday host name>.workday.com/ccx/service/<tenant name>/<service-name>

https://MY-INSTANCE.workday.com/ccx/service/MY-TenantID/Human_Resources

Load WSDL in SoapUI

Now it’s time to load WSDL and craft some SOAP Request and test. For detailed steps check this (watch Video to learn SoapUI tool) .

1. Open SoapUI tool
2. Click new File > New Soap Project
3. Enter WSDL Path and Name of the project
4. You will see many different operations in the imported project. For example below screenshot shows Human_Resources service and Get Employee API call. You can fill out necessary parameters in your request / remove optional parameters from XML as needed.
   

   Create SOAP Request Body from WSDL (Using SoapUI tool)

Configure Soap Request Body, URL and Credentials in SoapUI

Now it’s time to configure SOAP Request service URL and Credentials. To generate correct Credentials Header in SOAP Request use tool like SoapUI. Check this article to learn more how to generate WSS Security header.

1. Go to the API method you like to call and double click on the default Request node. Check this article to learn more.
2. Change default request parameters (Add /Remove as per your need). You can click on Re-generate icon in the toolbar to reset the request and create optional parameters.
3. Enter the Service URL
   Example:  https://myinstance0001.workday.com/ccx/service/AB11111111/Human_Resources
4. In the properties window select WSS-Password Type as PasswordDigest, Enter workday username and password
5. Once you done click Play button to execute the request. If things go well then you should see response on your right panel as below.
   

   SoapUI – Call Workday SOAP API (Configure WSS Security, Service URL, Body, Test Request / Response)

Creating ODBC DSN for Workday Connection

Once you test correct parameters in your XML Request we can create ODBC DSN for Workday.

1. Open start menu and search for “ODBC 64”. Launch ODBC Data sources for 64 bit
   

   Open ODBC Data Sources from Startup menu

2. Click Add and select ZappySys XML Driver
   

   ZappySys ODBC Driver for XML / SOAP API

3. Once driver UI is visible change default DSN name (e.g. change something to  WorkdayDSN)
4. Enter API EndPoint URL (Get it from SoapUI) and Select connection type as HTTP.
   Example URL:  https://YOUR-INSTANCE.workday.com/ccx/service/YOUR-TENANT/Human_Resources
5. Configure HTTP connection as below.
   

   Configure Workday Connection in ODBC DSN – XML / SOAP Driver

6. Click OK to save connection.
7. Now Configure other settings as below.
   1. Select Method as POST
   2. Select Content Type as text/xml
   3. Enter one required Header for SOAP API as below
      

      SOAPAction :  ""

   4. Enter Body (Get it from SoapUI ). Here is example body for Get_Employee request.
      

      <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:bsvc="urn:com.workday/bsvc">
         <soapenv:Header/>
         <soapenv:Body>
            <bsvc:Employee_Get>
               <bsvc:Employee_Reference>
                  <bsvc:Integration_ID_Reference>
                     <bsvc:ID>XXXX-1234-CHANGE-THIS-XXXXXXX</bsvc:ID>
                  </bsvc:Integration_ID_Reference>
               </bsvc:Employee_Reference>
            </bsvc:Employee_Get>
         </soapenv:Body>
      </soapenv:Envelope>

      NOTE: Change ID to your own record id.
   5. ( Optional ) If your API returns more than one record (e.g. if you call API like Get_Workers) then select Filter as per below screenshot (Click Yes if asked to treat selection as array). For example in below screenshot we called Get_Workers request which returns multiple records under wd:Worker node. Look for Array Icon in the Filter selection. Skip this step if no array extraction needed.
8. After Configuration your screen may look like below.
   

   Configure Workday SOAP Request – URL, Body, Filter

9. Now go to Preview Tab and you will see default SQL query generated for you. Here is example query (Replace Body part as per your need)
   

   SELECT * FROM $
   /*Below WITH clause is optional, uncomment parameter(s) values you like to set 
    which are different than DSN properties.
   */
   WITH(
   	Src='https://xxxxxxxxxx.workday.com/ccx/service/yyyyyyyyy/Human_Resources'
   	--,ElementsToTreatAsArray='wd:Worker'
   	--,DataConnectionType='HTTP'
   	--,SoapWssPasswordType='PasswordDigest'
   	--,UserName='testuser001'
   	--,CredentialType='SoapWss'
   	--,Filter='$.env:Envelope.env:Body.wd:Get_Workers_Response.wd:Response_Data.wd:Worker[*]'
   	--,RequestContentTypeCode='TextXmlUtf8'
   	--,RequestMethod='POST'
   	--,Header='SOAPAction: ""'	
   	,RequestData='
   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:bsvc="urn:com.workday/bsvc">
      <soapenv:Header/>
      <soapenv:Body>
           .......... .......... .......... 
           .......... FROM SOAPUI .........  
           .......... .......... ..........        
      </soapenv:Body>
   </soapenv:Envelope>'
   )

10. Click Preview button to see data. Copy your SQL Query for later step.
    

    Preview Workday Data (Test SQL query for ODBC XML / SOAP Driver)

11. Click OK to save DSN

Import Workday data in Power BI

Now let’s move to the final step. We will use SQL query and DSN created in previous step for data load in Power BI.

1. Open Power BI Desktop
2. Click on Get Data > More Data… > Other > ODBC
   

   Import SOAP API in Power BI (Using ZappySys XML ODBC Driver)

3. Select your ODBC DSN Source from the dropdown (Assuming you have created a ODBC Data Source using ZappySys XML Driver – See the previous Section)
4. Expand Advanced Options to enter custom Query.
5. Enter your SOAP API call SQL query here. For example purpose, you can use below query to get all JOB categories.
   

   SELECT * FROM $
   WITH(
   	 Src='https://xxxxxxxxxx.workday.com/ccx/service/yyyyyyyyyy/Human_Resources'	
   	--,DataConnectionType='HTTP'
   	--,SoapWssPasswordType='PasswordText'
   	--,UserName='testuser001'
   	--,Password='###########'
   	--,CredentialType='SoapWss'
   	,RequestMethod='POST'
   	,RequestContentTypeCode='TextXmlUtf8'
   	,Filter='$.soapenv:Envelope.soapenv:Body.bsvc:Get_Job_Categories_Response.bsvc:Response_Data.bsvc:Job_Category[*]'
   	,ElementsToTreatAsArray='bsvc:Job_Category'
   	,RequestData='
   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:bsvc="urn:com.workday/bsvc">
      <soapenv:Body>
         <bsvc:Get_Job_Categories_Request/>
      </soapenv:Body>
   </soapenv:Envelope>'
   )

    
   

   Import Workday SOAP Web Service Data in Power BI (ZappySys XML Driver)

6. Once you see data preview click OK to import.
7. After data import is done you can edit your dataset (e.g. remove unwanted columns)
8. You can also edit Source query after data is imported (See below)
   

   Edit Power BI Data Source SQL after REST / SOAP Web Service Import

9. That’s it. You can now create your dashboards using data we just imported. See below example.
   

   Import Workday data in Power BI dashboard

Workday API Pagination – Retrieve more than 100 rows

By default Workday API will return only 100 records unless you specify Page / Count in Response_Filter . Regardless it will only send you 999 maximum rows. To fetch more records after first page you have to setup pagination in XML Driver. Below screenshot

1. Select Paging Mode= POST Data Mode
2. Enter Page Number Indicator (e.g. <%page%> )
3. Click browse button for Max Pages Expression   (select node which indicates how many pages in response. e.g. Total_Pages)
   

   Workday SOAP API Pagination Settings in ZappySys XML / SOAP Driver

4. Once this is done go to Settings Tab. Edit your Body and make sure you supply Response_Filter. Pass Response_Filter parameter as below. See how we used <%page%> placeholder. You can use tool like fiddler to see multiple response at runtime. Run package and check fiddler. In the below example we are fetching maximum 300 rows per response. It will keep reading until all pages are done. For example if you have 1000 rows to read then it will take 4 response behind the scene.
   

   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:bsvc="urn:com.workday/bsvc">
      <soapenv:Body>
         <bsvc:Get_Job_Categories_Request bsvc:version="">
           <bsvc:Response_Filter>
               <bsvc:Page><%page%></bsvc:Page>
               <bsvc:Count>300</bsvc:Count>
            </bsvc:Response_Filter>
         </bsvc:Get_Job_Categories_Request>
      </soapenv:Body>
   </soapenv:Envelope>

   

   Workday API Pagination – Supply Response_Filter (Page Size and Current Page Number)

That’s it. Now if you run your query it will fetch all pages and you can now read more than 1000 rows. Make sure you configure Count at reasonable size (Rows per Page) in above screenshot we use 5 just to demo small sample. We suggest you to to set around 300.

How to pass parameters to SQL query in Power BI

let
    vKey=paraAPIKey,
    Source = Odbc.Query(
"dsn=ZS - OData Customers", 
"SELECT * FROM $ WITH (SRC='http://httpbin.org/post'," 
& "METHOD='POST'," 
& "HEADER='Content-Type:application/json'," 
& "BODY=@'{""CallerId"":1111, ""ApiKey"":""" & vKey & """}')")
in
    Source

How to debug Web Requests using Fiddler

There will be a time you like to see RAW HTTP Requests generated by driver for NetSuite SOAP API.  If you like to do that check this article.

Conclusion

Consuming data from Workday API or any XML / SOAP based API in your Reporting / ETL apps can be tricky. But we have seen how to solve this challenging task using ZappySys XML Driver in few clicks. Download ODBC PowerPack and explore many other API integration scenarios not discussed in this article.

The post Connect Workday in Power BI – Import SOAP Data appeared first on ZappySys Blog.

In this post we will learn how to read data from NetSuite in SSIS. We will use ZappySys XML Source for SOAP API access.

What is NetSuite CRM?

NetSuite is a CRM / ERP product. It gives you scalable cloud CRM / ERP solution targeted at high-growing, mid-sized businesses and large enterprises. It automates front- and back-office processes including: financial management, revenue management, fixed assets, order management, billing, and inventory management.

Preparing for NetSuite WebService Call in SSIS

For NetSuite API call we have to make sure certain settings and obtain Application ID and AccountID. So let’s get started.

How to enable NetSuite Web Service Feature

Before you call any NetSuite SOAP API (i.e. SuiteTask Web Service), First thing to get started with NetSuite SOAP API (i.e. ) is to enable Web Service Feature. Once you enable Web Service, you can programmatically access NetSuite data in your App (e.g. Power BI, Tableau, Informatica). Click this link to see steps to enable Web Service Feature

Enable NetSuite Web Service Feature (SuiteTalk API)

Obtain NetSuite Account ID

NetSuite Account ID is needed by Login API call. So perform following steps if you dont know account id.

1. Login to your NetSuite Portal
2. Click the Setup Tab
3. In the drop-down click Integration >  Web services preferences
4. On that page you will find your Account ID (example account ID = TSTDRV1234567 )

Obtain NetSuite Account ID (Needed for Web Service Login)

Obtain NetSuite Application ID

Once you have account ID now we need to get Application ID. For that follow these steps.

1. Login to your NetSuite Portal
2. Click the Setup Tab
3. In the drop-down click Integration >  Manage Integrations
4. On that page you will find your existing Applications along with Account ID / Status (example Application ID = AE2EACFD-1234-1222-1111-12345544555 )
5. If you don’t have any application created yet then just click on New.

Create new App / Get NetSuite Application ID

Understanding how NetSuite SOAP API Call works

In this section we will briefly understand how NetSuite SOAP API call works. When you use ZappySys Driver / Component it will take care many details for you but still you have to configure correct XML Body. There are two steps involved before you can call any SOAP API in NetSuite.

1. Call Login method and extract JSESSIONID from response Cookie (This token expires after some time)
2. Call any other API (Pass obtained JSESSIONID token in Cookie header)

NetSuite Login Call

Request

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml;charset=UTF-8
SOAPAction: "login"
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
  </soap:Header>
  <soap:Body>
    <login xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <passport>
        <email xmlns="urn:core_2017_1.platform.webservices.netsuite.com">xxxxxxxxxxxxx</email>
        <password xmlns="urn:core_2017_1.platform.webservices.netsuite.com">xxxxxxxxxxxx</password>
        <account xmlns="urn:core_2017_1.platform.webservices.netsuite.com">TSTxxxxxxxxxxxxxxx</account>
        <role internalId="3" xmlns="urn:core_2017_1.platform.webservices.netsuite.com" />
      </passport>
    </login>
  </soap:Body>
</soap:Envelope>

Response Notice JSESSIONID=6xI6xxxxxxxxx in the login call response below. This Session ID can be used to call next call. ZappySys HTTP Connection Manager configured with Dynamic Token option can extract Cookie and reuse it until its expired (1 hour by default).

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 16:50:41 GMT
Content-Length: 1676
Content-Type: text/xml; charset=utf-8
X-N-OperationId: 777274f6-49c6-4883-bcd1-1d70df90d78e
Strict-Transport-Security: max-age=31536000
NS_RTIMER_COMPOSITE: 13633xxxxxxxxx:70617xxxxxxxxxxxxx36F6D:80
Set-Cookie: JSESSIONID=6xI61rdFt26pxxxxxxxxxxxxxxxqvJIMlu!-17981123445; path=/; secure; HttpOnly
Set-Cookie: NS_VER=2018.1.0; domain=webservices.netsuite.com; path=/
Set-Cookie: NS_ROUTING_VERSION=LAGGING; path=/
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent
Keep-Alive: timeout=10, max=990
Connection: Keep-Alive

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_xxxxxxxxxxxxxxxxx_9fc719</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <loginResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <sessionResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
        <ns1:status isSuccess="true" xmlns:ns1="urn:core_2017_1.platform.webservices.netsuite.com"/>
        <ns2:wsRoleList xmlns:ns2="urn:core_2017_1.platform.webservices.netsuite.com">
          <ns2:wsRole>
            <ns2:role internalId="3">
              <ns2:name>Administrator</ns2:name>
            </ns2:role>
            <ns2:isDefault>false</ns2:isDefault>
            <ns2:isInactive>false</ns2:isInactive>
          </ns2:wsRole>
          <ns2:wsRole>
            <ns2:role internalId="15">
              <ns2:name>Employee Center</ns2:name>
            </ns2:role>
            <ns2:isDefault>false</ns2:isDefault>
            <ns2:isInactive>false</ns2:isInactive>
          </ns2:wsRole>
      </sessionResponse>
    </loginResponse>
  </soapenv:Body>
</soapenv:Envelope>

 

NetSuite API Call ( getAll method )

Now once you have JSESSIONID extracted you can pass it to call next API call as below. For example here is how to call getAll method to obtain currency list and exchange rates. If you are not sure how to craft correct Request XML then use tool like SoapUI. You can use NetSuite WSDL file from here and import it in SoapUI.   Request See how Cookie is passed for authentication purpose. Again you can pass this manually or use HTTP connection with Dynamic Token to pass it automatically along with each API call.

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml;charset=UTF-8
SOAPAction: "getAll"
Cookie: JSESSIONID=6xI61rxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxMlu!-1798141234
Accept-Encoding: gzip, deflate

<soapenv:Envelope 
   xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
   xmlns:urn="urn:messages_2018_1.platform.webservices.netsuite.com" 
   xmlns:urn1="urn:core_2018_1.platform.webservices.netsuite.com">
   <soapenv:Body>
      <urn:getAll>
         <record recordType="currency"/> 
      </urn:getAll>
   </soapenv:Body>
</soapenv:Envelope>

Response

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 16:50:42 GMT
Content-Length: 9648
Content-Type: text/xml; charset=utf-8
X-N-OperationId: 1b2989e4-b930-4a72-b471-00a8e131dfd4
NS_RTIMER_COMPOSITE: 21403xxxxxx:706172746E65xxxxxxxxxxxx6F6D:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_TSTDRV1439151_100320189871507121000128715_6ae8e8</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <getAllResponse xmlns="urn:messages_2018_1.platform.webservices.netsuite.com">
      <platformCore:getAllResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>11</platformCore:totalRecords>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>USD</listAcct:name>
            <listAcct:symbol>USD</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>$</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_unitedStatesEnglish</listAcct:locale>
            <listAcct:formatSample>$1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>1.0</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>GBP</listAcct:name>
            <listAcct:symbol>GBP</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>£</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_unitedKingdomEnglish</listAcct:locale>
            <listAcct:formatSample>£1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>1.33165497</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
          <platformCore:record internalId="3" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>CAD</listAcct:name>
            <listAcct:symbol>CAD</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>$</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_canadaEnglish</listAcct:locale>
            <listAcct:formatSample>$1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>0.76457273</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
        </platformCore:recordList>
      </platformCore:getAllResult>
    </getAllResponse>
  </soapenv:Body>
</soapenv:Envelope>

NetSuite Search API (Pagination Example)

Now let's look at more complex API calls where Pagination is needed. For example when you call search API to pull data from specific table (e.g. Account) you need to loop through multiple responses until all records are consumed. Request (Page-1)

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml; charset=utf-8
SOAPAction: "search"
Cookie: JSESSIONID=BKQ7xxxxxxxxxxxx4Ug!-995xxxxxx
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
    <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <pageSize>50</pageSize>
    </searchPreferences>
  </soap:Header>
  <soap:Body>
    <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:AccountSearch">
        <q1:basic>
          <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
            <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">x</searchValue>
          </description>
          <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
            <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">a</searchValue>
          </name>
        </q1:basic>
      </searchRecord>
    </search>
  </soap:Body>
</soap:Envelope>

  Response (Page-1) In below response notice 2 things. SearchId is like a cursor id which you have to pass for each page you request after first page. And you can stop looping after 4th page based on totalPages count. platformCore:searchId = WEBSERVICES_ABCD_12345-xxxxxxxxx platformCore:totalPages = 4

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 18:40:33 GMT
Content-Length: 49266
Content-Type: text/xml; charset=utf-8
X-N-OperationId: c1e79336-63b7-4eb7-989e-5c3d24c56891
NS_RTIMER_COMPOSITE: 118xxxxxxxx:70617xxxxxxxxxxxxxxxxxxxxxD:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="utf-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <searchResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformCore:searchResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>193</platformCore:totalRecords>
        <platformCore:pageSize>50</platformCore:pageSize>
        <platformCore:totalPages>4</platformCore:totalPages>
        <platformCore:pageIndex>1</platformCore:pageIndex>
        <platformCore:searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformCore:searchId>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>1111</listAcct:acctNumber>
            <listAcct:acctName>Checking</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>xxxxxxxxxxxxxxxxxxx</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Checking</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>22222</listAcct:acctNumber>
            <listAcct:acctName>Savings</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>yyyyyyyyy</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Savings</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          .........
          .........
          .........
          .........		  
        </platformCore:recordList>
      </platformCore:searchResult>
    </searchResponse>
  </soapenv:Body>
</soapenv:Envelope>

  Request (Page-2 and onwards) Here is the second page request. Notice few things here. In heraders we are now sending SOAPAction: "searchMoreWithId" rather than "search".  Also our SOAP Body is also different compared to first page request. Now we only sending pageIndex we want to fetch and searchId. We have also change <search> tag with <searchMoreWithId> under Body.

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml; charset=utf-8
SOAPAction: "searchMoreWithId"
Cookie: JSESSIONID=BKQ7xxxxxxxxxxxx4Ug!-995xxxxxx
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
    <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <pageSize>50</pageSize>
    </searchPreferences>
  </soap:Header>
  <soap:Body>
    <searchMoreWithId xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</searchId>
      <pageIndex>2</pageIndex>
    </searchMoreWithId>
  </soap:Body>
</soap:Envelope>

Response (Page-2 and onwards)

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 18:40:33 GMT
Content-Length: 49266
Content-Type: text/xml; charset=utf-8
X-N-OperationId: c1e79336-63b7-4eb7-989e-5c3d24c56891
NS_RTIMER_COMPOSITE: 118xxxxxxxx:70617xxxxxxxxxxxxxxxxxxxxxD:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="utf-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <searchMoreWithIdResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformCore:searchResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>193</platformCore:totalRecords>
        <platformCore:pageSize>50</platformCore:pageSize>
        <platformCore:totalPages>4</platformCore:totalPages>
        <platformCore:pageIndex>2</platformCore:pageIndex>
        <platformCore:searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformCore:searchId>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>8000</listAcct:acctNumber>
            <listAcct:acctName>Checking</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>xxxxxxxxxxxxxxxxxxx</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Checking</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>8001</listAcct:acctNumber>
            <listAcct:acctName>Savings</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>yyyyyyyyy</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Savings</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          .........
          .........
          .........
          .........		  
        </platformCore:recordList>
      </platformCore:searchResult>
    </searchMoreWithIdResponse>
  </soapenv:Body>
</soapenv:Envelope>

   

Getting Started

Now let’s check step by step instructions on how to load NetSuite data in SSIS. In this exercise we will focus on calling search API but you can call virtually any API using the same technique.

For detailed help on NetSuite SOAP API and other useful information check this documentation. To browse schema you can use this link. From schema browser we will use Search Type name later on in this article. For example later on in this article to search accounts we will use AccountSearch as below example. This name can be found from here(click search tab) You can change it to something else. For now dont worry too much until we get to that point.

Configure SSIS HTTP Connection for NetSuite

Now let’s look at how to configure SSIS HTTP Connection for NetSuite API login. We will use Dynamic Token Approach.

1. 1. Open SSIS Package
   2. Drag Data flow from SSIS Toolbox and double click it
   3. Inside data flow SSIS toolbox, find ZS XML Source and drag it on the surface.
   4. Double click it to configure. First enter NetSuite API URL. If you need to use latest WebService version e.g. NetSuitePort_2019_1 then you must use company specific endpoint.
      

      https://webservices.netsuite.com/services/NetSuitePort_2017_1

      –OR–

      To use

      NetSuitePort_2019_1

        or higher use company specific endpoint like below otherwise you will get error. NOTE: You must change Body in coming section to match Version if you use below way.

      https://111222333444.suitetalk.api.netsuite.com/services/NetSuitePort_2019_1

   5. Check Use Credentials option and Click New HTTP connection.
   6. Configure general properties of connections as below.
      1. Select Credentials Type as Dynamic Token
      2. Use any URL (It will be ignored)
      3. Enter your NetSuite UserID, Password
      4. Enter Header as Cookie
      5. Enter AUth Scheme as JSESSIONID=
   7. First Screen will look like as below. Click Configure.
      

      Configure SSIS NetSuite Connection

   8. On Dynamic Token > Request Tab we have to set Token URL, Method, Body, Header.
      1. In the URL enter same URL again
         

         https://webservices.netsuite.com/services/NetSuitePort_2017_1

      2. In the Body enter as below. Change APPLICATION-ID and ACCOUNT-ID with correct ID we obtained earlier. [$userid$] and [$password$] is special placeholders will be replaced automatically at runtime.
         NOTE: Try to match your URL version in the Body Request. If you used For example if you used 2019_1 in URL then change below body to match that version

         <?xml version="1.0" encoding="utf-8"?>
         <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
           <soap:Header>
             <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
               <applicationId>xxxxxxYOUR-APPLICATION-IDxxxxxx</applicationId>
             </applicationInfo>
           </soap:Header>
           <soap:Body>
             <login xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
               <passport>
                 <email xmlns="urn:core_2017_1.platform.webservices.netsuite.com">[$userid$]</email>
                 <password xmlns="urn:core_2017_1.platform.webservices.netsuite.com">[$password$]</password>
                 <account xmlns="urn:core_2017_1.platform.webservices.netsuite.com">xxxxxxYOUR-ACCOUNT-IDxxxxxx</account>
                 <role internalId="3" xmlns="urn:core_2017_1.platform.webservices.netsuite.com" />
               </passport>
             </login>
           </soap:Body>
         </soap:Envelope>

         NOTE: Remove below line from above Body if you get INVALID_ROLE error later in this tutorial. If if you get INVALID_LOGIN_CREDENTIALS error then refer here.

         <role internalId="3" xmlns="urn:core_2017_1.platform.webservices.netsuite.com" />

      3. Select Content Type as text/xml
      4. In the Headers click on Raw Edit Button and enter like below.
         

         SOAPAction: "login"

   9. Once all settings done your Request Tab will look like below
      

      SSIS NetSuite Connection – Configure Token Request

   10. On Response Tab configure like below. You have to enter Cookie name as JSESSIONID
       

       SSIS NetSuite Connection – Configure Token Response

   11. Click OK to Save Connection.
   12. We will configure XML Source in next section.

Configure SSIS XML Source to read NetSuite data

Now let’s continue configuration of XML Source. In this section we will show you how to get data without pagination. In next section we will see how to modify XML Source Settings to pagination request to read many records (e.g. 1000+ rows).

1. On your XML Source enter NetSuite Webservice URL like below.
   

   https://webservices.netsuite.com/services/NetSuitePort_2017_1

2. Now check Use Credentials and select previously created HTTP connection.
3. Select Method as  POST and content type as text/xml
4. Edit Request Body and enter like below (or any other XML body for API you like to call. Make sure SOAPAction in next step updated correctly too for other action)
   NOTE: Try to match your URL version in the Body Request. If you used For example if you used 2019_1 in URL then change below body to match that version

   <soapenv:Envelope 
      xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
      xmlns:urn="urn:messages_2018_1.platform.webservices.netsuite.com" 
      xmlns:urn1="urn:core_2018_1.platform.webservices.netsuite.com">
      <soapenv:Body>
         <urn:getAll>
            <record recordType="currency"/> 
         </urn:getAll>
      </soapenv:Body>
   </soapenv:Envelope>

5. NOTE: If you used NetSuitePort_2019_1 or higher version in URL then make sure you changed version  2017_1 to 2019_1 in below body
6. Now you click Raw Edit above headers grid and enter method name in SOAPAction (e.g. getAll in our case)
   

   SOAPAction: "getAll"

   NOTE: SOAPAction Header is very important so make sure you change it to match Action you like to call. You can Use Correct WebService WSDL File (Metadata)  in Tool like SOAPUI as explained here

   Version 2017_1 WSDL Link
   Version 2018_1 WSDL Link
   Version 2019_1 WSDL Link
   For Other Versions Just Change URL Version Part to different Number e.g. 2019_2_0

7. Now enter Filter like below or navigate by clicking Select Filter button.
   

   $.soapenv:Envelope.soapenv:Body.getAllResponse.platformCore:getAllResult.platformCore:recordList.platformCore:record[*]

   

   Configure XML Source – NetSuite SOAP API getAll (Body, Header, ContentType, Method, Filter)

8. Now click preview data.
   

   Preview NetSuite data in XML Source

9. Click OK to save XML Source UI
10. Now you can connect XML Source to target like OLEDB Destination and load NetSuite data into SQL Server.

Configure SSIS XML Source with Pagination

So in previous section we saw how to configure XML Source for basic API calls (without pagination). In this section we will see how to call more complex API (e.g. search ) which requires pagination. Some ideas about NetSuite Pagination discussed here too.

Foe pagination setup using XML Source we have to configure following properties via Properties Grid or Advanced Edit rather than Normal UI. May of these properties not available via UI as of now.

1. Create new data flow and drag new ZS XML Source.
2. Right click on XML Source and click Properties in Data flow designer. Find property by name in the Grid and set correct value.
   NOTE: In newer version v3.1.2 or higher you can use XML Source UI to enter some multi line properties for ease of use (e.g. FirstPageBodyPart and NextPageBodyPart can be entered on Advanced Pagination Tab, all other properties are easier to enter from Grid Mode if you sort alphabetically)

<search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:AccountSearch">
        <q1:basic>
          <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
            <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">x</searchValue>
          </description>
          <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
            <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">a</searchValue>
          </name>
        </q1:basic>
      </searchRecord>
    </search>

<searchMoreWithId xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <searchId>[$pagetoken$]</searchId>
      <pageIndex>[$pagenumber$]</pageIndex>
    </searchMoreWithId>

Once above items are set in property grid. Double click XML source to edit few more items from UI.

1. Enter URL as below
   

   https://webservices.netsuite.com/services/NetSuitePort_2017_1

2. Check Use Credentials and select HTTP connection we created in previous section.
3. Click Raw Edit button (above headers grid) and enter below
   

   SOAPAction: "search[$tag$]"

4. Click on Edit button next to Body. Enter following XML fragment for search request. Change ApplicationId and adjust PageSize if needed.
   

   <?xml version="1.0" encoding="utf-8"?>
   <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
     <soap:Header>
       <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <applicationId>ABCD-1234-XXXXXXXXXXXXXXX</applicationId>
       </applicationInfo>
       <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <pageSize>100</pageSize>
       </searchPreferences>
     </soap:Header>
     <soap:Body>
   [$tag$]
     </soap:Body>
   </soap:Envelope>

5. Select POST method, change content type to XML (text / xml, charset=UTF-8)
6. That’s it. Click ‘Preview to confirm.
7. Click ok to save XML Source.
   

   Read NetSuite data in SSIS with Pagination ( search, searchMoreWithId API call)

Configure SQL Server Destination to load NetSuite data

Once you have XML Source configured you can connect XML Source to Target (OLEDB Destination) and load data into SQL Server.

Conclusion

In this post we explored many advanced options of HTTP Connection and XML Source to enable NetSuite API integration in SSIS. Download SSIS PowerPack today for FREE to uncover many other data integration scenarios not discussed in this article.

The post How to read data from NetSuite in SSIS (SimpleTalk SOAP API) appeared first on ZappySys Blog.

In our previous blog post we saw how to read JSON REST API / XML SOAP in Informatica using ZappySys ODBC Drivers. Now lets look at API Write scenario. We will learn how to to POST data to API in Informatica (Insert / Update). We will use SQL Transformation to fire ODBC Driver Queries to create / update or delete records in Target System by calling Web API (i.e. REST or SOAP).

For example purpose we will use JSON API but if you are calling XML / SOAP API then use ZappySys XML Driver instead of JSON Driver.

Requirements

This article assumes that you have full filled following basic requirements.

1. Download Install ZappySys ODBC PowerPack (Drivers for JSON and REST API)
2. Install Informatica PowerCenter Client Tools (e.g. Workflow and Mapping Designers)
3. Access to a Relational database such as SQL Server (or use any of your choice e.g. Oracle, MySQL, DB2 ). If nothing available then you can use flat file target.

Video Tutorial – POST data to API in Informatica

Here is the full Video Tutorial with many API tips / tricks to show how to POST data to REST / SOAP API in Informatica PowerCenter. It also covers how to Upload File to Web API.

Getting Started

So let’s get started with basic setup on how to POST data to your API to create new records or update / delete records. In the following demo we are trying to create new Customer records by calling POST API.

For that we will use following setup.

Create ODBC Connection for API

Check this section from previous article to learn how to configure API Connection using ZappySys ODBC Driver for JSON / XML / CSV.

Create Relational Connection in Workflow Designer

Check this section from previous article to learn how to create and configure Relational Connection of ODBC Type which points to DSN (ZappySys JSON Driver).

Configure JSON connection in Informatica for REST API / JSON File connectivity – Using ZappySys JSON ODBC Driver

Prepare POST Query and generate Metadata File

Before you can call POST API in Informatica Mapping, you need to generate Metadata file using ODBC Data Source UI. Metadata file creation is important to avoid double calls (One for Metadata and One for Getting Data). When you dont Supply Metadata file to ZappySys Driver then it will try to call API twice (First time read 300 rows to guess datatypes / response columns and then call API again to read all remaining rows from response).

Its OK to allow two reads for the same request if its GET request (Read scenario) but its not OK if you are creating a New records because same request is sent twice and you may end up creating duplicate records.

Obtaining metadata is simple using Query Preview UI.

1. Go to ODBC Data Source and setup properties on Driver UI (e.g. URL, Connection, Filter, Headers etc)
2. Once everything is configured go to Preview Tab and click Preview Button to see data.
3. Once Data is returned as per your need you can click on Save Metadata. Copy SQL Query for later use.
   

   Generate Metadata File for REST API POST Operation using ZappySys API Driver (JSON / XML)

Create Informatica Mapping to POST data to REST API / SOAP Web Service

Once Metadata File is we can proceed to Informatica Mapping designer. For making instructions simple we will skip few details (e.g. Importing Source  /Target Definitions). Refer to Previous article to read instructions in depth.

Created Now lets look at how to read data from relational source (E.g. SQL Server, Oracle) and send data to REST API (Call POST API).

1. Drag Source into Mapping Designer
2. Drag Expression Transform  into Mapping Designer and drop desired columns on Expression Transform as Pass through Columns.
3. Create one additional Output Port for API URL as below (You may not need this if API URL is not Dynamic. Skipping URL or any other setting in ODBC SQL Query will use DSN Settings you configured in previous section)
4. Drag SQL Transform and select following Option (Mode = Query, Connection Type =ODBC)
   

   Create new SQL Transform for API Call (POST Data to API using Informatica and ZappySys ODBC Driver)

5. Now connect input columns from Expression Transform to SQL Transformation. These input columns will be used as parameters in API POST Body and SQL Query WITH Clause to craft JSON / XML API request.
6. Now double click on SQL Transform and go to SQL Ports Tab
7. Click somewhere in the Output Columns Group and Add desired columns (Tip: Add same number of columns you define in the Metadata File in Previous section. For Demo purpose you can remove some columns from Metadata file so you have only hadful columns to deal with. When you remove columns from Metadata File make sure JSON is Valid. Many times user forget to remove last comma)
8. Once you define Input and Output columns for your SQL Transform you can Edit SQL Query button and configure your SQL Query as below. In below example query we have supplied URL,  Body, HTTP Headers. Also notice that Body JSON is crafted by using Input Columns from Upstream. We used only CustomerID as an example but in real words you may add many more in your JSON Body.
   
   For Old Version of PowerCenter- V8.5 or lower:  It’s important to Pass URL via Substitution else you may get error (http://abc.com is output as http:/abc.com – only one slash)Here is a sample Query for SQL Transform (Added String Substitutions) 
   Few things to notice:
   – Output Ports defined in SQL Transform must match order and column names of SQL query. Best thing is to avoid SELECT * FROM and use explicit column names like below. If you already took care column orders in Metadata file then you can use * to shorten your SQL statement.
   – Slash (i.e. \ ) is treated as escape character so replace slash with double slash anywhere in SQL text(e.g. c:\temp\meta.txt should be replaced with c:\\temp\\meta.txt).
   – If you use ? in URL (e.g. http://myapi/?id=111) then replace ? with \?  because ? has special meaning in SQL Transform SQL Text.  ? is used in Parameter Binding (e.g. ?my_input_port?) .
   – You may use Format Specifiers

   <<~P_URL~,FUN_JSONENC>>

     if you want to encode special characters in your data (e.g. Tabs, Double quotes, New lines). Check this link
   – Avoid using string specifier before back slash (e.g. c:\\data\\~FileName~ ). because \~ cause issue with escape sequence triggered by \ . To solve this issue Use Expression Transform before SQL Transform to build Dynamic Path or other type of string.
   

   SELECT url,origin FROM $
   WITH
   (
     META='c:\\temp\\meta.txt'
    ,METHOD='POST'
    ,HEADER='Content-Type:text/plain||x-hdr1:AAA'
    ,SRC='~P_URL~'
    ,BODY='
    {
       "ID": "~P_CustomerID~",
       "Link": "<<~P_URL~,FUN_JSONENC>>",
       "Notes": "Line1\\r\\nLine2"
    }'
   )

   

   Configure Informatica SQL Transform to call ODBC SQL Queries – JSON /XML API POST Data (Create / Update Records)

9. In the above Screenshot Notice Following items in SQL Query.
   1. We have specified META property (Use the same Metadata File we generated   in the previous section. Meta Property can take direct file path, meta name (settings stored in DSN) or supply direct JSON for Metadata. Notice we escaped \ with \\
      

      META='[ {"Name": "MyCol1", "Type": "String", "Length": 100}, {"Name": "MyCol2", "Type": "String", "Length": 100} ...... ]'
      -- OR --
      META='c:\\meta\\my_post_query_metadata.txt'
      -- OR --
      META='MyMetaName'

   2. Second thing you will Notice that in the Body we have used String Substitution in SRC and BODY.  If your URL is not Dynamic and you like to use setting defined in DSN then omit SRC attribute from WITH clause. Substitution must start and end with tild (~). For example Input column P_URL can be used as ~P_URL~ and at runtime it will be replaced. Make sure to escape  single quote with double quote (We replaced ‘ with ” in Sam”s) in Input Parameter Values else SQL Query will fail. You can use Expression Transform to escape data if needed using REPLACE function.  We also replaced \r\n with \\r\\n
      

      SRC='~P_URL~'

      and
      

      BODY='
       {
          "ID": "~P_CustomerID~",
          "Notes": "Line1\\r\\nLine2, Sam''s Club"
       }'

10. Now connect your SQL Transformation output Ports to Target (e.g. Flat File Target or Relational Target)
11. Create new Workflow and Session using this Mapping
12. Go to Session properties and on the Mapping Tab make sure to change COnnection Type for SQL Transformation to Relational Connection.
    

    Configure Session Mapping Properties – Set ODBC COnnection Type for SQL Transform to call API requests

13. In our example we changed Target to File Writer but you can save SQL Transform output to some relational table too.
14. Now you can execute Workflow and check your session log for any error. You can also debug web requests in Fiddler (Must be installed on Server where actual Job is running).
    

    Monitor Session Execution in Informatica – Debug Web API requests using Fiddler

Upload File to API (Read Body from Local File)

Now let’s look at scenario to POST Body content from File (i.e. Upload File to API). Below simple query will Upload file content to API server (POST request where Body is coming from file)

SELECT url,origin FROM $
WITH
(
  META='c:\\temp\\meta.txt'
 ,METHOD='POST'
 ,HEADER='Content-Type:text/plain||x-hdr1:AAA'
 ,SRC='~P_URL~'
 ,BODY='@~P_FilePath~'
 ,IsMultiPart='True'
)

Notice how we have used IsMultiPart = ‘True’  and used @ before FilePath in Body. Our File Path is coming from String Substitution (i.e. Input Port  ~P_FilePath~)

Debugging REST / SOAP API using Fiddler

Check this article on how to debug REST / SOAP API calls using fiddler. You can also enable Log to Debug Mode on Advanced Tab of ODBC DSN UI.

Conclusion

In this post we saw how to call API POST requests in Informatica and send data from relational system to Web API (Create / Update / Delete Records using JSON / XML API calls). You can explore many other API related scenarios using ZappySys ODBC Drives for JSON / XML / CSV. Click here to try ZappySys ODBC Drivers for FREE and check out how easy it is to read / write JSON / XML API in Informatica or any other ODBC Compliant Apps (e.g Power BI, SSIS, Qlik).

The post POST data to API in Informatica using SQL Transformation appeared first on ZappySys Blog.

In our previous blog we discussed how to import REST / SOAP API data in Power BI. Now we will use that knowledge to read NetSuite data in Power BI (NetSuite CRM – SuiteTalk). We will use ZappySys XML Driver to call Search function in NetSuite SOAP API. If you are not familiar with SOAP API concept then we highly recommend you read previous blog post. In this post we will use Power BI as an example App to consume NetSuite data but you can use steps listed in this article to load data in any other ODBC compliant ETL / reporting app (e.g. Excel, Qlik, Informatica, SSRS) or programming language (e.g. C#, JAVA, Python) or SQL Server (Linked Server).

We will look at each step in depth, how to enable SOAP Api access in NetSuite, How to register Application, how to configure NetSuite ODBC DSN, and finally how to load NetSuite data into Power BI dashboard.

So let’s get started. The approach listed in this article can be used to call other SOAP based API such as Salesforce, Workday, SAP, Sage etc.

Prerequisite

Before you get started make sure following requirements are met.

1. NetSuite CRM Login (UserID and Password)
2. Enable Web Service Feature in NetSuite
3. Obtain NetSuite Account ID
4. Obtain Application ID if you have created an App. Else Register new Netsuite Application and get Application ID
5. Download ZappySys ODBC PowerPack (for SOAP / XML Driver)
6. Download Power BI Desktop (FREE) if you have not already done it.

What is NetSuite CRM?

NetSuite is a CRM / ERP product. It gives you scalable cloud CRM / ERP solution targeted at high-growing, mid-sized businesses and large enterprises. It automates front- and back-office processes including: financial management, revenue management, fixed assets, order management, billing, and inventory management.

Preparing for NetSuite WebService Call in Power BI

For NetSuite API call we have to make sure certain settings and obtain Application ID and AccountID. So let’s get started.

How to enable NetSuite Web Service Feature

Before you call any NetSuite SOAP API (i.e. SuiteTask Web Service), First thing to get started with NetSuite SOAP API (i.e. ) is to enable Web Service Feature. Once you enable Web Service, you can programmatically access NetSuite data in your App (e.g. Power BI, Tableau, Informatica). Click this link to see steps to enable Web Service Feature

Enable NetSuite Web Service Feature (SuiteTalk API)

Obtain NetSuite Account ID

NetSuite Account ID is needed by Login API call. So perform following steps if you dont know account id.

1. Login to your NetSuite Portal
2. Click the Setup Tab
3. In the drop-down click Integration >  Web services preferences
4. On that page you will find your Account ID (example account ID = TSTDRV1234567 )

Obtain NetSuite Account ID (Needed for Web Service Login)

Obtain NetSuite Application ID

Once you have account ID now we need to get Application ID. For that follow these steps.

1. Login to your NetSuite Portal
2. Click the Setup Tab
3. In the drop-down click Integration >  Manage Integrations
4. On that page you will find your existing Applications along with Account ID / Status (example Application ID = AE2EACFD-1234-1222-1111-12345544555 )
5. If you don’t have any application created yet then just click on New.

Create new App / Get NetSuite Application ID

Understanding how NetSuite SOAP API Call works

In this section we will briefly understand how NetSuite SOAP API call works. When you use ZappySys Driver / Component it will take care many details for you but still you have to configure correct XML Body. There are two steps involved before you can call any SOAP API in NetSuite.

1. Call Login method and extract JSESSIONID from response Cookie (This token expires after some time)
2. Call any other API (Pass obtained JSESSIONID token in Cookie header)

NetSuite Login Call

Request

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml;charset=UTF-8
SOAPAction: "login"
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
  </soap:Header>
  <soap:Body>
    <login xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <passport>
        <email xmlns="urn:core_2017_1.platform.webservices.netsuite.com">xxxxxxxxxxxxx</email>
        <password xmlns="urn:core_2017_1.platform.webservices.netsuite.com">xxxxxxxxxxxx</password>
        <account xmlns="urn:core_2017_1.platform.webservices.netsuite.com">TSTxxxxxxxxxxxxxxx</account>
        <role internalId="3" xmlns="urn:core_2017_1.platform.webservices.netsuite.com" />
      </passport>
    </login>
  </soap:Body>
</soap:Envelope>

Response Notice JSESSIONID=6xI6xxxxxxxxx in the login call response below. This Session ID can be used to call next call. ZappySys HTTP Connection Manager configured with Dynamic Token option can extract Cookie and reuse it until its expired (1 hour by default).

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 16:50:41 GMT
Content-Length: 1676
Content-Type: text/xml; charset=utf-8
X-N-OperationId: 777274f6-49c6-4883-bcd1-1d70df90d78e
Strict-Transport-Security: max-age=31536000
NS_RTIMER_COMPOSITE: 13633xxxxxxxxx:70617xxxxxxxxxxxxx36F6D:80
Set-Cookie: JSESSIONID=6xI61rdFt26pxxxxxxxxxxxxxxxqvJIMlu!-17981123445; path=/; secure; HttpOnly
Set-Cookie: NS_VER=2018.1.0; domain=webservices.netsuite.com; path=/
Set-Cookie: NS_ROUTING_VERSION=LAGGING; path=/
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent
Keep-Alive: timeout=10, max=990
Connection: Keep-Alive

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_xxxxxxxxxxxxxxxxx_9fc719</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <loginResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <sessionResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
        <ns1:status isSuccess="true" xmlns:ns1="urn:core_2017_1.platform.webservices.netsuite.com"/>
        <ns2:wsRoleList xmlns:ns2="urn:core_2017_1.platform.webservices.netsuite.com">
          <ns2:wsRole>
            <ns2:role internalId="3">
              <ns2:name>Administrator</ns2:name>
            </ns2:role>
            <ns2:isDefault>false</ns2:isDefault>
            <ns2:isInactive>false</ns2:isInactive>
          </ns2:wsRole>
          <ns2:wsRole>
            <ns2:role internalId="15">
              <ns2:name>Employee Center</ns2:name>
            </ns2:role>
            <ns2:isDefault>false</ns2:isDefault>
            <ns2:isInactive>false</ns2:isInactive>
          </ns2:wsRole>
      </sessionResponse>
    </loginResponse>
  </soapenv:Body>
</soapenv:Envelope>

 

NetSuite API Call ( getAll method )

Now once you have JSESSIONID extracted you can pass it to call next API call as below. For example here is how to call getAll method to obtain currency list and exchange rates. If you are not sure how to craft correct Request XML then use tool like SoapUI. You can use NetSuite WSDL file from here and import it in SoapUI.   Request See how Cookie is passed for authentication purpose. Again you can pass this manually or use HTTP connection with Dynamic Token to pass it automatically along with each API call.

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml;charset=UTF-8
SOAPAction: "getAll"
Cookie: JSESSIONID=6xI61rxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxMlu!-1798141234
Accept-Encoding: gzip, deflate

<soapenv:Envelope 
   xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
   xmlns:urn="urn:messages_2018_1.platform.webservices.netsuite.com" 
   xmlns:urn1="urn:core_2018_1.platform.webservices.netsuite.com">
   <soapenv:Body>
      <urn:getAll>
         <record recordType="currency"/> 
      </urn:getAll>
   </soapenv:Body>
</soapenv:Envelope>

Response

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 16:50:42 GMT
Content-Length: 9648
Content-Type: text/xml; charset=utf-8
X-N-OperationId: 1b2989e4-b930-4a72-b471-00a8e131dfd4
NS_RTIMER_COMPOSITE: 21403xxxxxx:706172746E65xxxxxxxxxxxx6F6D:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_TSTDRV1439151_100320189871507121000128715_6ae8e8</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <getAllResponse xmlns="urn:messages_2018_1.platform.webservices.netsuite.com">
      <platformCore:getAllResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>11</platformCore:totalRecords>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>USD</listAcct:name>
            <listAcct:symbol>USD</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>$</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_unitedStatesEnglish</listAcct:locale>
            <listAcct:formatSample>$1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>1.0</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>GBP</listAcct:name>
            <listAcct:symbol>GBP</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>£</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_unitedKingdomEnglish</listAcct:locale>
            <listAcct:formatSample>£1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>1.33165497</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
          <platformCore:record internalId="3" xsi:type="listAcct:Currency" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:name>CAD</listAcct:name>
            <listAcct:symbol>CAD</listAcct:symbol>
            <listAcct:isBaseCurrency>true</listAcct:isBaseCurrency>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:overrideCurrencyFormat>false</listAcct:overrideCurrencyFormat>
            <listAcct:displaySymbol>$</listAcct:displaySymbol>
            <listAcct:symbolPlacement>_beforeNumber</listAcct:symbolPlacement>
            <listAcct:locale>_canadaEnglish</listAcct:locale>
            <listAcct:formatSample>$1,234.56</listAcct:formatSample>
            <listAcct:exchangeRate>0.76457273</listAcct:exchangeRate>
            <listAcct:fxRateUpdateTimezone>_americaNewYork</listAcct:fxRateUpdateTimezone>
            <listAcct:currencyPrecision>_two</listAcct:currencyPrecision>
          </platformCore:record>
        </platformCore:recordList>
      </platformCore:getAllResult>
    </getAllResponse>
  </soapenv:Body>
</soapenv:Envelope>

NetSuite Search API (Pagination Example)

Now let's look at more complex API calls where Pagination is needed. For example when you call search API to pull data from specific table (e.g. Account) you need to loop through multiple responses until all records are consumed. Request (Page-1)

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml; charset=utf-8
SOAPAction: "search"
Cookie: JSESSIONID=BKQ7xxxxxxxxxxxx4Ug!-995xxxxxx
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
    <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <pageSize>50</pageSize>
    </searchPreferences>
  </soap:Header>
  <soap:Body>
    <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:AccountSearch">
        <q1:basic>
          <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
            <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">x</searchValue>
          </description>
          <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
            <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">a</searchValue>
          </name>
        </q1:basic>
      </searchRecord>
    </search>
  </soap:Body>
</soap:Envelope>

  Response (Page-1) In below response notice 2 things. SearchId is like a cursor id which you have to pass for each page you request after first page. And you can stop looping after 4th page based on totalPages count. platformCore:searchId = WEBSERVICES_ABCD_12345-xxxxxxxxx platformCore:totalPages = 4

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 18:40:33 GMT
Content-Length: 49266
Content-Type: text/xml; charset=utf-8
X-N-OperationId: c1e79336-63b7-4eb7-989e-5c3d24c56891
NS_RTIMER_COMPOSITE: 118xxxxxxxx:70617xxxxxxxxxxxxxxxxxxxxxD:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="utf-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <searchResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformCore:searchResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>193</platformCore:totalRecords>
        <platformCore:pageSize>50</platformCore:pageSize>
        <platformCore:totalPages>4</platformCore:totalPages>
        <platformCore:pageIndex>1</platformCore:pageIndex>
        <platformCore:searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformCore:searchId>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>1111</listAcct:acctNumber>
            <listAcct:acctName>Checking</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>xxxxxxxxxxxxxxxxxxx</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Checking</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>22222</listAcct:acctNumber>
            <listAcct:acctName>Savings</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>yyyyyyyyy</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Savings</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          .........
          .........
          .........
          .........		  
        </platformCore:recordList>
      </platformCore:searchResult>
    </searchResponse>
  </soapenv:Body>
</soapenv:Envelope>

  Request (Page-2 and onwards) Here is the second page request. Notice few things here. In heraders we are now sending SOAPAction: "searchMoreWithId" rather than "search".  Also our SOAP Body is also different compared to first page request. Now we only sending pageIndex we want to fetch and searchId. We have also change <search> tag with <searchMoreWithId> under Body.

POST https://webservices.netsuite.com/services/NetSuitePort_2017_1
Content-Type: text/xml; charset=utf-8
SOAPAction: "searchMoreWithId"
Cookie: JSESSIONID=BKQ7xxxxxxxxxxxx4Ug!-995xxxxxx
Accept-Encoding: gzip, deflate

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Header>
    <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <applicationId>xxxxxxxxxxxxxxxxxx</applicationId>
    </applicationInfo>
    <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <pageSize>50</pageSize>
    </searchPreferences>
  </soap:Header>
  <soap:Body>
    <searchMoreWithId xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</searchId>
      <pageIndex>2</pageIndex>
    </searchMoreWithId>
  </soap:Body>
</soap:Envelope>

Response (Page-2 and onwards)

HTTP/1.1 200 OK
Date: Wed, 03 Oct 2018 18:40:33 GMT
Content-Length: 49266
Content-Type: text/xml; charset=utf-8
X-N-OperationId: c1e79336-63b7-4eb7-989e-5c3d24c56891
NS_RTIMER_COMPOSITE: 118xxxxxxxx:70617xxxxxxxxxxxxxxxxxxxxxD:80
Strict-Transport-Security: max-age=31536000
P3P: CP="CAO PSAa OUR BUS PUR"
Vary: User-Agent

<?xml version="1.0" encoding="utf-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header>
    <platformMsgs:documentInfo xmlns:platformMsgs="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformMsgs:nsId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformMsgs:nsId>
    </platformMsgs:documentInfo>
  </soapenv:Header>
  <soapenv:Body>
    <searchMoreWithIdResponse xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
      <platformCore:searchResult xmlns:platformCore="urn:core_2017_1.platform.webservices.netsuite.com">
        <platformCore:status isSuccess="true" />
        <platformCore:totalRecords>193</platformCore:totalRecords>
        <platformCore:pageSize>50</platformCore:pageSize>
        <platformCore:totalPages>4</platformCore:totalPages>
        <platformCore:pageIndex>2</platformCore:pageIndex>
        <platformCore:searchId>WEBSERVICES_ABCD_12345-xxxxxxxxx</platformCore:searchId>
        <platformCore:recordList>
          <platformCore:record internalId="1" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>8000</listAcct:acctNumber>
            <listAcct:acctName>Checking</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>xxxxxxxxxxxxxxxxxxx</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Checking</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          <platformCore:record internalId="2" xsi:type="listAcct:Account" xmlns:listAcct="urn:accounting_2017_1.lists.webservices.netsuite.com">
            <listAcct:acctType>_bank</listAcct:acctType>
            <listAcct:acctNumber>8001</listAcct:acctNumber>
            <listAcct:acctName>Savings</listAcct:acctName>
            <listAcct:includeChildren>false</listAcct:includeChildren>
            <listAcct:currency internalId="1">
              <platformCore:name>USD</platformCore:name>
            </listAcct:currency>
            <listAcct:generalRate>_current</listAcct:generalRate>
            <listAcct:parent internalId="246">
              <platformCore:name>yyyyyyyyy</platformCore:name>
            </listAcct:parent>
            <listAcct:cashFlowRate>_average</listAcct:cashFlowRate>
            <listAcct:description>Savings</listAcct:description>
            <listAcct:isInactive>false</listAcct:isInactive>
            <listAcct:inventory>false</listAcct:inventory>
            <listAcct:eliminate>false</listAcct:eliminate>
            <listAcct:revalue>true</listAcct:revalue>
          </platformCore:record>
          .........
          .........
          .........
          .........		  
        </platformCore:recordList>
      </platformCore:searchResult>
    </searchMoreWithIdResponse>
  </soapenv:Body>
</soapenv:Envelope>

   

Getting Started

Now let’s check step by step instructions on how to load NetSuite data in Power BI. In this exercise we will focus on calling search API but you can call virtually any API using the same technique.

For detailed help on NetSuite SOAP API and other useful information check this documentation. To browse schema you can use this link. From schema browser we will use Search Type name later on in this article. For example later on in this article to search accounts we will use AccountSearch as below example. This name can be found from here(click search tab) You can change it to something else. For now dont worry too much until we get to that point.

Creating NetSuite ODBC Connection (Using ZappySys XML / SOAP Driver)

Once you have all necessary information handy we can create ODBC DSN for NetSuite which can be used by any ODBC compliant app such as Power BI. Perform the following steps to set NetSuite Connection which we will use to query data (e.g. Accounts, Budget, Campaign etc).

1. Open start menu and search for “ODBC 64”. Launch ODBC Data sources for 64 bit
2. Click Add and select ZappySys XML Driver
   

   ZappySys ODBC Driver for XML / SOAP API

3. Once driver UI is visible change default DSN name (e.g. change to ZS – NetSuite)
4. Click on the Load Connection String button and enter the following string after replacing below placeholders to configure the NetSuite connection. If you are unsure how to get Account ID and Application ID then check previous sections. For now dont worry about various options in the connection string. We will cover them later.
   Replace $$$YOUR-APPLICATION-ID$$$  (2 times) with your own Application ID
   Replace $$$YOUR-ACCOUNT-ID$$$ with your own Account ID
   Replace $$$YOUR-USER-ID$$$  with your own Login ID (e.g myuser@mycompany.com)
   Replace $$$YOUR-PASSWORD$$$  with your own Password
   Replace <pageSize>50</pageSize>  with your own value. We suggest not to go more than 200 rows per page for better performance.
   Replace c:\temp\netsuite.cache.db with your own path for Cache File. Caching must be ON in order to Import data in Power BI correct way. We also use CacheTtl=120 seconds (Cache Expires in 120 seconds) but adjust it longer if your data import takes long time.
   

   DRIVER={ZappySys XML Driver};ElementsToTreatAsArray='platformCore:record';DataPath='https://webservices.netsuite.com/services/NetSuitePort_2017_1';DataConnectionType=HTTP;AuthScheme='JSESSIONID=';TokenUrl='https://webservices.netsuite.com/services/NetSuitePort_2017_1';TokenRequestData='<?xml version="1.0" encoding="utf-8"?>
   <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
     <soap:Header>
       <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <applicationId>$$$YOUR-APPLICATION-ID$$$</applicationId>
       </applicationInfo>
     </soap:Header>
     <soap:Body>
       <login xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <passport>
           <email xmlns="urn:core_2017_1.platform.webservices.netsuite.com">[$userid$]</email>
           <password xmlns="urn:core_2017_1.platform.webservices.netsuite.com">[$password$]</password>
           <account xmlns="urn:core_2017_1.platform.webservices.netsuite.com">$$$YOUR-ACCOUNT-ID$$$</account>
           <role internalId="3" xmlns="urn:core_2017_1.platform.webservices.netsuite.com" />
         </passport>
       </login>
     </soap:Body>
   </soap:Envelope>';TokenRequestMethod='POST';TokenRequestContentType=TextXmlUtf8;TokenResponseValueExtractMode=Cookie;TokenRequestHeaders='SOAPAction:"login"';TokenResponseCookie='JSESSIONID';TokenAuthHeader='Cookie';UserName='$$$YOUR-USER-ID$$$';CredentialType=TokenDynamic;Password='$$$YOUR-PASSWORD$$$';Filter='$.soapenv:Envelope.soapenv:Body.search[$tag$]Response.platformCore:searchResult.platformCore:recordList.platformCore:record[*]';RequestData='<?xml version="1.0" encoding="utf-8"?>
   <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
     <soap:Header>
       <applicationInfo xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <applicationId>$$$YOUR-APPLICATION-ID$$$</applicationId>
       </applicationInfo>
       <searchPreferences xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <pageSize>50</pageSize>
       </searchPreferences>
     </soap:Header>
     <soap:Body>
   [$tag$]
     </soap:Body>
   </soap:Envelope>';RequestMethod='POST';NextUrlAttributeOrExpr='$.soapenv:Envelope.soapenv:Body.search[$tag$]Response.platformCore:searchResult.platformCore:searchId';FirstPageBodyPart='    <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:AccountSearch">
           <q1:basic>
   		<!--
             <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">x</searchValue>
             </description>
             <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">a</searchValue>
             </name>
   		 --> 
           </q1:basic>
         </searchRecord>
       </search>';NextPageBodyPart='    <searchMoreWithId xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchId>[$pagetoken$]</searchId>
         <pageIndex>[$pagenumber$]</pageIndex>
       </searchMoreWithId>';PagingMaxPagesExpr='$.soapenv:Envelope.soapenv:Body.searchResponse.platformCore:searchResult.platformCore:totalPages';HasDifferentNextPageInfo=True;EnablePageTokenForBody=True;PagePlaceholders='header=|MoreWithId;filter=|MoreWithId';RequestHeaders='Content-Type: text/xml; charset=utf-8 || SOAPAction: "search[$tag$]"';CachingMode=All;CacheStorage=File;CacheEntryTtl=120;CacheFileLocation='c:\temp\netsuite.cache.db'

    
   

   ZappySys ODBC Driver Load Connection String – NetSuite Connection Example

5. After configuration your screen may look like below. You can also click on Advanced View (Radio button) to see some Advanced properties (e.g. Caching Properties such as CachingMode, CacheFileLocation and Advanced Pagination Properties such as PagingMaxPagesExpr, HasDifferentNextPageInfo, NextPageBodyPart, FirstPageBodyPart, EnablePageTokenForBody, PagePlaceholders )
   
   Notice [$tag$] in Body, Header and Filter. This gets replaced at runtime based on some settings we did for pagination (see next 2 screenshots). NetSuite API has different Header, Body and Filter need for First page vs Next Pages so it needs special configuration. [$tag$] in Body gets replacement based on FirstPageBodyPart and NextPageBodyPart properties found on Advanced View under Pagination. [$tag$] used in Filter, Header or any other place gets its replacements from PagePlaceHolders Property found on Advanced View (see filter=firstPartString|nextPartString;header=firstPartString|nextPartString). Use pipe (vertical bar to separate first part placeholder and next part placeholder) In our case we use blank for first part.
   

   NetSuite ODBC Connection Settings (ZappySys XML Driver)

6. Here is what it looks like on Advanced Tab.
   

   ODBC XML Driver – Ádvanced View (Pagination, Caching, Logging and other properties not found in Simple Mode)

7. Now go to preview tab and click Preview Data button. You will see upto 100 sample records. Using below query it will use all options from UI (No override in query).SELECT * from $
   

   Preview NetSuite data (Query Accounts) using ODBC Driver

8. Now let’s change our default query a bit  to get list of Expense Category (i.e. ExpenseCategorySearch). As you see we change FirstRequestBody a little bit (change q1:ExpenseCategorySearch). We also removed some unwanted columns from preview by setting IncludeParentColumns=’false’
   

   SELECT * FROM $
   WITH(
   	FirstPageBodyPart='    <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:ExpenseCategorySearch">
           <q1:basic>
             <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">Some-String-Goes-Here</searchValue>
             </description>
             <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">Some-String-Goes-Here</searchValue>
             </name>
           </q1:basic>
         </searchRecord>
       </search>',
       IncludeParentColumns='False'
   )

   We have commented SearchQuery portion to make it simple but you can learn more about how to write search query from here

   Here is one more example of query which combines Basic Search and Advanced Search

   Pay close attention to namespace prefix in SOAP code you copy from Help center. It may be using different prefix than our original connection string (e.g. we use q1:xxxxx at some places but online sample may use ns1, ns2, ns3 ….)

   SELECT * FROM $
   WITH(
   	FirstPageBodyPart='
       <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchRecord xsi:type="ns4:ContactSearch" xmlns:ns4="urn:relationships_2017_1.lists.webservices.netsuite.com">
           <ns4:basic xsi:type="ns5:ContactSearchBasic" xmlns:ns5="urn:common_2017_1.platform.webservices.netsuite.com">
             <ns5:email operator="contains" xsi:type="ns6:SearchStringField" xmlns:ns6="urn:core_2017_1.platform.webservices.netsuite.com">
               <ns6:searchValue xsi:type="xsd:string">.com</ns6:searchValue>
             </ns5:email>
           </ns4:basic>
           <ns4:customerJoin xsi:type="ns7:CustomerSearchBasic" xmlns:ns7="urn:common_2017_1.platform.webservices.netsuite.com">
             <ns7:entityId operator="contains" xsi:type="ns8:SearchStringField" xmlns:ns8="urn:core_2017_1.platform.webservices.netsuite.com">
               <ns8:searchValue xsi:type="xsd:string">C</ns8:searchValue>
             </ns7:entityId>
           </ns4:customerJoin>
         </searchRecord>
       </search>',
       IncludeParentColumns='False'
   )

    
   

   NetSuite Search API Example (Mixing BasicSearch / AdvancedSearch with JOIN)

9. Copy your Final Query you like to use to import NetSuite data in Power BI
10. Click OK to save DSN

Import NetSuite data in Power BI

Now lets perform the final step to import our NetSuite Query to Power BI. Assume that we want to import Contacts in Power BI to report them by demographics or some other criteria.

1. Open Power BI Desktop
2. Click on Get Data > More Data… > Other > ODBC
   

   Import SOAP API in Power BI (Using ZappySys XML ODBC Driver)

3. Select your ODBC DSN Source from the dropdown (Assuming you have created a ODBC Data Source using ZappySys XML Driver – See the previous Section)
4. Expand Advanced Options to enter custom Query.
5. Enter your SOAP API call SQL query here. For example purpose, we will use below query to get all account using NetSuite Search Web service.
   

   SELECT * FROM $
   WITH(
   	FirstPageBodyPart='    <search xmlns="urn:messages_2017_1.platform.webservices.netsuite.com">
         <searchRecord xmlns:q1="urn:accounting_2017_1.lists.webservices.netsuite.com" xsi:type="q1:AccountSearch">
           <q1:basic>
           <!--
             <description operator="doesNotContain" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">x</searchValue>
             </description>
             <name operator="contains" xmlns="urn:common_2017_1.platform.webservices.netsuite.com">
               <searchValue xmlns="urn:core_2017_1.platform.webservices.netsuite.com">a</searchValue>
             </name>
             -->
           </q1:basic>
         </searchRecord>
       </search>',
       IncludeParentColumns='False'
   )

    
   

   Import NetSuite SOAP Web Service Data in Power BI (ZappySys XML Driver)

6. Once you see data preview click OK to import.
7. After data import is done you can edit your dataset (e.g. remove unwanted columns)
8. You can also edit Source query after data is imported (See below)
   

   Edit Power BI Data Source SQL after REST / SOAP Web Service Import

9. That’s it. You can now create your dashboards using NetSuite data. See below example.
   

   Import NetSuite data in Power BI dashboard

How to debug Web Requests using Fiddler

There will be a time you like to see RAW HTTP Requests generated by driver for NetSuite SOAP API.  If you like to do that check this article.

How to call other NetSuite SOAP API

So far we have seen only search API to query data. But there are many other SOAP APIs which can be consumed using approach listed in this article. Check this article to learn more how to call any SOAP API using ZappySys XML Driver. It shows techniques to use tool called SoapUI. You can use NetSuite WSDL file from here copy link listed on that page and use it to import in SoapUI.

Conclusion

Consuming data from NetSuite API or any XML / SOAP based API in your Reporting / ETL apps can be tricky. But we have seen how to solve this challenging task using ZappySys XML Driver in few clicks. Download ODBC PowerPack and explore many other API integration scenarios not discussed in this article.

The post How to read NetSuite data in Power BI appeared first on ZappySys Blog.

In this article, we will learn to create a Qlik REST connector to OneDrive, Google, and other services.

Qlik is a very powerful tool to analyze data from multiple sources and visualize, create relations and charts. It is a simple, but powerful tool used to visualize data.

In this tutorial, we will show how to get data from the REST API in Qlik. We will use the ZappySys ODBC PowerPack which is a very useful tool used to get data from REST API, WEB API, JSON files, XML files, SOAP and OData using simple SQL queries to the data.
This article will show how to extract data from Facebook, Twitter, and Google Sheets in Qlik using REST API.

Requirements for Qlik REST connector

1. First of all, you will need Qlik Sense for Desktop.
2. Secondly, ZappySys ODBC PowerPack installed.

Getting started – Reading data form API

We will first show how to connect to OData in JSON format.

Configure ODBC Data Source for API Connection

1. First of all, we will access the following URL:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Orders?$format=json

2. Secondly, in the windows start menu, Search for “ODBC” open the ODBC Data Sources.
3. Also, in the ODBC Administrator, press Add and select the ZappySys JSON
   Driver:
   

   Add ODBC JSON

4. Finally, specify the URL of step 1 and save the configuration:
   

   URL ODBC

Reading REST API data in Qlik (via ODBC Connector)

1. First of all, open the Qlik Sense Desktop:
   

   Open Qlik Sense
2. Secondly, enter your login and password:
   

   Login to Qlik Sense

3. Also, create a new App:
   

   Create new app Qlik Sense

4. Additionally, enter your App name:
   

   Add new Qlik create app

5. In addition, select the option Add data from files and other sources:
   

   Add data from files and folders

6. Select the ODBC data source:
   

   Select ODBC connector

7. Also, select the ZappySys DSN created in the Create Qlik REST connector to OData:
   

   Qlik REST connector

8. Finally, you will be able to see the data and create graphs and more. For more information about Qlik Sense refer to this link.
   

   Qlik to Odata

Other REST API Examples in Qlik

Now lets look at a few more examples of other popular REST API services (JSON or XML API).

Google SpreadSheet / Drive API Example in Qlik

In this new example, we will connect to a Google Sheet. The Sheet is the
following:

Qlik to Google Sheet

1. First of all, create a new ZappySys JSON Driver Connection in the ODBC.
2. Secondly, you will need to specify the following link:
   

   https://sheets.googleapis.com/v4/spreadsheets/1yfNChaljZvAL-HXF4lmM4o16Paa_BZuBWSflgBssZAQ/values/A1:B4

   where 1yfNChaljZvAL-HXF4lmM4o16Paa_BZuBWSflgBssZAQ is the ID of the Google Sheet and A1:B4 are the ranges of cells to retrieve:
   

   Google file id

3. Also, in filter enter $.values[*] and select the OAuth Connection Type and press the link to configure the OAuth connection:
   

   Google drive url

4. In addition, select Google OAuth Provider and specify the following scope and Generate a Token:
   

   OAuth REST API Qlik Sense

5. Finally, create an Application in Qlik Sense Desktop and add the ODBC DSN created in the previous step and you will have the Google Sheet data:
   

   Display data REST

Twitter REST API Example in Qlik

In this new section, we will show how to connect to Twitter and get the
information in Qlik using REST.

1. First of all, we will use the ODBC Administrator to use the ZappySys ODBC PowerPack and select the ZappySys JSON Driver.
2. Secondly, we will enter the following URL in the ZS JSON Drive:
   

   https://api.twitter.com/1.1/statuses/user_timeline.json?

3. Also in connection type, select OAuth:
   

   Select OAuth options

4. In Addition, in the OAuth parameters, select the Twitter OAuth provider and generate a Token. The application will ask for your Twitter credentials:
   

   Qlik connection information

5. Finally, create a new app in Qlik Sense Desktop and add the ODBC DSN just created and you will have your Twitter data in Qlik Sense.
   

   Qlik sense to Twitter

Facebook REST API Example in Qlik

Facebook is another popular social network with interesting information. The following example will show how to show the number of Facebook friends in Qlik using REST API.

1.  First of all, open the ODBC Administrator in Windows and add the ZappySys JSON Driver.
2. Secondly, you will need a Facebook application. The following link shows you how to configure the application.
   

   Connect Facebook Qlik

3. In your ZappySys ODBC driver, add your client id and client secret that can be obtained from the Facebook Application:
   

   Facebook Connection information for qlik

4. Finally add your ODBC connection to Facebook in Qlik Sense as we did in previous examples.

ZappySys JSON /REST API Driver Query Examples

Reading from XML files or API can be done using the same way as previous sections except you have to use ZappySys XML Driver. Read help file here to see json query examples.

ZappySys XML / SOAP Driver Query Examples

Reading from XML files or API can be done using the same way as previous sections except you have to use ZappySys XML Driver. Read help file here to see xml query examples.

Calling XML SOAP Web Service in Qlik

So far we have looked at examples to consume data using JSON driver. Now lets look at an example, to call XML SOAP Web Service in Qlik.

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

POST data to REST API URL from file in Qlik

Above example was POST data to API URL but what if your Request Body is large and you have saved that to file? Well here is the way to get your request body from a file (Use @ symbol before path).

SELECT * FROM $
WITH 
(METHOD='POST' 
,HEADER='Content-Type:text/plain || x-hdr1:AAA'
,SRC='http://httpbin.org/post'
,BODY='@c:\files\dump.xml'
,IsMultiPart='True'
)

REST API Pagination in Qlik Sense / QlikView

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

Error Handling in REST API / SOAP

Sometimes errors occur... they just do and there is nothing you can do! Or can you? Actually, in ODBC PowerPack you can handle them in two ways.

METHOD 1 - Using Error Handling Options

When to use?

You may want to use them when your source is a resource located on the Internet; e.g. a file on a website, a file on an FTP server or just a plain API HTTP response. By default, when a remote server returns an error, data retrieval is stopped, an error is raised and no data is given back to you. This might not be always desirable.

Scenario 1

Imagine a scenario, that there is a web server which each day at 12 AM releases a new JSON file with that day's date as filename, e.g. http://www.some-server.com/data/2018-06-20.json. And, of course, you want to download it and use it daily in your Power BI report. But you have a problem: Power BI report data sources are refreshed each hour and you may get HTTP 404 status code (no file was found) when a file is not released yet. Which consequentially means other data sources won't be updated as well and you will see old and cached data on the report. That's where you could use Continue on any error or Continue when Url is invalid or missing (404 Errors) to avoid an error being raised and let other data sources to be updated.

Scenario 2

Another scenario is when you expect a web server to raise some kind of HTTP error when accessing a URL. You don't want ODBC Data Source to raise an error but instead, you want to get response data. That's where you can use Continue on any error or alike together with  Get response data on error to continue on an error and get the data:

METHOD 2 - Using Connection [Retry Settings]

Another scenario you may run into is a buggy web server. You ask it to give you some file or data and it, like a snotty kid, just doesn't give it to you! You have to ask twice or thrice before it does its job. If that's the case, you have to retry HTTP requests using Connection:

REST API / SOAP Connection Types in Qlik (OAuth / HTTP)

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Performance consideration for Web API Calls

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Other considerations for Web API calls in Qlik

Conclusion

To conclude, we can say that Qlik Sense with the help of ZappySys ODBC PowerPack can be useful to access to different data using REST API. We learned how to connect Qlik Sense with ODBC to connect to OData, Google Drive Sheets, Facebook information and Twitter. If you enjoy it. Try our product.

References

Finally, for more information about Qlik Sense and REST API with ODBC, refer
to these links:

• ODBC PowerPack – Drivers for REST API, JSON, XML, OData
• Qlik Sense – Getting Started Webinar

The post Qlik REST Connector Examples – Read JSON / XML API appeared first on ZappySys Blog.