ODBC App Integration Archives | ZappySys Blog

Import SAP S/4HANA OData Service Data Into Ms Access via ODBC Driver

ZappySys — Sat, 27 Jul 2019 06:34:32 +0000

Introduction

In our previous blog we saw how to read JIRA data in SQL Server. Now let’s learn how to Import SAP S/4HANA OData Service Data Into MS Access. SAP S/4HANA provides OData REST API interface to access data in your application using HTTP Protocol. We will use ODBC XML Driver to read SAP data and load into MS Access.

So, let’s get started.

Requirements

A first requirement, make sure that Microsoft Access installed
Finally, make sure to have ZappySys ODBC PowerPack installed.

About SAP HANA / OData REST API Service

You can expose your SAP Data using ODATA REST API Service. Here is a good article which shows how to expose data as OData Service. For more information on SAP OData Service feature check this SAP help page. If you are new to OData Standard then read here to know more how OData can facilitate data extraction using HTTP REST API. If you need Sample XML based OData Service then use below test URLs.

https://services.odata.org/Northwind/Northwind.svc/
https://services.odata.org/Northwind/Northwind.svc/Customers
https://services.odata.org/Northwind/Northwind.svc/Orders
https://services.odata.org/Northwind/Northwind.svc/Invoices
https://services.odata.org/Northwind/Northwind.svc/Products

Now let’s look at how to read SAP Data using ODBC Driver. At this point we assume you have exposed your data as OData Endpoint.

Read SAP S/4HANA data using XML Driver

First let’s look at steps to configure XML Driver. We will read data from SAP S/4HANA OData Service and then in next section we will look at how to load data into MS Access or other target.

First, Windows search, write ODBC and select the ODBC Data sources (32 bits)

Open ODBC Data Source
As a second step, in ODBC Data source Administrator press the Add button.
Add ZappySys
In this step, create the new data source, select ZappySys XML Driver.
Add new zappysys xml driver
Now edit that XML data source to configure it. Enter your OData Service URL its typically like below. Replace 3 parts with your own value (i.e. replace {MY-INSTANCE}, {MY-PROJECT}, {MY-TABLE})
```
https://{MY-INSTANCE}/sap/opu/odata/sap/{MY-PROJECT}/{MY-TABLE}
```
For example if you are hosting SAP HANA in Cloud Instance then your URL may look like below
```
https://myXXXXXX-api.s4hana.ondemand.com/sap/opu/odata/sap/MyTestProject/PurchaseOrders
```
And create New ZS-HTTP connection in it select Basic Authentication and enter your SAP HANA UserID / Password to call OData Service and select the desire filter.

XML Driver : SAP S/4HANA OData Service Configuration
Now go to Data Format / Compression (Zip/GZip) tab and select Data Format as OData to get all the records.
Data Format : Odata
Finally, now using Query Builder and Code Generator we will generate the query and click on Preview the data.
Generate Query
That’s it we are ready to load SAP S/4HANA OData Service data to MS Access.

Load SAP S/4HANA OData Service in MS Access

In MS Access, go to External Data Ribbon and select New Data Source and select From Other Sources and ODBC Database.
A configuration in MS Access: Import Data From REST API
Select the Source and destination of the data, select Import the source data into a new table in the current database and press OK.

Get External Data – ODBC Database
Select your newly created JSON Driver Data Source and click OK.

Get External Data – Select your newly created ODBC Data Source (JSON Driver)
Select tables from the list of tables in the Import Objects screen and click OK.

Get External Data – Select tables from Import Objects Screen
Select Save Import Steps from the screen and click on Save Import button.
Get External Data – Select Save Import Steps
That’s it If everything is OK, you will be able to see the data.
Data Imported Successfully in MS Access DB Table

How to link REST API data to Access

The linking process is very similar to importing data into the Access table. Follow steps of the previous paragraph – How to import REST API data to Access – but on step 2 select the second option to create a linked table instead. Then at last it’s asked to select a unique record identifier, don’t select anything:

Linked table approach – Don’t select a unique Record identifier

SAP S/4HANA Integration with Other BI Tools

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

Troubleshooting Errors

While running in Access\Excel\other and reading data from DSN created with ODBC PowerPack, if you get this error "License type [ODBC_PP_TRIAL] not found or its expired"

Please refer to this article for the same: Troubleshooting "License type [ODBC_PP_TRIAL] not found or its expired" error in Microsoft Access

Conclusion

So in this blog, we learned how to Import SAP S/4HANA OData Service Data in Access using ODBC XML / SOAP API Driver in a very simple way. You can achieve many more functionalities with this tool. Check our blogs/articles on XML File / REST API Driver to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the use of SAP S/4HANA OData Service with our tools:

SAP API Business Hub
Landing Page for ODBC XML / SOAP API Driver, you can also find Tutorial Video here.
Help File: Documentation of XML Driver.
Blog/articles link: https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/xml-soap-api-driver/

The post Import SAP S/4HANA OData Service Data Into Ms Access via ODBC Driver appeared first on ZappySys Blog.

How to Export REST API to MS Access using VBA Command Button

ZappySys — Sat, 29 Jun 2019 09:34:55 +0000

Introduction

In our previous blog we saw how to export REST API to CSV using c# or Python. In this post we will look at specific example on How to Export REST API to MS Access using VBA Command Button, along with few other topics such as how to make REST API Call, how to read all Customers data from Odata using our ODBC JSON Driver and load it into the Ms Access Table.

We will go through the steps to read data from Odata and Load into MS Access using VBA Command Button.

In nutshell, this post will focus on how to make ODBC Driver Call using VBA.

So let’s get started.

Requirements

First of all, you will need ZappySys ODBC PowerPack installed.
Secondly, Make sure that Microsoft Access installed for VBA example.

An introduction to Rest API and OData

In this example, we will use OData (Open Data Protocol) to consume REST API. REST API (Representational State Transfer Application Program Interface) allows handling the interoperability between computers and the internet.

In REST API we can handle web services in different formats. In this example, we will work with the Northwind example. The Northwind example is available in this URL:

https://services.odata.org/V3/Northwind/Northwind.svc

By default, the data is displayed in XML format. To show the data in JSON use this URL:
```
https://services.odata.org/V3/Northwind/Northwind.svc/?$format=json
```
There are collections of data like Categories, CustomerDemographic, Customers, Invoices, etc. For example, the following URL will show the data of the categories collection:
```
https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json
```
In the next steps, we will use ZappySys drivers to connect to this URL and query using OData.

Configure ODBC DSN for ZappySys JSON Driver

ODBC driver can be accessed in two modes.

Using DSN
Without DSN (Supply direct Connection String e.g. DRIVER={ZappySys JSON Driver}; …….. )

In this article, we will use the DSN approach (User DSN). We will first add the ZappySys JSON Driver in the ODBC Data Source Administrator.

Follow these steps to accomplish the task:

First, Windows search, write ODBC and select the ODBC Data sources (32 bits)

Open ODBC Data Source
As a second step, in ODBC Data source Administrator press the Add button.

Add ZappySys
In this step, create the new data source, select ZappySys JSON Driver.

Add new zappysys json driver
Here we have several properties, write a data source name. In this example, the name will be ZappySys JSON to Excel.
The Data Source (URL or file path) can specify the URL of the source or if it is a local file, you can specify the local path. In this example, the URL is:
```
https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json
```
You can also specify a local file path as Data SourceFor a single file: c:\data\myfile_1.json
For multiple files: c:\data\myfile_*.json
Configure the ODBC JSON Driver like this and test the connection:
Configure JSON Driver for REST API Call
Now click on Copy Connection string Button.
ODBC Driver : Copy Connection String
That’s it Connection string is copied to clipboard. Now let’s use this connection string in the VBA code.

How to import REST API data to Access using VBA

Open the access database and create the table with the desire columns to get API data in it.
Create Table in Access
Now close the table, and go to Create menu and click on Form Design to add the new blank form in design view.
Access : Create a New Form in Design View
It will open the new form in the design view, now add new button with the desire text.
Access : Add Command Button
Now right click on button and go to properties and click on … builder button and select Code Builder.
Access : Code builder

When you click the button, it will open a new VBA instance, executing the command event associated with that button click.
Next, insert the provided code inside the Command5_Click() function. In this example, the button is named Command5, and that’s why it’s added within this click function.
The provided code encompasses the following steps:

Ensure the table is closed:
It’s crucial to close the table beforehand; otherwise, executing the subsequent command may result in an error.
Truncate the table:
Clear the existing data from the table. If you want to add fresh data each time
Make the API call:
Utilize the ODBC Driver connection string to initiate an API call.
Insert into the Access table:
Use the retrieved data from the API call to insert new records into the Access table.
Open the table:
Reopen the table to inspect the newly inserted data.

' Pass your ZappySys Driver Connection String: Refer to below link to learn how to obtain the ZappySys Driver connection string:
    ' https://community.zappysys.com/t/how-to-copy-the-zappysys-driver-connection-string/172
    Dim zsConnStr As String
    zsConnStr = "DRIVER={ZappySys JSON Driver};DataPath='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json';Filter='$.value[*]';IncludeParentColumns=0;RequestMethod='GET';RequestHeaders='cache-control: no-cache || Accept: */*'"
        
    'ZappySys Driver Query
    Dim zsDriverQuery As String
    zsDriverQuery = "Select * from $"
 
    'MS Access Table Name
    Dim tableName As String
    tableName = "tblCustomers"

    ' 1. Close Table If Open
    CloseTableIfOpen tableName

    ' 2. Truncate Table (Optional): If you want to load fresh data each time, uncomment the line below.
    'TruncateTable tableName

    ' 3. Import Data From API Into Table
    ImportDataFromAPIIntoTable zsConnStr, zsDriverQuery, tableName

    ' 4. Open Table for Viewing (Optional)
    OpenTableForViewing tableName

    ' Show Message after Loading Data Successfully
    MsgBox "Data Loaded Successfully!"

VBA Button OnClick Function

Below the Private Sub Command5_Click() function, please add the necessary functions and procedures as provided below:

Sub ImportDataFromAPIIntoTable(zsConnStr As String, zsDriverQuery As String, tableName As String)
    Dim dscn As New ADODB.Connection
    dscn.Open zsConnStr

    Dim rs As Object
    Set rs = New ADODB.Recordset
    
    ''ZappySys Driver Query Goes here
    rs.Open zsDriverQuery, dscn

    Dim strQuery As String
    strQuery = ""
    ' Check if the recordset is not empty
    If Not rs.EOF Then
        ' Get the field names
        Dim fieldNames As String
        fieldNames = ""

        For i = 1 To rs.Fields.Count
            fieldNames = fieldNames & rs.Fields(i - 1).Name
            If i < rs.Fields.Count Then
                fieldNames = fieldNames & ","
            End If
        Next i

        ' Generate the INSERT script
        Dim insertScript As String
        ' Iterate through the recordset to get the values
        Do Until rs.EOF
            ' Generate Insert script
            insertScript = "INSERT INTO " & tableName & " (" & fieldNames & ") VALUES"
            insertScript = insertScript & vbCrLf & "("
            For i = 1 To rs.Fields.Count
                ' Assuming all fields are text, you might need to handle different data types appropriately
                
                If IsNull(rs.Fields(i - 1).Value) Then
                    insertScript = insertScript & "''"
                Else
                    insertScript = insertScript & "'" & EscapeSingleQuote(rs.Fields(i - 1).Value) & "'"
                End If
                
                
                If i < rs.Fields.Count Then
                    insertScript = insertScript & ","
                End If
            Next i
            insertScript = insertScript & ")"
            
            'Insert data
            CurrentProject.Connection.Execute insertScript
            
            rs.MoveNext
        Loop
    End If
    
    ' Close the recordset and connection
    rs.Close
    dscn.Close
    Set rs = Nothing
    Set conn = Nothing
    
End Sub

Sub OpenTableForViewing(tableName As String)
    If Not IsTableOpen(tableName) Then
        DoCmd.OpenTable tableName
    End If
End Sub

Public Function IsTableOpen(strName As String) As Boolean
     IsTableOpen = SysCmd(acSysCmdGetObjectState, acTable, strName)
End Function

Sub CloseTableIfOpen(tableName As String)
    If IsTableOpen(tableName) Then
        DoCmd.SelectObject acTable, tableName
        DoCmd.RunCommand acCmdSaveRecord
        DoCmd.Close
    End If
End Sub


Sub TruncateTable(tableName As String)
    Dim strSql As String
    strSql = "DELETE FROM " & tableName & ";"
    CurrentProject.Connection.Execute strSql
End Sub

Public Function EscapeSingleQuote(strData As String) As String
     EscapeSingleQuote = Replace(strData, "'", "''")
End Function

Now, save the VBA form code and provide some appropriate form names.
Save form vba code
Make sure to add the reference to the Microsoft ActiveX Data Objects X.0 Library in the project settings.In Microsoft Access VBA,
You can add the reference to the Microsoft ActiveX Data Objects X.0 Library by following these steps:
1. In the VBA editor, go to “Tools” in the menu and select “References.”
2. In the References dialog box, scroll down the list to find “Microsoft ActiveX Data Objects X.0 Library” (where “X.0” corresponds to the version number, such as 6.1, 2.8, etc.).
3. Check the box next to the appropriate version to enable the reference.
4. Click “OK” to save the changes and close the References dialog box.
Now, your Access project has a reference to the Microsoft ActiveX Data Objects library, and you can use its features in your VBA code.

Access : Add reference

If the MS ActiveX reference is not added, an error will be thrown.

VBA : Error Type not defined
Close the code project, open the form in Form View, and click on the button.
Access : Click Command Button
After clicking the button, it will initiate a REST API call, insert the data into the Access table, and open the table to display the inserted records along with a success message.
Access : Open the table

Troubleshooting Errors

While running in Access\Excel\other and reading data from DSN created with ODBC PowerPack, if you get this error "License type [ODBC_PP_TRIAL] not found or its expired"

Please refer to this article for the same: Troubleshooting "License type [ODBC_PP_TRIAL] not found or its expired" error in Microsoft Access

Conclusion

In this article, we saw how to use VBA to call REST API. We show how to do REST API calls in Access using the ZappySys ODBC driver. Also, we show how to write VBA Code in MS Access for command button to make REST API call, truncate access table and insert record in the access table using VBA code. If you liked this article and you want to try, you can download the ZappySys ODBC installer here.

References

Form VBA Code:

After completing the migration of the higher-level VBA code, you’ll notice a transformation in the structure of your forms’ code.

Option Compare Database

Private Sub Command5_Click()
 
    ' Pass your ZappySys Driver Connection String: Refer to below link to learn how to obtain the ZappySys Driver connection string:
    ' https://community.zappysys.com/t/how-to-copy-the-zappysys-driver-connection-string/172
    Dim zsConnStr As String
    zsConnStr = "DRIVER={ZappySys JSON Driver};DataPath='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json';Filter='$.value[*]';IncludeParentColumns=0;RequestMethod='GET';RequestHeaders='cache-control: no-cache || Accept: */*'"
        
    'ZappySys Driver Query
    Dim zsDriverQuery As String
    zsDriverQuery = "Select * from $"
 
    'MS Access Table Name
    Dim tableName As String
    tableName = "tblCustomers"

    ' 1. Close Table If Open
    CloseTableIfOpen tableName

    ' 2. Truncate Table (Optional): If you want to load fresh data each time, uncomment the line below.
    'TruncateTable tableName

    ' 3. Import Data From API Into Table
    ImportDataFromAPIIntoTable zsConnStr, zsDriverQuery, tableName

    ' 4. Open Table for Viewing (Optional)
    OpenTableForViewing tableName

    ' Show Message after Loading Data Successfully
    MsgBox "Data Loaded Successfully!"
    
End Sub

Sub ImportDataFromAPIIntoTable(zsConnStr As String, zsDriverQuery As String, tableName As String)
    Dim dscn As New ADODB.Connection
    dscn.Open zsConnStr

    Dim rs As Object
    Set rs = New ADODB.Recordset
    
    ''ZappySys Driver Query Goes here
    rs.Open zsDriverQuery, dscn

    Dim strQuery As String
    strQuery = ""
    ' Check if the recordset is not empty
    If Not rs.EOF Then
        ' Get the field names
        Dim fieldNames As String
        fieldNames = ""

        For i = 1 To rs.Fields.Count
            fieldNames = fieldNames & rs.Fields(i - 1).Name
            If i < rs.Fields.Count Then
                fieldNames = fieldNames & ","
            End If
        Next i

        ' Generate the INSERT script
        Dim insertScript As String
        ' Iterate through the recordset to get the values
        Do Until rs.EOF
            ' Generate Insert script
            insertScript = "INSERT INTO " & tableName & " (" & fieldNames & ") VALUES"
            insertScript = insertScript & vbCrLf & "("
            For i = 1 To rs.Fields.Count
                ' Assuming all fields are text, you might need to handle different data types appropriately
                
                If IsNull(rs.Fields(i - 1).Value) Then
                    insertScript = insertScript & "''"
                Else
                    insertScript = insertScript & "'" & EscapeSingleQuote(rs.Fields(i - 1).Value) & "'"
                End If
                
                
                If i < rs.Fields.Count Then
                    insertScript = insertScript & ","
                End If
            Next i
            insertScript = insertScript & ")"
            
            'Insert data
            CurrentProject.Connection.Execute insertScript
            
            rs.MoveNext
        Loop
    End If
    
    ' Close the recordset and connection
    rs.Close
    dscn.Close
    Set rs = Nothing
    Set conn = Nothing
    
End Sub

Sub OpenTableForViewing(tableName As String)
    If Not IsTableOpen(tableName) Then
        DoCmd.OpenTable tableName
    End If
End Sub

Public Function IsTableOpen(strName As String) As Boolean
     IsTableOpen = SysCmd(acSysCmdGetObjectState, acTable, strName)
End Function

Sub CloseTableIfOpen(tableName As String)
    If IsTableOpen(tableName) Then
        DoCmd.SelectObject acTable, tableName
        DoCmd.RunCommand acCmdSaveRecord
        DoCmd.Close
    End If
End Sub


Sub TruncateTable(tableName As String)
    Dim strSql As String
    strSql = "DELETE FROM " & tableName & ";"
    CurrentProject.Connection.Execute strSql
End Sub

Public Function EscapeSingleQuote(strData As String) As String
     EscapeSingleQuote = Replace(strData, "'", "''")
End Function

The post How to Export REST API to MS Access using VBA Command Button appeared first on ZappySys Blog.

ODBC PowerPack v1.1.3 Released

ZappySys — Tue, 25 Jun 2019 13:35:30 +0000

What’s New

In this release we focused on adding URL JOIN Features which may allow many multi steps scenarios in one query. Bing Ads, Amazon MWS, Google BigQuery, Amazon Athena all these supports Job style queries which needs 3 step process (Create job, keep checking status, once done read report data). We also improved compatibility with Informatica and MS Access.

Version 1.1.3.10621 [Jun 24, 2019]

New Features/Improvements

NEW: Add option to replace placeholders in File content if you use IsMultiPart = True
NEW: All API Drivers – Add option EnableBodyPlaceholderForMultiPart to support replacing placeholders in file content in IsMultiPart mode (E.g. replace <%access_token%> or [$token$] or [$alias.MyJoinColumn$] )
NEW: All API Drivers – Add option OutputAsRawDocumentSingleRow (Returns __DOCUMENT__ field with entire File / Response string in one field)
NEW: All API Drivers – Add Options to parse from any unstructured data – RawOutputExtractMode, RawOutputDataRowTemplate, RawOutputFilterExpr
NEW: All API Drivers – Add Status Check feature for all API drivers (Keep sending request until desired value found) – Options added EnableStatusCheck, StatucCheckMaxWaitSeconds, StatucCheckIterationWaitSeconds, StatusSuccessValue, StatusFailureValue
NEW: All API Drivers – Add support for ConnectionType in URL Join
NEW: All API Drivers – Add Support for HMAC Authentication
NEW: All ODBC Drivers – Add support for Parameters
NEW: Data Gateway – Add Code Generator UI for SQL / Java
NEW: Data Gateway – Allow to add firewall rule via UI
NEW: Data Gateway – Better logging about connected devices
NEW: Salesforce Driver – Add support for Select * FROM

Bug fixes

FIX: All API Drivers – joinXX_DataConnectionType Option not supported in JOIN URL
FIX: All API Drivers- Show meaningful error when same option used twice in WITH clause (Right now you see odd error – An item with the same key has already been added)
FIX: Apps – Informatica – Driver may output many extra empty rows in some cases
FIX: Apps – MS Access – Linked table throwing call fail error (due to missing support for parameters in current driver)
FIX: Data Gateway – Certain errors not reported correctly back to caller
FIX: Data Gateway – Enabling caching options may hang query in some case
FIX: Data Gateway – Generic ODBC Connection Type fails with error Connection is Busy
FIX: Data Gateway – When you use Meta option in WITH clause you may see empty rowset rather error in some cases
FIX: Salesforce Driver – Can not load table list in Excel / Power BI
FIX: Salesforce Driver – Failed to preview data in Power BI
FIX: Salesforce Driver – Load ConnectionString doesnt set Username
FIX: Salesforce Driver – Nested queries not supported – You may get syntax error
FIX: Salesforce Driver – SSIS ODBC destination showing bad Table names in list

The post ODBC PowerPack v1.1.3 Released appeared first on ZappySys Blog.

Read / Write REST API data in Talend (JSON / XML / SOAP)

ZappySys — Sat, 12 Jan 2019 20:18:31 +0000

Introduction

UPDATE: Please visit Talend Studio Connectors page in API Integration Hub to find updated articles on various Talend Studio integrations, including integrations with REST API, SOAP API, JSON file, and XML file.

In this post we will learn how to read / write REST API data in Talend Open Studio. We will create a simple Talend Job using ZappySys JSON Driver to read from REST API / JSON Files and load into Target (e.g. File / DB). Techniques listed in this article can be also used to read from SOAP API / XML Files or CSV Files / API using XML Driver / CSV Driver.

These drivers support familiar SQL query language. Using SQL you can query virtually any API services just like relational database table. It can flatten nested hierarchy and provide output in rows / columns. Many complex REST API / SOAP API complexity is taken care automatically (e.g. Authentication, Pagination, Security, Error Handling).

So let’s get started.

Requirements

Download and install Talend Open Studio (FREE) from here. Skip this step if you already installed.
Download ZappySys ODBC PowerPack (JSON / XML Drivers)
Get Microsoft JDBC driver for SQL Server from here (Download sqljdbc_6.0.8112.200_enu.exe which is self extracting file you can run and extract to some folder)
After you extract jdbc files, go to sqljdbc_6.0\enu\jre8\ folder rename sqljdbc42.jar to mssql-jdbc.jar (name must be this). We will load this file in Talend later in this article.
Basic knowledge about REST API and JSON / XML format.

Configure Data Gateway

Now let's look at how to configure ZappySys Data Gateway. This feature acts as a bridge between Client App and ZappySys Drivers. Using data gateway you can use ZappySys Drivers inside applications / operating systems where ZappySys drivers may not be available directly for some reason (e.g. You don't have access to Server for Installation or System does not support ODBC drivers like JAVA programs). Click here to read more on various use cases of Data Gateway.

Configure Data Gateway User / Port

Now let's look at steps to configure Data Gateway after installation. We will also create a sample data source for ODATA API (i.e. JSON based REST API Service).

Assuming you have installed ZappySys ODBC PowerPack using default options (Which also enables Data Gateway Service)
Search "Gateway" in your start menu and click ZappySys Data Gateway

Open ZappySys Data Gateway
First make sure Gateway Service is running (Verify Start icon is disabled)
Also verify Port on General Tab

Port Number setting on ZappySys Data Gateway
Now go to Users tab. Click Add icon to add a new user. Check Is admin to give access to all data sources you add in future. If you don't check admin then you have to manually configure user permission for each data source.

Add Data Gateway User

Configure Data Source

After user is added, go to Data Sources tab. Click Add icon to create new data source. Select appropriate driver based on your API / File format. You can choose Generic ODBC option to read data from ODBC DSN or use Native Driver option.
```
NOTE: Whenever possible use native driver option for better performance / security and ease of use.
```
Add Gateway Data Source (Native JSON Driver)
Click on "Edit" under Data source and configure as per your need (e.g. Url, Connection, Request Method, Content Type, Body, Pagination etc.). For this demo we are going to pick simple JSON REST API which doesn't need any authentication. Enter following URL.
```
https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json
```
You can also view response structure and select default hierarchy (i.e. Filter) like below (Select Array Icon) for data extraction.

Configure JSON API Data source

Test SQL Query / Preview Data

Now go to Preview Tab. You can click Preview button to execute default query OR Select Table name from dropdown to generate SQL with column names.

JSON / REST API Driver Query Preview / Query Examples (Read REST API or JSON Files)
You can also click Query Builder to generate SQL using different options in WITH clause. ANy setting you specify in WITH clause will override UI settings we applied in previous steps.

Using SQL Query Builder (For Files or REST / SOAP API - JSON / XML / CSV Format)
There is another useful option for code generation. Select your Language and quickly copy code snippet. See below Example of XML Driver Query to call SOAP API.

Generate Example Code for ZappySys Driver
Click OK to Close Data Source UI
Once data source is tested and configured you can click Save button in the Gateway UI toolbar and click Yes for Restart Service.

Register MS SQL JDBC driver in Talend

Now lets register Microsoft JDBC Driver in Talend. This is very important step because MSSQL JDBC driver is used to communicate with ZappySys Data Gateway we configured in previous step.

If you missed steps mentioned in the Requirements section then make sure you first download JDBC driver using below steps.

Get Microsoft JDBC driver for SQL Server from here (Download sqljdbc_6.0.8112.200_enu.exe which is self extracting file you can run and extract to some folder). After you extract jdbc files, go to sqljdbc_6.0\enu\jre8\ folder rename sqljdbc42.jar to mssql-jdbc.jar (name must be this).

Now lets go through the steps to register MSSQL jdbc driver in Talend.

Open Talend Open Studio
Go to Windows > Click Show View
Talend – Show View
When you see Popup selection under Talend > Select Modules
Talend – Select Modules Window
When Module window is visible click on Little Jar Icon (Bottle icon) in the toolbar.
Select mssql-jdbc.jar file we renamed earlier and load this file.
Talend – Import JAR file / module (Register MS SQL JDBC Driver Example)
That’s it. Now we ready to make API calls / read from JSON / XML in the next section.

Setup Talend REST API Connection (JSON / XML / CSV)

Now let’s configure REST API Connection in Talend. To read from JSON / XML Files you can use same steps too. We will use MSSQL JDBC Driver to connect to ZappySys Data Gateway.

In Talend Go to Metadata > Db Connections (Right click) > Create Connection
Talend – Create new DB connection (JSON / REST API Example)
On the connection Wizard specify following attributes.
DB Version : Microsoft
Login : username you setup in zappysys data gateway
Password : password of data gateway user
Server : machine name or IP where zappysys data gateway is running
Port : Port on which zappysys data gateway is listening
DataBase : Data source name you setup in zappysys data gateway (case-sensitive)

Talend – Connect to JSON / REST API using ZappySys Gateway (Use MS SQL JDBC Driver)
That’s all we need to do to setup a connection which can be used to read / write REST API data in Talend. In the next section we will see how to create a job to read data from REST API Service using this connection.

Read from REST API in Talend

Now let’s look at how to read data from REST API source or JSON / XML File using the connection we configured in the previous section.

Configure REST API Source

Create a Talend JOB and double click to open designer
Now drag and drop MSSQL Connection we created for ZappySys Data gateway, drop it on the designer surface. It will popup UI like below.
Select tDBInput (Microsoft SQL Server). Remember that we are using MSSQL JDBC Driver to connect to ZappySys Data Gateway for REST API Call. This gateway uses Microsoft TDS Protocol so MSSQL JDBC driver is used to communicate.
Now rename Source to something meaningful (e.g. Read from JSON REST API)
Double click REST Source to configure.
Enter Query like below (Make sure to enter between double quotes). See below examples to read from URL or File. If you have double quote in SQL then escape using \” character (e.g. select \”my col\” from $ )Read From REST API Url
```
"SELECT * FROM value 
WITH (SRC='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json')"
```
Read from Local File (Wildcard Pattern Search allowed)
```
"SELECT * FROM value 
WITH (SRC='c:\data\Customers_*.json')"
```
Click on Guess Schema button and Click OK to accept detected schema.
Talend – Configure REST API / JSON Source (Enter SQL Query / Guess Schema)
Now we will configure target in the next section.

Configure Target (Delimited File)

Now search for “FileOut” in the toolbox (Hit Enter). You will see tFileOutputDelimited so just select that for now and drag on the surface.
Talend – Add File Output (tFileOutputDelimited)
Double click it to configure.
Enter correct file path (e.g. “C:/Talend/workspace/rest-api-output.csv” )
On Advanced Tab you can configure some additional settings (e.g. Throw an error if file already exists)

Connect and Run

Once you have configured Source and Target its time to connect them
Drag Source Port to Target to connect like below.
Talend – Connect REST / JSON Source to File Target
Run the job
Run Talend Job – Loading REST API data into File (Read JSON / XML / CSV)
That’s it. So in few clicks you loaded data from REST API to File in Talend Open Studio.

Write / Send data to REST API (POST Example)

There will be a time when you want to POST data to REST API service. Let’s check how to write POST query to submit data to REST API.

Just like how we did Read query in previous example, we can set POST Body in the SQL Query to send data. Use query like below and click Guess Schema button. If Blank Filter gives you no data error then make sure you remove Filter on Data Gateway Data source. (Notice we used \” to escape double quote inside query )

"SELECT * FROM _root_
WITH (
 METHOD='POST'
,HEADER='Content-Type: text/plain || first-header: AAA || second-header: BBB'
,SRC='http://httpbin.org/post'
,BODY='{id:1,notes:\"Some notes\"}'
,FILTER=''
)
"

Talend – POST data to REST API URL

SQL Query Examples

Click on below link to learn more writing SQL Query using ZappySys Drivers.

JSON / REST Driver – SQL Query Examples

XML / SOAP Driver – SQL Query Examples

CSV / REST Driver – SQL Query Examples

REST API / XML SOAP Pagination Settings for Talend

Even we set up ODBC Data Source to get the data, it may not be enough. Usually, if you are getting a huge data set from API provider, it won't give it to you in one HTTP response. Instead, it gives back only a subset of data and provides a mechanism for data pagination. The good news is that ZappySys ODBC Driver includes many options to cover virtually any pagination method.

Below you will find a few examples of API pagination. If you need something more sophisticated check the below link (the article was written for SSIS PowerPack but UI options and concepts apply to ODBC Driver too):

https://zappysys.com/blog/ssis-rest-api-looping-until-no-more-pages-found/

Paginate by Response Attribute

This example shows how to paginate API calls where you need to paginate until the last page detected. In this example, next page is indicated by some attribute called nextlink (found in response). If this attribute is missing or null then it stops fetching the next page.

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

Paginate by URL Parameter (Loop until certain StatusCode)

This example shows how to paginate API calls where you need to pass page number via URL. The driver keeps incrementing page number and calls next URL until the last page detected (401 error). There are few ways to indicate the last page (e.g. By status code, By row count, By response size). If you don't specify end detection then it will use the default (i.e. No records found).

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

Paginate by URL Path (Loop until no record)

This example shows how to paginate API calls where you need to pass page number via URL Path. The driver keeps incrementing page number and calls next URL until the last page is detected. There are few ways to indicate the last page (e.g. By status code, By row count, By response size). If you don't specify end detection then it will use the default (i.e. No records found).

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

Paginate by Header Link (RFC 5988)

API like GitHub / Wordpress use Next link in Headers (RFC 5988)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

REST API / SOAP Web Service Connection Settings for Talend

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

HTTP
OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

SOAP WSS (when accessing a SOAP WebService)
Static Token / API Key (when need to pass an API key in HTTP header)
Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same):
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Other settings for REST API / SOAP XML Call in Talend

There are few settings you can coder while calling Web API

API Limit / Throttling

While calling public API or other external web services one important aspect you have to check, how many requests are allowed by your API. Especially when you use API pagination options to pull many records you have to slow down based on API limits. For example, your API may allow you only 5 requests per second. Use Throttling Tab on Driver UI to set delay after each request.

2D Array Transformation

If you are using JSON or XML API Driver then possible you may have to transform your data using 2D array transformation feature. Check this link for more information.

REST API / XML SOAP Performance Tips for Talend

While calling APIs you may face some performance issues. There are a few tips you can consider to speed up things.

Use Server-side filtering if possible in URL or Body Parameters

Many API supports filtering your data by URL parameters or via Body. Whenever possible try to use such features. Here is an example of odata API, In the below query the first query is faster than the second query because in the first query we filter at the server.

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

Avoid Special features in SQL Query (e.g. WHERE, Group By, Order By)

ZappySys API engine triggers client-side processing if special features are used in Query. Following SQL Features will trigger Client-Side processing which is several times slower than server-side processing. So always try to use simple query (Select col1, col2 .... from mytable )

WHERE Clause
GROUP BY Clause
HAVING Clause
ORDER BY
FUNCTIONS (e.g. Math, String, DateTime, Regex... )

LIMIT clause does not trigger client-side processing.

Consider using pre-generated Metadata / Cache File

Use META option in WITH Clause to use static metadata (Pre-Generated)There are two more options to speedup query processing time. Check this article for details.

select * from value WITH( meta='c:\temp\meta.txt' )
--OR--
select * from value WITH( meta='my-meta-name' )
--OR--
select * from value WITH( meta='[ {"Name": "col1", "Type": "String", Length: 100}, {"Name": "col2", "Type": "Int32"} ...... ]' )

Enable Data Caching Options (Found on Property Grid > Advanced Mode Only )

Consider using Metadata / Data Caching Option

ZappySys API drivers support Caching Metadata and Data rows to speed up query processing. If your data doesn't change often then you can enable this option to speed up processing significantly. Check this article for details how to enable Data cache / metadata cache feature for datasource level or query level. To define cache option at query level you can use like below.

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

Use --FAST Option to enable Stream Mode

ZappySys JSON / XML drivers support --FAST suffix for Filter. By using this suffix after Filter driver enables Stream Mode, Read this article to understand how this works.

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Calling SOAP Web Service in Talend

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

Getting WSDL file or URL
Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='

   
   
      
         
         UnitedStates
      
   
'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

Open SoapUI and click SOAP button to create new SOAP Project
Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL
```
http://www.dneonline.com/calculator.asmx?wsdl
```
Create new SOAP API Project in SoapUI tool for SOAP API Testing
Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
  
  Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
  Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
Now you can test your request first Double-click on the request node to open request editor.
Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
Create SOAP Request XML (With Optional Parameters)
Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
Goto System DSN Tab (or User DSN which is not used by Service account)
Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
(This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

Conclusion

In this article, we used ZappySys Drivers to read data from JSON REST API / File. You can use same technique to consume SOAP / XML API or File. Try ODBC PowerPack for FREE and check out how easy it is to consume virtually any REST / SOAP API or Read from JSON / XML / CSV Files in Talend Open Studio.

The post Read / Write REST API data in Talend (JSON / XML / SOAP) appeared first on ZappySys Blog.

Read JIRA data in SSRS Reports (SQL Server Reporting Services)

ZappySys — Thu, 27 Dec 2018 13:04:45 +0000

Introduction

UPDATE: ZappySys has released a brand new API Connector for Jira Online which makes it much simpler to Read/Write Jira Data in ODBC compared to the steps listed in this article. You can still use steps from this article but if you are new to API or want to avoid learning curve with API then use newer approach.

Please visit this page to see all Pre-Configured ready to use API connectors which you can use in SSIS API Source / SSIS API Destination OR API ODBC Driver (for non-SSIS Apps such as Excel, Power BI, Informatica).

In our previous blog we saw how to read rest api in SSRS reports. We recommend you go through that article first. In this post we will look at specific example on How to read JIRA data in SSRS Reports (SQL Server Reporting Services), along with few other topics such as how to generate an API Token using Atlassian Account for JIRA REST API Call, how to read all Issues data from JIRA using our ODBC Driver and load it into the SSRS reports.

We will go through the steps to read data from JIRA and Load into SQL Server using JSON Driver of ODBC PowerPack

In nutshell, this post will focus on how to call JIRA API using ODBC PowerPack and how to create SSRS Report.

So let’s get started.

Requirements

Make sure you have SSRS designer installed (You can download SSDT BI for Visual Studio).
Finally, make sure to have ZappySys ODBC PowerPack installed.

What is JIRA

JIRA is Atlassian’s popular project- and issue-tracking platform used by development and technical support teams to get more work done and faster.

It combines:

issue tracking – a software application that allows to record and follow the progress of every problem or “issue” that user identifies until the problem is resolved.
agile project management – an iterative approach to planning and guiding project processes. customizable workflow, and a pluggable integration – integrates with Freshdesk, GitHub, Zendesk, Zapbook, Asana and so on.

Getting Started

First of all, we will see several examples. ZappySys includes an SSIS JSON Source Connector that will help you to call JIRA API, read various data from JIRA with SSIS and do many more actions using REST API. To learn more about JIRA API check this help file.

Generate API Token

To access JIRA data using REST API call you need to generate an API Token in your Atlassian account. Follow the steps mentioned below to generate API Token:

you can directly login to the JIRA Atlassian Cloud site with your credentials, and redirect to the JIRA Atlassian Cloud Account's Profile Page for generating an API Token.
Click API tokens, then Create API token.

Generate API Token - Step - 1
Create API Token - Give Label name to your API token which you are generating now

Generate API Token - Step - 2
Use Copy to the clipboard, and paste the token into your or elsewhere:
Note: * For security reasons it isn't possible to view the token after closing the creation dialog; if necessary, create a new token. * You should store the token securely, just as for any password.

Generate API Token - Step - 3
If you forgot API token, then you can Revoke your old API token and create new API token OR you can create multiple API tokens too.

Generate API token - Step - 4

Now we have successfully generated an API token. Let´s start with an example. In this example, we will show how to read Tasks/Issues data from JIRA and store that data into a SQL table with SSIS.

About JIRA REST API

JIRA On-premises and Cloud Version both offers REST API. Here are some examples of JIRA REST API. For reading JIRA REST API data we will use JIRA REST API Search method which returns data in JSON format. Search API support use of JQL Syntax to filter issues. If you don't specify JQL parameters in /search endpoint then all issues will be returned. Check this link to learn how to build JQL for JIRA REST API Call. If you don't have JIRA hosted instance yet but still want to try REST API Examples on public JIRA Server then try below URL (Its official JIRA site which lists their own issues). For example, try to copy below URL in a browser and hit enter.

https://jira.atlassian.com/rest/api/2/search?jql=project=JRASERVER

Step-by-Step: Import JIRA data using REST API

Now once you have API token in the JIRA Atlassian Cloud website we can move to ODBC JSON Driver Configuration Part. For this example purpose you can use the following URL:

https://YOUR-DOMAIN.atlassian.net/rest/api/2/search?startAt=0&maxResults=100&jql=project=MYPROJECT

hence, your-domain is your Atlassian Cloud account’s domain name (we are using zappysys2.atlassian.net). Therefore startAt is the index of the first item returned in the page of results. And another maxResults is the maximum number of items that can be returned per page. Above all, each API endpoint may have a different limit for the number of items returned, and these limits may change without notice. (For more See this JIRA API Documentation for Pagination)

Configure Data Source in Gateway (ZappySys JSON / REST API Driver Example)

Configure Data Gateway User / Port

Now let's look at steps to configure Data Gateway after installation. We will also create a sample data source for ODATA API (i.e. JSON based REST API Service).

Assuming you have installed ZappySys ODBC PowerPack using default options (Which also enables Data Gateway Service)
Search "Gateway" in your start menu and click ZappySys Data Gateway

Open ZappySys Data Gateway
First make sure Gateway Service is running (Verify Start icon is disabled)
Also verify Port on General Tab

Port Number setting on ZappySys Data Gateway
Now go to Users tab. Click Add icon to add a new user. Check Is admin to give access to all data sources you add in future. If you don't check admin then you have to manually configure user permission for each data source.

Add Data Gateway User

Configure Data Source

After user is added, go to Data Sources tab. Click Add icon to create new data source. Select appropriate driver based on your API / File format. You can choose Generic ODBC option to read data from ODBC DSN or use Native Driver option.
```
NOTE: Whenever possible use native driver option for better performance / security and ease of use.
```
Add Gateway Data Source (Native JSON Driver)
Now it's time to connect with JIRA. Let's use JSON Driver Data source with the following URL. Change domain with your own domain.
```
https://{yourdomain}.atlassian.net/rest/api/2/search?startAt=0&maxResults=100
```
ODBC JSON Driver: Enter Request URL
It is time to configure the Other Settings. Select HTTP Connection Type from the Connection Type drop-down and Click on Click to Configure Link to Configure the Connection Manager.
[caption id="attachment_5596" align="alignnone" width="1245"] JIRA REST API Connection - Use ZappySys JSON Driver HTTP Connection[/caption] JIRA REST API Connection - Use ZappySys JSON Driver HTTP Connection
Now, we are going to set Pagination related configurations for JIRA Search request. Page number indicator must be startAt and maxResults value in URL must match with Increment By (i.e. 100)
ODBC JSON Driver - JIRA request Pagination Configurations Settings
After adding pagination parameters let's add JQL= search parameter to get specific data. Here we are searching the data by Project name and status. You can find more information about JQL on this link: Search JIRA like a boss with JQL Here is example JQL search query for Public JIRA site. Replace jira.atlassian.com and project JRASERVER with your own values.
```
https://jira.atlassian.com/rest/api/2/search?jql=project=JRASERVER
https://jira.atlassian.com/rest/api/2/search?jql=project=JRASERVER AND status=Open
https://jira.atlassian.com/rest/api/2/search?jql=project=JRASERVER AND status in (Open,Done)
https://jira.atlassian.com/rest/api/2/search?jql=project=JRASERVER AND status=Open order by lastViewed DESC
```
Search using JQL parameter
Click on Select Filter to set the Filter.
ODBC JSON Driver for JIRA API: Select Filter
You are done with the Configurations, Click on Test Connection to test the Connection.
ODBC JSON Driver - Test Connection
After that select the table from where you want to get data.
JSON Driver: Create Query of JSON data
Click on the Preview Data button to preview the data.
Preview issues table data
And that’s it, Now we will move forward to the next Phase.

Build your Own JQL Search queries for JIRA Issue

In previous sections we saw some basic JQL queries to search JIRA Issues using API call. Now lets look at how to build such queries using online UI (Think like a query builder). from JIRA.

JIRA has its own public Issue repository which you can browse any time by going here https://jira.atlassian.com/issues/
Now try to change filter settings for search criteria (E.g. change Project, Status etc)
You will notice when you make change your URL keeps changing too. From that URL you can extract ?jql=xxxxxxx part and use it as template for your own JIRA API search queries.

For more information check how to build JQL for JIRA REST API Call.

Create and configure new SSRS Reports (Using ODBC Connection)

Let’s create a new SSRS project.

In SSRS, in a new Reporting Service Project, add a new Data Source:
Add SSRS data source
Also, specify a name for the data source and in type select ODBC. Press Edit to select the ODBC connection for JSON and REST API and select the ZappySys Data Source name that we created before.
Add ODBC JSON Data Source
Now let’s create a new SSRS report which uses the API data source we created in the previous section.
Right-click on the Reports node and click Add New Report (Wizard mode)
Add new SSRS report
In New report, wizard window clicks on next button until you reach the step of query builder like below screen. Now Press the query builder button.
Note: You can also copy and paste the query directly into the Query string box which we created in the previous phase.
```
SELECT * FROM "issues"
```
Query builder
Now, press the Add table icon and go to Views. Select issues(DATA) and press Add.
SSRS Add Table
Now remove DATA. prefix from the query so that our query look’s like as below and Click Ok button:
```
SELECT issues.* FROM issues
```
Now press Next Button and select Tabular Report Type and go to next in the following option until the completing wizard and specify a name for the report and finally press the Finish button.
Now click on the Preview a report. It shows the report and exports the REST API data to Word, Excel, PowerPoint, etc:
SSRS Preview a Report
That’s all we have successfully imported the JIRA data in SSRS Reports (SQL Server Reporting Services).

Conclusion

In this blog, we have learned how to retrieve the data from JIRA using ODBC JSON Driver and then generated the SSRS Reports using those data with ODBC DSN in a very simple way. You can achieve many more functionalities with the use of ODBC PowerPack.

References

Finally, you can use the following links for more information about the use of JIRA API with our tools:

JIRA Documentation for API
Landing Page for ODBC JSON / REST API Driver, you can also find Tutorial Video here.
Help File: Documentation of JSON Driver.
Blog/articles link: https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/json-rest-api-driver/

The post Read JIRA data in SSRS Reports (SQL Server Reporting Services) appeared first on ZappySys Blog.

How to Import JIRA data in Power BI

ZappySys — Wed, 26 Dec 2018 07:53:23 +0000

Introduction

UPDATE: ZappySys has released a brand new API Connector for Jira Online which makes it much simpler to Read/Write Jira Data in Power BI compared to the steps listed in this article. You can still use steps from this article but if you are new to API or want to avoid learning curve with API then use newer approach.

In our previous blog we saw how to import rest api in Power BI. We recommend you go through that article first. In this article, we will learn how to Import JIRA data in Power BI. For that, we will create a successful HTML Connection then request for reading JIRA data and loading into Power BI

Power BI is a very popular Business Analytic tool used to get business information. It is very popular because it is easy to install, simple to learn and very intuitive. Also, REST API is very popular these days and we wanted to show you a way to integrate them in Power BI with the help of the ZappySys ODBC PowerPack. This software includes powerful drivers to query REST API URL, Local XML / JSON files and XML SOAP Web Service with simple SQL queries.

In nutshell, this post will focus on how to call JIRA API using Zappysys ODBC PowerPack. So let’s get started

Prerequisite

First, you will need to have our ZappySys ODBC Power Pack.
In the second place, you will need to have internet access.
On third place, Power BI Desktop (FREE) installed.
JIRA Atlassian account.

What is JIRA

JIRA is Atlassian’s popular project- and issue-tracking platform used by development and technical support teams to get more work done and faster.

It combines:

Getting Started

In order to start, we will show several examples. ZappySys ODBC PowerPack includes the ZappySys JSON Driver that will help you to call JIRA API, retrieve various data from JIRA and load that data in Power BI. To learn more about JIRA API check this help file.

Generate API Token

To access JIRA data using REST API call you need to generate an API Token in your Atlassian account. Follow the steps mentioned below to generate API Token:

you can directly login to the JIRA Atlassian Cloud site with your credentials, and redirect to the JIRA Atlassian Cloud Account's Profile Page for generating an API Token.
Click API tokens, then Create API token.

Generate API Token - Step - 1
Create API Token - Give Label name to your API token which you are generating now

Generate API Token - Step - 2
Use Copy to the clipboard, and paste the token into your or elsewhere:
Note: * For security reasons it isn't possible to view the token after closing the creation dialog; if necessary, create a new token. * You should store the token securely, just as for any password.

Generate API Token - Step - 3
If you forgot API token, then you can Revoke your old API token and create new API token OR you can create multiple API tokens too.

Generate API token - Step - 4

Now we have successfully generated an API token. Let´s start with an example. In this example, we will show how to read Tasks/Issues data from JIRA and store that data into a SQL table with SSIS.

About JIRA REST API

https://jira.atlassian.com/rest/api/2/search?jql=project=JRASERVER

Step-by-Step: Import JIRA data using REST API

Now once you have API token in the JIRA Atlassian Cloud website we can move to ODBC JSON Driver Configuration Part. For this example purpose you can use the following URL:

https://YOUR-DOMAIN.atlassian.net/rest/api/2/search?startAt=0&maxResults=100&jql=project=MYPROJECT

Create ODBC – Zappysys JSON Driver Data Source

To do this, we will first open the ODBC Data Source : Select 64-Bit for 64bit apps and 32bit if your app / process doesnt support 64bit.
Windows ODBC Data Sources
Click on Add button in User DSN Tab and then select JSON Driver from the driver list and click on Finish button to Create a new JSON Driver.
ODBC User DSN Tab: Add new Driver Screen
Now it's time to connect with JIRA. Let's use JSON Driver Data source with the following URL. Change domain with your own domain.
```
https://{yourdomain}.atlassian.net/rest/api/2/search?startAt=0&maxResults=100
```
ODBC JSON Driver: Enter Request URL
It is time to configure the Other Settings. Select HTTP Connection Type from the Connection Type drop-down and Click on Click to Configure Link to Configure the Connection Manager.
JIRA REST API Connection - Use ZappySys JSON Driver HTTP Connection
Now, we are going to set Pagination related configurations for JIRA Search request. Page number indicator must be startAt and maxResults value in URL must match with Increment By (i.e. 50)
ODBC JSON Driver - JIRA request Pagination Configurations Settings
After adding pagination parameters let's add JQL= search parameter to get specific data. Here we are searching the data by Project name and status. You can find more information about JQL on this link: Search JIRA like a boss with JQL Here is example JQL search query for Public JIRA site. Replace jira.atlassian.com and project JRASERVER with your own values.
```
https://jira.atlassian.com/rest/api/2/search?jql=project=JRASERVER
https://jira.atlassian.com/rest/api/2/search?jql=project=JRASERVER AND status=Open
https://jira.atlassian.com/rest/api/2/search?jql=project=JRASERVER AND status in (Open,Done)
https://jira.atlassian.com/rest/api/2/search?jql=project=JRASERVER AND status=Open order by lastViewed DESC
```
Search using JQL parameter
Click on Select Filter to set the Filter.
ODBC JSON Driver for JIRA API: Select Filter
You are done with the Configurations, Click on Test Connection to test the Connection.
ODBC JSON Driver - Test Connection
After that select the table from where you want to get data.
JSON Driver: Create Query of JSON data
Click on the Preview Data button to preview the data.
Preview issues table data
And that’s it, Now we will move forward to the next Phase.

Build your Own JQL Search queries for JIRA Issue

In previous sections we saw some basic JQL queries to search JIRA Issues using API call. Now lets look at how to build such queries using online UI (Think like a query builder). from JIRA.

JIRA has its own public Issue repository which you can browse any time by going here https://jira.atlassian.com/issues/
Now try to change filter settings for search criteria (E.g. change Project, Status etc)
You will notice when you make change your URL keeps changing too. From that URL you can extract ?jql=xxxxxxx part and use it as template for your own JIRA API search queries.

For more information check how to build JQL for JIRA REST API Call.

Load JIRA data into Power BI (Connect JIRA REST API)

In the previous section, we configured and added the ZappySys drivers in the ODBC Driver Administrator with information to connect to REST API. We queried the REST API data in JSON / XML format. Now let's look at how to import REST API data in Power BI using from ODBC connection.

Open Power BI Desktop and select the Get data option.

Get data using Power BI

Once Get data is clicked, Go to Other and select ODBC.

Import ODBC data source in Power BI
After that select ODBC DSN name from the DSN dropdown
Now it's time to import data. Basically, there are two modes to import data. Table Mode and Query Mode. Query mode is the most common but we will show you both ways.
Import using Power BI Query Mode: Select your DSN and click Advanced Option to enter custom SQL Query to Import your REST API data. You can use ODBC DSN Data sources Preview tool to generate SQL Query. For example you can enter query like below. If you are not sure use Query builder (Found on Driver Preview Window)
```
SELECT * FROM $
WITH(SRC='https://my-api-url')
```
Import JSON / REST API data in Power BI using SQL Query Mode
Import using Power BI Table Mode: If you don't specify SQL query under advanced options then you may get a chance to select Table name to import and Credentials. If Power BI asks for credentials then select Windows and connect.

Select credentials for ODBC Source
Once you enter select table, you can choose Select data to import and click OK (Below screeenshot is for the Table mode import when no SQL specified. You can pick desired table to import)

Select Table to import in Power BI
You can also display data in map using Map visualization like below. Press the map and check Address this option will display the addresses in a map.

See Address data in power bi map

Map visualization in Power BI
Now, let's display data in Table Format. Select in values more columns and select the data grid to visualize the data.

Select columns in Power BI
Once data is displayed, Right click on the data and select Show Data.

Show data in Power BI
The data will be displayed

read json information data

Conclusion

In this blog, we have learned how to retrieve the data from JIRA using ODBC JSON Driver and then Load those data in Power BI with ODBC DSN in a very simple way. You can achieve many more functionalities with the use of ODBC PowerPack.

References

Finally, you can use the following links for more information about the use of JIRA API with our tools:

JIRA Documentation for API
Landing Page for ODBC JSON / REST API Driver, you can also find Tutorial Video here.
Help File: Documentation of JSON Driver.
Blog/articles link: https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/json-rest-api-driver/

The post How to Import JIRA data in Power BI appeared first on ZappySys Blog.

Get Office 365 data in Power BI using Microsoft Graph API and ODBC

ZappySys — Wed, 19 Dec 2018 17:57:00 +0000

Introduction

In this article, we will get Office 365 data in Power BI using Microsoft Graph API and ODBC drivers. Specifically, we will get Excel file data in OneDrive, events from a Calendar and finally, a list in a note in OneNote and then load that data into a Power BI report. We will achieve that by creating an ODBC data source using ZappySys ODBC PowerPack and then querying corresponding Microsoft Graph API methods. ODBC PowerPack allows to query REST/SOAP APIs in an easy way and lets to view responses in a tabular format, which will be perfect to accomplish our goal.

Prerequisites

A Power BI account.
Wield basic Power BI skills.
Power BI Desktop installed.
ZappySys ODBC PowerPack installed.

Step By Step – Get Office 365 data in Power BI using Microsoft Graph API and ODBC

Register Application (OAuth2 App for Graph API)

The first step to access any Office 365 API / Graph API is to register an OAuth App in the Azure Portal. After following these steps, you will get the following two items to use in the next section:

Application Id
Application Secret

In this section, you will learn how to register a custom app in Microsoft Azure portal that will allow access to the OneDrive. So, let's get started -

Log into Microsoft Azure portal to register a custom app.
Register a new application by clicking New Registration link.
Provide the name of the custom app and who can access the app in the organization.
Go to the App overview and add a Redirect URL.
Click on "Add a Platform" under Platform Configuration section and then select "Web" under Web applications section to enter a Redirect URL.
```
https://zappysys.com/oauth
```
Create a Client Secret key which will be used to Authenticate the custom Azure app.
Note - Take a note of Client Secret, it will be required while configuring OAuth connection in the SSIS later
Add API Permissions.
Select following permissions from the Delegated Permissions section.
```
User.ReadBasic.All
Files.Read
offline_access
```
Take a note of Client ID, it will be required while configuring OAuth connection in the SSIS later.

Open ODBC Data Source Administrator

Hit Windows Start button and search for "odbc".
You will see two versions of ODBC Data Source administrator: 64-bit and 32-bit. Choose:
1. - 64-bit if your client application is a 64-bit program and you plan to create a System DSN.
  - 32-bit if your client application is a 32-bit program and you plan to create a System DSN.
  - 32-bit or 64-bit version if you plan to create a User DSN.
Windows ODBC Data Sources
Once you open, you should see a similar window:

ODBC Data Source Administrator

Set up an ODBC data source based on ZappySys JSON Driver

Add a data source

Go to System DSN and press Add to add a system DSN. If you use apps like Power BI which needs access only your User account then you can create User DSN.
Add a data source based on ZappySys JSON Driver:

Configure the data source

Once a window appears, give a name to the data source, e.g. System - JSON:
Then input https://graph.microsoft.com/v1.0/me in the URL field.
Select OAuth as the Connection Type.
Click Click to Configure button/link to open OAuth configuration window.
Select Custom as OAuth Provider.
Then input Client Id, Client Secret which we got when created the Graph API application.
Set https://login.microsoftonline.com/common/oauth2/v2.0/authorize as Authorization Url.
Set https://login.microsoftonline.com/common/oauth2/v2.0/token as Access Token Url.
In Scopes / Permissions section set these scopes, separated by a new line. E.g. offline_access, Files.Read.All, Calendars.Read, Notes.Read, User.Read
In Advanced tab configure Callback/Redirect Url and set it to the one we configured in Graph API app:
Finally, click Generate Token button to generate a token.
Click OK to close the window.

Set up an ODBC data source based on ZappySys XML Driver

Before adding another data source based on ZappySys XML Driver, copy the connection string of the data source we just created:
Close the window and then add a new data source, but this time based on ZappySys XML Driver.
Once a window is opened, give it a name, e.g. System - XML.
Then press Load connection string button to load the connection string.
Paste the copied connection string and replace JSON text to XML, so that you have { ZappySys XML Driver } in DRIVER property.
Close the window.

Create a Power BI report

Reading files from OneDrive (getting Excel data)

Let’s say you have Stock.xlsx stored in Sample Documents folder, OneDrive with this kind of data and you want to show it in a Power BI report:

To have that data in a Power BI report, perform these steps:

Open Power BI Desktop and create a new Power BI report.
Create a data source based on System – JSON DSN (since we get a JSON response):
Then in SQL statement area enter this query:
SELECT * FROM $ WITH(Src='https://graph.microsoft.com/v1.0/me/drive/root:/YOUR FOLDER/Stock.xlsx:/workbook/worksheets/SHEET NAME/usedRange' ,Filter='$.values[1:]' ,ArrayTransformType='TransformSimpleTwoDimensionalArray' ,ArrayTransColumnNameFilter='$.values[0][*]' )Using WITH clause lets us override the properties. You can generate the query above using the Query Builder:
In this step we are overriding Url, Filter and array transformation fields you configured in JSON DSN. The Src value overrides the URL. The Filter property enables us to select the data we want to see in the report. We use [1:] in $.values[1:] to skip the first row because the first row contains the headers and we want only the data. You can configure this option in the Filter Options tab (read more about JSON path).We also have to make the transformation on the JSON and turn JSON arrays into rows. To accomplish that, we override ArrayTransformType. With ArrayTransColumnNameFilter property and [0][*] text, we specify that the first array of parent array will be the headers of the columns. You can read more about transformations in Parse multi-dimensional JSON array article.
Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other OneDrive endpoints.
Click OK.
Drag a table into the report, select the columns and you should see a similar view:

Reading calendar entries

Now, let’s say you use a calendar in Office 365 and you want your appointments to be visible in a Power BI report:

To start, create a new Power BI report.
Then add a data source based on System – JSON DSN (again, we get the response in JSON):
In SQL statement section enter this query:
SELECT * FROM $ WITH(Src='https://graph.microsoft.com/v1.0/me/events?$select=subject,start,end,location' ,Filter='$.value[*]' ,Header='Prefer: outlook.timezone="FLE Standard Time"' )Here we also are overriding HTTP Header so that we get the event times not in UTC/GMT but for the specified timezone. It must be specified in Prefer: outlook.timezone=”{time-zone-string}” format (check an article on how to get a list of supported time zones).
Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other Calendar endpoints.
Click OK.
Drag a table control into the report, select several columns and you should see a similar view:

Reading notes from OneNote

Suppose, you have these notes in OneNote and you want “Groceries list” to be shown in a Power BI report:

First, we will need to find NOTE ID we are trying to get. To do that, open ODBC Administrator, find System – JSON data source and hit Configure.
Then go to the Preview tab, and input this SQL query and hit Preview Data:
SELECT * FROM $ WITH (Src='https://graph.microsoft.com/v1.0/me/onenote/pages' ,Filter='$.value[*]' )
Once you get a result set, find your note name and copy its ID:
Then create a new Power BI report.
Create a new data source based on System – XML DSN (we are using XML-based DSN because the response will be in HTML):
In SQL statement section enter this query (use the NOTE ID you retrieved in the previous step):
SELECT * FROM $ WITH(ElementsToTreatAsArray='div,p' ,Src='https://graph.microsoft.com/v1.0/me/onenote/pages/NOTE ID/content' ,Filter='$.html.body.div[0].p[1:]' )

Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other OneNote endpoints.
Click OK.
Drag and drop Multi-row card visual, select #text column and you should see a similar view:
We finished! Now you can publish your Power BI reports to Power BI service.

Working with Gateways in Power BI (Schedule Import)

If the data needs to be updated, it is necessary to create a gateway on-premises. In this new section, we will install a Power BI Gateway and in the next section schedule it to update the REST API information.

In the last section, we Published the report. Power BI may ask you to SIGN IN.

Sign in in Power BI
Select the Workspace and select Datasets
[caption id="attachment_10110" align="alignnone" width="702"] Go to workspace and dataset[/caption]
Go to workspace and dataset
Right-click the report and select Settings.

Specify settings for the report
The system will ask for a Gateway. Stay here.

Add Power BI Gateway

Use the following link to install a Data Gateway:

https://docs.microsoft.com/en-us/power-bi/service-gateway-onprem

Run the installer and press Next

Gateway installer
Select the option On-premises data gateway (recommended). This option allows access to multiple users and can be used by more applications than Power BI.

Choose Power BI option
The installer will show a warning message.

Warning message during installation
Select the path to install and check the I accept the terms.

Accept terms
Specify the email address to use the gateway.

Register email address
After entering the email, write the gateway name and a recovery key. Make sure to confirm the recovery key.

Enter recovery key

Manage gateways and configure the schedule

Once that the gateway is installed we will configure it and add the connection strings.

The next step is to go to manage gateway

Power BI - Manage Gateway Setting
In order to get the connection string, we will need the connection string of the ZappySys JSON Driver. In the first section of this post, we explained how to configure it. Press Copy Connection String

Connection properties
Once that the data is copied, add a New data Source. In Data Source Name, enter the Data Source Name of the ZappySys JSON driver in step 13 and in Data Source Type, select ODBC. In connection string copy and paste from the clipboard of the step 13 and press Add.
[caption id="attachment_10113" align="alignnone" width="720"] ZappySys connection properties in Power BI[/caption]
ZappySys connection properties in Power BI
Once added the gateway. You can see the schedule refresh to On and Add another time to add the time where you want to refresh the data.

Schedule gateway

Conclusion

Today we learned how to call Microsoft Graph API methods using ZappySys ODBC PowerPack and load response data into Power BI without any ETL procedures. Particularly, we used OneDrive, Calendar and OneNote Graph API endpoints to achieve that. We created two ODBC data sources based on JSON and XML ODBC Driver to query those endpoints, which let us read data directly in a Power BI report. Finally, we examined how to refresh data source on Power BI using on-premises Power BI gateway.

References

https://powerbi.microsoft.com/en-us/

https://apps.dev.microsoft.com/

https://developer.microsoft.com/en-us/graph/graph-explorer/

https://docs.microsoft.com/en-us/graph/overview

https://docs.microsoft.com/en-us/graph/onedrive-concept-overview

https://docs.microsoft.com/en-us/graph/outlook-calendar-concept-overview

https://docs.microsoft.com/en-us/graph/integrate-with-onenote

The post Get Office 365 data in Power BI using Microsoft Graph API and ODBC appeared first on ZappySys Blog.

Tableau Parameters – Fix Metadata / Dynamic SQL Columns Issue

ZappySys — Mon, 05 Nov 2018 17:17:42 +0000

Introduction

Many time you have to use Dynamic SQL in your Stored Procedure but some tools (e.g. Tableau) may reject it because metadata is not returned when sp_describe_first_result_set is called in SQL Prepare phase. You may see error like below in that case.

Metadata Prepare Call

exec sp_describe_first_result_set N'EXEC [dbo].[usp_GetInvoicesByCountry] @country=''Germany'''

Error due to Dynamic SQL

Msg 11514, Level 16, State 1, Procedure sp_describe_first_result_set, Line 1 [Batch Start Line 29]
The metadata could not be determined because statement 'EXECUTE (@sqlFull)' in procedure 'usp_GetInvoicesByCountry' contains dynamic SQL. Consider using the WITH RESULT SETS clause to explicitly describe the result set.

Here is example Stored Procedure which may throw above error. We will re-write same stored proc in later section.

Example of Dynamic SQL in Stored Procedure

--DROP PROC dbo.usp_GetInvoicesByCountry
--GO
/*
Purpose: Parameterize REST API call via SQL. Call ZappySys Drivers inside SQL Server.
*/
CREATE PROC dbo.usp_GetInvoicesByCountry
	@country varchar(100) 
AS 

DECLARE @sql varchar(max)
--//Escape single ticks carefully
SET @sql =  'SELECT OrderID,CustomerID,Country,Quantity FROM $
WITH (Src=''https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json&filter=Country eq '+ @country +'''
	 ,Filter=''$.value[*]''
	 ,DataFormat=''OData''
)'

DECLARE @sqlFull varchar(max)
SET @sqlFull='SELECT * FROM OPENQUERY( LS , ''' + REPLACE( @sql, '''', '''''' ) + ''' )'
PRINT @sqlFull --//For DEBUG purpose
EXECUTE (@sqlFull)

GO
-- Example call
EXEC dbo.usp_GetInvoicesByCountry @country='Germany'

How to Fix Metadata error for Dynamic SQL

To fix metadata error due to dynamic SQL you have add WITH RESULT SETS statement after EXECUTE call. This feature was added in SQL 2012 and later. There are two ways to describe Stored Procedure / using Dynamic SQL

Method-1

Add WITH RESULT SETS clause inside Stored Proc if you are using EXECUTE or sp_Executesql like below.

--DROP PROC dbo.usp_GetInvoicesByCountry
--GO
/*
Purpose: Parameterize REST API call via SQL. Call ZappySys Drivers inside SQL Server.
*/
ALTER PROC dbo.usp_GetInvoicesByCountry
	@country varchar(100) 
AS 

DECLARE @sql varchar(max)
--//Escape single ticks carefully
SET @sql =  'SELECT OrderID,CustomerID,Country,Quantity FROM $
WITH (Src=''https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json&filter=Country eq '+ @country +'''
	 ,Filter=''$.value[*]''
	 ,DataFormat=''OData''
)'

DECLARE @sqlFull varchar(max)
SET @sqlFull='SELECT * FROM OPENQUERY( LS , ''' + REPLACE( @sql, '''', '''''' ) + ''' )'
PRINT @sqlFull --//For DEBUG purpose

EXECUTE (@sqlFull)
WITH RESULT SETS
(
	(OrderID int,CustomerID varchar(100),Country varchar(100),Quantity int) --//describe first result
)
GO

Method-2

Another way to describe resultset is add WITH RESULT SETS statement right after you call Stored proc each time from any client tool (see below). This way you don’t have to touch Source stored Proc. First method is still preferred way.

EXEC [dbo].[usp_GetInvoicesByCountry] @country='Germany' WITH RESULT SETS ( (OrderID int,CustomerID varchar(100),Country varchar(100),Quantity int))

How to generate column names and field datatypes for WITH RESULT SETS

If you have many columns to describe then it becomes tedious to type 200 columns by hand. Lets see how to make it simple.

First step is you need to load your result into some static table.
```
SELECT * INTO _tmp FROM ( some sub query ) t
```

Once you have data we can use it few ways to get datatypes. Lets create a view to query metadata

CREATE VIEW dbo.vwDataTypes
AS
SELECT 
   c.name as ColumnName,
   t.name +
   CASE WHEN t.name IN ('char', 'varchar','nchar','nvarchar') THEN '('+
             CASE WHEN c.max_length=-1 THEN 'MAX'
                  ELSE CONVERT(VARCHAR(4),
                               CASE WHEN t.name IN ('nchar','nvarchar')
                               THEN  c.max_length/2 ELSE c.max_length END )
                  END +')'
          WHEN t.name IN ('decimal','numeric')
                  THEN '('+ CONVERT(VARCHAR(4),c.precision)+','
                          + CONVERT(VARCHAR(4),c.Scale)+')'
                  ELSE '' END
	      as FriendlyDataType,
   t.name as DataType,
   c.[object_id] as [TblObjectId],
   c.max_length MaxLengthBytes,
   c.precision as [Precision],
   c.scale  as [Scale],
   c.is_nullable as IsNullable,
   ISNULL(i.is_primary_key, 0) as IsPrimaryKey
FROM    
   sys.columns c
INNER JOIN 
   sys.types t ON c.user_type_id = t.user_type_id
LEFT OUTER JOIN 
   sys.index_columns ic ON ic.object_id = c.object_id AND ic.column_id = c.column_id
LEFT OUTER JOIN 
   sys.indexes i ON ic.object_id = i.object_id AND ic.index_id = i.index_id
--WHERE    c.object_id = OBJECT_ID('_tmp')
go

Now you can write below query to generate Comma separated list of columns names and datatypes

DECLARE @csv VARCHAR(MAX)
SELECT @csv = COALESCE(@csv+', ' ,'') + '[' + ColumnName + '] ' + FriendlyDataType
FROM vwDataTypes WHERE [TblObjectId]= OBJECT_ID('_tmp')
SELECT @csv

Example :

[CustomerID] nvarchar(10), [CustomerName] nvarchar(68), ......... [Freight] nvarchar(16)

Use result of above query and paste in WITH RESULT SETS as below

EXECUTE( @sqlFull )
WITH RESULT SETS
(
  (  your-comma-separated-list-goes-here )
)

The post Tableau Parameters – Fix Metadata / Dynamic SQL Columns Issue appeared first on ZappySys Blog.

Import REST API in Tableau – Read JSON, SOAP XML, CSV

ZappySys — Sun, 04 Nov 2018 22:52:06 +0000

Introduction

Tableau is one of the most popular Reporting / Visualization tool for BI / Data analytics. It comes with many out-of the box connectors to pull data from some popular data sources but still it seriously lakes capability to consume data from millions of other REST / SOAP data sources out there for which Tableau not going to create native connectors.

In this article we will cover how to import REST API in Tableau, We will also look at various examples to read from Files or any JSON / XML SOAP / CSV API in few clicks. We will use ZappySys API Drivers to read data from Web API or Local XML / JSON / CSV Files without doing any coding or ETL.

ZappySys API drivers are most advanced API drivers available in the market with many options to connect to virtually any REST / SOAP API data source. For demo purpose we will load ODATA API (JSON Format) using ZappySys JSON Driver. Concepts / steps listed in this articles are mostly same for other type of API formats too (e.g. XML API or CSV API).

Requirements

Before we get started for step by step tutorial, make sure following requirements are met.

You have installed Tableau Desktop
Make sure SQL Server Instance is installed somewhere. We will use Linked Server to access REST API. You can download FREE SQL Express Version (DB Server) and SSMS from here (Client).
Download and Install ZappySys ODBC PowerPack (API Drivers) – These drivers must be installed on Windows Machine (Can be different than Tableau Desktop / Server)

Step By Step – Loading REST API data in Tableau

Now let’s get started. To load REST API data in Tableau using ZappySys drivers basically there are two approaches.

Approach#1 You can use direct ODBC Connection to call ZappySys Drivers
Approach#2 Use SQL Server Connection in Tableau to send ZappySys Driver queries to ZappySys Data Gateway (using Linked Server OPENQUERY).

We recommend Approach#2 for any ZappySys driver use case in Tableau. Now you must be wondering why use approach#2 and not #1 ? Tableau doesn’t play well when it comes to 3rd party ODBC drivers because behind the scene Tableau generate complex SQL statements which may not be supported by ODBC driver its trying to call (e.g. JOIN Syntax, CAST functions, Sub queries). See some issues like this and this one. So if we use SQL Server to wrap those complex queries then it becomes much simpler because Tableau has very good support for SQL Server compared to 3rd party ODBC Drivers.

To access REST API in tableau using SQL Server Connection, you have to perform the following high level steps.

Make sure you have access to Microsoft SQL Server instance. If you don’t have one you can always use lightweight FREE SQL Express Edition (Download). This is free for Production use too.
Install ZappySys ODBC PowerPack (API Drivers)
Configure ZappySys Data Gateway (Create API connection) – We will see in depth, later in this article.
Create Linked Server in Microsoft SQL Server and connect to ZappySys Data Gateway
Import API data in Tableau using Microsoft SQL Server Connector by calling Linked Server SQL Queries

Assuming you have already done first step lets get started with remaining steps. You can also read this article for other example of calling SQL Server Queries via ZappySys Data Gateway.

Configure ZappySys Data Gateway

Configure Data Gateway User / Port

Now let's look at steps to configure Data Gateway after installation. We will also create a sample data source for ODATA API (i.e. JSON based REST API Service).

Assuming you have installed ZappySys ODBC PowerPack using default options (Which also enables Data Gateway Service)
Search "Gateway" in your start menu and click ZappySys Data Gateway

Open ZappySys Data Gateway
First make sure Gateway Service is running (Verify Start icon is disabled)
Also verify Port on General Tab

Port Number setting on ZappySys Data Gateway
Now go to Users tab. Click Add icon to add a new user. Check Is admin to give access to all data sources you add in future. If you don't check admin then you have to manually configure user permission for each data source.

Add Data Gateway User

Configure Data Source

After user is added, go to Data Sources tab. Click Add icon to create new data source. Select appropriate driver based on your API / File format. You can choose Generic ODBC option to read data from ODBC DSN or use Native Driver option.
```
NOTE: Whenever possible use native driver option for better performance / security and ease of use.
```
Add Gateway Data Source (Native JSON Driver)
Click on "Edit" under Data source and configure as per your need (e.g. Url, Connection, Request Method, Content Type, Body, Pagination etc.). For this demo we are going to pick simple JSON REST API which doesn't need any authentication. Enter following URL.
```
https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json
```
You can also view response structure and select default hierarchy (i.e. Filter) like below (Select Array Icon) for data extraction.

Configure JSON API Data source

Test SQL Query / Preview Data

Now go to Preview Tab. You can click Preview button to execute default query OR Select Table name from dropdown to generate SQL with column names.

JSON / REST API Driver Query Preview / Query Examples (Read REST API or JSON Files)
You can also click Query Builder to generate SQL using different options in WITH clause. ANy setting you specify in WITH clause will override UI settings we applied in previous steps.

Using SQL Query Builder (For Files or REST / SOAP API - JSON / XML / CSV Format)
There is another useful option for code generation. Select your Language and quickly copy code snippet. See below Example of XML Driver Query to call SOAP API.

Generate Example Code for ZappySys Driver
Click OK to Close Data Source UI
Once data source is tested and configured you can click Save button in the Gateway UI toolbar and click Yes for Restart Service.

Create Linked Server in SQL Server

Once REST API Data source is configured in ZappySys Data Gateway, we can move forward to creating Linked Server part. At this point, you must be wondering why I have to use SQL Server to call ZappySys ODBC Driver? Why can’t I directly use ZappySys ODBC Driver in Tableau?

Technically, you can use Generic ODBC Connector in Tableau to call ZappySys ODBC Drivers for JSON API / XML API or CSV API but there are many problems with it.

Tableau Generic ODBC Connection option generates many unsupported SQL Syntax (e.g. Temp tables, Subqueries, date functions) which may not be supported by most non-relational ODBC drivers (e.g. ZappySys API drivers). Using SQL Server Linked Server we can bridge between ZappySys Drivers and Tableau to access any REST API. So let’s see how to do this.

Once you configured data source in Gateway, we can now setup Linked Server in SQL Server to query API data.

Assuming you have installed SQL Server and SSMS. If not then get both for FREE from here: Get SQL Server Express and Get SSMS
Open SSMS and connect to SQL Server.
Go to Root > Server Objects > Linked Servers node. Right click and click New Linked Server...

Add Linked Server in SQL Server
Now enter linked server name, select Provider as SQL Native Client
Enter data source as GatewayServerName,PORT_NUMBER where server name is where ZappySys Gateway is running (Can be same as SQL Server machine or remote machine). Default PORT_NUMBER is 5000 but confirm on Data gateway > General tab incase its different.
Enter Catalog Name. This must match name from Data gateway Data sources grid > Name column

Configure Linked Server Provider, Catalog, Server, Port for ZappySys Data Gateway Connection
INFO:
- For SQL Server 2012, 2014, 2016, 2017, and 2019, use the SQL Server Native Client 11.0 as the Provider.
- For SQL Server 2022 or higher, use the Microsoft OLE DB Driver for SQL Server as the Provider.
Click on Security Tab and select last option "Be made using this security context". Enter your gateway user account here.
Optional: Under the Server Options Tab, Enable RPC and RPC Out and Disable Promotion of Distributed Transactions(MSDTC).

RPC and MSDTC Settings

You need to enable RPC Out if you plan to use EXEC(...) AT [MY_LINKED_SERVER_NAME] rather than OPENQUERY.
If don't enabled it, you will encounter the 'Server "MY_LINKED_SERVER_NAME" is not configured for RPC' error.

Query Example: EXEC('Select * from Products') AT [MY_LINKED_SERVER_NAME]

If you plan to use 'INSERT INTO...EXEC(....) AT [MY_LINKED_SERVER_NAME]' in that case you need to Disable Promotion of Distributed Transactions(MSDTC).
If don't disabled it, you will encounter the 'The operation could not be performed because OLE DB provider "SQLNCLI11/MSOLEDBSQL" for linked server "MY_LINKED_SERVER_NAME" was unable to begin a distributed transaction.' error.

Query Example:
```
Insert Into dbo.Products 
EXEC('Select * from Products') AT [MY_LINKED_SERVER_NAME]
```
Click OK to save Linked Server

In SSMS execute below SQL query to test your connectivity.

SELECT * FROM OPENQUERY( MY_LINKED_SERVER_NAME, 'SELECT * FROM $')

--OR--

SELECT * FROM OPENQUERY( MY_LINKED_SERVER_NAME, 
'SELECT * FROM $
 WITH (Src=''https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json''
 ,Filter=''$.value[*]''
 ,DataFormat=''OData''
)');

Here is the preview after you run some REST API query in SQL Server. Notice that you can override default configuration by supplying many parameters in WITH clause (second query example in screenshot).

SSMS Output - Query REST API via Linked Server OPENQUERY statement (Connect to ZappySys Data Gateway)
You can wrap your queries inside View or wrap inside Stored procedure to parameterize. Here is an example of create view which calls REST API queries. Below View can be consumed like a normal table from any Tools or Programming Language which supports connectivity to SQL Server.
```
CREATE VIEW dbo.vwApiInvoices 
AS 
/*Call REST API inside SQL Server View*/
SELECT * FROM OPENQUERY( LS , 
'SELECT * FROM $
WITH (Src=''https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json''
	 ,Filter=''$.value[*]''
	 ,DataFormat=''OData''
)');

GO
```
Notice in above approach if you parameterize Stored Procedure then check this article to understand Dynamic Metadata.
That's it. We are now ready to move forward with more interesting things in next section.

Import REST API data in Tableau using Microsoft SQL Server Connector

Now we are ready to call REST API in Tableau. Perform following steps to Connect REST API to Tableau.

Open Tableau Desktop and click File > New
To create new Connection click More > Microsoft SQL Server > Enter your credentials to connect to SQL Server.
Connect to REST API in Tableau using Microsoft SQL Server (Linked Server OPENQUERY approach)
Once connection is created for SQL Server we can read REST API data 3 different ways
1. Query View which contains OPENQUERY to Linked Server for REST API data
2. Use direct SQL Query using OPENQUERY
3. Use Stored Procedure (Mostly useful to parameterize calls
See below example to pull data from REST API in Tableau using SQL View approach
Create new Data Source in Tableau to Query REST API or file data (JSON / XML / CSV )

Here is how to Create Data Source using direct SQL Query. You can enter following Example Query.

SELECT * FROM OPENQUERY( YOUR_LINKED_SERVER , 
'SELECT * FROM $
WITH (Src=''https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json''
	 ,Filter=''$.value[*]''
)');

Create Tableau REST API Datasource using direct SQL Query

Once your data sources are created you can click on Sheet1 and drag fields to create visualizations for Tableau Dashboard.
Create Tableau Dashboard from REST API data (JSON / XML / CSV)

Passing Parameters to REST API calls in Tableau (Dynamic SQL)

Now let’s look at scenario where you have to pass parameters to build Dynamic Dashboard. You can try to insert Parameters in your Direct SQL when you build Dynamic SQL but we found some issues with that so we are going to suggest Stored Procedure approach. For more information on Known issue on Dynamic Metadata Check this post.

First lets create a stored procedure in SQL Server for Parameter Example. Notice how we added WITH RESULT SETS in the code to describe metadata.

--DROP PROC dbo.usp_GetInvoicesByCountry
--GO
/*
Purpose: Parameterize REST API call via SQL. Call ZappySys Drivers inside SQL Server.
*/
CREATE PROC dbo.usp_GetInvoicesByCountry
	@country varchar(100) 
AS 

DECLARE @sql varchar(max)
--//Escape single ticks carefully
SET @sql =  'SELECT OrderID,CustomerID,Country,Quantity FROM $
WITH (Src=''https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json&filter=Country eq '+ @country +'''
	 ,Filter=''$.value[*]''
	 ,DataFormat=''OData''
)'

DECLARE @sqlFull varchar(max)
SET @sqlFull='SELECT * FROM OPENQUERY( LS , ''' + REPLACE( @sql, '''', '''''' ) + ''' )'
PRINT @sqlFull --//For DEBUG purpose

EXECUTE (@sqlFull) 
WITH RESULT SETS ( 
 (OrderID int,CustomerID varchar(100),Country varchar(100),Quantity int) --//describe first result. If you don't do this then wont work in Tableau 
)
GO
-- Example call
EXEC dbo.usp_GetInvoicesByCountry @country='Germany'

Once you create a stored procedure go to Tableau datasource and select Database which contains the stored procedure we just created.
Now find your stored proc and drag it on the datasource pane. You will see parameters UI as below. You can create new parameter – Select New Parameter under Value Column.
Parameterize Tableau REST API datasource (Stored Procedure Parameters)
Thats it now you can reuse your parameterized datasource anywhere in Dashboard.
If you have need to select Parameters from predefined values rather than free text then edit your parameter and select List option. Define values you like to select from as below.
Allow Tableau Parameter selection from multiple values
When you create Tableau Dashboard you will see Parameter dropdown (If you selected List) elase you may see Textbox to enter custom value.
Tableau Dashboard Example – REST API Source with Parameterized Datasource

Read data from JSON Files in Tableau (Single or Multiple)

So far we have seen how to read data from REST API but there will be a time to read data from JSON Files (Single or Multiple). Also files may be compressed (GZip or Zip). Regardless no worries if you are using ZappySys drivers. See below sample query you can submit via SQL Linked Server to read JSON files stored on network share.

Single File Example

SELECT * FROM OPENQUERY([MY_LINKED_SERVER]
, 'SELECT * FROM $
WITH(
   Src=''c:\ssis\large_2018.json''
  ,Filter=''$.Branches[*]''
)')

Multiple Files Example

SELECT * FROM OPENQUERY([MY_LINKED_SERVER]
, 'SELECT * FROM $
WITH(
   Src=''c:\ssis\large_*.json''
  ,Filter=''$.Branches[*]''
)')

Read Compressed Files (GZip / Zip Format)

SELECT * FROM OPENQUERY([MY_LINKED_SERVER]
, 'SELECT * FROM $
WITH(
   Src=''c:\ssis\large_*.json.gz''
  --Src=''https://zappysys.com/downloads/files/test/large_file_10k.json.gz''
  ,Filter=''$.Branches[*]''
  ,FileCompressionType=''GZip''  --OR-- Zip --OR-- None
)')

Read CSV Files or URL in Tableau

If you like to read data from CSV files or URL then you can use ZappySys CSV Driver same way.

Here is sample query for CSV data.

Reading Compressed CSV Files in Tableau

SELECT * FROM OPENQUERY([MY_API_SERVICE]
, 'SELECT * FROM $
WITH(
  Src=''https://zappysys.com/downloads/files/test/customerdata.csv.gz''
  ,FileCompressionType=''GZip''  --OR-- Zip --OR-- None
  ,ColumnDelimiter=''|''
  ,HasColumnHeaderRow=''False''
)')

Read multiple files (TAB delimited) in Tableau

SELECT * FROM OPENQUERY([MY_API_SERVICE]
, 'SELECT * FROM $
WITH(
  Src=''''c:\ssis\customerdata_*.csv''''
  ,ColumnDelimiter=''{tab}'' --or use \t for tab delimited files
  ,HasColumnHeaderRow=''True''
)')

Read XML Files or API in Tableau

If you like to read data from XML files or URL then you can use ZappySys XML Driver same way. See below example to read some data from XML Web API. We have also passed credentials along with it for demo purpose.

SELECT * FROM OPENQUERY([MY_API_SERVICE]
, 'SELECT * FROM $
WITH(
	 ElementsToTreatAsArray=''item,slide''
	,Src=''http://httpbin.org/xml''
	,DataConnectionType=''HTTP''
	,UserName=''user1''
	,CredentialType=''Basic''
	,Password=''P@11s#''
	,Filter=''$.slideshow.slide[*]''
)')

Read local XML files

SELECT * FROM OPENQUERY([MY_API_SERVICE]
, 'SELECT * FROM $ WITH (
 SRC=''C:\Data\CustomerData.xml''
, Filter=''$.Root.Row[*]''
, ElementsToTreatAsArray=''Row''
)')

Call POST REST API in Tableau

There will be a time when you like to submit API request as POST rather than GET. See below example how to write POST API Request to send Body along with Custom Headers and Method.

POST Body, Set Method, Headers

SELECT * FROM OPENQUERY([MY_API_SERVICE]
, 'SELECT * FROM $
WITH 
(METHOD=''POST'', HEADER=''Content-Type: text/plain || x-hdr1: AAA''
,SRC=''http://httpbin.org/post''
,BODY=''
{
    id:1,
    notes:"Line1\r\nLine2"
}''
)')

Call SOAP API using XML Driver (POST data) – Token based Authentication

Here is another interesting example to call XML SOAP API. It first authenticates and get Token from SOAP Response. Then use that token for API call. To read more about Dynamic Token Method check this article.

SELECT * FROM OPENQUERY([MY_API_SERVICE]
, 'SELECT * FROM $
WITH(
	 ElementsToTreatAsArray=''urn:Row''
	,Src=''https://zappysys.com/downloads/files/test/soap-getdata.aspx''
	,DataConnectionType=''HTTP''
	,AuthScheme=''{none}''
	,TokenUrl=''https://zappysys.com/downloads/files/test/soap-login.aspx''
	,TokenRequestData=''

   
      
         [$userid$]
         [$password$]
      
   
''

	,TokenRequestMethod=''POST''
	,TokenResponseContentFilter=''//*[local-name() = ''''sessionid'''']''
	,TokenRequestHeaders=''Content-Type:text/xml|Accept:*/*|Cache-Control:no-cache''
	,TokenResponseContentType=''Xml''
	,UserName=''MyUser001''
    ,Password=''P@$$w0rdAAc12''
	,CredentialType=''TokenDynamic''
	,Filter=''$.soapenv:Envelope.soapenv:Body.urn:Row[*]''
	,RequestData=''

   
      [$token$]
   
''

	,Header=''Content-Type: text/xml;charset=UTF-8 || SOAPAction: "https://zappysys.com/soap-getdata" || Accept: */* || Cache-Control: no-cache''
	,RequestMethod=''POST''
)')

REST API / XML SOAP Pagination Settings for Tableau

https://zappysys.com/blog/ssis-rest-api-looping-until-no-more-pages-found/

Paginate by Response Attribute

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

Paginate by URL Parameter (Loop until certain StatusCode)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

Paginate by URL Path (Loop until no record)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

Paginate by Header Link (RFC 5988)

API like GitHub / Wordpress use Next link in Headers (RFC 5988)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

REST API / SOAP Web Service Connection Settings for Tableau

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

HTTP
OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

SOAP WSS (when accessing a SOAP WebService)
Static Token / API Key (when need to pass an API key in HTTP header)
Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
JWT Token (As per RFC 7519)

OAuth

Other settings for REST API / SOAP XML Call in Tableau

There are few settings you can coder while calling Web API

API Limit / Throttling

2D Array Transformation

If you are using JSON or XML API Driver then possible you may have to transform your data using 2D array transformation feature. Check this link for more information.

REST API / XML SOAP Performance Tips for Tableau

While calling APIs you may face some performance issues. There are a few tips you can consider to speed up things.

Use Server-side filtering if possible in URL or Body Parameters

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

Avoid Special features in SQL Query (e.g. WHERE, Group By, Order By)

WHERE Clause
GROUP BY Clause
HAVING Clause
ORDER BY
FUNCTIONS (e.g. Math, String, DateTime, Regex... )

LIMIT clause does not trigger client-side processing.

Consider using pre-generated Metadata / Cache File

Use META option in WITH Clause to use static metadata (Pre-Generated)There are two more options to speedup query processing time. Check this article for details.

select * from value WITH( meta='c:\temp\meta.txt' )
--OR--
select * from value WITH( meta='my-meta-name' )
--OR--
select * from value WITH( meta='[ {"Name": "col1", "Type": "String", Length: 100}, {"Name": "col2", "Type": "Int32"} ...... ]' )

Enable Data Caching Options (Found on Property Grid > Advanced Mode Only )

Consider using Metadata / Data Caching Option

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

Use --FAST Option to enable Stream Mode

ZappySys JSON / XML drivers support --FAST suffix for Filter. By using this suffix after Filter driver enables Stream Mode, Read this article to understand how this works.

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Calling SOAP Web Service in Tableau

What is SOAP Web Service?

Getting WSDL file or URL
Knowing exact Web Service URL

What is WSDL

Example SQL Query for SOAP API call using ZappySys XML Driver

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='

   
   
      
         
         UnitedStates
      
   
'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Using SoapUI to test SOAP API call / Create Request Body XML

Open SoapUI and click SOAP button to create new SOAP Project
Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL
```
http://www.dneonline.com/calculator.asmx?wsdl
```
Create new SOAP API Project in SoapUI tool for SOAP API Testing
Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
  
  Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
  Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
Now you can test your request first Double-click on the request node to open request editor.
Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
Create SOAP Request XML (With Optional Parameters)
Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
Goto System DSN Tab (or User DSN which is not used by Service account)
Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
(This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

Conclusion

Tableau has no out of the box features to consume many Web API / JSON / XML data sources out there. ZappySys has developed most innovative drivers which can connect to virtually any REST API / SOAP data sources in Tableau. These drivers also capable reading local files in JSON / CSV or XML format. Feel free to download ZappySys Drivers here and try yourself . You can always contact ZappySys Support Team here if you need any API integration help.

The post Import REST API in Tableau – Read JSON, SOAP XML, CSV appeared first on ZappySys Blog.

How to call REST API in MicroStrategy (JSON / XML SOAP)

ZappySys Team — Sat, 13 Oct 2018 21:19:14 +0000

Introduction

In this new article, we will show how to call REST API in MicroStrategy. MicroStrategy is a Business Intelligence company that provides powerful software to create nice charts, reports to take decisions. It started in 1993 and now it is one of the most popular Business Intelligence tools in the world because it is simple, intuitive and friendly. Also, It can connect to almost any database. We will use ZappySys ODBC PowerPack. Zappysys ODBC is a pretty nice software used to connect to REST API. ZappySys ODBC is an excellent plugin for MicroStrategy, because, with this plugin, you can connect to almost any application that supports REST API.

Requirements

First of all, we will need MicroStrategy Desktop installed.
Secondly, you will require ZappySys PowerPack for ODBC installed.

Getting started

First of all, we will need to configure the ODBC ZappySys to connect to REST API. Once the ODBC connection is created, we will invoke the ODBC connection in Microstrategy.

Configuring the ODBC Zappysys

Search “odbc” in start menu and select ODBC Data sources (64-bit)
First of all, we will open the ODBC Data Source Administrator in the System DSN and press Add:
ODBC Connect to REST API

Use 64 bits if your machine supports it.
Secondly, add a new ZappySys JSON Driver:
List of drivers supported
Also, we will add the following URL to the Simple View:
```
https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json
```
ODBC configuration to connect to REST API

Configuring MicroStrategy

In MicroStrategy Desktop, press the New Dossier button:
Create a new dossier in MicroStrategy
Secondly, in the Dossier, press the New Data button:
Add new data in Microstrategy
Also, in Data Sources select Databases:

Select the option to connect to Databases in Microstrategy
In addition, select the option to Type a Query:
Create a query to get Database data
Next, in Data Sources press the + icon:
Add Data Source in MicroStrategy
Finally, in Data Source, select DSN Data Sources and select the data source created in the previous section using the ZappySys ODBC Driver. In version, select Generic DBMS and write any username and password:
Select DSN, version

Call REST API in MicroStrategy

You can write different queries using the ZappySys ODBC Driver. The driver supports SQL queries to REST API data including WHERE clauses, GROUP BY, HAVING, ORDER BY, LIMIT WITH:
```
SELECT ShipName FROM value  
WHERE ShipName="Around the Horn"
```
Query to REST API using MicroStrategy
Secondly, you can connect online (Live) or use an in-memory Dataset:
MicroStrategy connect Live
Finally, you can use the data for your MicroStrategy reports:
MicroStrategy report created

Invoking a JSON file in MicroStrategy using ODBC

With ZappySys PowerPack, you can not only CALL REST API in Microstrategy, but you can also create queries to plain JSON files stored locally or in a web address. We will use the following JSON file to test:

First of all, in order to connect, you will need to specify on the ZappySys JSON Driver the path of the local JSON file:
Configure a JSON file connection with ODBC
Secondly, you can create queries in MicroStrategy in the same way that we did in the Call REST API in MicroStrategy section:
Query to JSON file in MicroStrategy

Call REST API in MicroStrategy to get Gmail information

It is also possible to invoke and query Gmail data using REST API in MicroStrategy using ZappySys ODBC PowerPack. With the ODBC PowerPack, you can write SQL queries and call REST API in MicroStrategy using OAuth or HTTP
connections.

First of all, in the ZappySys ODBC JSON driver, write the following URL:
```
https://www.googleapis.com/gmail/v1/users/me/messages/
```
Secondly, in connection type select OAuth Provider.
Also, in the OAuth Parameter in OAuth Provider, select Google and in scope https://mail.google.com and press generate token button and specify your credentials:
Configuration to connect MicroStrategy with Gmail
Additionally, in Zappysys you can write queries in MicroStrategy as we did with other REST API URLs:
```
SELECT * FROM messages
```
Query to Gmail information in MicroStrategy

Problems with the boolean data type in MicroStrategy

In addition, there is known issue related to boolean data in Microstrategy:

Error in accessing information from the external data source. Database Instance: (DataImport) Error Fetch Failed Error type: Invalid operation. Data of a string cell was truncated.

https://community.microstrategy.com/s/article/KB20514-How-to-map-a-boolean-datatype-as-integer-with-PostgreSQL

We have a workaround for this situation.

First of all, you will need to change the metadata. The following article shows how to change the metadata:https://zappysys.com/blog/metadata-odbc-api-drivers-query/Also, in order to fix, save your current metadata:
Driver ODBC metadata
Additionally, modify your data from Boolean to String:
MicroStrategy boolean to string metadata

REST API / SOAP API Pagination Settings for MicroStrategy

https://zappysys.com/blog/ssis-rest-api-looping-until-no-more-pages-found/

Paginate by Response Attribute

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

Paginate by URL Parameter (Loop until certain StatusCode)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

Paginate by URL Path (Loop until no record)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

Paginate by Header Link (RFC 5988)

API like GitHub / Wordpress use Next link in Headers (RFC 5988)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

REST API / SOAP Web Service Connection Settings for MicroStrategy

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

HTTP
OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

SOAP WSS (when accessing a SOAP WebService)
Static Token / API Key (when need to pass an API key in HTTP header)
Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
JWT Token (As per RFC 7519)

OAuth

Other settings for REST API / SOAP XML Call in MicroStrategy

There are few settings you can coder while calling Web API

API Limit / Throttling

2D Array Transformation

If you are using JSON or XML API Driver then possible you may have to transform your data using 2D array transformation feature. Check this link for more information.

REST API / XML SOAP Performance Tips for MicroStrategy

While calling APIs you may face some performance issues. There are a few tips you can consider to speed up things.

Use Server-side filtering if possible in URL or Body Parameters

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

Avoid Special features in SQL Query (e.g. WHERE, Group By, Order By)

WHERE Clause
GROUP BY Clause
HAVING Clause
ORDER BY
FUNCTIONS (e.g. Math, String, DateTime, Regex... )

LIMIT clause does not trigger client-side processing.

Consider using pre-generated Metadata / Cache File

Use META option in WITH Clause to use static metadata (Pre-Generated)There are two more options to speedup query processing time. Check this article for details.

select * from value WITH( meta='c:\temp\meta.txt' )
--OR--
select * from value WITH( meta='my-meta-name' )
--OR--
select * from value WITH( meta='[ {"Name": "col1", "Type": "String", Length: 100}, {"Name": "col2", "Type": "Int32"} ...... ]' )

Enable Data Caching Options (Found on Property Grid > Advanced Mode Only )

Consider using Metadata / Data Caching Option

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

Use --FAST Option to enable Stream Mode

ZappySys JSON / XML drivers support --FAST suffix for Filter. By using this suffix after Filter driver enables Stream Mode, Read this article to understand how this works.

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Calling SOAP Web Service in MicroStrategy

What is SOAP Web Service?

Getting WSDL file or URL
Knowing exact Web Service URL

What is WSDL

Example SQL Query for SOAP API call using ZappySys XML Driver

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='

   
   
      
         
         UnitedStates
      
   
'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Using SoapUI to test SOAP API call / Create Request Body XML

Open SoapUI and click SOAP button to create new SOAP Project
Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL
```
http://www.dneonline.com/calculator.asmx?wsdl
```
Create new SOAP API Project in SoapUI tool for SOAP API Testing
Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
  
  Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
  Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
Now you can test your request first Double-click on the request node to open request editor.
Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
Create SOAP Request XML (With Optional Parameters)
Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
Goto System DSN Tab (or User DSN which is not used by Service account)
Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
(This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

Conclusion

To conclude, we can say that the process to call REST API in MicroStrategy using ZappySys ODBC PowerPack is straightforward. With the REST API and Web service support, Microstrategy can connect to almost any source. If you liked this article and you want to try our product, you can download it here.

References

Finally, if you need more information about the process to call REST API in MicroStrategy, refer to the following links:

The post How to call REST API in MicroStrategy (JSON / XML SOAP) appeared first on ZappySys Blog.