ZappySys provides high-performance drag and drop connectors for Amazon AWS Integration. In this post, we will see How to Read / Write / Delete Amazon SQS Queue data in SSIS using ZS Amazon SQS components. ZappySys developed many AWS related components but in this article, we will look at 2 components for Amazon SQS Queue integration Scenarios (Read and Write). We will discuss how to use SSIS Amazon SQS Source and SSIS Amazon SQS Destination. Amazon SQS Source can be used to read messages from Amazon AWS SQS (Simple Queue Storage) and Amazon SQS Destination can be used to write/delete data from Amazon SQS (Simple Queue Storage) using SSIS.

Prerequisite

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

• First, you will need to have SSIS installed.
• Secondly, make sure to have SSDT.
• Thirdly, you have obtained Amazon SQS Credentials.
• Finally, do not forget to install ZappySys SSIS PowerPack.

Component Mention in this article

	Amazon SQS Source Connector
	Amazon SQS Destination Connector

What is Amazon SQS (Simple Queue Service)

Before we talk How to Read / Write / Delete Amazon SQS Queue data in SSIS lets talk What is Amazon SQS (Simple Queue Service) or sometimes referred to as AWS SQS. Amazon SQS offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues and costs allocation tags. It provides a generic web services API and it can be accessed by any programming language that the AWS SDK supports.

SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

Getting Started

In order to start, we will show several examples. ZappySys includes an Amazon SQS Source and Amazon SQS Destination, that will help you How to Read / Write / Delete Amazon SQS Queue data in SSIS. You can connect to your Amazon SQS account by entering credentials.

Amazon SQS Source can be used to read messages from Amazon AWS SQS (Simple Queue Storage) using SSIS. Read Amazon SQS Messages (AWS SQS) in bulk. Amazon SQS Destination can be used to write/delete data from Amazon SQS (Simple Queue Storage) using SSIS. Write messages to Amazon SQS (AWS SQS) in bulk. Support for Visibility Timeout (Hide messages from others for certain time after you read), data preview and max rows, reading message attributes in separate output columns and adjusting UTC DateTime in local time.

Write Data using Amazon SQS Destination in SSIS

Overview

In this tutorial, you will learn how to write data in Amazon SQS Queue storage. You can use ZS DummyData Source for practice it or select suitable ZS Source Adapters. In this case its from ZS DummyData Source. So let’s start with an example, In this section, you will learn how to write messages into Amazon SQS Queue. For this example, we are assuming that you have already SQS Queue Created.

Create Amazon SQS Connection

1. Firstly, In order to connect to Amazon SQS from SSIS, you will need Access Key and Secret Key (Login to AWS portal and Create Access/Secret Key with SQS service read access). If you are not sure about this ask your SysAdmin or responsible person to provide that to you. Your key will look something like this: (This is just an example key which may differ in your case).
   
   

   AccessKey: AKIAIOSFODNN7EXAMPLE
   SecretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

   Click here to read more about how to get your AccessKey and Secret Key.
2. Secondly, Once you have Account Key (it’s like UserID) and Secret Key (its like password) you may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
3. Right-click in Connection Manager Panel and Select [New Connection] menu item.
   

   SSIS Create AWS Storage Connection

4. Now Select ZS-AWS-STORAGE connection type.
   

   Select ZS-AWS-STORAGE Connection

5. Let’s configure the Amazon SQS connection. Set Storage Service to SQS and Credential Type to Key-based access control. In the General tab enter your Amazon SQS Credentials and Set Default region as per you have.
   

   SSIS AWS Storage Connection – Configure

6. In the last, click on Test Connection and OK button to Save connection configure setting UI.

Write Data into Amazon SQS

1. Once you have created the Amazon SQS Connection successfully, let’s drag and Drop Data Flow Task from SSIS Toolbox on the control flow designer surface.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Double click on the Data Flow Task to see Data Flow Designer surface.
3. Here, from the SSIS Toolbox drag and drop ZS DummyData Source on the data flow designer surface.
   

   SSIS ZS Dummy Data Source – Drag and Drop

4. Double click on Dummy Data Source to configure it.
   

   SSIS ZS Dummy Data Source – Configure

5. Click on OK button to save configure setting UI.
6. From the SSIS toolbox drag and drop Amazon SQS Destination on the Data flow designer surface and single click on the Dummy Data Source, Once you see the blue arrow from the source. connect it to Amazon SQS Destination.
   

   SSIS ZS Amazon SQS Destination

7. Now double click on Amazon SQS Destination to configure it.
8. In the [Connection Manager] tab select SQS connection manager we have created before it.
   

   SSIS Amazon SQS Destination – Configure Connection Manager

9. Click on [Component Properties] tab and Select Target Queue from the dropdown list.
   

   SSIS ZS Amazon SQS Destination – Configure Component Properties

10. Click on [Mappings] tab and verify mapping. Map document column from upstream to the MessageText target column.
    

    SSIS ZS Amazon SQS Destination – Column Mappings

11. In the last, click on the OK button to save configure setting UI.
12. After all, Execute the Package and verify target data by going to your AWS console under SQS service section.
    

    Run or Execute SSIS Package

    

    Verify Amazon SQS Destination

Read Data using Amazon SQS Source in SSIS

Overview

In this tutorial, you will learn how to read data from Amazon SQS storage in SSIS using ZS SSIS Amazon SQS Source Adapter. In this SSIS we will read data. So let’s begin it.

Read Data from Amazon SQS

1. Firstly, create a new SSIS package. In BIDS/SSDT create a new SSIS package.
2. From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
   

   SSIS Data Flow Task – Drag and Drop

3. Double click on the DataFlow task to see DataFlow designer surface.
4. From the SSIS toolbox drag and drop ZS Amazon SQS Source on the dataflow designer surface.
   
5. Double click Amazon SQS Source to configure it.
6. Select Amazon SQS Connection we have created before and select queue from the dropdown list.
   

   SSIS Amazon SQS Source – Configure

7.  Click on Preview button to see Data Preview and Click on  OK button to save settings.
8. From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface or you can use any of our suitable ZS Destination Adapter.
   

   SSIS ZS Trash Destination – Drag and Drop

9. Now single click on the Amazon SQS Source once you see the blue arrow from the source … connect it to Trash Destination.
10. Double click on ZS Trash Destination to Configure it.
    

    SSIS ZS Trash Destination – Configure

11. Click on OK button to save Trash Destination configure setting UI.
12. After all, right-click on the path and Add Data Viewer and Execute the package and verify source data in the data viewer.
    

    Run or Execute The SSIS Package

13. After messages read from SQS Queue, the same message cannot be read again until VisibilityTimeout. By default Queue, visibility timeout is used (-1 means use default) but you can override it on the source UI.

Read all messages from SQS FIFO Queue

If you are using Normal SQS Queue then by setting Max Rows=0 will read all messages from your queue. However reading all messages from FIFO Queue is not that simple. Amazon SQS wont return more messages until you delete previously read Messages from same Message Group (identified by MessageGroupId). Due to this constraint you have to try below technique of Loop to read 10 messages until all messages Read > Processed  > Deleted.

Here is note from Amazon Documentation about maximum 10 messages limitation on FIFO Queue.

When you receive a message with a message group ID, no more messages for the same message group ID are returned unless you delete the message or it becomes visible.

Community Reference:

https://stackoverflow.com/questions/51661834/aws-sqs-fifo-how-to-get-more-than-10-messages-at-a-time

If you want to add fault tolerance then you can set Max Retry for NULL Receive like below (e.g. Setting MaxRetry=3 will try max 3 times if no message is read …. It will try every 2 seconds for 3 times before giving up to read more rows).

Read all messages from Amazon SQS FIFO Queue

Read , Process and Delete Amazon SQS FIFO Queue Message in SSIS Data flow

Delete Message from Queue using Amazon SQS Destination in SSIS

Overview

In this tutorial, you will learn how to Delete Messages from the Queue using the ZS SSIS Amazon SQS Destination Adapter. So let’s begin it.

Delete Message from Queue

1. From the SSIS toolbox drag and drop Amazon SQS Destination on the Data flow designer surface and connect it to Amazon SQS Destination.
2. Now double click on Amazon SQS Destination to configure it.
3. In the [Connection Manager] tab select SQS connection manager we have created before it.
   

   Select Amazon SQS Connection

4. Click on [Component Properties] tab, Set Operation to Delete and Select Target Queue from the dropdown list.
   

   Select Delete Operation and Queue

5. Click on [Column Mappings] tab and verify mapping. Map document column from upstream to the MessageText target column.
   

   Do Columns Mapping

6. In the last, click on the OK button to save configure setting UI.
7. After all, Execute the Package and verify target data by going to your AWS console under SQS service section.

Purge All Messages from SQS Queue (Clear Queue)

There will be a time you like to clear entire queue (i.e. Delete All Messages from Queue). Amazon recently added PurgeQueue API feature so now its possible without doing Delete message technique mentioned in previous section. ZappySys support calling any AWS API using REST API Task along with AWS Provider in OAuth Connection. You can use same technique to call PurgeQueue API like below.

1. Open SSIS Package and Drag ZS REST API Task from Control Flow Toolbox
2. Click on URL mode dropdown and change to URL from Connection.
3. From COnnection Dropdown select NEW ZS-OAuth connection
4. Configure OAuth connection to use Provider=Amazon AWS v4
   1. Enter Access Key and Secret Key and click OK to save connection
5. Enter below URL on REST API Task (Replace Queue ID and name with your own values) – Refer to your AWS Console / SQS Queue Properties to get this URL
   

   https://sqs.ENTER-REGION-HERE.amazonaws.com/ENTER-QUEUE-ID-HERE/ENTER-QUEUE-NAME-HERE/?Action=PurgeQueue

   Here is an example of Sample Queue from ZappySys Demo
   

   https://sqs.us-east-1.amazonaws.com/718883169616/zappysystest.fifo/?Action=PurgeQueue

6. Click ok to Save REST API Task

How to Purge Amazon SQS Queue in SSIS (Clear Queue / Delete All messages using PurgeQueue REST API)

Conclusion

So, In this blog, we learn How to Read / Write / Delete Amazon SQS Queue data in SSIS. We used Amazon SQS Destination for write data and Amazon SQS Source to read data. Additionally, you can download SSIS PowerPack here to try many other scenarios that are not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Amazon SQS Source
• Amazon SQS Destination
• Documents of Amazon SQS
• Help File:
  • Amazon SQS Source
  • Amazon SQS Destination

The post How to Read / Write / Delete Amazon SQS Queue data in SSIS appeared first on ZappySys Blog.

In our previous blog, we saw you how to read/write Azure Table Storage data. Now in this blog, we will see How to Read/Write Azure Queue Storage data in SSIS. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like perform other operations on Azure Blob Storage Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on How to Read/Write Azure Queue Storage data in SSIS using the following Source/Destination.

Component Mentioned in this article

	Azure Queue Storage Source
	Azure Queue Storage Destination

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Download and install Microsoft Azure Storage Emulator
4. Download and install Microsoft Azure Storage Explorer
5. Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Storage) then you can skip Step #3

What are Azure Queues?

Azure Queue storage is a service for storing large numbers of messages. You access messages from anywhere in the world via authenticated calls using HTTP or HTTPS. A queue message can be up to 64 KB in size. A queue may contain millions of messages, up to the total capacity limit of a storage account.

Common uses

Microsoft Azure Queue Concept

Common uses of Queue storage include:

• Creating a backlog of work to process asynchronously
• Passing messages from an Azure web role to an Azure worker role
• The Queue service contains the following components:

Getting Started

In order to start, we will show several examples. Zappysys includes an SSIS Azure Queue Storage Source that will help you in reading data of Queue and SSIS Azure Queue Storage Destination that will help you to write data in Queue. Here we are showing you is, how to Read/Write Azure Queue Storage data in SSIS.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of the local Azure Storage Emulator.

Setup Azure Storage client tools and Create Queue

1. Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
   

   Azure Storage Emulator Physical Location

2. If you can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, you have to download and install Microsoft Azure Storage Explorer and then you can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Explorer Location.

4. If you don’t have Azure Storage account then it’s easy to get FREE Trial Azure Storage account or use your MSDN to get credit each month which will be more than enough for real testing. If you prefer to avoid all this hassle then Microsoft provides another great way to test Azure Storage functionality is totally offline mode on your local machine. You can just download Azure Storage Emulator and start testing.
5. Click here to learn more about how to configure Azure Storage Emulator for Testing and Development purposes. In order to connect to Azure Storage Service from SSIS, you will need Storage AccountName and an AccessKey. Ask your SysAdmin or responsible person to provide that information to you. Here are sample Account Name and Access Key(this is just example key which may differ in your case)
   

   Account Name: mystorageaccount 
   Access Key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

6. For Creating a Queue, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Queues. Right-click on Queues and Select Create Queue and give a name you want.
   

   Microsoft Azure Storage Explorer – Create Queue

Write data using ZS Azure Queue Storage Destination

In this tutorial, you will learn how to write data into Azure Queue Storage. We will use ZS DummyData Source for practice it but you can use a suitable Source Adapter. So, Let’s start, In this SSIS we will write data into Azure Queue Storage using ZS Azure Queue Storage Destination.

1. Once you have created Azure Storage Queue and you got valid Account Name and AccessKey (for the local emulator you don’t need this) you may proceed to create new SSIS package. In BIDS/SSDT create a new SSIS package.
2. From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface and double click on it too.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

3. Click here to create Azure Storage Connection.
4. From the SSIS toolbox drag and drop Dummy Data Source on the dataflow designer surface.
   

   DummyData Source – Drag and Drop

5. Double click Dummy Data Source to configure it. From Template pick JsonDocuments and enter row count=100 to generate 100 JSON documents and click OK to save.
   

   DummyData Source – Configure

6. Let’s drag and drop Azure Queue Storage Destination on the dataflow designer surface from the SSIS toolbox.
7. Now single click on the Dummy Data Source, once you see the blue arrow from the source connect it to Azure Queue Destination.
8. Double click Azure Queue Destination to configure it. Select the connection we have created before.
   

   Configure Connection Manager

9. Click on [Component Properties] tab and select the Target Queue from the dropdown list.
   

   Configure Component Properties

10. Click on [Mappings] tab and verify mapping. Map Document column from upstream to the MessageText target column.
    

    Configure Column Mappings

11. Click OK to save settings.
12. Execute the package and verify target data by going to your Azure Queue. We use Azure Storage Explorer.
    

    SSIS Execute Package

    

    Microsoft Azure Explorer – Verify Queue Data

Read data using ZS Azure Queue Storage Source

1. Here you may proceed In BIDS/SSDT to create a new SSIS package.
2. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   SSIS Data Flow Task – Drag and Drop

3. Double click on the Data Flow task to see the Data Flow designer surface.
4. From the SSIS toolbox drag and drop ZS Azure Queue Storage Source on the dataflow designer surface.
   

   SSIS ZS Azure Queue Source – Drag and Drop

5. Double click Azure Queue Storage Source to configure it.
6. Select Azure Storage Connection we have created before and select queue from the dropdown and click on preview.
   

   Configure Azure Queue Source

7. Click on OK button to save configure.
8. Now, you can use a suitable Destination Adapter from here. In this case, we used Trash Destination.
9. From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface.
   

   SSIS ZS Trash Destination – Drag and Drop

10. Now single click on the Azure Queue Storage Source once you see the blue arrow from the source … connect it to Trash Destination.
11. Double click on ZS Trash Destination to Configure it.
    

    SSIS ZS Trash Destination – Configure

12. Click on OK button to save Trash Destination configure setting UI.
13. Execute the package and verify source data in the data viewer.
    

    SSIS ZS Azure Queue Source – Execute the Package

14. After messages read from Azure Queue, the same message cannot be read again until VisibilityTimeout. By default Queue, visibility timeout is used (-1 means use default) but you can override it on the source UI.

Conclusion

In this blog, we learn How to Read/Write Azure Queue Storage data in SSIS. We used the Azure Queue Destination for write data and Azure Queue Source to read data. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Azure Queue Source
• Azure Queue Destination
• Microsoft Azure Queues
• Help File: Azure Queue Source, Azure Queue Destination, Trash Destination

The post How to Read/Write Azure Queue Storage data in SSIS appeared first on ZappySys Blog.

In this article, we will see how to read How to write data into CSV file in SSIS from SQL Server. We will also explain how to compress (gzip) and split  CSV files into multiple files. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. We will use the ZS CSV File Destination component for this article.

What is CSV File Destination?

CSV File Destination can be used to write data in CSV / TSV file format. You can also split large files by row count or size at runtime. It also supports writing files directly in a compressed format such as GZip (*.gz).

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).

Component Mentioned in this article

CSV File Destination

How to Extract Data from SQL Server Table to CSV File Destination.

In this tutorial, we are going to load data from SQL Server Table using OLE DB Source and ZS CSV File Destination. You can use ZS DummyData Source too for practice it.

1. Firstly, you need to Download and Install SSIS ZappySys PowerPack.
2. Once you finished the first step, Open Visual Studio and Create New SSIS Package Project.
3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

4. Double click on the Data Flow task to see the Data Flow designer surface.
5. Here, In Visual Studio, drag and drop the OLE DB Source and CSV File Destination in the design surface and join the tasks with the blue arrow.
   

   SSIS OLE DB Source and CSV File Destination – Drag and Drop

6. Here, you can use our ZS Data Source too.
7. But, we are going through OLE DB Source so need OLE DB Connection and File Connection for CSV File Destination.

How to create OLE DB Connection.

1. Lets, create the first OLE DB Connection.
2. Let’s, Right-click on Connection Managers Panel to Create OLE DB Connection, so you can use Source and Context Menu will appear, Select New OLE DB Connection from the Context Menu.
   

   Create OLE DB Connection

3. Now, click on New Button to create Connection.
   

   Create New OLE DB Connection

4. Let’s Configure Connection Manager, just Follow steps one by one as we have created.
   

   Configure OLE DB Connection Manager

5. Click on Test Connection to see correct configure it.
6. Click on OK button to save connection configure setting UI.

How to Create a File Connection.

1. Let’s, Right-click on Connection Managers Panel to Create File Connection, so you can use Destination Path, and Context Menu will appear, Select New Connection from the Context Menu. In the File Connection Manager, select the Usage type and specify the path of the CSV file.
   CSV File: You can download good one CSV file from here for practice.
   

   https://zappysys.com/downloads/files/test/invoices.csv
   https://zappysys.com/downloads/files/test/invoices.csv.gz
   https://zappysys.com/downloads/files/test/invoices.csv.zip

   

   Create a New File Connection

2. Click on OK button to save connection configure setting UI.

Let’s see how to write data from SQL Server Table to CSV File.

1. Double click on OLE DB Source for configure it.
2. Let’s Configure in Connection Manager, just follow below image steps.
   

   Configure OLE DB Source Editor

3. Click OK to Save OLE DB Source Editor UI Settings.
4. Let’s configure CSV File Destination, Double click on it, In the File Connection Manager, select the file connection created before.
   

   SSIS ZS CSV File Destination – Configure Connection Manager Tab

5. In the Component Properties, set the Append property to False, enter the File Path and set the Overwrite file if exists to True.
   

   SSIS ZS CSV File Destination – Configure Component Properties Tab

6. In the Input Columns Tab, check the columns that you want as load data.
   

   SSIS ZS CSV File Destination – Configure Input column Tab

7. Click on OK button to save configure setting UI.
8. Finally, run the package and the data will be exported from source to destination immediately.
   

   SSIS ZS CSV FIle Destination – Execute the Package

How to do file split using CSV File Destination.

1. Double click on CSV File Destination to configure it.
2. In the Component Properties set Enable File Split to True and File Split Mode BySize.
   

   SSIS ZS CSV File Destination – File Split

3. Click OK button to save configure setting UI.
4. That’s all, Now you can run or execute the package and you will see numbers of CSV file in Destination Folder.

How to do file compression using CSV File Destination.

1. Double click on CSV File Destination to configure it.
2. In the Component Properties set Compression Type to GZip or Zip and File Split Mode BySize.
   

   SSIS ZS CSV File Destination – Compression Type

3. Click OK button to save configure setting UI. and You can Run or Execute.

How to do the maximum number of rows per file using CSV File Destination.

1. Double click on CSV File Destination to configure it.
2. In the Component Properties, you can set Max Rows Per File and Max Size Per File as per your need.
   

   SSIS ZS CSV File Destination – Max Rows Per File

3. Click OK button to save configure setting UI. and You can Run or Execute.

Conclusion

After all, we saw you how to write data into CSV File from SQL Server Table using OLE DB Connection, File Connection, and load into CSV File. We also learned, how to do various things like split CSV File, Compression File and the maximum number of rows per file of data. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following URL for more information.

• Help File: CSV File Destination

The post How to write data into CSV file in SSIS (GZip / Split) appeared first on ZappySys Blog.

In this article, we will learn how to import REST API in Power BI.

Power BI is a very popular Business Analytic tool used to get business information. It is very popular because it is easy to install, simple to learn and very intuitive. Also, REST API is very popular these days and we wanted to show you a way to integrate them in Power BI with the help of the ZappySys ODBC PowerPack. This software includes powerful drivers to query REST API URL, Local XML / JSON files and XML SOAP Web Service with simple SQL queries.

The following video will show how to import REST API in Power BI (JSON or XML Data):
[youtube https://www.youtube.com/watch?v=-3OPfhMY1aI&w=720&h=405]

Requirements

1. First, you will need to have Power BI Desktop (FREE) installed
2. In the second place, you will need to have internet access
3. On third place, our ZappySys ODBC Power Pack

Step-by-Step: Import REST API into Power BI

We’ll walk through the steps to import a REST API into Power BI. The steps outlined below focus on working with JSON APIs, but the same principles can be applied when importing SOAP XML Web Services or local JSON/XML files.

For demonstration purposes, we will utilize a REST API to retrieve data in JSON format. The example URL we’ll use is:

https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json

This URL leverages OData and presents information in JSON format. Our objective is to use the ZappySys ODBC Power Pack to establish a connection to this URL and retrieve the information seamlessly into Power BI.

Create ODBC DSN – JSON Driver

DRIVER={DRIVER-NAME-GOES-HERE}[;Prop1=xxxxxx][;Prop2=xxxxxx] .... [;PropN=xxxxxx]

DRIVER={ZappySys JSON Driver}
DRIVER={ZappySys JSON Driver};DataPath='c:\mydata.json'
DRIVER={ZappySys XML Driver};DataPath='http://myserver/api/xml/getOrders'
DRIVER={ZappySys XML Driver};DataPath='c:\mydata.xml'

DRIVER={DRIVER-NAME-GOES-HERE}[;Prop1=xxxxxx][;Prop2=xxxxxx] .... [;PropN=xxxxxx]

DSN=MyJsonAPI_DSN
DSN=MyJsonAPI_DSN;DataPath='http://myserver/api/json/getOrders'

Connect to REST API data source in Power BI (Connect JSON / XML data)

Publish Power BI dashboard

1. In order to publish the report, press Publish
2. Press Save
   

   Save changes in Power BI:

    

3. Select a workspace for the report
   

   Select Power BI workplace.

4. And that is all. You have now your report ready.

Import data using ODBC ConnectionString (DSN-less)

Let’s explore the process of importing data using the ODBC Driver without the need for configuring an ODBC DSN.

When importing data, you have the option to either use an ODBC DSN or a Direct ConnectionString. This section will guide you through the steps of utilizing the ODBC ConnectionString.

After opting for the ODBC option to retrieve data, you will encounter the “Select ODBC Data Source” screen. To proceed without a DSN, follow these steps:

1. Change the DSN dropdown to “(none).”
2. Enter the full ODBC ConnectionString, adhering to the syntax below. You can construct your own connection string using the ODBC Driver UI and utilize the “Copy ConnectionString” option.
   Syntax: 

   Driver={Your Driver Name}[;Property1=value][;Property2=value]......

3. Here is a screenshot with ConnectionString Setting
   

   Import Data from ODBC Driver – DSN less mode (Use Connection String)

4. Click “Next” to select the authentication method. Choose “Windows” and input the password value if necessary. If your connection string already includes the Password attribute, there’s no need to worry.
   

   Import Data from ODBC Driver – using Connection String – With Password Attribute

POST data to REST API URL

Up until now, we’ve explored the process of fetching data from URLs and importing it into Power BI. However, in certain scenarios, APIs may necessitate the use of the PUT or POST method. These methods involve submitting parameters in the body and receiving a response.

For a more in-depth understanding of how to seamlessly POST data to a URL in Power BI, watch the informative video below.

Calling XML SOAP Web Service in Power BI

So far, we have looked at examples to consume data using JSON driver. Now let’s look at an example, to call XML SOAP Web Service in Power BI.

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

SOAP / REST API pagination in Power BI

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

SOAP / REST API Error Handling in Power BI

Sometimes errors occur... they just do and there is nothing you can do! Or can you? Actually, in ODBC PowerPack you can handle them in two ways.

METHOD 1 - Using Error Handling Options

When to use?

You may want to use them when your source is a resource located on the Internet; e.g. a file on a website, a file on an FTP server or just a plain API HTTP response. By default, when a remote server returns an error, data retrieval is stopped, an error is raised and no data is given back to you. This might not be always desirable.

Scenario 1

Imagine a scenario, that there is a web server which each day at 12 AM releases a new JSON file with that day's date as filename, e.g. http://www.some-server.com/data/2018-06-20.json. And, of course, you want to download it and use it daily in your Power BI report. But you have a problem: Power BI report data sources are refreshed each hour and you may get HTTP 404 status code (no file was found) when a file is not released yet. Which consequentially means other data sources won't be updated as well and you will see old and cached data on the report. That's where you could use Continue on any error or Continue when Url is invalid or missing (404 Errors) to avoid an error being raised and let other data sources to be updated.

Scenario 2

Another scenario is when you expect a web server to raise some kind of HTTP error when accessing a URL. You don't want ODBC Data Source to raise an error but instead, you want to get response data. That's where you can use Continue on any error or alike together with  Get response data on error to continue on an error and get the data:

METHOD 2 - Using Connection [Retry Settings]

Another scenario you may run into is a buggy web server. You ask it to give you some file or data and it, like a snotty kid, just doesn't give it to you! You have to ask twice or thrice before it does its job. If that's the case, you have to retry HTTP requests using Connection:

Security / Connection Types (Options for HTTP, OAuth, SOAP) in Power BI

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Import data into Power BI from SQL Query

If you prefer to import data from an SQL query instead of selecting a table name, you can leverage advanced options during the import steps. Here’s a step-by-step guide:

1. After selecting the DSN, proceed to the import steps.
2. Click on the “Advanced Options” to access the SQL Query editor.

By utilizing these advanced options, you gain the flexibility to tailor your import process by specifying custom SQL queries to retrieve the exact data you need.

Import data into Power BI using SQL Query (ODBC Data source)

Edit Query / Using Parameters in Power BI (Dynamic Query)

let
    vKey=paraAPIKey,
    Source = Odbc.Query(
"dsn=ZS - OData Customers", 
"SELECT * FROM $ WITH (SRC='http://httpbin.org/post'," 
& "METHOD='POST'," 
& "HEADER='Content-Type:application/json'," 
& "BODY=@'{""CallerId"":1111, ""ApiKey"":""" & vKey & """}')")
in
    Source

Edit Query Settings after Import

There will be a time you need to change initial Query after dataset import in Power BI. Not to worry, just follow these steps to edit your SQL.

Edit Power BI Data Source SQL after REST / SOAP Web Service Import

Performance Tips for REST API / XML SOAP Calls

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Conclusion

In this article, we guide you through the process of configuring the ZappySys Power Pack, focusing on the ZappySys driver for JSON. This driver is instrumental in extracting data from REST APIs or JSON files. Follow these steps to seamlessly add the extracted data to Power BI and generate insightful reports.

Additionally, we’ll walk you through the configuration of the Power BI gateway, ensuring that the data is refreshed on a specified schedule for up-to-date and accurate reporting. To try this configuration yourself, download ODBC PowerPack and embark on a journey of efficient data extraction and reporting.

References

• What is Power BI?
• Keep your dashboards and reports up-to-date with your on-premises data sources
• Power BI – Read REST API / JSON File / XML File / SOAP (Pagination, OAuth, OData)

The post How to import REST API in Power BI (Load JSON / SOAP XML) appeared first on ZappySys Blog.