Sometimes Data Flow source does not contain an ID column, or a number indicating a row number, which you need in further steps. To solve this, you may use a free ZappySys PowerPack component Template Transform. So let’s take a look at how to accomplish that.

Tip

Add Template Transform

Go to the Data Flow, drag and drop Template Transform from SSIS toolbox, and connect it right after the source, which does not provide ID/row number:

Configure Template Transform

Template Transform will provide the row number we want. So configure it like this: in the text area enter <<FUN_SEQUENCE>> and optionally you can give this column a name, e.g. RowNumber.

Configure Template Transform to get row numbers for each row in SSIS

By default Template Transform’s output type is DT_NTEXT. So if you wish to change the type to, for example, integer, use the Derived Column connector and CAST function to change it, e.g.:

(DT_I4)(DT_WSTR,100)RowNumber

Derived column added to convert DT_NTEXT column to an integer type.

Result

In the data viewer you would see that you have a column with a different row number in each row:

Results: How to add row numbers to rows in SSIS data flow

The post How to add row numbers in SSIS data flow appeared first on ZappySys Blog.

Using SOAP API with SharePoint in SSIS is a common requirement to Administer SharePoint. SharePoint on-premises is a nice Microsoft application used to share documents and collaborate with the company. You can have schedules, projects, documents, and more shared using a Web platform.

In addition, it is a very useful tool that your company may need. It is also possible to automate, administer tasks using SOAP API.
In this article, we will learn how to create SOAP API queries to SharePoint on-premises.

Also, we will show our ZappySys SSIS PowerPack that includes very powerful tools to export SharePoint data from REST API to any other source.

	SSIS REST API Web Service Task
	SSIS XML Source (File, REST, SOAP Web Service)

Requirements

1. Firstly, you will need SSDT installed.
2. Secondly, SSIS PowerPack installed.
3. SoapUI installed.

Getting started

We will start with SoapUI application. It will help us to get all the possible methods SharePoint SOAP API allows. Also, it will show the inner workings of how the requests should look, which we will later replicate on XML Source. SoapUI will be our polestar.

How to call SharePoint SOAP API (On-Premises)

To call SOAP API in SSIS you have to perform the following steps at a high level. Detailed steps are described in the next section

1. Import WSDL file or URL for your Service in SoapUI for testing.
   1. File > New SOAP Project.
   2. In the Initial WSDL field enter
      http://your.domain.com/_vti_bin/Lists.asmx?wsdl
      or
      http://your.domain.com/my-site/_vti_bin/Lists.asmx?wsdl
2. Add credentials for SOAP Web service (Basic Auth or WSS)
3. Create SOAP Request XML (Edit default parameters) and Click Play Button to test your request. Associative photo:
   

   Create SOAP Request Body from WSDL (Using SoapUI tool)

4. If your test works in SoapUI then you are ready to move to SSIS part.
5. Copy Service URL, Request Body and Headers to Notepad — we will use it in SSIS.
   For Header see Raw Tab on SoapUI > Copy two lines Content-Type and SOAPAction.
   

   Get ContentType and SOAPAction Headers from SoapUI Raw tab

In the next section, we will see how to call SharePoint SOAP Web Service in SSIS.

How to read data from a SharePoint List via SOAP Web Service

Now let’s move onto SSIS and get some data from SharePoint using XML Source.

1. Open SSIS, create a new Integration project and create a new SSIS package.
2. Drag and drop Data Flow onto the Control Flow from SSIS Toolbox.
3. Then drag and drop XML Source onto Data Flow (you may also use other similar connectors such as REST API Task or Web API Destination).
4. Then configure XML Source in this way:
   
   1. Enter URL:
      https://my.sharepoint.site.com/sites/mysite/_vti_bin/Lists.asmx
      or
      https://my.sharepoint.site.com/_vti_bin/Lists.asmx (depends if you have many sites or just one).
   2. Check Use Credentials checkbox and create ZS-HTTP connection. Use Basic or NTLM authorization scheme.
   3. Then copy configuration from SoapUI to XML Source. E.g. HTTP Request Method, Body, Body Content-Type, HTTP Headers (e.g. SOAPAction header).
   4. Use $.soap:Envelope.soap:Body.GetListItemsResponse.GetListItemsResult.listitems.rs:data.z:row[*] as filter. If it doesn’t work, use Select Filter button to select the path manually and select row node.
5. What’s left is to connect XML Source with OLEDB Destination or a similar destination component:
   

Conclusion

To conclude, we can say that using SSIS PowerPack component XML Source and SoapUI we could achieve our goal of getting data from SharePoint On-Premises. The next step could be writing data to SharePoint via the same SOAP API interface, but this time using Web API Destination component.

The post Read data from SharePoint List in SSIS (On-Premises SOAP API) appeared first on ZappySys Blog.

In this article, we will cover the points on how to load 10 million rows from SQL Server to Snowflake in just 3 minutes.

Snowflake is a data warehousing platform that resides in a cloud. Basically, it is a data warehouse software exposed as a service. It allows integrating many data sources via internal Snowflake Partner apps and loading them into the Snowflake storage engine. Another part of Snowflake is a computing engine that is responsible for serving your SQL queries. Both engines can work independently thus users that are querying a data warehouse are not affected by a data load that is happening at the same time. Snowflake is an elastic service, which means you pay only for the resources used. Specifically, you pay only for each second of the processing time you use.

To show how data loading works in Snowflake, we will take 10 million rows from SQL Server and load them into Snowflake using SSIS and ZappySys SSIS PowerPack. These SSIS PowerPack connectors will be used to achieve the task:

Export CSV File Task

Step-by-step – How to load 10 million rows from SQL Server to Snowflake in 3 minutes

Getting started

To achieve the goal, we will use a slightly modified Northwind database (example database from Microsoft) and SnowSQL – a command-line tool provided by Snowflake. So in the first steps, you will need to install the Northwind database and SnowSQL. After that, we will proceed to create a table in Snowflake – a table we will load the data into – and a file format which will be used to load data from the Snowflake staging area into the destination table. Once that’s complete, we will proceed in creating an SSIS package, adding and configuring the connectors, and finally running it to get the results. Let’s proceed!

Install a Northwind database

Download and run the creation script of a modified Northwind database. The only thing that was modified is a “CustomersForSnowflake” view added which returns 10M rows.

Install and configure SnowSQL command-line tool

Download and install SnowSQL; you will find the instructions on how to do that in https://docs.snowflake.net/manuals/user-guide/snowsql-install-config.html.

Once installed, you will need to configure the default Snowflake account name, user name, and password:

1. Go to Windows File Explorer and enter this path:
   %USERPROFILE%\.snowsql\
   Usually, when entered the path looks similar to this one:
   C:\Users\myUserName\.snowsql
2. Find a file named config and open it.
3. Then configure your credentials:

Create a table in Snowflake

Login to your Snowflake account, open a Worksheet and execute this query:

create or replace table Customers(
ID number,
CustomerID nchar(5),
CompanyName nvarchar(40),
ContactName nvarchar(30),
ContactTitle nvarchar(30),
Address nvarchar(60),
City nvarchar(15),
Region nvarchar(15),
PostalCode nvarchar(10),
Country nvarchar(15),
Phone nvarchar(24),
Fax nvarchar(24)
)

We will load 10M customers from the Northwind database and load them into this table.

Create a file format in Snowflake

Then in the same Worksheet create a file format for zipped CSV files by executing this query:

create or replace file format GzipCsvFormat
type = csv
field_delimiter = ','
null_if = ('NULL', 'null')
empty_field_as_null = true
compression = gzip
field_optionally_enclosed_by = '"'

We will use this file format when loading data from a Snowflake stage to the Snowflake destination table. This file format defines CSV format which is used by ZappySys Export CSV Task in SSIS.

Create an SSIS package

We are ready to create a new SSIS package and load some data into Snowflake. We are going to use two methods of how we load data into Snowflake. In one method we will load data into Snowflake’s local storage, while in the second one we will stage data in Amazon S3 bucket. Decide how you want to stage files, and then choose the appropriate workflow when creating a new package:

Loading 10 million rows from SQL Server to Snowflake

Add & configure ZappySys Export CSV Task

Firstly, drag and drop ZappySys Export CSV Task from SSIS toolbox and follow the instructions on how to configure it:

Configure source

1. Proceed with configuring the data source, from which you will be exporting data. We will use the Northwind database as an example:
   

   Export CSV Task: configuring OLE DB Connection to export data from SQL Server to Snowflake

2. Then move on and select it and enter the query or table name you want to export data from:
   

   Export CSV Task: configuring source SQL query for data loading from SQL Server to Snowflake

   We will be using this query:
   select top 10000000 * from CustomersForSnowflake

3. Then in the Split Options tab split the exported CSV into many files, e.g. into 50 MB chunks:
   

   Using data split options in the Export CSV Task

   NOTE: Snowflake recommends having files that are 10-100 MB in size when compressed. So it is perfectly safe to configure the value to 100 MB and above.

Configure target

Depending on which staging approach you chose to use, set the export target to a local path or S3 bucket:

Using local Snowflake storage

Just set the appropriate Save Mode and a file path:

Saving files locally to upload them to Snowflake local stage later

Using S3 storage

For storing staging files in S3, follow these steps:

1. Set Save Mode to Save to Connection and select <New ZS-AWS-STORAGE Connection>:
   

   Export CSV Task: configuring Amazon S3 endpoint as the target for data export

2. Then select S3 as Storage Service and fill in Access and Secret Key, select your region (optional):
   

   Configuring Amazon Connection Manager to store staging data in S3

3. Then in the Export CSV Task window, in Target tab configure it similarly:
   

   Export CSV Task: configuring target to store staging data in S3

   Another option you may consider is to use Amazon Storage Task or Azure Storage Task to upload files to S3 in a separate step after the Export CSV Step. For this perform the following steps.

   1. On the Target tab, set Save Mode to Save to Path (Local Disk) instead of Connection to S3 or Azure.
   2. Drag & Drop Amazon Storage Task or Azure Storage Task  and connect with the previous Export CSV Step
   3. Configure Storage Task to upload Local files to S3 or Azure Blob Storage.
   4. Continue to the next section for more instructions.

Add Execute Process Task to create a staging area

We are ready to add Execute Process Task to create a staging area in Snowflake. Again, depending on where you will store staging files, Snowflake’s local storage or Amazon S3, use one of the approaches below:

Create Snowflake local staging

Creating a local stage in Snowflake using SnowSQL command-line tool and SSIS

File path:

C:\Program Files\Snowflake SnowSQL\snowsql.exe

Arguments:

-q "CREATE OR REPLACE STAGE CustomersStaging" -d DEMO_DB -s Public

Create Amazon S3 staging

Creating an Amazon S3 stage in Snowflake using SnowSQL command-line tool and SSISFile path:

C:\Program Files\Snowflake SnowSQL\snowsql.exe

Arguments:

CREATE OR REPLACE STAGE CustomersStaging url='s3://your-bucket-name/destinationFolder/' credentials=(aws_key_id='AKIAXXXXXXXXXXXXXXXX' aws_secret_key='6p1ayaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaLXz88')

Add Execute Process Task to upload files to the staging area (local staging approach only)

We still need to upload CSV files that were saved locally for local Snowflake staging. So add another Execute Process Task and configure it similarly:

Uploading local CSV files to Snowflake stage using SnowSQL command-line tool and SSISFile path:

C:\Program Files\Snowflake SnowSQL\snowsql.exe

Arguments:

-q "PUT file://e:/temp/*.csv @DEMO_DB.PUBLIC.CustomersStaging"

Add Execute Process Task to copy files from the Snowflake/Amazon S3 stage to Snowflake table

Finally, add the final Execute Process Task to issue a command to load files from the stage into a real Snowflake table:

Copying data from a staging area to a Snowflake table using SSIS

File path:

C:\Program Files\Snowflake SnowSQL\snowsql.exe

Arguments:

-q "COPY INTO CUSTOMERS FROM @CustomersStaging file_format = (format_name = 'GZIPCSVFORMAT')" -d DEMO_DB -s Public

Execute the package

We are ready to execute the package, so just run it. In this example you see the workflow when using Amazon S3 as stage:

Executing an SSIS package to load data from Snowflake to SQL Server

The Results

Once the package executes, we can check the results:

Successful execution of SSIS package when loading data from Snowflake to SQL Server

If we go to Snowflake and execute the SELECT query, we see all 10M customers loaded:

Successful loading of 10 million rows from SQL Server to Snowflake using SSIS and ZappySys SSIS PowerPack

All done in less than 3 mins:

Conclusion

We achieved just what we aimed for – we loaded 10 million rows from SQL Server to Snowflake; all done in less than 3 minutes! Two approaches were taken into consideration: in one we staged CSV files in the local Snowflake stage and in the other we staged files in the Amazon S3 bucket. To accomplish this we used ZappySys Export CSV Task, which allowed us to export data from the SQL Server view to CSV format, split files into smaller ones, zip them and store them locally or upload to the Amazon S3 bucket. Finally, we used a standard SSIS Task “Execute Process Task” to issue Snowflake commands such as creating the stage area in Snowflake, uploading CSVs into the local stage and loading data into the Snowflake table.

Download a sample package

Load 10M rows from SQL Server into Snowflake.dtsx.zip

References

https://docs.snowflake.net/manuals/user-guide/data-load-bulk.html

https://docs.snowflake.net/manuals/user-guide/snowsql.html

https://docs.snowflake.net/manuals/user-guide/data-load-s3-create-stage.html

https://docs.snowflake.net/manuals/sql-reference/sql/create-file-format.html

https://docs.snowflake.net/manuals/sql-reference/sql/put.html

https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-table.html

The post Load 10M rows from SQL Server to Snowflake in 3 minutes appeared first on ZappySys Blog.

In the previous article, we learned how to get data from AdWords in SSIS and basically control your AdWords account programmatically, outside the dashboard. In this one, we will continue on how to download AdWords report by using Google Ads API, ODBC, and ZappySys XML Driver. Let’s not waste our time and cut to the chase!

Prerequisites

1. Google Ads account up and running.
2. ZappySys ODBC PowerPack installed.

Step-by-Step – Download AdWords report

Get Developer Token

The first important step to access any Google Ads Account is to request a developer token. Click this link for step by step information. Watch a short video to get an overview of the entire process to access data via API.

Get Client Customer Id

Login into a non-manager account at https://ads.google.com and get your Client Customer Id:

Getting Google Ads (formerly AdWords) Client Customer Id

Copy it to a notepad, we will use it later.

Open ODBC Data Sources Manager

Search for "odbc" in your start menu and click on ODBC (64 bits). If you can't find this then you can also go to Start Menu > ZappySys > ODBC PowerPack > Click on ODBC Data Sources (64-Bit).

Open ODBC Data Source

Create ODBC Data Source based on ZappySys XML Driver

Select tab User DSN if you want to create a data source only for your user, or select System DSN to create a data source for all users or for OS services (e.g. SQL Server Agent):

Creating ZappySys XML Driver based data source

Configure authentication in the ODBC data source

Let’s authenticate first:

1. Select OAuth in Connection Type property.
2. Click Click to Configure link.
3. In OAuth Provider property select Google.
4. Then either select Use Default OAuth App or Use Custom OAuth App. For the latter, you will need to create a Google OAuth App yourself. It’s perfectly safe to use the default application, but if your company policy does not allow it, you may consider creating it yourself.
5. In Scopes enter:
   https://www.googleapis.com/auth/adwords
6. Then hit Generate Token and hit allow once asked.

Download AdWords report – Configuring authentication in ODBC data source

Proceed with HTTP request configuration

Then go on with configuring the data source:

Configuring HTTP request in ODBC data source to download an AdWords report

1. Give your data source a name
2. Then input this URL to download the report:
   https://adwords.google.com/api/adwords/reportdownload/v201809
   If the above URL does not work, get the working one from Google’s documentation.
3. Input POST as HTTP Request Method.
4. In Body enter this text:
   __rdxml=<reportDefinition xmlns="https://adwords.google.com/api/adwords/cm/v201809">
<selector>
<fields>CampaignId</fields>
<fields>Impressions</fields>
<fields>Clicks</fields>
<fields>Cost</fields>
<predicates>
<field>CampaignStatus</field>
<operator>IN</operator>
<values>ENABLED</values>
<values>PAUSED</values>
</predicates>
</selector>
<reportName>My Campaign Performance Report</reportName>
<reportType>CAMPAIGN_PERFORMANCE_REPORT</reportType>
<dateRangeType>LAST_7_DAYS</dateRangeType>
<downloadFormat>XML</downloadFormat>
</reportDefinition>
   This will display the data of CAMPAIGN_PERFORMANCE_REPORT report (campaign Id, impressions, clicks, and cost) for the last 7 days, for all campaigns that are enabled or paused. Make sure to visit the Google documentation page, which contains the information about how to download a different kind of report, how to configure predicates, etc.
5. In Body Content Type property select Form (application/x-www-form-urlencoded) option.
6. In HTTP Headers enter your developerToken and clientCustomerId retrieved in previous steps.

Select data filter in ODBC data source

Then hit Select Filter button and select row node, so that Array Filter is set to $.report.table.row[*]

Filtering on AdWords report data

If asked whether you want to treat the row element as an array, click “yes”. You can also go to the Xml Array Handling tab and see if the element row is specified there.

The Results

To view the results press Preview tab and then click Preview Data:

Previewing AdWords report data

Now you can use this data source and get the data into an Excel sheet, Power BI report, SQL Server (would need to create a data source in ZappySys Data Gateway instead) or elsewhere.

Reusing the data source

If your client application supports, you can override data source settings, such as HTTP body, in SQL query using WITH clause. In that case, you won’t need to create multiple data sources for different kind of reports; e.g.:

SELECT * FROM $ (RequestData='@c:\files\report-definition.txt')

SELECT * FROM $ (RequestData='__rdxml=<reportDefinition xmlns="https://adwords.google.com/api/adwords/cm/v201809">
<selector>
<fields>CampaignId</fields>
<fields>Cost</fields>
</selector>
<reportName>My Campaign Performance Report</reportName>
<reportType>CAMPAIGN_PERFORMANCE_REPORT</reportType>
<dateRangeType>LAST_7_DAYS</dateRangeType>
<downloadFormat>XML</downloadFormat>
</reportDefinition>')

Build a SQL query using the Query Builder:

Use Query Builder to construct SQL query and override data source properties

Conclusion

In this article, we learned how to download AdWords report using ODBC PowerPack and Google API. We created an ODBC data source based on XML Driver, then configured the OAuth authentication, HTTP request and data filtering without a single line of code. Then we previewed AdWords “Campaign Performance Report” data and finally, considered how to reuse the data source we created. Visit https://zappysys.com/blog/category/odbc-powerpack/odbc-app-integration/ for examples of how to integrate data sources based on ZappySys Drivers into applications.

References

https://developers.google.com/adwords/api/docs/guides/reporting

https://developers.google.com/adwords/api/docs/appendix/reports

https://developers.google.com/adwords/api/docs/appendix/reports/campaign-performance-report

The post Download AdWords report using Google Ads API, ODBC, and XML Driver appeared first on ZappySys Blog.

In the previous article, we learned how to copy settings from curl to ZappySys products, and in this article, we will continue on how to migrate Postman API call to SSIS / ODBC PowerPack. Postman is a very popular client tool to test API calls. Many API providers document how to call their API using Postman. In this article we will focus on how to use the same settings in ZappySys Products to implement API integration in SSIS or other apps like Power BI, Informatica, Excel, Tableau, SSRS… many ODBC apps.

When you are connecting to an API you may have examples of how to do that in Postman. If you know how to do it in Postman, you are one step away from making it work in SSIS PowerPack and ODBC PowerPack (JSON, XML, CSV Drivers, and other). Most of the examples will show how to do that specifically using ODBC PowerPack and JSON Driver, but since ODBC PowerPack UI is very similar to the connectors of SSIS PowerPack, it will be quite clear where to use it in SSIS PowerPack as well. In the examples where UI differs significantly, both – ODBC and SSIS PowerPack – examples will be presented.

Prerequisites

In this article we assume you meet these requirements:

When migrating Postman API call to SSIS PowerPack

1. Postman installed.
2. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
3. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
4. ZappySys SSIS PowerPack is installed (download it).

When migrating Postman API call to to ODBC PowerPack

1. Postman installed.
2. ZappySys ODBC PowerPack installed (download it).

The Basics

This section shows how to migrate simple options such as URL, HTTP Request Method, and HTTP Headers.

How to configure the URL and HTTP Request Method

Migrate Postman API call to SSIS / ODBC (API URL setting to ODBC Driver (GET / POST) )

How to configure HTTP Headers

Migrate Postman HTTP Headers setting to ODBC Driver

Posting data

In this section, you will find how to post data in ZappySys products. You will learn how to post different formats of data, upload files and send multi-part data.

How to configure POST raw body

Migrate Postman Body / Content-Type setting to ODBC Driver

How to configure POST JSON body

Migrate Postman Body / Content-Type setting to ODBC Driver

Use this table to determine what kind of Body Content Type to configure:

How to configure POST form/x-www-form-urlencoded data

Migrate Postman Body (x-www-form-urlencoded) setting to ODBC Driver

How to configure multipart form POST data

Migrate Postman Multi-Part File Upload setting to ODBC Driver (Upload File)

Authentication

This section will cover all common authentication methods, starting from using API Key to OAuth 2.0.

How to authenticate using API Key in HTTP headers

Authentication – Migrate Postman API Key (via header) setting to ODBC Driver

How to authenticate using API Key as a parameter in URL

Authentication – Migrate Postman API Key (via URL query parameter) setting to ODBC Driver

How to authenticate using Bearer Token

Authentication – Migrate Postman Bearer Token setting to SSIS / ODBC Driver

How to authenticate using Basic Authentication

In ODBC PowerPack

Authentication – Migrate Postman Basic Auth setting to SSIS / ODBC Driver (Login using Userid / password)

In SSIS PowerPack

In SSIS PowerPack use the same configuration as above, just create ZS-HTTP Connection Manager using instructions below:

Authentication – Migrate Postman Basic Auth setting to SSIS JSON / XML Source (Login using Userid / password)

How to authenticate using OAuth

In ODBC PowerPack

1. Configure main options:
   

   Authentication – Migrate Postman OAuth setting to SSIS / ODBC Drivers ( JSON / XML / CSV API) – Client ID / Secret / Scope / Auth and Token Url (Client Credential Grant Type, Password Grant Type, Code Authorization Grant Type)

2. Configure Callback/Return Url:
   

   Authentication – Migrate Postman OAuth setting to SSIS / ODBC Drivers ( Callback / Redirect URL )

If you are using a known, public API service, you may try locating it in the OAuth Providers list, which will save you from configuring Authorization and Access Token URLs:

For simplicity and if your company policy allows it or at least for testing purposes, you can use a Default OAuth Application created by ZappySys:

In SSIS PowerPack

In SSIS PowerPack use the same configuration as above, just create ZS-OAUTH Connection Manager using instructions below:

Authentication – Migrate Postman OAuth setting to SSIS JSON / XML Source

How to authenticate using AWS Signature

In ODBC PowerPack

Authentication – Migrate Postman AWS API call setting to SSIS / ODBC Drivers ( JSON / XML / CSV API call)

In SSIS PowerPack

In SSIS PowerPack use the same configuration as above, just create ZS-OAUTH Connection Manager using instructions below:

How to authenticate using NTLM Authentication

In ODBC PowerPack

In SSIS PowerPack

In SSIS PowerPack use the same configuration as above, just create ZS-HTTP Connection Manager using instructions below:

Conclusion

This article was quite self-explanatory, but if you have any questions or you have an authentication method that is not listed here and you don’t know how to proceed, please, contact us and we will try to help you: https://zappysys.com/support/.

The post How to migrate Postman API call to SSIS / ODBC PowerPack appeared first on ZappySys Blog.

There might be a case when you have many CSV, JSON or XML files in Amazon S3 bucket and you want them to be imported straight into a SQL Server table. Here come ZappySys ODBC PowerPack and ZappySys Data Gateway (part of ODBC PowerPack) which will enable you to accomplish that. ZappySys ODBC PowerPack includes powerful Amazon S3 CSV, Amazon S3 JSON and Amazon S3 XML drivers that let you connect to an S3 bucket and read the contents from the files in the bucket. Bringing in ZappySys Data Gateway allows doing that right from a SQL Server. Let’s begin and see how to import Amazon S3 files into SQL Server.

These drivers are related to this article:

Amazon S3 Driver (for XML Files)

Amazon S3 Driver (for JSON Files)

Amazon S3 Driver (for CSV Files)

Prerequisites

1. Have an AWS account.
2. Existing Amazon S3 bucket.
3. SQL Server instance installed (can be a SQL Server Express LocalDB instance).
4. SQL Server Management Studio (SSMS) installed.
5. ZappySys ODBC PowerPack installed (must be installed on a Windows machine; can be a different machine than SQL Server is installed on).

Getting started

We will start from a simple example of how to load contacts from a JSON that are located in Amazon S3 bucket, then we will proceed by loading contacts from many XML files that are compressed in ZIP, and finally, we will traverse many folders and subfolders to gather CSVs and load contacts from them.

Step-by-Step – Import JSON file located at Amazon S3 into SQL Server

Open and configure ZappySys Data Gateway

Create an Amazon S3 JSON data source in ZappySys Data Gateway

1. The first thing you will have to do is to create a data source in ZappySys Data Gateway. Just click Add button, give the data source a name, e.g. “MyContactsJSON“, and then select Native – ZappySys Amazon S3 JSON Driver:
   

   Adding a JSON data source in ZappySys Data Gateway to load it from Amazon S3 bucket into SQL Server

2. Then click Edit and add the Data Gateway user you created in Users tab. We will use this user later when adding a Linked Server to the Data Gateway to authenticate:
   

   Adding a user to the newly created Amazon S3 JSON data source

3. Once you do that, then click Edit to configure the data source:
   

   Preparing to configure the Amazon S3 JSON data source to load a JSON from Amazon S3 bucket to SQL Server

4. When a window opens, click Click here to Configure the Connection and enter your Access Key and Secret Key:
   

   Configuring the authentication to Amazon S3 bucket in ZappySys Data Gateway

5. After that, select a JSON file you want to load, and then click Select Filter button to choose data you want to be displayed in SQL Server:
   

   Configuring the JSON data source to load a JSON from Amazon S3 bucket into SQL Server

6. Go to the Preview tab and click the Preview Data button to make sure everything is configured correctly and preview the results:
   

   Previewing JSON data in the data source based on Amazon S3 JSON Driver

Now you are ready to create a Linked Server and connect to the data source you just created. If you used “MyContactsJSON” as the data source name, make sure to use the same name when creating a Linked Server.

Set up a SQL Server Linked Server

Execute the SQL query

Once you created the Linked Server to ZappySys Data Gateway, you are ready to execute the SQL query and load data into SQL Server. Supposedly, you created the Linked Server with the name “GATEWAY”, then open SSMS and execute this query:

SELECT * INTO MyContacts FROM openquery([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyContacts

The results

You should see similar results after you execute the query:

The results of getting JSON data straight from Amazon S3 bucket using ZappySys Amazon S3 JSON driver

Now we are ready to proceed to the next section and import many XML files at once.

Step-by-Step – Import many XML files located at Amazon S3 into SQL Server

Overview

Supposedly, you have many XMLs where each one is zipped and you want to load them all into a SQL Server table:

Zipped XML files located in Amazon S3 bucket to be loaded into SQL Server

The first thing you will have to do is to create a data source, based on ZappySys Amazon S3 XML Driver. Let’s proceed and just do that.

Create an Amazon S3 XML data source in ZappySys Data Gateway

Follow the same steps as in Create an Amazon S3 JSON data source in ZappySys Data Gateway section when adding a new data source, except for these two steps:

1. When adding a data source select Native – ZappySys Amazon S3 XML Driver as Connector Type:
   

   Choosing “ZappySys Amazon S3 XML Driver” to load XMLs to SQL Server

2. Then click Edit and edit the data source similarly like in this window:
   

   Configuring data source based on ZappySys Amazon S3 XML Driver to load XMLs into SQL Server

Now we are ready to set up a Linked Server to this newly created data source.

Set up a SQL Server Linked Server

Follow the very same steps from the section above where we added the Linked Server to a JSON data source, except that change Datasource property of the Linked Server to match the name of the Amazon S3 XML data source you created. Once you do that, you are ready to load XMLs into your SQL Server.

Execute the SQL query

Again, if you created the Linked Server with name “GATEWAY”, execute this SQL query in SSMS:

SELECT * INTO MyContacts FROM openquery([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyContacts

The results

You should see a similar view once you execute the query:

The data of many XMLs loaded from Amazon S3 bucket into SQL Server

Now we are ready to move to the next section and recursively scan CSVs and load them into SQL Server.

Step-by-Step – Import CSV file located at Amazon S3 into SQL Server

Overview

Let’s say you have many CSVs that are located in a folder tree, including subfolders and you want them all in your SQL Server:

Many CSV files located in Amazon S3 bucket folders and subfolders to be loaded into SQL Server

We will follow the same steps to accomplish that as we did for loading JSON and XMLs.

Create an Amazon S3 CSV data source in ZappySys Data Gateway

Follow the steps as in Create an Amazon S3 JSON data source in ZappySys Data Gateway section when adding a new data source, except for these two steps:

1. When adding a CSV data source select Native – ZappySys Amazon S3 CSV Driver as Connector Type:
   

   Choosing “ZappySys Amazon S3 CSV Driver” to load CSVs to SQL Server

2. Then click Edit and edit the data source similarly:
   

   Configuring data source based on ZappySys Amazon S3 CSV Driver to load CSVs into SQL Server

Now we are ready to proceed and add a Linked Server to this data source.

Set up a SQL Server Linked Server

Again, follow the very same steps from the section above where we added the Linked Server to a JSON data source, except that change Datasource property of the Linked Server to match the name of the Amazon S3 CSV data source we created. Once you do that, you are ready to load CSVs into your SQL Server.

Execute the SQL query

Once again, if you created the Linked Server with name “GATEWAY”, execute this SQL query in SSMS:

SELECT * INTO MyContacts FROM openquery([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyContacts

The results

You should see a similar view, once you execute the query:

The results of loading many CSVs from Amazon S3 bucket into SQL Server

Conclusion

We had a goal to import Amazon S3 files into SQL Server from SQL Server itself. We learned how to load JSON, XML and CSV files into SQL Server, which some of them were zipped and were located at different folders and subfolders. It was possible to accomplish that using ODBC PowerPack, ZappySys Data Gateway and ODBC PowerPack drivers – Amazon S3 JSON Driver, Amazon S3 XML Driver, and Amazon S3 CSV Driver. At first, we created data sources in the Data Gateway, then we created Linked Servers in SQL Server and connected to ZappySys Data Gateway, which finally made it possible to load data from Amazon S3 bucket straight from SQL queries in SSMS.

Check Amazon S3 Integration with Other BI Tools (Power BI, Excel, SSRS, MS Access, etc.)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

The post Import Amazon S3 files into SQL Server (CSV/JSON/XML Driver) appeared first on ZappySys Blog.

In this article, you will learn how to refresh a Power BI dataset with Power BI REST API using SSIS and ZappySys SSIS PowerPack. Power BI is a Microsoft reporting product that can get data from virtually any source and display it nicely in a report or a dashboard. Each Power BI report contains a dataset, which can be refreshed to show relevant data in the report. Power BI also has a Scheduled refresh capability, in case you want to refresh a dataset periodically. However, you are limited to only 8 specific times per day you can refresh a dataset (as of May 23, 2019). In the Power BI Premium license you can do it 48 times per day. But what if you need to refresh it more frequently? Or if you desire to refresh it at a non-predefined time; which is not known in advance? E.g. after an ETL process has finished? Here Power BI REST API comes into rescue and lets you refresh a dataset on demand.

Here are a few use cases when you may want to refresh a dataset from a SSIS package, using Power BI REST API:

• The need to refresh a dataset more often than Power BI allows.
• The requirement to refresh a dataset just after some process is finished; e.g. after an ETL process.
• The ability to have absolute control when to refresh multiple datasets. Sometimes, refreshing multiple huge datasets at once could cause memory problems in the Power BI service. So, having an orchestrated refresh would solve this problem.

We will use these ZappySys SSIS PowerPack connectors/connections in this article:

OAuth Connection Manager

REST API Task

Let’s begin!

Prerequisites

1. You have a Power BI account and you have basic Power BI skills.
2. An uploaded Power BI report to Power BI service.
3. A Microsoft Azure subscription (there is a free option with a 200 USD credit).
4. Azure Active Directory set up.
5. SSIS designer installed. Sometimes it is referred to as SSDT or BIDS (download it from the Microsoft site).
6. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
7. ZappySys SSIS PowerPack is installed (download it).

Getting started

Suppose you have a simple report and you want to refresh its dataset using Power BI REST API and SSIS. So far, the dataset has not been refreshed at all:

Preparing to refresh the Power BI dataset using SSIS and Power BI REST API.

To start, create an SSIS package and follow the steps below to accomplish the goal.

Step By Step – How to refresh a Power BI dataset with REST API using SSIS

Create an OAuth application

Create OAuth Connection Manager in SSIS package

1. Drag and drop REST API Task from SSIS Toolbox, double click, select Url from connection, and then select <New ZS-OAUTH Connection> item:
   

   Creating OAuth Connection Manager to connect to Power BI OAuth Application from SSIS.

2. Input Client ID from Step 2 when we were creating an Azure Active Directory OAuth Application.
   

   Configuring OAuth Connection Manager to authenticate to Power BI OAuth Application using SSIS.

3. Then proceed and input the Client Secret from Step 3, Create an Azure Active Directory OAuth application section.
4. Please go on, and fill in Authorization Url and Access Token Url properties with values we copied in Step 4, Create an Azure Active Directory OAuth application section.
5. In Scopes / Permissions field input these scopes:
   https://analysis.windows.net/powerbi/api/Dataset.ReadWrite.All
offline_access
6. Then go to the Advanced tab and in the Callback/Return Url field input this value:
   https://login.microsoftonline.com/common/oauth2/nativeclient
   

   Setting Callback/Return Url (Redirect URI) in OAuth Connection Manager to authenticate to Power BI OAuth Application using SSIS.

7. Then go back to the tab General and click the Generate Token button.
8. Once asked for credentials, log in with your organization user; this will generate Refresh and Access tokens.

Retrieve the ID of the Power BI dataset you want to refresh

1. Go to Power BI datasets.
2. Then click on Datasets.
3. Proceed and click the dataset you want to refresh.
4. Finally, copy the ID of the dataset from the URL, we will use it in the next section.

Getting Power BI dataset ID to refresh it in an HTTP request to Power BI REST API.

Configure REST API Task to refresh a Power BI dataset

We configured OAuth Connection Manager and retrieved the ID of the dataset we want to refresh, now it’s time to configure REST API Task and refresh the dataset. So do the following:

1. Go to the SSIS package and open the REST API Task for configuration.
2. Input URL in such a format:
   https://api.powerbi.com/v1.0/myorg/datasets/DATASET-ID/refreshes Replace DATASET-ID with an actual ID of the dataset. E.g.:
   https://api.powerbi.com/v1.0/myorg/datasets/a38c78fc-8c87-45e3-b961-6dcd3ac1eada/refreshes
   NOTE: If your dataset resides in a Workspace, use this format instead:
   https://api.powerbi.com/v1.0/WORKSPACE-ID/datasets/DATASET-ID/refreshes
3. Then set POST as HTTP Request Method.
4. In the Body field, input this JSON:
   { "notifyOption": "MailOnFailure" }
5. Press the Test Request/Response button to test the request and refresh the dataset at the same time.

Configuring a REST API Task to refresh a Power BI dataset using Power BI REST API and SSIS.

Run the package

Hit F5 and execute the package:

Executing SSIS package and refreshing Power BI dataset.

The Results

Now it’s time to check the results and see if the dataset was refreshed successfully:

1. Again, go to Power BI datasets.
2. Then click on Datasets.
3. Click the dataset you refreshed.
4. Then click “Refresh history” to check the refresh history. We expect two manual refreshes to be there.

Results: Successful Power BI dataset refresh using Power BI REST API and SSIS.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Deployment to Production

Conclusion

You learned how to refresh a Power BI dataset with REST API using SSIS. To accomplish the goal, firstly, we created the Azure Active Directory application and configured OAuth Connection Manager based on it. Then we retrieved the Power BI dataset ID and used it in the ZappySys SSIS PowerPack connector REST API Task. Finally, using the same REST API Task we made the HTTP request to Power BI REST API “refreshes” method and successfully refreshed the dataset. If you need to make a call to another Power BI REST API method, then just configure the REST API Task according to the documentation (for getting data use JSON Source or Web API Destination instead). Finally, if you make many requests to the Power BI REST API service, make sure you check its limitations.

References

https://powerbi.microsoft.com/en-us/

https://docs.microsoft.com/en-us/rest/api/power-bi/datasets/refreshdataset

https://docs.microsoft.com/en-us/rest/api/power-bi/

https://docs.microsoft.com/en-us/power-bi/developer/api-rest-api-limitations

The post How to refresh Power BI dataset with REST API using SSIS appeared first on ZappySys Blog.

In this article, you will learn how to load data from Exact Online into SQL Server using SSIS. Exact Online is a popular CRM, ERP and HRM tool which also lets you manage your finances. As an example, we will use Exact Online contacts as data to load it into a SQL Server database. We will use ZappySys SSIS PowerPack connector to get contacts from Exact Online and then use standard SSIS connector OLE DB Destination to load data into a table. This SSIS PowerPack connector will be used to get the data:

JSON Source

Step-by-step – Load data from Exact Online into SQL Server using SSIS

Getting started

We will get these contacts and load them into a SQL Server database:

Loading Exact Online contacts into SQL Server

The first thing you will need to do is to create an OAuth application in Exact App Center and then we will proceed in getting the data and loading it into a SQL Server database.

Create OAuth application in Exact App Center

1. Visit Exact App Center and login:
   

   Exact App Center login window

2. Once logged in, click on Manage my apps link:
   

   Exact App Center login area

3. Create a new OAuth application:
   

   Exact Online API applications management page

4. Give it a name.
5. In Redirect URI field enter: https://www.zappysyscom/oauth2
   
   

   Registering Exact Online API application to load data into SQL Server

6. Hit Register.
7. Don’t close the browser, we will use App information later.

Create OAuth Connection Manager to connect to Exact Online

1. Create a new SSIS package.
2. Create a new OAuth Connection Manager, by right-clicking on the Connection Managers pane:
   

   Creating OAuth Connection Manager to connect to Exact Online API application

3. Select ZS-OAUTH to create one:
   

   Selecting ZS-OAUTH option to create OAuth Connection Manager

4. Then open your Exact App Center API application configuration we left open in the previous step. We will use that data from that window.
5. Select Custom as OAuth Provider.
6. Select Authorization Code Grant as OAuth Grant Type.
7. Copy Client ID and paste it into Client Id field.
8. Copy Client secret and paste it into Client Secret field.
9. Set Authorization Url to https://start.exactonline.nl/api/oauth2/auth.
10. Set Access Token Url to https://start.exactonline.nl/api/oauth2/token.
    
    

    Configuring OAuth Connection Manager to access Exact Online data via API

11. Then open Advanced tab and copy paste Redirect URI to Callback/Return Url field:
    

    Configuring OAuth application Callback/Return URL

12. Then click on OAuth2 Grant Options tab and check option below.
13. Finally, since Exact Online refresh token keeps changing, enter a filename where to store the refresh token:
    

    Configuring OAuth2 Grant specifics

    NOTE: Can be a non-existent file or must be an empty, existing file.

Getting data from Exact Online

1. Drag and drop Data Flow and then JSON Source into it:
   

   Dragging and dropping Data Flow Task into Control Flow

2. Double click on JSON Source to modify it.
3. Check Use Credentials checkbox and select OAuth 1 as your Connection Manager.
4. Add HTTP Header named Accept and set the value to application/json.
5. In Array Filter field enter $.d.results[*].
6. In Enter Path or Web URL field enter this URL:
   https://start.exactonline.nl/api/v1/current/Me?$select=CurrentDivision
7. Hit Preview button.
8. Copy your division number, you will need it when constructing a URL for getting contacts.
   

   Getting division number

9. Now enter this URL in the field:
   https://start.exactonline.nl/api/v1/{division}/crm/Contacts?$select=FirstName,LastName
   Replace {division} with your division number.

Configuring JSON Source to get Exact Online data and load it into SQL Server

Pagination

To paginate through the results, perform these steps:

1. Open Pagination tab.
2. In Next Link/Cursor Expression field enter $.d.__next

Configuring pagination in JSON Source to paginate through Exact Online contacts

Loading Exact Online data into SQL Server

1. Drag and drop OLE DB Destination or similar connector.
2. Connect JSON Source with OLE DB Destination.
3. Create a new connection.
4. Click New to create a new table.
5. Hit OK.
6. Select Mappings to map the columns.
7. Hit OK to close the window.
8. Execute the package.

Configuring OLE DB Destination to load Exact Online data into SQL Server

The results

Once you execute the package, you should see similar results in Visual Studio and SQL Server Management Studio:

Executing SSIS package and loading Exact Online data  into SQL Server

Successfully loading Exact Online data into SQL Server

As you see, these are the same Exact Online contacts showed in the Getting Started section.

Conclusion

You learned how to load data from Exact Online into SQL Server using SSIS and ZappySys SSIS PowerPack. At first, we created an Exact Online OAuth application, then created an OAuth Connection Manager based on its information and finally loaded contacts using JSON Source.

References

Exact App Center

API methods

Get contacts API method

Pagination

The post Load data from Exact Online into SQL Server using SSIS appeared first on ZappySys Blog.

In this article, we will get Office 365  data in Power BI using Microsoft Graph API and ODBC drivers. Specifically, we will get Excel file data in OneDrive, events from a Calendar and finally, a list in a note in OneNote and then load that data into a Power BI report. We will achieve that by creating an ODBC data source using ZappySys ODBC PowerPack and then querying corresponding Microsoft Graph API methods. ODBC PowerPack allows to query REST/SOAP APIs in an easy way and lets to view responses in a tabular format, which will be perfect to accomplish our goal.

Prerequisites

1. A Power BI account.
2. Wield basic Power BI skills.
3. Power BI Desktop installed.
4. ZappySys ODBC PowerPack installed.

Step By Step – Get Office 365 data in Power BI using Microsoft Graph API and ODBC

Set up an ODBC data source based on ZappySys JSON Driver

Add a data source

Configure the data source

Set up an ODBC data source based on ZappySys XML Driver

1. Before adding another data source based on ZappySys XML Driver, copy the connection string of the data source we just created:
2. Close the window and then add a new data source, but this time based on ZappySys XML Driver.
3. Once a window is opened, give it a name, e.g. System - XML.
4. Then press Load connection string button to load the connection string.
5. Paste the copied connection string and replace JSON text to XML, so that you have { ZappySys XML Driver } in DRIVER property.
6. Close the window.

Create a Power BI report

Reading files from OneDrive (getting Excel data)

Let’s say you have Stock.xlsx stored in Sample Documents folder, OneDrive with this kind of data and you want to show it in a Power BI report:

To have that data in a Power BI report, perform these steps:

1. Open Power BI Desktop and create a new Power BI report.
2. Create a data source based on System – JSON DSN (since we get a JSON response):
   
3. Then in SQL statement area enter this query:
   SELECT * FROM $
WITH(Src='https://graph.microsoft.com/v1.0/me/drive/root:/YOUR FOLDER/Stock.xlsx:/workbook/worksheets/SHEET NAME/usedRange'
,Filter='$.values[1:]'
,ArrayTransformType='TransformSimpleTwoDimensionalArray'
,ArrayTransColumnNameFilter='$.values[0][*]'
)
Using WITH clause lets us override the properties. You can generate the query above using the Query Builder:
   In this step we are overriding Url, Filter and array transformation fields you configured in JSON DSN. The Src value overrides the URL. The Filter property enables us to select the data we want to see in the report. We use [1:] in $.values[1:] to skip the first row because the first row contains the headers and we want only the data. You can configure this option in the Filter Options tab (read more about JSON path).We also have to make the transformation on the JSON and turn JSON arrays into rows. To accomplish that, we override ArrayTransformType. With ArrayTransColumnNameFilter property and [0][*] text, we specify that the first array of parent array will be the headers of the columns. You can read more about transformations in Parse multi-dimensional JSON array article.
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other OneDrive endpoints.
4. Click OK.
5. Drag a table into the report, select the columns and you should see a similar view:
   

Reading calendar entries

Now, let’s say you use a calendar in Office 365 and you want your appointments to be visible in a Power BI report:

1. To start, create a new Power BI report.
2. Then add a data source based on System – JSON DSN (again, we get the response in JSON):
   
3. In SQL statement section enter this query:
   SELECT * FROM $
WITH(Src='https://graph.microsoft.com/v1.0/me/events?$select=subject,start,end,location'
,Filter='$.value[*]'
,Header='Prefer: outlook.timezone="FLE Standard Time"'
)Here we also are overriding HTTP Header so that we get the event times not in UTC/GMT but for the specified timezone. It must be specified in Prefer: outlook.timezone=”{time-zone-string}” format (check an article on how to get a list of supported time zones).
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other Calendar endpoints.
4. Click OK.
5. Drag a table control into the report, select several columns and you should see a similar view:
   

Reading notes from OneNote

Suppose, you have these notes in OneNote and you want “Groceries list” to be shown in a Power BI report:

1. First, we will need to find NOTE ID we are trying to get. To do that, open ODBC Administrator, find System – JSON data source and hit Configure.
2. Then go to the Preview tab, and input this SQL query and hit Preview Data:
   SELECT * FROM $
WITH (Src='https://graph.microsoft.com/v1.0/me/onenote/pages'
,Filter='$.value[*]'
)
3. Once you get a result set, find your note name and copy its ID:
   
4. Then create a new Power BI report.
5. Create a new data source based on System – XML DSN (we are using XML-based DSN because the response will be in HTML):
   
6. In SQL statement section enter this query (use the NOTE ID you retrieved in the previous step):
   SELECT * FROM $
WITH(ElementsToTreatAsArray='div,p'
,Src='https://graph.microsoft.com/v1.0/me/onenote/pages/NOTE ID/content'
,Filter='$.html.body.div[0].p[1:]'
)

   
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other OneNote endpoints.
7. Click OK.
8. Drag and drop Multi-row card visual, select #text column and you should see a similar view:
   
9. We finished! Now you can publish your Power BI reports to Power BI service.

Conclusion

Today we learned how to call Microsoft Graph API methods using ZappySys ODBC PowerPack and load response data into Power BI without any ETL procedures. Particularly, we used OneDrive, Calendar and OneNote Graph API endpoints to achieve that. We created two ODBC data sources based on JSON and XML ODBC Driver to query those endpoints, which let us read data directly in a Power BI report. Finally, we examined how to refresh data source on Power BI using on-premises Power BI gateway.

References

https://powerbi.microsoft.com/en-us/

https://apps.dev.microsoft.com/

https://developer.microsoft.com/en-us/graph/graph-explorer/

https://docs.microsoft.com/en-us/graph/overview

https://docs.microsoft.com/en-us/graph/onedrive-concept-overview

https://docs.microsoft.com/en-us/graph/outlook-calendar-concept-overview

https://docs.microsoft.com/en-us/graph/integrate-with-onenote

The post Get Office 365 data in Power BI using Microsoft Graph API and ODBC appeared first on ZappySys Blog.

In this article, you will learn how to push data into a Power BI dataset from SQL Server. Power BI is a Microsoft reporting product that can get data from virtually any source and display it nicely in a report or a dashboard. Among its dataset types, it has a streaming dataset into which you can push data from a Power BI REST API. Once pushed, data will be immediately refreshed in a visual, pinned to a dashboard. Thus this approach of sending data to a streaming dataset is a perfect way for displaying real-time data. Yet it is not possible to push data to a dataset straight from SQL Server out of the box. However, with ZappySys ODBC PowerPack it is possible to achieve that quite easily. We will employ one of its apps – ZappySys Data Gateway – to make HTTP requests and push data to a streaming dataset.

The use cases when you may want to push data to a streaming dataset from a SQL Server are:

• Displaying real-time data from many SQL Server instances at once (and you cannot use the DirectQuery option in a report).
• Displaying real-time data when many sources write to a dataset, and a SQL Server is one of them.
• Any other case.

Let’s begin and write some data to a Power BI streaming dataset!

Prerequisites

1. You have a Power BI account and you have basic Power BI skills.
2. SQL Server instance installed (can be a SQL Server Express LocalDB instance).
3. SQL Server Management Studio (SSMS) installed.
4. ZappySys ODBC PowerPack installed (must be installed on a Windows machine; can be a different machine than SQL Server is installed on).

Step By Step – Push data into a Power BI dataset from a SQL Server

Add a data source in the Gateway

It’s time to add a data source, which we will use to POST data to Power BI REST API:

1. In the Gateway, click on Data sources tab, click Add, give data source a name and set Connector Type to Native – ZappySys JSON Driver:
2. Then click Edit to modify the data source:
   
3. Finally, paste into Url textbox Push URL of the dataset which we saved in a previous step. Also, set HTTP Request Method to POST:
   

   Configure REST API Connection for Power BI dataset insert (Using ZappySys JSON / REST Driver)

Add a Linked Server to the Gateway

Writing data into Power BI dataset from SQL Server

Once you add a Linked Server you can execute a query to push data into a Power BI streaming dataset:

1. In the same SQL Server Management Studio create a new query and paste the code below. In Body attribute, we pass the JSON in the same format which was shown when we created a Power BI dataset:
   

   SELECT * FROM
   OPENQUERY([GATEWAY], 'SELECT * FROM $
   WITH (Body=''[ 
   	{ "Timestamp" :"2018-11-15T02:57:04.547Z", "Temperature" : 50.5 },
   	{ "Timestamp" :"2018-11-16T02:57:04.547Z", "Temperature" : 55.5 },
   	{ "Timestamp" :"2018-11-17T02:57:04.547Z", "Temperature" : 60.5 },
   	{ "Timestamp" :"2018-11-18T02:57:04.547Z", "Temperature" : 57.5 },
   	{ "Timestamp" :"2018-11-19T02:57:04.547Z", "Temperature" : 62.3 }
   ]''
   ,meta=''[ { "Name": "Dummy", "Type": "String" }]'' )')

   Note: Use the same Linked Server name you created in a previous step. In our case, it is “GATEWAY”.
2. Execute the query.

The results: creating a Power BI report

Now it’s time to check the results and see if data went through:

1. Go to Power BI portal.
2. Select Datasets and click on the add new report icon:
   

   Create new Power BI Report for real time dataset

3. Add a Line Chart, a Table and link them to the dataset.
4. You should see the data we pushed to the dataset:
   

   Showing real time data in Power BI dashboard

Conclusion

You learned how to push data into a Power BI streaming dataset directly from SQL Server. It was possible to achieve that with ZappySys Data Gateway and SQL Server Linked Server option. At first, we created a streaming dataset in Power BI, then configured a datasource in the Gateway, followed with adding a Linked Server to the Gateway, executed T-SQL query in SSMS and pushed some data to the dataset and finally created a Power BI report to check the received data.

When we set Historic data analysis option when creating a dataset, we, in turn, created a dataset of type Push (if we left it off, it would be of type Streaming). It allowed us to create a report easily on top of it, but it has certain limitations. Just be aware when using this option and check Power BI service documentation for more info.

Finally, in SSMS, we executed a query with predefined data. Therefore in real life, if you want to push data from an existing table, you will have to build JSON from it. Make sure you construct a valid JSON and escape the symbols that need to be escaped (such as a double-quote or a backslash).

References

https://powerbi.microsoft.com/en-us/

https://docs.microsoft.com/en-us/rest/api/power-bi/

https://docs.microsoft.com/en-us/power-bi/service-real-time-streaming

https://docs.microsoft.com/en-us/power-bi/developer/api-rest-api-limitations

The post Push data into a Power BI dataset from SQL Server appeared first on ZappySys Blog.