In this article, you will learn how to create simple or complex JSON file in Informatica just in a few clicks. We will use the Powerful Export JSON Command line Builder from ZappyShell. Using this Portable / xCopy style command line you can build nested/complex JSON / XML or even CSV from single / multiple SQL Queries as Input or Select Database Tables as Source. Later you can copy generated command line from Tool and execute in any tool like Informatica / Talend / DataStage or Automation Script like Python / PowerShell.

Prerequisite

This article assumes you have access to Informatica Developer Tools and basic knowledge of command-line tools.

1. Download ZappyShell Command line utility (Click on Download Button). You can get msi and Zip (Poratble / xCopy version)
2. If you want to use the Portable version just extract ZappyShell-xCopy zip to some folder lets say (C:\ZappyShell). If you installed MSI then its usually in C:\Program Files (x86)\ZappySys
3. Double click on DbCommandBuilder.exe as listed here  to get started. At the end of this exercise, you will be able to generate something like the below JSON File in few clicks in Informatica or any other Tool / Programming Language you prefer which has option to invoke command line…
4. For all possible command line options Refer this
5. So lets get started

Sample JSON Created by ZappyShell from Informatica

Sample JSON Created by Informatica Workflow (Create From Database SQL Query or Tables)

Create Command Line to Export JSON File from Database SQL Query / Tables

In this section, we will see how to build Command Line which you can then execute in Informatica Command Time and later Schedule Export Job to Produce JSON File on a predefined schedule or manually.

1. Open DbCommandBuilder.exe and configure the Connection / Data source(s) as listed here .
2. Select JSON Export Option from the Dropdown. You can also choose XML, Excel or CSV
3. You can use the Custom Layout Option if you wish to Export using custom Output Layout else just choose tables you like to export or type SQL query.
   
4. Configure your Input Dataset(s) (can be SQL Queries or Tables) and define Desired Layout for your Output File.
5. Here is just an example of how to define an Array for a nested layout in a JSON or XML file.
6. t the end of configuration you may see command line like below (more examples listed here)Example: Simple mode Export (Microsoft SQL Server)
   

   db.exe export customers --json -o c:\data\cust.json --connstr "Data Source=localhost;Initial Catalog=Northwind;Integrated Security=SSPI;" -y

   Example: For Simple mode Export (MySQL SQL Server – ODBC Driver)
   

   export customers --out c:\data\customers.json --json --overwrite --connstr "Driver={MySQL ODBC 5.2 ANSI Driver};Server=your_server_name_or_ip;Port=3306;Uid=user1;pwd=SomeP@ssword;Database=your_db_name"

   Example: For custom layout mode Export
   
   

   export --map c:\data\MySampleMap.xml -o c:\data\customers.json --json --connstr "Data Source=localhost;Initial Catalog=Northwind;Integrated Security=SSPI;" -y

7. Once the Command line is created you can click Run to test and then copy it into tools like Informatica Command Task (see next section).

Export / Create JSON File In Informatica

1. Once the command line is prepared (see previous section) we can open Informatica Developer (D)
2. Connect to your Repository and create a new workflow
3. Drag new Command Task, from Toolbox and double click to edit
   

   Informatica Command Task

4. Enter the sample command line we obtained from the previous step
5. Save the workflow and run it… That’s it you will see the JSON file generated

Export / Create XML File In Informatica

Same Steps as above except choose File format as XML and you will see output like the below once you run the command line.

Export / Create XML File In Informatica

Export / Create CSV File In Informatica

Same Steps as above except choose File format as CSV and you will see output like the below once you run the command line.

Generate CSV Files in Informatica

Export / Create Excel File In Informatica

Same Steps as above except choose File format as Excel and you will see output like the below once you run the command line.

Conclusion

In this post, we just saw how easy it is to generate JSON / XML / Excel or CSV files in tools like Informatica, Talend, python, PowerShell in few clicks without coding. You can schedule job to generate JSON / XML files from Database tables or SQL Query. Download ZappyShell Today and try it to see what else you can do with it.

The post Export / Create JSON File in Informatica (from SQL Query / Database Tables) appeared first on ZappySys Blog.

Using SOAP API with SharePoint in SSIS is a common requirement to Administer SharePoint. SharePoint on-premises is a nice Microsoft application used to share documents and collaborate with the company. You can have schedules, projects, documents, and more shared using a Web platform.

In addition, it is a very useful tool that your company may need. It is also possible to automate, administer tasks using SOAP API.
In this article, we will learn how to create SOAP API queries to SharePoint on-premises.

Also, we will show our ZappySys SSIS PowerPack that includes very powerful tools to export SharePoint data from REST API to any other source.

	SSIS REST API Web Service Task
	SSIS XML Source (File, REST, SOAP Web Service)

Requirements

1. Firstly, you will need SSDT installed.
2. Secondly, SSIS PowerPack installed.
3. SoapUI installed.

Getting started

We will start with SoapUI application. It will help us to get all the possible methods SharePoint SOAP API allows. Also, it will show the inner workings of how the requests should look, which we will later replicate on XML Source. SoapUI will be our polestar.

How to call SharePoint SOAP API (On-Premises)

To call SOAP API in SSIS you have to perform the following steps at a high level. Detailed steps are described in the next section

1. Import WSDL file or URL for your Service in SoapUI for testing.
   1. File > New SOAP Project.
   2. In the Initial WSDL field enter
      http://your.domain.com/_vti_bin/Lists.asmx?wsdl
      or
      http://your.domain.com/my-site/_vti_bin/Lists.asmx?wsdl
2. Add credentials for SOAP Web service (Basic Auth or WSS)
3. Create SOAP Request XML (Edit default parameters) and Click Play Button to test your request. Associative photo:
   

   Create SOAP Request Body from WSDL (Using SoapUI tool)

4. If your test works in SoapUI then you are ready to move to SSIS part.
5. Copy Service URL, Request Body and Headers to Notepad — we will use it in SSIS.
   For Header see Raw Tab on SoapUI > Copy two lines Content-Type and SOAPAction.
   

   Get ContentType and SOAPAction Headers from SoapUI Raw tab

In the next section, we will see how to call SharePoint SOAP Web Service in SSIS.

How to read data from a SharePoint List via SOAP Web Service

Now let’s move onto SSIS and get some data from SharePoint using XML Source.

1. Open SSIS, create a new Integration project and create a new SSIS package.
2. Drag and drop Data Flow onto the Control Flow from SSIS Toolbox.
3. Then drag and drop XML Source onto Data Flow (you may also use other similar connectors such as REST API Task or Web API Destination).
4. Then configure XML Source in this way:
   
   1. Enter URL:
      https://my.sharepoint.site.com/sites/mysite/_vti_bin/Lists.asmx
      or
      https://my.sharepoint.site.com/_vti_bin/Lists.asmx (depends if you have many sites or just one).
   2. Check Use Credentials checkbox and create ZS-HTTP connection. Use Basic or NTLM authorization scheme.
   3. Then copy configuration from SoapUI to XML Source. E.g. HTTP Request Method, Body, Body Content-Type, HTTP Headers (e.g. SOAPAction header).
   4. Use $.soap:Envelope.soap:Body.GetListItemsResponse.GetListItemsResult.listitems.rs:data.z:row[*] as filter. If it doesn’t work, use Select Filter button to select the path manually and select row node.
5. What’s left is to connect XML Source with OLEDB Destination or a similar destination component:
   

Conclusion

To conclude, we can say that using SSIS PowerPack component XML Source and SoapUI we could achieve our goal of getting data from SharePoint On-Premises. The next step could be writing data to SharePoint via the same SOAP API interface, but this time using Web API Destination component.

The post Read data from SharePoint List in SSIS (On-Premises SOAP API) appeared first on ZappySys Blog.

In our previous blog we saw How to export XML from SQL Server using SSIS. Now in this blog, we will Parse XML string into multiple columns and rows using SSIS XML Parser Transform can (Helpful to extract data from raw XML string stored as database column or coming from other source). Ability to de-normalize nested XML data into flat structure. Support for expression to extract nested data and convert single node into multiple rows (e.g. extract orders from customer document using expression $.Customer.Orders[*])

In nutshell, this post will focus on how to Parse XML string into multiple columns and rows using respective XML Parser Transform.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Finally, do not forget to install ZappySys SSIS PowerPack

How to Extract Data from SQL Server Table and Parse xml string into multiple columns and rows.

Let´s start with an example. In this SSIS XML Parser Transform example, we will Parse database column string into multiple columns and rows.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. Furthermore, drag and drop the OLE DB Source.
   

   OLE DB Source – Drag and Drop

3. Double click on OLE DB Source for configure it and click on OK.
   

   Configure OLE DB Source Editor

4. Now drag and drop XML Parser Transform and connect it with the ole db source.
5. Double click on XML Parser Transform and select that column from the drop down and enter sample data and click on Preview.
   

   SSIS XML Parser Transform – Parse XML Document

6. That’s it, now you can use that data in desire destination.
   

   SSIS XML Parser Transform – Parse XML Document, Convert XML to CSV or save to RDBMS (e.g. SQL Server, MySQL)

Conclusion

After all, we saw you how to Parse string into multiple columns and rows using SSIS XML Parser Transform from SQL Server Table using OLE DB Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following URL for more information.

• Help File: XML Parser Transform

The post Parse XML string into multiple columns and rows using SSIS appeared first on ZappySys Blog.

There might be a case when you have many CSV, JSON or XML files in Amazon S3 bucket and you want them to be imported straight into a SQL Server table. Here come ZappySys ODBC PowerPack and ZappySys Data Gateway (part of ODBC PowerPack) which will enable you to accomplish that. ZappySys ODBC PowerPack includes powerful Amazon S3 CSV, Amazon S3 JSON and Amazon S3 XML drivers that let you connect to an S3 bucket and read the contents from the files in the bucket. Bringing in ZappySys Data Gateway allows doing that right from a SQL Server. Let’s begin and see how to import Amazon S3 files into SQL Server.

These drivers are related to this article:

Amazon S3 Driver (for XML Files)

Amazon S3 Driver (for JSON Files)

Amazon S3 Driver (for CSV Files)

Prerequisites

1. Have an AWS account.
2. Existing Amazon S3 bucket.
3. SQL Server instance installed (can be a SQL Server Express LocalDB instance).
4. SQL Server Management Studio (SSMS) installed.
5. ZappySys ODBC PowerPack installed (must be installed on a Windows machine; can be a different machine than SQL Server is installed on).

Getting started

We will start from a simple example of how to load contacts from a JSON that are located in Amazon S3 bucket, then we will proceed by loading contacts from many XML files that are compressed in ZIP, and finally, we will traverse many folders and subfolders to gather CSVs and load contacts from them.

Step-by-Step – Import JSON file located at Amazon S3 into SQL Server

Open and configure ZappySys Data Gateway

Create an Amazon S3 JSON data source in ZappySys Data Gateway

1. The first thing you will have to do is to create a data source in ZappySys Data Gateway. Just click Add button, give the data source a name, e.g. “MyContactsJSON“, and then select Native – ZappySys Amazon S3 JSON Driver:
   

   Adding a JSON data source in ZappySys Data Gateway to load it from Amazon S3 bucket into SQL Server

2. Then click Edit and add the Data Gateway user you created in Users tab. We will use this user later when adding a Linked Server to the Data Gateway to authenticate:
   

   Adding a user to the newly created Amazon S3 JSON data source

3. Once you do that, then click Edit to configure the data source:
   

   Preparing to configure the Amazon S3 JSON data source to load a JSON from Amazon S3 bucket to SQL Server

4. When a window opens, click Click here to Configure the Connection and enter your Access Key and Secret Key:
   

   Configuring the authentication to Amazon S3 bucket in ZappySys Data Gateway

5. After that, select a JSON file you want to load, and then click Select Filter button to choose data you want to be displayed in SQL Server:
   

   Configuring the JSON data source to load a JSON from Amazon S3 bucket into SQL Server

6. Go to the Preview tab and click the Preview Data button to make sure everything is configured correctly and preview the results:
   

   Previewing JSON data in the data source based on Amazon S3 JSON Driver

Now you are ready to create a Linked Server and connect to the data source you just created. If you used “MyContactsJSON” as the data source name, make sure to use the same name when creating a Linked Server.

Set up a SQL Server Linked Server

Execute the SQL query

Once you created the Linked Server to ZappySys Data Gateway, you are ready to execute the SQL query and load data into SQL Server. Supposedly, you created the Linked Server with the name “GATEWAY”, then open SSMS and execute this query:

SELECT * INTO MyContacts FROM openquery([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyContacts

The results

You should see similar results after you execute the query:

The results of getting JSON data straight from Amazon S3 bucket using ZappySys Amazon S3 JSON driver

Now we are ready to proceed to the next section and import many XML files at once.

Step-by-Step – Import many XML files located at Amazon S3 into SQL Server

Overview

Supposedly, you have many XMLs where each one is zipped and you want to load them all into a SQL Server table:

Zipped XML files located in Amazon S3 bucket to be loaded into SQL Server

The first thing you will have to do is to create a data source, based on ZappySys Amazon S3 XML Driver. Let’s proceed and just do that.

Create an Amazon S3 XML data source in ZappySys Data Gateway

Follow the same steps as in Create an Amazon S3 JSON data source in ZappySys Data Gateway section when adding a new data source, except for these two steps:

1. When adding a data source select Native – ZappySys Amazon S3 XML Driver as Connector Type:
   

   Choosing “ZappySys Amazon S3 XML Driver” to load XMLs to SQL Server

2. Then click Edit and edit the data source similarly like in this window:
   

   Configuring data source based on ZappySys Amazon S3 XML Driver to load XMLs into SQL Server

Now we are ready to set up a Linked Server to this newly created data source.

Set up a SQL Server Linked Server

Follow the very same steps from the section above where we added the Linked Server to a JSON data source, except that change Datasource property of the Linked Server to match the name of the Amazon S3 XML data source you created. Once you do that, you are ready to load XMLs into your SQL Server.

Execute the SQL query

Again, if you created the Linked Server with name “GATEWAY”, execute this SQL query in SSMS:

SELECT * INTO MyContacts FROM openquery([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyContacts

The results

You should see a similar view once you execute the query:

The data of many XMLs loaded from Amazon S3 bucket into SQL Server

Now we are ready to move to the next section and recursively scan CSVs and load them into SQL Server.

Step-by-Step – Import CSV file located at Amazon S3 into SQL Server

Overview

Let’s say you have many CSVs that are located in a folder tree, including subfolders and you want them all in your SQL Server:

Many CSV files located in Amazon S3 bucket folders and subfolders to be loaded into SQL Server

We will follow the same steps to accomplish that as we did for loading JSON and XMLs.

Create an Amazon S3 CSV data source in ZappySys Data Gateway

Follow the steps as in Create an Amazon S3 JSON data source in ZappySys Data Gateway section when adding a new data source, except for these two steps:

1. When adding a CSV data source select Native – ZappySys Amazon S3 CSV Driver as Connector Type:
   

   Choosing “ZappySys Amazon S3 CSV Driver” to load CSVs to SQL Server

2. Then click Edit and edit the data source similarly:
   

   Configuring data source based on ZappySys Amazon S3 CSV Driver to load CSVs into SQL Server

Now we are ready to proceed and add a Linked Server to this data source.

Set up a SQL Server Linked Server

Again, follow the very same steps from the section above where we added the Linked Server to a JSON data source, except that change Datasource property of the Linked Server to match the name of the Amazon S3 CSV data source we created. Once you do that, you are ready to load CSVs into your SQL Server.

Execute the SQL query

Once again, if you created the Linked Server with name “GATEWAY”, execute this SQL query in SSMS:

SELECT * INTO MyContacts FROM openquery([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyContacts

The results

You should see a similar view, once you execute the query:

The results of loading many CSVs from Amazon S3 bucket into SQL Server

Conclusion

We had a goal to import Amazon S3 files into SQL Server from SQL Server itself. We learned how to load JSON, XML and CSV files into SQL Server, which some of them were zipped and were located at different folders and subfolders. It was possible to accomplish that using ODBC PowerPack, ZappySys Data Gateway and ODBC PowerPack drivers – Amazon S3 JSON Driver, Amazon S3 XML Driver, and Amazon S3 CSV Driver. At first, we created data sources in the Data Gateway, then we created Linked Servers in SQL Server and connected to ZappySys Data Gateway, which finally made it possible to load data from Amazon S3 bucket straight from SQL queries in SSMS.

Check Amazon S3 Integration with Other BI Tools (Power BI, Excel, SSRS, MS Access, etc.)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

The post Import Amazon S3 files into SQL Server (CSV/JSON/XML Driver) appeared first on ZappySys Blog.

There might be a case when you have many CSV, JSON or XML files in Azure Blob and you want them to be imported straight into a SQL Server table. Here come ZappySys ODBC PowerPack and ZappySys Data Gateway (part of ODBC PowerPack) which will enable you to accomplish that. ZappySys ODBC PowerPack includes powerful Azure Blob CSV, Azure Blob JSON and Azure Blob XML drivers that let you connect to an Azure Storage Explorer and read the contents from the files in the Container. Bringing in ZappySys Data Gateway allows doing that right from a SQL Server. Let’s begin and see how to Import Azure Blob files into SQL Server (CSV/JSON/XML Driver).

These drivers are related to this article:

Azure Blob Driver (for CSV Files)

Azure Blob Driver (for JSON Files)

Azure Blob Driver (for XML Files)

Prerequisites

1. Firstly, Download and install Microsoft Azure Storage Emulator and Microsoft Azure Storage Explorer.
2. SQL Server instance installed (can be a SQL Server Express LocalDB instance).
3. SQL Server Management Studio (SSMS) installed.
4. ZappySys ODBC PowerPack installed (must be installed on a Windows machine; can be a different machine than SQL Server is installed on).

NOTE: If you want to use Live account (Azure Storage) then you can skip Step #3

Getting started

In order to start, we will show several examples. ZappySys includes Data Gateway that will help you in reading data of Azure Blob files. Here we are showing you is, how to Import Azure Blob files into SQL Server (CSV/JSON/XML Driver).

You can connect to Azure Storage Service from SSIS, you will need Storage Account Name and Access Key. Ask your SysAdmin or responsible person to provide that information to you. Click here to read more about how to get your Storage Account Name and Access Key. Here are sample Credentials.

Account Name: mystorageaccount
Access Key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

If you don’t have Azure Storage account then you can try offline mode on your local machine. You can just download Azure Storage Emulator and start testing.

Setup Azure Storage Explorer

Setup Azure Storage client tools we will start from a simple example of how to load Invoice Data from a CSV that is located in Azure Blob, then we will load JSON contacts data from them and finally, we will proceed by loading contacts from many XML files those are compressed in ZIP.

1. Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.

   

   Azure Storage Emulator Physical Location

2. If you can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.

   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, you have to download and install Microsoft Azure Storage Explorer and then you can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.

   

   Microsoft Azure Storage Explorer Location.

4. For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).
   

   Microsoft Azure Storage Explorer: Create a Blob Container

Open and Configure ZappySys Data Gateway

Import CSV file located at Azure Blob into SQL Server

Overview

Let’s say you have CSV file that is located in Azure Blob Container, and you want them all data into your SQL Server. We will follow the same steps for that we will do for import JSON and XML file into SQL Server.

Azure Blob Container – CSV File Located

Create an Azure Blob CSV data source in ZappySys Data Gateway

1. The first thing you will have to do is to create a data source in ZappySys Data Gateway. In the Data Source tab, Just click on Add button, give the data source a name, e.g. “MyInvoiceCSV“, and then select Native – ZappySys Azure Blob CSV Driver.
   

   Create Data Source – Azure Blob CSV Driver

2. Then click on Edit and add the Data Gateway user you created in the Users tab. We will use this user later when adding a Linked Server to the Data Gateway to authenticate:
3. Now, click on the Edit button to configure “MyInvoiceCSV” Azure Blob CSV Data Source.
   

   Configure Data Gateway

4. When a window open, click on Click here to Configure the Connection and select Use the Microsoft Azure Storage Emulator. If you have Online Storage Account and Account Key then Select Enter Storage Account Credentials and enter it.
   
   

   Create Azure Storage Connection

5. Now, Select Azure Blob Container and file from it and In the Data Format / Compression (Zip / GZip) tab set suitable file Compression Format (Zip or GZip).
   

   ZappySys Azure Blob CSV – Configure Driver

6. Go to the Preview tab and click on the Preview Data button to make sure everything is configured correctly and preview the results.
   

   ZappySys Azure Blob CSV – Preview Data

7. Now, Click on OK button also Save and Restart Service.
   

   ZappySys Data Gateway Service – Save and Restart

8. That’s all, now we are ready to proceed and add a Linked Server to this data source.

Set up a SQL Server Linked Server

Execute the SQL query

Once again, if you created the Linked Server with name “GATEWAY”, execute this SQL query in SSMS:

SELECT * INTO MyInvoices FROM OPENQUERY([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyInvoices

The results

You should see a similar view, once you execute the query:

The results of loading many CSVs from Azure Blob into SQL Server

Import JSON file located at Azure Blob into SQL Server

Overview

Let’s say you have JSON file that is located in Azure Blob Container, and you want them all data into your SQL Server. We will follow the same steps of the import CSV file into SQL Server and we will do for XML file into SQL Server.

Create an Azure Blob JSON data source in ZappySys Data Gateway

Follow the steps as in Create an Azure Blob CSV data source in ZappySys Data Gateway section when adding a new data source, except for these two steps:

1. The first thing you will have to do is to create a data source in ZappySys Data Gateway. Just click Add button, give the data source a name, e.g. “MyContactsJSON“, and then select Native – ZappySys Azure Blob JSON Driver.
   

   ZappySys Data Gateway – Create New Data Source
2. After that, select a JSON file from container you want to load, and then click Select Filter button to choose data you want to be displayed in SQL Server.
   

   Configuring the JSON data source to load a JSON from Azure Blob container into SQL Server

Now you are ready to create a Linked Server and connect to the data source you just created. If you used “MyContactsJSON” as the data source name, make sure to use the same name when creating a Linked Server.

Set up a SQL Server Linked Server

Again, follow the very same steps from the section above where we added the Linked Server to a CSV data source, except that change Catalog property of the Linked Server to match the name of the Azure Blob JSON data source we created. Once you do that, you are ready to load JSON into your SQL Server.

Execute the SQL query

Once you created the Linked Server to ZappySys Data Gateway, you are ready to execute the SQL query and load data into SQL Server. Supposedly, you created the Linked Server with the name “GATEWAY”, then open SSMS and execute the following query:

SELECT * INTO MyContacts FROM openquery([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyContacts

The results

You should see similar results after you execute the query:

The results of getting JSON data straight from Azure Blob JSON

Now we are ready to proceed to the next section and import many XML files at once.

Import many XML files located at Azure Blob into SQL Server

Overview

Supposedly, you have many XMLs where each one is zipped and you want to load them all into a SQL Server table.

Zipped XML files located in Azure Blob container to be loaded into SQL Server

The first thing you will have to do is to create a data source, based on ZappySys Azure Blob XML Driver. Let’s proceed and just do that.

Create an Azure Blob XML data source in ZappySys Data Gateway

Follow the same steps as in Create an Azure Blob JSON data source in ZappySys Data Gateway section when adding a new data source, except for these two steps:

1. When adding a data source select Native – ZappySys Azure Blob XML Driver as Connector Type.
   

   Choosing “ZappySys Azure Blob XML Driver” to load XMLs to SQL Server
2. Then click on Edit and select Azure Blob Container and file from it and In the Data Format / Compression (Zip / GZip) tab set suitable file Compression Format (Zip or GZip).
   

   Configuring data source based on ZappySys Azure Blob XML Driver to load XMLs into SQL Server

3. Now we are ready to set up a Linked Server to this newly created data source.

Set up a SQL Server Linked Server

Follow the very same steps from the section above where we added the Linked Server to a CSV data source, except that change Catalog property of the Linked Server to match the name of the Azure Blob XML data source you created. Once you do that, you are ready to load XMLs into your SQL Server.

Execute the SQL query

Again, if you created the Linked Server with name “GATEWAY”, execute the following SQL query in SSMS:

SELECT * INTO MyContacts FROM openquery([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyContacts

The results

You should see a similar view once you execute the query:

The data of many XMLs loaded from Azure Blob Container into SQL Server

Now we are ready to move to the next section and recursively scan CSVs and load them into SQL Server.

Conclusion

We had a goal to import Azure Blob files into SQL Server from SQL Server itself. We learned how to Import Azure Blob files into SQL Server (CSV/JSON/XML Driver), which some of them were zipped and were located at different folders and subfolders. It was possible to accomplish that using ODBC PowerPack, ZappySys Data Gateway, and ODBC PowerPack drivers – Azure Blob JSON Driver, Azure Blob XML Driver, and Azure Blob CSV Driver. At first, we created data sources in the Data Gateway, then we created Linked Servers in SQL Server and connected to ZappySys Data Gateway, which finally made it possible to load data from Azure Blob straight from SQL queries in SSMS.

References

Finally, you can use the following links for more information:

• Azure Blob CSV Driver
• Azure Blob JSON Driver
• Azure Blob XML Driver

The post Import Azure Blob files into SQL Server (CSV/JSON/XML Driver) appeared first on ZappySys Blog.

In our previous blog, we saw how to read/write Azure Blob Storage Files in SSIS (CSV, JSON, XML Format files). Now in this blog, we will see how to Read/Write Azure Table Storage data in SSIS. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like to perform other operations on Azure Blob Storage Files (e.g. Download, Upload, Create, Delete) then check these articles.

Component Mentioned in this article

	Azure Table Storage Source
	Azure Table Storage Destination

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Download and install Microsoft Azure Storage Emulator
4. Download and install Microsoft Azure Storage Explorer
5. Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Table Storage?

Azure Table storage stores large amounts of structured data. The service is a NoSQL datastore which accepts authenticated calls from inside and outside the Azure cloud. Azure tables are ideal for storing structured, non-relational data. Common uses of Table storage include:

Microsoft Azure Table Storage Introduction

• Storing TBs of structured data capable of serving web-scale applications
• Storing datasets that don’t require complex joins, foreign keys, or stored procedures and can be denormalized for fast access
• Quickly querying data using a clustered index
• Accessing data using the OData protocol and LINQ queries with WCF Data Service .NET Libraries

You can use Table storage to store and query huge sets of structured, non-relational data, and your tables will scale as demand increases.

Getting Started

In order to start, we will show several examples. Zappysys includes an SSIS Azure Table Storage Source. Here we are showing you is, how to Read/Write Azure Table Storage data in SSIS.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of the local Azure Storage Emulator.

Setup Azure Storage client tools and Create Table

1. Once you have downloaded and installed a storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
   

   Azure Storage Emulator Physical Location

2. If you can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, you have to download and install Microsoft Azure Storage Explorer and then you can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Explorer Location.

4. If you don’t have Azure Storage account then it’s easy to get FREE Trial Azure Storage account or use your MSDN to get credit each month which will be more than enough for real testing. If you prefer to avoid all this hassle then Microsoft provides another great way to test Azure Storage functionality is totally offline mode on your local machine. You can just download Azure Storage Emulator and start testing.
5.  Click here to learn more about how to configure Azure Storage Emulator for Testing and Development purposes. In order to connect to Azure Storage Service from SSIS, you will need Storage AccountName and an AccessKey. Ask your SysAdmin or responsible person to provide that information to you. Here are sample Account Name and Access Key(this is just example key which may differ in your case)
   

   Account Name: mystorageaccount 
   Access Key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

6. For Creating a Tables, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> Emulator – Default Ports(Key) –> Tables. Right-click on Tables and Select Create Table and put your table name you want.
   

   Create Tables in Azure Table Storage Emulator

Write data using ZS Azure Table Storage Destination

In this tutorial, you will learn how to write data into Azure Table Storage. You can use ZS DummyData Source for practice it(In this case it’s from SQL Server Source). So, Let’s start with an example. In this SSIS we will write data into Azure Table Storage using ZS Azure Table Storage Destination.

1. Once you have created Azure Storage Table and you got valid Account Name and AccessKey (for the local emulator you don’t need this) you may proceed to create a new SSIS package. In BIDS/SSDT create new SSIS package
2. From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

3. Double click on the DataFlow task to see DataFlow designer surface.
4. Here, In Visual Studio, drag and drop the OLE DB Source and Azure Table Storage Destination in the design surface and join the components with the blue arrow.
   

   SSIS Azure Table Storage Destination – Drag and Drop

5. Now, we need two connection, OLE DB Connection, and Azure Storage Connection. Click here to create Azure Storage Connection.

How to create OLE DB Connection

1. Let’s, Right-click on Connection Managers Panel to Create OLE DB Connection, so you can use Source and Context Menu will appear, Select New OLE DB Connection from the Context Menu.
   

   OLE DB Connection – Create New Connection

2. Now, click on New Button to create Connection.
   

   OLE DB Connection – Create New Connection

3. Let’s Configure Connection Manager, just Follow steps one by one as we have created.
   

   OLE DB Connection – Configure Connection

4. Click on the OK button to save configure Setting UI.

Write data into Azure Storage Table in SSIS

1. Double click on OLE DB Source for configure it.
2. Let’s Configure in Connection Manager, just follow the below image steps.
   

   OLE DB Source – Configure

3. Click OK to Save OLE DB Source Editor UI Settings.
4. Now double click Azure Table Storage Destination to configure it.
5. On [Connection Manager] tab select Azure Storage Connection manager.
   

   ZS Azure Table Storage Destination – Connection Manager

6. Click on [Component Properties] tab and select AccessMode=Table and Pick destination Table from the Tables dropdown list.
   
   If you choose Reload option then target table is first truncated before inserting records so be careful with Reload option.
   

   ZS Azure Table Storage Destination – Component Properties

7. Click on [Mappings] tab and verify mapping. Drop CustomerID and RecordID mappings and reconnect as shown below. This is just to make sure PrimaryKey columns are mapped.
   

   ZS Azure Table Storage Destination – Column Mappings

8. Click OK to save settings.
9. Execute the package and verify the target data.
   

   ZS Azure Table Storage Destination – Execute the Package

   

   ZS Azure Table Storage – Verify Customers Table Data

Read data using ZS Azure Table Storage Source

In this tutorial, you will learn how to read data from Azure Table Storage in SSIS using the ZS Azure Table Storage Source adapter. In this SSIS we will read data with SQL Query and Table Mode. So Let’s begin it.

1. You may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
2. From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
   

   SSIS Data Flow Task – Drag and Drop

3. Double click on the DataFlow task to see DataFlow designer surface.
4. From the SSIS toolbox drag and drop Azure Table Storage Source on the data flow designer surface.
   

   ZS Azure Table Storage Source – Drag and Drop

5. Now, we need an Azure Storage connection. Click here to Create Connection.

Extract data with Query Mode

1. Here, we are getting data using Query Mode with Dynamic expression value.
2. Lets, create a Variable with correct DataType and Value, use the following image.
   

   SSIS Variables – Create Variable and Store Value

3. Double click Azure Table Storage Source to configure it.
4. Select the connection we have created before, set AccessMode to Query. You can use the following SQL Query and select Variable. Click here for more Azure Table Storage Query Examples.
   

   SELECT * FROM Customers
   WHERE Country = '{{User::varCountry}}'

   

   ZS Azure Table Storage Source – Configure Query Mode

5. Click on the Preview button to see Data Preview and OK button to save configure setting UI.
6. From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface.
   

   ZS Trash Destination – Drag and Drop

7. Now single click on the Azure Table Storage Source, once you see the blue arrow from the source … connect it to Trash Destination.
8. Double click on ZS Trash Destination to Configure it.
   

   ZS Trash Destination – Configure

9. Click on the OK button to save Trash Destination configure setting UI.
10. Execute the package and verify source data in the data viewer.
    

    SSIS ZS Azure Table Storage Source – Execute the Package

Extract data with Table Mode

1. So, Double click Azure Table Storage Source to configure it.
2. In the Azure Table Storage Source, Select the connection we have created, set AccessMode to Table and Select Table from the Tables dropdown.
   

   SSIS ZS Azure Table Source – Configure Table Mode

3. Click on the Preview button to see Data Preview and OK button to save configure setting UI.
4. From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface.
   

   ZS Trash Destination – Drag and Drop

5. Now single click on the Azure Table Storage Source, once you see the blue arrow from the source … connect it to Trash Destination.
6. Double click on ZS Trash Destination to Configure it.
   

   ZS Trash Destination – Configure

7. Click on the OK button to save Trash Destination configure setting UI.
8. In the last, Execute the package and verify source data in the data viewer.
   

   ZS Azure Table Storage Source – Execute the Package

Conclusion

After all, In this blog, we learned how to write data in Azure Table Storage in SSIS. We used the Azure Table Storage Destination for write and Azure Table Storage Source to read data from Microsoft Azure Table Storage using Query and Table mode. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

There are a few details you should use the following links for more information:

• Azure Table Storage Destination
• Azure Table Storage Source
• ZS Trash Destination
• Microsoft Azure Table Storage
• Help File:
  • Azure Table Storage Destination
  • Azure Table Storage Source
  • ZS Trash Destination

The post How to Read/Write Azure Table Storage data in SSIS appeared first on ZappySys Blog.

In our previous blog we saw how to read JIRA data in SQL Server. Now let’s learn how to Import SAP S/4HANA OData Service Data Into MS Access. SAP S/4HANA provides OData REST API interface to access data in your application using HTTP Protocol. We will use ODBC XML Driver to read SAP data and load into MS Access.

So, let’s get started.

About SAP HANA / OData REST API Service

You can expose your SAP Data using ODATA REST API Service. Here is a good article which shows how to expose data as OData Service.   For more information on SAP OData Service feature check this SAP help page. If you are new to OData Standard then read here to know more how OData can facilitate data extraction using HTTP REST API. If you need Sample XML based OData Service then use below test URLs.

https://services.odata.org/Northwind/Northwind.svc/
https://services.odata.org/Northwind/Northwind.svc/Customers
https://services.odata.org/Northwind/Northwind.svc/Orders
https://services.odata.org/Northwind/Northwind.svc/Invoices
https://services.odata.org/Northwind/Northwind.svc/Products

Now let’s look at how to read SAP Data using ODBC Driver. At this point we assume you have exposed your data as OData Endpoint.

Read SAP S/4HANA data using XML Driver

First let’s look at steps to configure XML Driver. We will read data from SAP S/4HANA OData Service and then in next section we will look at how to load data into MS Access or other target.

1. First, Windows search, write ODBC and select the ODBC Data sources (32 bits)
   

   Open ODBC Data Source

2. As a second step, in ODBC Data source Administrator press the Add button.
   

   Add ZappySys

3. In this step, create the new data source, select ZappySys XML Driver.
   

   Add new zappysys xml driver

4. Now edit that XML data source to configure it. Enter your OData Service URL its typically like below. Replace 3 parts with your own value (i.e. replace {MY-INSTANCE},  {MY-PROJECT}, {MY-TABLE})
   

   https://{MY-INSTANCE}/sap/opu/odata/sap/{MY-PROJECT}/{MY-TABLE}

   For example if you are hosting SAP HANA in Cloud Instance then your URL may look like below
   

   https://myXXXXXX-api.s4hana.ondemand.com/sap/opu/odata/sap/MyTestProject/PurchaseOrders

   And create New ZS-HTTP connection in it select Basic Authentication and enter your SAP HANA UserID / Password to call OData Service and select the desire filter.
   

   XML Driver : SAP S/4HANA OData Service Configuration

5. Now go to Data Format / Compression (Zip/GZip) tab and select Data Format as OData to get all the records.
   

   Data Format : Odata

6. Finally, now using Query Builder and Code Generator we will generate the query and click on Preview the data.
   

   Generate Query

7. That’s it we are ready to load SAP S/4HANA OData Service data to MS Access.

Load SAP S/4HANA OData Service in MS Access

1. In MS Access, go to External Data Ribbon and select New Data Source and select From Other Sources and ODBC Database.
   

   A configuration in MS Access: Import Data From REST API

2. Select the Source and destination of the data, select Import the source data into a new table in the current database and press OK.
   

   Get External Data – ODBC Database

3. Select your newly created JSON Driver Data Source and click OK.
   
   

   Get External Data – Select your newly created ODBC Data Source (JSON Driver)

    

4. Select tables from the list of tables in the Import Objects screen and click OK.
   
   

   Get External Data – Select tables from Import Objects Screen

5. Select Save Import Steps from the screen and click on Save Import button.
   

   Get External Data – Select Save Import Steps

6. That’s it If everything is OK, you will be able to see the data.
   

   Data Imported Successfully in MS Access DB Table

How to link REST API data to Access

The linking process is very similar to importing data into the Access table. Follow steps of the previous paragraph – How to import REST API data to Access – but on step 2 select the second option to create a linked table instead. Then at last it’s asked to select a unique record identifier, don’t select anything:

Linked table approach – Don’t select a unique Record identifier

SAP S/4HANA Integration with Other BI Tools

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

So in this blog, we learned how to Import SAP S/4HANA OData Service Data in Access using ODBC XML / SOAP API Driver in a very simple way. You can achieve many more functionalities with this tool. Check our blogs/articles on XML File / REST API Driver to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the use of SAP S/4HANA OData Service with our tools:

• SAP API Business Hub
• Landing Page for ODBC XML / SOAP API Driver, you can also find Tutorial Video here.
• Help File: Documentation of XML Driver.
• Blog/articles link: https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/xml-soap-api-driver/

The post Import SAP S/4HANA OData Service Data Into Ms Access via ODBC Driver appeared first on ZappySys Blog.

In our previous blog we saw how to read JIRA data in SQL Server. Now let’s learn how to Import SAP S/4HANA OData Service Data Into Sql Server. SAP S/4HANA provides OData REST API interface to access data in your application using HTTP Protocol. We will use ODBC XML Driver to read SAP data and load into SQL Server.

So, let’s get started.

About SAP HANA / OData REST API Service

You can expose your SAP Data using ODATA REST API Service. Here is a good article which shows how to expose data as OData Service.   For more information on SAP OData Service feature check this SAP help page. If you are new to OData Standard then read here to know more how OData can facilitate data extraction using HTTP REST API. If you need Sample XML based OData Service then use below test URLs.

https://services.odata.org/Northwind/Northwind.svc/
https://services.odata.org/Northwind/Northwind.svc/Customers
https://services.odata.org/Northwind/Northwind.svc/Orders
https://services.odata.org/Northwind/Northwind.svc/Invoices
https://services.odata.org/Northwind/Northwind.svc/Products

Now let’s look at how to read SAP Data using ODBC Driver. At this point we assume you have exposed your data as OData Endpoint.

Read SAP S/4HANA data using XML Driver

First let’s look at steps to configure XML Driver. We will read data from SAP S/4HANA OData Service and then in next section we will look at how to load data into SQL Server or other target.

1. To do this, first of all, we will open the ZappySys Data Gateway Configuration:
   

   Open ZappySys Data Gateway

2. Add the Native – ZappySys XML Driver Data source.
   

   Add Gateway Data Source

3. Now edit that XML data source to configure it. Enter your OData Service URL its typically like below. Replace 3 parts with your own value (i.e. replace {MY-INSTANCE},  {MY-PROJECT}, {MY-TABLE})
   

   https://{MY-INSTANCE}/sap/opu/odata/sap/{MY-PROJECT}/{MY-TABLE}

   For example if you are hosting SAP HANA in Cloud Instance then your URL may look like below
   

   https://myXXXXXX-api.s4hana.ondemand.com/sap/opu/odata/sap/MyTestProject/PurchaseOrders

   And create New ZS-HTTP connection in it select Basic Authentication and enter your SAP HANA UserID / Password to call OData Service and select the desire filter.
   

   XML Driver : SAP S/4HANA OData Service Configuration

4. Now go to Data Format / Compression (Zip/GZip) tab and select Data Format as OData to get all the records.
   

   Data Format : Odata

5. Finally, now using Query Builder and Code Generator we will generate the query.
   

   Generate Query

6. That’s it we are ready to load SAP S/4HANA OData Service data to SQL Server.

Load SAP S/4HANA OData Service in MS SQL Server

SAP S/4HANA Integration with Other BI Tools (Power BI, Excel, SSRS, MS Access…)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

So in this blog, we learned how to Import SAP S/4HANA OData Service Data in MS SQL Server using ODBC XML / SOAP API Driver in a very simple way. You can achieve many more functionalities with this tool. Check our blogs/articles on XML File / REST API Driver https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/xml-soap-api-driver/ to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the use of SAP S/4HANA OData Service with our tools:

• SAP API Business Hub
• Landing Page for ODBC XML / SOAP API Driver, you can also find Tutorial Video here.
• Help File: Documentation of XML Driver.
• Blog/articles link: https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/xml-soap-api-driver/

The post Import SAP S/4HANA OData Service Data Into Sql Server via ODBC Driver appeared first on ZappySys Blog.

In our previous blog post we saw how to import REST / SOAP API in SQL Server. Using same concepts let’s look at how to import Bing Ads data into SQL Server. We will explore many techniques to call Bing Ads API and learn how to automate data extraction without doing any ETL. You can call Bing Ads API just using T-SQL code (Yes you heard it right). At the end of this article you will learn how to Download Performance Reports from Bing Ads Account without any coding (See sample screenshot below). Please go through full article carefully.

NOTE: Bing Ads now renamed as Microsoft Advertising but this article will use popular name for time being.

Import Bing Ads data into SQL Server (Using SSMS T-SQL Code)

About Bing Ads API / SOAP Web Service

If you are new to Bing Ads API then start from here. Bing Ads APIs are SOAP XML APIs. You can call it to automate many scenarios for Bing Ads. In this article we will mainly focus on read scenario (e.g. Read Performance Data) but you can use these techniques to write / update (e.g. Create New Campaign).

If you are new to SOAP WebService then Check this article to learn more. It will explain you how to use tools like SoapUI tool to craft SOAP Requests for Bing API or any other SOAP API.

For Bing API you can use Service WSDL files found here. Use URL ending ?wsdl to import in SoapUI.

Example Bing API Request

Here is raw API request for Bing API. This sample assumes you have completed 3-Legged OAuth Authorization to obtain AccessToken and RefreshToken (Explained later in this article). Below request uses AccessToken you might have already obtained via process like this. If you are using ZappySys tools then this will be generated automatically for you.

Request Method / URL:

POST https://clientcenter.api.bingads.microsoft.com/Api/CustomerManagement/v13/CustomerManagementService.svc

Headers:
Notice that for each Api you may need correct SOAPAction in Header. If you are not sure then use SoapUI Tool. Execute request in SoapUI and then go to Raw tab like this one.

Content-Type: text/xml; charset=utf-8
SOAPAction: "GetUser"

Body:
Notice that you have to replace AuthenticationToken and DeveloperToken in below request. AccessToken is placed inside AuthenticationToken tag. This Auth token is short lived and extracted via OAuth Process and DeveloperToken you can extract as mentioned in the previous section or get it from here.

  
    GetUser
    <%access_token%>
    1052Bxxxxxxxxxxxxxx
  
  
    
      
    
  

Response:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Header>
    <h:TrackingId xmlns:h="https://bingads.microsoft.com/Customer/v13">fddf3cb9-e4c2-4e75-9546-f3bc20ea91a8</h:TrackingId>
  </s:Header>
  <s:Body>
    <GetUserResponse xmlns="https://bingads.microsoft.com/Customer/v13">
      <User xmlns:a="https://bingads.microsoft.com/Customer/v13/Entities" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
        <a:ContactInfo>
          <a:Address>
            <a:City i:nil="true" />
            <a:CountryCode>US</a:CountryCode>
            <a:Id i:nil="true" />
            <a:Line1 i:nil="true" />
            <a:Line2 i:nil="true" />
            <a:Line3 i:nil="true" />
            <a:Line4 i:nil="true" />
            <a:PostalCode i:nil="true" />
            <a:StateOrProvince i:nil="true" />
            <a:TimeStamp i:nil="true" />
            <a:BusinessName i:nil="true" />
          </a:Address>
          <a:ContactByPhone>false</a:ContactByPhone>
          <a:ContactByPostalMail>false</a:ContactByPostalMail>
          <a:Email>someone@zappysys.com</a:Email>
          <a:EmailFormat i:nil="true" />
          <a:Fax i:nil="true" />
          <a:HomePhone i:nil="true" />
          <a:Id>48045678</a:Id>
          <a:Mobile i:nil="true" />
          <a:Phone1>111-222-3333</a:Phone1>
          <a:Phone2 i:nil="true" />
        </a:ContactInfo>
        <a:CustomerId>19112345</a:CustomerId>
        <a:Id>48012345</a:Id>
        <a:JobTitle i:nil="true" />
        <a:LastModifiedByUserId>48012345</a:LastModifiedByUserId>
        <a:LastModifiedTime>2019-06-04T14:22:35.38</a:LastModifiedTime>
        <a:Lcid>EnglishUS</a:Lcid>
        <a:Name>
          <a:FirstName>Someone</a:FirstName>
          <a:LastName>Good</a:LastName>
          <a:MiddleInitial i:nil="true" />
        </a:Name>
        <a:Password i:nil="true" />
        <a:SecretAnswer i:nil="true" />
        <a:SecretQuestion>None</a:SecretQuestion>
        <a:UserLifeCycleStatus>Active</a:UserLifeCycleStatus>
        <a:TimeStamp>AAAAAJiX/cU=</a:TimeStamp>
        <a:UserName>someone@zappysys.com</a:UserName>
        <a:ForwardCompatibilityMap i:nil="true" xmlns:b="http://schemas.datacontract.org/2004/07/System.Collections.Generic" />
      </User>
      <CustomerRoles xmlns:a="https://bingads.microsoft.com/Customer/v13/Entities" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
        <a:CustomerRole>
          <a:RoleId>41</a:RoleId>
          <a:CustomerId>19160000</a:CustomerId>
          <a:AccountIds i:nil="true" xmlns:b="http://schemas.microsoft.com/2003/10/Serialization/Arrays" />
          <a:LinkedAccountIds xmlns:b="http://schemas.microsoft.com/2003/10/Serialization/Arrays" />
          <a:CustomerLinkPermission i:nil="true" />
        </a:CustomerRole>
      </CustomerRoles>
    </GetUserResponse>
  </s:Body>
</s:Envelope>

POST http://myhost/api/v2/customers/new
X-SecretKey: Abcdxxxxxxxxxxxxxx
Content-Type: application/json

{ firstname: "Someone", lastname: "Good" }

{
 request_status: "CreatedOK",
 new_customer_id: "1234566"
}

Step-By-Step: How to Import Bing Ads Data into SQL Server

Now let’s look at how to import Bing Ads Data into SQL Server Table. This tutorial will require 10-15 mins of your time. At the end of this tutorial you will be able to Read Bing Ads Performance Report and Import data into SQL Server. Step listed in this tutorial may help you to call Bing API in other tools like SSIS, Power BI, SSRS, Tableau and so on.

So let’s get started.

Register OAuth App for Bing Ads API (Azure Active Directory App)

Very first thing to call any Bing API is to register OAuth App (Under Azure Active Directory Tab).

Here is how to register OAuth App which can be used to call Bing Ads API later on.

1. Login to Azure Portal:
   • Navigate to the Azure Portal and log in using your credentials.
2. Access Azure Active Directory:
   • In the left-hand menu, click on Azure Active Directory.
3. Register a New Application:
   • Go to App registrations and click on New registration.
     
   • Application Name: Enter a name for your application.
     
   • Supported Account Types: Choose the account types your app will support. For example, select Accounts in this organizational directory only if you need access to data in your organization only.
   • Redirect URI:
     • Set the type to Web.
     • In the textbox enter https://login.microsoftonline.com/common/oauth2/nativeclient as the Redirect URI or any other valid redirect URL, e.g., https://zappysys.com/oauth.
     • Use this Redirect URI in the Redirect URL grid row.
4. Save Client ID:
   • After registering the app, copy the Application (client) ID and paste it into the Client ID field in the API Connection Manager configuration.
5. Set Authorization & Token URLs:
   • Click on the Endpoints link in the App registration overview.
   • Authorization URL: Copy the OAuth 2.0 authorization endpoint (v2) URL (e.g., https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/authorize) and paste it into the Authorization URL field in the configuration grid.
   • Token URL: Copy the OAuth 2.0 token endpoint (v2) URL (e.g., https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/token) and paste it into the Token URL field.
6. Create a Client Secret:
   • In the Certificates & secrets tab, click New client secret.
   • Set an expiration period for the secret.
   • Copy the generated client secret and paste it into the Client Secret field in the API Connection Manager configuration.
7. Configure API Permissions:
   • Go to the API Permissions section.
   • Click on Add a permission, select Microsoft Graph, and choose Delegated Permissions.
   • Add the required permissions:
     • offline_access
     • openid
     • profile
     • Sites.Read.All
     • Sites.ReadWrite.All
     • User.Read
     • email
     

     Azure AD App – Required API Permissions

   • Grant Admin Consent for the permissions that require it.
8. Generate Tokens:
   • Use the Generate Token feature in the API Connection Manager to generate authentication tokens.
9. Use a Generic Account for Automation:
   NOTE: If you are planning to use your current data connection/token for automated processes, we recommend that you use a generic account for token generation when the login box appears (e.g. sales_automation@mycompany.com instead of bob_smith@mycompany.com). When you use a personal account which is tied to a specific employee profile and that employee leaves the company, the token may become invalid and any automated processes using that token will fail. Another potentially unwanted effect of using a personal token is incorrect logging; the API calls (e.g. Read, Edit, Delete, Upload) made with that token will record the specific user as performing the calls instead of an automated process.
10. That’s it!

Get Bing API Developer Token

Once we have OAuth App registered next thing is to obtain Developer Token for Bing Ads API. Click here to request new token or view existing Developer token.

Create new Bing Ads API Data Source in Gateway

Once you setup Data Gateway user we can move to next step. To call Bing API we need to setup Connection for XML Driver in ZappySys Data Gateway.

1. Click on Add New Data source. Name as BingAds and select Native – ZappySys XML Driver option.
   

   Add Gateway Data Source

2. Now click Edit to Configure Settings. For now we will only care OAuth Connection settings. All other options are overwritten in SQL Query.
   

   Edit Gateway Data Source Settings

3. When UI opens Click on Load ConnectionString and Paste below connection string after changing some items (Screenshot may be differ). Make sure to replace following attributes.RefreshTokenFilePath – Enter some valid file path which will hold refresh token
   ClientId – get Application Id from Azure Portal under App we created earlier)
   DeveloperToken – Replace 1052Bxxxxxxxxxxxx with your own devloper token.
   

   Load ZappySys Driver ConnectionString to configure UI

   Use below string and replace parameters 

   DRIVER={ZappySys XML Driver};
   DataPath='https://clientcenter.api.bingads.microsoft.com/Api/CustomerManagement/v13/CustomerManagementService.svc';
   DataConnectionType=OAuth;
   AuthUrl='https://login.microsoftonline.com/common/oauth2/v2.0/authorize';
   TokenUrl='https://login.microsoftonline.com/common/oauth2/v2.0/token';
   ReturnUrl='https://login.microsoftonline.com/common/oauth2/nativeclient';
   RefreshTokenFilePath='c:\xxxxxx_some_folder_xxxxxxx\BingAds_RefreshToken.txt';
   ScopeSeparator='{space}';
   Scope='https://ads.microsoft.com/ads.manage offline_access';
   ClientId='xxxxxxxxxxxxxx';
   UseCustomApp=True;
   RequestData='<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
     <s:Header>
       <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Customer/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Customer/v13"><%access_token%></h:AuthenticationToken>
       <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Customer/v13">1052Bxxxxxxxxxxxx</h:DeveloperToken>
       <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Customer/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Customer/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
     </s:Header>
     <s:Body>
       <GetUserRequest xmlns="https://bingads.microsoft.com/Customer/v13">
         <UserId i:nil="true" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       </GetUserRequest>
     </s:Body>
   </s:Envelope>';
   RequestContentTypeCode=TextXml;
   RequestMethod='POST';
   RequestHeaders='SOAPAction: "GetUser"'

4. Next to OAuth Click on Configure Settings
5. By default most of settings on OAuth connections should be pre-populated for you but still lets Notice few things. This is how to setup redirect URI
6. 

   Configure Redirect URI for Bing Ads API OAuth Connection

7. Here is where you setup Changing Refresh Token File path. if you dont supply this path then your refresh token extracted first time will expire in 60 days.
   

   Refresh Token File path (Only needed if API issues new Refresh Token when you request AccessToken)

8. Once you verify all settings click Generate Token and Finish the process. It will populate AccessToken and RefreshToken Fields if everything goes well.
   

   Configure Bing Ads API Connection (OAuth 2.0)

9. Click OK to save OAuth Connection UI to go back to main UI
10. You can now Click Test Connection to confirm everything
11. If you need to access Gateway from Other machine then go to Firewall tab and Click Add Rule
12. Click Save button on Gateway UI to save and restart the service.

Create a new Linked Server for Bing Ads API

Once we setup setup Bing Ads Data Source in Gateway we can move to next step which is create a Linked Server which points to Data gateway Data source we created. This will create a bridge between ZappySys Driver and SQL Server so you can call API via T-SQL.

1. If you are using Latest ODBC PowerPack then on Data gateway UI you will See App Integration Tab. Copy Code and run it in SSMS to create a new linked server.
   

   ZappySys Data Gateway code generator for Linked Server

2. If things go well when you execute code you may see some result in grid.

Run sample Queries to call Bing API in SQL Server / Import Data

Once Linked Server is created we can run various queries like below.

Example-1

--use all default settings from gateway data source UI
--Read
Select * from OPENQUERY( [YourLinkedServer] , 'select * from $') 

/*
--import
Select * into #tempTable from OPENQUERY( [YourLinkedServer] , 'select * from $') 
Select * from #tempTable
*/

Example-2

--use Filter to extract specific Node from response XML
Select * from OPENQUERY( [YourLinkedServer] , 
'select * from $
WITH(Filter=''$.s:Envelope.s:Body.GetUserRequest[*]'', 
  ElementsToTreatAsArray=''GetUserResponse'' 
   )'
)

Example-3

--Override URL, Headers and Body Settings. Change DeveloperToken in below request
--Header, Filter, URL, Request data is different for each Endpoint

SELECT * FROM OPENQUERY([YourLinkedServer]
, 'SELECT * FROM $
WITH(
	 ElementsToTreatAsArray=''GetUserResponse''
	,Src=''https://clientcenter.api.bingads.microsoft.com/Api/CustomerManagement/v13/CustomerManagementService.svc''
	,Filter=''$.s:Envelope.s:Body.GetUserResponse[*]''	
	,Header=''SOAPAction: "GetUser"''	
	,RequestData=''<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v13="https://bingads.microsoft.com/Customer/v13">
<soapenv:Header>
   <v13:DeveloperToken>1052xxxxxxxxxxxxx</v13:DeveloperToken>
   <v13:AuthenticationToken><%access_token%></v13:AuthenticationToken>
</soapenv:Header>
<soapenv:Body>
   <v13:GetUserRequest>
      <v13:UserId xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/>
   </v13:GetUserRequest>
</soapenv:Body>
</soapenv:Envelope>''
)')

Download Bing Ads Performance Data in SQL Server Table

Now lets look at more complex example to read data from Bing Performance Report. Calling This API is tricky due to many steps involved in data extraction. But not to worry if you use ZappySys Drivers. This is API is somewhat similar as Amazon MWS Report Request (See last section).

Here are high level steps needs to be performed to call API for Bing Ads Report (e.g. Performance Report)

1. Call SubmitGenerateReportResponse API to request new Report. This returns ReportRequestId
2. Now we have to keep checking every few seconds see report is ready or not. Call PollGenerateReport until Status field is Success from Pending
3. Once previous step returns Success extract Report Download URL (Its Zip File which may contain data in CSV or XML)

So lets get started to implement above login in a single T-SQL Query and load data into Table. There is a known limitation with Query Size we can send to data gateway from SSMS so we can use some tricks to read Body from File rather than hard coding in SQL. We can use IsMultiPart=True and then supply Filepath in RequestBody (Path must start with @ symbol)

1. Bing Report API needs 2 additional values along with DeveloperToken we got earlier. You need to supply AccountId and CustomerId part of Body so obtain it by visiting below page in your bing account.
   Visit https://ads.microsoft.com/cc/accounts
   

   Obtain Bing Ads AccountId, ClientId for APi calls

2. Create new XML file for first step in a Notepad. Enter below content and save as C:\Requests\BingAds\sql_body1.xml
   Replace Following Values as per your settingsCustomerAccountId (Found 2 places)
   CustomerId
   DeveloperToken
   

   <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
     <s:Header>
       <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Reporting/v13"><%access_token%></h:AuthenticationToken>
       <h:CustomerAccountId xmlns:h="https://bingads.microsoft.com/Reporting/v13">481xxxxxxxxxxx</h:CustomerAccountId>
       <h:CustomerId xmlns:h="https://bingads.microsoft.com/Reporting/v13">191xxxxxxxxxxx</h:CustomerId>
       <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Reporting/v13">1052Bxxxxxxxxxxx</h:DeveloperToken>
       <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
     </s:Header>
     <s:Body>
       <SubmitGenerateReportRequest xmlns="https://bingads.microsoft.com/Reporting/v13">
         <ReportRequest i:type="CampaignPerformanceReportRequest" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
           <ExcludeColumnHeaders>false</ExcludeColumnHeaders>
           <ExcludeReportFooter>false</ExcludeReportFooter>
           <ExcludeReportHeader>false</ExcludeReportHeader>
           <Format>Xml</Format>
           <ReportName>My Campaign Performance Report</ReportName>
           <ReturnOnlyCompleteData>false</ReturnOnlyCompleteData>
           <Aggregation>Hourly</Aggregation>
           <Columns>
             <CampaignPerformanceReportColumn>TimePeriod</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AccountId</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>CampaignId</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>CampaignName</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>CampaignStatus</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>DeviceType</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>BidMatchType</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>QualityScore</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AdRelevance</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>LandingPageExperience</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Revenue</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Assists</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>ExpectedCtr</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>DeliveredMatchType</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AveragePosition</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Conversions</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AdDistribution</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Network</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Clicks</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Impressions</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Ctr</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>AverageCpc</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>Spend</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>LowQualityClicks</CampaignPerformanceReportColumn>
             <CampaignPerformanceReportColumn>LowQualityConversionRate</CampaignPerformanceReportColumn>
           </Columns>
           <Filter>
             <AccountStatus i:nil="true" />
             <AdDistribution i:nil="true" />
             <DeviceOS i:nil="true" />
             <DeviceType i:nil="true" />
             <Status i:nil="true" />
           </Filter>
           <Scope>
             <AccountIds xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
               <a:long>481xxxxxxxxxxx</a:long>
             </AccountIds>
             <Campaigns i:nil="true" />
           </Scope>
           <Time>
             <CustomDateRangeEnd i:nil="true" />
             <CustomDateRangeStart i:nil="true" />
             <PredefinedTime>Yesterday</PredefinedTime>
             <ReportTimeZone>PacificTimeUSCanadaTijuana</ReportTimeZone>
           </Time>
         </ReportRequest>
       </SubmitGenerateReportRequest>
     </s:Body>
   </s:Envelope>

    
3. Create another XML file for second step in a Notepad. Enter below content and save as C:\Requests\BingAds\sql_body2.xml
   Replace following values as per your need.
   

   <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
     <s:Header>
       <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Reporting/v13"><%access_token%></h:AuthenticationToken>
       <h:CustomerAccountId xmlns:h="https://bingads.microsoft.com/Reporting/v13">480xxxxxxxxxxx</h:CustomerAccountId>
       <h:CustomerId xmlns:h="https://bingads.microsoft.com/Reporting/v13">191xxxxxxxxxxx</h:CustomerId>
       <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Reporting/v13">1052Bxxxxxxxxxxx</h:DeveloperToken>
       <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
       <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
     </s:Header>
     <s:Body>
       <PollGenerateReportRequest xmlns="https://bingads.microsoft.com/Reporting/v13">
         <ReportRequestId>[$a.ReportRequestId$]</ReportRequestId>
       </PollGenerateReportRequest>
     </s:Body>
   </s:Envelope>

    
4. Now try to execute below query. This should generate Performance report and download it in a SQL table for you.
   

   select * into tmpBingAdsReport from OPENQUERY(Zs_BingAds,
   'SELECT c.* FROM $
   WITH(
   	--submit report request for XML format
   
   	 alias=''a'' 
   	,Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
   	,Filter=''$.s:Envelope.s:Body.SubmitGenerateReportResponse''
   	,RequestData=''@C:\Requests\BingAds\sql_body1.xml''
   	,IsMultiPart=''True''
   	,RequestContentTypeCode=''TextXml''
   	,Header=''SOAPAction: "SubmitGenerateReport"''
   	,RequestMethod=''POST''
   	,EnableBodyPlaceholderForMultiPart=''True''
   	,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''
   
   	--keep checking status until report is ready - returns final url when done
   
   	,join1_alias=''b''
   	,join1_Filter=''$.s:Envelope.s:Body.PollGenerateReportResponse.ReportRequestStatus''
   	,join1_RequestData=''@C:\Requests\BingAds\sql_body2.xml''
   	,join1_IsMultiPart=''True''
   	,join1_Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
   	,join1_RequestContentTypeCode=''TextXml''
   	,join1_Header=''SOAPAction: "PollGenerateReport"''
   	,join1_RequestMethod=''POST''
   	,join1_EnableBodyPlaceholderForMultiPart=''True''
   	,join1_EnableStatusCheck=''True'', 
   	,join1_StatucCheckMaxWaitSeconds=300, --wait max 5 mins?
   	,join1_StatucCheckIterationWaitSeconds=5, --check every 5 sec?
   	,join1_StatusSuccessValue=''Success'', --look for success word in response
   
   	--Download report (Zip file) and Parse rows
   
   	,join2_alias=''c''
   	,join2_DataConnectionType=''Default''
   	,join2_Src=''[$b.ReportDownloadUrl$]''
   	,join2_Filter=''$.Report.Table.Row[*]''
   	,join2_ElementsToTreatAsArray=''Row''	
   	,join2_RequestMethod=''GET''
   	,join2_IsMultiPart=''False''
   	,join2_FileCompressionType=''Zip''	
   )')
   
   select * from tmpBingAdsReport

   

   Import Bing Ads data into SQL Server

Multi Step Query to download Report

So in previous section we saw how to write a single query to get your Performance Report from Bing Ads. However there will be time you want to split 3 steps into seperate queries.

Here is how to do.

/*
RPC and RPC OUT options must be turned on to run this proc

EXEC master.dbo.sp_serveroption @server=N'Zs_BingAds', @optname=N'rpc', @optvalue=N'true'
EXEC master.dbo.sp_serveroption @server=N'Zs_BingAds', @optname=N'rpc out', @optvalue=N'true'
--Below needed to support EXEC + INSERT (dynamic query)
EXEC master.dbo.sp_serveroption @server=N'Zs_BingAds, @optname=N'remote proc transaction promotion', @optvalue=N'false'
Also 
1. Change Body file path
2. Change CustomerAccountId, CustomerId, DeveloperToken
*/

ALTER proc usp_Api_BingAds_Reports
as
--///////////////////////////////////////////////////////////////////////////////
--Step-1 - submit report request for XML format
--///////////////////////////////////////////////////////////////////////////////
create table #tmpBingAdsReport_Step1(ReportRequestId bigint)
INSERT into #tmpBingAdsReport_Step1
EXEC(
'SELECT * FROM $
WITH(
 Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,Filter=''$.s:Envelope.s:Body.SubmitGenerateReportResponse''
,RequestData=''@c:\BWProjects\Requests\BingAds\sql_body1.xml''
,IsMultiPart=''True''
,RequestContentTypeCode=''TextXml''
,Header=''SOAPAction: "SubmitGenerateReport"''
,RequestMethod=''POST''
,EnableBodyPlaceholderForMultiPart=''True''
,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''
)') AT zs_BingAds


--///////////////////////////////////////////////////////////////////////////////
--Step2 - keep checking status until report is ready - returns final url when done
--///////////////////////////////////////////////////////////////////////////////

declare @reportid varchar(100)
select top 1 @reportid=cast(ReportRequestId as varchar(100)) from #tmpBingAdsReport_Step1

create table #tmpBingAdsReport_Step2(ReportDownloadUrl varchar(500))
INSERT into #tmpBingAdsReport_Step2 
EXEC(
'SELECT * FROM $
WITH(
 Filter=''$.s:Envelope.s:Body.PollGenerateReportResponse.ReportRequestStatus''
,RequestData=''<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Header>
    <h:ApplicationToken i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
    <h:AuthenticationToken xmlns:h="https://bingads.microsoft.com/Reporting/v13"><%access_token%></h:AuthenticationToken>
    <h:CustomerAccountId xmlns:h="https://bingads.microsoft.com/Reporting/v13">480xxxxxxxxx</h:CustomerAccountId>
    <h:CustomerId xmlns:h="https://bingads.microsoft.com/Reporting/v13">191xxxxxxxxx</h:CustomerId>
    <h:DeveloperToken xmlns:h="https://bingads.microsoft.com/Reporting/v13">105xxxxxxxxxxx</h:DeveloperToken>
    <h:Password i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
    <h:UserName i:nil="true" xmlns:h="https://bingads.microsoft.com/Reporting/v13" xmlns:i="http://www.w3.org/2001/XMLSchema-instance" />
  </s:Header>
  <s:Body>
    <PollGenerateReportRequest xmlns="https://bingads.microsoft.com/Reporting/v13">
      <ReportRequestId>'+ @reportid +'</ReportRequestId>
    </PollGenerateReportRequest>
  </s:Body>
</s:Envelope>''
--,IsMultiPart=''True''
,Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,RequestContentTypeCode=''TextXml''
,Header=''SOAPAction: "PollGenerateReport"''
,RequestMethod=''POST''
--,EnableBodyPlaceholderForMultiPart=''True''
,EnableStatusCheck=''True'', 
,StatucCheckMaxWaitSeconds=300, --wait max 5 mins?
,StatucCheckIterationWaitSeconds=5, --check every 5 sec?
,StatusSuccessValue=''Success'', --look for success word in response
,Meta=''[{"Name": "ReportDownloadUrl","Type": "String",Length:"300"}]''
)') AT zs_BingAds


declare @reportpath varchar(500)
select top 1 @reportpath=ReportDownloadUrl from #tmpBingAdsReport_Step2

select * from #tmpBingAdsReport_Step1
select * from #tmpBingAdsReport_Step2

--///////////////////////////////////////////////////////////////////////////////
--Step-3 - Get report file
--///////////////////////////////////////////////////////////////////////////////
--Static table
--//INSERT INTO BingAdsReport
EXEC(
'SELECT * FROM $
WITH(
 DataConnectionType=''Default''
,Src='''+ @reportpath +'''
,Filter=''$.Report.Table.Row[*]''
,ElementsToTreatAsArray=''Row''	
,RequestMethod=''GET''
,IsMultiPart=''False''
,FileCompressionType=''Zip''	
)') AT zs_BingAds


--select * from tmpBingAdsReport_Step3
--select * from openquery(Bing, 'select * from $')


/*
select * into tmpBingAdsReport from OPENQUERY(zs_BingAds,
'SELECT c.* FROM $
WITH(
--submit report request for XML format

alias=''a'' 
,Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,Filter=''$.s:Envelope.s:Body.SubmitGenerateReportResponse''
,RequestData=''@c:\BWProjects\Requests\BingAds\sql_body1.xml''
,IsMultiPart=''True''
,RequestContentTypeCode=''TextXml''
,Header=''SOAPAction: "SubmitGenerateReport"''
,RequestMethod=''POST''
,EnableBodyPlaceholderForMultiPart=''True''
,Meta=''[{"Name": "ReportRequestId","Type": "Int64"}]''

--keep checking status until report is ready - returns final url when done

,join1_alias=''b''
,join1_Filter=''$.s:Envelope.s:Body.PollGenerateReportResponse.ReportRequestStatus''
,join1_RequestData=''@c:\BWProjects\Requests\BingAds\sql_body2.xml''
,join1_IsMultiPart=''True''
,join1_Src=''https://reporting.api.bingads.microsoft.com/Api/Advertiser/Reporting/v13/ReportingService.svc''
,join1_RequestContentTypeCode=''TextXml''
,join1_Header=''SOAPAction: "PollGenerateReport"''
,join1_RequestMethod=''POST''
,join1_EnableBodyPlaceholderForMultiPart=''True''
,join1_EnableStatusCheck=''True'', 
,join1_StatucCheckMaxWaitSeconds=300, --wait max 5 mins?
,join1_StatucCheckIterationWaitSeconds=5, --check every 5 sec?
,join1_StatusSuccessValue=''Success'', --look for success word in response

--Download report (Zip file) and Parse rows

,join2_alias=''c''
,join2_DataConnectionType=''Default''
,join2_Src=''[$b.ReportDownloadUrl$]''
,join2_Filter=''$.Report.Table.Row[*]''
,join2_ElementsToTreatAsArray=''Row''	
,join2_RequestMethod=''GET''
,join2_IsMultiPart=''False''
,join2_FileCompressionType=''Zip''	
)')

select * from tmpBingAdsReport

select * from openquery(Bing, 'select * from $')
*/
go

Conclusion

In this article we saw how to import Bing Ads API data in SQL Server without any coding using ZappySys ODBC PowerPack (SOAP / XML Driver). Download it it and try it for FREE to explore many API integration features not discussed in this article.

The post Import Bing Ads data into SQL Server (Performance Reports) appeared first on ZappySys Blog.

In our previous blog, we saw how to perform SFTP / FTP File Operations in SSIS. Now in this blog, we will see How to Read SFTP / FTP Files in SSIS (CSV, JSON, XML Format files). To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like to perform file operations on FTP Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on how to Read files from secure FTP Storage (CSV, JSON and XML Files) and load into SQL Server Table in few clicks.

Components Mentioned in this article

	Secure FTP Source for JSON File
	Secure FTP Source for XML File
	Secure FTP Source for CSV File
	Secure FTP Destination for CSV File

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. You have obtained FTP/SFTP Credential.
4. Finally, do not forget to install ZappySys SSIS PowerPack

What is FTP, FTPS, SFTP, FTP/SSL?

Many people still have confusion over various protocols around FTP terminology. Most common terms you hear are as below.

• FTP
• Classic FTP
• Secure FTP
• SFTP
• FTPS
• FTP/SSL
• FTP over SSL
• FTP Over SSH

You must be overwhelmed by now? So are they all same or different? See below for quick clarification in nutshell.

Mainly there are 2 protocols for FTP and yes they are very different but still used to do Files Transfer to/from the remote server.

1. Classic FTP – sometimes referred as just FTP (Typically server runs on Port 21)
2. SFTP – Sometimes referred as Secure FTP or FTP over SSH (Typically server runs on Port 22)

So in short SFTP is not the same as the Classic FTP protocol. SFTP is more secure because traffic is always encrypted. On the other hand, Classic FTP is not encrypted by default but you can use FTP over SSL to request encrypted traffic in classic FTP. FTP over SSL also referred to as FTPS or FTP/SSL.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Secure FTP Source for CSV/JSON/XML File that will help you in reading CSV, JSON and XML Files from FTP to the Local machine, Upload files(s) to FTP Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to download files from FTP Storage.

You can connect to your FTP by entering your FTP credentials.

Read SFTP / FTP Files in SSIS (CSV, JSON, XML)

Let´s start with an example. First of all, In this SSIS FTP Source for CSV/JSON/XML File task example, we will read CSV/JSON/XML files from FTP to SQL Server database.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Now drag and drop relevant Secure FTP for CSV/JSON/XML File Task from the SSIS Toolbox.
   

   Drag and Drop FTP

3. Create a connection for Secure FTP Account.
   

   Create FTP Connection

4. Likewise, select the relevant single file to read from FTP in their relevant source of CSV/JSON/XML File Task.
   

   Select File From FTP Storage

5. Similarly, we can also read the multiple files stored in FTP Storage using wildcard pattern supported e.g. dbo.tblNames*.csv / dbo.tblNames*.json / dbo.tblNames*.xml in relevant source task
   

   Use wildcard pattern .* to read multiple files data

6. We can also read the zip and gzip compressed files also without extracting it in the specific FTP Storage for CSV/JSON/XML File Task.
   

   Reading zip and gzip compressed files (stream mode)

7. Finally, we are ready to load this file(s) data into the SQL Server.

Load Secure FTP files data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

Above all, in this blog, we learned how to Read Secure FTP Storage Files in SSIS. Even more, we used Secure FTP Source for CSV File, Secure FTP Source for JSON File and Secure FTP Source for XML File to read the file(s) from FTP Storage and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, You can use the following links for more information:

• SSIS Secure FTP Source for CSV/JSON/XML File
• About SSIS PowerPack
• About Secure FTP Task

The post Read SFTP / FTP Files in SSIS (CSV, JSON, XML) appeared first on ZappySys Blog.