In our previous blog we saw How to write data into CSV file in SSIS (GZip / Split). Now in this blog, we will see How to Pivot CSV Data in SSIS using CSV Source. It also supports Pivot mode so you can convert single CSV string value into Rows. In this article we also see how make some CSV columns as parent and not consider it in pivot and load it as it is using JSON Generator and JSON Parser Transform Pivot Option.

In nutshell, this post will focus on how to Pivot CSV Data into multiple columns and rows.

How to Pivot CSV Data in SSIS using CSV File Source

Let´s start with an example. In this SSIS CSV Parser Transform example, we will Parse database column string into multiple columns and rows.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. Furthermore, drag and drop the CSV Source and Double click on it to configure it.
   

   Drag and Drop : CSV Source from SSIS Toolbox

3. Enter the CSV file path and select desire column delimiter most common in CSV is comma and check the First Row as headers and click on preview.
   

   CSV Source Preview

4. Now, go to Pivot Columns to Rows Tab and checked Enable Pivoting and click on Preview Data to view the data.
   

   CSV Source Pivoted Data Preview

5. That’s it we see how to read data from CSV File and also Pivot the CSV Data using CSV Source.
6. Now In the next section we will see how to how make some CSV columns as parent and not consider it in pivot and load it as it is.

How make some CSV columns as parent.

Sometimes you just need to make child columns in to rows and want to leave parent level columns as it is.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. Secondly, drag and drop the CSV Source and Double click on it to configure it.
   

   Drag and Drop : CSV Source from SSIS Toolbox

3. Furthermore, enter the file path and click on preview.
   

   CSV Source Preview

4. For example if we want to pivot Email and Phone only in pivoting and wants to exclude parent level columns FirstName and LastName as it is.
5. For that drag and drop JSON Generator Transform and Select Output Mode as Output One JSON Per Parent Input Row  and Add Unbound Nested Element and named it as Persons and click on OK.
   

   JSON Generator Add Unbound Nested Element

6. Now add that two parent level fields FirstName and LastName in the Persons.
   

   JSON Generator Add Multiple Elements

7. Now select Person Element and right click and Add Unbound Nested Element and named it as Contacts and click on OK.
   

   JSON Generator Add Unbound Nested Element

8. Now add that two child level fields Email and Phone in the Contacts.
   

   JSON Generator Add Multiple Elements

9. Therefore JSON Generator Transform looks like below screen. Now Copy that Sample Preview JSON String to use it in the JSON Parser Transom as sample string.
   

   JSON Generator Transform – Sample Data

10. Similarly, drag and drop JSON Parser Transform and select the Input JSON Column and Paste that copied Sample JSON String and select the Contacts filter to read the Child data and click on Include Parent.
    

    JSON Parser Transform – Select Filter

11. Now go to Pivot Columns to Rows Tab and checked the Enable Pivoting and click on the Preview.
    

    JSON Parser Transform – Enable Pivoting

12. Finally, we are ready to load this data.
    

    CSV Pivoted Data

13. Let’s load this data in the SQL Server.

Load Pivoted CSV File data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to read CSV file data, pivot the data and Similarly, using SSIS JSON Generator and Parser Transform Pivot the CSV Data excluding some columns from pivoting.  To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following URL for more information.

• Help File: CSV Source

The post How to Pivot CSV Data in SSIS appeared first on ZappySys Blog.

In this new article, we will show how to read WordPress data using REST API. REST API is a very popular software architectural style for Web Services.
SSIS is a very popular ETL software used to integrate almost any data (SQL Server, Oracle, Excel, DB2, CSV files, Big Data, Facebook, Gmail, etc).

Requirements

For this example, we will use a popular plug-in. The name is ZappySys PowerPack for SSIS which includes some nice generic connectors to REST API. You can download it here:

• ZappySys PowerPack for SSIS Installer
• SSDT for Business Intelligence

Getting started

We will be using this ZappySys SSIS PowerPack component to make things work:

	SSIS REST API Web Service Task
	SSIS JSON Source (File, REST API, OData Connector)
	SSIS JSON File Destination (Create JSON File)

Read WordPress data using SSIS with REST API

REST API is also the name of one of the ZappySys components. These tools can connect to thousands of REST API Servers. In this example, we will connect to WordPress.

1. First, drag and drop the REST API task to the design pane.
2. Secondly, write the following URL in the REST API task URL:
   
   

   http://demo.wp-api.org/wp-json/wp/v2/pages

3. After that, press Test Request/Response.
4. That URL shows the page’s information in WordPress. This is a demo API. For more information about the WordPress links, check here.
   

   Call WordPress in SSIS

5. Next, you can also check the Tags, Comments, Categories, Posts, Taxonomies, Media. Here you have some examples:
   • To get the tags in WordPress, use this URL:
     

     http://demo.wp-api.org/wp-json/wp/v2/tags

   • In order to read comments in WordPress using SSIS and REST API, use this URL:
     

     http://demo.wp-api.org/wp-json/wp/v2/comments

   • For the categories, use this URL to read WordPress data using SSIS:
     

     http://demo.wp-api.org/wp-json/wp/v2/categories

6. In addition, for your blog, you need to add /wp-json/wp/v2/ and pages, posts or tags in your WordPress URL.
7. Finally, if you want to save the results, in Response Settings, enable save response content and select the option save to file, and specify a file path:
   

   Save response WordPress API

How to read WordPress data using SSIS and store it into a SQL Server database

In the next example, we will use the JSON Source component. This component is similar to the REST API, but this is used in the Data Flow to save the REST API results in any destination of your preference (SQL Server, PostgreSQL, MongoDB, etc). Also, this example will read WordPress data using SSIS and store it into a SQL Server database.

1. First of all, in the Control flow, drag and drop the Data Flow:
   

   Data Flow Task

2. Drag and drop the JSON Source in the Data flow with this configuration:
   

   JSON Source to WordPress API

3. Finally, you can save to an OLEDB Destination to save into a SQL Database:
   

   Store WordPress into SQL Server

How to read WordPress data using SSIS and modify the JSON structure

Sometimes we need to modify the WordPress JSON output to a custom output in SSIS. We will now use the JSON Generator Transform this time, which is a very powerful tool for this purpose.

1. First of all, we will use the JSON Source connecting to the following
   

   URL: http://demo.wp-api.org/wp-json/wp/v2/categories

2. Secondly, we need to convert the JSON output to the following format:
   

   {
      "id" : 
        {
            "count" : 1474359596826126768,
            "description" : 
             {
               "link" : "xxx"
             }
       }
   }

3. Next, we will add an element first:
   

   SSIS WordPress API

4. Also, we will select the id attribute:
   

   id attributes SSIS

5. In addition, we will add a static element to the id:
   

   Add JSON element

6. Furthermore, we will add the count element.
   

   Add element in JSON with SSIS

7. Finally, repeat the same process for the description and link. The JSON structure will be the following:

How to read WordPress data using SSIS and use pagination

If it is a lot of data in REST API, pagination allows showing the results in pages. Let’s take a look at it.

1. First of all, we will use page 2. To do it, we will use the following URL in the JSON Source task in the data flow:
   

   http://demo.wp-api.org/wp-json/wp/v2/posts?per_page=2

   

   SSIS URL pagination REST API

2. Secondly, there is a pagination tab that can be used to get the data using pagination. For more information about pagination, refer to our ZappySys Pagination article related.
   

   SSIS pagination in SSIS

How to write WordPress data using SSIS

1. In order to write WordPress data using SSIS, we will use the REST API task with the following URL:
   

   POST http://demo.wp-api.org/wp-json/wp/v2/posts/

2. Secondly, we have the post with id equal to 2 and we will use the body request as follows:

Conclusion

To conclude, in this article, we learned how to write and read WordPress data using SSIS. In addition, we learned also how to get pagination and how to change the format of the JSON results. If you want to try it yourself, do not hesitate to download ZappySys PowerPack!

References

Finally, for more information about how to read WordPress Data using SSIS, refer to these links:

• REST API Handbook
• WordPress REST API Tutorial (Real Examples)

The post Write/Read WordPress data using SSIS and REST API appeared first on ZappySys Blog.

In our previous article we saw how to POST data to REST API using few different ways. Now let’s go one step further and discuss another common scenario to batch REST API requests in SSIS. For efficient data transfer many API provides you to submit multiple records in a single request. In this article we will look at few ways to batch our requests using SSIS JSON Generator Transform and  SSIS Web API destination.

Method 1 – Batch records using SSIS JSON / XML Generator Transform

First let’s look at common way to group multiple records and generate single JSON or XML document for desired batch size. We will also see another technique where you can assign Unique BatchID to each document.

For example below screenshot shows how to product one JSON document for every 10 input rows. We have also use BatchID column on the root level. This will use use BatchID from first row of that batch. You can use XML Generator Transform if you like to produce XML Document rather than JSON.

To generate Unique BatchID you can use function like newid() in SQL Server.

SELECT newid() as BatchID,CustomerID,CompanyName FROM Customers

Using SSIS JSON Generator Transform with Batch Option (Multiple Records in a single document)

Here if full flow to batch your JSON Requests and Submit to REST API.

POST data to REST API in Bulk Mode (Batch multiple rows)

Method 2 – Using SSIS Web API destination Batch Mode with Header / Footer

Another method of batching is use Web API destination and turn on Batch mode.  You can use Fiddler to debug your web requests incase you are not sure how its generating and sending POST requests after this setting.

Configure SSIS Web API Destination for Bulk API call (Insert multiple rows)

Method 3 – Batch CSV records using CSV Generator Transform

So far we looked at how to batch JSON and XML format. Now let’s look at scenario where you have to POST CSV formatted data to REST API endpoint.

Assume that you have an API endpoint where you can submit data in multiple batches. Also you have few additional requirements

1. First row of First batch must have column names (any other batch must not contains CSV Header)
2. In each Batch we want to submit maximum 5 rows as per API call
3. Each batch must contain Unique Batch Number in URL

Here is high level flow.

Generate CSV using Batch Setting , Send data to REST API in multiple batches, Debug REST API in Fiddler

Include CSV Header in First Batch Only

Now let’s break down the problem and find out how to do. First problem is we need to include CSV Header only in First batch. To achieve this we use old school trick where we output column names as data row… and when we generate CSV in the next section we will turn off Column Headers option.

If you are consuming records from SQL Server or any other RDBMS you can write following style source query. If you don’t use relational source then use native UNION Transform in SSIS.

select 'CustomerID' as Column1, 'OrderID' as Column2
UNION ALL
select top 12 cast(CustomerID as varchar(1000)), cast(OrderID as varchar(1000)) from Invoices

Batch records and output CSV data

Now next step would be to generate CSV data but in batches… so in our example we have to output size of 5 rows per batch. For this perform the following steps

1. Drag ZS CSV Generator Transform (Included in v2.8 and Higher)
2. Connect source to CSV Generator Transform
3. Now before we edit in UI mode, right click and select Properties. Change following 3 properties  (You can also change QuoteAroundValue and Dimiliter if needed). In future version batch setting will be visible in UI mode too.
   1. ArrayBatchSize  = yourbatchsize (in our case it was just 5 rows per batch)
   2. OutputMode = SingleFileFormat
   3. FirstRowsHasColumnNames  =False
4. Now double click to configure ZS Generator Output
5. Right click on Mappings folder node and Add Elements > Select Multiple Columns  > Check columns you like to output
6. Click OK to Save UI

Generate CSV data in SSIS (Batch Output Setting)

Generate unique URL with Batch using Script Component

Now next task we have to achieve is generate Unique Batch Number. Put that Batch ID in URL like below…

http://httpbin.org/post?batch=0
http://httpbin.org/post?batch=1
http://httpbin.org/post?batch=2
.....

1. For this drag SSIS Script Component and select Transformation Mode
2. Now go to Inputs and Outputs tab > Select Output Columns > Click Add Column > Name it as URL > Configure DataType as DT_WSTR , Length = 500
3. Now got Script Tab and Click Edit. Remove old code and enter following code (Assuming You selected C#). This code will generate new URL with Unique batch ID each time. You can use this same technique to generate Unique row number too (in that case rather than DT_WSTR type you can use DT_I4 type)

   [Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
   public class ScriptMain : UserComponent
   {
       private static int _batch;
       public override void Input0_ProcessInputRow(Input0Buffer Row)
       {
           Row.Url = string.Format("http://httpbin.org/post?batch={0}",_batch);
           //If you have to send GUID rather than Integer for Batch ID then use below code
           //Row.Url = string.Format("http://httpbin.org/post?batch={0}",System.Guid.NewGuid());
   
           _batch = _batch+1;
       }
   }

4. Click OK to Save script.

Output Unique URL or Number using SSIS Script Transform

Configure Web API Destination to POST CSV Data

Now final step is to configure SSIS Web API Destination to POST CSV data to API.

1. Drag ZS Web API Destination and connect Source Transform to Web API destination.
2. Double click to configure
3. Select Connection Manager
4. Once connection setup, select Input Column for URL ( This will be same column from Script Transform)
5. Select column for Body (e.g. ZS_CSV_OUT)
6. Click OK to save

POST data to REST API in multiple batches using SSIS Web API destination

Debug Web API Requests using Fiddler

Now we ready to run entire flow. Once you execute data flow, observe Input rows before CSV Generator Transform and after. CSV Generator transform must produce less rows before we now grouping multiple rows in a single batch.

Another important aspect to play with API request is to debug it, We recommend you to to look at this article about using Fiddler to debug web requests.

Download Sample Package

Here is the Sample SSIS Package (2012 format) – Rest API_Batching

Conclusion

In this post, we saw how easy it is to handle complex REST API  / CSV /JSON / XML operations in SSIS. We configured REST API POST Operation using Batching Method. Hope this post helps you to achieve your API integration without coding.  Feel free to download SSIS PowerPack and try many other components not discussed in this article.

The post How to batch REST API requests in SSIS (Bulk Operation) appeared first on ZappySys Blog.

Google BigQuery is a fully managed Big Data platform to run queries against large scale data. In this article you will learn how to integrate Google BigQuery data into Microsoft SQL Server using SSIS. We will leverage highly flexible JSON based REST API Connector and OAuth Connection to import / export data from Google BigQuery API just in a few clicks.

If you are looking for a similar product inside Amazon AWS Cloud then check an article about Amazon Athena.

Prerequisites

Before we look into a Step-By-Step section to extract and load data from BigQuery to SQL Server let’s make sure you meet the following requirements:

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have basic familiarity with REST API concepts and Google BigQuery API. This post uses free Public Dataset to query BigQuery table so no billing is required to get your hands dirty 🙂 However, if you like to query your own dataset then make sure you have at least one BigQuery dataset and one table with some sample data created (this part does require billing enabled). Read a Google Quickstart article for more information on how to create a new BigQuery dataset and a table.

Understanding Google BigQuery Object Heirarchy

Google BigQuery has 3 main concepts below.

• Project
  • Dataset
    • Table
      • Query requests (each query creates a unique JobID – valid for 24 hours from which you can read data)

So in order to create BigQuery table you always need Project and then Dataset under that project. You can group objects around Project / Datasets. When you query you can supply fully qualified name of your table in FROM clause (e.g.

select count(*) from `bigquery-public-data.usa_names.usa_1910_2013`

So let’s get started.

Read data from Google BigQuery using SSIS

Basically you can query Google BigQuery data in two ways: In this article we will not cover 2nd method.

• Method-1: Query data using jobs/query method in BigQuery API. Use it if you expect to get a result in a fairly short amount of time. This API method generates a temp table which gets deleted after 24 hours. You can read data within that time frame using newly created JobId reference.
  
  NOTE: If you have records that sum up to more than 10MB of data or ~10,000 rows (assuming 1 row uses 1KB of data) then you need to proceed with two-step process as explained below.
• Method-2: Export SQL query result using jobs/insert method in BigQuery API. Use this method if you expect a query to take a long time to finish.

Now let’s see how to query Google BigQuery data using SSIS 🙂

Create Google API Project and Register OAuth App

To use any Google API firstly you have to finish two tasks:

1. Create Google API Project and obtain projectId (see next section).
2. Enable BigQuery API
3. Register your own Google OAuth App and obtain ClientId and Client Secret.

Check step-by-step instructions on how to create Google API Project and create OAuth App for Google. During instructions make sure you enable BigQuery API (screenshots may differ).

Get Google API Project ID

Once you have Google API Project created you can grab Project ID (see screenshot below). You will need Project ID in the next step when building API URL.

How to find Google API Project ID?

Configure OAuth Connection for Google BigQuery API

In order to call most of Google APIs you will need an OAuth App. If you want to use your own app then refer to how to register Google OAuth Application (Get ClientID and ClientSecret) article. For simplicity we will use Default OAuth App:

1. Right click inside Connection Managers area and click “New Connection…”
2. From the connection type list select “ZS-OAUTH” connection type.
   

   Create new SSIS OAuth API Connection Manager

3. For OAuth Provider select “Google”.
4. For OAuth App type options leave “Use Default OAuth App” selected. If you created custom OAuth App then you would need to select “Use Custom OAuth App” option.
5. For Scopes enter or select the following URLs (URL is just a permission name). Each URL must be in a separate line. For demo purposes we use only the first 3 scopes but we included a few more in case you like to test API in depth with a different permission set:
   

   https://www.googleapis.com/auth/bigquery
   https://www.googleapis.com/auth/cloud-platform
   https://www.googleapis.com/auth/cloud-platform.read-only
   https://www.googleapis.com/auth/bigquery.insertdata
   https://www.googleapis.com/auth/devstorage.full_control
   https://www.googleapis.com/auth/devstorage.read_only
   https://www.googleapis.com/auth/devstorage.read_write

6. Click Generate Token. Login using correct account if needed and then you will be prompted to click “Approve” OAuth App.
7. Once you click OK on the login screen you will see new tokens populated on the connection screen.
8. Click Test to make sure connection / tokens are valid and then click OK to save the connection.
   NOTE: Below screenshot uses Default App but we recommend you to to use Custom OAuth App (Your own Clientid / secret – Obtained in previous section)
   

   Create SSIS OAuth API Connection for Google BigQuery API

Start BigQuery Job and get JobId (Submit Query)

Once you have SSIS OAuth connection created for BigQuery API it’s time to read data from BigQuery. So, basically, there are two ways you can read BigQuery data: using query or insert method. For demo purposes we will use jobs/query method. If you want fire complex queries which can run for many minutes then refer to jobs/insert method.

If you are expecting more than 10MB of data (~10K records) then you have to split data extract in two steps:

1. Step-1 : Call jobs/query method using REST API Task to submit SQL query. This steps returns jobComplete: true if supplied query execution is done within timeoutMs parameter you included. By default its 10 seconds so if your query is going to take longer than 10 seconds then its good idea to add bigger timeout that way you dont need step#2. Once API call is done you will Get JobId in the response. Save that JobId in a variable for later use. This job result is valid for 24-hours only.
2. Step-2 (Optional) – You can add another optional REST API Task after previous step to Wait until Job is completed. For simplicity, this article will Omit setting up Status Check but its very simple….Use below settings
   1. Drag new REST API Task from toolbox. Connect it to Step-1 and double click to configure as below.
   2. Select URL From Connection mode,
   3. Select OAuth connection
   4. Enter URL as https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/jobs/{{User::JobId}} , Method: GET
   5. On Response Settings Tab, Select Response Content Type JSON and for content filter enter $.status.state
   6. On Status Check Tab, check Enable Status Check Loop and enter DONE in SuccessValue field
   7. Doing this setup will make sure we do not query data until Job Status is DONE (System keeps checking every 5 seconds)
3. Step-2 : Read Job result using JSON Source in a Data Flow Task.

Now lets see how to configure REST API Task to submit query and extract JobId (which will be used in next section).

1. In the Control Flow designer drag and drop ZS REST API Task from the SSIS toolbox.
2. Double click the task to configure it.
3. Change Request URL Access Mode to [Url from connection].
4. From the connection dropdown select OAuth connection created in the previous section.
5. Enter URL as below (replace YOUR-PROJECT-ID with a valid Project ID obtained in the previous section):
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-PROJECT-ID/queries

6. Select Method as POST.
7. Enter Body (Request Data) as below (change your query if needed; we used Public dataset for the demo): If you do not specify timeout then default is 10 seconds only. Also in this call we only care about JobID from response so we just added maxResults=10 because in 2nd step we will get all rows by doing pagination.
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000"
   }

   To use a Standard SQL query instead of Legacy SQL, add useLegacySql property and set it to false:
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000",
      "useLegacySql": false
   }

   For all possible parameters refer to jobs/query method documentation.
   NOTE: You can also supply location Parameter in the Body JSON to indicate where job should run. For non EU / US datacenters we suggest you to supply this parameter. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location.
   Example Use Of Location
   

   { "location":"us-east1",  "maxResults": 10, "query": "SELECT title FROM [bigquery-public-data:samples.wikipedia] LIMIT 10" }

8. Select Body Content Type as application/json.
9. Click Test Request/Response. If things go well you will see JSON content. Then just copy JobId from the response and save for later use:
   
10. Now go to Response Settings tab.
11. Select ContentType=Json.
12. Enter below expression to extract JobId from the response. You will need this Id to read SQL query output (this JobId is valid for 24 hrs):
    

    $.jobReference.jobId

13. Choose save response content to save to a variable. Select <New Variable…> option. When prompted give a name to your variable (i.e. vJobId) and for the value field paste JobId copied from the above step:
    
    
    NOTE: This JobId expires after 24-hrs so while you are designing a package and not running it then design time value is used for testing/previewing. So make sure your JobId is valid. If needed click Test Request/Response to grab a new JobId and update the variable with a new value so you can preview data without running a full package.
14. Click OK to save the UI.

Read BigQuery SQL result (for specified JobID)

Now let’s look at how to read data from BigQuery

Configure Google BigQuery Web Request (URL, Method, ContentType, Body etc.)

1. In the Control Flow designer drag and drop Data Flow Task from SSIS toolbox.
   
2. Double click Data Flow Task and drag and drop ZS JSON Source (For API/File) from SSIS toolbox.
   
3. Double click JSON Source to edit and configure as below
4. Enter URL as below. Replace YOUR-API-PROJECT-ID with the API Project ID obtained in the previous section.
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?maxResults=10000

   –OR– (Use below if your Job ran outside US / EU data center)
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?location=YOUR_REGION_ID&maxResults=10000

   NOTE: location Parameter indicates the geographic location of the job. This is Required parameter except for US and EU. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location. You can supply same location parameter for first step when you submit the query

   Failure to supply location parameter for non US /EU users may result in 404 NotFound Error.
   

5. Check Use Credentials option and select OAuth Connection Manager created in the previous section.
6. For HTTP Request Method select GET.
   

Read data from Google BigQuery from Temp Job result

Configure Filter

We need to transform a single JSON response into multiple rows so we need to apply a correct filter.

1. On JSON Source go to Filter Options tab.
2. In the Filter field enter $.rows[*] or click [Select Filter] button to browse hierarchy you want to extract.

Select Filter for JSON Response

Configure BigQuery API Pagination Settings

Most of modern APIs usually implement some sort of pagination technique so you get a part of data in each request rather than all in one go. Thus if you want more you can paginate through pages until the last page is reached. You can also read Understanding REST API Pagination in SSIS article to learn more about pagination in SSIS.

BigQuery API returns pageToken attribute in response JSON if more data is found for requested query result. You can then pass pageToken in the next URL in this format: http://my-api-url/?pageToken=xxxxxxxxxx

Now lets configure JSON Source to automate this pagination for us. On JSON Source go to Pagination Tab and enter the following two settings:

1. Set Next Link as $.pageToken.
2. Set Suffix for Next URL as &pageToken=<%nextlink%>.

See the screenshot below to get more clarity:

Configure BigQuery API Pagination on SSIS JSON Source

Configure Array Transformation

Now the last thing we have to configure is special 2-dimensional JSON array format used by BigQuery API:

1. On the JSON Source UI go to 2D Array Transformation tab.
2. Enter the following settings:
   1. For Transformation Type select Transform complex 2-dimensional array.
   2. For Column Name filter enter $.schema.fields[*].name.
   3. For Row Values Filter enter $.f[*].v.

JSON Array Transformation Options

Preview Data and Save UI

That’s it click Preview to see some data. If you entered sample JobID for your User::vJobID variable then you will see some data. In the next section we will see how to load Google BigQuery data into SQL Server. You can click Columns Tab to review Metadata.

Click OK to save UI and generate Metadata.

Configure Target – Load Google BigQuery data into SQL Server

Now you can connect your JSON Source to any target such as ZS Trash Destination or a real database destination such as OLEDB Destination.

If you wish to dump data from Google BigQuery to a SQL Server table then just perform the following steps:

1. Drag and drop OLEDB Destination from SSIS Toolbox. Rename it to something like SQL Server Table.
2. Double click on OLEDB Destination.
3. Click New to create a new connection > Configure connection > Click OK.
4. Click on New to create a new table > Rename default table name > Click OK.
5. Click on Mappings tab and click OK to save UI with default mappings.

Configure SSIS OLEDB Destination – Google BigQuery to SQL Server Import

Execute Package – Loading BigQuery data into SQL Server

SSIS Package Execution –
Loading Google BigQuery Data into SQL Server

Create / Delete Google BigQuery Dataset using API call

As we mentioned before, you need dataset before you can create a table. Most common way to create dataset is via User Interface but what if you like to automate from your SSIS Package or other workflow? Here is how you can create or delete. Basically you can use REST API Task to send CREATE or DROP command for Dataset object

Create Google Dataset

To create dataset configure REST API Task using below settings (We will call this API)

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/datasets

4. Request Method as POST
5. Enter Request Body as below (change YOUR_DATASET_NAME – e.g. TestDataset )
   

   {"datasetReference": { "datasetId": "YOUR_DATASET_NAME", "projectId": "{{User::ProjectId}}"} }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Dataset already exists then you can go to Error Handling Tab (See next section for screenshot) and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Delete Google BigQuery Dataset

For deleing dataset you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Create / Delete Google BigQuery Table using API call

Now let’s look at how to create /drop Table in Google BigQuery using API calls.

Create Google BigQuery Table

To send CREATE TABLE  SQL statement (DDL) we have to use same approach as we send normal SQL Query using this API call. So notice we used Standard SQL for this call by supplying  useLegacySql: false . DDL Statement must be all in one line

POST https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

Content-Type: application/json

>>>> BODY <<<<<

{
 "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
 "useLegacySql": false,
 "timeoutMs": 100000 
}

To create dataset configure REST API Task using below settings

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

4. Request Method as POST
5. Enter Request Body as below
   

   {
    "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
    "useLegacySql": false,
    "timeoutMs": 100000
   }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Table exists then you can go to enable Error Handling Tab and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Create Google BigQuery Table – API Call (Continue On Error Setting – Skip CREATE if Table / Dataset Already Exists)

Delete Google BigQuery Table

For deleing table you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Write data to Google BigQuery using SSIS – 1 Million row insert test (FAST)

Now let’s look at how easy it is to import data into Google BigQuery using SSIS. We will use the same OAuth connection we created before. To learn more about inserting data into BigQuery check tabledata/insertAll method documentation.

Make sure billing is enabled

Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

Streaming is not available via the free tier. If you attempt to use streaming without enabling billing, you receive the following error: BigQuery: Streaming insert is not allowed in the free tier.

As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call (Write  demo).

Configure OAuth Connection / Permissions

If you want to perform data insert operation in BigQuery using API calls then include the following scopes in your OAuth Connection Manager and generate a token (see our first section of this article – We already included scopes for Write operation but incase you didnt do then regenerate token with below scopes):

https://www.googleapis.com/auth/bigquery
https://www.googleapis.com/auth/cloud-platform
https://www.googleapis.com/auth/bigquery.insertdata

Create BigQuery Dataset (From UI)

For this demo first create a test dataset and one table under it like shown below (billing must be enabled on your Google API Project). To do that go to https://bigquery.cloud.google.com/welcome and configure them:

Insert data into BigQuery using API call

And here is a REST API example to insert data into a BigQuery Table:

Request URL:

https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

Request Headers:

Authorization: Bearer ya29.Gl0cBxxxxxxxxxxxxxxxxxxuEIhJIEnxE6GsQPHI
Content-Type: application/json

Request Body:

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
     {"json": {"RowId": 2,"CustomerName": "BBB"} }
   ]
}

Here is our data flow setup to achive very high throughput for Google BigQuery Data Load. We will show you how to insert one million rows in Google BigQuery in less than a minute based on below setup. We will use Multi Threading option and New Compression Option (Added in v3.1.4)

1. Dummy Data Source – Generate sample records
2. JSON Generator Transform – Generates JSON documents to send as POST request for above /insertAll API call.
3. Web API destination  – Call /insertAll API call to submit our data to BigQuery
4. (Optional) JSON Parser Transform – Parse Error Message for any response
5. (Optional) Trash Destination – Save any errors to text file for review

Google BigQuery Data Load Demo in SSIS – 1 Million Rows Insert with Multi Threads and Compression ON (Fast Upload)

Configure SSIS JSON Generator – Generate JSON for BigQuery Table Insert

Now let’s see how to build an HTTP request with JSON body and send it to BigQuery:

1. Drag Data Flow Task and double click on it.
   
2. Drag and configure your Source (for this demo we use Dummy Data Source with Customer Template). See previous section for configuration of Dummy Data Source.
   
3. Drag and drop ZS JSON Generator Transform to produce JSON from a database or file records. If your source is already sending a valid JSON then you can skip this step (e.g. SQL query is
   returning JSON). You can also read raw JSON from a very large file (new-line separated JSON) using JSON Source with Output as Raw Data option checked.
4. Connect Source to JSON Generator. Double click JSON Generator Transform to start configuring it like below.
5. Select Output Mode as Single Dataset Array and enter Batch Size 10000 (This is Max limit allowed by Google BigQuery API insertAll)
6. First right click on Mappings node and select Add Static Element
   

   Generate JSON for Google BigQuery InsertAll API request – Batch 10000 rows in a single API call

7. Enter Name as kind and value as bigquery#tableDataInsertAllRequest
   
   

   JSON Generator – Add Static Element

8. Now right click on Mappings node and select Add Document Array option
   

   JSON Generator – Add Document Array

9. Enter rows as array title and click OK
   

   JSON Generator – Name array

10. Select newly added array node and right click > Add unbound nested element  enter Output alias as json and click OK.
    

    JSON Generator – Add unbound nested element

11. Select json node and right click > Select Add Elements below this node and select multiple columns you like to send to BigQuery. Click OK to save.
    

    JSON Generator – Add Multiple Elements

12. Now let’s preview our JSON (Copy preview JSON to try in the next step – Web API destination)
    NOTE: Table name and column names are case-sensitive so make sure your JSON attribute matches exact same way.
    Here is the finished JSON Structure for next Step
    

    Sample JSON Request body for Google BigQuery insertAll API request

13. Click OK to save UI.

Configure SSIS Web API destination – Insert data into BigQuery Table

Once you have Input JSON prepared,  now let’s configure destination for BigQuery.

1. Drag and drop ZS Web API Destination.
2. Connect your JSON Generator Transform to Web API destination.
3. Configure general properties:
   

   SSIS Web API Destination –
   Configure for BigQuery Data load

4. Make sure to enter URL in this format:
   

   https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

   Make sure to replace 3 parts in above URL (MY_PROJECT_ID, MY_DATASET_ID, MY_TABLE_ID) with actual values from your Google Project and BigQuery dataset/table configuration.
5. Now you can enable Compression and Multiple Threads for higher throughput as below.
   NOTE: Compression Property was added in v3.1.4 so you may not see it if you have older version.
   

   Google BigQuery Data Loading Performance Optimization Options – Enable Multiple Threads and Compression Options

6. If you want to try test insert request from UI without running full package then go back to first tab and edit Body (Use Sample JSON generated by previous JSON Transform – You can grab from JSON Preview Panel on Generator Transform).Click Test Request / Response and confirm Success as below. You can go back to your BigQuery Portal and check one row is inserted after our test click. If everything looking good then run full package to insert all records.Sample JSON for Body:
   

   {
     "kind": "bigquery#tableDataInsertAllRequest",
     "rows": [
        {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
        {"json": {"RowId": 2,"CustomerName": "BBB"} }
      ]
   }

   

   Test Google BigQuery Table Insert – SSIS Web API Destination UI

7. Hit OK to save UI.
8. Run the package and verify data in Google BigQuery Console:
   

   Loading data into Google BigQuery using SSIS

Error Handling for BigQuery Data Load (Bulk Insert API Calls)

There will be a time when some records you insert may not go well in Google BigQuery. In such case you can read output from Web API destination and parse further with JSON Parser Transform. Check for certain values in the output. You must use JSON Parser after Web API destination (Connect Blue Arrow from Web API destination – Since its Soft Error it won’t redirect in Red Arrow ).

For example here is the sample JSON in POST Body for testing which produces error due to bad column name. When bad row found in batch all records will be rejected. Notice that error returns index of record in batch so you can identify which row went bad. It also returns column name in location attribute.

NOTE: Bad column name in 2nd record

Test Body (Bad):

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RecordID": 1,"CustomerID": "X1"} }, 
     {"json": {"Bad_Column": 2,"CustomerID": "X2"} }
     {"json": {"RecordID": 3,"CustomerID": "X3"} }, 
   ]
}

Response (For Bad Input):

{
  "kind": "bigquery#tableDataInsertAllResponse",
  "insertErrors": [
    {
      "index": 1,
      "errors": [
        {
          "reason": "invalid",
          "location": "bad_column",
          "debugInfo": "",
          "message": "no such field."
        }
      ]
    },
    {
      "index": 0,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    },
    {
      "index": 2,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    }
  ]
}

To configure error detection perform following steps.

1. Drag and drop ZS JSON Parser Transform after Web API destination
2. Click on Web API destination. Connect Blue arrow  to JSON Parser Transform
3. Configure JSON Parser Transform like below
4. Connect JSON Parser TRansform to some Destination to save error information (e.g. SQL Table or Trans destination)

Handling BigQuery Insert Errors in SSIS

Other Common Errors in BigQuery API calls

In this section we will talk about many common API errors in BigQuery.

Error: The project XXXXXXX has not enabled BigQuery

Sometimes you might get below error. To fix this error make sure you go to your Project and Enable BigQuery API

Status Code: BadRequest

Response Body: {
  "error": {
    "code": 400,
    "message": "The project bigquerytest-281915 has not enabled BigQuery.",
    "errors": [
      {
        "message": "The project bigquerytest-281915 has not enabled BigQuery.",
        "domain": "global",
        "reason": "invalid"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}

Error: 404 – Not Found: Table / Job xxxxxx

• Make sure you are setting GET request and not POST.
• Also make sure your jobId is valid because it expires after 24 hours.
• Make sure project-id supplied in URL is valid ID  (DO NOT Specify Alias, use internal ID for project)
• Make sure you supplied location parameter if your outside EU / US region. Reading data using this API call might fail if you failed to supply location in URL

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "notFound",
    "message": "Not Found: Table xxxxxx"
   }
  ],
  "code": 404,
  "message": "Not Found: Table xxxxxxx
 }
}

Error: 403 – Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier

If you trying call certain APIs on sandbox mode or free tier then you might get below error. To overcome this error enable billing on google bigquery. As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call.

Response Url: https://www.googleapis.com/bigquery/v2/projects/bigquerytest-281915/datasets/TestDataset/tables/Table1/insertAll

Status Code: Forbidden

Response Body: {
  "error": {
    "code": 403,
    "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
    "errors": [
      {
        "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
        "domain": "global",
        "reason": "accessDenied"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}

Debugging Web API Requests

If you need to debug actual requests made to Google server then you can use a tool like Fiddler. It’s a very handy tool to troubleshoot JSON format issues. It will allow to see how a request is made to a server.

Using Fiddler to debug Google BigQuery API requests in SSIS

Common Errors

Deployment to Production

Download Sample Package

Click Here to Download SSIS Sample Package – Google BigQuery API Read Write Create Delete (SSIS 2019, 2017, 2012)

Conclusion. What’s next?

In this article we have learned how to load data from Google BigQuery into SQL Server using SSIS (drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from Google BigQuery REST API using OAuth. JSON Source Connector makes it super simple to parse complex/large JSON files or any Web API response into rows and columns so you can load data into a database, e.g. SQL Server database. Download SSIS PowerPack to try many other automation scenarios that were not discussed in this article.

Keywords:

Google BigQuery Integration with SQL Server | How to extract data from google bigquery in SSIS? | How to read data from Google BigQuery API? | Loading BigQuery Data into SQL Server. | BigQuery to SQL Server | SSIS Google Big Query Integration | SSIS Google BigQuery Import  JSON File | SSIS Google BigQuery Export data

The post How to read / write data in Google BigQuery using SSIS appeared first on ZappySys Blog.

Elasticsearch is a powerful engine that allows you to store, aggregate, and, most importantly, search data in a very analytical way. In this tutorial, you will learn how to bulk load data from SQL Server to Elasticsearch with SSIS (part of SQL Server) and ZappySys PowerPack.

The scope of this article will be to show how to import records from SQL Server into Elasticsearch index as JSON documents using Elasticsearch Bulk API. In a nutshell, we will retrieve IDs and names of the products from a SQL table, transform each row into a JSON and lastly, index each JSON in Elasticsearch under record’s corresponding ID. This tutorial is going to be your stepping stone to use any Elasticsearch API as a destination.

We will be using these SSIS components of PowerPack to make things work:

	JSON Generator Transform
	Web API Destination

Let’s begin?

Prerequisites

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Northwind database deployed on your machine (we will use it to load data from; download it).
4. Elasticsearch instance up and running.
5. ZappySys SSIS PowerPack installed.

<New ZS-OAUTH Connection>

Step-by-Step – Bulk loading data from SQL Server to Elasticsearch with SSIS

Load data from SQL Server database first

In this section, you will perform several basic steps to start the package: load data from SQL Server and prepare it for further steps.

1. Create a new SSIS package and drag a Data Flow Task into the Control Flow from the SSIS Toolbox.
   

   Dragging and dropping Data Flow Task into Control Flow

2. Open Data Flow Task and then add OLE DB Source.
3. Configure OLE DB Source to take data from Products table, Northwind database.
4. Choose ProductID and ProductName as columns you will use.
   

   Load data to Elasticsearch using SSIS and ZappySys Web API Destination component.

Transform data into JSON documents

To add data into Elasticsearch firstly we need to prepare it with JSON Generator Transform. It will help us convert table rows into JSON documents.

JSON String to Create Index data for the Bulk API call

1. Close the window and then drag Derived Column component from the SSIS Toolbox.
2. Furthermore, Add two new columns and name them 

   RowHeader

    and 

   RowFooter

   :
   

   Derived Column configuration to add prefix and header and footer to a JSON

   In the Expression column give them values:

   • RowHeader:
     

     "{ \"index\": { \"_index\": \"shop\", \"_type\": \"products\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n"

   • RowFooter:  

     "\n"

3. Now drag JSON Generator Transform from the SSIS Toolbox and open it.
4. Right mouse click on item Mappings and select

   Add Element(s) (Below this node)

   :
   

   Convert SQL table rows into JSON using JSON Generator Transform

5. Once a dialog appears, select

   ProductName

    as Source Column and

   name

    as Output Alias:
   
   

   Map table column to JSON property using JSON Generator Transform

6. You should get a similar view:
   

   JSON Generator Transform configuration to create JSON from a SQL table

7. Then go to Header / Footer tab and:
   • Select

     Direct String

      option.
   • Input

     <%RowHeader%>

      in Header String option.
   • Input

     <%RowFooter%>

      in Footer String option.
   

   JSON Generator Transform configuration to convert rows into JSON suitable for Elasticsearch Bulk operation

8. Close the window.

JSON String to Update Index data for the Bulk API call

From the above section “JSON String to Create Index data for the Bulk API call” follows all the steps From Step-1 to Step-8.

Just for (2nd step) Step-2 use below expression for  

RowHeader

RowFooter

Derived Column configuration to add prefix and header and footer to a JSONIn the Expression column give them values:

• RowHeader:
  

  "{ \"update\": { \"_index\": \"shop\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n{ \"doc\" : "

• RowFooter:  

  "}\n"

JSON String to Delete Index data for the Bulk API call

From the above section “JSON String to Create Index data for the Bulk API call” follows steps Step-1 and Step-2.

Just for (2nd step) Step-2 use below expression for  

RowHeader

Derived Column configuration to add prefix and header and footer to a JSON

In the Expression column give them values:

• RowHeader:
  

  "{ \"delete\": { \"_index\": \"shop\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n"

Finally, Elasticsearch & SQL Server integration

1. Now drag Web API Destination SSIS component into the Data Flow and open it for editing.
2. In Select Connection section press

   <New ZS-HTTP Connection>

   .
   NOTE: If your ElasticSearch instance is hosted as AWS Managed Service then select  

   <New ZS-OAUTH Connection>

   . rather than ZS-HTTP. When OAuth UI launches select AWS v4 Provider. For more information on calling REST API on AWS check this article.
3. Once HTTP Connection Manager window opens configure connection to your Elasticsearch instance:
   • Set Web Url, which points to your Elasticsearch instance.
   • Set Credentials Type to

     Basic - UserID/Password

      (or other appropriate authentication method).
   • Finally, set User Name and Password:
     
     

     Configure SSIS HTTP Connection to connect to Elasticsearch

4. Close configuration window. Now it’s time to configure Web API Destination.
5. All that is separating you from getting those rows into Elasticsearch is:
   • Setting Input Column for Body to

     ZS_JSON_OUT

      for Create and Update Index Data and for Delete Index Data Input Column for Body to

     RowHeader [Derived Column]

     .
   • Setting URL to

     http://localhost:9200/_bulk

     .
   • Setting HTTP Request Method to

     POST

     .
   

   Index SQL Server Database data in Elasticsearch using SSIS component Web API Destination

6. Then open Batch Settings (For Body) tab:
   • Enable batch submission.
   • Set Body Batch Size, e.g. 1000. Make sure this number is even, otherwise you may get into problems (Elasticsearch Bulk request has special JSON request body format, as you perhaps noticed).
   

   Web API Destination batch settings configuration

7. Close the window and run the package! You should see green lights, telling you everything is OK:
   

   Successful data load from SQL Server to Elasticsearch

8. We can also query Elasticsearch by using one of its Search APIs – URI Search to see if we successfully indexed data:
   

   SQL Server data index in Elasticsearch

9. Rows number match in SSIS and in search results, thus everything is good. You are done.

What if I want more? After upserting data use Web API Destination further

Let’s say you have a requirement to do something with rows that were freshly indexed in Elasticsearch. Then you will need somehow to distinguish between created and updated records in Elasticsearch. And that won’t be difficult because Web API Destination acts not only as Destination but as Transformation as well. So one thing you have to do is connect JSON Parser Transform downstream to Web API Destination. It will parse Elasticsearch HTTP JSON response – which is returned by Web API Destination – into columns, which you can later easily redirect using Conditional Split:

1. Add JSON Parser Transform and connect it to Web API Destination:
   

   Use JSON Parser Transform to get Elasticsearch HTTP JSON response when integrating SQL Server & Elasticsearch

2. Make sure you have selected

   ResponseText

    as Select Input JSON Column.
3. Then go to Bulk API and copy/paste “the result of this bulk operation” JSON response:
   

   Using JSON Parser Transform to parse JSON response from REST HTTP request

4. Set

   $.items[*]

    as the filter.
5. Optional step. You won’t find a response sample for every Elasticsearch API call. In that case, you may want to use Postman to make an HTTP request to Elasticsearch and get a sample response:
   

   Use Postman to make a request to Elasticsearch and get a sample response to be used in JSON Parser Transform

   NOTE: Don’t forget to set the username and password in Authorization section!
6. Then use a Conditional Split and Trash Destination to redirect the rows:
   
   Input these clauses:
   • Inserted:
     

     REPLACENULL([index.result],"") == "created" || REPLACENULL([create.result],"") == "created"

   • Updated:
     

     REPLACENULL([index.result],"") == "updated" || REPLACENULL([create.result],"") == "updated"

7. As a result, new index records will be redirected to one destination, while updated records – to the other:
   

Overall you can use Web API Destination HTTP JSON response for other useful things as well, e.g. determine on how many replica shards record was indexed. JSON response will depend on which Elasticsearch API and which method you use.

Delete Index by making an API call.

If you want to delete the index by making Delete API call you can. Let’s make that call using the Rest API Task. Configure it like below screen and click on the Test Request button.

Rest API Delete Method

Things went bad: Error handling & debugging

Incidentally, you may incorrectly construct JSON for Web API Destination body. Elasticsearch nodes may go offline or go out of memory. In any case you may want to know when that happens and take actions accordingly. For that purpose you have to redirect failed requests from Web API Destination to some other destination:

1. Add a Derived Column above Web API Destination with expression

   (DT_WSTR,4000)ZS_JSON_OUT

    and name it

   JsonAsString

   . This will let you see what JSON you are actually passing.
2. After that, add database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don’t forget to set

   Redirect row

    option for both, Error and Truncation columns:
   
3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled in Elasticsearch

Call ElasticSearch API hosted on AWS

If your ElasticSearch instance is hosted as AWS Managed ElasticSearch Instance then select  

<New ZS-OAUTH Connection>

Call AWS Hosted ElasticSearch REST API in SSIS (V4 Request Signing)

Common Errors

Deployment to Production

Download a sample package

Be sure to download a sample SQL Server 2008 SSIS package, in case you want to try it right away (you can upgrade it to a higher version).

Conclusion. What’s next?

In conclusion, in this article we have learned how to bulk load data from SQL Server and upsert it in Elasticsearch index. We used JSON Generator Transform to help us convert database table rows into JSON documents. In addition, Web API Destination was very helpful in automatically creating and making HTTP requests to Elasticsearch instance and indexing our data as a result. This tutorial was quite simple and straightforward, yet it is a good starter to use other Elasticsearch APIs and their features. From now on Elasticsearch reference is your best friend in learning what APIs exist, what are their methods and how HTTP requests should look like.

Keywords

How to import data from SQL Server to Elasticsearch | How to load data into Elasticsearch | Using Elasticsearch to index SQL Server | Elasticsearch and SQL Server integration | How to use Elasticsearch together with SQL Server | Upsert SQL Server data into Elasticsearch

The post Loading data from SQL Server to Elasticsearch with SSIS appeared first on ZappySys Blog.

In our previous article, we discussed how to load data into MongoDB (Insert, Update, or Upsert Mode) with various options. In this post, we will focus on creating a MongoDB Document for the load process. We will see how to produce BSON (MongoDB-specific JSON that may include functions such as ObjectID and ISODate).

This article assumes you have installed SSIS PowerPack and watched the introduction videos of SSIS JSON Generator Transform and MongoDB Destination Connector

Create MongoDB document Example (Produce JSON, BSON for Load)

Here is the screenshot, which shows how to fetch SQL Server data > create a MongoDB document in JSON / BSON format, and load it into a MongoDB collection

The high-level steps we performed are

1. Used OLEDB Source to fetch data from SQL Server. Used a query like below
   

   select CustomerID, 'ISODate("' + convert(varchar(30),getdate(),126) + '")' as DOB from Customers

   In the above query, you will notice that we are building some BSON fragments, which will be treated as raw values in the JSON generator. Also, we are using the convert function from SQL Server to produce an ISO date format (Style=126 will produce an ISO date)
2. Used the SSIS JSON Generator Transform to produce a JSON document for each customer. Define a custom layout using the drag-and-drop interface. You can control how many documents you want to output using settings on Transform (Single document for all input rows or one document per row in the Main dataset)
3. Load JSON documents coming from upstream into MongoDB using the SSIS MongoDB Destination Connector. We changed Action=Upsert and set ColumnsForLookup to CustomerID (Consider this as Primary Key or your Join Column to find existing document)
4. On Target, we mapped __DOCUMENT__ rather than individual columns, so that we can load the full document.
   

   Sample SSIS Package – Create MongoDB Document in BSON format (JSON with functions). Update or Insert Mode

Configure JSON Generator Transform – Create JSON Document

If you want to learn how to configure the SSIS JSON Generator Transform, then watch the video here. Important thing to remember in the configuration is to check the Treat value of this column as Raw JSON

Create MongoDB Document in JSON or BSON format – Load into Collection (Upsert – Update or Insert)

Configure MongoDB Destination – For Upsert (Update or Insert Mode)

By default, the SSIS MongoDB Destination Connector performs an insert, but if you want to perform an Upsert (i.e., insert if not found, else update), you have to change a few settings.

1. Change Operation to UpdateOrInsert
2. Set ColumnsForLookup (e.g., in our case, it was just one CustomerID). If you have multiple columns for join, then use a comma-separated list (e.g., City, State, Zip)
3. If you want to affect various documents if more than one match is found for the join criteria, then use multi=true as shown in the screenshot below (see LoadOptions )
   

   Configure the SSIS MongoDB destination for the Upsert operation

Conclusion

In this post, you have seen how easy it is to create MongoDB Documents in JSON / BSON format and load them into a MongoDB collection using ZappySys SSIS PowerPack. This innovative drag-and-drop approach can simplify the entire ETL process if you are in a time crunch and don’t know scripting languages.

The post How to create MongoDB documents (JSON, BSON) for loading in SSIS appeared first on ZappySys Blog.

In this article we will look at how to Read / Write Amazon DynamoDB in SSIS. ZappySys developed many AWS related components but in this article we will look at 3 Tasks/Components for DynamoDB Integration Scenarios (Read, Write, Update, Bulk Insert, Create / Drop Table etc.). We will discuss how to use SSIS DynamoDB Source Connector , SSIS DynamoDB Destination Connector and SSIS DynamoDB ExecuteSQL Task . DynamoDB Source connector supports familiar SQL Like query language so any DBA or SQL user can learn it in no time. ZappySys developed highly sophisticated engine to turn your SQL Query into native DynamoDB query. For more information check this help file

Extract data from Amazon DynamoDB in SSIS

Video Tutorial – Read/Write/Update data in Amazon DynamoDB in SSIS

Below video tutorial will show you from start to end how to use ZappySys Components to perform various AWS DynamoDB Integration Tasks such as Reading Bulk data, Writing bulk data, Executing ad-hock Command for DynamoDB (such as Create/Drop Table). Get Row Count etc.

Using Amazon DynamoDB Emulator for Local Testing

If you don’t have AWS Cloud account and you still want to try DynamoDB related options in SSIS then you can use FREE Local Emulator for DynamoDB.

1.  Download it from here.
2. Extract to local folder (e.g. c:\tools\dynamodb_local
3. Create a batch file there call it RunDynamo.bat and you can put command like below.
   

   "java.exe" -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb

   Or (Java Home Path not set use below)
   

   "C:\Program Files\Java\jdk--YOUR-VERSION\bin\java.exe" -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb

4. Double click Batch file to launch DynamoDB Local instance

Connect to DynamoDB in SSIS

Now lets look at step by step how to Connect to DynamoDB in SSIS. For Simple demo we will use  ZS Amazon DynamoDB ExecuteSQL Task but in next section we will show you How to Read / Write DynamoDB Data using other SSIS Components.

Create SSIS DynamoDB Connection / List Tables Example

1. Installed SSIS PowerPack if you have not done it.
2. Open SSIS Package
3. Drag and drop ZS Amazon DynamoDB ExecuteSQL Task  from SSIS Toolbox
4. Double click to edit the task
5. Click New button next to Connection Dropdown
6. Select DynamoDB from Service Type. If you are connecting to Local DynamoDB then check DynamoDB Local option else enter your AWS Account Access Key and Secret Key to connect to Cloud Instance.
   

   Create new Amazon DynamoDB Connection in SSIS – Use Amazon DynamoDB Execute SQL Task – List Tables Example

7. Click Test Connection to confirm and click OK to save Connection.
8. Back to ExecuteSQL Task UI, Click on Examples dropdown and select command like List Tables.
9. Click Test / Preview button.

Create DynamoDB Table Example

Now lets look at how to create DynamoDB table in SSIS Amazon DynamoDB ExecuteSQL Task.

Change your Command Type Option to CreateTableSafe (Safe Suffix means, do not throw error if table already exists).

Enter the following Command and Click Test / Preview Result button to create new table called Customers. Primary Key for table is CustomerID.

{
    /*For more info visit : http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_CreateTable.html */

    TableName: 'Customers',
    Wait : true, /* Wait until table status is ACTIVE */ 
    AttributeDefinitions: [
        /*Below two columns used in Composite Primary Key */
        {
            AttributeName: 'CustomerID',
            AttributeType: 'S'
        }
    ],
    KeySchema: [
        {
            AttributeName: 'CustomerID',
            KeyType: 'HASH'
        }
    ],
    ProvisionedThroughput: {
        ReadCapacityUnits: 5,
        WriteCapacityUnits: 5
    }
}

Write data to Amazon DynamoDB in SSIS

Loading data into Amazon DynamoDB can be easily done using SSIS DynamoDB Destination Connector. It supports loading data two different ways. You can insert Raw JSON documents or Masp individual columns too. Let’s look at both ways.

Create Sample DynamoDB Table (via AWS Console)

Load data into DynamoDB (Column Mappings Mode)

First lets look at how to load sample Customers data into DynamoDB Table using Simple column mapping mode.

Insert JSON documents into DynamoDB (RAW Mode)

Previous simple approach can be used if you have simple JSON structure in DynamoDB (i.e. no nested array). Now let’s look at how to create nested JSON documents and load into DynamoDB using RAW loading approach. With this method you can create nested JSON from multiple SQL Server tables or other sources (e.g. File, Oracle, MySQL, DB2).

You can use JSON Generator Transform or Template Transform to generate JSON documents in Data flow and then load into DynamoDB. Check these article to learn more on JSON Generator use cases.

Read from Amazon DynamoDB in SSIS

Now lets move onto the next part – How to read data from DynamoDB in SSIS. DynamoDB is NoSQL database and it doesn’t have support for SQL Query language like any other RDBMS. But no worry.. If you using SSIS DynamoDB Source Connector then it supports custom SQL query language so it turns SQL into DynamoDB native API commands. This makes it super easy for any SQL Developer to use connector in drag and drop manner saving huge amount of time.

Let’s see how to read data from DyanmoDB Table

1. Drag data flow

Specifying Options in WITH Clause of DynamoDB Query:

• Specify SCAN mode using SCAN option
  

  select * from mytable where SomeColumn > 55 WITH(SCAN)

• Specify SCAN mode with LIMIT of 1000 rows
  

  select * from mytable where SomeColumn > 55 WITH(SCAN, LIMIT=1000)

• Specify QUERY mode using QUERY option
  

  select * from mytable where MyHashKeyColumn = 'XYZ' And MyRangeKey > '1223' WITH(QUERY)

• Specify QUERY mode using QUERY option, and use of INDEX
  

  select * from mytable where MyHashKeyColumn = 'XYZ' And MyRangeKey > '1223' and Phone LIKE '111-%' WITH(QUERY, INDEX=idxPhone)

Query Amazon DynamoDB with Date/Time column

There is no Datetime datatype in DynamoDB but if you are storing your date values in ISO 8601 DateTime Format as string then your query will understand automatically if you specify >, >=, <, <= or BETWEEN Filter Condition

select * from Orders where OrderDate > '2015-12-31T23:59:59.000Z' WITH(SCAN)

Using SSIS variable to query DynamoDB Date/Time column

If you want to make your query dynamic then you can use SSIS variable placeholder (use quote around it). If your variable DataType is datetime then you can format it using following way so it has ISO date format. If your variable DataType is String then make sure its correctly formatted.

select * from Orders where OrderDate > '{{User::vMyDate,yyyy-MM-ddTHH:mm:ssZ}}' WITH(SCAN)

Click here to see various ISO date formats

Amazon DynamoDB Query Examples (DynamoDB Filter) :

-- Selecting all columns 
select * from CustomerTest

-- Selecting multiple columns 
select CustomerID,CompanyName from CustomerTest

-- Selecting multiple columns, where some attribute name has space in it
select CustomerID,[Contact Title] from CustomerTest

-- Find records where City attribute is missing (Missing is different than actual NULL value {City: null, Id:1} -vs- {Id:1} )

select * from CustomerTest Where City IS  NULL 
-- Find records where City attribute is NULL (i.e. attribute is found but value is NULL e.g. {City: null, Id:1} )

select * from CustomerTest Where City =  NULL 

-- Using Where clause with simple condition, select all columns
select * from CustomerTest Where CustomerID='ALC3R'

-- Limiting records returned from query using TOP clause (Similar as LIMIT found in some RDBMS query engine)
select top 7 * from CustomerTest

-- Using OR  
select * from CustomerTest Where CustomerID = 'ALFKI' OR CustomerID = 'BOLID'

-- Using AND  
select * from CustomerTest Where CustomerID = 'ALFKI' AND Age >
 3

-- Using comparison operators 
select * from CustomerTest Where CustomerID <> 'ALFKI'

select * from CustomerTest Where CustomerID != 'ALFKI'

select * from CustomerTest Where Age > 5

select * from CustomerTest Where Age >= 5

select * from CustomerTest Where Age < 5

select * from CustomerTest Where Age = 5 AND CustomerID = 'C5'

Amazon DynamoDB Source Connector Screenshots

Amazon DynamoDB Source – Configure Connection

Query Amazon DynamoDB – Read from Table Mode

Query Amazon DynamoDB – Read from Table Mode

Amazon DynamoDB Source – Select Columns

Amazon DynamoDB Source – Select Columns

Query Amazon DynamoDB – Read from nested array using JSON Path

Query Amazon DynamoDB – Read from nested array using JSON Path

Amazon DynamoDB Source – Extract data from DynamoDB

Update Documents in DynamoDB

Now let’s look at how to Update Documents in DynamoDB. Amazon DynamoDB does not allow Bulk Update so you have to do row by row processing. You have to submit UpdateItem Command as per this document. We will update Document for matching condition and update specific columns in this example. By default Document is matched By Key unless you supply additional Condition in the command (e.g. “ConditionExpression”: “LastPostedBy = :val2” ) .

1. Drag ZS Dummy Data Source and configure like previous section with Customers Template. You can also use OLEDB Source for read DB data.
2. Now drag ZS Template Transform. You can enter command  like below to match record by Key and Update CompanyName and Phone fields (for demo we added -MY-NEW suffix in the data )
   

   {
       "TableName": "Customers",
       "Key": { "CustomerID": { "S": "<%CustomerID%>" } },
       "UpdateExpression": "set CompanyName = :val1, Phone = :val2",
       "ExpressionAttributeValues": {
           ":val1": {"S": "<%CompanyName,JSONENCODE%>-MY-NEW"},
           ":val2": {"S": "<%Phone,JSONENCODE%>-MY-NEW"}
       }
   }

   If you have multi Key then use like this (e.g. CustomerID + ProductID )
   

   "Key": { "CustomerID": { "S": "<%CustomerID%>" } ,  "ProductID": { "S": "<%ProductID%>" }   },

   Also you can add additional condition on the top of Key JOIN criteria like below (e.g. Match Country too)
   

   {
       "TableName": "Customers",
       "Key": { "CustomerID": { "S": "<%CustomerID%>" } },
       "UpdateExpression": "set CompanyName = :val1, Phone = :val2",
       "ConditionExpression": "Country = :val3",
       "ExpressionAttributeValues": {
           ":val1": {"S": "<%CompanyName,JSONENCODE%>-MY-NEW"},
           ":val2": {"S": "<%Phone,JSONENCODE%>-MY-NEW"},
           ":val3": {"S": "<%Country,JSONENCODE%>"}
       }
   }

    
   

   Update Amazon DynamoDB Table – Use SSIS Template Transform

3. Connect Template Transform to ZS DynamoDB Destination
4. Now drag ZS Amazon DynamoDB Destination
   1. On first tab Select Connection
   2. On second tab Change Action to Update, Select Table as Customers
   3. On Mappings tab Map TemplateOutput column to __DOCUMENT__ column. Remove any other mappings. If you dont see columns in Target just click Refresh.
5. Thats it now run Data flow to Update records. AFter execution you can review updated records in DynamoDB.

Call/Execute Ad-Hoc DynamoDB commands

Sometimes you have need to call ad-hoc Amazon DynamoDB commands which are different than read/write. For that you can use Amazon DynamoDB ExecuteSQL Task.

Call Amazon DynamoDB API Commands (Create/Drop Table, Get, Put)

Conclusion

Amazon DynamoDB is one of the most popular Managed NoSQL database in cloud at fraction of cost of traditional RDBMS. Extracting/Writing/Updating data into DynamoDB can be challenge without right toolset. ZappySys AWS Connectors/Tasks makes it super easy for various AWS Integration scenarios (S3, Redshift, DynamoDB, JSON, Redshift etc). Try SSIS PowerPack for FREE to uncover many more things you can do.

The post How to Read / Write Amazon DynamoDB in SSIS appeared first on ZappySys Blog.

SQL Server 2012 and all previous versions lacking native JSON support, same theory is true for their ETL Platform SSIS. If you are one of them who wants to export JSON from SQL Server or any other RDBMS like MySQL, Oracle then you can easily output complex JSON using any of the following components.

Five ways of creating JSON inside SSIS

ZappySys toolset provides multiple ways creating JSON inside SSIS. You can choose method suitable for your scenario. Each method has pros and cons so choose it as per your need.

JSON File Destination

SSIS JSON Destination can be used to generate simple or complex JSON files out of relational data source such as SQL Server, Oracle, MySQL. In few clicks you can create desired JSON Shape from single or multiple tables (Parent/Child Nested Documents).

PROS: Multiple data flow components can be attached. Which means unlike Export JSON Task you are not limited to OLEDB or ODBC Sources only.

CONS: When you remove upstream component it may break Dataset mappings. Also server side Lookup Option is not available. File Splitting Options not available.

Here are some articles for use case of this component.

JSON File Destination – Generate JSON in SSIS from multiple inputs

JSON Generator Transform

SSIS JSON Generator Transform can be used to generate single or multiple JSON documents from any type of data sources (e.g. SQL Server, MySQL, Flat File, Excel) inside data flow task. It takes multiple inputs and outputs JSON String Output Column which can be used to feed JSON to other downstream system.

PROS: Multiple data flow components can be attached. Which means unlike Export JSON Task you are not limited to OLEDB or ODBC Sources only.

CONS: When you remove upstream component it may break Dataset mappings. Also server side Lookup Option is not available. File Splitting Options not available. Output is JSON String Column.

Here are some articles for use case of this transform.

Preview JSON – Generate for BigQuery Table Insert

Export JSON Task

SSIS Export JSON File Task can be used to generate simple or complex JSON files out of relational data source such as SQL Server, Oracle, MySQL. It also support exporting directly to cloud (e.g. Azure Blob Storage, Amazon AWS S3, Secure FTP). In few clicks you can create desired JSON Shape from single or multiple tables (Parent/Child Nested Documents)

PROS: Simple to use with flexible options to export to file, variable, AWS S3, Azure Blob, FTP / SFTP location. Allows to split records. File Splitting Options available (Split By Size  / Row Count).

CONS: You can only use limited data sources as Dataset input (e.g. OLEDB / ODBC or SQL Server).

Here are some articles for use case of this task.

Template Transform

Template Transform can be used to create dynamic strings inside data flow. You can use upstream columns as placeholders or use SSIS variables anywhere in the template to create desired text (e.g. XML / JSON document).

PROS: Very Simple to use.

CONS: Cannot use multiple inputs. It may not be used to create Complex JSON and options like do not output NULL attributes is not possible.

Here are some articles for use case of this transform.

Create JSON for API Request (POST) using SSIS Template Transform

Logging Task (Control Flow Approach)

 Just like Template Transform you can use ZS Logging Task to create JSON string from Template and Variable Placeholders.

Here is how to use Logging Task to save JSON into a variable.

1. Select Variable which you like to Set on Logging Task
2. Enter your Template JSON String which you like to produce (see example below). Notice the use of yyyy-MM-dd format specifier and also JSONENCODE function.
   

   {
    ActionId: {{User::vActionId}},
    CreatedOn: "{{User::vOrderDate,yyyy-MM-ddTHH:mm:ss}}",
    Notes: "{{User::vNotes,JSONENCODE}}"
   }

3. Select Message Type = None
4. Uncheck Append, and Prefix options like below
   

   Creating JSON / XML Document using SSIS Logging Task

Here are some articles for use case of this task.

Create JSON using SSIS Export JSON Task

Now let’s look at Export JSON Task form example to create JSON. Steps listed in below sections are very similar to JSON File Destination or JSON Generator Transform.

ZappySys offers innovative Layout Editor for JSON / XML creation from multiple sources. You can use Layout Editor to produce desired JSON Layout (Layout is stored in XML format behind the scene) and then in later step you will see how to make it dynamic so values are changed at runtime.

You can read layout as direct value, read from variable or read from file from disk.

Select source connection for data

First lets create new OLEDB or ADO.net connection from where we will read data. In our case we reading from Northwind Database.

SSIS JSON Export Task – Select Source Database Connection

Create Datasets

Now add datasets which will be used to produce JSON File. One dataset will be always Root Dataset (Example: If your Root information is about customer then select customer table or query as your Main dataset. All other sections will be child sections)

Add SQL Dataset – Export JSON from SQL Query Option

Add Output Columns for Main JSON Document

Now its time to add columns for our output. If * is added by default then remove. * means output all columns. When you add columns first make sure you select correct parent node under which you want to add columns. In our case “Mappings” is parent node.

Add Output Columns for JSON Document – Column Selection Screen

Add JSON Array / JOIN Conditions for related document

Now let’s look at how to add JSON Array. For example Customer may have one or more Orders so we need to add node called Orders which may Produce JSON fragment like below

Orders : [ {...} , {...} ] 

Once customer related columns added, its time to add Orders section.

1. This is Array of documents so first highlight “Mappings” node
2. Click “Add Document Array” option from toolbar.
3. On Add/Edit Elements UI specify name for JSON Attribute and specify correct relationship to parent dataset. In our case Orders related to Customers using CustomerID key.

Add Child Section – nested JSON document , Define Parent-Child Relationship

Handling JSON Array with dummy root dataset (Multiple Arrays)

So in previous section we saw how to add array and define JOIN condition with parent dataset. Now what if you really have no parent dataset to join with ? For example we need to produce following sample JSON where Order array is related to Customer by common Column CustomerID as JOIN column. But Customer is not bound with any parent column. So how do you define JOIN condition? In such case you have to use below technique called dummy root dataset. (See next section on how to handle only one dataset using Single Dataset Array Mode)

Sample JSON (Dummy root pattern)

{
    "Customers": [
      {
        "CustomerID": "ALFKI",
        "CompanyName": "Alfreds Futterkiste",
        "Orders": [
          {
            "OrderID": 1000,
            "OrderDate": "2015-12-01"
          },
          {
            "OrderID": 1001,
            "OrderDate": "2015-12-01"
          }
        ]
      },

      {
        "CustomerID": "ANATR",
        "CompanyName": "Ana Trujillo Emparedados y helados",
        "Orders": [
          {
            "OrderID": 1002,
            "OrderDate": "2015-12-01"
          },
          {
            "OrderID": 1003,
            "OrderDate": "2015-12-01"
          }
        ]
      }

    ]
  }
}

Since root level always needs dataset, we must provide dummy dataset (only one row) even though there is no need for Parent Dataset JOIN in Customer Array. Typically you can write query like below to product dummy row in SQL Server or other database using OLEDB Source or Use Dummy Data Source with Max Rows=1 setting to produce one fake record.

SELECT 1 as Col1

Using Dummy DataSource as Input

For generating JSON which needs dummy root. Perform the following steps

1. Drag Dummy Data Source from SSIS Toolbox. Change How many rows to 1. Click OK
2. Now drag two more OLEDB sources and Configure (e.g. Customers and Orders)
3. Now drag ZS JSON FIle Destination component or ZS JSON Generator
4. Now connect Dummy datasource then Customers and then Orders
5. Double click JSON component to configure layout. You may notice that Dummy Input is set as Root by default.
6. You can now select Mappings node and click on Add Array Documents icon. Configure like below. Notice we have not selected any JOIN criteria because its joined with Dummy Root.
   

   Add array to join with dummy dataset

7. Now select newly added node and Click Add column icon to add CustomerID, CompanyName column under Customers array
8. Now select Customers array node and click Array Documents icon to Add Orders array.
9. Once Orders Array ia added you can Select Orders node and Insert new element to add OrderID and OrderDate columns
10. Finally it will look like below
    

    JSON File Destination – Generate JSON in SSIS from multiple inputs

Single dataset used as array

Now lets look at one more pattern just like described in the previous section but now we have just one nested level (Only customers and not using Orders). If you have such case then choose Single dataset pattern option in the output mode. On Export Task you may see as check box rather than dropdown. To produced desired JSON (see blow) perform these steps.

1. Select Output mode as Single Dataset Array
2. In your Layout builder click on Add Document Array Icon and name it “Customers”.
3. After that highlight newly added node > Right-Click > Add Elements under that node
4. That’s it you will now see preview like below.

{
    "Customers": [
      {
        "CustomerID": "ALFKI",
        "CompanyName": "Alfreds Futterkiste"
      },

      {
        "CustomerID": "ANATR",
        "CompanyName": "Ana Trujillo Emparedados y helados"
      }
    ]
  }
}

How to export JSON in 2D Array format

If you have to export JSON as 2D array format then you can change following option.

Here are examples of output for each option.
Select say you have source data with 2 columns id and name.

Default

[{id:1,name:"AA"}, {id:2,name:"BB"}]

Multicontent

{id:1,name:"AA"}{id:2,name:"BB"}

ArrayDocs

[{id:1,name:"AA"}, {id:2,name:"BB"}]

Array2D

[[1,"AA"],[2,"BB"]]

Array2DWithHeader

[["id","name"],[1,"AA"],[2,"BB"]]

ArrayLines

[1,"AA"][2,"BB"]

ArrayLinesWithHeader

["id","name"][1,"AA"][2,"BB"]

Preview JSON output using live preview

You can enable / disable Live preview of JSON layout as below.

JSON Output preview in SSIS Export JSON File Task

Make things dynamic (Using SSIS Variable for SQL or Layout)

There will be a time when you have use dynamic values in your JSON generation. You may have to Alias something from variable or you have to supply Variable in your SQL used for Dataset. Not to worry. ZappySys Supports multiple ways to make it dynamic. You can use SSIS Variables following 3 ways to make Layout XML dynamic.

1. Use SSIS variable inside Dataset SQL (Only for ZS Export JSON Task)
2. Anywhere inside Layout XML (Click Edit Code option)
3. Load Layout From Variable (Indirect Method)

Using Variable inside Dataset SQL

Here is how you can use Variable inside dataset SQL for ZS Export JSON Task

SSIS Export JSON Task – Using Variable for Dynamic SQL

Using Variable inside Layout XML

Here is how you can insert SSIS Variable anywhere in Layout XML. In below Example we are choosing Alias for one column from some SSIS Variable.

SSIS JSON File Destination – Using SSIS Variable for Dynamic Layout

Load Layout From Variable

If you have some need to load Layout XML from File or SSIS Variable. You can use Load Layout from Variable Option.

For example in below screenshot we are creating layout first and ave to some variable. Or you can load from external file using ZS Advanced File System Task.  Connect Logging task to Export JSON Task and you ready to produce JSON.

Set SSIS Variable value (multi-line content using placeholder)

Here is how to load Layout From any SSIS Variable. You can also load Layout from XML file or Variable which contains layout File Path.

JSON Layout from SSIS Variable

Other JSON Output Options

Execute package and verify generated JSON it should look like below (Date format may be different in your case if you have set different on JSON options tab). For clarity we have formatted JSON below but in your output it may be compact version so reduce file size.

Indent JSON Output

You can check Indent option to make JSON look pretty like below.

{
  "CustomerID": "ALFKI",
  "CompanyName": "Alfreds Futterkiste",
  "ContactName": null,
  "Orders": [
    {
      "OrderID": 10643,
      "OrderDate": "1997-08-25T00:00:00"
    },
    {
      "OrderID": 10692,
      "OrderDate": "1997-10-03T00:00:00"
    },
    {
      "OrderID": 10702,
      "OrderDate": "1997-10-13T00:00:00"
    },
    {
      "OrderID": 10835,
      "OrderDate": "1998-01-15T00:00:00"
    },
    {
      "OrderID": 10952,
      "OrderDate": "1998-03-16T00:00:00"
    },
    {
      "OrderID": 11011,
      "OrderDate": "1998-04-09T00:00:00"
    }
  ]
}

Hide NULL Attribute Option

There will be a time when you want to Hide NULL attribute rather than showing MyAttr: null this is also possible by checking Hide NULL Attribute Option on Options Tab.

Export JSON to Amazon S3, Azure Blob, Secure FTP (SFTP / FTPS)

Export JSON Task support some really good options to save file to other than local file system. If you want to export JSON directly to Amazon / Azure Cloud or FTP Server without saving to local disk then you can use Azure / FTP / Amazon Connection Manager on Target Tab of Export JSON Task. This option is only available for PRO Edition Users.

Export JSON / CSV / XML Files to Azure Blob, Amazon S3, Secure FTP Storage

Conclusion

If you want to output JSON from SQL Server or any other relational DB then its not simple task in SSIS specially when you have nested JSON documents (e.g. Customer->Orders). In this post you have seen how to use SSIS JSON Export Task to generate nested JSON from multiple datasets.

The post 5 Ways to Export JSON from SQL Server using SSIS appeared first on ZappySys Blog.