Google BigQuery is a fully managed Big Data platform to run queries against large scale data. In this article you will learn how to integrate Google BigQuery data into Microsoft SQL Server using SSIS. We will leverage highly flexible JSON based REST API Connector and OAuth Connection to import / export data from Google BigQuery API just in a few clicks.

If you are looking for a similar product inside Amazon AWS Cloud then check an article about Amazon Athena.

Prerequisites

Before we look into a Step-By-Step section to extract and load data from BigQuery to SQL Server let’s make sure you meet the following requirements:

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have basic familiarity with REST API concepts and Google BigQuery API. This post uses free Public Dataset to query BigQuery table so no billing is required to get your hands dirty 🙂 However, if you like to query your own dataset then make sure you have at least one BigQuery dataset and one table with some sample data created (this part does require billing enabled). Read a Google Quickstart article for more information on how to create a new BigQuery dataset and a table.

Understanding Google BigQuery Object Heirarchy

Google BigQuery has 3 main concepts below.

• Project
  • Dataset
    • Table
      • Query requests (each query creates a unique JobID – valid for 24 hours from which you can read data)

So in order to create BigQuery table you always need Project and then Dataset under that project. You can group objects around Project / Datasets. When you query you can supply fully qualified name of your table in FROM clause (e.g.

select count(*) from `bigquery-public-data.usa_names.usa_1910_2013`

So let’s get started.

Read data from Google BigQuery using SSIS

Basically you can query Google BigQuery data in two ways: In this article we will not cover 2nd method.

• Method-1: Query data using jobs/query method in BigQuery API. Use it if you expect to get a result in a fairly short amount of time. This API method generates a temp table which gets deleted after 24 hours. You can read data within that time frame using newly created JobId reference.
  
  NOTE: If you have records that sum up to more than 10MB of data or ~10,000 rows (assuming 1 row uses 1KB of data) then you need to proceed with two-step process as explained below.
• Method-2: Export SQL query result using jobs/insert method in BigQuery API. Use this method if you expect a query to take a long time to finish.

Now let’s see how to query Google BigQuery data using SSIS 🙂

Create Google API Project and Register OAuth App

To use any Google API firstly you have to finish two tasks:

1. Create Google API Project and obtain projectId (see next section).
2. Enable BigQuery API
3. Register your own Google OAuth App and obtain ClientId and Client Secret.

Check step-by-step instructions on how to create Google API Project and create OAuth App for Google. During instructions make sure you enable BigQuery API (screenshots may differ).

Get Google API Project ID

Once you have Google API Project created you can grab Project ID (see screenshot below). You will need Project ID in the next step when building API URL.

How to find Google API Project ID?

Configure OAuth Connection for Google BigQuery API

In order to call most of Google APIs you will need an OAuth App. If you want to use your own app then refer to how to register Google OAuth Application (Get ClientID and ClientSecret) article. For simplicity we will use Default OAuth App:

1. Right click inside Connection Managers area and click “New Connection…”
2. From the connection type list select “ZS-OAUTH” connection type.
   

   Create new SSIS OAuth API Connection Manager

3. For OAuth Provider select “Google”.
4. For OAuth App type options leave “Use Default OAuth App” selected. If you created custom OAuth App then you would need to select “Use Custom OAuth App” option.
5. For Scopes enter or select the following URLs (URL is just a permission name). Each URL must be in a separate line. For demo purposes we use only the first 3 scopes but we included a few more in case you like to test API in depth with a different permission set:
   

   https://www.googleapis.com/auth/bigquery
   https://www.googleapis.com/auth/cloud-platform
   https://www.googleapis.com/auth/cloud-platform.read-only
   https://www.googleapis.com/auth/bigquery.insertdata
   https://www.googleapis.com/auth/devstorage.full_control
   https://www.googleapis.com/auth/devstorage.read_only
   https://www.googleapis.com/auth/devstorage.read_write

6. Click Generate Token. Login using correct account if needed and then you will be prompted to click “Approve” OAuth App.
7. Once you click OK on the login screen you will see new tokens populated on the connection screen.
8. Click Test to make sure connection / tokens are valid and then click OK to save the connection.
   NOTE: Below screenshot uses Default App but we recommend you to to use Custom OAuth App (Your own Clientid / secret – Obtained in previous section)
   

   Create SSIS OAuth API Connection for Google BigQuery API

Start BigQuery Job and get JobId (Submit Query)

Once you have SSIS OAuth connection created for BigQuery API it’s time to read data from BigQuery. So, basically, there are two ways you can read BigQuery data: using query or insert method. For demo purposes we will use jobs/query method. If you want fire complex queries which can run for many minutes then refer to jobs/insert method.

If you are expecting more than 10MB of data (~10K records) then you have to split data extract in two steps:

1. Step-1 : Call jobs/query method using REST API Task to submit SQL query. This steps returns jobComplete: true if supplied query execution is done within timeoutMs parameter you included. By default its 10 seconds so if your query is going to take longer than 10 seconds then its good idea to add bigger timeout that way you dont need step#2. Once API call is done you will Get JobId in the response. Save that JobId in a variable for later use. This job result is valid for 24-hours only.
2. Step-2 (Optional) – You can add another optional REST API Task after previous step to Wait until Job is completed. For simplicity, this article will Omit setting up Status Check but its very simple….Use below settings
   1. Drag new REST API Task from toolbox. Connect it to Step-1 and double click to configure as below.
   2. Select URL From Connection mode,
   3. Select OAuth connection
   4. Enter URL as https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/jobs/{{User::JobId}} , Method: GET
   5. On Response Settings Tab, Select Response Content Type JSON and for content filter enter $.status.state
   6. On Status Check Tab, check Enable Status Check Loop and enter DONE in SuccessValue field
   7. Doing this setup will make sure we do not query data until Job Status is DONE (System keeps checking every 5 seconds)
3. Step-2 : Read Job result using JSON Source in a Data Flow Task.

Now lets see how to configure REST API Task to submit query and extract JobId (which will be used in next section).

1. In the Control Flow designer drag and drop ZS REST API Task from the SSIS toolbox.
2. Double click the task to configure it.
3. Change Request URL Access Mode to [Url from connection].
4. From the connection dropdown select OAuth connection created in the previous section.
5. Enter URL as below (replace YOUR-PROJECT-ID with a valid Project ID obtained in the previous section):
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-PROJECT-ID/queries

6. Select Method as POST.
7. Enter Body (Request Data) as below (change your query if needed; we used Public dataset for the demo): If you do not specify timeout then default is 10 seconds only. Also in this call we only care about JobID from response so we just added maxResults=10 because in 2nd step we will get all rows by doing pagination.
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000"
   }

   To use a Standard SQL query instead of Legacy SQL, add useLegacySql property and set it to false:
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000",
      "useLegacySql": false
   }

   For all possible parameters refer to jobs/query method documentation.
   NOTE: You can also supply location Parameter in the Body JSON to indicate where job should run. For non EU / US datacenters we suggest you to supply this parameter. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location.
   Example Use Of Location
   

   { "location":"us-east1",  "maxResults": 10, "query": "SELECT title FROM [bigquery-public-data:samples.wikipedia] LIMIT 10" }

8. Select Body Content Type as application/json.
9. Click Test Request/Response. If things go well you will see JSON content. Then just copy JobId from the response and save for later use:
   
10. Now go to Response Settings tab.
11. Select ContentType=Json.
12. Enter below expression to extract JobId from the response. You will need this Id to read SQL query output (this JobId is valid for 24 hrs):
    

    $.jobReference.jobId

13. Choose save response content to save to a variable. Select <New Variable…> option. When prompted give a name to your variable (i.e. vJobId) and for the value field paste JobId copied from the above step:
    
    
    NOTE: This JobId expires after 24-hrs so while you are designing a package and not running it then design time value is used for testing/previewing. So make sure your JobId is valid. If needed click Test Request/Response to grab a new JobId and update the variable with a new value so you can preview data without running a full package.
14. Click OK to save the UI.

Read BigQuery SQL result (for specified JobID)

Now let’s look at how to read data from BigQuery

Configure Google BigQuery Web Request (URL, Method, ContentType, Body etc.)

1. In the Control Flow designer drag and drop Data Flow Task from SSIS toolbox.
   
2. Double click Data Flow Task and drag and drop ZS JSON Source (For API/File) from SSIS toolbox.
   
3. Double click JSON Source to edit and configure as below
4. Enter URL as below. Replace YOUR-API-PROJECT-ID with the API Project ID obtained in the previous section.
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?maxResults=10000

   –OR– (Use below if your Job ran outside US / EU data center)
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?location=YOUR_REGION_ID&maxResults=10000

   NOTE: location Parameter indicates the geographic location of the job. This is Required parameter except for US and EU. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location. You can supply same location parameter for first step when you submit the query

   Failure to supply location parameter for non US /EU users may result in 404 NotFound Error.
   

5. Check Use Credentials option and select OAuth Connection Manager created in the previous section.
6. For HTTP Request Method select GET.
   

Read data from Google BigQuery from Temp Job result

Configure Filter

We need to transform a single JSON response into multiple rows so we need to apply a correct filter.

1. On JSON Source go to Filter Options tab.
2. In the Filter field enter $.rows[*] or click [Select Filter] button to browse hierarchy you want to extract.

Select Filter for JSON Response

Configure BigQuery API Pagination Settings

Most of modern APIs usually implement some sort of pagination technique so you get a part of data in each request rather than all in one go. Thus if you want more you can paginate through pages until the last page is reached. You can also read Understanding REST API Pagination in SSIS article to learn more about pagination in SSIS.

BigQuery API returns pageToken attribute in response JSON if more data is found for requested query result. You can then pass pageToken in the next URL in this format: http://my-api-url/?pageToken=xxxxxxxxxx

Now lets configure JSON Source to automate this pagination for us. On JSON Source go to Pagination Tab and enter the following two settings:

1. Set Next Link as $.pageToken.
2. Set Suffix for Next URL as &pageToken=<%nextlink%>.

See the screenshot below to get more clarity:

Configure BigQuery API Pagination on SSIS JSON Source

Configure Array Transformation

Now the last thing we have to configure is special 2-dimensional JSON array format used by BigQuery API:

1. On the JSON Source UI go to 2D Array Transformation tab.
2. Enter the following settings:
   1. For Transformation Type select Transform complex 2-dimensional array.
   2. For Column Name filter enter $.schema.fields[*].name.
   3. For Row Values Filter enter $.f[*].v.

JSON Array Transformation Options

Preview Data and Save UI

That’s it click Preview to see some data. If you entered sample JobID for your User::vJobID variable then you will see some data. In the next section we will see how to load Google BigQuery data into SQL Server. You can click Columns Tab to review Metadata.

Click OK to save UI and generate Metadata.

Configure Target – Load Google BigQuery data into SQL Server

Now you can connect your JSON Source to any target such as ZS Trash Destination or a real database destination such as OLEDB Destination.

If you wish to dump data from Google BigQuery to a SQL Server table then just perform the following steps:

1. Drag and drop OLEDB Destination from SSIS Toolbox. Rename it to something like SQL Server Table.
2. Double click on OLEDB Destination.
3. Click New to create a new connection > Configure connection > Click OK.
4. Click on New to create a new table > Rename default table name > Click OK.
5. Click on Mappings tab and click OK to save UI with default mappings.

Configure SSIS OLEDB Destination – Google BigQuery to SQL Server Import

Execute Package – Loading BigQuery data into SQL Server

SSIS Package Execution –
Loading Google BigQuery Data into SQL Server

Create / Delete Google BigQuery Dataset using API call

As we mentioned before, you need dataset before you can create a table. Most common way to create dataset is via User Interface but what if you like to automate from your SSIS Package or other workflow? Here is how you can create or delete. Basically you can use REST API Task to send CREATE or DROP command for Dataset object

Create Google Dataset

To create dataset configure REST API Task using below settings (We will call this API)

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/datasets

4. Request Method as POST
5. Enter Request Body as below (change YOUR_DATASET_NAME – e.g. TestDataset )
   

   {"datasetReference": { "datasetId": "YOUR_DATASET_NAME", "projectId": "{{User::ProjectId}}"} }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Dataset already exists then you can go to Error Handling Tab (See next section for screenshot) and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Delete Google BigQuery Dataset

For deleing dataset you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Create / Delete Google BigQuery Table using API call

Now let’s look at how to create /drop Table in Google BigQuery using API calls.

Create Google BigQuery Table

To send CREATE TABLE  SQL statement (DDL) we have to use same approach as we send normal SQL Query using this API call. So notice we used Standard SQL for this call by supplying  useLegacySql: false . DDL Statement must be all in one line

POST https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

Content-Type: application/json

>>>> BODY <<<<<

{
 "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
 "useLegacySql": false,
 "timeoutMs": 100000 
}

To create dataset configure REST API Task using below settings

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

4. Request Method as POST
5. Enter Request Body as below
   

   {
    "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
    "useLegacySql": false,
    "timeoutMs": 100000
   }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Table exists then you can go to enable Error Handling Tab and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Create Google BigQuery Table – API Call (Continue On Error Setting – Skip CREATE if Table / Dataset Already Exists)

Delete Google BigQuery Table

For deleing table you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Write data to Google BigQuery using SSIS – 1 Million row insert test (FAST)

Now let’s look at how easy it is to import data into Google BigQuery using SSIS. We will use the same OAuth connection we created before. To learn more about inserting data into BigQuery check tabledata/insertAll method documentation.

Make sure billing is enabled

Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

Streaming is not available via the free tier. If you attempt to use streaming without enabling billing, you receive the following error: BigQuery: Streaming insert is not allowed in the free tier.

As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call (Write  demo).

Configure OAuth Connection / Permissions

If you want to perform data insert operation in BigQuery using API calls then include the following scopes in your OAuth Connection Manager and generate a token (see our first section of this article – We already included scopes for Write operation but incase you didnt do then regenerate token with below scopes):

https://www.googleapis.com/auth/bigquery
https://www.googleapis.com/auth/cloud-platform
https://www.googleapis.com/auth/bigquery.insertdata

Create BigQuery Dataset (From UI)

For this demo first create a test dataset and one table under it like shown below (billing must be enabled on your Google API Project). To do that go to https://bigquery.cloud.google.com/welcome and configure them:

Insert data into BigQuery using API call

And here is a REST API example to insert data into a BigQuery Table:

Request URL:

https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

Request Headers:

Authorization: Bearer ya29.Gl0cBxxxxxxxxxxxxxxxxxxuEIhJIEnxE6GsQPHI
Content-Type: application/json

Request Body:

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
     {"json": {"RowId": 2,"CustomerName": "BBB"} }
   ]
}

Here is our data flow setup to achive very high throughput for Google BigQuery Data Load. We will show you how to insert one million rows in Google BigQuery in less than a minute based on below setup. We will use Multi Threading option and New Compression Option (Added in v3.1.4)

1. Dummy Data Source – Generate sample records
2. JSON Generator Transform – Generates JSON documents to send as POST request for above /insertAll API call.
3. Web API destination  – Call /insertAll API call to submit our data to BigQuery
4. (Optional) JSON Parser Transform – Parse Error Message for any response
5. (Optional) Trash Destination – Save any errors to text file for review

Google BigQuery Data Load Demo in SSIS – 1 Million Rows Insert with Multi Threads and Compression ON (Fast Upload)

Configure SSIS JSON Generator – Generate JSON for BigQuery Table Insert

Now let’s see how to build an HTTP request with JSON body and send it to BigQuery:

1. Drag Data Flow Task and double click on it.
   
2. Drag and configure your Source (for this demo we use Dummy Data Source with Customer Template). See previous section for configuration of Dummy Data Source.
   
3. Drag and drop ZS JSON Generator Transform to produce JSON from a database or file records. If your source is already sending a valid JSON then you can skip this step (e.g. SQL query is
   returning JSON). You can also read raw JSON from a very large file (new-line separated JSON) using JSON Source with Output as Raw Data option checked.
4. Connect Source to JSON Generator. Double click JSON Generator Transform to start configuring it like below.
5. Select Output Mode as Single Dataset Array and enter Batch Size 10000 (This is Max limit allowed by Google BigQuery API insertAll)
6. First right click on Mappings node and select Add Static Element
   

   Generate JSON for Google BigQuery InsertAll API request – Batch 10000 rows in a single API call

7. Enter Name as kind and value as bigquery#tableDataInsertAllRequest
   
   

   JSON Generator – Add Static Element

8. Now right click on Mappings node and select Add Document Array option
   

   JSON Generator – Add Document Array

9. Enter rows as array title and click OK
   

   JSON Generator – Name array

10. Select newly added array node and right click > Add unbound nested element  enter Output alias as json and click OK.
    

    JSON Generator – Add unbound nested element

11. Select json node and right click > Select Add Elements below this node and select multiple columns you like to send to BigQuery. Click OK to save.
    

    JSON Generator – Add Multiple Elements

12. Now let’s preview our JSON (Copy preview JSON to try in the next step – Web API destination)
    NOTE: Table name and column names are case-sensitive so make sure your JSON attribute matches exact same way.
    Here is the finished JSON Structure for next Step
    

    Sample JSON Request body for Google BigQuery insertAll API request

13. Click OK to save UI.

Configure SSIS Web API destination – Insert data into BigQuery Table

Once you have Input JSON prepared,  now let’s configure destination for BigQuery.

1. Drag and drop ZS Web API Destination.
2. Connect your JSON Generator Transform to Web API destination.
3. Configure general properties:
   

   SSIS Web API Destination –
   Configure for BigQuery Data load

4. Make sure to enter URL in this format:
   

   https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

   Make sure to replace 3 parts in above URL (MY_PROJECT_ID, MY_DATASET_ID, MY_TABLE_ID) with actual values from your Google Project and BigQuery dataset/table configuration.
5. Now you can enable Compression and Multiple Threads for higher throughput as below.
   NOTE: Compression Property was added in v3.1.4 so you may not see it if you have older version.
   

   Google BigQuery Data Loading Performance Optimization Options – Enable Multiple Threads and Compression Options

6. If you want to try test insert request from UI without running full package then go back to first tab and edit Body (Use Sample JSON generated by previous JSON Transform – You can grab from JSON Preview Panel on Generator Transform).Click Test Request / Response and confirm Success as below. You can go back to your BigQuery Portal and check one row is inserted after our test click. If everything looking good then run full package to insert all records.Sample JSON for Body:
   

   {
     "kind": "bigquery#tableDataInsertAllRequest",
     "rows": [
        {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
        {"json": {"RowId": 2,"CustomerName": "BBB"} }
      ]
   }

   

   Test Google BigQuery Table Insert – SSIS Web API Destination UI

7. Hit OK to save UI.
8. Run the package and verify data in Google BigQuery Console:
   

   Loading data into Google BigQuery using SSIS

Error Handling for BigQuery Data Load (Bulk Insert API Calls)

There will be a time when some records you insert may not go well in Google BigQuery. In such case you can read output from Web API destination and parse further with JSON Parser Transform. Check for certain values in the output. You must use JSON Parser after Web API destination (Connect Blue Arrow from Web API destination – Since its Soft Error it won’t redirect in Red Arrow ).

For example here is the sample JSON in POST Body for testing which produces error due to bad column name. When bad row found in batch all records will be rejected. Notice that error returns index of record in batch so you can identify which row went bad. It also returns column name in location attribute.

NOTE: Bad column name in 2nd record

Test Body (Bad):

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RecordID": 1,"CustomerID": "X1"} }, 
     {"json": {"Bad_Column": 2,"CustomerID": "X2"} }
     {"json": {"RecordID": 3,"CustomerID": "X3"} }, 
   ]
}

Response (For Bad Input):

{
  "kind": "bigquery#tableDataInsertAllResponse",
  "insertErrors": [
    {
      "index": 1,
      "errors": [
        {
          "reason": "invalid",
          "location": "bad_column",
          "debugInfo": "",
          "message": "no such field."
        }
      ]
    },
    {
      "index": 0,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    },
    {
      "index": 2,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    }
  ]
}

To configure error detection perform following steps.

1. Drag and drop ZS JSON Parser Transform after Web API destination
2. Click on Web API destination. Connect Blue arrow  to JSON Parser Transform
3. Configure JSON Parser Transform like below
4. Connect JSON Parser TRansform to some Destination to save error information (e.g. SQL Table or Trans destination)

Handling BigQuery Insert Errors in SSIS

Other Common Errors in BigQuery API calls

In this section we will talk about many common API errors in BigQuery.

Error: The project XXXXXXX has not enabled BigQuery

Sometimes you might get below error. To fix this error make sure you go to your Project and Enable BigQuery API

Status Code: BadRequest

Response Body: {
  "error": {
    "code": 400,
    "message": "The project bigquerytest-281915 has not enabled BigQuery.",
    "errors": [
      {
        "message": "The project bigquerytest-281915 has not enabled BigQuery.",
        "domain": "global",
        "reason": "invalid"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}

Error: 404 – Not Found: Table / Job xxxxxx

• Make sure you are setting GET request and not POST.
• Also make sure your jobId is valid because it expires after 24 hours.
• Make sure project-id supplied in URL is valid ID  (DO NOT Specify Alias, use internal ID for project)
• Make sure you supplied location parameter if your outside EU / US region. Reading data using this API call might fail if you failed to supply location in URL

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "notFound",
    "message": "Not Found: Table xxxxxx"
   }
  ],
  "code": 404,
  "message": "Not Found: Table xxxxxxx
 }
}

Error: 403 – Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier

If you trying call certain APIs on sandbox mode or free tier then you might get below error. To overcome this error enable billing on google bigquery. As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call.

Response Url: https://www.googleapis.com/bigquery/v2/projects/bigquerytest-281915/datasets/TestDataset/tables/Table1/insertAll

Status Code: Forbidden

Response Body: {
  "error": {
    "code": 403,
    "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
    "errors": [
      {
        "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
        "domain": "global",
        "reason": "accessDenied"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}

Debugging Web API Requests

If you need to debug actual requests made to Google server then you can use a tool like Fiddler. It’s a very handy tool to troubleshoot JSON format issues. It will allow to see how a request is made to a server.

Using Fiddler to debug Google BigQuery API requests in SSIS

Common Errors

Deployment to Production

Download Sample Package

Click Here to Download SSIS Sample Package – Google BigQuery API Read Write Create Delete (SSIS 2019, 2017, 2012)

Conclusion. What’s next?

In this article we have learned how to load data from Google BigQuery into SQL Server using SSIS (drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from Google BigQuery REST API using OAuth. JSON Source Connector makes it super simple to parse complex/large JSON files or any Web API response into rows and columns so you can load data into a database, e.g. SQL Server database. Download SSIS PowerPack to try many other automation scenarios that were not discussed in this article.

Keywords:

Google BigQuery Integration with SQL Server | How to extract data from google bigquery in SSIS? | How to read data from Google BigQuery API? | Loading BigQuery Data into SQL Server. | BigQuery to SQL Server | SSIS Google Big Query Integration | SSIS Google BigQuery Import  JSON File | SSIS Google BigQuery Export data

The post How to read / write data in Google BigQuery using SSIS appeared first on ZappySys Blog.

Elasticsearch is a powerful engine that allows you to store, aggregate, and, most importantly, search data in a very analytical way. In this tutorial, you will learn how to bulk load data from SQL Server to Elasticsearch with SSIS (part of SQL Server) and ZappySys PowerPack.

The scope of this article will be to show how to import records from SQL Server into Elasticsearch index as JSON documents using Elasticsearch Bulk API. In a nutshell, we will retrieve IDs and names of the products from a SQL table, transform each row into a JSON and lastly, index each JSON in Elasticsearch under record’s corresponding ID. This tutorial is going to be your stepping stone to use any Elasticsearch API as a destination.

We will be using these SSIS components of PowerPack to make things work:

	JSON Generator Transform
	Web API Destination

Let’s begin?

Prerequisites

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Northwind database deployed on your machine (we will use it to load data from; download it).
4. Elasticsearch instance up and running.
5. ZappySys SSIS PowerPack installed.

<New ZS-OAUTH Connection>

Step-by-Step – Bulk loading data from SQL Server to Elasticsearch with SSIS

Load data from SQL Server database first

In this section, you will perform several basic steps to start the package: load data from SQL Server and prepare it for further steps.

1. Create a new SSIS package and drag a Data Flow Task into the Control Flow from the SSIS Toolbox.
   

   Dragging and dropping Data Flow Task into Control Flow

2. Open Data Flow Task and then add OLE DB Source.
3. Configure OLE DB Source to take data from Products table, Northwind database.
4. Choose ProductID and ProductName as columns you will use.
   

   Load data to Elasticsearch using SSIS and ZappySys Web API Destination component.

Transform data into JSON documents

To add data into Elasticsearch firstly we need to prepare it with JSON Generator Transform. It will help us convert table rows into JSON documents.

JSON String to Create Index data for the Bulk API call

1. Close the window and then drag Derived Column component from the SSIS Toolbox.
2. Furthermore, Add two new columns and name them 

   RowHeader

    and 

   RowFooter

   :
   

   Derived Column configuration to add prefix and header and footer to a JSON

   In the Expression column give them values:

   • RowHeader:
     

     "{ \"index\": { \"_index\": \"shop\", \"_type\": \"products\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n"

   • RowFooter:  

     "\n"

3. Now drag JSON Generator Transform from the SSIS Toolbox and open it.
4. Right mouse click on item Mappings and select

   Add Element(s) (Below this node)

   :
   

   Convert SQL table rows into JSON using JSON Generator Transform

5. Once a dialog appears, select

   ProductName

    as Source Column and

   name

    as Output Alias:
   
   

   Map table column to JSON property using JSON Generator Transform

6. You should get a similar view:
   

   JSON Generator Transform configuration to create JSON from a SQL table

7. Then go to Header / Footer tab and:
   • Select

     Direct String

      option.
   • Input

     <%RowHeader%>

      in Header String option.
   • Input

     <%RowFooter%>

      in Footer String option.
   

   JSON Generator Transform configuration to convert rows into JSON suitable for Elasticsearch Bulk operation

8. Close the window.

JSON String to Update Index data for the Bulk API call

From the above section “JSON String to Create Index data for the Bulk API call” follows all the steps From Step-1 to Step-8.

Just for (2nd step) Step-2 use below expression for  

RowHeader

RowFooter

Derived Column configuration to add prefix and header and footer to a JSONIn the Expression column give them values:

• RowHeader:
  

  "{ \"update\": { \"_index\": \"shop\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n{ \"doc\" : "

• RowFooter:  

  "}\n"

JSON String to Delete Index data for the Bulk API call

From the above section “JSON String to Create Index data for the Bulk API call” follows steps Step-1 and Step-2.

Just for (2nd step) Step-2 use below expression for  

RowHeader

Derived Column configuration to add prefix and header and footer to a JSON

In the Expression column give them values:

• RowHeader:
  

  "{ \"delete\": { \"_index\": \"shop\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n"

Finally, Elasticsearch & SQL Server integration

1. Now drag Web API Destination SSIS component into the Data Flow and open it for editing.
2. In Select Connection section press

   <New ZS-HTTP Connection>

   .
   NOTE: If your ElasticSearch instance is hosted as AWS Managed Service then select  

   <New ZS-OAUTH Connection>

   . rather than ZS-HTTP. When OAuth UI launches select AWS v4 Provider. For more information on calling REST API on AWS check this article.
3. Once HTTP Connection Manager window opens configure connection to your Elasticsearch instance:
   • Set Web Url, which points to your Elasticsearch instance.
   • Set Credentials Type to

     Basic - UserID/Password

      (or other appropriate authentication method).
   • Finally, set User Name and Password:
     
     

     Configure SSIS HTTP Connection to connect to Elasticsearch

4. Close configuration window. Now it’s time to configure Web API Destination.
5. All that is separating you from getting those rows into Elasticsearch is:
   • Setting Input Column for Body to

     ZS_JSON_OUT

      for Create and Update Index Data and for Delete Index Data Input Column for Body to

     RowHeader [Derived Column]

     .
   • Setting URL to

     http://localhost:9200/_bulk

     .
   • Setting HTTP Request Method to

     POST

     .
   

   Index SQL Server Database data in Elasticsearch using SSIS component Web API Destination

6. Then open Batch Settings (For Body) tab:
   • Enable batch submission.
   • Set Body Batch Size, e.g. 1000. Make sure this number is even, otherwise you may get into problems (Elasticsearch Bulk request has special JSON request body format, as you perhaps noticed).
   

   Web API Destination batch settings configuration

7. Close the window and run the package! You should see green lights, telling you everything is OK:
   

   Successful data load from SQL Server to Elasticsearch

8. We can also query Elasticsearch by using one of its Search APIs – URI Search to see if we successfully indexed data:
   

   SQL Server data index in Elasticsearch

9. Rows number match in SSIS and in search results, thus everything is good. You are done.

What if I want more? After upserting data use Web API Destination further

Let’s say you have a requirement to do something with rows that were freshly indexed in Elasticsearch. Then you will need somehow to distinguish between created and updated records in Elasticsearch. And that won’t be difficult because Web API Destination acts not only as Destination but as Transformation as well. So one thing you have to do is connect JSON Parser Transform downstream to Web API Destination. It will parse Elasticsearch HTTP JSON response – which is returned by Web API Destination – into columns, which you can later easily redirect using Conditional Split:

1. Add JSON Parser Transform and connect it to Web API Destination:
   

   Use JSON Parser Transform to get Elasticsearch HTTP JSON response when integrating SQL Server & Elasticsearch

2. Make sure you have selected

   ResponseText

    as Select Input JSON Column.
3. Then go to Bulk API and copy/paste “the result of this bulk operation” JSON response:
   

   Using JSON Parser Transform to parse JSON response from REST HTTP request

4. Set

   $.items[*]

    as the filter.
5. Optional step. You won’t find a response sample for every Elasticsearch API call. In that case, you may want to use Postman to make an HTTP request to Elasticsearch and get a sample response:
   

   Use Postman to make a request to Elasticsearch and get a sample response to be used in JSON Parser Transform

   NOTE: Don’t forget to set the username and password in Authorization section!
6. Then use a Conditional Split and Trash Destination to redirect the rows:
   
   Input these clauses:
   • Inserted:
     

     REPLACENULL([index.result],"") == "created" || REPLACENULL([create.result],"") == "created"

   • Updated:
     

     REPLACENULL([index.result],"") == "updated" || REPLACENULL([create.result],"") == "updated"

7. As a result, new index records will be redirected to one destination, while updated records – to the other:
   

Overall you can use Web API Destination HTTP JSON response for other useful things as well, e.g. determine on how many replica shards record was indexed. JSON response will depend on which Elasticsearch API and which method you use.

Delete Index by making an API call.

If you want to delete the index by making Delete API call you can. Let’s make that call using the Rest API Task. Configure it like below screen and click on the Test Request button.

Rest API Delete Method

Things went bad: Error handling & debugging

Incidentally, you may incorrectly construct JSON for Web API Destination body. Elasticsearch nodes may go offline or go out of memory. In any case you may want to know when that happens and take actions accordingly. For that purpose you have to redirect failed requests from Web API Destination to some other destination:

1. Add a Derived Column above Web API Destination with expression

   (DT_WSTR,4000)ZS_JSON_OUT

    and name it

   JsonAsString

   . This will let you see what JSON you are actually passing.
2. After that, add database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don’t forget to set

   Redirect row

    option for both, Error and Truncation columns:
   
3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled in Elasticsearch

Call ElasticSearch API hosted on AWS

If your ElasticSearch instance is hosted as AWS Managed ElasticSearch Instance then select  

<New ZS-OAUTH Connection>

Call AWS Hosted ElasticSearch REST API in SSIS (V4 Request Signing)

Common Errors

Deployment to Production

Download a sample package

Be sure to download a sample SQL Server 2008 SSIS package, in case you want to try it right away (you can upgrade it to a higher version).

Conclusion. What’s next?

In conclusion, in this article we have learned how to bulk load data from SQL Server and upsert it in Elasticsearch index. We used JSON Generator Transform to help us convert database table rows into JSON documents. In addition, Web API Destination was very helpful in automatically creating and making HTTP requests to Elasticsearch instance and indexing our data as a result. This tutorial was quite simple and straightforward, yet it is a good starter to use other Elasticsearch APIs and their features. From now on Elasticsearch reference is your best friend in learning what APIs exist, what are their methods and how HTTP requests should look like.

Keywords

How to import data from SQL Server to Elasticsearch | How to load data into Elasticsearch | Using Elasticsearch to index SQL Server | Elasticsearch and SQL Server integration | How to use Elasticsearch together with SQL Server | Upsert SQL Server data into Elasticsearch

The post Loading data from SQL Server to Elasticsearch with SSIS appeared first on ZappySys Blog.

In our previous blog post we saw how to perform Read and Write operations in MongoDB using SSIS (i.e. Bulk Update, Delete, Upsert, Insert). In this post we specifically focus on how to update MongoDB Array items / elements using SSIS.

To make things simple to follow we have used JSON Source to produce sample JSON documents for MongoDB update (NOTE: Output as Document option is checked for JSON Source). However in real world you may have to use SSIS JSON Generator Transform to produce input documents for MongoDB. To learn more about generating JSON documents for MongoDB Load process check this blog post or watch video found on this page.

Video Tutorial – Example of insert/update documents inside MongoDB Array (Use $set, $push operator)

Basic Concepts: Performing MongoDB CRUD operations using SSIS

If you never heard term called CRUD stands for Create, Read, Update, Delete. These are the most common operations you have to do with any data source. SSIS PowerPack comes with many MongoDB Components to help you with CRUD operations. You can use one or more following components to achieve drag and drop MongoDB CRUD operations in SSIS. When you use SSIS PowerPack for MongoDB all operations will be Bulk Operations by default which means it will provide very high throughput.

• SSIS MongoDB Source
• SSIS MongoDB Destination
• SSIS MongoDB ExecuteSQL Task

Download Example SSIS Package

Here is the link to download Sample SSIS Package

SSIS Example : Update MongoDB Array Items / Elements (Update, Delete, Insert – Using $set, $pull, $push)

Update MongoDB Array Items ($set operator)

In next couple of sections we will see how to perform Update, Insert and Delete operations with MongoDB Array Items using various update operators and array operators. Consider the following Sample JSON documents for MongoDB. First we will see how to use MongoDB Shell commands to perform CRUD operations on MongoDB Array items.

{
    "_id" : ObjectId("5883ebe669e0f22ba890b49b"),
    "CustomerID" : "BOLID",
    "Name" : "Some name",
    "Orders" : [ 
        {
            "OrderID" : 200,
            "ShipCountry" : "USA",
            "City" : "Atlanta"
        }, 
        {
            "OrderID" : 201,
            "ShipCountry" : "USA",
            "City" : "New York"
        }
    ]
}

Update Array element using Shell command

Assume you want to update ShipCountry attribute of one of the Orders (OrderID=200) and for CustomerID=”BOLID” in above sample document.

Shell Command for Update Array Item

Use below shell command in your favorite MongoDB Client Tool (e.g. RoboMongo). Notice two things in below command because we will apply same concepts when we take this approach to SSIS. In below command Actual JSON document is wrapped inside. This is MongoDB Update operator. Click here to learn more about $set. Also notice that rather than nested Array syntax we used Orders.$.ShipCountry to update ShipCountry which is part of Array element. MongoDB provides special way to navigate array elements using   $ Operator (MongoDB Positional Update)

db.MyCollection.update(
	{ "CustomerID" : "BOLID", "Orders.OrderID" : 200 }, 
	{ $set : { "Orders.$.ShipCountry" : "Test1"} } , 
	{ upsert: false, multi: false}
)

After Update (MongoDB Document)

After you run above shell command notice all other attributes remain there in Array document and only ShipCountry is changed. This is because we used $ Operator (MongoDB Positional Update) rather than supplying Raw JSON for Array document (i.e. Orders : [ { … } ] ).

{
    "_id" : ObjectId("5883ebe669e0f22ba890b49b"),
    "CustomerID" : "BOLID",
    "Name" : "Some name",
    "Orders" : [ 
        {
            "OrderID" : 200,
            "ShipCountry" : "Test1",
            "City" : "Atlanta"
        }, 
        {
            "OrderID" : 201,
            "ShipCountry" : "USA",
            "City" : "New York"
        }
    ]
}

The issue with this approach its not Bulk operation. Assume that you have 100,000 records to update in various documents and its nested array items then row by row operation is not effective. To solve this issue we will use SSIS MongoDB Destination which performs Bulk operations (e.g. Bulk Update, Delete, Insert and Upsert).

Update MongoDB Array Items using SSIS

Now lets implement above scenario using SSIS. Assume that you have sample MongoDB document (as per above example – Before Update). We want to update ShipCountry for OrderID=200 for Customer BOLID. To do that perform following steps in SSIS. If you are updating Second level array (i.e. Root >> Orders >> OrderItems)  then you have follow slight different process. Check next section in this article about updating second level array (Insert items)

1. Download and Install SSIS PowerPack
2. Create new SSIS Project with one data flow task
   

   Dragging and dropping Data Flow Task into Control Flow

3. Drag ZS JSON Source and check Output as Document option (as below sceenshot). You can use JSON Source is your JSON is stored inside file or Variable or coming from Some API call which eventually go into MongoDB. You can also use JSON Source for quick testing where you Hardcode JSON to feed to MongoDB.
4. Enter the following JSON in the text area. Click OK to save.
   

   {"CustomerID":"BOLID", "Orders.$.OrderID":201, "Orders.$.ShipCountry": "Test1"}
   {"CustomerID":"BOLID", "Orders.$.OrderID":202, "Orders.$.ShipCountry": "Test2"}
   /**Below is extra record not found in destination - will be ignored in Update**/
   {"CustomerID":"BOLID", "Orders.$.OrderID":203, "Orders.$.ShipCountry": "Test3"}

5. Right click in Connections area and Create new ZS-MONGODB connection. Specify credentials and click Test before you hit OK to save.
6. Now drag ZS MongoDB Destination from SSIS Toolbox.
7. Select Runtime connection.
8. On the properties tab edit following properties
   1. Set ColumnsForLookup property as below (Notice how we use 2 columns for JOIN. Also used AS keywords for 2nd Column so we have <column-from-input-doc> AS <column-for-server-side-find>
      CustomerID,Orders.$.OrderID AS Orders.OrderID
   2. Set LoadOptions as below (Op is Update Operator you like to use. In our case its $set, Multi=True means if multiple match found then all matching documents will be updated). If you want to use multiple update operators then you have to supply that from Input document and use op=none option rather op=$set.
      op=$set;multi=true
   3. Set Operation property to Update
9. On the Mappings tab Map __DOCUMENT__ from upstream (In our case you will see same name column from JSON Source if you checked Output as Document option)
10. Click OK to save MongoDB Destination UI
11. Run package. You will notice after execution two orders of CustomerID=BOLID will be updated.

SSIS JSON Source – Read JSON (Raw Documents)

SSIS MongoDB Destination – Update MongoDB Array Item using $set update operator

Insert MongoDB Array Item ($push / $addToSet Operator)

Now lets look at how to insert new item into existing Array. Lets say we already have two orders for a customer and now we want to add new order. For this you can either use  $push operator or $addToSet operator. $push operator doesn’t check for duplicate item so its little faster. If you wish to skip append if item already exists (by matching all attributes) then use $addToSet operator.

Using Shell Command : Insert MongoDB Array Item (Append to array)

Here is the shell command which will push new item to the array

db.MyCollection.update(
	{CustomerID:"BOLID"},
	{$push:{Orders: {"OrderID":202,"ShipCountry":"USA","ShipCity":"Atlanta" } }},
	{upsert: false,multi: false}
)

Using Shell Command : Insert MongoDB Array Item at second level

Here is the shell command which will push new item to the array which is stored at second level e.g. { Orders :  [ { Items : [ insert-here ] } ] }

db.MyCollection.update(
	{CustomerID:"BOLID", "Orders.OrderID" : 202 },
	{$push:{ "Orders.$.Items" : {"ProductID":1001,"Quantity":3} } },
	{upsert: false,multi: false}
)

Using SSIS : Insert MongoDB Array Item at Second Level

Now lets look at some example how to Insert Items inside Orders array. Since it requires two JOIN columns to perform second level array insert you may need to add metadata for join.

First step is create documents for update. There are atleast 4 ways you can create input documents for MongoDB Destination.

1. You can use JSON generator Transform to create documents for load process
2. You can use Export JSON Task to generate input documents file and then use JSON Source to Read Documents (Check Output as Document option on JSON Source to read as RAW JSON)
3. You can use JSON Source to feed direct JSON (This approach is demonstrated in the below example sceenshot)
4. You can also use SQL query to output JSON Strings along with columns used for JOIN condition
   

   select CustomerID, OrderID as 'Orders.$.OrderID', DOC = '{Orders.$.Items : {ItemID: 1, Qty: 5} }'

To make it simple lets use #3 approach from above list. We will create few documents and insert New orders Items

Sample Document for MongoDB Collection

For testing purpose you can create MongoDB collection called test with following one document. As you see Items array is empty for both orders but once you run sample each will have 2 items.

{
    "_id" : ObjectId("58c01bee0610fea8ddda04b9"),
    "CustomerID" : "ALFKI",
    "Orders" : [ 
        {
            "OrderID" : 100,
            "Items" : [ ]
        }, 
        {
            "OrderID" : 101,
            "Items" : [ ]
        }
    ]
}

Now lets look at how to configure SSIS package

Step-1 : Create JSON Documents for Array Insert

Lets use JSON Source like below. Use following Sample JSON

/*Insert new items into order#100*/
{CustomerID:"ALFKI","Orders.$.OrderID" : 100, Doc: "{ \"Orders.$.Items\" : {ItemID: 1, Qty: 10} }" }
{CustomerID:"ALFKI","Orders.$.OrderID" : 100, Doc: "{ \"Orders.$.Items\" : {ItemID: 2, Qty: 11} }" }

/*Insert new items into order#101*/
{CustomerID:"ALFKI","Orders.$.OrderID" : 101, Doc: "{ \"Orders.$.Items\" : {ItemID: 1, Qty: 20} }" }
{CustomerID:"ALFKI","Orders.$.OrderID" : 101, Doc: "{ \"Orders.$.Items\" : {ItemID: 2, Qty: 21} }" }

Create Sample JSON Documents for MongoDB Array Update (2nd level array)

Step-2 : Configure MongoDB Destination for $push operation on Second level Array

Now lets connect source to Destination and configure MongoDB Destination. Select connection on first tab. Then On second tab change properties like below.

Configure MongoDB Destination for second level array update (Insert item using $push update operator)

Step-3 : Add missing JOIN column for Mapping

When you click on Mappings tab you will notice only root level columns are listed there for mapping but if you are JOINing based on Nested column found inside array then you have to add missing mapping column by hand like below screenshot.  In our case we Join by CustomerID and OrderID where OrderID is found in Orders array which is missing on mappings tab. So lets add it. Just make sure name new column using $ positional operator… e.g. Orders.$.OrderID  this will tell MongoDB that our OrderID column must be lookup under Array called Orders: [  ]

MongoDB Destination – Add missing metadata column for JOIN

Step-4 : Map JOIN columns and Input DOCUMENT

Now finally map all columns

SSIS MongoDB Destination – Column Mappings

Using SSIS : Insert MongoDB Array Item (Append to array)

Now lets look at how to perform same operation to insert array item in SSIS using $push operator. In this case we will do little different than Update Item example ($set operator). First we have not included CustomerID as part of Input columns and we mapped it on Mapping Screen rather than supplying columns names in ColumnsForLookup property (On Component Properties Tab of Destination UI). If you are using JSON Generator Transform to generate your JSON then you have option to Hide/Include certain columns part of your flow. Check Include column to downstream option if you want to use for JOIN column.

Set following settings

• Set LoadOptions as below
  op=$push;multi=true
• Set Operation=Update
• Leave ColumnsForLookup as empty
• On the mappings tab map __DOCUMENT__ and CustomerID in the Target. If you don’t see CustomerID column in the mapping create atleast one sample doc in MongoDB Collection so metadata is detected.

Download Sample for more detail.

SSIS JSON Source – generate JSON for MongoDB Array Insert

SSIS MongoDB Destination – Insert MongoDB Array Item (Use $push, or $addToSet operator)

Delete MongoDB Array Item ($pull operator)

If you wish to remove array item then you can use $pull operator. Only difference is rather than specifying two conditions in first argument… you just specify JOIN columns without array column (i.e. OrderID). To specify Array item condition you have to set inside $pull : { … }

Using Shell Command for MongoDB Delete Array Item (Remove by condition)

Below example shows how to remove OrderID=201 from Orders array for CustomerID=BOLID.

db.MyCollection.update(
	{CustomerID:"BOLID"},
	{$pull:{Orders: {"OrderID":201} }},
	{upsert: false,multi: false}
)

Using SSIS for MongoDB Delete Array Item

Now lets look at how to implement Delete logic for MongoDB Array Item in SSIS. As you see in below screenshots You have to modify following things

• Set ColumnsForLookup property to CustomerID (no need to add OrderID here because its part of input document)
• Set LoadOptions property to op=none;multi=false. op=none means we are supplying Update operator part of input document.

Assume you have doc like below in MongoDB  and you like to remove Order#101 from array

{
    "_id" : ObjectId("58c01bee0610fea8ddda04b9"),
    "ParentID" : 1,
    "Orders" : [ 
        100, 
        101, 
        102
    ]
}

For this supply 2 columns from upstream
col1  = ParentID
col2  = DOCUMENT

Document column should look like below
{Orders: 101}

1. On MongoDB destination change LoadOptions property to  op=$pull;multi=true
2. Change Operation to Update
3. Map ParentID and DOCUMENT columns on Mapping screen
4. Run the package … It will remove Order# 101 from array

Array Upsert – Update / Insert using Custom JOIN condition (such as $ne )

If you like to perform Upsert Operation on MongoDB Array Items then refer to below blog post.

Update or Insert – Upsert MongoDB Array Items using SSIS

Conclusion

MongoDB integration can be challenging if you are new to NoSQL world. If you are using SSIS as your primary ETL tool then not to worry because SSIS PowerPack can give super power needed to complete your project on time with drag and drop high performance connectors.

Keywords: mongodb update array item | mongodb update array elements | mongodb update array documents | MongoDB $ operator | MongoDB update sub document items | MongoDB CRUD operations | MongoDB Bulk Updates | MongoDB bulk updates to array items

The post Update MongoDB Array Items using SSIS appeared first on ZappySys Blog.

In our previous article, we discussed how to load data into MongoDB (Insert, Update, or Upsert Mode) with various options. In this post, we will focus on creating a MongoDB Document for the load process. We will see how to produce BSON (MongoDB-specific JSON that may include functions such as ObjectID and ISODate).

This article assumes you have installed SSIS PowerPack and watched the introduction videos of SSIS JSON Generator Transform and MongoDB Destination Connector

Create MongoDB document Example (Produce JSON, BSON for Load)

Here is the screenshot, which shows how to fetch SQL Server data > create a MongoDB document in JSON / BSON format, and load it into a MongoDB collection

The high-level steps we performed are

1. Used OLEDB Source to fetch data from SQL Server. Used a query like below
   

   select CustomerID, 'ISODate("' + convert(varchar(30),getdate(),126) + '")' as DOB from Customers

   In the above query, you will notice that we are building some BSON fragments, which will be treated as raw values in the JSON generator. Also, we are using the convert function from SQL Server to produce an ISO date format (Style=126 will produce an ISO date)
2. Used the SSIS JSON Generator Transform to produce a JSON document for each customer. Define a custom layout using the drag-and-drop interface. You can control how many documents you want to output using settings on Transform (Single document for all input rows or one document per row in the Main dataset)
3. Load JSON documents coming from upstream into MongoDB using the SSIS MongoDB Destination Connector. We changed Action=Upsert and set ColumnsForLookup to CustomerID (Consider this as Primary Key or your Join Column to find existing document)
4. On Target, we mapped __DOCUMENT__ rather than individual columns, so that we can load the full document.
   

   Sample SSIS Package – Create MongoDB Document in BSON format (JSON with functions). Update or Insert Mode

Configure JSON Generator Transform – Create JSON Document

If you want to learn how to configure the SSIS JSON Generator Transform, then watch the video here. Important thing to remember in the configuration is to check the Treat value of this column as Raw JSON

Create MongoDB Document in JSON or BSON format – Load into Collection (Upsert – Update or Insert)

Configure MongoDB Destination – For Upsert (Update or Insert Mode)

By default, the SSIS MongoDB Destination Connector performs an insert, but if you want to perform an Upsert (i.e., insert if not found, else update), you have to change a few settings.

1. Change Operation to UpdateOrInsert
2. Set ColumnsForLookup (e.g., in our case, it was just one CustomerID). If you have multiple columns for join, then use a comma-separated list (e.g., City, State, Zip)
3. If you want to affect various documents if more than one match is found for the join criteria, then use multi=true as shown in the screenshot below (see LoadOptions )
   

   Configure the SSIS MongoDB destination for the Upsert operation

Conclusion

In this post, you have seen how easy it is to create MongoDB Documents in JSON / BSON format and load them into a MongoDB collection using ZappySys SSIS PowerPack. This innovative drag-and-drop approach can simplify the entire ETL process if you are in a time crunch and don’t know scripting languages.

The post How to create MongoDB documents (JSON, BSON) for loading in SSIS appeared first on ZappySys Blog.

MongoDB is one of the most poplar NoSQL database out there. In this article you will learn how to use ZappySys MongoDB Destination Connector to perform various bulk operations on MongoDB such as Insert, Update, Delete and Upsert. If you wish to learn how to Update MongoDB Array items then check this blog post instread.

MongoDB data read/write Example SSIS Package

Before we talk more about loading data into MongoDB lets look at screenshot and steps involved in our sample package for this tutorial. Below sample SSIS package uses 4 Custom SSIS Tasks

1. SSIS MongoDB Source Connector (Used to read data from MongoDB Collection)
2. SSIS MongoDB Destination Connector (Used to write data into MongoDB Collection also you can do other bulk operations such as MongoDB update, MongoDB upsert and MongoDB delete)
3. SSIS MongoDB ExecuteSQL Task (Used to call any MongoDB Shell commands including MongoDB DDL or DML type operations such as CREATE/DROP, Insert, Update, Delete, Read, you can also call server side JavaScript)
4. Dummy Data Source (Used to generate sample JSON Documents which can be loaded in MongoDB)

MongoDB Examples

Here is the list of various MongoDB examples included in attached sample package (see at the end for download links)

• MongoDB example – How to read data from MongoDB collection using SQL query language
• MongoDB example – How to load raw JSON document into MongoDB Collection (Bulk Insert with Reload)
• MongoDB example – How to do MongoDB Upsert (Bulk Update or Insert) for specific fields or entire document
• MongoDB example – How to perform MongoDB Delete ( Bulk Delete based on Matching Rows)
• MongoDB example – How to create new MongoDB collection (Only if not exist – Use safe option)
• MongoDB example – How to fetch MongoDB collection record count and save to SSIS variable
• MongoDB example – How to fetch MongoDB collection size and save to SSIS variable
• MongoDB example – How to get collection names and loop through them using ForEach Loop Task
• MongoDB example – How to save query output into Recordset variable and loop though records using ForEach Loop Task

Video Tutorial – Update/Delete/Write/Bulk Load data into MongoDB from SQL Server/ORACLE

In below video tutorial you will see how easy it is to load data into MongoDB from any Datasource such as SQL Server, Oracle, MySQL, Flatfile etc. You will also learn how to product nested JSON from multiple tables using JSON Generator Transform. You can also use JSON Source with Output as Document Mode to load raw JSON data.

Video Tutorial – Read data from MongoDB

In the following video tutorial you will learn how to consume data from MongoDB Collection using SSIS MongoDB Source Connector. You will notice that it uses SQL query syntax rather than MongoDB specific native query syntax (JSON query). This makes it super easy to query data without any learning curve. Data stored in MongoDB is in JSON document format but Data coming out from MongoDB Source Connector is Flat table structure (It de-normalize nested nodes). You can also Query inner Hierarchy using JSON Path expression (e.g. query Orders from Customer Document using $.Customer.Orders[*] ) .

SSIS Example – Loading data into MongoDB, Read from MongoDB, Upsert, Upsert, Delete Insert JSON Documents, Execute Shell Commands

MongoDB Insert – Bulk Loading data into MongoDB

Using MongoDB Destination Connector you can easily Insert JSON documents in Bulk. There are two input modes for inserting records into MongoDB.

Simple Mode – Loading data in simple mode (array not allowed)

In simple loading mode you can map source columns to target column in MongoDB destination. If you Insert data using this mode then you cant load complex documents with array.

Document Mode – Loading JSON documents into MongoDB

In this mode you can insert/update/delete documents by mapping inbuilt __DOCUMENT__ input column which appears on mapping tab (Target columns). When you perform Insert you have two options (Insert (i.e. Append) and Reload (Truncate + Insert). When you select Operation=Reload in that case Collection is first Truncated and then new records are loaded.

In JSON document load mode if you don’t supply _id column part of your input JSON then new _id is automatically generated by MongoDB for newly inserted document.

Loading JSON Documents into MongoDB Collection (Map Raw Document Column)

Loading JSON files into MongoDB

Another scenario could be loading JSON files into MongoDB. You can load JSON Files into MongoDB two ways.

• Use Flat File Source with just one column with DT_NTEXT datatype (use this method when you each JSON documents in one line and they are separated by new line as below screenshot)
• Use JSON Source Connector with Output as Document option checked. You can map __DOCUMENT__ source column to Target MongoDB collection as below. Advantage of this method is you extract JSON from array too and specify many other advanced options.
  

  SSIS Extract JSON Documents or Sub Documents from JSON file or JSON Array

Specifying LoadOptions

MongoDB Destination Connector supports following LoadOptions property (see Component Properties Tab on UI) which controls how target data gets modified. LoadOptions property is ignored for Reload and Insert operations.

• op : this option specify update operators (Valid for Update, Replace and Upsert operation (i.e. UpdateOrInsert) ).  See this link for more information on various Update operators available in MongoDB. Most common operators listed below. Refer official MongoDB help for more information.
  • op=none : Replace entire document
  • op=$set  : Update only those fields specified in mapping. If Specified mapping field not available in target then its added to target document.
  • op=$push : Adds new item to destination array. See following examples (How to insert single or multiple documents into MongoDB Array by supplying JSON into  __DOCUMENT__ column
    

    /* Insert single item into array - Construct input JSON as below */
    {YourArrayField : 111 }
    
    /* Insert multiple items into array - Construct document as below */
    {YourArrayField : { $each: [ 111, 222, 333 ] } }
    
    /* Insert multiple documents into array */
    {YourArrayField : { $each: [ {OrderID:1, Total:20.00}, {OrderID:2, Total:12.00} ] } }

• multi : This option controls if you want to Update or Delete multiple records for matching condition for each upstream input record.  For example if you joining by Country and City columns for update and target system finds 5 records then by default they all will be updated if multi=true is set in LoadOptions property. If you don’t want to update multiple records

MongoDB Update

MongoDB Destination Connector supports Batch Update and Batch Upsert Operations. There are few things to remember how Update operation works with MongoDB Destination Connector.

• To perform Update operation you have to specify Join Criteria for record lookup on target (see ColumnsForLookup property).
• Specify JOIN columns in ColumnsForLookup property. Use comma to specify multiple columns (e.g. CustomerID,OrderID). If you dont specify columns in this property then any columns mapped other than __DOCUMENT__ will be considered as JOIN column.
• With MongoDB destination you can only specify simple Update condition by mapping JOIN input columns. Behind the scene it will construct search query for update. For example if you Map 3 input columns (__DOCUMENT__ , Country, State) and you specify LoadOptions : op=$set;multi=true then its similar as below SQL Statement.
  

  UPDATE MyCollection
  SET [properties specified in __DOCUMENT__]
  WHERE Country=[input value] AND State=[input value]

• Join columns can be individually mapped or can be supplied as part of JSON document mapped to __DOCUMENT__ column.
• If __DOCUMENT__ column is not mapped then you must specify at least one column name in ColumnsForLookup property to identify search column.
• If __DOCUMENT__ column is mapped and column name is not specified in ColumnsForLookup property then you must map at least one more input column which can treated as JOIN column

Update specific fields  ($set operator)

To update specific fields in target MongoDB collection you can map JOIN column(s) and then map fields you want to update or map __DOCUMENT__ if you suppling JSON document with fields you want to update.

See below screenshot where its updating Customers record with new coordinates  based on City and Country information.

SSIS MongoDB Update Bulk – $set update operator – Update specific fields – Update single or multiple documents

Update single or multiple target records for single input row

If you want to update all matching records in target then set LoadOptions property to op=$set,multi=true
If you want to update only one record per matching criteria in target then set LoadOptions property to op=$set,multi=false

Update entire document (i.e. Overwrite/Replace documents)

If you like to replace entire target document then you can do two ways. Few things you have to change for MongoDB Destination.

Approach#1 

1. Select Operation=Replace (Rather than Update or Upsert)
2. Specify Join column in ColumnsForLookup
3. On the mapping tab Map __DOCUMENT__ (Assuming you have Raw JSON coming from Upstream)

Approach#2

Another approach is use $set operator in LoadOptions property then it will only update specified fields in mappings. But if you want to overwrite entire document (Replace document) then set LoadOptions property to op=none,multi=true or op=none,multi=false (If you want to update one record for matching condition)

Insert single item into MongoDB Array using $push operator

If you want to insert new items into array then set LoadOptions property to op=$push,multi=true or op=$push,multi=false (If you want to update one record for matching condition). You have to map __DOCUMENT__ column on target mappings tab to supply item(s) or document(s) you want to insert into target array.

/* Insert single number or string into MongoDB array - Construct input JSON as below */
{YourArrayField : 111 }

/* Insert single document into MongoDB array - Construct input JSON as below */
{YourArrayField : {OrderID:1, Total:20.00} }

Insert multiple items into MongoDB Array using $push operator along with $each

Inserting multiple items into array requires use of $each modifier as below. Use above instructions except input document (e.g. data passed to __DOCUMENT__ column) will look like below (see how $each used).

/* Insert multiple items into MongoDB array - Construct document as below */
{YourArrayField : { $each: [ 111, 222, 333 ] } }

/* Insert multiple documents into MongoDB array */
{YourArrayField : { $each: [ {OrderID:1, Total:20.00}, {OrderID:2, Total:12.00} ] } }

Other MongoDB Update Operators

So far we have seen $set and $push update operators. To learn more about other operators see this link

MongoDB Update Array Items ($pull, $push, $addToSet)

If you wish to update items found inside nested array of document then Check this full length article

MongoDB Upsert – Bulk Update or Insert JSON documents

If you wish to perform Bulk Upsert using MongoDB Destination connector then select Upsert Action from dropdown (Action=UpdateOrInsert). In Upset if document is not found for matching criteria then new document is inserted in MongoDB Collection. If document is found then Update operation occurs. To set specific fields use op=$set in LoadOptions   and if you wish to overwrite entire document then use $op=none in LoadOptions

SSIS MongoDB Upsert – Bulk Update or Insert JSON documents

MongoDB Delete – Bulk delete MongoDB JSON documents

To delete documents from collection you can set operation=Delete on Component properties page. For delete you can simply map _id column on target or just supply columns based on you want to search records you want to delete.

To delete maximum one record for each input row set LoadOptions property to op=none,multi=false or set op=none,multi=false if you want to delete more than one record for matching condition.

MongoDB Collection Reload – Remove All (Truncate collection)

To reload collection set operation to Reload on Component properties page. Reload operation first clears target collection and perform bulk Insert operation

Generate JSON for MongoDB data load

Now lets talk about different ways to produce simple or complex JSON using ZappySys components.  There are 3 ways you can produce JSON which can be loaded into MongoDB Collection. Following section will describe each approach and you can select which one best suited for you.

Generate JSON – Using Export JSON Task

Export JSON Task can export JSON to File or variable which you can consume via JSON Source Connector (Check Output as Document option) and supply to MongoDB Destination Connector (map to __DOCUMEN__).

Check this link for more info :  Export Json from sql server using ssis

Generate JSON – Using JSON Generator Transform

JSON generator transform is useful when you want to Produce JSON inside pipeline from single or multiple input.

Check this link for more info :  JSON generator Transform (Watch Video on this page)

JSON Generator Transform – Generate JSON documents and load into MongoDB

Generate JSON – Using Dummy Data Source (Fake JSON for Testing)

If you need to produce JSON for testing purpose then use Dummy Data Source (Select JsonDocuments Template from dropdown and click ok)

Query/Read data from MongoDB

Visit following links to learn more about reading data from MongoDB.

SSIS MongoDB Source Homepage
MongoDB SQL Query Reference
MongoDB data extract – from nested sub documents and array

Execute MongoDB Shell commands in SSIS (e.g. Create/Drop collection)

MongoDB SOurce and Destination Connectors are great for read/write data. But what if you want to issue ad-hoc MongoDB shell command such as create collection/drop collection ? No worry.. SSIS PowerPack v2.0 released new Task called SSIS MongoDB ExecuteSQL Task This task comes with many examples listed on the UI itself (Look for Examples dropdown menu on UI). You can use SSIS MongoDB ExecuteSQL Task to perform many operations such as

• Create/Drop MongoDB collections
• Get collection list as ADO.net recordset so you can loop through using ForEachLoop task
• Call serverside JavaScript
• Fetch data as ADO.net recordset and store in SSIS variable so you can loop using ForEachLoop task
• Drop database
• Call many shell commands

Download MongoDB Example Files used in this tutorial

To test this package you first need to download SSIS PowerPack
Once SSIS PowerPack installed you can download following SSIS sample project file(s).

Download sample files – SSIS 2008
Download sample files – SSIS 2012
Download sample files – SSIS 2014
Download sample files – SSIS 2016

Conclusion

MongoDB makes it super easy to store JSON documents and gives you whole new way of manipulating documents with ease of use. ZappySys SSIS Connectors and Tasks for MongoDB makes it possible to implement various data integration scenarios with MongoDB. Click here to Download SSIS PowerPack and try it your self.

Keywords:
Loading data into MongoDB | mongodb upsert array | mongodb upsert example | mongodb insert or update | mongo insert or update | mongo upsert | mongo insert JSON | mongo upsert JSON | update data into MongoDB

The post SSIS – Loading data into MongoDB (Upsert, Delete, Update) appeared first on ZappySys Blog.