In this post you will learn how to access Microsoft Graph API  (Office 365 REST API / Sharepoint API) inside SSIS without any coding. Microsoft Graph API is a unified way to access many Microsoft services API including Office 365 API. In this post we will use SSIS JSON Source / REST API Connector to load data from Office 365 to SQL Server.

Use Case of Microsoft Graph API

Here are some use cases why you want to use Microsoft Graph API

• Read/Write events from Outlook Calendar
• Send email  / Read emails
• Get list of files from OneDrive
• Upload/ Download files to OneDrive
• Read/Write Excel Sheet (Use range or specific range of specific sheet)
• Read Sharepoint Document library
• Search content from Sharepoint document library
• Get AD users for your Organization (AD Accounts)
• Update OneNote
• Track changes to Users, Events, Calendar Items using delta API

Step-By-Step : Access Outlook.com Emails using SSIS

Now lets look at examples to access your outlook.com emails using SSIS. You can use same techniques to call pretty much any Office 365 API.

Configure SSIS OAuth Connection for Graph REST API

First step to access any Graph API is configure OAuth connection

1. Open Visual Studio
2. Open existing SSIS Project or create new using File > New > Project > Choose “Integration Services Project” Type under Business Intelligence template category
3. Open Package
4. Right click inside Connection Managers area and click “New Connection…”
5. From the connection type list select “ZS-OAUTH” connection type.
   

   Create new SSIS OAuth API Connection Manager

6. On the OAuth Connection Manager configure following options
   1. Select “Custom” from Provider dropdown
   2. Select OAuth2 from OAuth version
   3. Enter your Application Id and Secret Key (i.e. App Password) obtained in the previous section from here.
   4. Enter following URL in the Authorization URL
      

      https://login.microsoftonline.com/common/oauth2/v2.0/authorize

   5. Enter the following URL in the token URL field
      

      https://login.microsoftonline.com/common/oauth2/v2.0/token

   6. Enter the following Graph API Scopes (each scope must be entered in a new line).
      

      user.read
      mail.read
      offline_access

      NOTE: Always include offline_access scope returns refreh_token which allows you to renew token without going through login process again. For more information about which scopes needed always refer to API help page (e.g. if you want to read mail using this API then you will see it says you can use Mail.Read or Mail.ReadWrite scope).
   7. Go to Advanced tab and enter the following URL in the Callback/Return URL (Assuming exact same URL used when you registered App in the previous section)
      

      https://zappysys.com/oauth

   8. Now go back to first tab and Click Generate Token. When prompted Login using your Personal Microsoft Account or Work Account (Office 365 or AD login)
   9. After login you will see Accept option just click it. If things go right then you will see Access Token and Refresh Token fields will be populated.
7. Once all configured you can click Test to make sure connection works.
   

   SSIS OAuth Connection Manager for Office 365 REST API / Microsoft Graph API

Configure SSIS JSON/REST API Source

Once you configure OAuth connection you are ready to read data from Office 365 API. Lets look at step by step how to read emails using Mail API and save to SQL Server.

1. Drag and drop Data flow task from SSIS toolbox
2. Goto data flow designer and drag and drop ZS JSON Source/REST API Connector on the designer
3. Double click to edit JSON Source.
4. In the URL field enter below URL
   

   https://graph.microsoft.com/v1.0/me/messages

   To customize URL with additional parameters check this help link. It’s a standard OData Protocol so you can leverage many common OData features (e.g. Use $top parameters to set pageSize. Default is 10 for List Mail but you can include more records per response by changing it such as …/messages?$top=50).
5. Check Use credentials option
6. From the connections dropdown select OAuth Connection manager created in the previous section
7. Change Data Format option to OData 
8. In the Filter enter following or click “Select Filter” and select Value Array node > Click OK.
   

   $.value[*]

9. In the Max rows set to 200 so we don’t pull all data for now 🙂 … This is our hello world example.
10. Now click Preview to see data. If all is well then Save UI by clicking OK
    

    SSIS JSON Source – Get data from Microsoft Graph API (Office 365 API – Read Mail Example)

Loading Graph API / Office 365 data to SQL Server

Now last step is loading Outlook data to SQL Server. For example purpose we will load data into SQL Server but you can load into any Target (e.g. Flat file, Oracle, Excel) using Microsoft or ZappySys Destination connectors

To load Office 365 data into SQL Server perform the following steps

1. Drag OLEDB destination on Data Flow surface
2. Connect ZS JSON Source to Destination
3. On OLEDB destination select / create new SQL Connection and then Click “New Table”
4. Click on Mappings tab and click OK to save
5. Execute Package

Configure SSIS OLEDB Destination – Loading Xero Data into SQL Server Table

Graph API / Office 365 to SQL Server Column Mappings for OLEDB Destination

Loading Graph REST API  / Office 365 data to SQL Server in SSIS

POST data to Graph API (Insert or Update)

So far we have seen how to read data from Office 365 API. Now let’s look at how to write data to Office 365.

There are two ways you can achieve this.

1. Use Web API Destination in Data Flow
2. Use REST API Task in Control Flow

Read this article to learn more about API POST using SSIS

Using SSIS Web API Task (Write data from SQL Server to Office 365 / Graph API)

Using REST API Task to POST data to Graph API

If you have JSON/XML data already prepared and if you like to POST it to Graph API then REST API Task would be easy to use.

REST API Task Body can be direct string, variable (e.g. {{User::varSomeData}} or Body can come from File 

SSIS REST API Task – POST data to Graph API

Download file from OneDrive

To download file from OneDrive perform these steps

https://zappysys.zendesk.com/hc/en-us/articles/115005010573-How-to-download-file-from-OneDrive-using-Graph-API

Upload file to OneDrive

https://zappysys.zendesk.com/hc/en-us/articles/115004893713-How-to-upload-file-to-OneDrive-using-Office-365-Graph-API

Reading Office Excel Sheets from OneDrive

JSON Source also support reading API response which returns 2D arrays. Such as Office Excel Spreadsheet API to read excel data. Check this blog post to learn similar scenario to parse 2D arrays. There are few other options for parsing 2D arrays which are documented here.

https://graph.microsoft.com/v1.0/me/drive/items/01CYZLFJDYxxxxxxx/workbook/worksheets('test')/range(address='A1:B200')

Testing Office 365 API using Graph API Explorer

If you like to test API request/response inside browser then Microsoft provides really great tool to called Graph API Explorer. Click here to try Graph API Explorer.

See below screenshot

Common Errors

Deployment to Production

Conclusion. What’s next?

In this article we have learned how to load data from Microsoft Graph API to SQL Server using SSIS ( drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from Outlook Mail using OAuth. JSON Source makes it super simple to parsing complex / large JSON Files or any Web API Response into rows and column so you can load into database like SQL Server. Download SSIS PowerPack to try many other automation scenarios not discussed in this article.

Keywords:

Office 365 API Integration with SQL Server | How to extract Office 365 data in SSIS? | How to read outlook mail, calendar event and contacts? | Calling Microsoft Graph API using SSIS. | Office 365 to SQL Server | SQL Server to Office 365 | SSIS Office 365 API Integration | SSIS Graph API Integration

The post How to use Office 365 API with SSIS (Mail, Calendar, Contacts, OneDrive, Excel) appeared first on ZappySys Blog.

Google BigQuery is a fully managed Big Data platform to run queries against large scale data. In this article you will learn how to integrate Google BigQuery data into Microsoft SQL Server using SSIS. We will leverage highly flexible JSON based REST API Connector and OAuth Connection to import / export data from Google BigQuery API just in a few clicks.

If you are looking for a similar product inside Amazon AWS Cloud then check an article about Amazon Athena.

Prerequisites

Before we look into a Step-By-Step section to extract and load data from BigQuery to SQL Server let’s make sure you meet the following requirements:

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have basic familiarity with REST API concepts and Google BigQuery API. This post uses free Public Dataset to query BigQuery table so no billing is required to get your hands dirty 🙂 However, if you like to query your own dataset then make sure you have at least one BigQuery dataset and one table with some sample data created (this part does require billing enabled). Read a Google Quickstart article for more information on how to create a new BigQuery dataset and a table.

Understanding Google BigQuery Object Heirarchy

Google BigQuery has 3 main concepts below.

• Project
  • Dataset
    • Table
      • Query requests (each query creates a unique JobID – valid for 24 hours from which you can read data)

So in order to create BigQuery table you always need Project and then Dataset under that project. You can group objects around Project / Datasets. When you query you can supply fully qualified name of your table in FROM clause (e.g.

select count(*) from `bigquery-public-data.usa_names.usa_1910_2013`

So let’s get started.

Read data from Google BigQuery using SSIS

Basically you can query Google BigQuery data in two ways: In this article we will not cover 2nd method.

• Method-1: Query data using jobs/query method in BigQuery API. Use it if you expect to get a result in a fairly short amount of time. This API method generates a temp table which gets deleted after 24 hours. You can read data within that time frame using newly created JobId reference.
  
  NOTE: If you have records that sum up to more than 10MB of data or ~10,000 rows (assuming 1 row uses 1KB of data) then you need to proceed with two-step process as explained below.
• Method-2: Export SQL query result using jobs/insert method in BigQuery API. Use this method if you expect a query to take a long time to finish.

Now let’s see how to query Google BigQuery data using SSIS 🙂

Create Google API Project and Register OAuth App

To use any Google API firstly you have to finish two tasks:

1. Create Google API Project and obtain projectId (see next section).
2. Enable BigQuery API
3. Register your own Google OAuth App and obtain ClientId and Client Secret.

Check step-by-step instructions on how to create Google API Project and create OAuth App for Google. During instructions make sure you enable BigQuery API (screenshots may differ).

Get Google API Project ID

Once you have Google API Project created you can grab Project ID (see screenshot below). You will need Project ID in the next step when building API URL.

How to find Google API Project ID?

Configure OAuth Connection for Google BigQuery API

In order to call most of Google APIs you will need an OAuth App. If you want to use your own app then refer to how to register Google OAuth Application (Get ClientID and ClientSecret) article. For simplicity we will use Default OAuth App:

1. Right click inside Connection Managers area and click “New Connection…”
2. From the connection type list select “ZS-OAUTH” connection type.
   

   Create new SSIS OAuth API Connection Manager

3. For OAuth Provider select “Google”.
4. For OAuth App type options leave “Use Default OAuth App” selected. If you created custom OAuth App then you would need to select “Use Custom OAuth App” option.
5. For Scopes enter or select the following URLs (URL is just a permission name). Each URL must be in a separate line. For demo purposes we use only the first 3 scopes but we included a few more in case you like to test API in depth with a different permission set:
   

   https://www.googleapis.com/auth/bigquery
   https://www.googleapis.com/auth/cloud-platform
   https://www.googleapis.com/auth/cloud-platform.read-only
   https://www.googleapis.com/auth/bigquery.insertdata
   https://www.googleapis.com/auth/devstorage.full_control
   https://www.googleapis.com/auth/devstorage.read_only
   https://www.googleapis.com/auth/devstorage.read_write

6. Click Generate Token. Login using correct account if needed and then you will be prompted to click “Approve” OAuth App.
7. Once you click OK on the login screen you will see new tokens populated on the connection screen.
8. Click Test to make sure connection / tokens are valid and then click OK to save the connection.
   NOTE: Below screenshot uses Default App but we recommend you to to use Custom OAuth App (Your own Clientid / secret – Obtained in previous section)
   

   Create SSIS OAuth API Connection for Google BigQuery API

Start BigQuery Job and get JobId (Submit Query)

Once you have SSIS OAuth connection created for BigQuery API it’s time to read data from BigQuery. So, basically, there are two ways you can read BigQuery data: using query or insert method. For demo purposes we will use jobs/query method. If you want fire complex queries which can run for many minutes then refer to jobs/insert method.

If you are expecting more than 10MB of data (~10K records) then you have to split data extract in two steps:

1. Step-1 : Call jobs/query method using REST API Task to submit SQL query. This steps returns jobComplete: true if supplied query execution is done within timeoutMs parameter you included. By default its 10 seconds so if your query is going to take longer than 10 seconds then its good idea to add bigger timeout that way you dont need step#2. Once API call is done you will Get JobId in the response. Save that JobId in a variable for later use. This job result is valid for 24-hours only.
2. Step-2 (Optional) – You can add another optional REST API Task after previous step to Wait until Job is completed. For simplicity, this article will Omit setting up Status Check but its very simple….Use below settings
   1. Drag new REST API Task from toolbox. Connect it to Step-1 and double click to configure as below.
   2. Select URL From Connection mode,
   3. Select OAuth connection
   4. Enter URL as https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/jobs/{{User::JobId}} , Method: GET
   5. On Response Settings Tab, Select Response Content Type JSON and for content filter enter $.status.state
   6. On Status Check Tab, check Enable Status Check Loop and enter DONE in SuccessValue field
   7. Doing this setup will make sure we do not query data until Job Status is DONE (System keeps checking every 5 seconds)
3. Step-2 : Read Job result using JSON Source in a Data Flow Task.

Now lets see how to configure REST API Task to submit query and extract JobId (which will be used in next section).

1. In the Control Flow designer drag and drop ZS REST API Task from the SSIS toolbox.
2. Double click the task to configure it.
3. Change Request URL Access Mode to [Url from connection].
4. From the connection dropdown select OAuth connection created in the previous section.
5. Enter URL as below (replace YOUR-PROJECT-ID with a valid Project ID obtained in the previous section):
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-PROJECT-ID/queries

6. Select Method as POST.
7. Enter Body (Request Data) as below (change your query if needed; we used Public dataset for the demo): If you do not specify timeout then default is 10 seconds only. Also in this call we only care about JobID from response so we just added maxResults=10 because in 2nd step we will get all rows by doing pagination.
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000"
   }

   To use a Standard SQL query instead of Legacy SQL, add useLegacySql property and set it to false:
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000",
      "useLegacySql": false
   }

   For all possible parameters refer to jobs/query method documentation.
   NOTE: You can also supply location Parameter in the Body JSON to indicate where job should run. For non EU / US datacenters we suggest you to supply this parameter. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location.
   Example Use Of Location
   

   { "location":"us-east1",  "maxResults": 10, "query": "SELECT title FROM [bigquery-public-data:samples.wikipedia] LIMIT 10" }

8. Select Body Content Type as application/json.
9. Click Test Request/Response. If things go well you will see JSON content. Then just copy JobId from the response and save for later use:
   
10. Now go to Response Settings tab.
11. Select ContentType=Json.
12. Enter below expression to extract JobId from the response. You will need this Id to read SQL query output (this JobId is valid for 24 hrs):
    

    $.jobReference.jobId

13. Choose save response content to save to a variable. Select <New Variable…> option. When prompted give a name to your variable (i.e. vJobId) and for the value field paste JobId copied from the above step:
    
    
    NOTE: This JobId expires after 24-hrs so while you are designing a package and not running it then design time value is used for testing/previewing. So make sure your JobId is valid. If needed click Test Request/Response to grab a new JobId and update the variable with a new value so you can preview data without running a full package.
14. Click OK to save the UI.

Read BigQuery SQL result (for specified JobID)

Now let’s look at how to read data from BigQuery

Configure Google BigQuery Web Request (URL, Method, ContentType, Body etc.)

1. In the Control Flow designer drag and drop Data Flow Task from SSIS toolbox.
   
2. Double click Data Flow Task and drag and drop ZS JSON Source (For API/File) from SSIS toolbox.
   
3. Double click JSON Source to edit and configure as below
4. Enter URL as below. Replace YOUR-API-PROJECT-ID with the API Project ID obtained in the previous section.
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?maxResults=10000

   –OR– (Use below if your Job ran outside US / EU data center)
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?location=YOUR_REGION_ID&maxResults=10000

   NOTE: location Parameter indicates the geographic location of the job. This is Required parameter except for US and EU. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location. You can supply same location parameter for first step when you submit the query

   Failure to supply location parameter for non US /EU users may result in 404 NotFound Error.
   

5. Check Use Credentials option and select OAuth Connection Manager created in the previous section.
6. For HTTP Request Method select GET.
   

Read data from Google BigQuery from Temp Job result

Configure Filter

We need to transform a single JSON response into multiple rows so we need to apply a correct filter.

1. On JSON Source go to Filter Options tab.
2. In the Filter field enter $.rows[*] or click [Select Filter] button to browse hierarchy you want to extract.

Select Filter for JSON Response

Configure BigQuery API Pagination Settings

Most of modern APIs usually implement some sort of pagination technique so you get a part of data in each request rather than all in one go. Thus if you want more you can paginate through pages until the last page is reached. You can also read Understanding REST API Pagination in SSIS article to learn more about pagination in SSIS.

BigQuery API returns pageToken attribute in response JSON if more data is found for requested query result. You can then pass pageToken in the next URL in this format: http://my-api-url/?pageToken=xxxxxxxxxx

Now lets configure JSON Source to automate this pagination for us. On JSON Source go to Pagination Tab and enter the following two settings:

1. Set Next Link as $.pageToken.
2. Set Suffix for Next URL as &pageToken=<%nextlink%>.

See the screenshot below to get more clarity:

Configure BigQuery API Pagination on SSIS JSON Source

Configure Array Transformation

Now the last thing we have to configure is special 2-dimensional JSON array format used by BigQuery API:

1. On the JSON Source UI go to 2D Array Transformation tab.
2. Enter the following settings:
   1. For Transformation Type select Transform complex 2-dimensional array.
   2. For Column Name filter enter $.schema.fields[*].name.
   3. For Row Values Filter enter $.f[*].v.

JSON Array Transformation Options

Preview Data and Save UI

That’s it click Preview to see some data. If you entered sample JobID for your User::vJobID variable then you will see some data. In the next section we will see how to load Google BigQuery data into SQL Server. You can click Columns Tab to review Metadata.

Click OK to save UI and generate Metadata.

Configure Target – Load Google BigQuery data into SQL Server

Now you can connect your JSON Source to any target such as ZS Trash Destination or a real database destination such as OLEDB Destination.

If you wish to dump data from Google BigQuery to a SQL Server table then just perform the following steps:

1. Drag and drop OLEDB Destination from SSIS Toolbox. Rename it to something like SQL Server Table.
2. Double click on OLEDB Destination.
3. Click New to create a new connection > Configure connection > Click OK.
4. Click on New to create a new table > Rename default table name > Click OK.
5. Click on Mappings tab and click OK to save UI with default mappings.

Configure SSIS OLEDB Destination – Google BigQuery to SQL Server Import

Execute Package – Loading BigQuery data into SQL Server

SSIS Package Execution –
Loading Google BigQuery Data into SQL Server

Create / Delete Google BigQuery Dataset using API call

As we mentioned before, you need dataset before you can create a table. Most common way to create dataset is via User Interface but what if you like to automate from your SSIS Package or other workflow? Here is how you can create or delete. Basically you can use REST API Task to send CREATE or DROP command for Dataset object

Create Google Dataset

To create dataset configure REST API Task using below settings (We will call this API)

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/datasets

4. Request Method as POST
5. Enter Request Body as below (change YOUR_DATASET_NAME – e.g. TestDataset )
   

   {"datasetReference": { "datasetId": "YOUR_DATASET_NAME", "projectId": "{{User::ProjectId}}"} }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Dataset already exists then you can go to Error Handling Tab (See next section for screenshot) and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Delete Google BigQuery Dataset

For deleing dataset you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Create / Delete Google BigQuery Table using API call

Now let’s look at how to create /drop Table in Google BigQuery using API calls.

Create Google BigQuery Table

To send CREATE TABLE  SQL statement (DDL) we have to use same approach as we send normal SQL Query using this API call. So notice we used Standard SQL for this call by supplying  useLegacySql: false . DDL Statement must be all in one line

POST https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

Content-Type: application/json

>>>> BODY <<<<<

{
 "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
 "useLegacySql": false,
 "timeoutMs": 100000 
}

To create dataset configure REST API Task using below settings

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

4. Request Method as POST
5. Enter Request Body as below
   

   {
    "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
    "useLegacySql": false,
    "timeoutMs": 100000
   }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Table exists then you can go to enable Error Handling Tab and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Create Google BigQuery Table – API Call (Continue On Error Setting – Skip CREATE if Table / Dataset Already Exists)

Delete Google BigQuery Table

For deleing table you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Write data to Google BigQuery using SSIS – 1 Million row insert test (FAST)

Now let’s look at how easy it is to import data into Google BigQuery using SSIS. We will use the same OAuth connection we created before. To learn more about inserting data into BigQuery check tabledata/insertAll method documentation.

Make sure billing is enabled

Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

Streaming is not available via the free tier. If you attempt to use streaming without enabling billing, you receive the following error: BigQuery: Streaming insert is not allowed in the free tier.

As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call (Write  demo).

Configure OAuth Connection / Permissions

If you want to perform data insert operation in BigQuery using API calls then include the following scopes in your OAuth Connection Manager and generate a token (see our first section of this article – We already included scopes for Write operation but incase you didnt do then regenerate token with below scopes):

https://www.googleapis.com/auth/bigquery
https://www.googleapis.com/auth/cloud-platform
https://www.googleapis.com/auth/bigquery.insertdata

Create BigQuery Dataset (From UI)

For this demo first create a test dataset and one table under it like shown below (billing must be enabled on your Google API Project). To do that go to https://bigquery.cloud.google.com/welcome and configure them:

Insert data into BigQuery using API call

And here is a REST API example to insert data into a BigQuery Table:

Request URL:

https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

Request Headers:

Authorization: Bearer ya29.Gl0cBxxxxxxxxxxxxxxxxxxuEIhJIEnxE6GsQPHI
Content-Type: application/json

Request Body:

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
     {"json": {"RowId": 2,"CustomerName": "BBB"} }
   ]
}

Here is our data flow setup to achive very high throughput for Google BigQuery Data Load. We will show you how to insert one million rows in Google BigQuery in less than a minute based on below setup. We will use Multi Threading option and New Compression Option (Added in v3.1.4)

1. Dummy Data Source – Generate sample records
2. JSON Generator Transform – Generates JSON documents to send as POST request for above /insertAll API call.
3. Web API destination  – Call /insertAll API call to submit our data to BigQuery
4. (Optional) JSON Parser Transform – Parse Error Message for any response
5. (Optional) Trash Destination – Save any errors to text file for review

Google BigQuery Data Load Demo in SSIS – 1 Million Rows Insert with Multi Threads and Compression ON (Fast Upload)

Configure SSIS JSON Generator – Generate JSON for BigQuery Table Insert

Now let’s see how to build an HTTP request with JSON body and send it to BigQuery:

1. Drag Data Flow Task and double click on it.
   
2. Drag and configure your Source (for this demo we use Dummy Data Source with Customer Template). See previous section for configuration of Dummy Data Source.
   
3. Drag and drop ZS JSON Generator Transform to produce JSON from a database or file records. If your source is already sending a valid JSON then you can skip this step (e.g. SQL query is
   returning JSON). You can also read raw JSON from a very large file (new-line separated JSON) using JSON Source with Output as Raw Data option checked.
4. Connect Source to JSON Generator. Double click JSON Generator Transform to start configuring it like below.
5. Select Output Mode as Single Dataset Array and enter Batch Size 10000 (This is Max limit allowed by Google BigQuery API insertAll)
6. First right click on Mappings node and select Add Static Element
   

   Generate JSON for Google BigQuery InsertAll API request – Batch 10000 rows in a single API call

7. Enter Name as kind and value as bigquery#tableDataInsertAllRequest
   
   

   JSON Generator – Add Static Element

8. Now right click on Mappings node and select Add Document Array option
   

   JSON Generator – Add Document Array

9. Enter rows as array title and click OK
   

   JSON Generator – Name array

10. Select newly added array node and right click > Add unbound nested element  enter Output alias as json and click OK.
    

    JSON Generator – Add unbound nested element

11. Select json node and right click > Select Add Elements below this node and select multiple columns you like to send to BigQuery. Click OK to save.
    

    JSON Generator – Add Multiple Elements

12. Now let’s preview our JSON (Copy preview JSON to try in the next step – Web API destination)
    NOTE: Table name and column names are case-sensitive so make sure your JSON attribute matches exact same way.
    Here is the finished JSON Structure for next Step
    

    Sample JSON Request body for Google BigQuery insertAll API request

13. Click OK to save UI.

Configure SSIS Web API destination – Insert data into BigQuery Table

Once you have Input JSON prepared,  now let’s configure destination for BigQuery.

1. Drag and drop ZS Web API Destination.
2. Connect your JSON Generator Transform to Web API destination.
3. Configure general properties:
   

   SSIS Web API Destination –
   Configure for BigQuery Data load

4. Make sure to enter URL in this format:
   

   https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

   Make sure to replace 3 parts in above URL (MY_PROJECT_ID, MY_DATASET_ID, MY_TABLE_ID) with actual values from your Google Project and BigQuery dataset/table configuration.
5. Now you can enable Compression and Multiple Threads for higher throughput as below.
   NOTE: Compression Property was added in v3.1.4 so you may not see it if you have older version.
   

   Google BigQuery Data Loading Performance Optimization Options – Enable Multiple Threads and Compression Options

6. If you want to try test insert request from UI without running full package then go back to first tab and edit Body (Use Sample JSON generated by previous JSON Transform – You can grab from JSON Preview Panel on Generator Transform).Click Test Request / Response and confirm Success as below. You can go back to your BigQuery Portal and check one row is inserted after our test click. If everything looking good then run full package to insert all records.Sample JSON for Body:
   

   {
     "kind": "bigquery#tableDataInsertAllRequest",
     "rows": [
        {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
        {"json": {"RowId": 2,"CustomerName": "BBB"} }
      ]
   }

   

   Test Google BigQuery Table Insert – SSIS Web API Destination UI

7. Hit OK to save UI.
8. Run the package and verify data in Google BigQuery Console:
   

   Loading data into Google BigQuery using SSIS

Error Handling for BigQuery Data Load (Bulk Insert API Calls)

There will be a time when some records you insert may not go well in Google BigQuery. In such case you can read output from Web API destination and parse further with JSON Parser Transform. Check for certain values in the output. You must use JSON Parser after Web API destination (Connect Blue Arrow from Web API destination – Since its Soft Error it won’t redirect in Red Arrow ).

For example here is the sample JSON in POST Body for testing which produces error due to bad column name. When bad row found in batch all records will be rejected. Notice that error returns index of record in batch so you can identify which row went bad. It also returns column name in location attribute.

NOTE: Bad column name in 2nd record

Test Body (Bad):

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RecordID": 1,"CustomerID": "X1"} }, 
     {"json": {"Bad_Column": 2,"CustomerID": "X2"} }
     {"json": {"RecordID": 3,"CustomerID": "X3"} }, 
   ]
}

Response (For Bad Input):

{
  "kind": "bigquery#tableDataInsertAllResponse",
  "insertErrors": [
    {
      "index": 1,
      "errors": [
        {
          "reason": "invalid",
          "location": "bad_column",
          "debugInfo": "",
          "message": "no such field."
        }
      ]
    },
    {
      "index": 0,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    },
    {
      "index": 2,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    }
  ]
}

To configure error detection perform following steps.

1. Drag and drop ZS JSON Parser Transform after Web API destination
2. Click on Web API destination. Connect Blue arrow  to JSON Parser Transform
3. Configure JSON Parser Transform like below
4. Connect JSON Parser TRansform to some Destination to save error information (e.g. SQL Table or Trans destination)

Handling BigQuery Insert Errors in SSIS

Other Common Errors in BigQuery API calls

In this section we will talk about many common API errors in BigQuery.

Error: The project XXXXXXX has not enabled BigQuery

Sometimes you might get below error. To fix this error make sure you go to your Project and Enable BigQuery API

Status Code: BadRequest

Response Body: {
  "error": {
    "code": 400,
    "message": "The project bigquerytest-281915 has not enabled BigQuery.",
    "errors": [
      {
        "message": "The project bigquerytest-281915 has not enabled BigQuery.",
        "domain": "global",
        "reason": "invalid"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}

Error: 404 – Not Found: Table / Job xxxxxx

• Make sure you are setting GET request and not POST.
• Also make sure your jobId is valid because it expires after 24 hours.
• Make sure project-id supplied in URL is valid ID  (DO NOT Specify Alias, use internal ID for project)
• Make sure you supplied location parameter if your outside EU / US region. Reading data using this API call might fail if you failed to supply location in URL

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "notFound",
    "message": "Not Found: Table xxxxxx"
   }
  ],
  "code": 404,
  "message": "Not Found: Table xxxxxxx
 }
}

Error: 403 – Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier

If you trying call certain APIs on sandbox mode or free tier then you might get below error. To overcome this error enable billing on google bigquery. As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call.

Response Url: https://www.googleapis.com/bigquery/v2/projects/bigquerytest-281915/datasets/TestDataset/tables/Table1/insertAll

Status Code: Forbidden

Response Body: {
  "error": {
    "code": 403,
    "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
    "errors": [
      {
        "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
        "domain": "global",
        "reason": "accessDenied"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}

Debugging Web API Requests

If you need to debug actual requests made to Google server then you can use a tool like Fiddler. It’s a very handy tool to troubleshoot JSON format issues. It will allow to see how a request is made to a server.

Using Fiddler to debug Google BigQuery API requests in SSIS

Common Errors

Deployment to Production

Download Sample Package

Click Here to Download SSIS Sample Package – Google BigQuery API Read Write Create Delete (SSIS 2019, 2017, 2012)

Conclusion. What’s next?

In this article we have learned how to load data from Google BigQuery into SQL Server using SSIS (drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from Google BigQuery REST API using OAuth. JSON Source Connector makes it super simple to parse complex/large JSON files or any Web API response into rows and columns so you can load data into a database, e.g. SQL Server database. Download SSIS PowerPack to try many other automation scenarios that were not discussed in this article.

Keywords:

Google BigQuery Integration with SQL Server | How to extract data from google bigquery in SSIS? | How to read data from Google BigQuery API? | Loading BigQuery Data into SQL Server. | BigQuery to SQL Server | SSIS Google Big Query Integration | SSIS Google BigQuery Import  JSON File | SSIS Google BigQuery Export data

The post How to read / write data in Google BigQuery using SSIS appeared first on ZappySys Blog.

Elasticsearch is a powerful engine that allows you to store, aggregate, and, most importantly, search data in a very analytical way. In this tutorial, you will learn how to bulk load data from SQL Server to Elasticsearch with SSIS (part of SQL Server) and ZappySys PowerPack.

The scope of this article will be to show how to import records from SQL Server into Elasticsearch index as JSON documents using Elasticsearch Bulk API. In a nutshell, we will retrieve IDs and names of the products from a SQL table, transform each row into a JSON and lastly, index each JSON in Elasticsearch under record’s corresponding ID. This tutorial is going to be your stepping stone to use any Elasticsearch API as a destination.

We will be using these SSIS components of PowerPack to make things work:

	JSON Generator Transform
	Web API Destination

Let’s begin?

Prerequisites

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Northwind database deployed on your machine (we will use it to load data from; download it).
4. Elasticsearch instance up and running.
5. ZappySys SSIS PowerPack installed.

<New ZS-OAUTH Connection>

Step-by-Step – Bulk loading data from SQL Server to Elasticsearch with SSIS

Load data from SQL Server database first

In this section, you will perform several basic steps to start the package: load data from SQL Server and prepare it for further steps.

1. Create a new SSIS package and drag a Data Flow Task into the Control Flow from the SSIS Toolbox.
   

   Dragging and dropping Data Flow Task into Control Flow

2. Open Data Flow Task and then add OLE DB Source.
3. Configure OLE DB Source to take data from Products table, Northwind database.
4. Choose ProductID and ProductName as columns you will use.
   

   Load data to Elasticsearch using SSIS and ZappySys Web API Destination component.

Transform data into JSON documents

To add data into Elasticsearch firstly we need to prepare it with JSON Generator Transform. It will help us convert table rows into JSON documents.

JSON String to Create Index data for the Bulk API call

1. Close the window and then drag Derived Column component from the SSIS Toolbox.
2. Furthermore, Add two new columns and name them 

   RowHeader

    and 

   RowFooter

   :
   

   Derived Column configuration to add prefix and header and footer to a JSON

   In the Expression column give them values:

   • RowHeader:
     

     "{ \"index\": { \"_index\": \"shop\", \"_type\": \"products\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n"

   • RowFooter:  

     "\n"

3. Now drag JSON Generator Transform from the SSIS Toolbox and open it.
4. Right mouse click on item Mappings and select

   Add Element(s) (Below this node)

   :
   

   Convert SQL table rows into JSON using JSON Generator Transform

5. Once a dialog appears, select

   ProductName

    as Source Column and

   name

    as Output Alias:
   
   

   Map table column to JSON property using JSON Generator Transform

6. You should get a similar view:
   

   JSON Generator Transform configuration to create JSON from a SQL table

7. Then go to Header / Footer tab and:
   • Select

     Direct String

      option.
   • Input

     <%RowHeader%>

      in Header String option.
   • Input

     <%RowFooter%>

      in Footer String option.
   

   JSON Generator Transform configuration to convert rows into JSON suitable for Elasticsearch Bulk operation

8. Close the window.

JSON String to Update Index data for the Bulk API call

From the above section “JSON String to Create Index data for the Bulk API call” follows all the steps From Step-1 to Step-8.

Just for (2nd step) Step-2 use below expression for  

RowHeader

RowFooter

Derived Column configuration to add prefix and header and footer to a JSONIn the Expression column give them values:

• RowHeader:
  

  "{ \"update\": { \"_index\": \"shop\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n{ \"doc\" : "

• RowFooter:  

  "}\n"

JSON String to Delete Index data for the Bulk API call

From the above section “JSON String to Create Index data for the Bulk API call” follows steps Step-1 and Step-2.

Just for (2nd step) Step-2 use below expression for  

RowHeader

Derived Column configuration to add prefix and header and footer to a JSON

In the Expression column give them values:

• RowHeader:
  

  "{ \"delete\": { \"_index\": \"shop\", \"_id\" : \"" + (DT_WSTR,100)ProductID + "\" } }\n"

Finally, Elasticsearch & SQL Server integration

1. Now drag Web API Destination SSIS component into the Data Flow and open it for editing.
2. In Select Connection section press

   <New ZS-HTTP Connection>

   .
   NOTE: If your ElasticSearch instance is hosted as AWS Managed Service then select  

   <New ZS-OAUTH Connection>

   . rather than ZS-HTTP. When OAuth UI launches select AWS v4 Provider. For more information on calling REST API on AWS check this article.
3. Once HTTP Connection Manager window opens configure connection to your Elasticsearch instance:
   • Set Web Url, which points to your Elasticsearch instance.
   • Set Credentials Type to

     Basic - UserID/Password

      (or other appropriate authentication method).
   • Finally, set User Name and Password:
     
     

     Configure SSIS HTTP Connection to connect to Elasticsearch

4. Close configuration window. Now it’s time to configure Web API Destination.
5. All that is separating you from getting those rows into Elasticsearch is:
   • Setting Input Column for Body to

     ZS_JSON_OUT

      for Create and Update Index Data and for Delete Index Data Input Column for Body to

     RowHeader [Derived Column]

     .
   • Setting URL to

     http://localhost:9200/_bulk

     .
   • Setting HTTP Request Method to

     POST

     .
   

   Index SQL Server Database data in Elasticsearch using SSIS component Web API Destination

6. Then open Batch Settings (For Body) tab:
   • Enable batch submission.
   • Set Body Batch Size, e.g. 1000. Make sure this number is even, otherwise you may get into problems (Elasticsearch Bulk request has special JSON request body format, as you perhaps noticed).
   

   Web API Destination batch settings configuration

7. Close the window and run the package! You should see green lights, telling you everything is OK:
   

   Successful data load from SQL Server to Elasticsearch

8. We can also query Elasticsearch by using one of its Search APIs – URI Search to see if we successfully indexed data:
   

   SQL Server data index in Elasticsearch

9. Rows number match in SSIS and in search results, thus everything is good. You are done.

What if I want more? After upserting data use Web API Destination further

Let’s say you have a requirement to do something with rows that were freshly indexed in Elasticsearch. Then you will need somehow to distinguish between created and updated records in Elasticsearch. And that won’t be difficult because Web API Destination acts not only as Destination but as Transformation as well. So one thing you have to do is connect JSON Parser Transform downstream to Web API Destination. It will parse Elasticsearch HTTP JSON response – which is returned by Web API Destination – into columns, which you can later easily redirect using Conditional Split:

1. Add JSON Parser Transform and connect it to Web API Destination:
   

   Use JSON Parser Transform to get Elasticsearch HTTP JSON response when integrating SQL Server & Elasticsearch

2. Make sure you have selected

   ResponseText

    as Select Input JSON Column.
3. Then go to Bulk API and copy/paste “the result of this bulk operation” JSON response:
   

   Using JSON Parser Transform to parse JSON response from REST HTTP request

4. Set

   $.items[*]

    as the filter.
5. Optional step. You won’t find a response sample for every Elasticsearch API call. In that case, you may want to use Postman to make an HTTP request to Elasticsearch and get a sample response:
   

   Use Postman to make a request to Elasticsearch and get a sample response to be used in JSON Parser Transform

   NOTE: Don’t forget to set the username and password in Authorization section!
6. Then use a Conditional Split and Trash Destination to redirect the rows:
   
   Input these clauses:
   • Inserted:
     

     REPLACENULL([index.result],"") == "created" || REPLACENULL([create.result],"") == "created"

   • Updated:
     

     REPLACENULL([index.result],"") == "updated" || REPLACENULL([create.result],"") == "updated"

7. As a result, new index records will be redirected to one destination, while updated records – to the other:
   

Overall you can use Web API Destination HTTP JSON response for other useful things as well, e.g. determine on how many replica shards record was indexed. JSON response will depend on which Elasticsearch API and which method you use.

Delete Index by making an API call.

If you want to delete the index by making Delete API call you can. Let’s make that call using the Rest API Task. Configure it like below screen and click on the Test Request button.

Rest API Delete Method

Things went bad: Error handling & debugging

Incidentally, you may incorrectly construct JSON for Web API Destination body. Elasticsearch nodes may go offline or go out of memory. In any case you may want to know when that happens and take actions accordingly. For that purpose you have to redirect failed requests from Web API Destination to some other destination:

1. Add a Derived Column above Web API Destination with expression

   (DT_WSTR,4000)ZS_JSON_OUT

    and name it

   JsonAsString

   . This will let you see what JSON you are actually passing.
2. After that, add database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don’t forget to set

   Redirect row

    option for both, Error and Truncation columns:
   
3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled in Elasticsearch

Call ElasticSearch API hosted on AWS

If your ElasticSearch instance is hosted as AWS Managed ElasticSearch Instance then select  

<New ZS-OAUTH Connection>

Call AWS Hosted ElasticSearch REST API in SSIS (V4 Request Signing)

Common Errors

Deployment to Production

Download a sample package

Be sure to download a sample SQL Server 2008 SSIS package, in case you want to try it right away (you can upgrade it to a higher version).

Conclusion. What’s next?

In conclusion, in this article we have learned how to bulk load data from SQL Server and upsert it in Elasticsearch index. We used JSON Generator Transform to help us convert database table rows into JSON documents. In addition, Web API Destination was very helpful in automatically creating and making HTTP requests to Elasticsearch instance and indexing our data as a result. This tutorial was quite simple and straightforward, yet it is a good starter to use other Elasticsearch APIs and their features. From now on Elasticsearch reference is your best friend in learning what APIs exist, what are their methods and how HTTP requests should look like.

Keywords

How to import data from SQL Server to Elasticsearch | How to load data into Elasticsearch | Using Elasticsearch to index SQL Server | Elasticsearch and SQL Server integration | How to use Elasticsearch together with SQL Server | Upsert SQL Server data into Elasticsearch

The post Loading data from SQL Server to Elasticsearch with SSIS appeared first on ZappySys Blog.