Write data in Google BigQuery using Google BigQuery Connector in SSIS

In this section we will learn how to configure and use Google BigQuery Connector in the API Destination to write data in the Google BigQuery.

Video Tutorial - Write or lookup data to Google BigQuery using SSIS

This video covers following and more so watch carefully. After watching this video follow the steps described in this article.

How to download SSIS PowerPack for Google BigQuery integration in SSIS
How to configure connection for Google BigQuery
How to Write or lookup data to Google BigQuery
Features about SSIS API Destination
Using Google BigQuery Connector in SSIS

How to write or lookup data to Google BigQuery in SSIS (Import data)

In upper section we learned how to read data, now in this section we will learn how to configure Google BigQuery in the API Source to Post data to the Google BigQuery.

Read the data from the source, being any desired source component. In example we will use ZappySys Dummy Data Source component.
From the SSIS Toolbox drag and drop API Destination (Predefined Templates) on the Data Flow Designer surface and connect source component with it, and double click to edit it.
Select New Connection to create a new connection:

API Destination - Google BigQuery

Read / write Google BigQuery data inside your app without coding using easy to use high performance API Connector
Use a preinstalled Google BigQuery Connector from Popular Connector List or press Search Online radio button to download Google BigQuery Connector. Once downloaded simply use it in the configuration:

Google BigQuery

Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed. You may press a link Steps to Configure which will help set certain parameters. More info is available in Authentication section.

User Account [OAuth]

Steps to get Google BigQuery Credentials
This connection can be configured using two ways. Use Default App (Created by ZappySys) OR Use Custom App created by you.
To use minimum settings you can start with ZappySys created App. Just change UseCustomApp=false on the properties grid so you dont need ClientID / Secret. When you click Generate Token you might see warning about App is not trusted (Simply Click Advanced Link to expand hidden section and then click Go to App link to Proceed).

To register custom App, perform the following steps (Detailed steps found in the help link at the end)

Go to Google API Console
From the Project Dropdown (usually found at the top bar) click Select Project
On Project Propup click CREATE PROJECT
Once project is created you can click Select Project to switch the context (You can click on Notification link or Choose from Top Dropdown)
Click ENABLE APIS AND SERVICES
Now we need to Enable two APIs one by one (BigQuery API and Cloud Resource Manager API).
Search BigQuery API. Select and click ENABLE
Search Cloud Resource Manager API. Select and click ENABLE
Go to back to main screen of Google API Console
Click OAuth consent screen Tab. Enter necessary details and Save.
1. Choose Testing as Publishing status
2. Set application User type to Internal, if possible
3. If MAKE INTERNAL option is disabled, then add a user in Test users section, which you will use in authentication process when generating Access and Refresh tokens
Click Credentials Tab
Click CREATE CREDENTIALS (some where in topbar) and select OAuth Client ID option.
When prompted Select Application Type as Desktop App and click Create to receive your ClientID and Secret. You can use this information now to configure Connection with UseCustomApp=true.

Google BigQuery

User Account [OAuth]

https://www.googleapis.com/bigquery/v2

Required Parameters
UseCustomApp	Fill in the parameter...
ProjectId (Choose after [Generate Token] clicked)	Fill in the parameter...
DatasetId (Choose after [Generate Token] clicked and ProjectId selected)	Fill in the parameter...
Optional Parameters
ClientId	Fill in the parameter...
ClientSecret	Fill in the parameter...
Scope	Fill in the parameter...
RetryMode	Fill in the parameter...
RetryStatusCodeList	Fill in the parameter...
RetryCountMax	Fill in the parameter...
RetryMultiplyWaitTime	Fill in the parameter...
Job Location	Fill in the parameter...

Service Account (Using Private Key File) [OAuth]

Steps to get Google BigQuery Credentials
Use these steps to authenticate as service account rather than Google / GSuite User. Learn more about service account here

Basically to call Google API as Service account we need to perform following steps listed in 3 sections (Detailed steps found in the help link at the end)

Create Project

First thing is create a Project so we can call Google API. Skip this section if you already have Project (Go to next section)

Go to Google API Console
From the Project Dropdown (usually found at the top bar) click Select Project
On Project Propup click CREATE PROJECT
Once project is created you can click Select Project to switch the context (You can click on Notification link or Choose from Top Dropdown)
Click ENABLE APIS AND SERVICES
Now we need to Enable two APIs one by one (BigQuery API and Cloud Resource Manager API).
Search BigQuery API. Select and click ENABLE
Search Cloud Resource Manager API. Select and click ENABLE

Create Service Account

Once Project is created and APIs are enabled we can now create a service account under that project. Service account has its ID which looks like some email ID (not to confuse with Google /Gmail email ID)

Go to Create Service Account
From the Project Dropdown (usually found at the top bar) click Select Project
Enter Service account name and Service account description
Click on Create. Now you should see an option to assign Service Account permissions (See Next Section).

Give Permission to Service Account

By default service account cant access BigQuery data or List BigQuery Projects so we need to give that permission using below steps.

After you Create Service Account look for Permission drop down in the Wizard.
Choose BigQuery -> BigQuery Admin role so we can read/write data. (NOTE: If you just need read only access then you can choose BigQuery Data Viewer)
Now choose one more Project -> Viewer and add that role so we can query Project Ids.
Click on Continue. Now you should see an option to Create Key (See Next Section).

Create Key (P12)

Once service account is created and Permission is assigned we need to create key file.

In the Cloud Console, click the email address for the service account that you created.
Click Keys.
Click Add key, then click Create new key.
Click Create and select P12 format. A P12 key file is downloaded to your computer. We will use this file in our API connection.
Click Close.
Now you may use downloaded *.p12 key file as secret file and Service Account Email as Client ID (e.g. some_name@some_name.iam.gserviceaccount.com).

Manage Permissions / Give Access to Other Projects

We saw how to add permissions for Service Account during Account Creation Wizard but if you ever wish to edit after its created or you wish to give permission for other projects then perform forllowing steps.

From the top Select Project for which you like to edit Permission.
Go to IAM Menu option (here)
Link to IAM: https://console.cloud.google.com/iam-admin/iam
Goto Permissions tab. Over there you will find ADD button.
Enter Service account email for which you like to grant permission. Select role you wish to assign.

Google BigQuery

Service Account (Using Private Key File) [OAuth]

https://www.googleapis.com/bigquery/v2

Required Parameters
Service Account Email	Fill in the parameter...
Service Account Private Key Path (i.e. *.p12)	Fill in the parameter...
ProjectId	Fill in the parameter...
DatasetId (Choose after ProjectId)	Fill in the parameter...
Optional Parameters
Scope	Fill in the parameter...
RetryMode	Fill in the parameter...
RetryStatusCodeList	Fill in the parameter...
RetryCountMax	Fill in the parameter...
RetryMultiplyWaitTime	Fill in the parameter...
Job Location	Fill in the parameter...

Select the desired endpoint, change/pass the properties values, and go to the Mappings tab to map the columns.

API Destination - Google BigQuery

Read / write Google BigQuery data inside your app without coding using easy to use high performance API Connector
Finally, map the the desired columns:

API Destination - Google BigQuery

Read / write Google BigQuery data inside your app without coding using easy to use high performance API Connector
That's it; we successfully configured the POST API Call. In a few clicks we configured the Google BigQuery API call using ZappySys Google BigQuery Connector