Write data in Google BigQuery using Google BigQuery Connector in SSIS
In this section we will learn how to configure and use Google BigQuery Connector in the API Destination to write data in the Google BigQuery.
Video Tutorial - Write or lookup data to Google BigQuery using SSIS
This video covers following and more so watch carefully. After watching this video follow the steps described in this article.
- How to download SSIS PowerPack for
Google BigQuery integration in SSIS - How to configure connection for
Google BigQuery - How to Write or lookup data to
Google BigQuery - Features about SSIS API Destination
- Using
Google BigQuery Connector in SSIS
How to write or lookup data to Google BigQuery in SSIS (Import data)
In upper section we learned how to read data, now in this section we will learn how to configure Google BigQuery in the API Source to Post data to the Google BigQuery.
-
Read the data from the source, being any desired source component. In example we will use ZappySys Dummy Data Source component.
-
From the SSIS Toolbox drag and drop API Destination (Predefined Templates) on the Data Flow Designer surface and connect source component with it, and double click to edit it.
-
Select New Connection to create a new connection:
API Destination - Google BigQueryRead / write Google BigQuery data inside your app without coding using easy to use high performance API Connector
-
Use a preinstalled Google BigQuery Connector from Popular Connector List or press Search Online radio button to download Google BigQuery Connector. Once downloaded simply use it in the configuration:
Google BigQuery
-
Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed. You may press a link Steps to Configure which will help set certain parameters. More info is available in Authentication section.
Steps to get Google BigQuery Credentials
This connection can be configured using two ways. Use Default App (Created by ZappySys) OR Use Custom App created by you.
To use minimum settings you can start with ZappySys created App. Just change UseCustomApp=false on the properties grid so you dont need ClientID / Secret. When you click Generate Token you might see warning about App is not trusted (Simply Click Advanced Link to expand hidden section and then click Go to App link to Proceed). To register custom App, perform the following steps (Detailed steps found in the help link at the end)- Go to Google API Console
- From the Project Dropdown (usually found at the top bar) click Select Project
- On Project Propup click CREATE PROJECT
- Once project is created you can click Select Project to switch the context (You can click on Notification link or Choose from Top Dropdown)
- Click ENABLE APIS AND SERVICES
- Now we need to Enable two APIs one by one (BigQuery API and Cloud Resource Manager API).
- Search BigQuery API. Select and click ENABLE
- Search Cloud Resource Manager API. Select and click ENABLE
- Go to back to main screen of Google API Console
Click OAuth consent screen Tab. Enter necessary details and Save.
- Choose Testing as Publishing status
- Set application User type to Internal, if possible
- If MAKE INTERNAL option is disabled, then add a user in Test users section, which you will use in authentication process when generating Access and Refresh tokens
- Click Credentials Tab
- Click CREATE CREDENTIALS (some where in topbar) and select OAuth Client ID option.
- When prompted Select Application Type as Desktop App and click Create to receive your ClientID and Secret. Later on you can use this information now to configure Connection with UseCustomApp=true.
- Go to OAuth Consent Screen tab. Under Publishing Status click PUBLISH APP to ensure your refresh token doesnt expire often. If you planning to use App for Private use then do not have to worry about Verification Status after Publish.
Google BigQueryUser Account [OAuth]https://www.googleapis.com/bigquery/v2Required Parameters UseCustomApp Fill in the parameter... ProjectId (Choose after [Generate Token] clicked) Fill in the parameter... DatasetId (Choose after [Generate Token] clicked and ProjectId selected) Fill in the parameter... Optional Parameters ClientId Fill in the parameter... ClientSecret Fill in the parameter... Scope Fill in the parameter... RetryMode Fill in the parameter... RetryStatusCodeList Fill in the parameter... RetryCountMax Fill in the parameter... RetryMultiplyWaitTime Fill in the parameter... Job Location Fill in the parameter... Redirect URL (Only for Web App) Fill in the parameter... 
Steps to get Google BigQuery Credentials
Use these steps to authenticate as service account rather than Google / GSuite User. Learn more about service account here Basically to call Google API as Service account we need to perform following steps listed in 3 sections (Detailed steps found in the help link at the end)Create Project
First thing is create a Project so we can call Google API. Skip this section if you already have Project (Go to next section)- Go to Google API Console
- From the Project Dropdown (usually found at the top bar) click Select Project
- On Project Propup click CREATE PROJECT
- Once project is created you can click Select Project to switch the context (You can click on Notification link or Choose from Top Dropdown)
- Click ENABLE APIS AND SERVICES
- Now we need to Enable two APIs one by one (BigQuery API and Cloud Resource Manager API).
- Search BigQuery API. Select and click ENABLE
- Search Cloud Resource Manager API. Select and click ENABLE
Create Service Account
Once Project is created and APIs are enabled we can now create a service account under that project. Service account has its ID which looks like some email ID (not to confuse with Google /Gmail email ID)- Go to Create Service Account
- From the Project Dropdown (usually found at the top bar) click Select Project
- Enter Service account name and Service account description
- Click on Create. Now you should see an option to assign Service Account permissions (See Next Section).
Give Permission to Service Account
By default service account cant access BigQuery data or List BigQuery Projects so we need to give that permission using below steps.- After you Create Service Account look for Permission drop down in the Wizard.
- Choose BigQuery -> BigQuery Admin role so we can read/write data. (NOTE: If you just need read only access then you can choose BigQuery Data Viewer)
- Now choose one more Project -> Viewer and add that role so we can query Project Ids.
- Click on Continue. Now you should see an option to Create Key (See Next Section).
Create Key (P12)
Once service account is created and Permission is assigned we need to create key file.- In the Cloud Console, click the email address for the service account that you created.
- Click Keys.
- Click Add key, then click Create new key.
- Click Create and select P12 format. A P12 key file is downloaded to your computer. We will use this file in our API connection.
- Click Close.
- Now you may use downloaded *.p12 key file as secret file and Service Account Email as Client ID (e.g. some_name@some_name.iam.gserviceaccount.com).
Manage Permissions / Give Access to Other Projects
We saw how to add permissions for Service Account during Account Creation Wizard but if you ever wish to edit after its created or you wish to give permission for other projects then perform forllowing steps.- From the top Select Project for which you like to edit Permission.
- Go to IAM Menu option (here)
Link to IAM: https://console.cloud.google.com/iam-admin/iam - Goto Permissions tab. Over there you will find ADD button.
- Enter Service account email for which you like to grant permission. Select role you wish to assign.
Google BigQueryService Account (Using Private Key File) [OAuth]https://www.googleapis.com/bigquery/v2Required Parameters Service Account Email Fill in the parameter... P12 Service Account Private Key Path (i.e. *.p12) Fill in the parameter... ProjectId Fill in the parameter... DatasetId (Choose after ProjectId) Fill in the parameter... Optional Parameters Scope Fill in the parameter... RetryMode Fill in the parameter... RetryStatusCodeList Fill in the parameter... RetryCountMax Fill in the parameter... RetryMultiplyWaitTime Fill in the parameter... Job Location Fill in the parameter...
-
Select the desired endpoint, change/pass the properties values, and go to the Mappings tab to map the columns.
API Destination - Google BigQueryRead / write Google BigQuery data inside your app without coding using easy to use high performance API Connector
-
Finally, map the desired columns:
API Destination - Google BigQueryRead / write Google BigQuery data inside your app without coding using easy to use high performance API Connector
-
That's it; we successfully configured the POST API Call. In a few clicks we configured the Google BigQuery API call using ZappySys Google BigQuery Connector
