How to read Google BigQuery data in MS SQL/SSRS/Java using ZappySys Data Gateway?

In this section we will learn how to configure and use Google BigQuery Connector in the API Driver to extract data from the Google BigQuery.

Create Data Source in ZappySys Data Gateway based on API Driver

Download and install ZappySys ODBC PowerPack.
Search for gateway in start menu and Open ZappySys Data Gateway:
Go to Users Tab to add our first Gateway user. Click Add; we will give it a name tdsuser and enter password you like to give. Check Admin option and click OK to save. We will use these details later when we create linked server:
Now we are ready to add a data source. Click Add, give data source a name (Copy this name somewhere, we will need it later) and then select Native - ZappySys API Driver. Finally, click OK. And it will create the Data Set for it and open the ZS driver UI.

GoogleBigQueryDSN
When the Configuration window appears give your data source a name if you haven't done that already, then select "Google BigQuery" from the list of Popular Connectors. If "Google BigQuery" is not present in the list, then click "Search Online" and download it. Then set the path to the location where you downloaded it. Finally, click Continue >> to proceed with configuring the DSN:

GoogleBigQueryDSN

Google BigQuery

Now it's time to configure the Connection Manager. Select Authentication Type, e.g. Token Authentication. Then select API Base URL (in most cases, the default one is the right one). More info is available in the Authentication section.

User Account [OAuth]

Steps to get Google BigQuery Credentials
This connection can be configured using two ways. Use Default App (Created by ZappySys) OR Use Custom App created by you.
To use minimum settings you can start with ZappySys created App. Just change UseCustomApp=false on the properties grid so you dont need ClientID / Secret. When you click Generate Token you might see warning about App is not trusted (Simply Click Advanced Link to expand hidden section and then click Go to App link to Proceed).

To register custom App, perform the following steps (Detailed steps found in the help link at the end)

Go to Google API Console
From the Project Dropdown (usually found at the top bar) click Select Project
On Project Propup click CREATE PROJECT
Once project is created you can click Select Project to switch the context (You can click on Notification link or Choose from Top Dropdown)
Click ENABLE APIS AND SERVICES
Now we need to Enable two APIs one by one (BigQuery API and Cloud Resource Manager API).
Search BigQuery API. Select and click ENABLE
Search Cloud Resource Manager API. Select and click ENABLE
Go to back to main screen of Google API Console
Click OAuth consent screen Tab. Enter necessary details and Save.
1. Choose Testing as Publishing status
2. Set application User type to Internal, if possible
3. If MAKE INTERNAL option is disabled, then add a user in Test users section, which you will use in authentication process when generating Access and Refresh tokens
Click Credentials Tab
Click CREATE CREDENTIALS (some where in topbar) and select OAuth Client ID option.
When prompted Select Application Type as Desktop App and click Create to receive your ClientID and Secret. Later on you can use this information now to configure Connection with UseCustomApp=true.
Go to OAuth Consent Screen tab. Under Publishing Status click PUBLISH APP to ensure your refresh token doesnt expire often. If you planning to use App for Private use then do not have to worry about Verification Status after Publish.

Fill in all required parameters and set optional parameters if needed:

GoogleBigQueryDSN

Google BigQuery

User Account [OAuth]

https://www.googleapis.com/bigquery/v2

Required Parameters
UseCustomApp	Fill in the parameter...
ProjectId (Choose after [Generate Token] clicked)	Fill in the parameter...
DatasetId (Choose after [Generate Token] clicked and ProjectId selected)	Fill in the parameter...
Optional Parameters
ClientId	Fill in the parameter...
ClientSecret	Fill in the parameter...
Scope	Fill in the parameter...
RetryMode	Fill in the parameter...
RetryStatusCodeList	Fill in the parameter...
RetryCountMax	Fill in the parameter...
RetryMultiplyWaitTime	Fill in the parameter...
Job Location	Fill in the parameter...
Redirect URL (Only for Web App)	Fill in the parameter...

Service Account (Using Private Key File) [OAuth]

Steps to get Google BigQuery Credentials
Use these steps to authenticate as service account rather than Google / GSuite User. Learn more about service account here

Basically to call Google API as Service account we need to perform following steps listed in 3 sections (Detailed steps found in the help link at the end)

Create Project

First thing is create a Project so we can call Google API. Skip this section if you already have Project (Go to next section)

Go to Google API Console
From the Project Dropdown (usually found at the top bar) click Select Project
On Project Propup click CREATE PROJECT
Once project is created you can click Select Project to switch the context (You can click on Notification link or Choose from Top Dropdown)
Click ENABLE APIS AND SERVICES
Now we need to Enable two APIs one by one (BigQuery API and Cloud Resource Manager API).
Search BigQuery API. Select and click ENABLE
Search Cloud Resource Manager API. Select and click ENABLE

Create Service Account

Once Project is created and APIs are enabled we can now create a service account under that project. Service account has its ID which looks like some email ID (not to confuse with Google /Gmail email ID)

Go to Create Service Account
From the Project Dropdown (usually found at the top bar) click Select Project
Enter Service account name and Service account description
Click on Create. Now you should see an option to assign Service Account permissions (See Next Section).

Give Permission to Service Account

By default service account cant access BigQuery data or List BigQuery Projects so we need to give that permission using below steps.

After you Create Service Account look for Permission drop down in the Wizard.
Choose BigQuery -> BigQuery Admin role so we can read/write data. (NOTE: If you just need read only access then you can choose BigQuery Data Viewer)
Now choose one more Project -> Viewer and add that role so we can query Project Ids.
Click on Continue. Now you should see an option to Create Key (See Next Section).

Create Key (P12)

Once service account is created and Permission is assigned we need to create key file.

In the Cloud Console, click the email address for the service account that you created.
Click Keys.
Click Add key, then click Create new key.
Click Create and select P12 format. A P12 key file is downloaded to your computer. We will use this file in our API connection.
Click Close.
Now you may use downloaded *.p12 key file as secret file and Service Account Email as Client ID (e.g. some_name@some_name.iam.gserviceaccount.com).

Manage Permissions / Give Access to Other Projects

We saw how to add permissions for Service Account during Account Creation Wizard but if you ever wish to edit after its created or you wish to give permission for other projects then perform forllowing steps.

From the top Select Project for which you like to edit Permission.
Go to IAM Menu option (here)
Link to IAM: https://console.cloud.google.com/iam-admin/iam
Goto Permissions tab. Over there you will find ADD button.
Enter Service account email for which you like to grant permission. Select role you wish to assign.

Fill in all required parameters and set optional parameters if needed:

GoogleBigQueryDSN

Google BigQuery

Service Account (Using Private Key File) [OAuth]

https://www.googleapis.com/bigquery/v2

Required Parameters
Service Account Email	Fill in the parameter...
P12 Service Account Private Key Path (i.e. *.p12)	Fill in the parameter...
ProjectId	Fill in the parameter...
DatasetId (Choose after ProjectId)	Fill in the parameter...
Optional Parameters
Scope	Fill in the parameter...
RetryMode	Fill in the parameter...
RetryStatusCodeList	Fill in the parameter...
RetryCountMax	Fill in the parameter...
RetryMultiplyWaitTime	Fill in the parameter...
Job Location	Fill in the parameter...

Once the data source has been configured, you can preview data. Select the Preview tab and use settings similar to the following to preview data:
Click OK to finish creating the data source.

Reading data from client application

Firstly, to get data from ODBC data source based on ZappySys ODBC driver, in your client application, you would need to connect to ODBC source and then from the list select the data source.
Finally, to read the data just read tables/views in your app or enter a SQL statement to extract data, e.g.:
SELECT * FROM [$parent.tableReference.tableId$]

Google BigQuery Connector Examples

The ZappySys API Driver is a user-friendly interface designed to facilitate the seamless integration of various applications with the Google BigQuery API. With its intuitive design and robust functionality, the ZappySys API Driver simplifies the process of configuring specific API endpoints to efficiently read or write data from Google BigQuery.

Click here to find more Google BigQuery Connector examples designed for seamless integration with the ZappySys API ODBC Driver under ODBC Data Source (36/64) or ZappySys Data Gateway, enhancing your ability to connect and interact with Prebuilt Connectors effectively.

Consume Data inside your App / Programming Language

Once you know how to load data from Google BigQuery Connector, you can click on one of the below links to learn the steps how to consume data inside your App / Programming Language from Google BigQuery Connector.

ODBC inside ETL / Reporting / BI Tools

ODBC inside Programming Languages

Key features of the ZappySys API Driver include:

The API ODBC driver facilitates the reading and writing of data from numerous popular online services (refer to the complete list here) using familiar SQL language without learning complexity of REST API calls. The driver allows querying nested structure and output as a flat table. You can also create your own ODBC / Data Gateway API connector file and use it with this driver.

Intuitive Configuration: The interface is designed to be user-friendly, enabling users to easily set up the specific API endpoints within Google BigQuery without requiring extensive technical expertise or programming knowledge.
Customizable Endpoint Setup: Users can conveniently configure the API endpoint settings, including the HTTP request method, endpoint URL, and any necessary parameters, to precisely target the desired data within Google BigQuery.
Data Manipulation Capabilities: The ZappySys API Driver allows for seamless data retrieval and writing, enabling users to fetch data from Google BigQuery and perform various data manipulation operations as needed, all through an intuitive and straightforward interface.
Secure Authentication Integration: The driver provides secure authentication integration, allowing users to securely connect to the Google BigQuery API by inputting the necessary authentication credentials, such as API tokens or other authentication keys.
Error Handling Support: The interface is equipped with comprehensive error handling support, ensuring that any errors or exceptions encountered during the data retrieval or writing process are efficiently managed and appropriately communicated to users for prompt resolution.
Data Visualization and Reporting: The ZappySys API Driver facilitates the seamless processing and presentation of the retrieved data from Google BigQuery, enabling users to generate comprehensive reports and visualizations for further analysis and decision-making purposes.

Overall, the ZappySys API Driver serves as a powerful tool for streamlining the integration of applications with Google BigQuery, providing users with a convenient and efficient way to access and manage data, all through a user-friendly and intuitive interface.