How to read Google BigQuery data in ZappySys Data Gateway?
In this section we will learn how to configure and use Google BigQuery Connector in the API Driver to extract data from the Google BigQuery.
Create Data Source in ZappySys Data Gateway based on ZappySys API Driver
-
Download and install ZappySys ODBC PowerPack.
-
Search for gateway in start menu and Open ZappySys Data Gateway:
-
Go to Users Tab to add our first Gateway user. Click Add; we will give it a name tdsuser and enter password you like to give. Check Admin option and click OK to save. We will use these details later when we create linked server:
-
Now we are ready to add a data source. Click Add, give data source a name (Copy this name somewhere, we will need it later) and then select Native - ZappySys API Driver. Finally, click OK. And it will create the Data Set for it and open the ZS driver UI.
GoogleBigQueryDSN
-
When the Configuration window appears give your data source a name if you haven't done that already, then select "Google BigQuery" from the list of Popular Connectors. If "Google BigQuery" is not present in the list, then click "Search Online" and download it. Then set the path to the location where you downloaded it. Finally, click Continue >> to proceed with configuring the DSN:
GoogleBigQueryDSNGoogle BigQuery
-
Now it's time to configure the Connection Manager. Select Authentication Type, e.g. Token Authentication. Then select API Base URL (in most cases, the default one is the right one). More info is available in the Authentication section.
Steps to get Google BigQuery Credentials
This connection can be configured using two ways. Use Default App (Created by ZappySys) OR Use Custom App created by you.
To use minimum settings you can start with ZappySys created App. Just change UseCustomApp=false on the properties grid so you dont need ClientID / Secret. When you click Generate Token you might see warning about App is not trusted (Simply Click Advanced Link to expand hidden section and then click Go to App link to Proceed). To register custom App, perform the following steps (Detailed steps found in the help link at the end)- Go to Google API Console
- From the Project Dropdown (usually found at the top bar) click Select Project
- On Project Propup click CREATE PROJECT
- Once project is created you can click Select Project to switch the context (You can click on Notification link or Choose from Top Dropdown)
- Click ENABLE APIS AND SERVICES
- Now we need to Enable two APIs one by one (BigQuery API and Cloud Resource Manager API).
- Search BigQuery API. Select and click ENABLE
- Search Cloud Resource Manager API. Select and click ENABLE
- Go to back to main screen of Google API Console
Click OAuth consent screen Tab. Enter necessary details and Save.
- Choose Testing as Publishing status
- Set application User type to Internal, if possible
- If MAKE INTERNAL option is disabled, then add a user in Test users section, which you will use in authentication process when generating Access and Refresh tokens
- Click Credentials Tab
- Click CREATE CREDENTIALS (some where in topbar) and select OAuth Client ID option.
- When prompted Select Application Type as Desktop App and click Create to receive your ClientID and Secret. You can use this information now to configure Connection with UseCustomApp=true.
Fill in all required parameters and set optional parameters if needed:
GoogleBigQueryDSNUser Account [OAuth]https://www.googleapis.com/bigquery/v2Required Parameters UseCustomApp Fill in the parameter... ProjectId (Choose after [Generate Token] clicked) Fill in the parameter... DatasetId (Choose after [Generate Token] clicked and ProjectId selected) Fill in the parameter... Optional Parameters ClientId Fill in the parameter... ClientSecret Fill in the parameter... Scope Fill in the parameter... RetryMode Fill in the parameter... RetryStatusCodeList Fill in the parameter... RetryCountMax Fill in the parameter... RetryMultiplyWaitTime Fill in the parameter... Job Location Fill in the parameter... 
Steps to get Google BigQuery Credentials
Use these steps to authenticate as service account rather than Google / GSuite User. Learn more about service account here Basically to call Google API as Service account we need to perform following steps listed in 3 sections (Detailed steps found in the help link at the end)Create Project
First thing is create a Project so we can call Google API. Skip this section if you already have Project (Go to next section)- Go to Google API Console
- From the Project Dropdown (usually found at the top bar) click Select Project
- On Project Propup click CREATE PROJECT
- Once project is created you can click Select Project to switch the context (You can click on Notification link or Choose from Top Dropdown)
- Click ENABLE APIS AND SERVICES
- Now we need to Enable two APIs one by one (BigQuery API and Cloud Resource Manager API).
- Search BigQuery API. Select and click ENABLE
- Search Cloud Resource Manager API. Select and click ENABLE
Create Service Account
Once Project is created and APIs are enabled we can now create a service account under that project. Service account has its ID which looks like some email ID (not to confuse with Google /Gmail email ID)- Go to Create Service Account
- From the Project Dropdown (usually found at the top bar) click Select Project
- Enter Service account name and Service account description
- Click on Create. Now you should see an option to assign Service Account permissions (See Next Section).
Give Permission to Service Account
By default service account cant access BigQuery data or List BigQuery Projects so we need to give that permission using below steps.- After you Create Service Account look for Permission drop down in the Wizard.
- Choose BigQuery -> BigQuery Admin role so we can read/write data. (NOTE: If you just need read only access then you can choose BigQuery Data Viewer)
- Now choose one more Project -> Viewer and add that role so we can query Project Ids.
- Click on Continue. Now you should see an option to Create Key (See Next Section).
Create Key (P12)
Once service account is created and Permission is assigned we need to create key file.- In the Cloud Console, click the email address for the service account that you created.
- Click Keys.
- Click Add key, then click Create new key.
- Click Create and select P12 format. A P12 key file is downloaded to your computer. We will use this file in our API connection.
- Click Close.
- Now you may use downloaded *.p12 key file as secret file and Service Account Email as Client ID (e.g. some_name@some_name.iam.gserviceaccount.com).
Manage Permissions / Give Access to Other Projects
We saw how to add permissions for Service Account during Account Creation Wizard but if you ever wish to edit after its created or you wish to give permission for other projects then perform forllowing steps.- From the top Select Project for which you like to edit Permission.
- Go to IAM Menu option (here)
Link to IAM: https://console.cloud.google.com/iam-admin/iam - Goto Permissions tab. Over there you will find ADD button.
- Enter Service account email for which you like to grant permission. Select role you wish to assign.
Fill in all required parameters and set optional parameters if needed:
GoogleBigQueryDSNService Account (Using Private Key File) [OAuth]https://www.googleapis.com/bigquery/v2Required Parameters Service Account Email Fill in the parameter... Service Account Private Key Path (i.e. *.p12) Fill in the parameter... ProjectId Fill in the parameter... DatasetId (Choose after ProjectId) Fill in the parameter... Optional Parameters Scope Fill in the parameter... RetryMode Fill in the parameter... RetryStatusCodeList Fill in the parameter... RetryCountMax Fill in the parameter... RetryMultiplyWaitTime Fill in the parameter... Job Location Fill in the parameter...
-
Once the data source has been configured, you can preview data. Select the Preview tab and use settings similar to the following to preview data:
-
Click OK to finish creating the data source.
Reading data from client application
- Firstly, to get data from ODBC data source based on ZappySys ODBC driver, in your client application, you would need to connect to ODBC source and then from the list select the data source.
-
Finally, to read the data just read tables/views in your app or enter a SQL statement to extract data, e.g.:
SELECT * FROM [$parent.tableReference.tableId$]
Google BigQuery Connector Examples
The ZappySys API Driver is a user-friendly interface designed to facilitate the seamless integration of various applications with the Google BigQuery API. With its intuitive design and robust functionality, the ZappySys API Driver simplifies the process of configuring specific API endpoints to efficiently read or write data from Google BigQuery.
Click here to find more Google BigQuery Connector examples designed for seamless integration with the ZappySys API ODBC Driver under ODBC Data Source (36/64) or ZappySys Data Gateway, enhancing your ability to connect and interact with Prebuilt Connectors effectively.
Consume Data inside your App / Programming Language
Once you know how to load data from Google BigQuery Connector, you can click on one of the below links to learn the steps how to consume data inside your App / Programming Language from Google BigQuery Connector.ODBC inside ETL / Reporting / BI Tools
- Power BI
- Tableau
- Excel
- Informatica PowerCenter
- QlikView
- SQL Server / Linked Server
- SSRS (SQL Server Reporting Services)
ODBC inside Programming Languages
Key features of the ZappySys API Driver include:
The API ODBC driver facilitates the reading and writing of data from numerous popular online services (refer to the complete list here) using familiar SQL language without learning complexity of REST API calls. The driver allows querying nested structure and output as a flat table. You can also create your own ODBC / Data Gateway API connector file and use it with this driver.
Intuitive Configuration: The interface is designed to be user-friendly, enabling users to easily set up the specific API endpoints within Google BigQuery without requiring extensive technical expertise or programming knowledge.
Customizable Endpoint Setup: Users can conveniently configure the API endpoint settings, including the HTTP request method, endpoint URL, and any necessary parameters, to precisely target the desired data within Google BigQuery.
Data Manipulation Capabilities: The ZappySys API Driver allows for seamless data retrieval and writing, enabling users to fetch data from Google BigQuery and perform various data manipulation operations as needed, all through an intuitive and straightforward interface.
Secure Authentication Integration: The driver provides secure authentication integration, allowing users to securely connect to the Google BigQuery API by inputting the necessary authentication credentials, such as API tokens or other authentication keys.
Error Handling Support: The interface is equipped with comprehensive error handling support, ensuring that any errors or exceptions encountered during the data retrieval or writing process are efficiently managed and appropriately communicated to users for prompt resolution.
Data Visualization and Reporting: The ZappySys API Driver facilitates the seamless processing and presentation of the retrieved data from Google BigQuery, enabling users to generate comprehensive reports and visualizations for further analysis and decision-making purposes.
Overall, the ZappySys API Driver serves as a powerful tool for streamlining the integration of applications with Google BigQuery, providing users with a convenient and efficient way to access and manage data, all through a user-friendly and intuitive interface.