Gmail Connector for Talend Studio

Read / search Gmail messages, download attachments, send mail and more using Gmail REST API.

In this article you will learn how to quickly and efficiently integrate Gmail data in Talend Studio without coding. We will use high-performance Gmail Connector to easily connect to Gmail and then access the data inside Talend Studio.

Let's follow the steps below to see how we can accomplish that!

Download Documentation

Gmail Connector for Talend Studio is based on ZappySys API Driver which is part of ODBC PowerPack. It is a collection of high-performance ODBC drivers that enable you to integrate data in SQL Server, SSIS, a programming language, or any other ODBC-compatible application. ODBC PowerPack supports various file formats, sources and destinations, including REST/SOAP API, SFTP/FTP, storage services, and plain files, to mention a few.

Create Data Source in ZappySys Data Gateway based on API Driver

Download and install ODBC PowerPack.
Search for gateway in start menu and Open ZappySys Data Gateway:
Go to Users Tab to add our first Gateway user. Click Add; we will give it a name tdsuser and enter password you like to give. Check Admin option and click OK to save. We will use these details later when we create linked server:
Now we are ready to add a data source. Click Add, give data source a name (Copy this name somewhere, we will need it later) and then select Native - ZappySys API Driver. Finally, click OK. And it will create the Data Set for it and open the ZS driver UI.

GmailDSN
When the Configuration window appears give your data source a name if you haven't done that already, then select "Gmail" from the list of Popular Connectors. If "Gmail" is not present in the list, then click "Search Online" and download it. Then set the path to the location where you downloaded it. Finally, click Continue >> to proceed with configuring the DSN:

GmailDSN

Gmail

Now it's time to configure the Connection Manager. Select Authentication Type, e.g. Token Authentication. Then select API Base URL (in most cases, the default one is the right one). More info is available in the Authentication section.

User Account

Gmail authentication

To make this work you will have to create a Google Cloud project. This project forms the basis for creating, enabling, and using all Google Cloud services, including managing APIs, enabling billing, adding and removing collaborators, and managing permissions.

Go to the Google Cloud Console
In the Google Cloud console, go to menu IAM & Admin, and then Create a Project.
Enter the name of the new project into the Project Name field and select the appropriate organization, then click Create.
The console navigates to the Dashboard page and your project is created within a few minutes.
In the Google Cloud console, go to menu More products, Google Workspace, and then Product Library.
Now we need to enable the Gmail API.
Search for Gmail API. Select it and then click ENABLE.
Select the Credentials tab on the left side of the screen.
Select the + CREATE CREDENTIALS link to reveal a drop down menu.
In the dropdown menu, select OAuth client ID.
If presented, click the CONFIGURE CONSENT SCREEN button and then elect the appropriate User Type based on your business needs. Otherwise, continue to step 16.
In the OAuth consent screen tab fill in your app's website and domain information, and then click SAVE AND CONTINUE.
In the Scopes section of the app registration, click ADD OR REMOVE SCOPES and add the following scopes:
- openid
- https://mail.google.com/
- https://www.googleapis.com/auth/userinfo.email
- https://www.googleapis.com/auth/userinfo.profile
- https://www.googleapis.com/auth/gmail.labels
- https://www.googleapis.com/auth/gmail.send
- https://www.googleapis.com/auth/gmail.modify
- https://www.googleapis.com/auth/gmail.compose
- https://www.googleapis.com/auth/gmail.readonly
After the scopes have been added, click SAVE AND CONTINUE.
Complete the wizard setting up scopes.
Select the Credentials tab on the left side of the screen.
Select the + CREATE CREDENTIALS link to reveal a drop down menu.
In the dropdown menu, select OAuth client ID.
In the Application type drop down, select Desktop application and then give your app client a name.
Select the CREATE button.
A popup window will appear that provides important information you should record for your records:
- Client ID
- Client secret
And then click OK.
Go to OAuth Consent Screen tab. Under Publishing Status click PUBLISH APP to ensure your refresh token doesnt expire often. If you planning to use App for Private use then do not have to worry about Verification Status after Publish.
Back to Connection UI, Enter the client ID and client secret into the required parameters.

API Connection Manager configuration

Just perform these simple steps to finish authentication configuration:

Set Authentication Type to User Account [OAuth]
Optional step. Modify API Base URL if needed (in most cases default will work).
Fill in all the required parameters and set optional parameters if needed.
Press Generate Token button to generate the tokens.
Finally, hit OK button:

GmailDSN

Gmail

User Account [OAuth]

https://gmail.googleapis.com/

Required Parameters
UseCustomApp	Fill-in the parameter...
Authorization URL	Fill-in the parameter...
Token URL	Fill-in the parameter...
Default API Version	Fill-in the parameter...
Default User Id	Fill-in the parameter...
Scope	Fill-in the parameter...
Optional Parameters
Client Id
Client Secret
RetryMode	RetryWhenStatusCodeMatch
RetryStatusCodeList	429
RetryCountMax	5
RetryMultiplyWaitTime	True
Redirect URL (Only for Web App)

Once the data source connection has been configured, it's time to configure the SQL query. Select the Preview tab and then click Query Builder button to configure the SQL query:

ZappySys API Driver - Gmail

Read / search Gmail messages, download attachments, send mail and more using Gmail REST API.

GmailDSN
Start by selecting the Table or Endpoint you are interested in and then configure the parameters. This will generate a query that we will use in Talend Studio to retrieve data from Gmail. Hit OK button to use this query in the next step.
```
SELECT * FROM Messages
```
Some parameters configured in this window will be passed to the Gmail API, e.g. filtering parameters. It means that filtering will be done on the server side (instead of the client side), enabling you to get only the meaningful data much faster.
Now hit Preview Data button to preview the data using the generated SQL query. If you are satisfied with the result, use this query in Talend Studio:
ZappySys API Driver - Gmail

Read / search Gmail messages, download attachments, send mail and more using Gmail REST API.

GmailDSN
```
SELECT * FROM Messages
```
You can also access data quickly from the tables dropdown by selecting <Select table>.

A WHERE clause, LIMIT keyword will be performed on the client side, meaning that the whole result set will be retrieved from the Gmail API first, and only then the filtering will be applied to the data. If possible, it is recommended to use parameters in Query Builder to filter the data on the server side (in Gmail servers).
Click OK to finish creating the data source.
Most Important Step: Now, After creating or modifying a data source in ZappySys Data Gateway, make sure to click the SAVE button to preserve your changes.
Then, RESTART the Data Gateway service to ensure all changes are properly applied.

Skipping either step may result in the new settings not taking effect.

Read Gmail data in Talend Studio

To read Gmail data in Talend Studio, we'll need to complete several steps. Let's get through them all right away!

This article is compatible with Talend Open Studio (a free version, currently retired by Qlik). If you don't have it, you can still purchase a shareware version of Talend Studio from Qlik.

Create connection for input

First of all, open Talend Studio
Create a new connection:
Select Microsoft SQL Server connection:
Name your connection:
Fill-in connection parameters and then click Test connection:

GmailDSN
If the List of modules not installed for this operation window shows up, then download and install all of them:
Review and accept all additional module license agreements during the process
Finally, you should see a successful connection test result at the end:

Add input

Once we have a connection to ZappySys Data Gateway created, we can proceed by creating a job:
Simply drag and drop ZappySys Data Gateway connection onto the job:
Then create an input based on ZappySys Data Gateway connection:
Continue by configuring a SQL query and click Guess schema button:
Finish by configuring the schema, for example: