In this article, you will learn how to register Facebook OAuth App (Graph API) for data access using SSIS PowerPack REST API Connectors. Using SSIS PowerPack you can consume/manage Facebook data by calling Graph API. These connectors have drag and drop UI so you can do data integration without any coding (e.g. C#, Java, Python, Ruby).

JSON /REST Source Connector and REST API Task they both support a wide range of options for any type of REST API integration scenarios including Facebook REST API integration into your ETL Workflow. It also supports the use of OAuth 1.0 and 2.0 Protocols used by Twitter, Facebook, Google, LinkedIn, and many other popular Apis.

Facebook Graph API can be used to read or manage Facebook data via REST API Calls. To access Graph API you will have to create OAuth App inside Facebook Developer Portal. Let’s take a look step by step.

Step-By-Step: Create Facebook OAuth App (i.e Register OAuth App)

1. Go to www.facebook.com and log in with an account which you want to use to access Facebook data. You can access your own data using API without issue but if you have to access someone else data via App then it will require additional steps listed in this article in “Create Public Facebook OAuth App” section
2. Once you login to Facebook type https://developers.facebook.com/apps in the Address bar
3. Click on Create New App. If you never access developer portal then you may be prompted to Register as a developer.
   

   Create Facebook OAuth App for Graph API access (For SSIS PowerPack REST Connectors)

4. It creates the app by given name and asked you to choose some options like ads and FB login. Click on skip to skip that step.
5. Now go to basic settings and there you can change app name, contact email and select category (e.g. Business). There you can also found OAuth App ID and App Secret store it somewhere for the next phase.
   

   Register Facebook OAuth App for Custom Desktop Application (Basic Setup)

6. No Click on Add Products and add Facebook Login. Because our app is used for SSIS PowerPack which has custom login flow (using Embedded Browser). We have to Add Facebook Login support and check Embedded browser option (see below).
   

   Adding Embedded browser support for Facebook OAuth Application

7. Now you ready to use your App to use OAuth App ID and App Secret.

Test Facebook OAuth App (Consume graph api in SSIS)

If you using SSIS PowerPack to manage your Facebook data using Graph API then perform the following steps to test your connectivity.

1. In the SSIS Designer in connection manager page .. right click and add ZS-OAUTH
2. Select Facebook provider, Configure App ID and OAuth App Secret
3. Select Permission (i.e. Scope) or type by hand (1 per each line)
4. Click generate Token. You will be redirected to login page and then it will show Authorize App confirmation. Click Accept.
5. Test Connection (If things go well it will show green success)
   

   Testing Facebook OAuth Connectivity in SSIS (Use Graph API)

Fetch from Facebook Graph API (Load data into SQL Server)

Below is sample screenshot how you can use JSON Source to pull data from Facebook REST API and load into SQL Server. Refer this article for step-by-step information.

Extract data from Facebook using Graph API (SSIS JSON/REST API Connector) – Load into SQL Server Table

The post How to register Facebook OAuth App for Graph API Access appeared first on ZappySys Blog.

In our previous article we understood little bit about OAuth 2.0 Protocol. We talked how to use Default App for ease of use. However In this post you will learn how to register Google OAuth Application so you can access various Google Services programmatically (e.g Youtube, Google Analytics, AdWords, DoubleClick etc.). Here is the link to find all services: Google API Services (Using OAuth 2.0)

Recently google started supporting OAuth 2.0 Protocol. There are many articles written explaining OAuth 2.0 protocol so we are not going to cover that in this article.

Following SSIS Connectors support OAuth Connectivity. When you use below connectors no coding required to pull data from OAuth enabled services.

SSIS Tasks / Components which Supports OAuth Connection / Google API

	API Source
	Google Analytics Source
	JSON Source Connector (File, REST API)
	REST API Webservice Task
	XML Source (File, REST, SOAP)
	Web API Destination

ODBC Drivers which Supports OAuth Connection / Google API (Call in Power BI, Excel, Informatica, MS Access)

	API ODBC Driver
	JSON ODBC Driver
	XML ODBC Driver
	CSV ODBC Driver

Create Google Project

Registering OAuth App in Google first requires you to create API Project. API Project is a container which can host many OAuth Apps or other API Keys.

1. To create API Project Visit https://console.developers.google.com/project
   If you already have existing API Project then once you click above link you can pick desired API Project from dropdown (Found in the top bar). If you using existing API Project then you may skip next few steps and directly move to step which explains [how to create OAuth ClientId].
2. To create new project Click on “Create Project” button. Once you click Create it will take few seconds to create project for you.
   

   Create New Google API Project (Google API – Register new OAuth App)

3. Now give some friendly name and change default Project ID (optional) for your project and click Create (See below).
   
   NOTE: Before you click create you can change default Project ID suggested by Google by clicking Edit. This Project ID is used everywhere when calling many APIs like this one so copy this ID somewhere (You can also see this on project screen so dont worry if you miss it).Project ID is unique across google system so you may not get your preferred ID but if you can get one then its easy to remember for sure.
   

   Create new Google Api Project for OAuth App (Set project name / edit default project ID)

Enable Google API for Integration

Once your Project is created, next step is enable APIs you like to call. By default most Google APIs are disabled so make sure you enable each APi category manually using below steps.

1. Once project is created it will redirect you to the API Home page where you will see many common Google API you can use.If you do not see that you can always click below option to enable API manually.Visit : https://console.cloud.google.com/apis/library
   

   Enable Google APIs / Services

2. By default API you like to call might not be enabled. Search for API you wish to enable, click on the API name and click Enable.Some common Google API Examples:
   1. To enable Google YouTube API
      > search for “YouTube“. It has few APIs so pick the one you need under this,
      > For example To manage/view channel use Data API, for Reporting use Analytics API)
   2. To enable Google Drive API
      search for “Drive” and click Enable
   3. To enable Google Sheets API
      search for “Sheet” (Some Sheets API operations need Drive API too see enable that as well)
   4. To enable Google BigQuery API
      > search for “BigQuery“
   5. To enable Google Analytics (GA3)
      > search “Analytics“ and enable Google Analytics API
   6. To enable Google Analytics (GA4)
      > search for Google Analytics Data API and click Enable, Search for Google Analytics Admin API and click enable (needed this to query available properties)
      

      Search Google API for OAuth Application Access

       

      

      Enable Google API for OAuth Application Access

       

Create OAuth Credentials (Client ID / Secret)

Once we create project and enable APIs our next step is to obtain OAuth ClientID / Secret. Before you can obtain that you have to configure few properties of your OAuth Consent Screen. Most fields are optional on this screen but fill out them if you plan to publish this API App for other users (i.e. Public App).

1. Now click on the credentials tab and click “OAuth consent screen”. Fill out necessary information which will be visible to the user (such as App Name). Once you are done, continue on with credentials creation:Google OAuth API – Create OAuth App – Set Application Name on Consent Screen
2. Click on [Create credentials] button and select OAuth Client Id option.
   

   Create OAuth Client ID Credentials for Google API access

3. Next you will be asked to select Application Type. Pick “Desktop app” if your Application is Desktop style application. If your app is web site or running browser then pick Web Application. Click on Create and you will see your ClientID and ClientSecret which you can use during OAuth Authentication. Keep these both information secret. This is used to call OAuth Approval Screen by your Desktop App or Web App… and then user can obtain Token which can be used to call desired Google API.
   NOTE: Make sure you do not choose Web App and only select Desktop App. Desktop App doesnt have Redirect URL making it simple to setup and use with ZappySys software.
   

   Create Google OAuth 2.0 Credentials – Desktop App

   

   Generate ClientID and Client Secret for Google OAuth Application

Publish App

The last step in App creation is to go to the OAuth Consent Screen tab. Under Publishing Status, click PUBLISH APP to ensure your refresh token doesn’t expire often. If you plan to use the App for Private use, you do not have to worry about the Verification Status after Publishing (Which is the case for most users who access their own data).

Publish Google App (This avoids Refresh Token Expiration every 7 days)

Using Client ID and Client Secret in ZappySys OAuth Connection

You can use generated ClientID and ClientSecret to initiate OAuth Authorization process. We are not going to cover how that works, but below is a simple screenshot of how these two pieces of information are used to call the Consent screen with some desired permission requested by calling Application:

ZappySys OAuth Connection can be used by many components suggested in the beginning of this article. So refer them for more information.

Google OAuth Application: Generating Access and Refresh tokens using SSIS and OAuth Connection Manager.

Using System Default Browser to get the token

If embedded browser has some issue to extract your token then you can close the window by pressing [X] in the title bar. Once you do that you will be prompted to use System Default browser to finish the token extraction. If you click Yes then Full browser will be launched.

NOTE: Below feature is added in version published after Dec 31, 2019 (v3.0.0.11231 or later) so make sure you have latest version.

Using System Default Browser to extract OAuth Token

Untrusted App Warning

Sometimes Google might show you below warning. In order to continue the flow you might have to Click on Advanced and then click Go to YourAppName (unsafe) like below

How to ignore Google Unsafe App Warning (OAuth Flow)

OR, in some cases, you may see modern UI like below. In either case, choose continue.

Google App not Verified Warning (New UI)

Select the account

Using System Browser

Copy code and finish the flow

Manually Copy Code and Get OAuth Refresh / Access Token

Why I didn’t get refresh token back?

For some reason if you just see Access token and not refresh token then try following

1. Logout current session in browser and close all browser windows and try again
2. If you use Custom Provider rather than Google on OAuth connection then change Auth URL to include prompt and access_type like below
   

   https://accounts.google.com/o/oauth2/auth?access_type=offline&prompt=consent

What to do if Google Refresh Token expires every 7 days

Technically, once you have a refresh token, you should not worry about renewing as long as the API call is made atleast once every few months (max 6 months idle time). However, in some cases, your refresh token can expire sooner, and you have to click Generate Token manually

One main reason that is not well documented is if the Google Oauth Private App you created to access your own data is still in “testing” mode and not “published,” then as per Google, you may get this error. Perform these steps to publish your app, which is in testing mode, so you do not have to renew the refresh token every 7 days.

1. Go to your Project in Google API console
   Generally here
2. No API call has been made using the generated refresh_token for six months or longer
   SOLUTION:  Make at least some API in six months to keep refresh_token active
3. Your OAuth App is in “testing” status and not published yet
   SOLUTION: Just click the Publish button on your App OAuth Consent screen. Do not worry about getting verified if you plan to use it for internal use; after this step, your refresh_token may not expire every 7 days See this thread.DO NOT WORRY ABOUT GETTING VERIFIED – Just publish it to avoid refresh token expiration every 7 days.
   

   Publish Google App (This avoids Refresh Token Expiration every 7 days)

What is Google OAuth AccessToken

AccessToken is short lived token which you can pass along with each API Service call. once this token is expired you can get new AccessToken as long as you have obtained and saved RefreshToken (Returned to caller after very first time user go through Approval process on Consent screen). If you are using ZappySys OAuth connection manager

What is Google OAuth RefreshToken

May OAuth services support short lived token which expires after few minutes or hours. Google API by default expires token after 3600 seconds … So how to get new token ? Do you always go through that annoying Consent screen… what if your application is Automated Service? Well OAuth has solution for that too… its called RefreshToken.

First time to setup OAuth connection user have to go through Approval screen  (i.e. Consent screen) once they click Approve button they may receive two types of token (AccessToken and RefreshToken).  RefreshToken is used only when you need to get new AccessToken without going through Consent screen. So technically if OAuth supports RefreshToken (This is optional) then you can Save this piece of information in database somewhere and when you get token_expired error or you detect expiration time you can get new AccessToken without involving user consent (because you already did once).

Conclusion

Hope this article helped you to understand how to register Google OAuth Application. Download SSIS PowerPack to try various OAuth data integration scenarios using drag and drop high performance connectors. With SSIS PowerPack OAuth support you can virtually pull data from any API service (XML REST API or JSON REST API). Such as Facbook, Twitter, SurveyMonkey, LinkedIn and many more.

The post How to register Google OAuth Application (Get ClientID and ClientSecret) appeared first on ZappySys Blog.

In this article you will how to Read Twitter data in SSIS using SSIS JSON Source and SSIS REST API Web Service Task. You will also learn about latest OAuth 2.0 Protocol to simplify REST API access.

Twitter REST API Authentication

In order to fetch any data from twitter using OAuth REST API calls you have to obtain two Keys (Consumer Key [Like UserID] and Consumer Secret [Like Password]). There are mainly two methods to read data from Twitter. Using Application-user authentication and Application Only authentication. Use correct method depending what type of information you want to pull from twitter. Most common method is Application-user authentication . If you want to read more about other methods then click here

Method 1 – Read Twitter data in SSIS using Application-user authentication

In this method you will need twitter user account to connect Twitter API. Once first time authorization is done you don’t have to re-authenticate. Certain type of API calls only allowed by this method (such as POST new tweet using API). For any API call in Twitter very first step is create OAuth App (i.e. Register new App in Twitter developer portal).

Using Default OAuth App Created by ZappySys

To make your life easy ZappySys provides default App for certain OAuth providers (e.g. Google, Twitter). If you decide to create your own app for whatever reason then check next section on how to register twitter OAuth Application. See below screenshot how to use Twitter OAuth connection using Default App. Once you click Generate Token it will prompt you to login using Twitter account and then you can grant permission.

Step-By-Step

1. Download and Install SSIS PowerPack FREE Trial from this link
2. Create new SSIS Project and add new package.
3. Open Package. Go to control flow. Drag and drop Data Flow task from SSIS Toolbox
   

   Dragging and dropping Data Flow Task into Control Flow

4. Go to Data flow designer.Drag and drop ZS JSON Source Component and double on it to configure it.
   

   Drag and Drop JSON Source Component

5. Enter any twitter API URL you want to call such as below
   

   https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=zappysys&count=4

6. Now check use credentials option
7. Select new ZS-OAuth connection from dropdown
8. When new connection dialogbox pops up select Twitter from providers dropdown. Select Default App option for now.
9. Click Generate Token. It may ask you to login using your Twitter account credentials. If prompted click Approve.
10. If twitter OAuth grant approved then you will see Access Token and Access Token Secret both populated
11. Click Test Connection and if it works then click OK to close connection
12. On JSON Source you can now click Preview to see data from Twitter
13. You can optionally specify Filter expression [Click Select Filter button] to select result from specific JSON Node. For example if you want to select all Hashtags used inside each Tweet then select Filter like $.entities.hashtags[*]
14. Click OK to close the UI
15. Drag ZS Trash Destination from Toolbox (Or you can use OLEDB Destination if you want to load inside SQL Server)
16. Connect JSON Source to ZS Trash Destination
17. Execute SSIS package

Call Twitter REST API in SSIS – using OAuth Connection (Default App)

Using Custom OAuth App Created by you

For some reason if you dont want to use Default Twitter App then you can register Custom App. It requires few extra steps listed in next section but it wont take more than few minutes. In Custom App option you have to specify ClientId and Client Secret (in above screenshot). Once you click Generate Token it will prompt you to login using Twitter account and then you can grant permission.

Register/Create OAuth Application for twitter API Access

Now lets look at how to register OAuth Application for Twitter API access.

Goto https://apps.twitter.com/app/new and create new app by providing necessary information. Don’t get confused by calling it App, its just way in twitter to create multiple API Access Keys so you can grant different level of access to different users. Once app is created you will be taken to page where your Consumer Key and Secret Key will be listed.

Create Twitter Application – REST API Access for Developer

Obtaining Consumer Key and Consumer Secret from Twitter

Once app is created you can go to Keys and Access Tokens tab. Here you will find Consumer Key and Consumer Secret.
Consumer Key can be Public but Consumer Secret must not be shared (think it like a password).

Obtain Twitter API Consumer Key and Consumer Secret – REST API Access for Developer

Method 2 – Read Twitter data in SSIS – Application-only authentication

For some reason if you don’t want to use Twitter user account to access data (e.g. Giving access to consultant so he can access you company Twitter Account via API). In this scenario you have to use Application Only method. In this method you don’t authorize application using Login form.

Here is our basic flow to access twitter data.

1. Get Bearer Access Token by calling https://api.twitter.com/oauth2/token service (Use POST method and Pass BASE64 encoded ConsumerKey and Consumer Secret)
2. Call any twitter service by passing Bearer AccessToken we retrieved in previous step.NOTE: Any access to twitter service is over HTTPS so automatically your tokens passed along request is encrypted before sending over wire unless someone know how to hack SSL 🙂

Application Only Authentication using REST API task – Get Access Token

Once we create SSIS Package – First step to access any twitter data will be get Access Token. As you can see from below screenshot we have called https://api.twitter.com/oauth2/token service URL with POST method. Notice how we have supplied POST data and 2 headers. Authorization header contains BASE64 encoded value of YourConsumerKey:YourConsumerSecret

SSIS Twitter REST API Get Token

Fetch data from Twitter using JSON Source – Deformalize nested JSON

Once we have Authentication Token we are ready pull twitter data using JSON Source. Check below screenshot how we have supplied Token in Authorization Header. JSON Source can make your JSON look like normal table (It also de-normalize nested JSON into flat dataset. If you want to extract subset of JSON then simply specify JSON Path expression e.g. $.data.users[*]

Read JSON response of Twitter REST API in SSIS

Load Twitter JSON data to SQL Server

You can easily connect your SSIS JSON Source to OLEDB Destination if you want to load Twitter Data to some RDBMS such as SQL Server, MySQL.

Handling paging of large REST API result set with twitter data – looping/cursoring

Most of REST API limit total data sent in single response. So if you wish to get all records then you have to loop through multiple results. Twitter provides looping mechanism using CursorClick here to read more

In our case SSIS JSON Source Supports Paging very well so we are covered. To loop through multiple result sets of twitter data simply configure following 3 properties. See below screenshot.

Twitter REST API – Paging Example -Loop through resultset using cursor

Word of caution about too many requests

Twitter does not allow you request too much data too quickly so be careful how many requests you make :). Check their official page on twitter Rate Limit for REST API. You can add delay after each request if you doing pagination. Go to Throttling tab of JSON/XML Source. You can change setting there based on API restriction. For example if API allows only 30 requests per minute then adding 2 seconds delay will make sure you wont exceed 30 requests in 1 minute.

To learn more about rate limit of Twitter API check this table

Handling Twitter API Date Format (Parse Twitter Date/Time)

Twitter returns date in the following format e.g. Fri May 03 15:22:09 +0000 2013 to parse this to correct date/time datatype you can use Date/Time Handling Tab of JSON/XML Source.

Change Custom Date Format as below and preview Twitter data.

Parsing Twitter API date/time format

Download related files

Below is sample SSIS package
Twitter Demo SSIS 2012

Conclusion

So you have now seen how easy it is to access twitter data with OAuth 2.0 using SSIS JSON Source and SSIS REST API Web Service Task. We have also seen how to loop through large resultset using inbuilt Paging support of JSON Source.

The post How to read Twitter data in SSIS using REST API Call appeared first on ZappySys Blog.