Connect to YouTube API in SSIS (OAuth Connection)

Very first step to call any Google API is to create an API project for Google Service and then enable APIs you like to call in Console. In our case we need to enable YouTube API.

Check this article to learn how to create API Project and obtain Client ID and Client Secret to connect to YouTube API. For ease of use use Default Application provided by ZappySys but we still recommend using your own Custom Application  on OAuth Connection.

Before we try our first YouTube API example, we recommend you to get familiar with YouTube API here. Once you done with above steps you can perform the following steps to call YouTube API in SSIS.

1. Open existing SSIS Project or Create new One in Visual Studio (i.e. SSDT)
2. Create a new SSIS Package. Right click in the Connection Manager Panel and click “New Connection…”.
   
3. When Dialog box opens select ZS-OAUTH connection type.
   

   SSIS Connection Managers List: Select O-AUTH Connection Manager

4. Once OAuth UI opens, Configure Google connection like below.
   1. Select Service Type as Google
   2. You can select Default App for ease of use or Select Custom Application (Enter Client Id and Secret Obtained using these steps )
   3. In the Scope Enter below – One scope per line for calling correct YouTube API. Refer to YouTube API documentation incase you need extra permissions. For read only operation we just need below scopes. Last scope is only needed if you need to use YouTube Analytics Reports
      

      https://www.googleapis.com/auth/youtube
      https://www.googleapis.com/auth/youtube.readonly
      https://www.googleapis.com/auth/yt-analytics.readonly

5. Click Generate Token to obtain Refresh Token (You may see Login Prompt, and then Grant Permission Confirmation Screen)
   

   Connect to Google API using OAuth 2.0 (YouTube API Example)

6. Click OK to save UI.

Call YouTube REST API Example

Now lets look at how to call YouTube API REST API Task this task is more suitable to call GET, POST, DELETE, PUT requests without parsing data in Rows and Columns. If you need to parse data and load into Database table then see next section (Use JSON Source)

1. From SSIS Toolbox look for items starting with “ZS”. Drag and Drop [ZS Rest API Task] to Designer Surface.
   
2. Double click it to configure like below.
   1. Select URL From Connection option
   2. Select OAuth connection created in previous section.
   3. Enter URL like below (For example get all playlists). Replace xxxxxxxxxxx with your own channel ID (Its usually found in URL when you visit Youtube Channel Homepage)
      

      https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&channelId=xxxxxxxxxxxxx

   4. Click Test Request Button
3. Here is how it looks like
   

   Call Youtube API in SSIS Example (Dynamic URL)

That’s it we have successfully configured Connection for YouTube API in SSIS. In the next section we will see how to use this connection and read various data from YouTube API.

Read YouTube Playlists in SSIS

Once we done creating OAuth Connection Manager we can move forward to read YouTube data inside Data Flow.

Configure SSIS JSON / REST API Source

1. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
2. Double click on the Data Flow task to see Data Flow designer surface.
3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   
4. Double click JSON Source and enter URL as below (Use variable or Hardcode Channel Id). maxResults controls how many rows you want to get in each response. Rep
   

   https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&maxResults=50&channelId={{User::varChannelId}}
   --OR-- 
   https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&maxResults=50&channelId=xxxxxxxxxxxxxxxxxxxxxx

5. Check Use Credentials and select existing OAuth connection or Create New one
6. Select Array Filter (Items node) or type

   $.items[*]

     as below.
   

   Get Playlists for YouTube Channel

7. That’s it you can now Click Preview to see sample data. In the next section we will configure Pagination to read all records if you have more than 100 rows.

Configure YouTube REST API Pagination

Most REST APIs dont return data in one big response. So you have to make sure you implement looping / pagination to read more records. Luckily ZappySys supports many different Pagination Settings. Here is how to configure YouTube REST API Pagination.

1. Go to Pagination Tab found on JSON Source and select following 3 settings
2. Select Pagination Mode as

   Response Attribute Mode

3. Enter Next Link as

   $.nextPageToken

4. Enter Suffix for Next URL as

   &pageToken=<%nextlink%>

YouTube REST API Pagination Settings

Read YouTube Playlist Videos in SSIS

So in previous section we saw simple API call to read all playlists from YouTube. Now lets see how to get information about Playlist Videos.

For that everything should remain same as previous section except change URL as below. Where xxxxxxxxxxxxxxxxxxxxx is your Playlist ID obtained from Previous Call.

https://www.googleapis.com/youtube/v3/playlistItems?part=snippet,id&maxResults=50&playlistId=xxxxxxxxxxxxxxxxxxxxx

Get all Videos for YouTube Channel

If you want to get all video name and ID for specific channel then use below URL. Click here to read more. Replace xxxxxxxxxxxxxxxxxxx with your Channel ID.

https://www.googleapis.com/youtube/v3/search?channelId=xxxxxxxxxxxxxxxxxxx&part=snippet,id&order=date&maxResults=50

Read YouTube Video Views, Likes, Dislikes (Extended Videos)

Another common request from YouTube API is get Video Information such as Views, Likes, Dislikes, Comment Count etc. For that you can use this api . However the issue is this endpoint requires you to pass Comma separated Ids of Video. You can only pass 25 Ids in one requests so if you have 200 videos for which you need to get information then you have to make 8 requests (25 Ids in each Request). In below screenshot you can see we used Fiddler to debug web requests.

So here is how high level process to fetch extended information for all Videos in your channel. Full SSIS Package is attached here (SSIS 2012 Format).

1. Get all videos for your channel by calling (Assuming you will paginate in below request.
   

   https://www.googleapis.com/youtube/v3/search?channelId=xxxxxxxxxxxxxxxxxxx&part=snippet,id&order=date&maxResults=50

2. Build Video Id groups for 25 Ids from above result and submit requests. For example if above request returns 200 Videos then your request to fetch extended information may look like below.
   

   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID1,ID2,ID3........ID25
   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID26,ID27,ID28........ID50
   ........
   ........
   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID176,ID177,ID178........ID200

3. Parse response coming from each Request Above…. Each Response will give you details about 25 videos you requesting.

Here is the flow of our complete data flow to read extended information for all videos. You can read extended information one by one too but it wont be good idea if you have many videos.

We have used Following Components to achieve Bulk Mode

• JSON Source (REST API Source)
• CSV Parser Transform
• Web API Destination
• JSON Parser Transform

Get YouTube Video View Clicks, Likes, Dislikes (Bulk Mode for all videos)

Read data from YouTube Analytics / Reporting API (Metrics, Dimensions Report)

So far we have seen basic YouTube APIs to extract your Channel information. However there is another very powerful API endpoint to query custom reports from YouTube. Check this YouTube Analytics API endpoint. In this section we will learn how to extract useful information from YouTube using Analytics API.

Step-By-Step

Let’s look at an example to read total views, estimated watch time, average view duration and some other metrics by date for channel you have access.

1. Go to Data Flow, Drag ZS JSON Source from SSIS Toolbox
2. Double click to configure.
3. Check Use Credentials. Select OAuth connection we created in previous section.
4. In the URL field, Enter Below Sample URL. Change Parameters as per your need or leave it default.
   For example:
   ids=channel==MINE   (where MINE can be replaced by other channel id or keep it as MINE to get data for your own channel)
   metrics (can be changed from any valid list here)
   dimensions (can be changed to any valid list from here)
   startDate (Must be YYYY-MM-DD format)
   endDate (Must be YYYY-MM-DD format)
   

   https://youtubeanalytics.googleapis.com/v2/reports?ids=channel==MINE&metrics=views,estimatedMinutesWatched,averageViewDuration,averageViewPercentage,subscribersGained&dimensions=day&startDate=2019-03-01&endDate=2019-04-01

5. Change Filter to

   $.rows[*]

6. Go to 2D Array Transform tab.
   Select Method as

   Simple 2-dimensional array

   EnterColumn Filter as 

   $.columnHeaders[*].name

   

   YouTube Analytics API Advanced Transformation

7. That’s it now Preview your data. See below over all steps.
   

   Read data from YouTube Analytics API (Custom Reports using Dimensions, Metrics Query)

Build YouTube Analytics / Reporting API Query (API Playground)

With Analytics API You can extract many interesting data. You can use Test Tool found on this page to build some interesting queries. Scroll to Common Use Cases section on that link. On the right side you can select Credentials Type as OAuth and click Execute.

https://developers.google.com/youtube/analytics/reference/reports/query

YouTube Analytics / Reporting API Explorer (Testing API Calls)

Loading YouTube API Data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Download Sample SSIS Package

SSIS Package is attached here (SSIS 2012 Format).

Conclusion

In this article we saw how to extract information from REST API such as YouTube REST API using OAuth 2.0. We also learned techniques like Pagination, JSON Parsing, Request Batching using multiple components to achieve full scale API integration. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

The post How to read YouTube API data in SSIS (Videos, Channels, Playlists) appeared first on ZappySys Blog.

In our previous article we saw how to POST data to REST API using few different ways. Now let’s go one step further and discuss another common scenario to batch REST API requests in SSIS. For efficient data transfer many API provides you to submit multiple records in a single request. In this article we will look at few ways to batch our requests using SSIS JSON Generator Transform and  SSIS Web API destination.

Method 1 – Batch records using SSIS JSON / XML Generator Transform

First let’s look at common way to group multiple records and generate single JSON or XML document for desired batch size. We will also see another technique where you can assign Unique BatchID to each document.

For example below screenshot shows how to product one JSON document for every 10 input rows. We have also use BatchID column on the root level. This will use use BatchID from first row of that batch. You can use XML Generator Transform if you like to produce XML Document rather than JSON.

To generate Unique BatchID you can use function like newid() in SQL Server.

SELECT newid() as BatchID,CustomerID,CompanyName FROM Customers

Using SSIS JSON Generator Transform with Batch Option (Multiple Records in a single document)

Here if full flow to batch your JSON Requests and Submit to REST API.

POST data to REST API in Bulk Mode (Batch multiple rows)

Method 2 – Using SSIS Web API destination Batch Mode with Header / Footer

Another method of batching is use Web API destination and turn on Batch mode.  You can use Fiddler to debug your web requests incase you are not sure how its generating and sending POST requests after this setting.

Configure SSIS Web API Destination for Bulk API call (Insert multiple rows)

Method 3 – Batch CSV records using CSV Generator Transform

So far we looked at how to batch JSON and XML format. Now let’s look at scenario where you have to POST CSV formatted data to REST API endpoint.

Assume that you have an API endpoint where you can submit data in multiple batches. Also you have few additional requirements

1. First row of First batch must have column names (any other batch must not contains CSV Header)
2. In each Batch we want to submit maximum 5 rows as per API call
3. Each batch must contain Unique Batch Number in URL

Here is high level flow.

Generate CSV using Batch Setting , Send data to REST API in multiple batches, Debug REST API in Fiddler

Include CSV Header in First Batch Only

Now let’s break down the problem and find out how to do. First problem is we need to include CSV Header only in First batch. To achieve this we use old school trick where we output column names as data row… and when we generate CSV in the next section we will turn off Column Headers option.

If you are consuming records from SQL Server or any other RDBMS you can write following style source query. If you don’t use relational source then use native UNION Transform in SSIS.

select 'CustomerID' as Column1, 'OrderID' as Column2
UNION ALL
select top 12 cast(CustomerID as varchar(1000)), cast(OrderID as varchar(1000)) from Invoices

Batch records and output CSV data

Now next step would be to generate CSV data but in batches… so in our example we have to output size of 5 rows per batch. For this perform the following steps

1. Drag ZS CSV Generator Transform (Included in v2.8 and Higher)
2. Connect source to CSV Generator Transform
3. Now before we edit in UI mode, right click and select Properties. Change following 3 properties  (You can also change QuoteAroundValue and Dimiliter if needed). In future version batch setting will be visible in UI mode too.
   1. ArrayBatchSize  = yourbatchsize (in our case it was just 5 rows per batch)
   2. OutputMode = SingleFileFormat
   3. FirstRowsHasColumnNames  =False
4. Now double click to configure ZS Generator Output
5. Right click on Mappings folder node and Add Elements > Select Multiple Columns  > Check columns you like to output
6. Click OK to Save UI

Generate CSV data in SSIS (Batch Output Setting)

Generate unique URL with Batch using Script Component

Now next task we have to achieve is generate Unique Batch Number. Put that Batch ID in URL like below…

http://httpbin.org/post?batch=0
http://httpbin.org/post?batch=1
http://httpbin.org/post?batch=2
.....

1. For this drag SSIS Script Component and select Transformation Mode
2. Now go to Inputs and Outputs tab > Select Output Columns > Click Add Column > Name it as URL > Configure DataType as DT_WSTR , Length = 500
3. Now got Script Tab and Click Edit. Remove old code and enter following code (Assuming You selected C#). This code will generate new URL with Unique batch ID each time. You can use this same technique to generate Unique row number too (in that case rather than DT_WSTR type you can use DT_I4 type)

   [Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
   public class ScriptMain : UserComponent
   {
       private static int _batch;
       public override void Input0_ProcessInputRow(Input0Buffer Row)
       {
           Row.Url = string.Format("http://httpbin.org/post?batch={0}",_batch);
           //If you have to send GUID rather than Integer for Batch ID then use below code
           //Row.Url = string.Format("http://httpbin.org/post?batch={0}",System.Guid.NewGuid());
   
           _batch = _batch+1;
       }
   }

4. Click OK to Save script.

Output Unique URL or Number using SSIS Script Transform

Configure Web API Destination to POST CSV Data

Now final step is to configure SSIS Web API Destination to POST CSV data to API.

1. Drag ZS Web API Destination and connect Source Transform to Web API destination.
2. Double click to configure
3. Select Connection Manager
4. Once connection setup, select Input Column for URL ( This will be same column from Script Transform)
5. Select column for Body (e.g. ZS_CSV_OUT)
6. Click OK to save

POST data to REST API in multiple batches using SSIS Web API destination

Debug Web API Requests using Fiddler

Now we ready to run entire flow. Once you execute data flow, observe Input rows before CSV Generator Transform and after. CSV Generator transform must produce less rows before we now grouping multiple rows in a single batch.

Another important aspect to play with API request is to debug it, We recommend you to to look at this article about using Fiddler to debug web requests.

Download Sample Package

Here is the Sample SSIS Package (2012 format) – Rest API_Batching

Conclusion

In this post, we saw how easy it is to handle complex REST API  / CSV /JSON / XML operations in SSIS. We configured REST API POST Operation using Batching Method. Hope this post helps you to achieve your API integration without coding.  Feel free to download SSIS PowerPack and try many other components not discussed in this article.

The post How to batch REST API requests in SSIS (Bulk Operation) appeared first on ZappySys Blog.