This tutorial walks you through the process of extracting blog posts and comments data from Blogger using the REST API and Zappysys SSIS connectors. In this guide, you will learn how to authenticate with the Blogger REST API, configure a Zappysys REST data source, and load extracted data into SQL Server. This approach eliminates the need for custom code and provides a graphical interface for managing complex API integrations.

Zappysys provides a powerful REST connector that simplifies integrating cloud-based APIs into your SQL Server Integration Services packages. We will demonstrate a practical ETL pattern using ZappySys REST connectivity to extract data from Blogger and load it into SQL Server within SSIS.

Prerequisites

Before you begin, make sure you have the following:

• First, you will need to have SSIS installed
• Secondly, make sure to have SSDT
• Thirdly, access to a Blogger blog with appropriate permissions to read public posts and comments.
• Finally, do not forget to install ZappySys SSIS PowerPack.

Steps

Create/Select a Project in the Google API Console

1. Navigate to the Google API Console.
2. Click on the Project Dropdown at the top bar and either select an existing project or create a new one by clicking CREATE PROJECT.
   

   Google Console – Create or select a project

3. Once the project is set, click ENABLE APIS AND SERVICES.
4. Search for the Blogger API v3 and click ENABLE.
   

   Google Console – Enable Blogger API

5. Return to the main screen and click on the OAuth Consent Screen tab.
6. Select the External user type and click CREATE.
7. Provide the application name, user support email, and developer contact information, then click SAVE AND CONTINUE.
8. Click through the remaining scopes and summary pages (no additional scopes are required for this tutorial), and click SAVE AND CONTINUE on each.
9. Move to the Credentials tab.
10. Click CREATE CREDENTIALS in the top bar, choose OAuth Client ID, select Desktop App as the Application Type, and click Create to obtain your Client ID and Secret.
    

    Google Console – Create credentials

11. Download the credentials JSON file and save it in a secure location on your development machine.

Create the OAuth connection for Blogger

1. Create or select your SSIS project, and open the package.dtsx file.
2. Drag and drop the ZS REST API task from the Toolbox.
3. Inside Connection Manager, right-click and select New connection…
4. Inside the SSIS connectors, select the ZS-OAuth
   
   

   Create a new SSIS OAuth API Connection Manager

5. Select the Google API provider
6. Enter the Client ID and Client Secret you generated in the past step
7. Use Blogger URL scope: https://www.googleapis.com/auth/blogger
8. Generate the token
9. Test the connection to ensure it is working correctly.
10. Click OK to save the connection.
    

    OAuth connection – Blogger API

Get information from a Blog

1. Create a string variable {{User::BlogID}} and enter the ID from the blog you want to get information from.
2. From the SSIS Toolbox, drag the ZS REST API Task onto the Control Flow designer.
3. Double-click the task to open the editor.
4. Select the connection manager you created in the previous step.
5. In the URL Path field, enter: https://www.googleapis.com/blogger/v3/blogs/{{User::BlogID}}/posts
6. Set the HTTP method to GET.
7. Click the Preview tab to test the API connection and view sample data.
8. If the preview is successful, click OK to configure the task.
   

   REST API task – Blogger example

Save the information in a SQL table

1. Drag and drop a Data Flow into the Control Flow and double-click it.
   

   Drag and drop Data flow.

2. In the Data Flow, drag and drop the JSON source, then double-click it to configure it.
   

   SSIS JSON Source – Drag and Drop

3. Use the same configuration as in the REST API task, but first enable the Use credentials option.
4. Enter the URL, select the OAuth connection, and use the Method GET.
5. You can use the filter option to get the object you want to get data from: $.Items[*]
6. Preview the data and click OK to save the configuration.
   

   JSON Source – Blogger Example Request
7. Now drag and drop a destination component. In our case, we will use the Upsert destination component.
   

   SSIS – Data Flow – Drag and Drop Upsert Destination Component

8. Double-click in the Upsert destination component to configure it.
9. Select the Action you want to use.
10. Create or select a database connector.
11. Create or select a table.
12. Use the options Insert or/and Update.
13. Map the column; if the columns have the same name, it will be automatic.
14. Select at least one column for a primary key.
15. Click OK to save the configuration.
    

    Upsert destination – Blogger example

16. Run the package, and you will see the data from Blogger in your database
    

    Blogger – result

Code Examples

Sample REST API Request

Here’s an example of retrieving blog posts using the Blogger REST API:

GET https://www.googleapis.com/blogger/v3/blogs/{blogId}/posts
Authorization: Bearer {access_token}
Content-Type: application/json

Example response:

{
  "kind": "blogger#posts",
  "items": [
    {
      "kind": "blogger#post",
      "id": "7109703471293209437",
      "title": "Getting started with Blogger API",
      "content": "Learn how to integrate Blogger with your applications...",
      "published": "2026-03-15T10:30:00Z",
      "updated": "2026-03-20T14:15:00Z",
      "url": "http://example.blogspot.com/2026/03/getting-started.html",
      "author": {
        "displayName": "John Doe"
      }
    }
  ]
}

SQL Table Structure

Recommended table structure for storing Blogger posts data:

CREATE TABLE BloggerPosts (
    PostID BIGINT PRIMARY KEY,
    BlogID BIGINT NOT NULL,
    Title NVARCHAR(MAX),
    Content NVARCHAR(MAX),
    AuthorName NVARCHAR(255),
    PublishedDate DATETIME,
    UpdatedDate DATETIME,
    PostURL NVARCHAR(MAX),
    CreatedDate DATETIME DEFAULT GETDATE(),
    ModifiedDate DATETIME DEFAULT GETDATE()
);

Conclusion

You have successfully learned how to use Zappysys SSIS connectors to extract data from the Blogger
platform using the REST API. With ZappySys SSIS PowerPack, connecting Blogger to your SQL Server becomes
straightforward and reliable. By configuring the REST connector for Blogger and using the native SSIS destination,
you can build a robust ETL pipeline for extracting blog data.

Explore the full capabilities of the ZappySys SSIS PowerPack to optimize your data integration
tasks. By applying this pattern, you can extend your solution to include additional transformations, data
validation, error handling, and scheduling through SQL Server Agent.

Need Help?

If you encounter any issues, our support team is here to help:

1. Live Chat — Use the chat widget (bottom-right corner of this page)
2. Email — support@zappysys.com
3. Support Center — Visit the ZappySys Support Portal

The post SSIS tutorial: How to connect Blogger with SSIS appeared first on ZappySys Blog.

In this article, you’ll learn how to connect SSIS PowerPack (ZappySys) to AliPay OpenAPI to call REST endpoints from SSIS. We’ll focus on the practical integration pattern AliPay requires: sending requests to the OpenAPI gateway with the correct parameters and a valid RSA2 signature, then parsing JSON responses in SSIS.

AliPay OpenAPI basics (what SSIS must send)

AliPay OpenAPI calls typically go through a single gateway endpoint:

• Sandbox gateway:
  

  https://openapi-sandbox.dl.alipaydev.com/gateway.do

A request is built from:

• Standard parameters (for example: app_id, method, charset, sign_type, timestamp, version)
• biz_content (usually JSON)
• sign (a signature generated from the parameters using RSA/RSA2—AliPay recommends RSA2)

AliPay requires you to construct a “string to sign” from parameters and attach sign and sign_type to the request, then validate/verify responses using the AliPay public key.

High-level integration pattern in SSIS

Because AliPay signing is mandatory, the most reliable SSIS approach is a two-step pattern:

1. Generate the signature (RSA2) for the request parameters.
2. Call the AliPay gateway from ZappySys (REST API Task or REST API Source), passing the signed parameters.

ZappySys components support custom HTTP methods, headers, request bodies, and advanced options—so they work well once you have the signed payload ready.

Steps

Start in the AliPay Sandbox

AliPay strongly recommends using the sandbox for development and debugging. In the sandbox, you must:

• An AliPay Developer Account
• An AliPay App ID
• Use the sandbox gateway URL
• RSA2 Private Key (generated in AliPay developer console)
• AliPay Public Key (for verification)
• Access to the AliPay Sandbox environment (recommended for testing)

Decide where you will generate the RSA2 signature

You have three common options:

• Option A (Recommended): SSIS Script Task (C#) to generate signature
  Use a Script Task before the ZappySys call to build parameters and compute sign.
• Option B: Use an AliPay SDK in a small helper app
  Run it via Execute Process Task to output signed parameters (JSON/text), then pass them to ZappySys.
• Option C: Use a signing microservice
  SSIS calls your internal service to get sign, then calls AliPay.

AliPay’s documentation emphasizes the signature process and verification using the AliPay public key.

Example: What a signed AliPay request looks like

AliPay’s gateway model is “parameters + signature.” Conceptually, your final request includes:

• app_id
• method (the API name you’re calling)
• charset (often utf-8)
• sign_type (recommended RSA2)
• timestamp
• version
• biz_content (JSON)
• sign (computed over the canonicalized parameter string)

AliPay documents that sign and sign_type are appended after creating the string to be signed, and verification uses the AliPay public key.

Configure the ZappySys REST API Task (Control Flow)

Use this when operating (e.g., creating a payment, refunding, or querying).

1. Drag ZappySys REST API Web Service Task into Control Flow.
2. Set the task:
   • Method: POST (commonly used with AliPay gateway)
   • URL:
     

     https://openapi-sandbox.dl.alipaydev.com/gateway.do

   • Headers: use
     

     Content-Type: application/x-www-form-urlencoded

   • Body: form-encoded parameters; use this format: key1=value1&key2=value2…

     app_id=YOUR_APP_ID&method=alipay.trade.query&charset=utf-8&sign_type=RSA2&timestamp=2026-02-20 10:00:00&version=1.0&biz_content={"out_trade_no":"202602200001"}&sign=GENERATED_SIGNATURE

3. Click Test Request/Response to confirm you get a valid JSON response.
   

   Alipay REST API

Troubleshooting common AliPay + SSIS issues

• invalid-signature / missing-signature: most often caused by parameter ordering, incorrect URL-encoding, wrong key pair, or using sandbox keys against production (or vice versa).
• Wrong gateway: sandbox and production are fully isolated—ensure your app_id,keys, and gateway host match the environment.
• Encoding issues: confirm UTF-8 everywhere and ensure parameters are encoded exactly once before submission.
• Parsing JSON: use ZappySys JSON Source preview and JSONPath filters to flatten nested objects.

Security best practices

• Store private keys securely (SSIS variables protected, encrypted config, or secret vaults).
• Do not hardcode secrets in packages.
• Use a sandbox for development, then switch to production only after full validation.

Conclusion

Download SSIS PowerPack today and see for yourself how easy it is to connect SSIS PowerPack (ZappySys) to AliPay is absolutely achievable once you treat AliPay’s OpenAPI as a signed gateway integration. The key is generating a correct RSA2 signature for every request, then using ZappySys REST components to send the parameters and parse responses.

Still need help?

If the issue persists, please get in touch with our support team:

• Live Chat: Open the chat widget (bottom right of this page)
• Email: support@zappysys.com
• Support Center: Support | ZappySys

The post SSIS tutorial: How to connect to AliPay appeared first on ZappySys Blog.

In this article, we’ll learn how to integrate a documented API in Swagger with SQL Server Integration Services (SSIS) using ZappySys PowerPack.

Swagger enables us to visualize and test REST endpoints, while ZappySys SSIS provides advanced components for consuming APIs and transforming data in ETL processes.

Steps

Get the Swagger endpoints

1. Review the server’s endpoint list. In this example, we will explore the Pet category.
   

   Swagger – Pet category

2. From the list, we can see multiple endpoints and HTTP methods, including GET, POST, PUT, and DELETE.
3. Click on an endpoint to expand it. For this example, we will use Add a new pet to the store.
   
   

   Swagger – Create a new pet

4. Click Try it out. Swagger will allow you to edit the request body and choose the content type (JSON or XML).
5. Click Execute. Swagger will display
   • The generated request in cURL format
   • The Request URL
   • The Response body
   • The HTTP status code
6. In the Response JSON, we can see the new pet with status code 200, indicating success.
7. Now, with the CURL syntax, we can use that inside our tool.
   

   Swagger – Create a new pet CURL

POST request: Create Pet Request

1. Open Visual Studio and create a new Integration Services Project.
2. After the project is created, locate the SSIS Toolbox on the left panel.
3. Drag and drop the REST API Task onto the Control Flow surface and double-click it to configure
   

   Drag and Drop REST API Task

4. Use the cURL generated from Swagger. For reference, you can review our article about converting cURL to an SSIS API Task.
   

   curl -X 'POST' \
     'https://petstore.swagger.io/v2/pet' \
     -H 'accept: application/json' \
     -H 'Content-Type: application/json' \
     -d '{
     "id": 123,
     "category": {
       "id": 456,
       "name": "Friendly"
     },
     "name": "Zappy",
     "photoUrls": [
       "string"
     ],
     "tags": [
       {
         "id": 0,
         "name": "tech"
       }
     ],
     "status": "available"
   }'

5. -X POST represents the HTTP method. Set the Request Method to POST.
6. Enter the URL from the cURL command.
7. Each -H represents a Header. Add the corresponding headers inside the REST API Task.
8. The -d parameter represents the Body. Paste the JSON payload into the Body section.
9. Click Test Request/Response to validate the configuration.
   

   SSIS – Create a new pet

10. You can use additional components such as JSON Source or XML Source, depending on how you plan to process the response.

POST request: Upload a file

1. Next, we will upload an image for the pet we created earlier
   

   Swagger – upload image

2. Generate the CURL code.
   

   curl -X 'POST' \
     'https://petstore.swagger.io/v2/pet/123/uploadImage' \
     -H 'accept: application/json' \
     -H 'Content-Type: multipart/form-data' \
     -F 'file=@new.jpg;type=image/jpeg'

3. Configure the REST API Task with:
   • Request Method: POST
   • URL
   • Headers
   • File parameter
4. Check the option File upload.
5. Specify the full file path (e.g., file=@D:\Zappysys\new.jpg).
6. Click Test Request to validate the upload.
7. You can find an article about uploading files here.
   

   SSIS – upload an image

GET request: Get the Pet information

1.  Now we will get the pet we created earlier. This is the example
   

   Swagger – get pet by id

2. We generate the CURL
   

   curl -X 'GET' \
     'https://petstore.swagger.io/v2/pet/123' \
     -H 'accept: application/json'

3. Inside the Visual Studio project, drag and drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow designer.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

4. From the SSIS toolbox, drag and drop a JSON Source onto the data flow designer surface, and double-click on it to edit it
   

   SSIS JSON Source – Drag and Drop

5. We use the method GET
6. We enter the URL with the ID from the earlier response.
7. We can use the filter to get the object we need, and now we can check the data.
8. Preview the data, and you will see the Pet’s information
   

   SSIS – get a pet by ID

PUT request: Update an existing pet

1. We go to the updated pet example and run the example
   

   Swagger – update a pet

2. Generate the CURL
   

   curl -X 'PUT' \
     'https://petstore.swagger.io/v2/pet' \
     -H 'accept: application/json' \
     -H 'Content-Type: application/json' \
     -d '{
     "id": 0,
     "category": {
       "id": 0,
       "name": "string"
     },
     "name": "doggie2",
     "photoUrls": [
       "string"
     ],
     "tags": [
       {
         "id": 0,
         "name": "string"
       }
     ],
     "status": "available"
   }'

3. Inside the JSON source, we configure the HTTP method to PUT.
4. Change the url and the header.
5. Inside the Body we can change the values from the existing pet.
6. Test the connection and check the result.
   

   SSIS – update a pet

DELETE request: Remove an existing pet

1. Now we will delete the pet using this example.
   

   Swagger – delete a pet

2. Generate the CURL to delete the pet
   

   curl -X 'DELETE' \
     'https://petstore.swagger.io/v2/pet/1234' \
     -H 'accept: application/json' \
     -H 'api_key: special-key'

3. Inside the JSON source, configure the HTTP method to DELETE.
4. Change the URL to include the pet’s ID.
5. Inside the header, we need to insert an API key.
6. Test the connection, and we will delete the pet.
   

   aSSIS – delete a pet

Authentication method

1. Some APIs require authentication. You can configure authentication using an HTTP Connection Manager.
2. Enter the login URL if required.
3. Select the credential type (e.g., Basic – Username/Password).
4. Enter the credentials.
5. Click OK to save the configuration.
6. This connection can be reused across multiple REST API Tasks or JSON Sources.
7. For more details on basic authentication, click here. For more details on the OAuth connection, click here.
   

   Swagger – SSIS basic authentication

Conclusion

Integrating Swagger-documented APIs with ZappySys SSIS becomes significantly easier when using the generated cURL command as a reference. Swagger provides a clear and interactive way to understand API endpoints, request structures, headers, and payload formats, while cURL offers a direct, executable representation of the HTTP request.

By translating the cURL command into the REST API Task or JSON Source configuration, developers can confidently replicate API calls inside SSIS without guesswork. This approach ensures accuracy, reduces troubleshooting time, and provides a consistent workflow for handling GET, POST, PUT, DELETE operations, file uploads, and authentication scenarios.

Download SSIS PowerPack today and see for yourself how easy it is to combine Swagger and ZappySys SSIS, enabling organizations to seamlessly integrate modern REST APIs into enterprise ETL processes and bridge the gap between cloud-based services and traditional SQL Server data platforms.

Still need help?

If the issue persists, please get in touch with our support team:

• Live Chat: Open the chat widget (bottom right of this page)
• Email: support@zappysys.com
• Support Center: Support | ZappySys

The post SSIS tutorial: How to connect to Swagger appeared first on ZappySys Blog.

In our previous post we saw how to read API data in SSIS. This post we will cover how to call NetSuite RESTlet to read / write data in SSIS or ODBC Apps. For demo purpose we will use SSIS PowerPack REST API Task but concepts and many UI elements remain same for ODBC JSON Driver too. This article assumes you know how to write Netsuite RESTlet and you have obtained Credentials to call RESTlet from extranal app.

What is NetSuite RESTlets

NetSuite added a really nice feature to expose business data and workflows as API by writing Javascript functions which they call it RESTlets. You can secure and customize these restlets as per your need.

You can automate CRUD (Create /Read / Write / Delete) operations in NetSuite by writing special functions in JavaScript and expose those functions as API URL which can be called by any external Systems (i.e. SSIS Packages , ODBC Apps like Power BI, Informatica, SSRS).

RESTlet can be secured few ways such as OAuth 1.0 and OAuth 2.0 authentication. We will cover both approach briefly in this article. You can choose any one as per your need

Creating NetSuite RESTlet / Obtain OAuth Credentials (Client ID and Secret)

Below video shows how to obtain necessary credentials to call Netsuites API (RESTlet).

At mark 4:30 seconds it will show you How to use OAuth 2.0 for calling RESTlet

Create NetSuite RESTlet Connection

There are few ways to authenticate NetSuite RESTlet. If you are using ZappySys Products then it makes it super easy to use either approach. Lets look at both approaches.

Using OAuth 2.0 (Recommended)

As per above video if you obtained Client ID and Secret for OAuth 2.0 you can follow below steps to configure OAuth connection to call RESTlet.

What you need to use OAuth 2.0.

• Your Account ID (it goes in URL)
• Your Client ID and Secret (Obtained from Netsuite Portal) – See above video Mark 04:30
• Register Redirect URL in NetSuite portal (we recommend you use https://zappysys.com/oauth as call back Url for Automatic Code extraction)

Steps to configure OAuth Connection

1. Right click on Connection Panel and choose Choose “New Connection..”. Select ZS-OAUTH type
   
   
   

   Create new SSIS OAuth API Connection Manager

    

    

2.  Enter following things
   ————————-
   On General Tab
   ————————-
   ClientID and Secret: <get it from NetSuite>
   Authorization URL: https://<accountID>.app.netsuite.com/app/login/oauth2/authorize.nl
   Token URL: https://<accountID>.suitetalk.api.netsuite.com/services/rest/auth/oauth2/v1/token
   Scope: restlet
   

   Configure NetSuite OAuth 2.0 Connection for API call

    

   ————————-
   On Advanced Tab
   ————————-
   Callback / Redirect URL:  https://zappysys.com/oauth  (this must be entered same way in NetSuite Portal)

   

   OAuth2 Connection – Callback / Redirect URL

3. Now go back to General Tab and click Generate Token button
4. Follow login screen and try to finish the flow. At the end it will generate Tokens and populate on UI
5. Click OK to save

Using OAuth 1.0

Another approach is use OAuth 1.0 (Older version). If you have enabled OAuth 1.0 in NetSuite then follow below steps (Create connection is same as before).

1. Setup General Tab and Advanced Tab as below.
   On ==== General Tab====:
   Enter ClientID, ClientSecret, AccessToken, AccessToken Secret
   On ==== Advanced Tab ====:
   Check Do Not Include Empty Key/Value Pairs
   Select Signature Method as HMAC-SHA256
   For Extra Token Attribute enter realm=<your-account-number>_SB1
2. Click OK to Save

Call NetSuite RESTlet in SSIS

1. Create a new SSIS Package
2. Drag REST API Task from SSIS Toolbox
3. Double click the Task to configure
4. Select URL from Connection
5. From Connection dropdown select OAuth connection we created in the earlier section (either OAuth 1 or OAuth 2)
6. Enter the URL as below. Assuming you copied URL from NetSuite Replace script ID and account number
   

   https://<account-number>-sb1.restlets.api.netsuite.com/app/site/hosting/restlet.nl?script=<scriptid>&deploy=1

   Enter <account-number>
   Enter <scriptid>
7. Change Method to POST if your script supports Parameters (Else you can supply id=xxxx and recordtype=xxxxx in the URL for GET request)
8. If its POST request then chnage ContentType to ApplicationJSON (application/json)
9. Enter your Input in the body. In below example our RESTlet takes sql query as input so we enter just one parameter named query but you can have different parameter(s)
   

   Syntax:  { param1_name : <value> , param2_name : <value> ....  }

10. Now click Test see everything works
    

    Calling NetSuite RESTlet in SSIS (Using REST API Task)

Conclusion

We saw how easy it is to call NetSuite RESTlets or other NetSuite API in SSIS / other BI Apps using ODBC Drivers provided by ZappySys. Download SSIS PowerPack and try your self see how easy it is to consume virtually any API in SSIS.

The post How to call NetSuite RESTlet API in SSIS / ODBC appeared first on ZappySys Blog.

OneDrive allows to store files on the cloud and and we have seen growing requirements in downloading/Uploading files from there. This post will provide step by step details of how to Upload/Download files from OneDrive.

Register Microsoft App for OAuth Authentication

In this section, you will learn how to register a custom app in Microsoft Azure portal that will allow access to the OneDrive. So, let’s get started –

1. Log into Microsoft Azure portal to register a custom app.
2. Register a new application by clicking New Registration link.
   

   New App Registration in Azure portal

3. Provide the name of the custom app and who can access the app in the organization.
   

   Register an OneDrive App

4. Go to the App overview and add a Redirect URL.
   

   Add a Redirect URL

5. Click on “Add a Platform” under Platform Configuration section and then select “Web” under Web applications section to enter a Redirect URL.
   

   https://zappysys.com/oauth

   

   Redirect URL

6. Create a Client Secret key which will be used to Authenticate the custom Azure app.
   

   Add a Client Secret

   

   Secret Key Expiration Period

   

   Specify Secret Key

   Note – Take a note of Client Secret, it will be required while configuring OAuth connection in the SSIS later.
7. Add API Permissions for OneDrive API.
   

   OneDrive API Permissions

8. Select the following permissions from the Delegated Permissions section.
   

   User.ReadBasic.All
   Files.Read
   offline_access

   

   Select Delegated Permissions

9. Take note of Client ID, it will be required while configuring OAuth connection in the SSIS later.
   

   Client ID

Get OneDrive File ID

In this section, you will learn how to get the File ID of the CSV file through OneDrive API. So, let’s get started –

1. Go to the Microsoft Graph Explorer.
2. Sign in to Graph Explorer.
   

   SSIS GraphExplorer SignIn

3. Run below OneDrive API to get a list of the files from the OneDrive.
   

   https://graph.microsoft.com/v1.0/me/drive/root/children

   

   Microsoft Graph Explorer

   NOTE: Take a note of File ID of the CSV file. For this example, we will download invoices.csv file.

Create OAuth Connection in SSIS

In this section, you will learn how to create an OAuth connection in SSIS to an Azure custom app. So, let’s get started –

1. Create a new ZS-OAuth connection in SSIS.
   

   ZappySys OAuth Connection in SSIS

2. Configure OAuth connection with following settings.

   Client ID	Provide Client ID from App Registration steps
   Client Secret	Provide Client Secret from App Registration steps
   Authorization URL	https://login.microsoftonline.com/organizations/oauth2/v2.0/authorize
   Access Token URL	https://login.microsoftonline.com/organizations/oauth2/v2.0/token
   Scope/Permissions	User.ReadBasic.All Files.Read.All offline_access

   

   OAuth Connection Configuration

3. Go to the Advanced tab and provide Return URL.
   

   https://zappysys.com/oauth

   

   Add Redirect URL

4. Click on Generate Token will give a login screen which will allow to generate access and refresh token for the app.
5. Approve the requested permissions bu Accepting it.
   

   Approve Permissions Requested

6. Token has been generate successfully.
   

   Refresh Token Successful

7. Let’s test the connection to make sure it is working fine.
   

   Connection Successful

Download CSV File Through OneDrive REST API in SSIS

In this section, you will learn how to download a CSV file through OneDrive API in SSIS and save it into a CSV file.

1. Here is the OneDrive API which allows to read content of the file.
   

   https://graph.microsoft.com/v1.0/me/drive/Items/<File ID>/content

2. Replace <File ID> in above step#1 with the file ID that we have got in the Get OneDrive File ID section. Final URL will look something like below.
   

   https://graph.microsoft.com/v1.0/me/drive/Items/01W7L3VCUBS27ILUQKENAK4LQGGTJ2R3TT/content

3. Let’s create a SSIS package with Data Flow Task from SSIS Toolbox.
4. Replace <File ID> in above step#1 with the file ID that we have got in the Get OneDrive File ID section. Final URL will look something like below.
   

   https://graph.microsoft.com/v1.0/me/drive/Items/01W7L3VCWMCGPYZLM2IRAYWROSH4XMQT7A/content

5. Configure REST API task as follows.
   

   Binary File Configuration in ZappySys REST API Task

6. Configure the responses setting as below.
   

   Binary File Configuration in ZappySys REST API Task

   NOTE: Ensure that the Treat response as the binary checkbox is checked otherwise the file will be downloaded but corrupted as zip/mp3 etc. file format is binary.
7. Check if the request is made correctly by clicking Test Request/Response.
   

   Example of Testing the Request

8. Let’s have a look at the folder where the file has been downloaded. Here is an example –
   

   Downloaded File Example

Download a Text File Through OneDrive API in SSIS

Here is another technique that can be used to download a CSV/text file through OneDrive API without knowing it’s file ID but file name. Please check this link for more details.

Download Excel Sheets Through OneDrive API in SSIS

JSON Source also supports reading API response which returns 2D arrays. Such as Office Excel Spreadsheet API to read excel data. Check this blog post to learn a similar scenario to parse 2D arrays. There are few other options for parsing 2D arrays which are documented here.

https://graph.microsoft.com/v1.0/me/drive/items/01CYZLFJDYxxxxxxx/workbook/worksheets('test')/range(address='A1:B200')

Upload a File Through OneDrive API in SSIS

In this section, you will learn how to upload a file through OneDrive API in SSIS. Uploading a file has few additional steps and has been documented separately. Please check this link for more details.

Delete a File Through OneDrive API in SSIS

In this section, you will learn how to delete a file from OneDrive through OneDrive APIs in SSIS.

1. Add a REST API task and configure as follows. Make sure to use the File ID of the file which needs to be deleted from OneDrive. Example URL –
   

   https://graph.microsoft.com/v1.0/me/drive/Items/01W7L3VCSEADQ457W54RBI3I47YGFJOD2I

   

   Delete OneDrive File

2. Execute the task and you will have file deleted from OneDrive.

Conclusion

In this article, we have learned how to Upload/Download/Delete data/file from OneDrive API in SSIS. We used SSIS REST API Task to extract data through OneDrive API in SSIS. Download SSIS PowerPack to try many other automation scenarios not discussed in this article.

The post How to call OneDrive API in SSIS (Upload, Download) appeared first on ZappySys Blog.

In our previous article, we see how to load data into Elastic Search using SSIS. Now let’s look at how to read data from ElasticSearch using SSIS and load response into SQL Server.

Elasticsearch is a powerful engine that allows you to store, aggregate, and, most importantly, search data in a very analytical way. In this tutorial, you will learn how to load Elasticsearch data to SQL Server with SSIS (part of SQL Server) and ZappySys PowerPack. We also see how to set the maximum result window of the index using the Rest API Task.

In a nutshell, this post will focus on how to Make Elasticsearch Search API call using SSIS.

Step-By-Step to Call Elasticsearch API call using SSIS

Read from Elasticsearch Search API call

Let’s make the Elasticsearch Search API call using the JSON source to get all the records of the index with pagination.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Dragging and dropping Data Flow Task into Control Flow

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to make Search API Call on the index.
   

   http://{elasticsearch-site-url}/{index}/{index-type}/_search

4. Now check Use credentials and Select Connection section press

   <New ZS-HTTP Connection>

   .
   

   SSIS JSON Source – Passing Basic Credentials (Supply UserID / Password) using HTTP Connection

5. Once the HTTP Connection Manager window opens configure the connection to your Elasticsearch instance:
   • Set Web Url, which points to your Elasticsearch instance.
   • Set Credentials Type to

     Basic - UserID/Password

      (or other appropriate authentication methods).
   • Finally, set User Name and Password:
     
     

     Configure SSIS HTTP Connection to connect to Elasticsearch

6. Now, select method as POST and set for and size value in the body to get data using pagination. Set the content type as JSON (application/json).
   

   {"from" : <%page%>, "size" : 100}

7. Furthermore, go to the pagination tab and configure pagination like the below screenshot makes sure we need to enter increment size the same as given in body size value therefor 100.
   

   Pagination Mode: POST data in the body

8. That’s it now select the desired filter and click on the Preview button.
9. That’s it we are ready to load the data in the SQL Server.

For more than 10000 rows

When you try to get more than 100000 rows you will get the next error message

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [11000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
            }
        ]
    },
    "status": 400
}

Here is how to configure your JSON Source to pull more than 10K rows using Scroll Method.

1. In Settings Tab: In the Body and URL rather than Index name use

   [$tag$]

2. In the filter type:

   $.hits.hits[*]._source

3. Uncheck Include Parent
   

   Elasticsearch example for more than 10000 rows

4. In Pagination Tab, Set Next Link as below

   $._scroll_id

5. Stop Indicator Attribute as below

   $.hits.hits[0]._id

6. Stop Indicator Value as

   regex=^$

   

   Elasticsearch pagination tab configuration

7. In Advanced Pagination Tab, Check Has Different Page Info and Enable Page Token
8. Enter Page Place Holders as below (change YOUR_INDEX_NAME)
   

   url=YOUR_INDEX_NAME/_search?scroll=10m|_search/scroll

9. First Page Body as below – Change query as needed else keep it match all to fetch all rows from Index
   

   { "size":1000, "query": { "match_all": { }  } }

10. Next Page Body as below 

    {  "scroll": "5m", "scroll_id": "[$pagetoken$]" }

    

    Configuration for Elasticsearch advanced pagination tab

Loading Elasticsearch API data into SQL Server / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Enable Deep Pagination Setting

By default index, the maximum result window size is 10,000 means you can paginate upto 10000 rows. If you ever want to get more than 10000 result then you have to perform the following steps to change default setting from 10000 to 500000 (that’s max).

We need to send PUT request to below URL:

http://{elasticsearch-site-url}/{your-index}/_settings

Need to pass below body as I am setting max result window size value as 500000.

{ "index" : { "max_result_window" : 500000 } }

And set content-type as JSON (application/json) and click on Test Request/Response and it will make the API call and set the max window size.

Rest API Task PUT Call

That’s it we have successfully made the call to set the maximum result window size of the index.

Common Errors

Deployment to Production

Conclusion

After all, we saw you How to Make Elasticsearch Search API call using SSIS JSON Source and load response into SQL Server. In this article, we have seen that how to make a setting call to set the maximum result window of the index. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File)
• Elasticsearch Search APIs

The post How to read data from ElasticSearch using SSIS appeared first on ZappySys Blog.

In our previous few posts we saw how to call various google apis in SSIS.  In this post lets learn how to call Google Search Console API in SSIS or other ODBC Compatible Apps such as Power BI, Informatica, SSRS using API Drivers for ODBC

About Google Search Console API (Google Webmaster API)

If you are new to Google Search Console API (i.e. Webmaster API) then start from this link. You can also check API Explorer here to test API requests.

Search API can be called by supplying either API key in URL  or use OAuth

Search Console V3 Covers following APIs (May add more in future)

Search Console API Call Example

Here is simple example of Search Console API call. Below example returns all sites you added under your search console account.

Request

GET https://www.googleapis.com/webmasters/v3/sites?key={YOUR_API_KEY}

Response

{
 "siteEntry": [
  {
   "siteUrl": "https://my-google-search-api-1.com/",
   "permissionLevel": "siteOwner"
  },
  {
   "siteUrl": "https://www.my-google-search-api-1.com/",
   "permissionLevel": "siteOwner"
  },
  {
   "siteUrl": "https://my-google-search-api-2.com/",
   "permissionLevel": "siteOwner"
  },
  {
   "siteUrl": "https://www.my-google-search-api-2.com/",
   "permissionLevel": "siteOwner"
  }
 ]
}

Step-By-Step – Call Search Console API in SSIS

Now lets look at very simple API call to list search console sites

1. Open SSIS Package
2. Drag and drop ZS REST API Task from SSIS tool box
   

   Drag and Drop ZS REST API Task from SSIS Toolbox

3. Now double click REST API Task and configure (Skip Step 5 if you use API Key instead of OAuth).
4. Enter API URL you like to call. For This example we will use simple URL as below
   OAuth based Authentication
   

   https://www.googleapis.com/webmasters/v3/sites/

   API Key based Authentication
   

   https://www.googleapis.com/webmasters/v3/sites/?key={YOUR_API_KEY}

5. If you want to use OAuth based credentials then perform the following steps
   1. Select URL from Connection 
   2. Click New ZS-OAUTH connection from Dropdown
   3. On the OAuth UI select Google as Provider
   4. Enter Scopes as below
      

      https://www.googleapis.com/auth/webmasters
      https://www.googleapis.com/auth/webmasters.readonly

   5. Click Generate Token. When you get Login Prompt enter your Google Account information and click Accept
   6. Click OK to Save OAuth UI
6. This is how it will look like if you use OAuth Connection
   

   Call Google Search Console API using SSIS REST API Task (OAuth Authentication)

7. Now click Test Request / Response to check if its working. You should get response like below

Read from Search Console Analytics API (query)

Now let’s look at how to call Search Console Analytics API (i.e. query) for Specified Site and extract data in tabular format using ZS JSON Source. In below example we will obtain search Impression, clicks, position etc for specified site. We will aggregate this over date and country (dimensions).

In this section you will learn how to use JSON Source Adapter to extract data from JSON file (In this case its Web URL).

1. Firstly, You need to Download and Install SSIS ZappySys PowerPack.
2. Once you finished first step, Open Visual Studio and Create New SSIS Package Project.
3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
4. Double click on the Data Flow task to see Data Flow designer surface.
5. From the SSIS toolbox drag and drop JSON Source on the Data Flow designer surface.
   
6. Double click JSON Source and configure like below
   1. Enter URL as below. Notice two things, first we used <<somedata,FUN_URLENC>> placeholder function to encode : and // in the URL so it become %2F %3A. We use

      --dont-escape--

        suffix at the end.
      For OAuth based Credentials
      

      https://www.googleapis.com/webmasters/v3/sites/<<https://mysite.com,FUN_URLENC>>/searchAnalytics/query--dont-escape--

      For API Key based Credentials
      

      https://www.googleapis.com/webmasters/v3/sites/<<https://mysite.com,FUN_URLENC>>/searchAnalytics/query/?key={YOUR_API_KEY}--dont-escape--

   2. Check Use Credentials and select OAuth connection (created earlier) (if you are using API key in URL then skip this step).
   3. Select POST as HTTP Method
   4. Enter Body for Query. Here are some examples for various queries you can use. For our case we want to group by date and country so use below
      

      {
      "startDate": "2019-01-01",
      "endDate": "2019-01-10",
      "dimensions" : ["date","country"]
      }

   5. Select Content Type as

      application/json

        from Content Type drop down.
   6. Select Filter or enter as 

      $.rows[*]

   7. Now go to Extract Multiple Arrays Tab and Check Array Flattening Option (on older version it was on the different Tab)
7. Click Preview and you will see your Google Search Console API data as below.
   

   Read from Google Search API Console (Analytics Query Endpoint) – JSON Source

Loading Google Search Console API data into SQL Server / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Import Google Search Console API data in Reporting / other ETL tools (ODBC Usecase)

There will be a time when you dont want to use SSIS connector like mentioned in previous sections but extract Google Search API data in Reporting Tools / Other ETL Platform.  Good news is you can use ZappySys ODBC Driver for JSON / REST API in ODBC Apps. See few popular ODBC Apps below.

• Power BI connection for Google Search Console API
• Excel connection for Google Search Console API
• Informatica connection for Google Search Console API
• MS Access connection for Google Search Console API
• C# , PowerShell, VB.net connection for Google Search Console API

Preview / Generate Query in JSON Driver / XML Driver

Load Google Search Console API in SQL Server without any ETL

Now let’s look at even more interesting integration scenario. If you have SQL Server and like to load Google Search Console API without any ETL then you can use Data Gateway

Read this article to learn how to load REST API in SQL Server without any coding (Use just T-SQL) .

You can write query like below to load data inside SQL Table.

SELECT * FROM OPENQUERY([MY_LINKED_SERVER]
, 'SELECT * FROM $
WITH(
	 Src=''https://www.googleapis.com/webmasters/v3/sites/<<https://mysite.com,FUN_URLENC>>/searchAnalytics/query/?key={YOUR_API_KEY}--dont-escape--''
	,Filter=''$.rows[*]''
	,RequestData=''{
"startDate": "2019-01-01",
"endDate": "2019-01-10",
"dimensions" : ["date","country"]
}''
	,RequestContentTypeCode=''ApplicationJson''
	,Header=''cache-control: no-cache || Accept: */*''
	,RequestMethod=''POST''
	,EnableArrayFlattening=''True''
)')

SSMS Output – Query REST API via Linked Server OPENQUERY statement (Connect to ZappySys Data Gateway)

Conclusion

So in this post we saw how easy it is to achieve total REST API integration in SSIS using JSON Connector. We also saw how to use ODBC Drivers to query REST API data inside Apps like Excel, MS Access, Informatica and other ODBC Apps using ODBC JSON / REST API driver. If you are SSIS User download SSIS PowerPack and try for FREE and if you don’t have SSIS in house and want to try more generic approach then Download ODBC PowerPack Drivers for FREE Trial.

The post Get data from Google Search Console API in SSIS and ODBC Apps appeared first on ZappySys Blog.

In this article, you will learn how to refresh a Power BI dataset with Power BI REST API using SSIS and ZappySys SSIS PowerPack. Power BI is a Microsoft reporting product that can get data from virtually any source and display it nicely in a report or a dashboard. Each Power BI report contains a dataset, which can be refreshed to show relevant data in the report. Power BI also has a Scheduled refresh capability, in case you want to refresh a dataset periodically. However, you are limited to only 8 specific times per day you can refresh a dataset (as of May 23, 2019). In the Power BI Premium license you can do it 48 times per day. But what if you need to refresh it more frequently? Or if you desire to refresh it at a non-predefined time; which is not known in advance? E.g. after an ETL process has finished? Here Power BI REST API comes into rescue and lets you refresh a dataset on demand.

Here are a few use cases when you may want to refresh a dataset from a SSIS package, using Power BI REST API:

• The need to refresh a dataset more often than Power BI allows.
• The requirement to refresh a dataset just after some process is finished; e.g. after an ETL process.
• The ability to have absolute control when to refresh multiple datasets. Sometimes, refreshing multiple huge datasets at once could cause memory problems in the Power BI service. So, having an orchestrated refresh would solve this problem.

We will use these ZappySys SSIS PowerPack connectors/connections in this article:

OAuth Connection Manager

REST API Task

Let’s begin!

Prerequisites

1. You have a Power BI account and you have basic Power BI skills.
2. An uploaded Power BI report to Power BI service.
3. A Microsoft Azure subscription (there is a free option with a 200 USD credit).
4. Azure Active Directory set up.
5. SSIS designer installed. Sometimes it is referred to as SSDT or BIDS (download it from the Microsoft site).
6. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
7. ZappySys SSIS PowerPack is installed (download it).

Getting started

Suppose you have a simple report and you want to refresh its dataset using Power BI REST API and SSIS. So far, the dataset has not been refreshed at all:

Preparing to refresh the Power BI dataset using SSIS and Power BI REST API.

To start, create an SSIS package and follow the steps below to accomplish the goal.

Step By Step – How to refresh a Power BI dataset with REST API using SSIS

Create an OAuth application

Create OAuth Connection Manager in SSIS package

1. Drag and drop REST API Task from SSIS Toolbox, double click, select Url from connection, and then select <New ZS-OAUTH Connection> item:
   

   Creating OAuth Connection Manager to connect to Power BI OAuth Application from SSIS.

2. Input Client ID from Step 2 when we were creating an Azure Active Directory OAuth Application.
   

   Configuring OAuth Connection Manager to authenticate to Power BI OAuth Application using SSIS.

3. Then proceed and input the Client Secret from Step 3, Create an Azure Active Directory OAuth application section.
4. Please go on, and fill in Authorization Url and Access Token Url properties with values we copied in Step 4, Create an Azure Active Directory OAuth application section.
5. In Scopes / Permissions field input these scopes:
   https://analysis.windows.net/powerbi/api/Dataset.ReadWrite.All
offline_access
6. Then go to the Advanced tab and in the Callback/Return Url field input this value:
   https://login.microsoftonline.com/common/oauth2/nativeclient
   

   Setting Callback/Return Url (Redirect URI) in OAuth Connection Manager to authenticate to Power BI OAuth Application using SSIS.

7. Then go back to the tab General and click the Generate Token button.
8. Once asked for credentials, log in with your organization user; this will generate Refresh and Access tokens.

Retrieve the ID of the Power BI dataset you want to refresh

1. Go to Power BI datasets.
2. Then click on Datasets.
3. Proceed and click the dataset you want to refresh.
4. Finally, copy the ID of the dataset from the URL, we will use it in the next section.

Getting Power BI dataset ID to refresh it in an HTTP request to Power BI REST API.

Configure REST API Task to refresh a Power BI dataset

We configured OAuth Connection Manager and retrieved the ID of the dataset we want to refresh, now it’s time to configure REST API Task and refresh the dataset. So do the following:

1. Go to the SSIS package and open the REST API Task for configuration.
2. Input URL in such a format:
   https://api.powerbi.com/v1.0/myorg/datasets/DATASET-ID/refreshes Replace DATASET-ID with an actual ID of the dataset. E.g.:
   https://api.powerbi.com/v1.0/myorg/datasets/a38c78fc-8c87-45e3-b961-6dcd3ac1eada/refreshes
   NOTE: If your dataset resides in a Workspace, use this format instead:
   https://api.powerbi.com/v1.0/WORKSPACE-ID/datasets/DATASET-ID/refreshes
3. Then set POST as HTTP Request Method.
4. In the Body field, input this JSON:
   { "notifyOption": "MailOnFailure" }
5. Press the Test Request/Response button to test the request and refresh the dataset at the same time.

Configuring a REST API Task to refresh a Power BI dataset using Power BI REST API and SSIS.

Run the package

Hit F5 and execute the package:

Executing SSIS package and refreshing Power BI dataset.

The Results

Now it’s time to check the results and see if the dataset was refreshed successfully:

1. Again, go to Power BI datasets.
2. Then click on Datasets.
3. Click the dataset you refreshed.
4. Then click “Refresh history” to check the refresh history. We expect two manual refreshes to be there.

Results: Successful Power BI dataset refresh using Power BI REST API and SSIS.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Deployment to Production

Conclusion

You learned how to refresh a Power BI dataset with REST API using SSIS. To accomplish the goal, firstly, we created the Azure Active Directory application and configured OAuth Connection Manager based on it. Then we retrieved the Power BI dataset ID and used it in the ZappySys SSIS PowerPack connector REST API Task. Finally, using the same REST API Task we made the HTTP request to Power BI REST API “refreshes” method and successfully refreshed the dataset. If you need to make a call to another Power BI REST API method, then just configure the REST API Task according to the documentation (for getting data use JSON Source or Web API Destination instead). Finally, if you make many requests to the Power BI REST API service, make sure you check its limitations.

References

https://powerbi.microsoft.com/en-us/

https://docs.microsoft.com/en-us/rest/api/power-bi/datasets/refreshdataset

https://docs.microsoft.com/en-us/rest/api/power-bi/

https://docs.microsoft.com/en-us/power-bi/developer/api-rest-api-limitations

The post How to refresh Power BI dataset with REST API using SSIS appeared first on ZappySys Blog.

Connect to YouTube API in SSIS (OAuth Connection)

Very first step to call any Google API is to create an API project for Google Service and then enable APIs you like to call in Console. In our case we need to enable YouTube API.

Check this article to learn how to create API Project and obtain Client ID and Client Secret to connect to YouTube API. For ease of use use Default Application provided by ZappySys but we still recommend using your own Custom Application  on OAuth Connection.

Before we try our first YouTube API example, we recommend you to get familiar with YouTube API here. Once you done with above steps you can perform the following steps to call YouTube API in SSIS.

1. Open existing SSIS Project or Create new One in Visual Studio (i.e. SSDT)
2. Create a new SSIS Package. Right click in the Connection Manager Panel and click “New Connection…”.
   
3. When Dialog box opens select ZS-OAUTH connection type.
   

   SSIS Connection Managers List: Select O-AUTH Connection Manager

4. Once OAuth UI opens, Configure Google connection like below.
   1. Select Service Type as Google
   2. You can select Default App for ease of use or Select Custom Application (Enter Client Id and Secret Obtained using these steps )
   3. In the Scope Enter below – One scope per line for calling correct YouTube API. Refer to YouTube API documentation incase you need extra permissions. For read only operation we just need below scopes. Last scope is only needed if you need to use YouTube Analytics Reports
      

      https://www.googleapis.com/auth/youtube
      https://www.googleapis.com/auth/youtube.readonly
      https://www.googleapis.com/auth/yt-analytics.readonly

5. Click Generate Token to obtain Refresh Token (You may see Login Prompt, and then Grant Permission Confirmation Screen)
   

   Connect to Google API using OAuth 2.0 (YouTube API Example)

6. Click OK to save UI.

Call YouTube REST API Example

Now lets look at how to call YouTube API REST API Task this task is more suitable to call GET, POST, DELETE, PUT requests without parsing data in Rows and Columns. If you need to parse data and load into Database table then see next section (Use JSON Source)

1. From SSIS Toolbox look for items starting with “ZS”. Drag and Drop [ZS Rest API Task] to Designer Surface.
   
2. Double click it to configure like below.
   1. Select URL From Connection option
   2. Select OAuth connection created in previous section.
   3. Enter URL like below (For example get all playlists). Replace xxxxxxxxxxx with your own channel ID (Its usually found in URL when you visit Youtube Channel Homepage)
      

      https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&channelId=xxxxxxxxxxxxx

   4. Click Test Request Button
3. Here is how it looks like
   

   Call Youtube API in SSIS Example (Dynamic URL)

That’s it we have successfully configured Connection for YouTube API in SSIS. In the next section we will see how to use this connection and read various data from YouTube API.

Read YouTube Playlists in SSIS

Once we done creating OAuth Connection Manager we can move forward to read YouTube data inside Data Flow.

Configure SSIS JSON / REST API Source

1. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
2. Double click on the Data Flow task to see Data Flow designer surface.
3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   
4. Double click JSON Source and enter URL as below (Use variable or Hardcode Channel Id). maxResults controls how many rows you want to get in each response. Rep
   

   https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&maxResults=50&channelId={{User::varChannelId}}
   --OR-- 
   https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&maxResults=50&channelId=xxxxxxxxxxxxxxxxxxxxxx

5. Check Use Credentials and select existing OAuth connection or Create New one
6. Select Array Filter (Items node) or type

   $.items[*]

     as below.
   

   Get Playlists for YouTube Channel

7. That’s it you can now Click Preview to see sample data. In the next section we will configure Pagination to read all records if you have more than 100 rows.

Configure YouTube REST API Pagination

Most REST APIs dont return data in one big response. So you have to make sure you implement looping / pagination to read more records. Luckily ZappySys supports many different Pagination Settings. Here is how to configure YouTube REST API Pagination.

1. Go to Pagination Tab found on JSON Source and select following 3 settings
2. Select Pagination Mode as

   Response Attribute Mode

3. Enter Next Link as

   $.nextPageToken

4. Enter Suffix for Next URL as

   &pageToken=<%nextlink%>

YouTube REST API Pagination Settings

Read YouTube Playlist Videos in SSIS

So in previous section we saw simple API call to read all playlists from YouTube. Now lets see how to get information about Playlist Videos.

For that everything should remain same as previous section except change URL as below. Where xxxxxxxxxxxxxxxxxxxxx is your Playlist ID obtained from Previous Call.

https://www.googleapis.com/youtube/v3/playlistItems?part=snippet,id&maxResults=50&playlistId=xxxxxxxxxxxxxxxxxxxxx

Get all Videos for YouTube Channel

If you want to get all video name and ID for specific channel then use below URL. Click here to read more. Replace xxxxxxxxxxxxxxxxxxx with your Channel ID.

https://www.googleapis.com/youtube/v3/search?channelId=xxxxxxxxxxxxxxxxxxx&part=snippet,id&order=date&maxResults=50

Read YouTube Video Views, Likes, Dislikes (Extended Videos)

Another common request from YouTube API is get Video Information such as Views, Likes, Dislikes, Comment Count etc. For that you can use this api . However the issue is this endpoint requires you to pass Comma separated Ids of Video. You can only pass 25 Ids in one requests so if you have 200 videos for which you need to get information then you have to make 8 requests (25 Ids in each Request). In below screenshot you can see we used Fiddler to debug web requests.

So here is how high level process to fetch extended information for all Videos in your channel. Full SSIS Package is attached here (SSIS 2012 Format).

1. Get all videos for your channel by calling (Assuming you will paginate in below request.
   

   https://www.googleapis.com/youtube/v3/search?channelId=xxxxxxxxxxxxxxxxxxx&part=snippet,id&order=date&maxResults=50

2. Build Video Id groups for 25 Ids from above result and submit requests. For example if above request returns 200 Videos then your request to fetch extended information may look like below.
   

   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID1,ID2,ID3........ID25
   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID26,ID27,ID28........ID50
   ........
   ........
   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID176,ID177,ID178........ID200

3. Parse response coming from each Request Above…. Each Response will give you details about 25 videos you requesting.

Here is the flow of our complete data flow to read extended information for all videos. You can read extended information one by one too but it wont be good idea if you have many videos.

We have used Following Components to achieve Bulk Mode

• JSON Source (REST API Source)
• CSV Parser Transform
• Web API Destination
• JSON Parser Transform

Get YouTube Video View Clicks, Likes, Dislikes (Bulk Mode for all videos)

Read data from YouTube Analytics / Reporting API (Metrics, Dimensions Report)

So far we have seen basic YouTube APIs to extract your Channel information. However there is another very powerful API endpoint to query custom reports from YouTube. Check this YouTube Analytics API endpoint. In this section we will learn how to extract useful information from YouTube using Analytics API.

Step-By-Step

Let’s look at an example to read total views, estimated watch time, average view duration and some other metrics by date for channel you have access.

1. Go to Data Flow, Drag ZS JSON Source from SSIS Toolbox
2. Double click to configure.
3. Check Use Credentials. Select OAuth connection we created in previous section.
4. In the URL field, Enter Below Sample URL. Change Parameters as per your need or leave it default.
   For example:
   ids=channel==MINE   (where MINE can be replaced by other channel id or keep it as MINE to get data for your own channel)
   metrics (can be changed from any valid list here)
   dimensions (can be changed to any valid list from here)
   startDate (Must be YYYY-MM-DD format)
   endDate (Must be YYYY-MM-DD format)
   

   https://youtubeanalytics.googleapis.com/v2/reports?ids=channel==MINE&metrics=views,estimatedMinutesWatched,averageViewDuration,averageViewPercentage,subscribersGained&dimensions=day&startDate=2019-03-01&endDate=2019-04-01

5. Change Filter to

   $.rows[*]

6. Go to 2D Array Transform tab.
   Select Method as

   Simple 2-dimensional array

   EnterColumn Filter as 

   $.columnHeaders[*].name

   

   YouTube Analytics API Advanced Transformation

7. That’s it now Preview your data. See below over all steps.
   

   Read data from YouTube Analytics API (Custom Reports using Dimensions, Metrics Query)

Build YouTube Analytics / Reporting API Query (API Playground)

With Analytics API You can extract many interesting data. You can use Test Tool found on this page to build some interesting queries. Scroll to Common Use Cases section on that link. On the right side you can select Credentials Type as OAuth and click Execute.

https://developers.google.com/youtube/analytics/reference/reports/query

YouTube Analytics / Reporting API Explorer (Testing API Calls)

Loading YouTube API Data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Download Sample SSIS Package

SSIS Package is attached here (SSIS 2012 Format).

Conclusion

In this article we saw how to extract information from REST API such as YouTube REST API using OAuth 2.0. We also learned techniques like Pagination, JSON Parsing, Request Batching using multiple components to achieve full scale API integration. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

The post How to read YouTube API data in SSIS (Videos, Channels, Playlists) appeared first on ZappySys Blog.

In this post we will learn how to read Freshdesk data in SSIS using ZappySys JSON / REST API Source. We will see how to authenticate, paginate / parse and load Freshdesk data into SQL Server Table in just few clicks. If you don’t use Freshdesk and considering using Zendesk instead then check our previous post on how to Read Zendesk data in SSIS.

Freshdesk is a cloud based / managed service like Zendesk to for Support / Customer service.

Getting Started

So let’s get started. We will see step by step instructions to configure Freshdesk API connection, setup JSON Source to read data from Freshdesk and finally load data into SQL Server table. We will use List Tickets API in this article for demo purpose but read other useful information on that page if you are new to REST API.

Setup Freshdesk REST API Connection

Very first step to call any Freshdesk API in SSIS is to configure HTTP connection (Basic Authentication – Using UserID/ Password).

1. Open Visual Studio (i.e. SSDT) to create new SSIS Package
2. From Control Flow SSIS Toolbox drag REST API task
3. Select URL From HTTP Connection Option from Drop down
4. Click New HTTP Connection
5. Configure HTTP connection like below.
6. Enter any Valid URL (This will be ignored if you Override URL on REST API Task)
7. Select Basic Authentication (Use User ID and Password)
   

   Configure Freshdesk REST API Connection in SSIS – Basic Authentication (User ID / Password)

8. Click OK to close HTTP Connection
9. On Main UI of REST API Task, check Use Direct URL and enter below URL (Change YourDomain Part)
   https://{your-domain}.freshdesk.com/api/v2/tickets
10. Click Test Request / Response. If you see OK status then you have successfully called Freshdesk API. That’s all.
    

    Calling Freshdesk REST API in SSIS ( Read Tickets example)

In the section we will see how to Parse REST API response and load data into SQL Table.

Read Freshdesk data in SSIS

Now let’s look at steps to configure SSIS JSON Source to read from Freshdesk REST API.

1. Drag Data Flow task from toolbox. Double click to go to Data Flow designer
2. Drag ZS JSON Source (For File / REST API)
3. Configure JSON Source like below.
4. Check Use Connection option and select HTTP connection created in the previous section.
5. Enter API URL from which you like to read data. In out example we want to read tickets created after 2019-01-01 so we will call API like below. You can skip ?updated_since=2019-01-01 if you like to read all tickets.
   

   https://{your-domain}.freshdesk.com/api/v2/tickets?updated_since=2019-01-01

6. You can also Use Variable Placeholder in URL to make things dynamic like below
   

   https://{your-domain}.freshdesk.com/api/v2/tickets?updated_since={{User::MyDateVar,yyyy-MM-dd}}

7. Click Preview to see sample data.
   

   Read Freshdesk data in SSIS – Fetch Tickets example (With REST API pagination)

8. Configure Pagination like below. Basically you have to select Pagination Mode = URL Parameter Mode. Enter Page num indicator as 

   page

   

   Configure REST API pagination (URL Parameter Mode)

9. Thats it. Now click OK to save UI.

Load Freshdesk data into SQL Server Table

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

In this article we saw how easy it is to read data from any REST API service like Freshdesk REST API and load into Relational DB like SQL Server. Try SSIS PowerPack for FREE and explore many other 70+ Tasks /Components not discussed in this article.

The post Read Freshdesk data in SSIS – REST API Call appeared first on ZappySys Blog.