This tutorial walks you through the process of extracting blog posts and comments data from Blogger using the REST API and Zappysys SSIS connectors. In this guide, you will learn how to authenticate with the Blogger REST API, configure a Zappysys REST data source, and load extracted data into SQL Server. This approach eliminates the need for custom code and provides a graphical interface for managing complex API integrations.

Zappysys provides a powerful REST connector that simplifies integrating cloud-based APIs into your SQL Server Integration Services packages. We will demonstrate a practical ETL pattern using ZappySys REST connectivity to extract data from Blogger and load it into SQL Server within SSIS.

Prerequisites

Before you begin, make sure you have the following:

• First, you will need to have SSIS installed
• Secondly, make sure to have SSDT
• Thirdly, access to a Blogger blog with appropriate permissions to read public posts and comments.
• Finally, do not forget to install ZappySys SSIS PowerPack.

Steps

Create/Select a Project in the Google API Console

1. Navigate to the Google API Console.
2. Click on the Project Dropdown at the top bar and either select an existing project or create a new one by clicking CREATE PROJECT.
   

   Google Console – Create or select a project

3. Once the project is set, click ENABLE APIS AND SERVICES.
4. Search for the Blogger API v3 and click ENABLE.
   

   Google Console – Enable Blogger API

5. Return to the main screen and click on the OAuth Consent Screen tab.
6. Select the External user type and click CREATE.
7. Provide the application name, user support email, and developer contact information, then click SAVE AND CONTINUE.
8. Click through the remaining scopes and summary pages (no additional scopes are required for this tutorial), and click SAVE AND CONTINUE on each.
9. Move to the Credentials tab.
10. Click CREATE CREDENTIALS in the top bar, choose OAuth Client ID, select Desktop App as the Application Type, and click Create to obtain your Client ID and Secret.
    

    Google Console – Create credentials

11. Download the credentials JSON file and save it in a secure location on your development machine.

Create the OAuth connection for Blogger

1. Create or select your SSIS project, and open the package.dtsx file.
2. Drag and drop the ZS REST API task from the Toolbox.
3. Inside Connection Manager, right-click and select New connection…
4. Inside the SSIS connectors, select the ZS-OAuth
   
   

   Create a new SSIS OAuth API Connection Manager

5. Select the Google API provider
6. Enter the Client ID and Client Secret you generated in the past step
7. Use Blogger URL scope: https://www.googleapis.com/auth/blogger
8. Generate the token
9. Test the connection to ensure it is working correctly.
10. Click OK to save the connection.
    

    OAuth connection – Blogger API

Get information from a Blog

1. Create a string variable {{User::BlogID}} and enter the ID from the blog you want to get information from.
2. From the SSIS Toolbox, drag the ZS REST API Task onto the Control Flow designer.
3. Double-click the task to open the editor.
4. Select the connection manager you created in the previous step.
5. In the URL Path field, enter: https://www.googleapis.com/blogger/v3/blogs/{{User::BlogID}}/posts
6. Set the HTTP method to GET.
7. Click the Preview tab to test the API connection and view sample data.
8. If the preview is successful, click OK to configure the task.
   

   REST API task – Blogger example

Save the information in a SQL table

1. Drag and drop a Data Flow into the Control Flow and double-click it.
   

   Drag and drop Data flow.

2. In the Data Flow, drag and drop the JSON source, then double-click it to configure it.
   

   SSIS JSON Source – Drag and Drop

3. Use the same configuration as in the REST API task, but first enable the Use credentials option.
4. Enter the URL, select the OAuth connection, and use the Method GET.
5. You can use the filter option to get the object you want to get data from: $.Items[*]
6. Preview the data and click OK to save the configuration.
   

   JSON Source – Blogger Example Request
7. Now drag and drop a destination component. In our case, we will use the Upsert destination component.
   

   SSIS – Data Flow – Drag and Drop Upsert Destination Component

8. Double-click in the Upsert destination component to configure it.
9. Select the Action you want to use.
10. Create or select a database connector.
11. Create or select a table.
12. Use the options Insert or/and Update.
13. Map the column; if the columns have the same name, it will be automatic.
14. Select at least one column for a primary key.
15. Click OK to save the configuration.
    

    Upsert destination – Blogger example

16. Run the package, and you will see the data from Blogger in your database
    

    Blogger – result

Code Examples

Sample REST API Request

Here’s an example of retrieving blog posts using the Blogger REST API:

GET https://www.googleapis.com/blogger/v3/blogs/{blogId}/posts
Authorization: Bearer {access_token}
Content-Type: application/json

Example response:

{
  "kind": "blogger#posts",
  "items": [
    {
      "kind": "blogger#post",
      "id": "7109703471293209437",
      "title": "Getting started with Blogger API",
      "content": "Learn how to integrate Blogger with your applications...",
      "published": "2026-03-15T10:30:00Z",
      "updated": "2026-03-20T14:15:00Z",
      "url": "http://example.blogspot.com/2026/03/getting-started.html",
      "author": {
        "displayName": "John Doe"
      }
    }
  ]
}

SQL Table Structure

Recommended table structure for storing Blogger posts data:

CREATE TABLE BloggerPosts (
    PostID BIGINT PRIMARY KEY,
    BlogID BIGINT NOT NULL,
    Title NVARCHAR(MAX),
    Content NVARCHAR(MAX),
    AuthorName NVARCHAR(255),
    PublishedDate DATETIME,
    UpdatedDate DATETIME,
    PostURL NVARCHAR(MAX),
    CreatedDate DATETIME DEFAULT GETDATE(),
    ModifiedDate DATETIME DEFAULT GETDATE()
);

Conclusion

You have successfully learned how to use Zappysys SSIS connectors to extract data from the Blogger
platform using the REST API. With ZappySys SSIS PowerPack, connecting Blogger to your SQL Server becomes
straightforward and reliable. By configuring the REST connector for Blogger and using the native SSIS destination,
you can build a robust ETL pipeline for extracting blog data.

Explore the full capabilities of the ZappySys SSIS PowerPack to optimize your data integration
tasks. By applying this pattern, you can extend your solution to include additional transformations, data
validation, error handling, and scheduling through SQL Server Agent.

Need Help?

If you encounter any issues, our support team is here to help:

1. Live Chat — Use the chat widget (bottom-right corner of this page)
2. Email — support@zappysys.com
3. Support Center — Visit the ZappySys Support Portal

The post SSIS tutorial: How to connect Blogger with SSIS appeared first on ZappySys Blog.

In this article, we’ll learn how to integrate a documented API in Swagger with SQL Server Integration Services (SSIS) using ZappySys PowerPack.

Swagger enables us to visualize and test REST endpoints, while ZappySys SSIS provides advanced components for consuming APIs and transforming data in ETL processes.

Steps

Get the Swagger endpoints

1. Review the server’s endpoint list. In this example, we will explore the Pet category.
   

   Swagger – Pet category

2. From the list, we can see multiple endpoints and HTTP methods, including GET, POST, PUT, and DELETE.
3. Click on an endpoint to expand it. For this example, we will use Add a new pet to the store.
   
   

   Swagger – Create a new pet

4. Click Try it out. Swagger will allow you to edit the request body and choose the content type (JSON or XML).
5. Click Execute. Swagger will display
   • The generated request in cURL format
   • The Request URL
   • The Response body
   • The HTTP status code
6. In the Response JSON, we can see the new pet with status code 200, indicating success.
7. Now, with the CURL syntax, we can use that inside our tool.
   

   Swagger – Create a new pet CURL

POST request: Create Pet Request

1. Open Visual Studio and create a new Integration Services Project.
2. After the project is created, locate the SSIS Toolbox on the left panel.
3. Drag and drop the REST API Task onto the Control Flow surface and double-click it to configure
   

   Drag and Drop REST API Task

4. Use the cURL generated from Swagger. For reference, you can review our article about converting cURL to an SSIS API Task.
   

   curl -X 'POST' \
     'https://petstore.swagger.io/v2/pet' \
     -H 'accept: application/json' \
     -H 'Content-Type: application/json' \
     -d '{
     "id": 123,
     "category": {
       "id": 456,
       "name": "Friendly"
     },
     "name": "Zappy",
     "photoUrls": [
       "string"
     ],
     "tags": [
       {
         "id": 0,
         "name": "tech"
       }
     ],
     "status": "available"
   }'

5. -X POST represents the HTTP method. Set the Request Method to POST.
6. Enter the URL from the cURL command.
7. Each -H represents a Header. Add the corresponding headers inside the REST API Task.
8. The -d parameter represents the Body. Paste the JSON payload into the Body section.
9. Click Test Request/Response to validate the configuration.
   

   SSIS – Create a new pet

10. You can use additional components such as JSON Source or XML Source, depending on how you plan to process the response.

POST request: Upload a file

1. Next, we will upload an image for the pet we created earlier
   

   Swagger – upload image

2. Generate the CURL code.
   

   curl -X 'POST' \
     'https://petstore.swagger.io/v2/pet/123/uploadImage' \
     -H 'accept: application/json' \
     -H 'Content-Type: multipart/form-data' \
     -F 'file=@new.jpg;type=image/jpeg'

3. Configure the REST API Task with:
   • Request Method: POST
   • URL
   • Headers
   • File parameter
4. Check the option File upload.
5. Specify the full file path (e.g., file=@D:\Zappysys\new.jpg).
6. Click Test Request to validate the upload.
7. You can find an article about uploading files here.
   

   SSIS – upload an image

GET request: Get the Pet information

1.  Now we will get the pet we created earlier. This is the example
   

   Swagger – get pet by id

2. We generate the CURL
   

   curl -X 'GET' \
     'https://petstore.swagger.io/v2/pet/123' \
     -H 'accept: application/json'

3. Inside the Visual Studio project, drag and drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow designer.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

4. From the SSIS toolbox, drag and drop a JSON Source onto the data flow designer surface, and double-click on it to edit it
   

   SSIS JSON Source – Drag and Drop

5. We use the method GET
6. We enter the URL with the ID from the earlier response.
7. We can use the filter to get the object we need, and now we can check the data.
8. Preview the data, and you will see the Pet’s information
   

   SSIS – get a pet by ID

PUT request: Update an existing pet

1. We go to the updated pet example and run the example
   

   Swagger – update a pet

2. Generate the CURL
   

   curl -X 'PUT' \
     'https://petstore.swagger.io/v2/pet' \
     -H 'accept: application/json' \
     -H 'Content-Type: application/json' \
     -d '{
     "id": 0,
     "category": {
       "id": 0,
       "name": "string"
     },
     "name": "doggie2",
     "photoUrls": [
       "string"
     ],
     "tags": [
       {
         "id": 0,
         "name": "string"
       }
     ],
     "status": "available"
   }'

3. Inside the JSON source, we configure the HTTP method to PUT.
4. Change the url and the header.
5. Inside the Body we can change the values from the existing pet.
6. Test the connection and check the result.
   

   SSIS – update a pet

DELETE request: Remove an existing pet

1. Now we will delete the pet using this example.
   

   Swagger – delete a pet

2. Generate the CURL to delete the pet
   

   curl -X 'DELETE' \
     'https://petstore.swagger.io/v2/pet/1234' \
     -H 'accept: application/json' \
     -H 'api_key: special-key'

3. Inside the JSON source, configure the HTTP method to DELETE.
4. Change the URL to include the pet’s ID.
5. Inside the header, we need to insert an API key.
6. Test the connection, and we will delete the pet.
   

   aSSIS – delete a pet

Authentication method

1. Some APIs require authentication. You can configure authentication using an HTTP Connection Manager.
2. Enter the login URL if required.
3. Select the credential type (e.g., Basic – Username/Password).
4. Enter the credentials.
5. Click OK to save the configuration.
6. This connection can be reused across multiple REST API Tasks or JSON Sources.
7. For more details on basic authentication, click here. For more details on the OAuth connection, click here.
   

   Swagger – SSIS basic authentication

Conclusion

Integrating Swagger-documented APIs with ZappySys SSIS becomes significantly easier when using the generated cURL command as a reference. Swagger provides a clear and interactive way to understand API endpoints, request structures, headers, and payload formats, while cURL offers a direct, executable representation of the HTTP request.

By translating the cURL command into the REST API Task or JSON Source configuration, developers can confidently replicate API calls inside SSIS without guesswork. This approach ensures accuracy, reduces troubleshooting time, and provides a consistent workflow for handling GET, POST, PUT, DELETE operations, file uploads, and authentication scenarios.

Download SSIS PowerPack today and see for yourself how easy it is to combine Swagger and ZappySys SSIS, enabling organizations to seamlessly integrate modern REST APIs into enterprise ETL processes and bridge the gap between cloud-based services and traditional SQL Server data platforms.

Still need help?

If the issue persists, please get in touch with our support team:

• Live Chat: Open the chat widget (bottom right of this page)
• Email: support@zappysys.com
• Support Center: Support | ZappySys

The post SSIS tutorial: How to connect to Swagger appeared first on ZappySys Blog.

This time we will explain how to get all URLS from emails using MS Outlook. Sometimes we need get all URLs from emails. In this post, we will show how to do this. Step by step using SSIS.

Microsoft Graph API is a unified way to access many Microsoft services API including Office 365 API

Prerequisites

Before we perform steps listed in this article, you will need to make sure following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Optional (If you want to Deploy and Schedule ) – Deploy and Schedule SSIS Packages

Steps-by-step process to download images from HTML using SSIS

Register Application (OAuth2 App for Graph API)

First, check our article about how to get the OAuth2 connection, click here for the article

Get the information for the body content from the emails

1. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.

Drag and drop Data flow task

2. Double click on the DataFlow task to see the DataFlow designer surface.

3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.

Drag and drop a JSON sourcec

4. Select the OAuth connection you created, then use this URL to get the body content

https://graph.microsoft.com/v1.0/me/messages?$select=subject,body,bodyPreview,uniqueBody

Getting the information from the emails

5. Drag and drop the trash destination and save the result in a file with the following configuration. Make sure to check the Overwrite target file if exists.

Save the body from the emails in a file

Now we have saved the email body in the file, to extract the links we need to go control flow and drag and drop the Regular Expression Parser Task and follow the steps from below section.

Getting all URLs from emails inside a variable

6. The next step is to save the URLs’. We will use the Regular Expression Parser Task for this.
Also, you need to use Regex and here is one example you can use, in this page Regex101 you can check more details about the expressions we are using:

Getting all URL from the emails

Expression:

href="(.*?)"{{*}}

Save the result in a file of the URLs from emails

7. Finally, use the Logging task to save the URLs in a file, use the following configuration

Uncheck all options and select message type as none

Conclusion

If everything is OK, you will be able to download the URLS from your emails. To do that, we read the list from the body of the emails. Then we get the URLs of the emails using expressions. Finally, we store them in a local file.

The post How to get all URLs from emails from Outlook appeared first on ZappySys Blog.

In this post, you will learn how to read / write ManageEngine ServiceDesk data in SSIS (Drag and drop approach without any coding).  We will use SSIS JSON/ REST API Source to extract data from Servicedesk API and use SSIS Web API Destination to write data to Zoho.

We will look at step-by-step instructions to read ServiceDesk  data (e.g. Request, Problems, Contacts) and load into SQL Server Database Table.

Authentication ServiceDesk REST API in SSIS (Use OAuth 2.0)

Very first step to call ServiceDesk API is you need to register OAuth App in Zoho Portal. Don’t get confused if you see Zoho Portal for ServiceDesk a Product from ManageEngine is in fact Zoho subsidiary (in other words Zoho is the parent company). ServiceDesk has On-Premises version too but this article is only about Cloud version.

Register Zoho OAuth App and get ClientID/ Secret (*** MUST READ ***)

Check this article to register your OAuth App and obtain Client ID  /Client Secret

Create ServiceDesk API Connection in SSIS

Once you create Zoho OAuth App now its time to call some Zoho API. Let’s create the OAuth Connection first. In the next section we will see how to load data in SQL Server.

1. Right click on Connection Panel and choose Choose “New Connection..”. Select ZS-OAUTH type
   

   Create new SSIS OAuth API Connection Manager

2. Configure your OAuth connection General Tab as below
3. OAuth Provider: Custom
4. OAuth Version: OAuth2
5. Client Id , Client Secret
6. Enter One Scope Per Line (For more info on scope read here) . For write you can change READ to WRITE or use ALL as per the help link.
   

   SDPOnDemand.requests.READ

7. Auth URL : https://accounts.zoho.com/oauth/v2/auth
8. Token URL:  https://accounts.zoho.com/oauth/v2/token
9. Configure Advanced tab as below
10. Enter Callback URL : https://zappysys.com/oauth
11. Enter Extra Attribute for /Auth : access_type=offline&prompt=consent
    This is needed to obtain refresh_token (if you ommit this then only access_token returned which cannot be renewed once expired after 1hr)

    

12. Now go back to general tab and Click Generate Token. Close the Popup and use Full browser instead if you see prompt.
13. Click OK to save the connection UI

Read data from ServiceDesk API in SSIS JSON Source

In this section we will learn how to read data from ServiceDesk using SSIS JSON Source. We will use OAuth connection created in earlier section.

Now lets look at step by step example of reading ManageEngine ServiceDesk data using SSIS (e.g. Requests).

ManageEngine provides JSON API for read action so we will Use SSIS JSON Source for this purpose.

Configure JSON Source

So below steps shows example of reading from ManageEngine Requests API

1. Open SSIS Package
2. Drag data flow task from Control flow SSIS Toolbox
   

   Drag data flow from SSIS Toolbox

3. Go to data flow and drag ZS JSON Source from SSIS Toolbox
   

   Drag and drop SSIS JSON Source (REST API or File)

4. Double click JSON Source to configure. Enter URL as below. You can supply few more parameters. Click Here to get full list of parameters and usage. In below example we are reading records from Leads Module but you can replace Leads to something else (e.g. Accounts). Refer to above help link for available Modules.
   

   https://sdpondemand.manageengine.com/api/v3/requests

5. Check Use Credentials. Select OAuth connection manager we created in the previous section.
6. Add the Accept header in HTTP Headers.
   

   Accept: application/vnd.manageengine.sdp.v3+json

7. Under filter options tab, enter Array Filter as  

   $.requests[*]

8. Now click Preview to see your data
9. Click OK to save UI. In the next section, we will see how to setup pagination and incremental extract of Servicedesk records. After that we will see how to load data into SQL Server Table.

Read ManageEngine ServiceDesk Plus data in SSIS (Cloud Version) – Zoho API

ServiceDesk API Pagination (Read all Requests)

Just like most API ServiceDesk also uses pagination. See below to learn how to configure pagination in JSON Source for ServiceDesk API.

1. Double click JSON Source
2. Change URL to include two things (Page size (how many rows per page and start row). Note <%page%> must match with Page Num Indicator you going enter in the next screen
   

   https://sdpondemand.manageengine.com/api/v3/requests?input_data={"list_info":{"row_count":100,"start_index":<%page%>}}

3. Go to Pagination Tab and select URL Path Mode
4. For Page Num Indicator : Enter

   <%page%>

5. For Increment By : Enter

   100

6. Page Start Number select New variable call it StartIndex and start default value 1

Once you configure this way now your API will return all records until last page is detected. For more information on pagination check this article

How to paginate ManageEngine ServiceDesk API

Loading ServiceDesk data into SQL Server Table / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Write data to ServiceDesk using OAuth 2.0

If you like to load data from any source (i.e. SQL Server) to Zoho CRM then you can use Records API (POST). Use a combination of the following Transforms  / Components. Later in this article we have some more examples (Older API) which may give you one more ways to load data into Zoho (Use of Template Transform).

Basically you have to build POST Body using JSON Generator Transform or Template Transform and then use that data in Web API Destination to call POST API request to push data to ServiceDesk. Basically screnshot below is just an example using some other Zoho API but you get the idea.

	Web API Destination
	JSON Generator Transform
	JSON Parser Transform

Create JSON for POST request (Single dataset pattern)

Insert data to Zoho Table using SSIS (Lead, Account Modules) – Bulk Load using POST API call

Write data to ServiceDesk using SSIS (Import SQL Server Table to ServiceDesk)

Now you know how to read data from Zoho CRM using JSON API Source next step is to load data to Zoho CRM. For writing data we will use following three components

	Web API Destination
	Template Transform
	XML Parser Transform

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Conclusion

Manageengine  Servicedesk API provides a great way to automate data read/write operations. However, to call Service API  you have to use SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Servicedesk API call via a  simple drag and drop approach without coding. Try  SSIS PowerPack for free and call virtually any REST API in a few clicks.

The post Read/Write ManageEngine ServiceDesk Data in SSIS (Zoho API) appeared first on ZappySys Blog.

If you have need to integrate REST API in SSIS (i.e. fetch JSON, XML, CSV via HTTP call) and wondering how to Read API data in SSIS and Load into SQL DB (i.e. SQL Server, MySQL, Oracle) then you are at the right place.

In this article we will cover simple step by step approach how to achieve API integration in few clicks. For detailed introduction please refer to our previous article.

In next few sections we will cover the following SSIS Components to read data from API like a Table (Parse in Rows / Column). Our instructions will be mostly for JSON Format but concepts are same for other formats (i.e. XML / CSV).

	JSON Source Connector (Read from REST API, JSON File or OData Service): Use this dataflow component when you have to fetch data from REST API webservice like a table. This component allows you to extract JSON data from webservice and de-normalize nested structure so you can save to Relational database such as SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local JSON files or direct JSON string (Wildcard pattern supported too e.g. c:\data\file*.json).
	XML Source Connector (SOAP, File, REST) : Use this dataflow component when you have to fetch data from XML or SOAP webservice and consume data like a table. This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local XML files or direct XML string.
	CSV Source Connector (File, REST) : Use this dataflow component when you have to fetch data from File or REST API which returns data in CSV format . This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local CSV files or direct CSV string.

There are few more API components not discussed in this articles. They are REST API Task (no Parser – Just raw API calls) and Web API Destination (Write / Send Data To API / Delete rows). They both have different use case to check them see it fits in your need.

Video Tutorial

Step-By-Step – Read API data in SSIS using JSON / XML / CSV Source

In this section you will learn how to use JSON Source Adapter to Read data from API URL.

Load API data into SQL Server Table / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

API Authentication

In our step by step example we did not cover Authentication to keep things simple but in real world you have to pass some sort of authentication details to make API calls (e.g. UserID/Password or Secure Token). There are many ways to authenticate with API Services. ZappySys offer mainly 2 Connection Managers to cover most Authentication scenarios.

1. HTTP Connection Manager
2. OAuth Connection Manager

Using HTTP Connection Manager

If you have use case of using UserID / Password or any other method supported by HTTP Connection Manager then use below option (Below example is for Basic Authentication)

SSIS JSON Source – Passing Basic Credentials (Supply UserID / Password) using HTTP Connection

Using OAuth Connection Manager

If your API support OAuth Standard for Authorization then use OAuth Connection Type as below.

Generate Token for Google API using OAuth 2.0

API Pagination

Pagination is another important concept to understand when you call API using ZappySys Connector. If you ready many records from your API then by default API might not return all rows so you have to paginate until all records are read (Like a Loop in Programming). ZappySys makes it easy to paginate. Check this article to configure many different Pagination Modes

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Conclusion

REST API is becoming more and more popular each day. With that Data integration from RESTful API services going to be challenge. Luckily ZappySys SSIS PowerPack provides a great way to integrate any API in SSIS via simple drag and drop approach without coding. Try SSIS PowerPack for FREE  see how much time / money you can save and to integrate virtually any REST API.

The post How to Read API data in SSIS and Load into SQL Table appeared first on ZappySys Blog.

In our previous posts we saw how to read and write Salesforce Data using SSIS. In this post we will see how to read Salesforce Metadata in SSIS using JSON Source.

Step-By-Step

In this section you will learn how to use ZappySys JSON Source Adapter to extract data from Salesforce REST API (JSON Format). You can call pretty much Any REST API to fetch data from salesforce but our example for this post would be read metadata (i.e. table list) from salesforce (we will call sobjects to get all table information) .

So lets get started.

1. In case you missed the previous section, You need to Download and Install SSIS ZappySys PowerPack.
2. Once you finished first step, Open Visual Studio and Create New SSIS Package Project.
3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
4. Double click on the Data Flow task to see Data Flow designer surface.
5. From the SSIS toolbox drag and drop JSON Source on the Data Flow designer surface.
   
6. Now double click JSON Source enter any valid REST API URL like below. Make sure you used correct instance name. Using wrong instance name might slow down connection or fail your request. If you are not sure then just login to salesforce portal using credentials you like to use and observe browser URL.. It will have instance number.
   

   https://na34.salesforce.com/services/data/v20.0/sobjects/Account/describe/

7. Check Use Connection and select New ZS-Salesforce Connection from the list.
   

   Create new REST API Connection in SSIS (For JSON, XML or CSV Source)

8. Configure your salesforce connection
   
9. Select or type filter. For example in this case you will type 

   $.sobjects[*]

10. Click OK to Save UI and attach your source to some target like OLEDB Destination and run your data flow to test.
11. Here is full UI after all configurations.
    

    Read Salesforce Metadata in SSIS (Get Table List Example – SOObject REST API Call)

How to read Salesforce Table Fields Metadata

If you have need to read fields and its information for tables then use below steps

1. Use almost identical steps as previous section. Except two things. Our URL and Filter would be different.
2. Configure your API URL as below
   

   https://YOUR-INSTANCE.salesforce.com/services/data/v20.0/sobjects/YOUR-TABLE/describe/

   Example:
   

   https://demo.salesforce.com/services/data/v20.0/sobjects/Account/describe/

3. Configure Filter as a 

   $.fields[*]

Get Salesforce Table Fields Metadata using REST API call in SSIS

The post How to read Salesforce Metadata in SSIS using REST API (JSON Source) appeared first on ZappySys Blog.

In our previous article, we saw how to get Office 365 data in Power BI. Now, let’s look at how to get Office 365 Mail Attachments using SSIS without any coding. Microsoft Graph API is a unified way to access many Microsoft services, including the Office 365 API. In this post, we will use the SSIS JSON/REST API Source Connector to retrieve the Outlook Messages list and its attachments.

Use Case of Microsoft Graph API

Here are some use cases for why you want to use the Microsoft Graph API

• Read/Write events from Outlook Calendar
• Send email  / Read emails
• Get the list of files from OneDrive
• Upload/ Download files to OneDrive
• Read/Write Excel Sheet (Use range or a specific range of a specific sheet)
• Read the SharePoint Document library
• Search content from the SharePoint document library
• Get AD users for your Organization (AD Accounts)
• Update OneNote
• Track changes to Users, Events, Calendar Items using the delta API

Step-By-Step to call the Microsoft Graph Rest API Using SSIS

Now, let’s look at examples to access your Outlook.com emails using SSIS. You can use the same techniques to call pretty much any Office 365 API.

Configure SSIS OAuth Connection for Graph REST API

The first step to access any Graph API is to configure an OAuth connection

1. Open Visual Studio
2. Open an existing SSIS Project or create a new one using File > New > Project > Choose “Integration Services Project” Type under Business Intelligence template category.
3. Create Package
4. Right-click inside the Connection Managers area and click New Connection…
5. From the connection type list, select the ZS-OAUTH connection type.
   

   Create a new SSIS OAuth API Connection Manager

6. On the OAuth Connection Manager, configure the following options
   1. Select “Custom” from the Provider dropdown
   2. Select OAuth2 from the OAuth version
   3. Enter your Application Id and Secret Key (i.e., App Password) obtained in the previous section from here.
   4. Enter the following URL in the Authorization URL
      

      https://login.microsoftonline.com/common/oauth2/v2.0/authorize

   5. Use the following URL in the token URL field
      

      https://login.microsoftonline.com/common/oauth2/v2.0/token

   6. Enter the following Graph API Scopes (each scope must be entered on a new line).
      

      user.read
      mail.read
      offline_access

      NOTE: Always include the offline_access scope, which returns a refresh_token, which allows you to renew the token without going through the login process again. For more information about which scopes are needed, always refer to the API help page (e.g., if you want to read mail using this API, then you will see it says you can use Mail.Read or Mail.ReadWrite scope).
   7. Go to the Advanced tab and enter the following URL in the Callback/Return URL (assuming the same URL used when you registered the App in the previous section)
      

      https://zappysys.com/oauth

   8. Now go back to the first tab and click Generate Token. When prompted, log in using your Personal Microsoft Account or Work Account (Office 365 or AD login)
   9. After logging in, you will see the Accept option. Just click it. If things go right, then you will see the Access Token and Refresh Token fields populated.
7. Once everything is configured, you can click Test Connection to verify the connection works.
   

   SSIS OAuth Connection Manager for Office 365 REST API Microsoft Graph API

Download the Messages Attachment Files from Outlook

Let’s start with an example. We use the SSIS JSON Source and Web API Destination components to make a call to the Microsoft Graph REST API.

1. First of all, drag and drop the Data Flow Task from the SSIS Toolbox and double-click it to edit.
   

   Drag and drop Data flow

2. From the SSIS toolbox, drag and drop the JSON Source onto the Data Flow Designer surface.
   

   SSIS JSON Source – Drag and Drop

3. Double-click the JSON Source, then enter the following URL to invoke the Messages API call with search options to retrieve emails with attachments.
   

   https://graph.microsoft.com/v1.0/me/messages?$select=sentDateTime,from,subject,hasAttachments&$search="hasAttachments:True"

   Set the filter to $.value[*] and click the preview.
   

   SSIS JSON Source – Get data from Microsoft Graph API (Office 365 API – Read Mail Example)

   To customize the URL with additional parameters, check this help link. It’s a standard OData Protocol, so you can leverage many common OData features (e.g., use $top parameters to set pageSize. Default is 10 for List Mail, but you can include more records per response by changing it, such as …/messages?$top=50).

4.  Now, let’s use Template Transform to set the URL using the message ID to make the Get attachment API call.

   https://graph.microsoft.com/v1.0/me/messages/<%id%>/attachments

   

   SSIS Template Transform sending attachment id
5. Now we need to use the Web API Destination to make the Get Attachment API call by message ID. Select the Input column for URL as that TemplateOutput.
   

   SSIS Web API Destination – Get data from Microsoft Graph API (Office 365 API – Get Attachment Example)

6. Now, the next step is to parse the JSON Response String of the get attachment api call. For that, we need to use a JSON Parser. Select the input as Web PAI Destination ResponseText and use the below sample JSON string to configure the metadata(Columns).
   

   {"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#users('support%40zappysys.com')/messages('AQMkADAwATMwMAItMDkyMC0zZTg4AC0wMAItMDAKAEYAAAP_JUy')/attachments"
   ,"value":[{"@odata.type":"#microsoft.graph.fileAttachment","@odata.mediaContentType":"text/plain"
   ,"id":"AQMkADAwATMwMAItMDkyMC0zZTg4AC0wMAItMDAKAEYAAAP_JUy_27M"
   ,"lastModifiedDateTime":"2020-02-21T07:36:16Z"
   ,"name":"Test.txt","contentType":"text/plain","size":870,"isInline":false
   ,"contentId":null,"contentLocation":null,"contentBytes":"VGVzdCBNZXNzYWdl"}]}

   

   JSON Parser Transform – using JSON example
7.  Now, in the JSON Parser, go to the Go-to columns Tab and set the contentBytes DataType to DT_IMAGE, and increase the other columns’ DT_WSTR lengths by 1000 and 1500. We set contentBytes to the DT_IMAGE  datatype, which is like varbinary(MAX).
   

   JSON Parser Transform Columns Tab fix the datatype

8. Furthermore, in the next step, we need to use an SSIS Derived Column to set the FilePath where we want to save the attachments, and then concatenate the local file path with the attachment name column.
   

   SSIS Derived Column: Add New Column

9. Finally, we need to use the Export Column Transform (Native) to export attachment bytes to a local file. It needs to know 2 upstream info Bytes and the target file path.
   

   Export Column configuration

10. Make sure to attach the export column to some destination (e.g., our ZS Trash Destination); the engine might remove it. At the same time, it optimizes the runtime. It may remove all transformations that don’t have a destination, such as deploying the package to SQL Server, or when you set optimize in Visual Studio.
11. That’s it, execute the package, and it will download all the email attachments.
    

    Downloaded Mail Attachments

Conclusion

After all, in this article, we learned how to make a Microsoft Graph API REST API call. We used the SSIS JSON / REST API Connector to extract data from Outlook Mail using OAuth. Made the Get attachment API call using the Web API Destination and parsed the JSON response using the JSON Parser. And using the Native SSIS Export Column, we saved the export attachment’s bytes to a local file. To explore many other scenarios not discussed in this article, download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • Help File: Web API Destination
  • Help File: JSON Parser Transform
  • Overview of the Microsoft Graph API

The post How to Get Office 365 Mail Attachments using SSIS appeared first on ZappySys Blog.

In our previous article, we see how to load data into Elastic Search using SSIS. Now let’s look at how to read data from ElasticSearch using SSIS and load response into SQL Server.

Elasticsearch is a powerful engine that allows you to store, aggregate, and, most importantly, search data in a very analytical way. In this tutorial, you will learn how to load Elasticsearch data to SQL Server with SSIS (part of SQL Server) and ZappySys PowerPack. We also see how to set the maximum result window of the index using the Rest API Task.

In a nutshell, this post will focus on how to Make Elasticsearch Search API call using SSIS.

Step-By-Step to Call Elasticsearch API call using SSIS

Read from Elasticsearch Search API call

Let’s make the Elasticsearch Search API call using the JSON source to get all the records of the index with pagination.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Dragging and dropping Data Flow Task into Control Flow

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to make Search API Call on the index.
   

   http://{elasticsearch-site-url}/{index}/{index-type}/_search

4. Now check Use credentials and Select Connection section press

   <New ZS-HTTP Connection>

   .
   

   SSIS JSON Source – Passing Basic Credentials (Supply UserID / Password) using HTTP Connection

5. Once the HTTP Connection Manager window opens configure the connection to your Elasticsearch instance:
   • Set Web Url, which points to your Elasticsearch instance.
   • Set Credentials Type to

     Basic - UserID/Password

      (or other appropriate authentication methods).
   • Finally, set User Name and Password:
     
     

     Configure SSIS HTTP Connection to connect to Elasticsearch

6. Now, select method as POST and set for and size value in the body to get data using pagination. Set the content type as JSON (application/json).
   

   {"from" : <%page%>, "size" : 100}

7. Furthermore, go to the pagination tab and configure pagination like the below screenshot makes sure we need to enter increment size the same as given in body size value therefor 100.
   

   Pagination Mode: POST data in the body

8. That’s it now select the desired filter and click on the Preview button.
9. That’s it we are ready to load the data in the SQL Server.

For more than 10000 rows

When you try to get more than 100000 rows you will get the next error message

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [11000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
            }
        ]
    },
    "status": 400
}

Here is how to configure your JSON Source to pull more than 10K rows using Scroll Method.

1. In Settings Tab: In the Body and URL rather than Index name use

   [$tag$]

2. In the filter type:

   $.hits.hits[*]._source

3. Uncheck Include Parent
   

   Elasticsearch example for more than 10000 rows

4. In Pagination Tab, Set Next Link as below

   $._scroll_id

5. Stop Indicator Attribute as below

   $.hits.hits[0]._id

6. Stop Indicator Value as

   regex=^$

   

   Elasticsearch pagination tab configuration

7. In Advanced Pagination Tab, Check Has Different Page Info and Enable Page Token
8. Enter Page Place Holders as below (change YOUR_INDEX_NAME)
   

   url=YOUR_INDEX_NAME/_search?scroll=10m|_search/scroll

9. First Page Body as below – Change query as needed else keep it match all to fetch all rows from Index
   

   { "size":1000, "query": { "match_all": { }  } }

10. Next Page Body as below 

    {  "scroll": "5m", "scroll_id": "[$pagetoken$]" }

    

    Configuration for Elasticsearch advanced pagination tab

Loading Elasticsearch API data into SQL Server / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Enable Deep Pagination Setting

By default index, the maximum result window size is 10,000 means you can paginate upto 10000 rows. If you ever want to get more than 10000 result then you have to perform the following steps to change default setting from 10000 to 500000 (that’s max).

We need to send PUT request to below URL:

http://{elasticsearch-site-url}/{your-index}/_settings

Need to pass below body as I am setting max result window size value as 500000.

{ "index" : { "max_result_window" : 500000 } }

And set content-type as JSON (application/json) and click on Test Request/Response and it will make the API call and set the max window size.

Rest API Task PUT Call

That’s it we have successfully made the call to set the maximum result window size of the index.

Common Errors

Deployment to Production

Conclusion

After all, we saw you How to Make Elasticsearch Search API call using SSIS JSON Source and load response into SQL Server. In this article, we have seen that how to make a setting call to set the maximum result window of the index. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File)
• Elasticsearch Search APIs

The post How to read data from ElasticSearch using SSIS appeared first on ZappySys Blog.

In our previous article we see How to read call REST API data in SSIS. Now in this article, we will see How to Make UPS API call in SSIS and load response into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call UPS REST API Call can be useful for any developer regardless of which programming language or tool set you use. We will also see how to configure POST API Call.

We will go through the steps to make Pickup Creation Request API call which Returns a PRN and Rate Result and we load it into MS SQL Server.

In nutshell, this post will focus on how to Make UPS REST API Call in SSIS.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of UPS API.

What is UPS?

United Parcel Service Founded in 1907, UPS is the world’s largest package delivery company and a leading global provider of specialised transportation and logistics services. Using advanced technology, access to global resources, and an integrated network of physical, technological, and human assets, UPS provides a powerful competitive advantage that can help you earn repeat customers and grow your business.

Where to get the UPS Credentials?

You can get Username and password by signup in UPS website and also need to get Access Key from UPS Developer Kit.

Call UPS REST API using SSIS

Let’s start with an example. We use SSIS JSON Source component to make the call to UPS API, we will make Pickup Creation Request and load response into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to make Pickup Creation Request API Call and select Method as POST, I am using Testing URL but we need to use production one for real time API call.
   Testing:
   

   https://wwwcie.ups.com/rest/Pickup

   Production:
   

   https://onlinetools.ups.com/rest/Pickup

   

   JSON Source Set URL

4. Furthermore, set below JSON string in the body and also replace your UPS username, password and access key in the placeholder, i used my dummy data you need to pass yours valid data.
   

   {
   	"UPSSecurity": {
   		"UsernameToken": {
   			"Username": "[YOUR USERNAME]",
   			"Password": "[PASSWORD]"
   		},
   		"ServiceAccessToken": {
   			"AccessLicenseNumber": "[ACCESS KEY]"
   		}
   	},
   	"PickupCreationRequest": {
   		"Request": {
   			"TransactionReference": {
   				"CustomerContext": "CustomerContext."
   			}
   		},
   		"RatePickupIndicator": "Y",
   		"TaxInformationIndicator": "Y",
   		"PickupDateInfo": {
   			"CloseTime": "1700",
   			"ReadyTime": "1000",
   			"PickupDate": "20191123"
   		},
   		"PickupAddress": {
   			"CompanyName": "EIVR TEST NON-ACCOUNT PL 3",
   			"ContactName": "EIVR TES PL",
   			"AddressLine": "12380 Morris Rd",
   			"City": "Louisville",
   			"StateProvince": "KY",
   			"PostalCode": "40201",
   			"CountryCode": "US",
   			"ResidentialIndicator": "N",
   			"Phone": {
   				"Number": "+48223803070"
   			}
   		},
   		"AlternateAddressIndicator": "N",
   		"PickupPiece": {
   			"ServiceCode": "003",
   			"Quantity": "1",
   			"DestinationCountryCode": "US",
   			"ContainerCode": "01"
   		},
   		"OverweightIndicator": "N",
   		"PaymentMethod": "05"
   	}
   }

   

   JSON Source set Body

5. Now click on Select Filter button to get the desire filter.
   

   JSON Source Select Filter

6. Finally click on Preview button to load the data.
   

   JSON Source Preview

7. That’s it we are ready to load UPS Pickup Creation Request API call response in SQL Server.

Load UPS REST API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to make UPS API call using SSIS JSON Source and load response into SQL Server. In this article we made Pickup Creation Request POST API call same way you can make other UPS API calls like Tracking API and Address Verification API and so on by passing relevant URL and JSON string. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • UPS Developer Kit

The post How to Make UPS API call in SSIS appeared first on ZappySys Blog.

In our previous article we Read data from Twilio API in SSIS . Now in this article, we will see example of how to Call SurveyGizmo API in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call SurveyGizmo API Call can be useful for any developer regardless of which programming language or tool set you use. we will also see how to store access token and secret in the variable and pass it in the URL using variable.

We will go through the steps to make Survey GetList REST API call which Get a list of all of your surveys and we load it into MS SQL Server.

In nutshell, this post will focus on how to Make SurveyGizmo REST API Call in SSIS.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of SurveyGizmo API.

What is SurveyGizmo?

SurveyGizmo takes data out of dashboards and puts it into the hands of people who take action. Through the systems they use every day. SurveyGizmo is an online survey solution that makes it easy for businesses of all sorts to create and conduct surveys, polls, quizzes, and questionnaires.  Reaching out to potential and existing customers, managing organizational and employee relationships and conducting academic researches has become convenient through SurveyGizmo.

• Click here to read SurveyGizmo Rest API Documentation.

Call SurveyGizmo REST API using SSIS JSON Source

Let’s start with an example. We use SSIS JSON Source component to make the call to SurveyGizmo API, we will read list of all available temporary lists and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Now let’s add variable for SurveyGizmo Access Token and Secret and Set it in as a value.
   

   SSIS Add Variable

4. Now double click on JSON Source and enter the following URL with variable as placeholder to get list of your surveys.
   

   https://restapi.surveygizmo.com/v5/survey?page=1&resultsperpage=100&api_token={{User::v_api_token}}&api_token_secret={{User::v_api_token_secret}}

   

   JSON Source : URL with variable as placeholder

5. Let’s configure the pagination, go to Pagination tab and select mode as URL Parameter and set the indicator as

   Page

    and configure it like below screen.
   

   JSON Source Pagination

6. You can refer to this article for more details on pagination here Understanding REST API Pagination in SSIS / ODBC Drivers
7. Now click on Select Filter button and select the desire filter from the window and click on Preview to get the data.
   

   JSON Source Preview

8. That’s it we are ready to load all SurveyGizmo Surveys data in SQL Server.

Load SurveyGizmo REST API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to extract information from REST API such as SurveyGizmo and load into SQL Server. We also learned techniques like How to use variable as placeholder in URL and configure the pagination. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • SurveyGizmo Rest API

The post Call SurveyGizmo API in SSIS appeared first on ZappySys Blog.