How to read OData in SSIS – REST API Example

Introduction

In this article, we will learn how to utilize the SSIS JSON Source Component to retrieve JSON data from an OData service and perform OData pagination on significant results. According to the OData specification, data can be presented in either JSON or XML format. For the XML format, refer to this article. This article primarily covers the OData JSON API example.

REST API Pagination concept

Most of the REST API Services limit the size of the dataset returned in a single request. When more data is found, it simply includes a pointer to the following result set. If you wish to fetch all data (e.g., loop through), you must check the following result set indicator. If it is not null, you can continue fetching until all data is retrieved. This article covers several pagination methods.

Reading OData API in SSIS using JSON Source

Let’s discuss how to read the OData API and Paginate Automatically. Below is a typical OData service Response (JSON Format). Notice how it includes the nextLink attribute, which points to the following URL to fetch more data. When you set Data Format to OData in the JSON Source, it will handle pagination for you.

Request URL:

Response:

Step-By-Step

In this section, you will learn how to use the JSON Source Adapter to extract data from the OData API.

1. Firstly, you need to download and install SSIS ZappySys PowerPack.
2. Once you have finished the first step, Open Visual Studio and create a New SSIS Package Project.
3. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox.
   

   SSIS Data Flow Task – Drag and Drop

4. Double-click on the Data Flow task to see the Data Flow designer surface.
5. From the SSIS toolbox, drag and drop the JSON Source onto the Data Flow Designer surface.
   

   SSIS JSON Source – Drag and Drop

6. Double-click JSON Source and enter URL for OData API (You may use the below example URL for demo)
   
   https://services.odata.org/V3/Northwind/Northwind.svc/Order_Details?$format=json

   1	https://services.odata.org/V3/Northwind/Northwind.svc/Order_Details?$format=json

    
7. Now configure Array Filter as below (you may see $.value[*] or some other node if URL is different)
   

   OData Paging using SSIS JSON Source

8. Click Preview data and OK to save
   

   Read JSON File data from Web URL Example in SSIS

9. Now click OK to save.
10. Connect Your Source with a target like ZS Trash destination and execute the package. You will notice in the log that it paginates to return all rows
    

    Fetch OData in SSIS (REST API Example)

Loading OData API into SQL Server / Other Target using SSIS

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Twitter(Now X) REST API Paging Example using SSIS JSON Source

The Twitter (Now X) REST API is not an OData format, but you can see a similar concept there as well. On Twitter, you don’t have a partial URL like OData for the following link, but it provides a unique identifier for some sort of record. Read here for more information on the Twitter REST API Paging technique.

In our case, SSIS JSON Source Supports Paging very well, so we are covered. To loop through multiple result sets of Twitter data, simply configure the following four properties. See the screenshot below.

Twitter(X) REST API – Paging Example -Loop through resultset using cursor

Conclusion

Based on the above examples, you can see that handling Raging in REST API calls is not straightforward, but using Components like SSIS JSON Source can take some work off your plate.