Read data from Twilio API in SSIS

Introduction

TwilioIn our previous article we see How to call REST API in SSIS . Now in this article, we will see example of how to Make Twilio API Call in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call Twilio API Call can be useful for any developer regardless of which programming language or tool set you use. we will also see how to configure HTTP basic connection.

We will go through the steps to make Accounts REST API call which Returns a list of all Accounts resources and we load it into MS SQL Server.

In nutshell, this post will focus on how to Make Twilio REST API Call in SSIS.

 

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

  1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
  2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
  3. Make sure ZappySys SSIS PowerPack is installed (download it).
  4. Credentials of Twilio API.

What is Twilio?

Twilio is a cloud communications platform as a service company based in San Francisco, California. It’s allows software developers programmatically to make and receive phone calls, send and receive text messages, and perform other communication functions using its web service APIs.

Make Call of Twilio REST API using SSIS

Let’s start with an example. We use SSIS JSON Source component to make the call to Twilio API, we will read list of all available temporary lists and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

  1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
    Drag and Drop SSIS Data Flow Task from SSIS Toolbox

    Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

  2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
    Drag and Drop JSON Source Component

    Drag and Drop JSON Source Component

  3. Double click JSON Source and enter the following URL as below to get multiple Account resources and create the HTTP Connection use the same url in the http connection also.
    Json Source HTTP connection

    Json Source HTTP connection

  4. Now click on Select Filter button and select the desire filter from the window.
    Json Source Select Filter

    Json Source Select Filter

  5. Go to Pagination Tab and select mode as Response attribute Mode – Read next page information from response and set up the next page filter.
    JSON Source Preview

    JSON Source Preview

  6. You can refer to this article for more details on pagination here Understanding REST API Pagination in SSIS / ODBC Drivers
  7. Now select the desire options to get the data and click on preview, as here i want parent and missing child also i checked both options and clicked on Preview. if you don’t want to get parent details you can uncheck it.
    JSON Source Preview

    JSON Source Preview

  8. That’s it we are ready to load all Twilio accounts resources in SQL Server.

Load Twilio REST API data into SQL Server

Now let's look at how to load data into target like SQL Server, Oracle or Flat File. In below example we will see loading data into SQL Server database but steps may remain same for other targets which can be accessed using OLEDB Drivers (e.g. Oracle).
  1. Inside Data Flow, Drag OLEDB Destination from SSIS Toolbox
  2. Connect our Source component to OLEDB Destination
  3. Double click OLEDB Destination to configure it
  4. Select Target Connection or click NEW to create new connection
    Configure SSIS OLEDB Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

    Configure SSIS OLEDB Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

  5. Select Target Table or click NEW to create new table based on source columns
  6. Click on Mappings Tab to Auto map columns by name. You can change mappings as you need
    Create NEW Table SSIS OLEDB Destination

    Create NEW Table SSIS OLEDB Destination

  7. Click OK to Save OLEDB Destination Settings
  8. That's it, You are now ready to run data flow. NOTE: If you wish to debug data flow and see records when you run, add data viewer by right click on blue arrow > Click Enable Data Viewer
  9. To execute data flow, Right click anywhere inside Data Flow Surface and click Execute Task
 

Conclusion

After all, we saw you how to extract information from REST API such as Twilio using HTTP Connection and load into SQL Server. We also learned techniques like How to create HTTP Connection and configure the pagination. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

Posted in REST API, REST API Integration, SSIS JSON Source (File/REST) and tagged , , , .