In this article, you’ll learn how to connect SSIS PowerPack (ZappySys) to AliPay OpenAPI to call REST endpoints from SSIS. We’ll focus on the practical integration pattern AliPay requires: sending requests to the OpenAPI gateway with the correct parameters and a valid RSA2 signature, then parsing JSON responses in SSIS.

AliPay OpenAPI basics (what SSIS must send)

AliPay OpenAPI calls typically go through a single gateway endpoint:

• Sandbox gateway:
  

  https://openapi-sandbox.dl.alipaydev.com/gateway.do

A request is built from:

• Standard parameters (for example: app_id, method, charset, sign_type, timestamp, version)
• biz_content (usually JSON)
• sign (a signature generated from the parameters using RSA/RSA2—AliPay recommends RSA2)

AliPay requires you to construct a “string to sign” from parameters and attach sign and sign_type to the request, then validate/verify responses using the AliPay public key.

High-level integration pattern in SSIS

Because AliPay signing is mandatory, the most reliable SSIS approach is a two-step pattern:

1. Generate the signature (RSA2) for the request parameters.
2. Call the AliPay gateway from ZappySys (REST API Task or REST API Source), passing the signed parameters.

ZappySys components support custom HTTP methods, headers, request bodies, and advanced options—so they work well once you have the signed payload ready.

Steps

Start in the AliPay Sandbox

AliPay strongly recommends using the sandbox for development and debugging. In the sandbox, you must:

• An AliPay Developer Account
• An AliPay App ID
• Use the sandbox gateway URL
• RSA2 Private Key (generated in AliPay developer console)
• AliPay Public Key (for verification)
• Access to the AliPay Sandbox environment (recommended for testing)

Decide where you will generate the RSA2 signature

You have three common options:

• Option A (Recommended): SSIS Script Task (C#) to generate signature
  Use a Script Task before the ZappySys call to build parameters and compute sign.
• Option B: Use an AliPay SDK in a small helper app
  Run it via Execute Process Task to output signed parameters (JSON/text), then pass them to ZappySys.
• Option C: Use a signing microservice
  SSIS calls your internal service to get sign, then calls AliPay.

AliPay’s documentation emphasizes the signature process and verification using the AliPay public key.

Example: What a signed AliPay request looks like

AliPay’s gateway model is “parameters + signature.” Conceptually, your final request includes:

• app_id
• method (the API name you’re calling)
• charset (often utf-8)
• sign_type (recommended RSA2)
• timestamp
• version
• biz_content (JSON)
• sign (computed over the canonicalized parameter string)

AliPay documents that sign and sign_type are appended after creating the string to be signed, and verification uses the AliPay public key.

Configure the ZappySys REST API Task (Control Flow)

Use this when operating (e.g., creating a payment, refunding, or querying).

1. Drag ZappySys REST API Web Service Task into Control Flow.
2. Set the task:
   • Method: POST (commonly used with AliPay gateway)
   • URL:
     

     https://openapi-sandbox.dl.alipaydev.com/gateway.do

   • Headers: use
     

     Content-Type: application/x-www-form-urlencoded

   • Body: form-encoded parameters; use this format: key1=value1&key2=value2…

     app_id=YOUR_APP_ID&method=alipay.trade.query&charset=utf-8&sign_type=RSA2&timestamp=2026-02-20 10:00:00&version=1.0&biz_content={"out_trade_no":"202602200001"}&sign=GENERATED_SIGNATURE

3. Click Test Request/Response to confirm you get a valid JSON response.
   

   Alipay REST API

Troubleshooting common AliPay + SSIS issues

• invalid-signature / missing-signature: most often caused by parameter ordering, incorrect URL-encoding, wrong key pair, or using sandbox keys against production (or vice versa).
• Wrong gateway: sandbox and production are fully isolated—ensure your app_id,keys, and gateway host match the environment.
• Encoding issues: confirm UTF-8 everywhere and ensure parameters are encoded exactly once before submission.
• Parsing JSON: use ZappySys JSON Source preview and JSONPath filters to flatten nested objects.

Security best practices

• Store private keys securely (SSIS variables protected, encrypted config, or secret vaults).
• Do not hardcode secrets in packages.
• Use a sandbox for development, then switch to production only after full validation.

Conclusion

Download SSIS PowerPack today and see for yourself how easy it is to connect SSIS PowerPack (ZappySys) to AliPay is absolutely achievable once you treat AliPay’s OpenAPI as a signed gateway integration. The key is generating a correct RSA2 signature for every request, then using ZappySys REST components to send the parameters and parse responses.

Still need help?

If the issue persists, please get in touch with our support team:

• Live Chat: Open the chat widget (bottom right of this page)
• Email: support@zappysys.com
• Support Center: Support | ZappySys

The post SSIS tutorial: How to connect to AliPay appeared first on ZappySys Blog.

In this article, we’ll learn how to integrate a documented API in Swagger with SQL Server Integration Services (SSIS) using ZappySys PowerPack.

Swagger enables us to visualize and test REST endpoints, while ZappySys SSIS provides advanced components for consuming APIs and transforming data in ETL processes.

Steps

Get the Swagger endpoints

1. Review the server’s endpoint list. In this example, we will explore the Pet category.
   

   Swagger – Pet category

2. From the list, we can see multiple endpoints and HTTP methods, including GET, POST, PUT, and DELETE.
3. Click on an endpoint to expand it. For this example, we will use Add a new pet to the store.
   
   

   Swagger – Create a new pet

4. Click Try it out. Swagger will allow you to edit the request body and choose the content type (JSON or XML).
5. Click Execute. Swagger will display
   • The generated request in cURL format
   • The Request URL
   • The Response body
   • The HTTP status code
6. In the Response JSON, we can see the new pet with status code 200, indicating success.
7. Now, with the CURL syntax, we can use that inside our tool.
   

   Swagger – Create a new pet CURL

POST request: Create Pet Request

1. Open Visual Studio and create a new Integration Services Project.
2. After the project is created, locate the SSIS Toolbox on the left panel.
3. Drag and drop the REST API Task onto the Control Flow surface and double-click it to configure
   

   Drag and Drop REST API Task

4. Use the cURL generated from Swagger. For reference, you can review our article about converting cURL to an SSIS API Task.
   

   curl -X 'POST' \
     'https://petstore.swagger.io/v2/pet' \
     -H 'accept: application/json' \
     -H 'Content-Type: application/json' \
     -d '{
     "id": 123,
     "category": {
       "id": 456,
       "name": "Friendly"
     },
     "name": "Zappy",
     "photoUrls": [
       "string"
     ],
     "tags": [
       {
         "id": 0,
         "name": "tech"
       }
     ],
     "status": "available"
   }'

5. -X POST represents the HTTP method. Set the Request Method to POST.
6. Enter the URL from the cURL command.
7. Each -H represents a Header. Add the corresponding headers inside the REST API Task.
8. The -d parameter represents the Body. Paste the JSON payload into the Body section.
9. Click Test Request/Response to validate the configuration.
   

   SSIS – Create a new pet

10. You can use additional components such as JSON Source or XML Source, depending on how you plan to process the response.

POST request: Upload a file

1. Next, we will upload an image for the pet we created earlier
   

   Swagger – upload image

2. Generate the CURL code.
   

   curl -X 'POST' \
     'https://petstore.swagger.io/v2/pet/123/uploadImage' \
     -H 'accept: application/json' \
     -H 'Content-Type: multipart/form-data' \
     -F 'file=@new.jpg;type=image/jpeg'

3. Configure the REST API Task with:
   • Request Method: POST
   • URL
   • Headers
   • File parameter
4. Check the option File upload.
5. Specify the full file path (e.g., file=@D:\Zappysys\new.jpg).
6. Click Test Request to validate the upload.
7. You can find an article about uploading files here.
   

   SSIS – upload an image

GET request: Get the Pet information

1.  Now we will get the pet we created earlier. This is the example
   

   Swagger – get pet by id

2. We generate the CURL
   

   curl -X 'GET' \
     'https://petstore.swagger.io/v2/pet/123' \
     -H 'accept: application/json'

3. Inside the Visual Studio project, drag and drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow designer.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

4. From the SSIS toolbox, drag and drop a JSON Source onto the data flow designer surface, and double-click on it to edit it
   

   SSIS JSON Source – Drag and Drop

5. We use the method GET
6. We enter the URL with the ID from the earlier response.
7. We can use the filter to get the object we need, and now we can check the data.
8. Preview the data, and you will see the Pet’s information
   

   SSIS – get a pet by ID

PUT request: Update an existing pet

1. We go to the updated pet example and run the example
   

   Swagger – update a pet

2. Generate the CURL
   

   curl -X 'PUT' \
     'https://petstore.swagger.io/v2/pet' \
     -H 'accept: application/json' \
     -H 'Content-Type: application/json' \
     -d '{
     "id": 0,
     "category": {
       "id": 0,
       "name": "string"
     },
     "name": "doggie2",
     "photoUrls": [
       "string"
     ],
     "tags": [
       {
         "id": 0,
         "name": "string"
       }
     ],
     "status": "available"
   }'

3. Inside the JSON source, we configure the HTTP method to PUT.
4. Change the url and the header.
5. Inside the Body we can change the values from the existing pet.
6. Test the connection and check the result.
   

   SSIS – update a pet

DELETE request: Remove an existing pet

1. Now we will delete the pet using this example.
   

   Swagger – delete a pet

2. Generate the CURL to delete the pet
   

   curl -X 'DELETE' \
     'https://petstore.swagger.io/v2/pet/1234' \
     -H 'accept: application/json' \
     -H 'api_key: special-key'

3. Inside the JSON source, configure the HTTP method to DELETE.
4. Change the URL to include the pet’s ID.
5. Inside the header, we need to insert an API key.
6. Test the connection, and we will delete the pet.
   

   aSSIS – delete a pet

Authentication method

1. Some APIs require authentication. You can configure authentication using an HTTP Connection Manager.
2. Enter the login URL if required.
3. Select the credential type (e.g., Basic – Username/Password).
4. Enter the credentials.
5. Click OK to save the configuration.
6. This connection can be reused across multiple REST API Tasks or JSON Sources.
7. For more details on basic authentication, click here. For more details on the OAuth connection, click here.
   

   Swagger – SSIS basic authentication

Conclusion

Integrating Swagger-documented APIs with ZappySys SSIS becomes significantly easier when using the generated cURL command as a reference. Swagger provides a clear and interactive way to understand API endpoints, request structures, headers, and payload formats, while cURL offers a direct, executable representation of the HTTP request.

By translating the cURL command into the REST API Task or JSON Source configuration, developers can confidently replicate API calls inside SSIS without guesswork. This approach ensures accuracy, reduces troubleshooting time, and provides a consistent workflow for handling GET, POST, PUT, DELETE operations, file uploads, and authentication scenarios.

Download SSIS PowerPack today and see for yourself how easy it is to combine Swagger and ZappySys SSIS, enabling organizations to seamlessly integrate modern REST APIs into enterprise ETL processes and bridge the gap between cloud-based services and traditional SQL Server data platforms.

Still need help?

If the issue persists, please get in touch with our support team:

• Live Chat: Open the chat widget (bottom right of this page)
• Email: support@zappysys.com
• Support Center: Support | ZappySys

The post SSIS tutorial: How to connect to Swagger appeared first on ZappySys Blog.

In our previous post we saw how to read API data in SSIS. This post we will cover how to call NetSuite RESTlet to read / write data in SSIS or ODBC Apps. For demo purpose we will use SSIS PowerPack REST API Task but concepts and many UI elements remain same for ODBC JSON Driver too. This article assumes you know how to write Netsuite RESTlet and you have obtained Credentials to call RESTlet from extranal app.

What is NetSuite RESTlets

NetSuite added a really nice feature to expose business data and workflows as API by writing Javascript functions which they call it RESTlets. You can secure and customize these restlets as per your need.

You can automate CRUD (Create /Read / Write / Delete) operations in NetSuite by writing special functions in JavaScript and expose those functions as API URL which can be called by any external Systems (i.e. SSIS Packages , ODBC Apps like Power BI, Informatica, SSRS).

RESTlet can be secured few ways such as OAuth 1.0 and OAuth 2.0 authentication. We will cover both approach briefly in this article. You can choose any one as per your need

Creating NetSuite RESTlet / Obtain OAuth Credentials (Client ID and Secret)

Below video shows how to obtain necessary credentials to call Netsuites API (RESTlet).

At mark 4:30 seconds it will show you How to use OAuth 2.0 for calling RESTlet

Create NetSuite RESTlet Connection

There are few ways to authenticate NetSuite RESTlet. If you are using ZappySys Products then it makes it super easy to use either approach. Lets look at both approaches.

Using OAuth 2.0 (Recommended)

As per above video if you obtained Client ID and Secret for OAuth 2.0 you can follow below steps to configure OAuth connection to call RESTlet.

What you need to use OAuth 2.0.

• Your Account ID (it goes in URL)
• Your Client ID and Secret (Obtained from Netsuite Portal) – See above video Mark 04:30
• Register Redirect URL in NetSuite portal (we recommend you use https://zappysys.com/oauth as call back Url for Automatic Code extraction)

Steps to configure OAuth Connection

1. Right click on Connection Panel and choose Choose “New Connection..”. Select ZS-OAUTH type
   
   
   

   Create new SSIS OAuth API Connection Manager

    

    

2.  Enter following things
   ————————-
   On General Tab
   ————————-
   ClientID and Secret: <get it from NetSuite>
   Authorization URL: https://<accountID>.app.netsuite.com/app/login/oauth2/authorize.nl
   Token URL: https://<accountID>.suitetalk.api.netsuite.com/services/rest/auth/oauth2/v1/token
   Scope: restlet
   

   Configure NetSuite OAuth 2.0 Connection for API call

    

   ————————-
   On Advanced Tab
   ————————-
   Callback / Redirect URL:  https://zappysys.com/oauth  (this must be entered same way in NetSuite Portal)

   

   OAuth2 Connection – Callback / Redirect URL

3. Now go back to General Tab and click Generate Token button
4. Follow login screen and try to finish the flow. At the end it will generate Tokens and populate on UI
5. Click OK to save

Using OAuth 1.0

Another approach is use OAuth 1.0 (Older version). If you have enabled OAuth 1.0 in NetSuite then follow below steps (Create connection is same as before).

1. Setup General Tab and Advanced Tab as below.
   On ==== General Tab====:
   Enter ClientID, ClientSecret, AccessToken, AccessToken Secret
   On ==== Advanced Tab ====:
   Check Do Not Include Empty Key/Value Pairs
   Select Signature Method as HMAC-SHA256
   For Extra Token Attribute enter realm=<your-account-number>_SB1
2. Click OK to Save

Call NetSuite RESTlet in SSIS

1. Create a new SSIS Package
2. Drag REST API Task from SSIS Toolbox
3. Double click the Task to configure
4. Select URL from Connection
5. From Connection dropdown select OAuth connection we created in the earlier section (either OAuth 1 or OAuth 2)
6. Enter the URL as below. Assuming you copied URL from NetSuite Replace script ID and account number
   

   https://<account-number>-sb1.restlets.api.netsuite.com/app/site/hosting/restlet.nl?script=<scriptid>&deploy=1

   Enter <account-number>
   Enter <scriptid>
7. Change Method to POST if your script supports Parameters (Else you can supply id=xxxx and recordtype=xxxxx in the URL for GET request)
8. If its POST request then chnage ContentType to ApplicationJSON (application/json)
9. Enter your Input in the body. In below example our RESTlet takes sql query as input so we enter just one parameter named query but you can have different parameter(s)
   

   Syntax:  { param1_name : <value> , param2_name : <value> ....  }

10. Now click Test see everything works
    

    Calling NetSuite RESTlet in SSIS (Using REST API Task)

Conclusion

We saw how easy it is to call NetSuite RESTlets or other NetSuite API in SSIS / other BI Apps using ODBC Drivers provided by ZappySys. Download SSIS PowerPack and try your self see how easy it is to consume virtually any API in SSIS.

The post How to call NetSuite RESTlet API in SSIS / ODBC appeared first on ZappySys Blog.

Downloading Gmail Attachments in SSIS is always a little more difficult than working with simple text or numeric columns. If you are not an SSIS expert, doing it for the first time may be challenging. That is why we created this article for you. This is a common request we receive, so we are bringing this article to you.

Gmail is the most popular email service in the world, and it can be helpful to automate and get valuable information from it. We will connect to Gmail with the REST API. For that, if you are not an experienced user of the REST API, do not worry. This article is for you. We will provide simple steps for the connection.

Requirements

To do it, we will need the following components:

1. SSDT for SSIS must be installed on your computer
2. ZappySys SSIS Powerpack should be installed. This Powerpack Plugin includes connectors to Gmail via the REST API.

Getting Started

Creating the connection

1. First of all, in SSDT, create a new SSIS project
2. To create a connection in SSDT, open the Connection Manager and select New Connection.
   

   Create a new connection

3. Select the ZS OAuth connection. This is a special connection included with the ZS.
   SSIS PowerPack installer that we will use to connect to Gmail.
   

   Select ZS-OAuth connector
4. In the ZS OAuth connection, select the Google API OAuth Provider and specify the following credentials and scope permissions:
   

   SSIS Gmail connection

   https://mail.google.com/
   https://www.googleapis.com/auth/gmail.modify
   https://www.googleapis.com/auth/gmail.readonly
   https://www.googleapis.com/auth/gmail.addons.current.message.readonly
   https://www.googleapis.com/auth/gmail.addons.current.message.action

Creating the package to download Gmail attachments in SSIS

1. First of all, drag and drop the Data Flow Task from the SSIS Toolbox and double-click it
   to edit.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Secondly, in the Data Flow, we will use the JSON Source to connect to Gmail and get the attachments. JSON Source is a ZappySys component used now only to connect to JSON files, but also to connect to REST API Servers that support JSON. In this case, Gmail REST API can send information in JSON Format. Use the connection created before.
3. Basically, we need to specify the URL of the Gmail attachment. In Gmail, the email attachment is inside the message. Then you need to specify the message-id and the Gmail attachment-id like this:
   

   Zappysys JSON source Gmail attachments configuration

   https://www.googleapis.com/gmail/v1/users/me/messages/{{User::email_id}}/attachments/{{User::attachment_id}}

4. Now, in the go-to columns tab, set the data type to DT_IMAGE and increase the length of the other columns’ DT_WSTR fields by 1000 and 1500. We set

   contentBytes

   to the

   DT_IMAGE

     datatype, it’s like

   varbinary(MAX)

   

   JSON Source Columns Tab – attachment datatype

5. Next, we will use the derived column to set the path to the local folder where the attachment will be stored. The derived column will be used to set the local path for the attachment. Make sure to specify valid paths with double backslashes, since backslashes are special characters. The derived column is a special transformation component that creates new columns. SSIS expressions.
   

   Derived columns – Gmail attachment file path

6. Also, we will require the Export column to download into a file. This component is specially used to get images or binary files into the file system.
   

   Export columns transformation – Gmail attachment

7. Finally, we will use the Trash destination to get the attachment and finish the flow. This one is also a ZappySys component ubiquitous.
8. Your package should be something like this:
   

   Final Result – Gmail attachment

9. The next step is to run it and test it.
10. If everything is fine, we will be able to see the file in your system.

Conclusion

In conclusion, we learned how to download Gmail attachments in SSIS. Basically, you need to use the ZappySys JSON Source to connect to Gmail via the REST API, then use the derived column component to specify the path to store the attachment. The next step is to use the Export column component to export the attachment. Finally, the ZS Trash destination is the attachment’s destination. If you liked this article, do not hesitate to continue testing and using ZappySys PowerPack for SSIS.

API Connector

Here are a few examples with our API connector for Gmail

API connector – Gmail get message attachments

API connector – Gmail download message attachments

The post Download Gmail Attachment in SSIS appeared first on ZappySys Blog.

If you have need to integrate REST API in SSIS (i.e. fetch JSON, XML, CSV via HTTP call) and wondering how to Read API data in SSIS and Load into SQL DB (i.e. SQL Server, MySQL, Oracle) then you are at the right place.

In this article we will cover simple step by step approach how to achieve API integration in few clicks. For detailed introduction please refer to our previous article.

In next few sections we will cover the following SSIS Components to read data from API like a Table (Parse in Rows / Column). Our instructions will be mostly for JSON Format but concepts are same for other formats (i.e. XML / CSV).

	JSON Source Connector (Read from REST API, JSON File or OData Service): Use this dataflow component when you have to fetch data from REST API webservice like a table. This component allows you to extract JSON data from webservice and de-normalize nested structure so you can save to Relational database such as SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local JSON files or direct JSON string (Wildcard pattern supported too e.g. c:\data\file*.json).
	XML Source Connector (SOAP, File, REST) : Use this dataflow component when you have to fetch data from XML or SOAP webservice and consume data like a table. This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local XML files or direct XML string.
	CSV Source Connector (File, REST) : Use this dataflow component when you have to fetch data from File or REST API which returns data in CSV format . This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local CSV files or direct CSV string.

There are few more API components not discussed in this articles. They are REST API Task (no Parser – Just raw API calls) and Web API Destination (Write / Send Data To API / Delete rows). They both have different use case to check them see it fits in your need.

Video Tutorial

Step-By-Step – Read API data in SSIS using JSON / XML / CSV Source

In this section you will learn how to use JSON Source Adapter to Read data from API URL.

Load API data into SQL Server Table / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

API Authentication

In our step by step example we did not cover Authentication to keep things simple but in real world you have to pass some sort of authentication details to make API calls (e.g. UserID/Password or Secure Token). There are many ways to authenticate with API Services. ZappySys offer mainly 2 Connection Managers to cover most Authentication scenarios.

1. HTTP Connection Manager
2. OAuth Connection Manager

Using HTTP Connection Manager

If you have use case of using UserID / Password or any other method supported by HTTP Connection Manager then use below option (Below example is for Basic Authentication)

SSIS JSON Source – Passing Basic Credentials (Supply UserID / Password) using HTTP Connection

Using OAuth Connection Manager

If your API support OAuth Standard for Authorization then use OAuth Connection Type as below.

Generate Token for Google API using OAuth 2.0

API Pagination

Pagination is another important concept to understand when you call API using ZappySys Connector. If you ready many records from your API then by default API might not return all rows so you have to paginate until all records are read (Like a Loop in Programming). ZappySys makes it easy to paginate. Check this article to configure many different Pagination Modes

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Conclusion

REST API is becoming more and more popular each day. With that Data integration from RESTful API services going to be challenge. Luckily ZappySys SSIS PowerPack provides a great way to integrate any API in SSIS via simple drag and drop approach without coding. Try SSIS PowerPack for FREE  see how much time / money you can save and to integrate virtually any REST API.

The post How to Read API data in SSIS and Load into SQL Table appeared first on ZappySys Blog.

In our previous article, we see how to load data into Elastic Search using SSIS. Now let’s look at how to read data from ElasticSearch using SSIS and load response into SQL Server.

Elasticsearch is a powerful engine that allows you to store, aggregate, and, most importantly, search data in a very analytical way. In this tutorial, you will learn how to load Elasticsearch data to SQL Server with SSIS (part of SQL Server) and ZappySys PowerPack. We also see how to set the maximum result window of the index using the Rest API Task.

In a nutshell, this post will focus on how to Make Elasticsearch Search API call using SSIS.

Step-By-Step to Call Elasticsearch API call using SSIS

Read from Elasticsearch Search API call

Let’s make the Elasticsearch Search API call using the JSON source to get all the records of the index with pagination.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Dragging and dropping Data Flow Task into Control Flow

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to make Search API Call on the index.
   

   http://{elasticsearch-site-url}/{index}/{index-type}/_search

4. Now check Use credentials and Select Connection section press

   <New ZS-HTTP Connection>

   .
   

   SSIS JSON Source – Passing Basic Credentials (Supply UserID / Password) using HTTP Connection

5. Once the HTTP Connection Manager window opens configure the connection to your Elasticsearch instance:
   • Set Web Url, which points to your Elasticsearch instance.
   • Set Credentials Type to

     Basic - UserID/Password

      (or other appropriate authentication methods).
   • Finally, set User Name and Password:
     
     

     Configure SSIS HTTP Connection to connect to Elasticsearch

6. Now, select method as POST and set for and size value in the body to get data using pagination. Set the content type as JSON (application/json).
   

   {"from" : <%page%>, "size" : 100}

7. Furthermore, go to the pagination tab and configure pagination like the below screenshot makes sure we need to enter increment size the same as given in body size value therefor 100.
   

   Pagination Mode: POST data in the body

8. That’s it now select the desired filter and click on the Preview button.
9. That’s it we are ready to load the data in the SQL Server.

For more than 10000 rows

When you try to get more than 100000 rows you will get the next error message

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [11000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
            }
        ]
    },
    "status": 400
}

Here is how to configure your JSON Source to pull more than 10K rows using Scroll Method.

1. In Settings Tab: In the Body and URL rather than Index name use

   [$tag$]

2. In the filter type:

   $.hits.hits[*]._source

3. Uncheck Include Parent
   

   Elasticsearch example for more than 10000 rows

4. In Pagination Tab, Set Next Link as below

   $._scroll_id

5. Stop Indicator Attribute as below

   $.hits.hits[0]._id

6. Stop Indicator Value as

   regex=^$

   

   Elasticsearch pagination tab configuration

7. In Advanced Pagination Tab, Check Has Different Page Info and Enable Page Token
8. Enter Page Place Holders as below (change YOUR_INDEX_NAME)
   

   url=YOUR_INDEX_NAME/_search?scroll=10m|_search/scroll

9. First Page Body as below – Change query as needed else keep it match all to fetch all rows from Index
   

   { "size":1000, "query": { "match_all": { }  } }

10. Next Page Body as below 

    {  "scroll": "5m", "scroll_id": "[$pagetoken$]" }

    

    Configuration for Elasticsearch advanced pagination tab

Loading Elasticsearch API data into SQL Server / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Enable Deep Pagination Setting

By default index, the maximum result window size is 10,000 means you can paginate upto 10000 rows. If you ever want to get more than 10000 result then you have to perform the following steps to change default setting from 10000 to 500000 (that’s max).

We need to send PUT request to below URL:

http://{elasticsearch-site-url}/{your-index}/_settings

Need to pass below body as I am setting max result window size value as 500000.

{ "index" : { "max_result_window" : 500000 } }

And set content-type as JSON (application/json) and click on Test Request/Response and it will make the API call and set the max window size.

Rest API Task PUT Call

That’s it we have successfully made the call to set the maximum result window size of the index.

Common Errors

Deployment to Production

Conclusion

After all, we saw you How to Make Elasticsearch Search API call using SSIS JSON Source and load response into SQL Server. In this article, we have seen that how to make a setting call to set the maximum result window of the index. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File)
• Elasticsearch Search APIs

The post How to read data from ElasticSearch using SSIS appeared first on ZappySys Blog.

In our previous article we see How to read call REST API data in SSIS. Now in this article, we will see How to Make UPS API call in SSIS and load response into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call UPS REST API Call can be useful for any developer regardless of which programming language or tool set you use. We will also see how to configure POST API Call.

We will go through the steps to make Pickup Creation Request API call which Returns a PRN and Rate Result and we load it into MS SQL Server.

In nutshell, this post will focus on how to Make UPS REST API Call in SSIS.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of UPS API.

What is UPS?

United Parcel Service Founded in 1907, UPS is the world’s largest package delivery company and a leading global provider of specialised transportation and logistics services. Using advanced technology, access to global resources, and an integrated network of physical, technological, and human assets, UPS provides a powerful competitive advantage that can help you earn repeat customers and grow your business.

Where to get the UPS Credentials?

You can get Username and password by signup in UPS website and also need to get Access Key from UPS Developer Kit.

Call UPS REST API using SSIS

Let’s start with an example. We use SSIS JSON Source component to make the call to UPS API, we will make Pickup Creation Request and load response into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to make Pickup Creation Request API Call and select Method as POST, I am using Testing URL but we need to use production one for real time API call.
   Testing:
   

   https://wwwcie.ups.com/rest/Pickup

   Production:
   

   https://onlinetools.ups.com/rest/Pickup

   

   JSON Source Set URL

4. Furthermore, set below JSON string in the body and also replace your UPS username, password and access key in the placeholder, i used my dummy data you need to pass yours valid data.
   

   {
   	"UPSSecurity": {
   		"UsernameToken": {
   			"Username": "[YOUR USERNAME]",
   			"Password": "[PASSWORD]"
   		},
   		"ServiceAccessToken": {
   			"AccessLicenseNumber": "[ACCESS KEY]"
   		}
   	},
   	"PickupCreationRequest": {
   		"Request": {
   			"TransactionReference": {
   				"CustomerContext": "CustomerContext."
   			}
   		},
   		"RatePickupIndicator": "Y",
   		"TaxInformationIndicator": "Y",
   		"PickupDateInfo": {
   			"CloseTime": "1700",
   			"ReadyTime": "1000",
   			"PickupDate": "20191123"
   		},
   		"PickupAddress": {
   			"CompanyName": "EIVR TEST NON-ACCOUNT PL 3",
   			"ContactName": "EIVR TES PL",
   			"AddressLine": "12380 Morris Rd",
   			"City": "Louisville",
   			"StateProvince": "KY",
   			"PostalCode": "40201",
   			"CountryCode": "US",
   			"ResidentialIndicator": "N",
   			"Phone": {
   				"Number": "+48223803070"
   			}
   		},
   		"AlternateAddressIndicator": "N",
   		"PickupPiece": {
   			"ServiceCode": "003",
   			"Quantity": "1",
   			"DestinationCountryCode": "US",
   			"ContainerCode": "01"
   		},
   		"OverweightIndicator": "N",
   		"PaymentMethod": "05"
   	}
   }

   

   JSON Source set Body

5. Now click on Select Filter button to get the desire filter.
   

   JSON Source Select Filter

6. Finally click on Preview button to load the data.
   

   JSON Source Preview

7. That’s it we are ready to load UPS Pickup Creation Request API call response in SQL Server.

Load UPS REST API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to make UPS API call using SSIS JSON Source and load response into SQL Server. In this article we made Pickup Creation Request POST API call same way you can make other UPS API calls like Tracking API and Address Verification API and so on by passing relevant URL and JSON string. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • UPS Developer Kit

The post How to Make UPS API call in SSIS appeared first on ZappySys Blog.

In our previous article we Read data from Twilio API in SSIS . Now in this article, we will see example of how to Call SurveyGizmo API in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call SurveyGizmo API Call can be useful for any developer regardless of which programming language or tool set you use. we will also see how to store access token and secret in the variable and pass it in the URL using variable.

We will go through the steps to make Survey GetList REST API call which Get a list of all of your surveys and we load it into MS SQL Server.

In nutshell, this post will focus on how to Make SurveyGizmo REST API Call in SSIS.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of SurveyGizmo API.

What is SurveyGizmo?

SurveyGizmo takes data out of dashboards and puts it into the hands of people who take action. Through the systems they use every day. SurveyGizmo is an online survey solution that makes it easy for businesses of all sorts to create and conduct surveys, polls, quizzes, and questionnaires.  Reaching out to potential and existing customers, managing organizational and employee relationships and conducting academic researches has become convenient through SurveyGizmo.

• Click here to read SurveyGizmo Rest API Documentation.

Call SurveyGizmo REST API using SSIS JSON Source

Let’s start with an example. We use SSIS JSON Source component to make the call to SurveyGizmo API, we will read list of all available temporary lists and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Now let’s add variable for SurveyGizmo Access Token and Secret and Set it in as a value.
   

   SSIS Add Variable

4. Now double click on JSON Source and enter the following URL with variable as placeholder to get list of your surveys.
   

   https://restapi.surveygizmo.com/v5/survey?page=1&resultsperpage=100&api_token={{User::v_api_token}}&api_token_secret={{User::v_api_token_secret}}

   

   JSON Source : URL with variable as placeholder

5. Let’s configure the pagination, go to Pagination tab and select mode as URL Parameter and set the indicator as

   Page

    and configure it like below screen.
   

   JSON Source Pagination

6. You can refer to this article for more details on pagination here Understanding REST API Pagination in SSIS / ODBC Drivers
7. Now click on Select Filter button and select the desire filter from the window and click on Preview to get the data.
   

   JSON Source Preview

8. That’s it we are ready to load all SurveyGizmo Surveys data in SQL Server.

Load SurveyGizmo REST API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to extract information from REST API such as SurveyGizmo and load into SQL Server. We also learned techniques like How to use variable as placeholder in URL and configure the pagination. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • SurveyGizmo Rest API

The post Call SurveyGizmo API in SSIS appeared first on ZappySys Blog.

In our previous article we see How to call REST API in SSIS . Now in this article, we will see example of how to Make Twilio API Call in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call Twilio API Call can be useful for any developer regardless of which programming language or tool set you use. we will also see how to configure HTTP basic connection.

We will go through the steps to make Accounts REST API call which Returns a list of all Accounts resources and we load it into MS SQL Server.

In nutshell, this post will focus on how to Make Twilio REST API Call in SSIS.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Twilio API.

What is Twilio?

Twilio is a cloud communications platform as a service company based in San Francisco, California. It’s allows software developers programmatically to make and receive phone calls, send and receive text messages, and perform other communication functions using its web service APIs.

• Twilio API Documentation : Click here to read Twilio Rest API Documentation.

Make Call of Twilio REST API using SSIS

Let’s start with an example. We use SSIS JSON Source component to make the call to Twilio API, we will read list of all available temporary lists and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to get multiple Account resources and create the HTTP Connection use the same url in the http connection also.
   

   https://api.twilio.com/2010-04-01/Accounts.json

   

   Json Source HTTP connection

4. Now click on Select Filter button and select the desire filter from the window.
   

   Json Source Select Filter

5. Go to Pagination Tab and select mode as Response attribute Mode – Read next page information from response and set up the next page filter.
   

   JSON Source Preview

6. You can refer to this article for more details on pagination here Understanding REST API Pagination in SSIS / ODBC Drivers
7. Now select the desire options to get the data and click on preview, as here i want parent and missing child also i checked both options and clicked on Preview. if you don’t want to get parent details you can uncheck it.
   

   JSON Source Preview

8. That’s it we are ready to load all Twilio accounts resources in SQL Server.

Load Twilio REST API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to extract information from REST API such as Twilio using HTTP Connection and load into SQL Server. We also learned techniques like How to create HTTP Connection and configure the pagination. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • Twilio Rest API

The post Read data from Twilio API in SSIS appeared first on ZappySys Blog.

In our previous article we see How to read call REST API data in SSIS. Now in this article, we will see How to Make MailPlus OAuth 1.0a REST API Call in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call MailPlus Oauth 1.0a REST API Call can be useful for any developer regardless of which programming language or tool set you use. We will also see how to configure OAuth1.0 connection.

We will go through the steps to make temporary lists API call which Returns a list of all available temporary lists and we load it into MS SQL Server.

In nutshell, this post will focus on how to Make MailPlus OAuth 1.0a REST API Call in SSIS.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of MailPlus API.

What is MailPlus?

MailPlus, founded in 2004, is the number one ESP (Email Service Provider) in the Netherlands. MailPlus is an award winning solution that enables companies to send out their online communication, such as newsletters, easily and professionally.

Where to get the MailPlus API Key and Secret?

After activation of the MailPlus REST API in a MailPlus account you can get unique Key and Secret from there. The MailPlus REST API uses Oauth 1.0a (one leg, see Oauth 1.0a (One Leg) ). A consumer key and secret are required to access your MailPlus account. These can be created in your MailPlus account. The Key and Secret are always available via the startpage of a MailPlus account. Click on the tab “MailPlus Settings” and then “MailPlus Rest API”. In MailPlus eCom an authentication is already available.

Call MailPlus Oauth 1.0a REST API using SSIS

Let’s start with an example. We use SSIS JSON Source component to make the call to MailPlus API, we will read list of all available temporary lists and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to get list of all available temporary lists and create the OAuth 1.0 Connection.
   

   https://restapi.mailplus.nl/integrationservice-1.1.0/templist

   

   MailPlus Oauth 1.0a Connection

4. Finally click on Preview button to load the data.
   

   MailPlus Preview Data

5. You can find more MailPlus API Lists here on MailPlus Interactive API Documentation link.
6. That’s it we are ready to load all available temporary lists in SQL Server.

Load MailPlus REST API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to extract information from REST API such as MailPlus REST API using OAuth1.0 Connection and load into SQL Server. We also learned techniques like How to create OAuth1.0 Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • MailPlus API

The post How to Make MailPlus OAuth 1.0a REST API Call in SSIS appeared first on ZappySys Blog.