In our previous article we saw how to call REST API or SOAP Web Service in SSIS. In this article we will only focus how to perform HTTP POST in SSIS using RESTful approach.

HTTP Protocol supports many methods also referred as verb (e.g. GET, PUT, POST, HEAD, DELETE). Based on this verb, target server (API server) can perform desired action. For example GET is usually used for read only data fetch (similar to select query in SQL). POST is usually used to create new record or update existing data.

For any HTTP operation you can submit parameters/data in 3 ways.

via URL query string
via HTTP Headers
via Body (Only valid for operations such as POST, PUT)

HTTP POST in SSIS

Traditionally whenever SSIS developers wanted to perform HTTP POST operation in SSIS they had to write C# Code using Script Task. This is no more case after ZappySys developed easy to use drag and drop REST API Tasks and Connectors. As of now there 4 main connectors/tasks which supports HTTP POST

1. SSIS Web API Destination
2. SSIS REST API Task
3. SSIS JSON Source
4. SSIS XML Source

SSIS PowerPack offers following two methods to POST data from your database to target API URL (e.g. SQL Server to API POST in JSON format).

Method-1 : HTTP POST using Web API destination (In Data Flow)

This method allows you to read data from any source and generate POST data and then submit to API endpoint using SSIS Web API Destination. Using this destination you can configure URL and API Body (e.g. JSON/XML document you want to submit) for each request. Check following screenshot to get idea how API POST works

Step-By-Step : Video Tutorial

You can also extract http response (i.e. JSON or XML) from Web API Destination and parse it into Rows and Columns using JSON Parser Transform like below. If you want to handle errors gracefully without stopping the execution then use error output (Drag red arrow from Web API destination and attach to downstream (Make sure you set Redirect for Error and Truncate both when prompted on attach)

Method-2 : HTTP POST using SSIS REST API Task (In Control Flow)

REST API Task support many advanced options too incase you want to tweak certain aspects of your POST request.

Here is the screenshot of HTTP POST in JSON format using ZappySys REST API Task

Perform HTTP POST operation using SSIS REST API Task – Send data in JSON content type

Parse response into rows/columns using SSIS JSON Source

If you wish to POST data and parse JSON response into rows/columns then you can use SSIS JSON Source for this purpose. JSON Source has great JSON Parser which supports parsing very large JSON  (stored in File or API URL or Direct string) into Rows and Columns. This is useful if you wish to save response into target such as SQL Server or CSV file.

Post JSON data to HTTP URL and parse response into columns and rows

Parse response into rows/columns using SSIS XML Source

If you wish to POST data and parse XML response into rows/columns then you can use SSIS XML Source for this purpose. XML Source supports parsing XML data (stored in File or API URL or Direct string) into Rows and Columns. This is useful if you wish to save response into target such as SQL Server or CSV file.

Passing credentials for Web Request for HTTP POST

Unfortunately there is no one standard for passing credentials to server but good thing is ZappySys REST API Task comes with many options to secure your request and pass credentials (Userid/Password or API Key or OAuth Token).

Basic Authentication

Most common approach of passing credentials is userid/password. Check this article how to use Url from connection if you wish to pass credentials using Basic authentication (i.e. Userid/password).

OAuth Authentication (1.0 or 2.0)

If your API support OAuth protocol then check these articles how to use OAuth with ZappySys REST connectors

Passing Parameters in POST Body (Make it dynamic)

SSIS PowerPack support variable placeholders for many editable text fields (usually indicated by variable icon or pencil icon).

If you wish to pass parameters in your data used in Body then use variable placeholder (e.g. {{User::varMyDate}}. When you edit POST Body you will see <<Insert Variable>> option.

Here is the complete list of available placeholder functions

Upload file using Multi-Part POST

If you have need to submit data to server from file then check this link (its shows few ways to upload file using HTTP POST)

Most common Content-Type for POST operation

There are many data formats which you can submit to server using HTTP POST but below is the most common data formats you will ever use for most of scenarios.

Data format is usually determined by server based on Content-Type header you include along with your request.

HTTP POST Form data – submit key/value pairs (application/x-www-form-urlencoded)

This is the most common data format for HTTP POST. Its also famous because any type you submit form on any website, most likely it will use below method for submitting your data to server. As you see its key/value pairs separated by ampersand [&]. Also if some special character found then it needs to be URL encoded (e.g. @ is replaced by %40)

POST https://myserver.com/submit
Content-Type: application/x-www-form-urlencoded

firstname=bob&lastname=smith&email=bob%40mycompany.com

HTTP POST JSON data (application/json content-type)

JSON is becoming very popular data format so if you are dealing with newer API then most likely you will have to submit your POST request using this content type like below. Assume that you creating new customer record then your request may look like below.

POST https://myserver.com/api/customer
Content-Type: application/json

{
 "firstname" : "bob",
 "lastname" : "smith",
 "email" : "bob@mycompany.com"
}

HTTP POST CSV data (text/plain content-type)

When API POST required data in CSV format then you have to specify this content type. Below is the example of HTTP POST with CSV data.

POST https://myserver.com/api/customer
Content-Type: text/plain

firstname,lastname,email
bob,smith,bob@mycompany.com

HTTP POST XML data (application/xml content-type)

When API POST required data in XML format then you have to specify (application/xml) content type. Below is the example of HTTP POST with XML data.

POST https://myserver.com/api/customer
Content-Type: application/xml

<?xml version="1.0"?>
<data>
 <firstname>bob</firstname>
 <lastname>smith</lastname>
 <email>bob@mycompany.com</email>
</data>

HTTP POST Binary data (Upload file)

If you are POSTing large amount of data from file (i.e. uploading file) to your API endpoint then check this article about How to Upload file using Multi-Part Upload

HTTP POST Raw data (no content type)

Do not pass any content-type header to send raw request

Advanced Read / Write Scenarios / API Error Handling

Now let’s look at some advanced scenario where ýou first have to call one API (GET or POST) using JSON Source or XML Source and then POST data to some other URL using Web API Destination.

Once you get response after POST/GET parse those records for further processing along with error handling. Such API calls can be achieved using following pattern.

Using Template Transform to build JSON or XML for POST Body

Check this article to learn more about some techniques to build JSON or XML for API POST Call (e.g. SOAP )

Encoding for POST Body

When using application/x-www-form-urlencoded content type (POST Form data) your server may require you to encode certain characters in property name or value using URL Encoding technique. As you see below key/value pairs separated by ampersand [&]. Also if some special character found then it needs to be URL encoded (e.g. @ is replaced by %40). There are many online tools found to encode or decode data using URL encoding technique.

firstname=bob&lastname=smith&email=bob%40mycompany.com

Troubleshooting / Debugging HTTP POST issues using Fiddler

If you wish to see what’s being passed to server via any client tool or when you perform POST request ? Use tool called fiddler. Check this article to learn how to use Fiddler for debugging purpose.

Conclusion

As we enter into cloud first world you will face more and more need to integrate various RESTful API into your ETL workflows. SSIS PowerPack comes with complete toolset to automate any API calls (Read, Write or Parse). Try it for free here to discover many hidden features not discussed in this article.

The post HTTP POST in SSIS – Send data to Web API url appeared first on ZappySys Blog.

In this post, you will learn how to Parse Cookies from a Response and Pass Cookies to any Web Requests. SSIS PowerPack v1.8.2 introduced a new feature that allows sending and receiving cookies with HTTP Web Request/Response.

We will use the SSIS REST API Task to parse cookie values and then send cookies along with an HTTP Web Request. Cookies are beneficial for maintaining session state across multiple web requests.

Understanding Cookies

Let’s look at an example of how Cookie works. A cookie is a way of communication. Which means you can pass values to the Server or the Server can pass values to you.

Getting Cookies from Server (Web Response)

Many web APIs or sites send you cookies. If you are using a Browser, then it automatically saves for later use. The most common example is login pages. When you authenticate, the server sends cookies back via the Set-Cookie header. The server may send one or more Set-Cookie headers.  When you use the SSIS REST API Task, you can map such a value on the Response Tab. However, one crucial difference is that the REST API Task will send only one Set-Cookie header, combining all cookies in Key1=Value1; Key2=Value2 … so on. You can use Cookie Tab to Parse each Cookie and save to a separate variable (Explained in the next section)

Web Request:

POST http://mysite.com/login
Content-Type: application/x-www-form-urlencoded

userid=SOME_USER_NAME&password=SOME_PASSWORD

Web Response:

Set-Cookie: SitePreference=MobileVersion
Set-Cookie: SessionId=12345
Set-Cookie: LoginToken=AAA111222; Expires=Wed, 09 Jun 2021 10:18:14 GMT
...
...

Passing Cookies to Server (Web Request)

The user can pass one or multiple values in the form of Cookies to the server. To pass Cookie (s) to the Server, you have to send the  Cookie header as follows

Web Request:

GET http://mysite.com/securepage.html
Cookie: LoginToken=AAA1112222; SessionId=1234
...
...

Reading Cookies from Web Response using SSIS

SSIS REST API Task supports reading cookies in three different ways.

Method-1 (for SSIS / ODBC): Dynamic Token with Retain Cookie Setting (Using Cookie from Login)

In the new version of the HTTP Connection UI, you can find Retain Cookie settings as shown below.

Dynamic token configuration

Dynamic token – sending the request

If you are not receiving a Token from the response and only want to use Cookies, then set the following way (Enter Fake Regular expression (i.e., word like “SOME-FAKE” is fine )

Dynamic token – save the response

Method-2 (for SSIS): Reading / Parsing Multiple Cookies (Save to SSIS variables)

If you like to save cookie value(s) into separate SSIS variable(s), then you can use SSIS REST API Task >> Cookie Tab to configure Mappings. Perform the following steps.

1. Configure REST API Task with necessary information (e.g., URL, Headers, Body)
2. Go to the Cookie Tab. In the Mapping Grid, enter each Cookie Name and map it to an SSIS Variable, as shown below. When you run the SSIS Package, you will see values stored in the variable.

REST API task – Cookies tab

Method-3 (for SSIS): Read as raw cookie string

Sometimes you may need to see all cookies in key/value pairs and save them to a variable. If that’s the case, then use the Response setting tab and map the Set-Cookie Response Header to some SSIS Variable.

REST API task – Raw Cookie

Final Result – cookies

Method-4 (for SSIS): Read Cookies  as CookieContainer

Now, let’s look at how to use the CookieContainer Variable to save the Session across multiple API calls. All you have to do is create an Object datatype variable and select the same variable across various tasks.

In the example below, the first task is “Set Cookies”.It’s similar to a Login call in a traditional API call, where you get cookies back from the server. Call web url “http://httpbin.org/cookies/set?k2=val2&k1=myval1”, this generates some cookies and places them into a variable called “varCookies” (Must be Object datatype)

In the next step, we call the URL “http://httpbin.org/cookies” and pass the previously captured cookies from “varCookies” (Set on the Cookies tab). And the 3rd step displays the response from the URL (this demo URL returns all supplied cookies in JSON format).

SSIS REST API Task – Send Cookies with HTTP Web Request, Store Cookies in Variable

Pass Cookies to Web API (for SSIS / ODBC)

Now, let’s look at how to Pass Cookie (s) along with your Web Request

To pass a cookie, you have to use the Cookie HTTP Header. You can pass multiple cookies in a single header as below

Cookie : Key1=value1; Key2=Value2

You can use an SSIS variable to pass a dynamic value, as shown in the screenshot below. (e.g. MyCookieName={{User::MyVariable}} )

REST API task – Cookies inside the header

The post Working with HTTP Cookies in SSIS or ODBC appeared first on ZappySys Blog.