Imagine this situation, you or users of your product wants to consume some REST API in ETL / BI / Programming Tools using ODBC / JDBC Driver interface but there are no ODBC / JDBC Driver available for that API and you dont even know coding? If you are in a situation No Problem !!! In this post you will learn how to create a custom ODBC Driver for API (i.e. REST / SOAP / OData) without writing code… and access in most BI / ETL / Reporting tools using Industry Standard such as ODBC Interface….. Yes you heard it right… 🙂

Generally speaking, creating an ODBC Driver requires a lot of coding effort and that’s why most vendors don’t invest in providing ODBC Driver and let you consume their API using code only. They assume that your company / team has really sharp C# / JAVA / C++ coders available 🙂 …. but we all know that coding is not cheap and not easy to maintain in long run.

Basically, the approach mentioned in this article requires you to create an XML File (or call it Connector File) and then consume it using API Driver (ODBC) So let’s get started!

API Connector File – The Concept

ZappySys has developed a highly flexible API Connector Framework which you can use to build API Connector Files which can be used by API ODBC Driver in different apps. This API Connector Framework is so flexible that it can handle pretty much any API out there.

Here are a few example connector files we built using the same framework. Don’t worry about various parts in those connector files for now, you might not need to implement all those. We will start simple and explain to you advanced use cases later.

Example Connector Files

• Youtube
• Stripe
• Jira
• Shopify

Basic Example of API Connector File

Here is a very simple example of some API to read Customers and Orders. Keep in mind most features are omitted for simplicity.

Below connector file shows few concepts which we will discuss later in this article. Such as API Base URL (i.e. Service URL), Authetication, Pagination, EndPoints, Metadata, Parameters, Templates, Tables so on.

1. Open Notepad
2. Copy / Paste below XMl in notepad and save as NorthwindConnector.xml
3. Then you can Create a new ODBC DSN as mentioned here and use our connector file for Testing (Use Saved Connector File Option on new DSN wizard).

NorthwindConnector.xml

<?xml version="1.0"?>
<ApiConfig Name="Northwind" Slug="northwind-api" Desc="Sample connector for Northwind OData Service" Logo="https://www.odata.org/assets/ODataLogo-96.png">
  <ServiceUrls>
    <ServiceUrl Name="V3_API" Url="https://services.odata.org/V3/Northwind/Northwind.svc"/>
    <ServiceUrl Name="V4_API" Url="https://services.odata.org/V4/Northwind/Northwind.svc"/>
  </ServiceUrls>


  <!-- Implement supported way to Authenticate API calls -->
  <Auths>
    <Auth Name="ApiKeyAuth" Type="Http" TestEndPoint="get_customers" ConnStr="CredentialType=Token;TokenAuthHeader=X-APIKEY;Password=[$ApiKey$]">
      <Params>
        <Param Name="ApiKey" Label="Enter your API  Key" Secret="True"/>
      </Params>
    </Auth>  
  </Auths>

  <!-- Reusable Templates (i.e. API Pagination Properties) -->
  <Template>
    <EndPoint Name="PaginatedCall">
      <Params>
        <Param Name="NextUrlAttributeOrExpr" Type="Property" Value="$.['odata.nextLink']"/>
      </Params>    
    </EndPoint>
  </Template>

  <!-- EndPoints -->
  <EndPoints>
    <EndPoint Name="get_customers" Template="PaginatedCall" Url="/Customers?$format=json" Filter="$.value[*]" Method="GET">
      <OutputColumns>
        <Column Name="CustomerID" Label="Id" DataType="DT_WSTR" Length="20"/>
        <Column Name="CompanyName" DataType="DT_WSTR" Length="100"/>
      </OutputColumns>
    </EndPoint>

    <EndPoint Name="get_orders" Template="PaginatedCall" Url="/Orders?$format=json" Filter="$.value[*]" Method="GET">

      <OutputColumns>
        <Column Name="OrderID" Label="Id" DataType="DT_I4"/>
        <Column Name="OrderDate" DataType="DT_DBTIMESTAMP"/>
        <Column Name="CustomerID" DataType="DT_WSTR" Length="20"/>
        <Column Name="Notes" DataType="DT_NTEXT"/>
      </OutputColumns>
    </EndPoint>
  </EndPoints>

  <!-- Tables (with SELECT, LOOKUP UPDATE, INSERT, DELETE operations. For Demo we have used just SELECT EndPoint) -->
  <Tables> 
    <Table Name="Orders" SelectEndPoint="get_orders"/>
    <Table Name="Customers" SelectEndPoint="get_customers"/>
  </Tables>
</ApiConfig>

Using API Connector File in Apps

Once API connector file is created

Use in ODBC Apps

Click here to see an example – Using API connector file in ODBC Apps (ODATA). For other popular apps check this page and click on desired app.

Here is how it may look like if we import the previous connector file in API Driver UI. Notice how we selected the V3 Base URL as API. Also, see how various connector file elements translated to Driver UI…. very cool right ?

Using Custom API Connector File – New ODBC DSN – ZappySys API Driver

Using Custom API Connector file in ZappySys API ODBC Driver

Here is the output from Preview Tab

Preview Data from Custom API Driver – Using ZappySys API Driver

Use in JAVA Apps

Click here to see an example – Using API connector file in JAVA based Apps (It uses ZappySys Data Gateway as Bridge rather than direct ODBC call in Java becuase JAVA doesnt support ODBC directly)

Use in SQL Server (T-SQL code)

Click here to see an example – Using API connector file in SQL Server (It uses ZappySys Data Gateway as Bridge rather than direct ODBC call in SQL Servr becuase of some known issues of direct ODBC)

Use in SSIS Package

Click here to see an example – Using API connector file in SSIS Packages. It uses native API Source / API Destination components.

Create / Edit Connector file using Wizard from API Driver UI

If you like to edit your file previously created using Wizard then you can do the following steps.

1. Assuming you have already setup ODBC DSN for API Connector File we previously created.
2. Open Odbc UI (search for “ODBC” in the start menu and select “ODBC Administrator”)
3. Double-click on the Data source you like to edit
4. On the Properties Tab > Go to Advanced Tab
5. Click on Customize – Current Connector File (If you wish to create a new API connector file then you can click on Create New Connector File Button instead)

Create / Edit Custom API Connector File – From API ODBC Driver UI

Create Connector file from SSIS JSON / XML Source UI Settings

If you are an SSIS user and you have configured JSON Source or XML Source then you convert those UI settings into a basic API connector file by going on API Template and clicking on Create New Connector File button like below.

Create a new Custom API Connector File – From SSIS JSON / XML Source UI

Create API Connector file from scratch

So far you have understood the basic ideas about connector files. Also explored creating / editing API Connector File using Wizard in ODBC / SSIS.

Now let’s look at concepts of creating connector file in detail.

Basic Properties (Name, Slug, Logo)

Coming Soon

Service URL (API Base URL)

Coming Soon

Authentication

Coming Soon

Templates

Coming Soon

Parameters

Coming Soon

EndPoints

Coming Soon

Metadata / OutputColumns

Coming Soon

Tables

Coming Soon

Contact Us – We will build Custom API Driver for you

If you are still not sure how to use all these on your own or you do not have time/experience… no worries. Contact us and we will provide you an estimate for building a custom API Driver / connector.

In the contact form above provide us following details (or as much as you can provide)

• Which connector do you like to build?
• How are you planning to consume that data (i.e. which apps e.g. Power BI, Tableau)?
• Your use case (e.g. read customer data from Zoho Invoice System)
• Include table / data feed names you prefer to read from (e.g. Read Invoices, Read/Write Customers)

Conclusion

In this article, we explored how easy it is to create your custom ODBC Driver for API using a powerful API Connector Framework. Download ODBC PowerPack and build your own API Driver in a few clicks.

The post How to create custom ODBC Driver for API without coding appeared first on ZappySys Blog.

The article shows how to download images from a web page using SSIS. Sometimes we need to download the images from a web page. In this post, we will show how to do this.

Prerequisites

Before we perform steps listed in this article, you will need to make sure following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Optional (If you want to Deploy and Schedule ) – Deploy and Schedule SSIS Packages

Steps-by-step process to download images from HTML using SSIS

Use REST API task to get the HTML body

1. Drag and drop the REST API Task from SSIS toolbox, select the html page you want and save it in a variable.

Select the page you want to get the images

2. Go to Response Settings. Check the option Save the response content. In Save Mode, select Save to File. In the option Enter File Path, write the path for the html file.

Save the page in a file

Parse the image with Regular Expression parser task

3. From the SSIS toolbox drag and drop Regular Expression Parser Task on the Control flow designer surface.

4. The next step is to save the source path from the images. You need to use Regex and here are two example you can use, in this page Regex101 you can check more details about the expressions we are using:

Expression 1:

<img.*?src="(.*?)"{{*}}

src="([a-z\-_0-9\/\:\.]*\.(png|jpg|jpeg|gif|png))"{{*}}

Regex expression to get the image code from the page

Read the image source with CSV source in order to download images from a Web page

5. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.

6. Double click on the DataFlow task to see DataFlow designer surface.

7. From the SSIS toolbox drag and drop CSV source and insert the variable you are using from the previous task

Use CSV source to read the variable

Get the image name, image full path and the destination folder to download images from a Web Page

8. From the SSIS toolbox drag and drop Derived Column transform to remove the HTML code and get the image name.

Expression for the Image name:

(DT_WSTR,1000)REVERSE(LEFT(REVERSE(REPLACE(REPLACE(Column1,"src=\"",""),"\"","")),(FINDSTRING(REVERSE(REPLACE(REPLACE(Column1,"src=\"",""),"\"","")),"/",1) - 1)))

Remove the HTML code:

REPLACE(REPLACE(Column1,"src=\"",""),"\"","")

Add a new column and remove HTML code

9. Now we will drag and drop another Derived Column, this one is for the file path for the images and validate if the image URL is valid. If the image URL has “http” in the stringm then it is valid otherwise you need to add the rest of the URL. Here is our example for the filePath and validate the URL image. If everything is OK, we can send the image URL.

File path: (DT_WSTR,2500)@[User::vDestinationFolder] + ImageName

Validate image URL: ((FINDSTRING(Column1,"http",1) > 0) ? Column1 : (DT_WSTR,2500)("https://www.urlexample/image/" + Column1))

Note: @[User::vDestinationFolder] is a variable we used for the local path

Add a new column and verify the HTML path

Send the request and save the image In the folder TO download images from a Web Page

10. Now we will drag and drop a web API destination and select the column in the input column to URL. If the request is right, we can save the image in the local file.

Send a request for all images you get

11. Now drag and drop an Export Column transformation and select the ResponseText from the request and the file path.

Select the image response and the local file path

12. Finally, we use a trash destination to close the flow.

The final result

Conclusion

If everything is OK, you will be able to download the images from your HTML page. To do that, we read the list of URLs from a list. Then we get the name of the images using expressions. Then we add a path for each image to store them. Finally, we save the images.

The post How to download images from a web page using SSIS appeared first on ZappySys Blog.

In our previous post we saw how to read API data in SSIS. This post we will cover how to call NetSuite RESTlet to read / write data in SSIS or ODBC Apps. For demo purpose we will use SSIS PowerPack REST API Task but concepts and many UI elements remain same for ODBC JSON Driver too. This article assumes you know how to write Netsuite RESTlet and you have obtained Credentials to call RESTlet from extranal app.

What is NetSuite RESTlets

NetSuite added a really nice feature to expose business data and workflows as API by writing Javascript functions which they call it RESTlets. You can secure and customize these restlets as per your need.

You can automate CRUD (Create /Read / Write / Delete) operations in NetSuite by writing special functions in JavaScript and expose those functions as API URL which can be called by any external Systems (i.e. SSIS Packages , ODBC Apps like Power BI, Informatica, SSRS).

RESTlet can be secured few ways such as OAuth 1.0 and OAuth 2.0 authentication. We will cover both approach briefly in this article. You can choose any one as per your need

Creating NetSuite RESTlet / Obtain OAuth Credentials (Client ID and Secret)

Below video shows how to obtain necessary credentials to call Netsuites API (RESTlet).

At mark 4:30 seconds it will show you How to use OAuth 2.0 for calling RESTlet

Create NetSuite RESTlet Connection

There are few ways to authenticate NetSuite RESTlet. If you are using ZappySys Products then it makes it super easy to use either approach. Lets look at both approaches.

Using OAuth 2.0 (Recommended)

As per above video if you obtained Client ID and Secret for OAuth 2.0 you can follow below steps to configure OAuth connection to call RESTlet.

What you need to use OAuth 2.0.

• Your Account ID (it goes in URL)
• Your Client ID and Secret (Obtained from Netsuite Portal) – See above video Mark 04:30
• Register Redirect URL in NetSuite portal (we recommend you use https://zappysys.com/oauth as call back Url for Automatic Code extraction)

Steps to configure OAuth Connection

1. Right click on Connection Panel and choose Choose “New Connection..”. Select ZS-OAUTH type
   
   
   

   Create new SSIS OAuth API Connection Manager

    

    

2.  Enter following things
   ————————-
   On General Tab
   ————————-
   ClientID and Secret: <get it from NetSuite>
   Authorization URL: https://<accountID>.app.netsuite.com/app/login/oauth2/authorize.nl
   Token URL: https://<accountID>.suitetalk.api.netsuite.com/services/rest/auth/oauth2/v1/token
   Scope: restlet
   

   Configure NetSuite OAuth 2.0 Connection for API call

    

   ————————-
   On Advanced Tab
   ————————-
   Callback / Redirect URL:  https://zappysys.com/oauth  (this must be entered same way in NetSuite Portal)

   

   OAuth2 Connection – Callback / Redirect URL

3. Now go back to General Tab and click Generate Token button
4. Follow login screen and try to finish the flow. At the end it will generate Tokens and populate on UI
5. Click OK to save

Using OAuth 1.0

Another approach is use OAuth 1.0 (Older version). If you have enabled OAuth 1.0 in NetSuite then follow below steps (Create connection is same as before).

1. Setup General Tab and Advanced Tab as below.
   On ==== General Tab====:
   Enter ClientID, ClientSecret, AccessToken, AccessToken Secret
   On ==== Advanced Tab ====:
   Check Do Not Include Empty Key/Value Pairs
   Select Signature Method as HMAC-SHA256
   For Extra Token Attribute enter realm=<your-account-number>_SB1
2. Click OK to Save

Call NetSuite RESTlet in SSIS

1. Create a new SSIS Package
2. Drag REST API Task from SSIS Toolbox
3. Double click the Task to configure
4. Select URL from Connection
5. From Connection dropdown select OAuth connection we created in the earlier section (either OAuth 1 or OAuth 2)
6. Enter the URL as below. Assuming you copied URL from NetSuite Replace script ID and account number
   

   https://<account-number>-sb1.restlets.api.netsuite.com/app/site/hosting/restlet.nl?script=<scriptid>&deploy=1

   Enter <account-number>
   Enter <scriptid>
7. Change Method to POST if your script supports Parameters (Else you can supply id=xxxx and recordtype=xxxxx in the URL for GET request)
8. If its POST request then chnage ContentType to ApplicationJSON (application/json)
9. Enter your Input in the body. In below example our RESTlet takes sql query as input so we enter just one parameter named query but you can have different parameter(s)
   

   Syntax:  { param1_name : <value> , param2_name : <value> ....  }

10. Now click Test see everything works
    

    Calling NetSuite RESTlet in SSIS (Using REST API Task)

Conclusion

We saw how easy it is to call NetSuite RESTlets or other NetSuite API in SSIS / other BI Apps using ODBC Drivers provided by ZappySys. Download SSIS PowerPack and try your self see how easy it is to consume virtually any API in SSIS.

The post How to call NetSuite RESTlet API in SSIS / ODBC appeared first on ZappySys Blog.

In this post, you will learn how to read / write ManageEngine ServiceDesk data in SSIS (Drag and drop approach without any coding).  We will use SSIS JSON/ REST API Source to extract data from Servicedesk API and use SSIS Web API Destination to write data to Zoho.

We will look at step-by-step instructions to read ServiceDesk  data (e.g. Request, Problems, Contacts) and load into SQL Server Database Table.

Authentication ServiceDesk REST API in SSIS (Use OAuth 2.0)

Very first step to call ServiceDesk API is you need to register OAuth App in Zoho Portal. Don’t get confused if you see Zoho Portal for ServiceDesk a Product from ManageEngine is in fact Zoho subsidiary (in other words Zoho is the parent company). ServiceDesk has On-Premises version too but this article is only about Cloud version.

Register Zoho OAuth App and get ClientID/ Secret (*** MUST READ ***)

Check this article to register your OAuth App and obtain Client ID  /Client Secret

Create ServiceDesk API Connection in SSIS

Once you create Zoho OAuth App now its time to call some Zoho API. Let’s create the OAuth Connection first. In the next section we will see how to load data in SQL Server.

1. Right click on Connection Panel and choose Choose “New Connection..”. Select ZS-OAUTH type
   

   Create new SSIS OAuth API Connection Manager

2. Configure your OAuth connection General Tab as below
3. OAuth Provider: Custom
4. OAuth Version: OAuth2
5. Client Id , Client Secret
6. Enter One Scope Per Line (For more info on scope read here) . For write you can change READ to WRITE or use ALL as per the help link.
   

   SDPOnDemand.requests.READ

7. Auth URL : https://accounts.zoho.com/oauth/v2/auth
8. Token URL:  https://accounts.zoho.com/oauth/v2/token
9. Configure Advanced tab as below
10. Enter Callback URL : https://zappysys.com/oauth
11. Enter Extra Attribute for /Auth : access_type=offline&prompt=consent
    This is needed to obtain refresh_token (if you ommit this then only access_token returned which cannot be renewed once expired after 1hr)

    

12. Now go back to general tab and Click Generate Token. Close the Popup and use Full browser instead if you see prompt.
13. Click OK to save the connection UI

Read data from ServiceDesk API in SSIS JSON Source

In this section we will learn how to read data from ServiceDesk using SSIS JSON Source. We will use OAuth connection created in earlier section.

Now lets look at step by step example of reading ManageEngine ServiceDesk data using SSIS (e.g. Requests).

ManageEngine provides JSON API for read action so we will Use SSIS JSON Source for this purpose.

Configure JSON Source

So below steps shows example of reading from ManageEngine Requests API

1. Open SSIS Package
2. Drag data flow task from Control flow SSIS Toolbox
   

   Drag data flow from SSIS Toolbox

3. Go to data flow and drag ZS JSON Source from SSIS Toolbox
   

   Drag and drop SSIS JSON Source (REST API or File)

4. Double click JSON Source to configure. Enter URL as below. You can supply few more parameters. Click Here to get full list of parameters and usage. In below example we are reading records from Leads Module but you can replace Leads to something else (e.g. Accounts). Refer to above help link for available Modules.
   

   https://sdpondemand.manageengine.com/api/v3/requests

5. Check Use Credentials. Select OAuth connection manager we created in the previous section.
6. Add the Accept header in HTTP Headers.
   

   Accept: application/vnd.manageengine.sdp.v3+json

7. Under filter options tab, enter Array Filter as  

   $.requests[*]

8. Now click Preview to see your data
9. Click OK to save UI. In the next section, we will see how to setup pagination and incremental extract of Servicedesk records. After that we will see how to load data into SQL Server Table.

Read ManageEngine ServiceDesk Plus data in SSIS (Cloud Version) – Zoho API

ServiceDesk API Pagination (Read all Requests)

Just like most API ServiceDesk also uses pagination. See below to learn how to configure pagination in JSON Source for ServiceDesk API.

1. Double click JSON Source
2. Change URL to include two things (Page size (how many rows per page and start row). Note <%page%> must match with Page Num Indicator you going enter in the next screen
   

   https://sdpondemand.manageengine.com/api/v3/requests?input_data={"list_info":{"row_count":100,"start_index":<%page%>}}

3. Go to Pagination Tab and select URL Path Mode
4. For Page Num Indicator : Enter

   <%page%>

5. For Increment By : Enter

   100

6. Page Start Number select New variable call it StartIndex and start default value 1

Once you configure this way now your API will return all records until last page is detected. For more information on pagination check this article

How to paginate ManageEngine ServiceDesk API

Loading ServiceDesk data into SQL Server Table / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Write data to ServiceDesk using OAuth 2.0

If you like to load data from any source (i.e. SQL Server) to Zoho CRM then you can use Records API (POST). Use a combination of the following Transforms  / Components. Later in this article we have some more examples (Older API) which may give you one more ways to load data into Zoho (Use of Template Transform).

Basically you have to build POST Body using JSON Generator Transform or Template Transform and then use that data in Web API Destination to call POST API request to push data to ServiceDesk. Basically screnshot below is just an example using some other Zoho API but you get the idea.

	Web API Destination
	JSON Generator Transform
	JSON Parser Transform

Create JSON for POST request (Single dataset pattern)

Insert data to Zoho Table using SSIS (Lead, Account Modules) – Bulk Load using POST API call

Write data to ServiceDesk using SSIS (Import SQL Server Table to ServiceDesk)

Now you know how to read data from Zoho CRM using JSON API Source next step is to load data to Zoho CRM. For writing data we will use following three components

	Web API Destination
	Template Transform
	XML Parser Transform

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Conclusion

Manageengine  Servicedesk API provides a great way to automate data read/write operations. However, to call Service API  you have to use SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Servicedesk API call via a  simple drag and drop approach without coding. Try  SSIS PowerPack for free and call virtually any REST API in a few clicks.

The post Read/Write ManageEngine ServiceDesk Data in SSIS (Zoho API) appeared first on ZappySys Blog.

In our previous posts we saw how to read and write Salesforce Data using SSIS. In this post we will see how to read Salesforce Metadata in SSIS using JSON Source.

Step-By-Step

In this section you will learn how to use ZappySys JSON Source Adapter to extract data from Salesforce REST API (JSON Format). You can call pretty much Any REST API to fetch data from salesforce but our example for this post would be read metadata (i.e. table list) from salesforce (we will call sobjects to get all table information) .

So lets get started.

1. In case you missed the previous section, You need to Download and Install SSIS ZappySys PowerPack.
2. Once you finished first step, Open Visual Studio and Create New SSIS Package Project.
3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
4. Double click on the Data Flow task to see Data Flow designer surface.
5. From the SSIS toolbox drag and drop JSON Source on the Data Flow designer surface.
   
6. Now double click JSON Source enter any valid REST API URL like below. Make sure you used correct instance name. Using wrong instance name might slow down connection or fail your request. If you are not sure then just login to salesforce portal using credentials you like to use and observe browser URL.. It will have instance number.
   

   https://na34.salesforce.com/services/data/v20.0/sobjects/Account/describe/

7. Check Use Connection and select New ZS-Salesforce Connection from the list.
   

   Create new REST API Connection in SSIS (For JSON, XML or CSV Source)

8. Configure your salesforce connection
   
9. Select or type filter. For example in this case you will type 

   $.sobjects[*]

10. Click OK to Save UI and attach your source to some target like OLEDB Destination and run your data flow to test.
11. Here is full UI after all configurations.
    

    Read Salesforce Metadata in SSIS (Get Table List Example – SOObject REST API Call)

How to read Salesforce Table Fields Metadata

If you have need to read fields and its information for tables then use below steps

1. Use almost identical steps as previous section. Except two things. Our URL and Filter would be different.
2. Configure your API URL as below
   

   https://YOUR-INSTANCE.salesforce.com/services/data/v20.0/sobjects/YOUR-TABLE/describe/

   Example:
   

   https://demo.salesforce.com/services/data/v20.0/sobjects/Account/describe/

3. Configure Filter as a 

   $.fields[*]

Get Salesforce Table Fields Metadata using REST API call in SSIS

The post How to read Salesforce Metadata in SSIS using REST API (JSON Source) appeared first on ZappySys Blog.

In the previous article, we learned how to copy settings from curl to ZappySys products, and in this article, we will continue on how to migrate Postman API call to SSIS / ODBC PowerPack. Postman is a very popular client tool to test API calls. Many API providers document how to call their API using Postman. In this article we will focus on how to use the same settings in ZappySys Products to implement API integration in SSIS or other apps like Power BI, Informatica, Excel, Tableau, SSRS… many ODBC apps.

When you are connecting to an API you may have examples of how to do that in Postman. If you know how to do it in Postman, you are one step away from making it work in SSIS PowerPack and ODBC PowerPack (JSON, XML, CSV Drivers, and other). Most of the examples will show how to do that specifically using ODBC PowerPack and JSON Driver, but since ODBC PowerPack UI is very similar to the connectors of SSIS PowerPack, it will be quite clear where to use it in SSIS PowerPack as well. In the examples where UI differs significantly, both – ODBC and SSIS PowerPack – examples will be presented.

Prerequisites

In this article we assume you meet these requirements:

When migrating Postman API call to SSIS PowerPack

1. Postman installed.
2. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
3. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
4. ZappySys SSIS PowerPack is installed (download it).

When migrating Postman API call to to ODBC PowerPack

1. Postman installed.
2. ZappySys ODBC PowerPack installed (download it).

The Basics

This section shows how to migrate simple options such as URL, HTTP Request Method, and HTTP Headers.

How to configure the URL and HTTP Request Method

Migrate Postman API call to SSIS / ODBC (API URL setting to ODBC Driver (GET / POST) )

How to configure HTTP Headers

Migrate Postman HTTP Headers setting to ODBC Driver

Posting data

In this section, you will find how to post data in ZappySys products. You will learn how to post different formats of data, upload files and send multi-part data.

How to configure POST raw body

Migrate Postman Body / Content-Type setting to ODBC Driver

How to configure POST JSON body

Migrate Postman Body / Content-Type setting to ODBC Driver

Use this table to determine what kind of Body Content Type to configure:

How to configure POST form/x-www-form-urlencoded data

Migrate Postman Body (x-www-form-urlencoded) setting to ODBC Driver

How to configure multipart form POST data

Migrate Postman Multi-Part File Upload setting to ODBC Driver (Upload File)

Authentication

This section will cover all common authentication methods, starting from using API Key to OAuth 2.0.

How to authenticate using API Key in HTTP headers

Authentication – Migrate Postman API Key (via header) setting to ODBC Driver

How to authenticate using API Key as a parameter in URL

Authentication – Migrate Postman API Key (via URL query parameter) setting to ODBC Driver

How to authenticate using Bearer Token

Authentication – Migrate Postman Bearer Token setting to SSIS / ODBC Driver

How to authenticate using Basic Authentication

In ODBC PowerPack

Authentication – Migrate Postman Basic Auth setting to SSIS / ODBC Driver (Login using Userid / password)

In SSIS PowerPack

In SSIS PowerPack use the same configuration as above, just create ZS-HTTP Connection Manager using instructions below:

Authentication – Migrate Postman Basic Auth setting to SSIS JSON / XML Source (Login using Userid / password)

How to authenticate using OAuth

In ODBC PowerPack

1. Configure main options:
   

   Authentication – Migrate Postman OAuth setting to SSIS / ODBC Drivers ( JSON / XML / CSV API) – Client ID / Secret / Scope / Auth and Token Url (Client Credential Grant Type, Password Grant Type, Code Authorization Grant Type)

2. Configure Callback/Return Url:
   

   Authentication – Migrate Postman OAuth setting to SSIS / ODBC Drivers ( Callback / Redirect URL )

If you are using a known, public API service, you may try locating it in the OAuth Providers list, which will save you from configuring Authorization and Access Token URLs:

For simplicity and if your company policy allows it or at least for testing purposes, you can use a Default OAuth Application created by ZappySys:

In SSIS PowerPack

In SSIS PowerPack use the same configuration as above, just create ZS-OAUTH Connection Manager using instructions below:

Authentication – Migrate Postman OAuth setting to SSIS JSON / XML Source

How to authenticate using AWS Signature

In ODBC PowerPack

Authentication – Migrate Postman AWS API call setting to SSIS / ODBC Drivers ( JSON / XML / CSV API call)

In SSIS PowerPack

In SSIS PowerPack use the same configuration as above, just create ZS-OAUTH Connection Manager using instructions below:

How to authenticate using NTLM Authentication

In ODBC PowerPack

In SSIS PowerPack

In SSIS PowerPack use the same configuration as above, just create ZS-HTTP Connection Manager using instructions below:

Conclusion

This article was quite self-explanatory, but if you have any questions or you have an authentication method that is not listed here and you don’t know how to proceed, please, contact us and we will try to help you: https://zappysys.com/support/.

The post How to migrate Postman API call to SSIS / ODBC PowerPack appeared first on ZappySys Blog.

Connect to YouTube API in SSIS (OAuth Connection)

Very first step to call any Google API is to create an API project for Google Service and then enable APIs you like to call in Console. In our case we need to enable YouTube API.

Check this article to learn how to create API Project and obtain Client ID and Client Secret to connect to YouTube API. For ease of use use Default Application provided by ZappySys but we still recommend using your own Custom Application  on OAuth Connection.

Before we try our first YouTube API example, we recommend you to get familiar with YouTube API here. Once you done with above steps you can perform the following steps to call YouTube API in SSIS.

1. Open existing SSIS Project or Create new One in Visual Studio (i.e. SSDT)
2. Create a new SSIS Package. Right click in the Connection Manager Panel and click “New Connection…”.
   
3. When Dialog box opens select ZS-OAUTH connection type.
   

   SSIS Connection Managers List: Select O-AUTH Connection Manager

4. Once OAuth UI opens, Configure Google connection like below.
   1. Select Service Type as Google
   2. You can select Default App for ease of use or Select Custom Application (Enter Client Id and Secret Obtained using these steps )
   3. In the Scope Enter below – One scope per line for calling correct YouTube API. Refer to YouTube API documentation incase you need extra permissions. For read only operation we just need below scopes. Last scope is only needed if you need to use YouTube Analytics Reports
      

      https://www.googleapis.com/auth/youtube
      https://www.googleapis.com/auth/youtube.readonly
      https://www.googleapis.com/auth/yt-analytics.readonly

5. Click Generate Token to obtain Refresh Token (You may see Login Prompt, and then Grant Permission Confirmation Screen)
   

   Connect to Google API using OAuth 2.0 (YouTube API Example)

6. Click OK to save UI.

Call YouTube REST API Example

Now lets look at how to call YouTube API REST API Task this task is more suitable to call GET, POST, DELETE, PUT requests without parsing data in Rows and Columns. If you need to parse data and load into Database table then see next section (Use JSON Source)

1. From SSIS Toolbox look for items starting with “ZS”. Drag and Drop [ZS Rest API Task] to Designer Surface.
   
2. Double click it to configure like below.
   1. Select URL From Connection option
   2. Select OAuth connection created in previous section.
   3. Enter URL like below (For example get all playlists). Replace xxxxxxxxxxx with your own channel ID (Its usually found in URL when you visit Youtube Channel Homepage)
      

      https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&channelId=xxxxxxxxxxxxx

   4. Click Test Request Button
3. Here is how it looks like
   

   Call Youtube API in SSIS Example (Dynamic URL)

That’s it we have successfully configured Connection for YouTube API in SSIS. In the next section we will see how to use this connection and read various data from YouTube API.

Read YouTube Playlists in SSIS

Once we done creating OAuth Connection Manager we can move forward to read YouTube data inside Data Flow.

Configure SSIS JSON / REST API Source

1. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
2. Double click on the Data Flow task to see Data Flow designer surface.
3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   
4. Double click JSON Source and enter URL as below (Use variable or Hardcode Channel Id). maxResults controls how many rows you want to get in each response. Rep
   

   https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&maxResults=50&channelId={{User::varChannelId}}
   --OR-- 
   https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&maxResults=50&channelId=xxxxxxxxxxxxxxxxxxxxxx

5. Check Use Credentials and select existing OAuth connection or Create New one
6. Select Array Filter (Items node) or type

   $.items[*]

     as below.
   

   Get Playlists for YouTube Channel

7. That’s it you can now Click Preview to see sample data. In the next section we will configure Pagination to read all records if you have more than 100 rows.

Configure YouTube REST API Pagination

Most REST APIs dont return data in one big response. So you have to make sure you implement looping / pagination to read more records. Luckily ZappySys supports many different Pagination Settings. Here is how to configure YouTube REST API Pagination.

1. Go to Pagination Tab found on JSON Source and select following 3 settings
2. Select Pagination Mode as

   Response Attribute Mode

3. Enter Next Link as

   $.nextPageToken

4. Enter Suffix for Next URL as

   &pageToken=<%nextlink%>

YouTube REST API Pagination Settings

Read YouTube Playlist Videos in SSIS

So in previous section we saw simple API call to read all playlists from YouTube. Now lets see how to get information about Playlist Videos.

For that everything should remain same as previous section except change URL as below. Where xxxxxxxxxxxxxxxxxxxxx is your Playlist ID obtained from Previous Call.

https://www.googleapis.com/youtube/v3/playlistItems?part=snippet,id&maxResults=50&playlistId=xxxxxxxxxxxxxxxxxxxxx

Get all Videos for YouTube Channel

If you want to get all video name and ID for specific channel then use below URL. Click here to read more. Replace xxxxxxxxxxxxxxxxxxx with your Channel ID.

https://www.googleapis.com/youtube/v3/search?channelId=xxxxxxxxxxxxxxxxxxx&part=snippet,id&order=date&maxResults=50

Read YouTube Video Views, Likes, Dislikes (Extended Videos)

Another common request from YouTube API is get Video Information such as Views, Likes, Dislikes, Comment Count etc. For that you can use this api . However the issue is this endpoint requires you to pass Comma separated Ids of Video. You can only pass 25 Ids in one requests so if you have 200 videos for which you need to get information then you have to make 8 requests (25 Ids in each Request). In below screenshot you can see we used Fiddler to debug web requests.

So here is how high level process to fetch extended information for all Videos in your channel. Full SSIS Package is attached here (SSIS 2012 Format).

1. Get all videos for your channel by calling (Assuming you will paginate in below request.
   

   https://www.googleapis.com/youtube/v3/search?channelId=xxxxxxxxxxxxxxxxxxx&part=snippet,id&order=date&maxResults=50

2. Build Video Id groups for 25 Ids from above result and submit requests. For example if above request returns 200 Videos then your request to fetch extended information may look like below.
   

   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID1,ID2,ID3........ID25
   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID26,ID27,ID28........ID50
   ........
   ........
   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID176,ID177,ID178........ID200

3. Parse response coming from each Request Above…. Each Response will give you details about 25 videos you requesting.

Here is the flow of our complete data flow to read extended information for all videos. You can read extended information one by one too but it wont be good idea if you have many videos.

We have used Following Components to achieve Bulk Mode

• JSON Source (REST API Source)
• CSV Parser Transform
• Web API Destination
• JSON Parser Transform

Get YouTube Video View Clicks, Likes, Dislikes (Bulk Mode for all videos)

Read data from YouTube Analytics / Reporting API (Metrics, Dimensions Report)

So far we have seen basic YouTube APIs to extract your Channel information. However there is another very powerful API endpoint to query custom reports from YouTube. Check this YouTube Analytics API endpoint. In this section we will learn how to extract useful information from YouTube using Analytics API.

Step-By-Step

Let’s look at an example to read total views, estimated watch time, average view duration and some other metrics by date for channel you have access.

1. Go to Data Flow, Drag ZS JSON Source from SSIS Toolbox
2. Double click to configure.
3. Check Use Credentials. Select OAuth connection we created in previous section.
4. In the URL field, Enter Below Sample URL. Change Parameters as per your need or leave it default.
   For example:
   ids=channel==MINE   (where MINE can be replaced by other channel id or keep it as MINE to get data for your own channel)
   metrics (can be changed from any valid list here)
   dimensions (can be changed to any valid list from here)
   startDate (Must be YYYY-MM-DD format)
   endDate (Must be YYYY-MM-DD format)
   

   https://youtubeanalytics.googleapis.com/v2/reports?ids=channel==MINE&metrics=views,estimatedMinutesWatched,averageViewDuration,averageViewPercentage,subscribersGained&dimensions=day&startDate=2019-03-01&endDate=2019-04-01

5. Change Filter to

   $.rows[*]

6. Go to 2D Array Transform tab.
   Select Method as

   Simple 2-dimensional array

   EnterColumn Filter as 

   $.columnHeaders[*].name

   

   YouTube Analytics API Advanced Transformation

7. That’s it now Preview your data. See below over all steps.
   

   Read data from YouTube Analytics API (Custom Reports using Dimensions, Metrics Query)

Build YouTube Analytics / Reporting API Query (API Playground)

With Analytics API You can extract many interesting data. You can use Test Tool found on this page to build some interesting queries. Scroll to Common Use Cases section on that link. On the right side you can select Credentials Type as OAuth and click Execute.

https://developers.google.com/youtube/analytics/reference/reports/query

YouTube Analytics / Reporting API Explorer (Testing API Calls)

Loading YouTube API Data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Download Sample SSIS Package

SSIS Package is attached here (SSIS 2012 Format).

Conclusion

In this article we saw how to extract information from REST API such as YouTube REST API using OAuth 2.0. We also learned techniques like Pagination, JSON Parsing, Request Batching using multiple components to achieve full scale API integration. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

The post How to read YouTube API data in SSIS (Videos, Channels, Playlists) appeared first on ZappySys Blog.

In this post we will learn how to read Freshdesk data in SSIS using ZappySys JSON / REST API Source. We will see how to authenticate, paginate / parse and load Freshdesk data into SQL Server Table in just few clicks. If you don’t use Freshdesk and considering using Zendesk instead then check our previous post on how to Read Zendesk data in SSIS.

Freshdesk is a cloud based / managed service like Zendesk to for Support / Customer service.

Getting Started

So let’s get started. We will see step by step instructions to configure Freshdesk API connection, setup JSON Source to read data from Freshdesk and finally load data into SQL Server table. We will use List Tickets API in this article for demo purpose but read other useful information on that page if you are new to REST API.

Setup Freshdesk REST API Connection

Very first step to call any Freshdesk API in SSIS is to configure HTTP connection (Basic Authentication – Using UserID/ Password).

1. Open Visual Studio (i.e. SSDT) to create new SSIS Package
2. From Control Flow SSIS Toolbox drag REST API task
3. Select URL From HTTP Connection Option from Drop down
4. Click New HTTP Connection
5. Configure HTTP connection like below.
6. Enter any Valid URL (This will be ignored if you Override URL on REST API Task)
7. Select Basic Authentication (Use User ID and Password)
   

   Configure Freshdesk REST API Connection in SSIS – Basic Authentication (User ID / Password)

8. Click OK to close HTTP Connection
9. On Main UI of REST API Task, check Use Direct URL and enter below URL (Change YourDomain Part)
   https://{your-domain}.freshdesk.com/api/v2/tickets
10. Click Test Request / Response. If you see OK status then you have successfully called Freshdesk API. That’s all.
    

    Calling Freshdesk REST API in SSIS ( Read Tickets example)

In the section we will see how to Parse REST API response and load data into SQL Table.

Read Freshdesk data in SSIS

Now let’s look at steps to configure SSIS JSON Source to read from Freshdesk REST API.

1. Drag Data Flow task from toolbox. Double click to go to Data Flow designer
2. Drag ZS JSON Source (For File / REST API)
3. Configure JSON Source like below.
4. Check Use Connection option and select HTTP connection created in the previous section.
5. Enter API URL from which you like to read data. In out example we want to read tickets created after 2019-01-01 so we will call API like below. You can skip ?updated_since=2019-01-01 if you like to read all tickets.
   

   https://{your-domain}.freshdesk.com/api/v2/tickets?updated_since=2019-01-01

6. You can also Use Variable Placeholder in URL to make things dynamic like below
   

   https://{your-domain}.freshdesk.com/api/v2/tickets?updated_since={{User::MyDateVar,yyyy-MM-dd}}

7. Click Preview to see sample data.
   

   Read Freshdesk data in SSIS – Fetch Tickets example (With REST API pagination)

8. Configure Pagination like below. Basically you have to select Pagination Mode = URL Parameter Mode. Enter Page num indicator as 

   page

   

   Configure REST API pagination (URL Parameter Mode)

9. Thats it. Now click OK to save UI.

Load Freshdesk data into SQL Server Table

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

In this article we saw how easy it is to read data from any REST API service like Freshdesk REST API and load into Relational DB like SQL Server. Try SSIS PowerPack for FREE and explore many other 70+ Tasks /Components not discussed in this article.

The post Read Freshdesk data in SSIS – REST API Call appeared first on ZappySys Blog.

Any time you send HTTP Request according to HTTP Standard using tools like ZappySys REST API Drivers or SSIS Connectors you may receive web response along with HTTP Status Codes. Dealing with REST / SOAP API calls you may face errors and you may find it challenging to understand generic error message from server along with some Numeric value (Called HTTP Status Code).  In this article we will try to decode certain status codes and discuss various way to solve it.

Example of HTTP Error and Status Code

Before we dive deep on this topic of HTTP Errors Status Codes let’s check an example HTTP Error with Status Code 400 (BadRequest). As you notice we passed some bad data in Request Body to produce this error. As you may see in below screenshot there are few parts we need to configure to call REST API request.

1. URL  (and optionally parameters in URL)
2. Request Method  (i.e. GET / POST)
3. Request Body (Optional – Not needed for GET request but for POST /PUT API requests you may need to pass)
4. Request Content Type ( Optional – i.e. application/xml . This is not needed for GET Request)
5. Request Headers (Optional – Many tools generate default headers behind the scene. In our example we passed only two headers but see how many more added automatically in below Raw View obtained by tools like Fiddler )

Sample Request:

POST
https://accounts.google.com/o/oauth2/token

>>>> HEADERS <<<<<

User-Agent: ZappySysApp/1.0.2019.10310
Content-Type: application/xml
Accept: */*
Cache-Control: no-cache
Host: accounts.google.com
Content-Length: 13
Expect: 100-continue
Accept-Encoding: gzip, deflate
Connection: Keep-Alive

>>>> BODY <<<<<

some bad data

Sample Response:

HTTP/1.1 400 Bad Request

{
  "error": {
    "code": 400,
    "message": "Invalid JSON payload received. Unexpected token.\nsome bad data\n^",
    "status": "INVALID_ARGUMENT"
  }
}

Here is a screenshot of SSIS REST API Task which can be used for calling XML / JSON API to automate ETL workflows.

Example of Calling REST API which returns HTTP Error Status Code 400 (Bad Request) – Invalid POST Body / Syntax Error

Common HTTP Error Status Codes / Reasons / Possible Fix

Status codes can be anywhere between 100 to 505. Most of them documented here also here (RFC2616 – HTTP / 1.1). Out of these status codes we will discuss few most common ones in this article and how to fix them.

Microsoft has also provided great explanation for most common HTTP Status Code here.

Status Code 400 – Bad Request

Possible Reason(s):

• One or more URL Parameter is wrong
• Request body may be wrong (For example: Either data is invalid in Body or Syntax is invalid – such as JSON Property name misspelled or missing closing brackets)

Possible Fix:

• Make sure you passing URL Parameters correct way (Refer your API help file or some sample code)
• For POST / PUT request check your Parameter names, Data format, JSON / XML syntax (Refer your API help file or some sample code)
• If you want to validate JSON / XML Syntax use some FREE online validators for JSON  or XML

Status Code 401 – Unauthorized

Possible Reason(s):

• You did not pass userid / password or session token
• Your credentials are invalid or session token is expired
• Some tool may not save Sensitive Information unless you set Save Sensitive Information option. For example in SSIS if you set PackageProtectionLevel= DoNotSaveSensitive then password will not be sent along with your API request.

Possible Fix:

• Make sure you passing credentials either directly via HTTP Headers or URL Parameter
• For ZappySys Products use HTTP or OAuth connection manager to pass credentials securely without worrying about manual HTTP Headers
• If you configured PackageProtectionLevel= DoNotSaveSensitive in SSIS then Check this and this  article to learn how to pass sensitive information in SSIS at runtime / design time.

Status Code 404 – Page Not Found

Possible Reason(s):

• You entered invalid URL
• URL is intranet only and you trying to access from outside network

Possible Fix:

• Check your URL and correct it so its not throwing 404 error when you test in Browser (It must return data or throw some other error – E.g. 400 – Bad request or 401-Unauthorized)

Status Code 405 – Method Is Not Allowed

Possible Reason(s):

• You selected invalid Request Method (see #2 in above screenshot)

Possible Fix:

• Select correct Request Method. Refer to your API see it supports the method you selected. For example You selected POSt method to update some data but your API is expecting PUT method rather than POST.

Status Code 411 – Length Required

Possible Reason(s):

• You selected request method as POST or PUT but did not pass any body

Possible Fix:

• Make sure you supply some value in the Request Body field as per API specification. See #4 in the screenshot above.

Status Code 415 – Media Type Is Not Supported

Possible Reason(s):

• You did not pass correct Content-Type Header or your API was expecting different content-type setting than what you passed.

Possible Fix:

• Pass HTTP Header Content-Type with correct value (Check API documentation). Typically for JSON API its

  application/json

    and for XML SOAP API its

  text/xml

• If you are using Zappysys REST API Drivers or SSIS Connectors you will see Content Type Dropdown on UI see #4 in above screenshot. Change it to correct setting. If your content type not in the dropdown then you can set direct HTTP Header
  Example =>

  Content-Type: application/json

    see #5 in screenshot.

Debugging HTTP Web Requests

If you want to make sure what’s being passed to web server you can use this tool (Fiddler) and debug your web requests. It will show you all parts of your HTTP Request / Response (i.e.  Headers, Body, URL Parameters)

The post Understand HTTP Status Code and Fix Common Errors in REST API appeared first on ZappySys Blog.

In our previous article we saw how to POST data to REST API using few different ways. Now let’s go one step further and discuss another common scenario to batch REST API requests in SSIS. For efficient data transfer many API provides you to submit multiple records in a single request. In this article we will look at few ways to batch our requests using SSIS JSON Generator Transform and  SSIS Web API destination.

Method 1 – Batch records using SSIS JSON / XML Generator Transform

First let’s look at common way to group multiple records and generate single JSON or XML document for desired batch size. We will also see another technique where you can assign Unique BatchID to each document.

For example below screenshot shows how to product one JSON document for every 10 input rows. We have also use BatchID column on the root level. This will use use BatchID from first row of that batch. You can use XML Generator Transform if you like to produce XML Document rather than JSON.

To generate Unique BatchID you can use function like newid() in SQL Server.

SELECT newid() as BatchID,CustomerID,CompanyName FROM Customers

Using SSIS JSON Generator Transform with Batch Option (Multiple Records in a single document)

Here if full flow to batch your JSON Requests and Submit to REST API.

POST data to REST API in Bulk Mode (Batch multiple rows)

Method 2 – Using SSIS Web API destination Batch Mode with Header / Footer

Another method of batching is use Web API destination and turn on Batch mode.  You can use Fiddler to debug your web requests incase you are not sure how its generating and sending POST requests after this setting.

Configure SSIS Web API Destination for Bulk API call (Insert multiple rows)

Method 3 – Batch CSV records using CSV Generator Transform

So far we looked at how to batch JSON and XML format. Now let’s look at scenario where you have to POST CSV formatted data to REST API endpoint.

Assume that you have an API endpoint where you can submit data in multiple batches. Also you have few additional requirements

1. First row of First batch must have column names (any other batch must not contains CSV Header)
2. In each Batch we want to submit maximum 5 rows as per API call
3. Each batch must contain Unique Batch Number in URL

Here is high level flow.

Generate CSV using Batch Setting , Send data to REST API in multiple batches, Debug REST API in Fiddler

Include CSV Header in First Batch Only

Now let’s break down the problem and find out how to do. First problem is we need to include CSV Header only in First batch. To achieve this we use old school trick where we output column names as data row… and when we generate CSV in the next section we will turn off Column Headers option.

If you are consuming records from SQL Server or any other RDBMS you can write following style source query. If you don’t use relational source then use native UNION Transform in SSIS.

select 'CustomerID' as Column1, 'OrderID' as Column2
UNION ALL
select top 12 cast(CustomerID as varchar(1000)), cast(OrderID as varchar(1000)) from Invoices

Batch records and output CSV data

Now next step would be to generate CSV data but in batches… so in our example we have to output size of 5 rows per batch. For this perform the following steps

1. Drag ZS CSV Generator Transform (Included in v2.8 and Higher)
2. Connect source to CSV Generator Transform
3. Now before we edit in UI mode, right click and select Properties. Change following 3 properties  (You can also change QuoteAroundValue and Dimiliter if needed). In future version batch setting will be visible in UI mode too.
   1. ArrayBatchSize  = yourbatchsize (in our case it was just 5 rows per batch)
   2. OutputMode = SingleFileFormat
   3. FirstRowsHasColumnNames  =False
4. Now double click to configure ZS Generator Output
5. Right click on Mappings folder node and Add Elements > Select Multiple Columns  > Check columns you like to output
6. Click OK to Save UI

Generate CSV data in SSIS (Batch Output Setting)

Generate unique URL with Batch using Script Component

Now next task we have to achieve is generate Unique Batch Number. Put that Batch ID in URL like below…

http://httpbin.org/post?batch=0
http://httpbin.org/post?batch=1
http://httpbin.org/post?batch=2
.....

1. For this drag SSIS Script Component and select Transformation Mode
2. Now go to Inputs and Outputs tab > Select Output Columns > Click Add Column > Name it as URL > Configure DataType as DT_WSTR , Length = 500
3. Now got Script Tab and Click Edit. Remove old code and enter following code (Assuming You selected C#). This code will generate new URL with Unique batch ID each time. You can use this same technique to generate Unique row number too (in that case rather than DT_WSTR type you can use DT_I4 type)

   [Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
   public class ScriptMain : UserComponent
   {
       private static int _batch;
       public override void Input0_ProcessInputRow(Input0Buffer Row)
       {
           Row.Url = string.Format("http://httpbin.org/post?batch={0}",_batch);
           //If you have to send GUID rather than Integer for Batch ID then use below code
           //Row.Url = string.Format("http://httpbin.org/post?batch={0}",System.Guid.NewGuid());
   
           _batch = _batch+1;
       }
   }

4. Click OK to Save script.

Output Unique URL or Number using SSIS Script Transform

Configure Web API Destination to POST CSV Data

Now final step is to configure SSIS Web API Destination to POST CSV data to API.

1. Drag ZS Web API Destination and connect Source Transform to Web API destination.
2. Double click to configure
3. Select Connection Manager
4. Once connection setup, select Input Column for URL ( This will be same column from Script Transform)
5. Select column for Body (e.g. ZS_CSV_OUT)
6. Click OK to save

POST data to REST API in multiple batches using SSIS Web API destination

Debug Web API Requests using Fiddler

Now we ready to run entire flow. Once you execute data flow, observe Input rows before CSV Generator Transform and after. CSV Generator transform must produce less rows before we now grouping multiple rows in a single batch.

Another important aspect to play with API request is to debug it, We recommend you to to look at this article about using Fiddler to debug web requests.

Download Sample Package

Here is the Sample SSIS Package (2012 format) – Rest API_Batching

Conclusion

In this post, we saw how easy it is to handle complex REST API  / CSV /JSON / XML operations in SSIS. We configured REST API POST Operation using Batching Method. Hope this post helps you to achieve your API integration without coding.  Feel free to download SSIS PowerPack and try many other components not discussed in this article.

The post How to batch REST API requests in SSIS (Bulk Operation) appeared first on ZappySys Blog.