In this blog post you will learn how to call Amazon AWS API using SSIS (virtually any API) without a single line of code (No more JAVA, C#, Ruby, Python). Yes you heard it right 🙂 . If you are a SSIS / ETL Developer or even coder everyone loves drag & drop interface. SSIS has many advantages over other approaches such as Programming SDKs, Command Lines. Main advantage is ease of use, security and long term maintenance without learning expensive coding approach.

In this post we will use following components to show various possibilities to implement Amazon AWS API integration scenarios inside your ETL workflows.

	JSON / REST API Source (REST API, JSON File or OData Service): Use this dataflow component when you have to fetch data from REST API webservice like a table. This component allows you to extract JSON data from webservice and de-normalize nested structure so you can save to Relational database such as SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local JSON files or direct JSON string (Wildcard pattern supported too e.g. c:\data\file*.json).
	Web API Destination (POST data to API URL) : Use this dataflow component when you have to call API inside Dataflow (see POST to URL). Possible Use case: Call Lambda function with various parameters for each input record found in database.
	REST API Task : Use this task when you don’t want to pull REST API data in tabular format but want to call rest API for POST data to server, DELETE data from server or things like download HTML page, extract Authentication tokens etc where you not necessarily dealing data in tabular format. This task also allows you many other options such as saving RAW response into variable or file.
	XML Source (SOAP, File, REST) : Use this dataflow component when you have to fetch data from XML or SOAP webservice and consume data like a table. This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local XML files or direct XML string.

Prerequisites

Before we do hello world demo for calling Amazon AWS API, you will need to make sure following prerequisites are met.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Access to valid AWS credentials (Access Key, Secret Key for your IAM User). Click here to learn more about IAM users and Access Key/Secret Key
4. Make sure SSIS PowerPack is installed. Click here to download.

Step-By-Step Example-1 (Call AWS API)

Now lets call some simple GET API call using SSIS REST API Task.

1. Install  SSIS PowerPack (Skip this step if you already installed SSIS PowerPack.
2. Open Visual Studio and create new Integration Services Project
3. Open SSIS Package and check your SSIS Toolbox you will see many tasks/components starting with ZS
4. From Control flow SSIS Toolbox Drag & drop ZS REST API Task.
5. Double click the REST API task to configure it.
6. On REST API Task change URL Access mode drop down to URL from Connection
7. Now in the connection dropdown click New ZS-OAUTH connection and configure connection as below
   1. Once you see OAuth connection dialog box change Provider Type from Custom to Amazon AWS API (v4)
   2. In the ClientId enter your AWS Access Key
   3. In the ClientSecret enter your AWS Secret Key
   4. Click OK to save connection
      

      Configure SSIS OAuth Connection – Use Amazon AWS API Provider, Enter Access Key, Secret Key

8. On REST API Task change few more settings as below
   1. Enter API URL you like to call (In our case we will use S3 API (Simple Storage Service). We assume you have ListBucket permission to make this call. If you dont have such permission try to get Full path of File (choose small file) . You have to tweak API url to adjust Service Type, Region,Bucket, Path
      

      https://s3.us-east-1.amazonaws.com
      -- OR -- Use below (list files) if you have single bucket permission---
      https://s3.us-east-1.amazonaws.com/YOUR-BUCKET

   2. Click Test Request. If you have valid Permission and setup looks ok then you will see Response window like below. Using this technique you can call any API to execute AWS operations (E.g. start EC2 VM, Create SQS Queue, Call Lambda Function, Drop or Update resource)
      

      Calling Amazon AWS API Using SSIS REST API Task Example. (Call any AWS API without SDK. such as EC2 API, SQS API, Lambda API, Redshift API, DynamoDB API)

Step-By-Step Example-2 (Loading data from AWS API to SQL Server)

Now lets do more interesting scenario. We will call AWS S3 API to get S3 File list from Bucket. After extract we will save that list to SQL Server Table. Since Amazon S3 API is XML based API we will use ZappySys SSIS XML Source. For JSON based API use JSON Source instead. XML Source / JSON Source both can parse API response into Rows and Columns so you can easily store it into SQL Server. Now lets see how to do this.

1. From Control flow SSIS Toolbox Drag & drop Data Flow Task
2. Double click Data Flow Task. From SSIS Toolbox Drag & drop ZS SSIS XML Source.
3. Double click the XML Source to configure it.
4. In the URL text box enter API URL like below to list S3 Files for specified bucket (Change YOUR-BUCKET to your own name)
   

   https://s3.us-east-1.amazonaws.com/YOUR-BUCKET

5. Now check Use Credentials and Select same Amazon API connection we created in previous example.
6. Click on Select Filter (This step allows us to flatten the XML hierarchy. Select the node which is Array icon like below. If prompted to treat selected node as array click Yes.
   

   Configure SSIS XML Source – Get Amazon S3 File List from specific bucket (AWS API)

7. Click Preview to see your data. Click OK to save.
   

   Preview Data in XML Source – Amazon S3 File List (Calling AWS API)

8. Now attach your XML source to target like OLEDB Destination to load data to SQL Server or other Target (e.g. Oracle, MySQL)
9. Execute SSIS Package to load data from Amazon AWS API to SQL Server.
   

   SSIS Package – Read Amazon S3 File List and load into SQL Server Table (Call AWS API Example)

File Upload Example – Low level API – Call PUT request

There will be a time when you want to take total control of your AWS API calls. One example is if you wish to Upload / Write data to S3 then components like ZappySys Amazon S3 CSV Destination or Amazon Storage Task might need additional permission such as HeadObject . If you have only write permission on bucket then this will fail to execute Task. In such case you can use REST API Task like below way.

Basically we exported data from Relation Database to CSV File using Export CSV File Task and then we uploaded file content using REST API Task.

Upload Text File to S3 (i.e. JSON, CSV, XML …)

Your URL Format can be like this. Region codes can be found here

https://YOUR-BUCKET.s3.YOUR-REGION.amazonaws.com/YOUR-FILE
https://YOUR-BUCKET.s3.YOUR-REGION.amazonaws.com/SOME-FOLDER/SOME-SUB-FOLDER/YOUR-FILE

Upload file to Amazon S3 – Call low level API (PUT request)

Upload Binary File to S3 (i.e. Zip, mp3, gzip, png, jpeg…)

Above method only works for Text Files. If you have Binary files (e.g. Zip file, mp3, png, jpeg) then you can use below workaround.

1. Check Is Multi Part / File Upload Option next to the Body editor (Read this Post for more info)
2. In the Body enter file path like below
   

   @c:\folder\some-file.xyz

3. That’s it, now your binary file can be uploaded to S3 same way as Text file we uploaded in earlier section.

NOTE: Binary file option only works in the latest Build Uploaded after 5/21/2020

Debugging AWS API Command Line Requests using Fiddler (Web Proxy)

Before we see more examples of calling AWS API lets first learn how to capture Request data using aws command line (CLI). We will use Fiddler to capture AWS API Requests.

1. Download and Install Fiddler (Free Tool)
2. Install  aws command line
3. Open command prompt and type aws configure command to set credentials.  For more info see configure aws credentials .. see below example.
   

   c:> aws configure
   AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
   AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
   Default region name [None]: us-west-2
   Default output format [None]: json

4. Once you set credentials launch Fiddler
5. On the Fiddler Tools menu > Click Fiddler Option > HTTPS > Check Decrypt HTTPS Traffic
6. Once you do that you may be asked to Trust Fiddler Certificate click OK
7. Close and Open Fiddler to apply setting
8. Now any command you type in aws command line will show up in fiddler. This command we can use in SSIS REST API Task or XML Source to call virtually Any API AWS supports.
9. For example to call some lambda function (supply input json data from file) use below command and watch Fiddler Trace. Notice we added –no-verify-ssl option so we can see requests in custom web proxy like fiddler (This option will trust Fiddler certificate).
   

   c:\>aws lambda invoke --function-name HelloWorld c:\temp\outputfile.txt --no-verify-ssl --payload file://c://test/customer.json

    
   

   Capture AWS Commandline (CLI) request using Fiddler

10. Once you have this information you can use it inside ZappySys Components which supports API calls (e.g. REST API Task, JSON Source, XML Source). For this example we will use REST API Task to call same Lambda Function. Things to change to call any API is URL, Method, ContentType, Body. If its GET call then you wont have Body.
    

    Calling Amazon AWS Lambda Function using SSIS REST API Task

     

11. That’s it.. You can now take this same concept and call virtually API AWS API right insight SSIS without any SDK or command line tools.

Call API Gateway Endpoint (Default URL)

If you wish to call API hosted on Amazon API Gateway Service then enter direct URL. You have to use OAuth Connection (AWS v4 Provider) as previous section.

Typical URL may look like as below if you calling API gateway Endpoint.

https://c5hhigf5mh.execute-api.us-east-1.amazonaws.com/prod/pets

Call AWS API with Region and Custom Service Name

There will be a time when you will have to call AWS API URL and it doesnt indicate Region / Service but you have to supply part of signature for Authentication.  In this case just enter AWS Service and Region along with ClientID, Secret Fields (Custom Service and  Region attributes were introduced in v3.0 or higher so if you are not seeing on OAuth Connection UI then you probably running older version).

Call Amazon Athena API

Check this article for detailed instructions

Call AWS EC2 API

Here is an example of calling EC2 API. This is example of listing EC2 instances.

Request:

POST https://ec2.us-east-1.amazonaws.com/
Content-Type: application/x-www-form-urlencoded; charset=utf-8

Action=DescribeInstances&Version=2016-11-15

Call AWS Lambda API

Here is some example of calling Lambda function

Request:

POST https://lambda.YOUR-REGION.amazonaws.com/2015-03-31/functions/YOUR-FUNCTION/invocations

{you-json-input-goes-in-body}

Call AWS ElasticSearch

If you are using AWS Hosted Managed ElasticSearch then also you can use OAuth AWS v4 Provider as below. Assuming you have configured correct Policy to allow your IAM User Account / IP address.

Call AWS Hosted ElasticSearch REST API in SSIS (V4 Request Signing)

Call AWS API using Native Task/Components

ZappySys provides many High quality Tasks / Components for AWS integration. See below list.

Amazon AWS Cloud Integration
Amazon S3 Task	Amazon Redshift Data Transfer Task	Amazon Redshift ExecuteSql Task
Amazon Redshift Cluster Management Task	Amazon DynamoDB Source	Amazon DynamoDB Destination
Amazon Redshift Source	Amazon SQS Queue Source	Amazon SQS Queue Destination

Conclusion

Amazon AWS Cloud integration from SSIS packages becoming more and more common scenario. ZappySys provides easy to use no coding connectors to achieve many time consuming scenarios. Having clean drag and drop approach is not only faster but more secure using inbuilt SSIS framework. Try SSIS PowerPack to explore many other scenarios not discussed in this article.

The post How to call Amazon AWS API using SSIS (EC2, Lambda, API Gateway, SQS) appeared first on ZappySys Blog.

Google BigQuery is a fully managed Big Data platform to run queries against large scale data. In this article you will learn how to integrate Google BigQuery data into Microsoft SQL Server using SSIS. We will leverage highly flexible JSON based REST API Connector and OAuth Connection to import / export data from Google BigQuery API just in a few clicks.

If you are looking for a similar product inside Amazon AWS Cloud then check an article about Amazon Athena.

Prerequisites

Before we look into a Step-By-Step section to extract and load data from BigQuery to SQL Server let’s make sure you meet the following requirements:

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have basic familiarity with REST API concepts and Google BigQuery API. This post uses free Public Dataset to query BigQuery table so no billing is required to get your hands dirty 🙂 However, if you like to query your own dataset then make sure you have at least one BigQuery dataset and one table with some sample data created (this part does require billing enabled). Read a Google Quickstart article for more information on how to create a new BigQuery dataset and a table.

Understanding Google BigQuery Object Heirarchy

Google BigQuery has 3 main concepts below.

• Project
  • Dataset
    • Table
      • Query requests (each query creates a unique JobID – valid for 24 hours from which you can read data)

So in order to create BigQuery table you always need Project and then Dataset under that project. You can group objects around Project / Datasets. When you query you can supply fully qualified name of your table in FROM clause (e.g.

select count(*) from `bigquery-public-data.usa_names.usa_1910_2013`

So let’s get started.

Read data from Google BigQuery using SSIS

Basically you can query Google BigQuery data in two ways: In this article we will not cover 2nd method.

• Method-1: Query data using jobs/query method in BigQuery API. Use it if you expect to get a result in a fairly short amount of time. This API method generates a temp table which gets deleted after 24 hours. You can read data within that time frame using newly created JobId reference.
  
  NOTE: If you have records that sum up to more than 10MB of data or ~10,000 rows (assuming 1 row uses 1KB of data) then you need to proceed with two-step process as explained below.
• Method-2: Export SQL query result using jobs/insert method in BigQuery API. Use this method if you expect a query to take a long time to finish.

Now let’s see how to query Google BigQuery data using SSIS 🙂

Create Google API Project and Register OAuth App

To use any Google API firstly you have to finish two tasks:

1. Create Google API Project and obtain projectId (see next section).
2. Enable BigQuery API
3. Register your own Google OAuth App and obtain ClientId and Client Secret.

Check step-by-step instructions on how to create Google API Project and create OAuth App for Google. During instructions make sure you enable BigQuery API (screenshots may differ).

Get Google API Project ID

Once you have Google API Project created you can grab Project ID (see screenshot below). You will need Project ID in the next step when building API URL.

How to find Google API Project ID?

Configure OAuth Connection for Google BigQuery API

In order to call most of Google APIs you will need an OAuth App. If you want to use your own app then refer to how to register Google OAuth Application (Get ClientID and ClientSecret) article. For simplicity we will use Default OAuth App:

1. Right click inside Connection Managers area and click “New Connection…”
2. From the connection type list select “ZS-OAUTH” connection type.
   

   Create new SSIS OAuth API Connection Manager

3. For OAuth Provider select “Google”.
4. For OAuth App type options leave “Use Default OAuth App” selected. If you created custom OAuth App then you would need to select “Use Custom OAuth App” option.
5. For Scopes enter or select the following URLs (URL is just a permission name). Each URL must be in a separate line. For demo purposes we use only the first 3 scopes but we included a few more in case you like to test API in depth with a different permission set:
   

   https://www.googleapis.com/auth/bigquery
   https://www.googleapis.com/auth/cloud-platform
   https://www.googleapis.com/auth/cloud-platform.read-only
   https://www.googleapis.com/auth/bigquery.insertdata
   https://www.googleapis.com/auth/devstorage.full_control
   https://www.googleapis.com/auth/devstorage.read_only
   https://www.googleapis.com/auth/devstorage.read_write

6. Click Generate Token. Login using correct account if needed and then you will be prompted to click “Approve” OAuth App.
7. Once you click OK on the login screen you will see new tokens populated on the connection screen.
8. Click Test to make sure connection / tokens are valid and then click OK to save the connection.
   NOTE: Below screenshot uses Default App but we recommend you to to use Custom OAuth App (Your own Clientid / secret – Obtained in previous section)
   

   Create SSIS OAuth API Connection for Google BigQuery API

Start BigQuery Job and get JobId (Submit Query)

Once you have SSIS OAuth connection created for BigQuery API it’s time to read data from BigQuery. So, basically, there are two ways you can read BigQuery data: using query or insert method. For demo purposes we will use jobs/query method. If you want fire complex queries which can run for many minutes then refer to jobs/insert method.

If you are expecting more than 10MB of data (~10K records) then you have to split data extract in two steps:

1. Step-1 : Call jobs/query method using REST API Task to submit SQL query. This steps returns jobComplete: true if supplied query execution is done within timeoutMs parameter you included. By default its 10 seconds so if your query is going to take longer than 10 seconds then its good idea to add bigger timeout that way you dont need step#2. Once API call is done you will Get JobId in the response. Save that JobId in a variable for later use. This job result is valid for 24-hours only.
2. Step-2 (Optional) – You can add another optional REST API Task after previous step to Wait until Job is completed. For simplicity, this article will Omit setting up Status Check but its very simple….Use below settings
   1. Drag new REST API Task from toolbox. Connect it to Step-1 and double click to configure as below.
   2. Select URL From Connection mode,
   3. Select OAuth connection
   4. Enter URL as https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/jobs/{{User::JobId}} , Method: GET
   5. On Response Settings Tab, Select Response Content Type JSON and for content filter enter $.status.state
   6. On Status Check Tab, check Enable Status Check Loop and enter DONE in SuccessValue field
   7. Doing this setup will make sure we do not query data until Job Status is DONE (System keeps checking every 5 seconds)
3. Step-2 : Read Job result using JSON Source in a Data Flow Task.

Now lets see how to configure REST API Task to submit query and extract JobId (which will be used in next section).

1. In the Control Flow designer drag and drop ZS REST API Task from the SSIS toolbox.
2. Double click the task to configure it.
3. Change Request URL Access Mode to [Url from connection].
4. From the connection dropdown select OAuth connection created in the previous section.
5. Enter URL as below (replace YOUR-PROJECT-ID with a valid Project ID obtained in the previous section):
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-PROJECT-ID/queries

6. Select Method as POST.
7. Enter Body (Request Data) as below (change your query if needed; we used Public dataset for the demo): If you do not specify timeout then default is 10 seconds only. Also in this call we only care about JobID from response so we just added maxResults=10 because in 2nd step we will get all rows by doing pagination.
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000"
   }

   To use a Standard SQL query instead of Legacy SQL, add useLegacySql property and set it to false:
   

   {
      "timeoutMs": 100000, 
      "maxResults": 10, 
      "query": "SELECT title id,language,wp_namespace,reversion_id ,comment ,num_characters FROM [bigquery-public-data:samples.wikipedia] LIMIT 100000",
      "useLegacySql": false
   }

   For all possible parameters refer to jobs/query method documentation.
   NOTE: You can also supply location Parameter in the Body JSON to indicate where job should run. For non EU / US datacenters we suggest you to supply this parameter. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location.
   Example Use Of Location
   

   { "location":"us-east1",  "maxResults": 10, "query": "SELECT title FROM [bigquery-public-data:samples.wikipedia] LIMIT 10" }

8. Select Body Content Type as application/json.
9. Click Test Request/Response. If things go well you will see JSON content. Then just copy JobId from the response and save for later use:
   
10. Now go to Response Settings tab.
11. Select ContentType=Json.
12. Enter below expression to extract JobId from the response. You will need this Id to read SQL query output (this JobId is valid for 24 hrs):
    

    $.jobReference.jobId

13. Choose save response content to save to a variable. Select <New Variable…> option. When prompted give a name to your variable (i.e. vJobId) and for the value field paste JobId copied from the above step:
    
    
    NOTE: This JobId expires after 24-hrs so while you are designing a package and not running it then design time value is used for testing/previewing. So make sure your JobId is valid. If needed click Test Request/Response to grab a new JobId and update the variable with a new value so you can preview data without running a full package.
14. Click OK to save the UI.

Read BigQuery SQL result (for specified JobID)

Now let’s look at how to read data from BigQuery

Configure Google BigQuery Web Request (URL, Method, ContentType, Body etc.)

1. In the Control Flow designer drag and drop Data Flow Task from SSIS toolbox.
   
2. Double click Data Flow Task and drag and drop ZS JSON Source (For API/File) from SSIS toolbox.
   
3. Double click JSON Source to edit and configure as below
4. Enter URL as below. Replace YOUR-API-PROJECT-ID with the API Project ID obtained in the previous section.
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?maxResults=10000

   –OR– (Use below if your Job ran outside US / EU data center)
   

   https://www.googleapis.com/bigquery/v2/projects/YOUR-API-PROJECT-ID/queries/{{User::vJobId}}?location=YOUR_REGION_ID&maxResults=10000

   NOTE: location Parameter indicates the geographic location of the job. This is Required parameter except for US and EU. See details at https://cloud.google.com/bigquery/docs/locations#specifying_your_location. You can supply same location parameter for first step when you submit the query

   Failure to supply location parameter for non US /EU users may result in 404 NotFound Error.
   

5. Check Use Credentials option and select OAuth Connection Manager created in the previous section.
6. For HTTP Request Method select GET.
   

Read data from Google BigQuery from Temp Job result

Configure Filter

We need to transform a single JSON response into multiple rows so we need to apply a correct filter.

1. On JSON Source go to Filter Options tab.
2. In the Filter field enter $.rows[*] or click [Select Filter] button to browse hierarchy you want to extract.

Select Filter for JSON Response

Configure BigQuery API Pagination Settings

Most of modern APIs usually implement some sort of pagination technique so you get a part of data in each request rather than all in one go. Thus if you want more you can paginate through pages until the last page is reached. You can also read Understanding REST API Pagination in SSIS article to learn more about pagination in SSIS.

BigQuery API returns pageToken attribute in response JSON if more data is found for requested query result. You can then pass pageToken in the next URL in this format: http://my-api-url/?pageToken=xxxxxxxxxx

Now lets configure JSON Source to automate this pagination for us. On JSON Source go to Pagination Tab and enter the following two settings:

1. Set Next Link as $.pageToken.
2. Set Suffix for Next URL as &pageToken=<%nextlink%>.

See the screenshot below to get more clarity:

Configure BigQuery API Pagination on SSIS JSON Source

Configure Array Transformation

Now the last thing we have to configure is special 2-dimensional JSON array format used by BigQuery API:

1. On the JSON Source UI go to 2D Array Transformation tab.
2. Enter the following settings:
   1. For Transformation Type select Transform complex 2-dimensional array.
   2. For Column Name filter enter $.schema.fields[*].name.
   3. For Row Values Filter enter $.f[*].v.

JSON Array Transformation Options

Preview Data and Save UI

That’s it click Preview to see some data. If you entered sample JobID for your User::vJobID variable then you will see some data. In the next section we will see how to load Google BigQuery data into SQL Server. You can click Columns Tab to review Metadata.

Click OK to save UI and generate Metadata.

Configure Target – Load Google BigQuery data into SQL Server

Now you can connect your JSON Source to any target such as ZS Trash Destination or a real database destination such as OLEDB Destination.

If you wish to dump data from Google BigQuery to a SQL Server table then just perform the following steps:

1. Drag and drop OLEDB Destination from SSIS Toolbox. Rename it to something like SQL Server Table.
2. Double click on OLEDB Destination.
3. Click New to create a new connection > Configure connection > Click OK.
4. Click on New to create a new table > Rename default table name > Click OK.
5. Click on Mappings tab and click OK to save UI with default mappings.

Configure SSIS OLEDB Destination – Google BigQuery to SQL Server Import

Execute Package – Loading BigQuery data into SQL Server

SSIS Package Execution –
Loading Google BigQuery Data into SQL Server

Create / Delete Google BigQuery Dataset using API call

As we mentioned before, you need dataset before you can create a table. Most common way to create dataset is via User Interface but what if you like to automate from your SSIS Package or other workflow? Here is how you can create or delete. Basically you can use REST API Task to send CREATE or DROP command for Dataset object

Create Google Dataset

To create dataset configure REST API Task using below settings (We will call this API)

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/datasets

4. Request Method as POST
5. Enter Request Body as below (change YOUR_DATASET_NAME – e.g. TestDataset )
   

   {"datasetReference": { "datasetId": "YOUR_DATASET_NAME", "projectId": "{{User::ProjectId}}"} }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Dataset already exists then you can go to Error Handling Tab (See next section for screenshot) and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Delete Google BigQuery Dataset

For deleing dataset you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Create / Delete Google BigQuery Table using API call

Now let’s look at how to create /drop Table in Google BigQuery using API calls.

Create Google BigQuery Table

To send CREATE TABLE  SQL statement (DDL) we have to use same approach as we send normal SQL Query using this API call. So notice we used Standard SQL for this call by supplying  useLegacySql: false . DDL Statement must be all in one line

POST https://www.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

Content-Type: application/json

>>>> BODY <<<<<

{
 "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
 "useLegacySql": false,
 "timeoutMs": 100000 
}

To create dataset configure REST API Task using below settings

1. Drag REST API Task from toolbox
   
2. Select URL from Connection mode and select connection as OAuth connection (Created in previous section)
3. Enter below URL (assuming you stored your ProjectID in a variable called ProjectId
   

   https://bigquery.googleapis.com/bigquery/v2/projects/{{User::ProjectId}}/queries

4. Request Method as POST
5. Enter Request Body as below
   

   {
    "query": "CREATE TABLE TestDataset.Table1 (RecordID INT64,CustomerID STRING,CustomerName STRING);",
    "useLegacySql": false,
    "timeoutMs": 100000
   }

6. Select Request Content Type as application/json from the dropdown
7. (Optional) – If you want to implement Continue if Table exists then you can go to enable Error Handling Tab and check Continue On Error Code option and set 409 status code
8. That’s it. Click Test Request/Response see it works.

Create Google BigQuery Table – API Call (Continue On Error Setting – Skip CREATE if Table / Dataset Already Exists)

Delete Google BigQuery Table

For deleing table you have to choose same steps as above except two things

1. Request Method as DELETE
2. Request Body as blank

Write data to Google BigQuery using SSIS – 1 Million row insert test (FAST)

Now let’s look at how easy it is to import data into Google BigQuery using SSIS. We will use the same OAuth connection we created before. To learn more about inserting data into BigQuery check tabledata/insertAll method documentation.

Make sure billing is enabled

Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

Streaming is not available via the free tier. If you attempt to use streaming without enabling billing, you receive the following error: BigQuery: Streaming insert is not allowed in the free tier.

As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call (Write  demo).

Configure OAuth Connection / Permissions

If you want to perform data insert operation in BigQuery using API calls then include the following scopes in your OAuth Connection Manager and generate a token (see our first section of this article – We already included scopes for Write operation but incase you didnt do then regenerate token with below scopes):

https://www.googleapis.com/auth/bigquery
https://www.googleapis.com/auth/cloud-platform
https://www.googleapis.com/auth/bigquery.insertdata

Create BigQuery Dataset (From UI)

For this demo first create a test dataset and one table under it like shown below (billing must be enabled on your Google API Project). To do that go to https://bigquery.cloud.google.com/welcome and configure them:

Insert data into BigQuery using API call

And here is a REST API example to insert data into a BigQuery Table:

Request URL:

https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

Request Headers:

Authorization: Bearer ya29.Gl0cBxxxxxxxxxxxxxxxxxxuEIhJIEnxE6GsQPHI
Content-Type: application/json

Request Body:

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
     {"json": {"RowId": 2,"CustomerName": "BBB"} }
   ]
}

Here is our data flow setup to achive very high throughput for Google BigQuery Data Load. We will show you how to insert one million rows in Google BigQuery in less than a minute based on below setup. We will use Multi Threading option and New Compression Option (Added in v3.1.4)

1. Dummy Data Source – Generate sample records
2. JSON Generator Transform – Generates JSON documents to send as POST request for above /insertAll API call.
3. Web API destination  – Call /insertAll API call to submit our data to BigQuery
4. (Optional) JSON Parser Transform – Parse Error Message for any response
5. (Optional) Trash Destination – Save any errors to text file for review

Google BigQuery Data Load Demo in SSIS – 1 Million Rows Insert with Multi Threads and Compression ON (Fast Upload)

Configure SSIS JSON Generator – Generate JSON for BigQuery Table Insert

Now let’s see how to build an HTTP request with JSON body and send it to BigQuery:

1. Drag Data Flow Task and double click on it.
   
2. Drag and configure your Source (for this demo we use Dummy Data Source with Customer Template). See previous section for configuration of Dummy Data Source.
   
3. Drag and drop ZS JSON Generator Transform to produce JSON from a database or file records. If your source is already sending a valid JSON then you can skip this step (e.g. SQL query is
   returning JSON). You can also read raw JSON from a very large file (new-line separated JSON) using JSON Source with Output as Raw Data option checked.
4. Connect Source to JSON Generator. Double click JSON Generator Transform to start configuring it like below.
5. Select Output Mode as Single Dataset Array and enter Batch Size 10000 (This is Max limit allowed by Google BigQuery API insertAll)
6. First right click on Mappings node and select Add Static Element
   

   Generate JSON for Google BigQuery InsertAll API request – Batch 10000 rows in a single API call

7. Enter Name as kind and value as bigquery#tableDataInsertAllRequest
   
   

   JSON Generator – Add Static Element

8. Now right click on Mappings node and select Add Document Array option
   

   JSON Generator – Add Document Array

9. Enter rows as array title and click OK
   

   JSON Generator – Name array

10. Select newly added array node and right click > Add unbound nested element  enter Output alias as json and click OK.
    

    JSON Generator – Add unbound nested element

11. Select json node and right click > Select Add Elements below this node and select multiple columns you like to send to BigQuery. Click OK to save.
    

    JSON Generator – Add Multiple Elements

12. Now let’s preview our JSON (Copy preview JSON to try in the next step – Web API destination)
    NOTE: Table name and column names are case-sensitive so make sure your JSON attribute matches exact same way.
    Here is the finished JSON Structure for next Step
    

    Sample JSON Request body for Google BigQuery insertAll API request

13. Click OK to save UI.

Configure SSIS Web API destination – Insert data into BigQuery Table

Once you have Input JSON prepared,  now let’s configure destination for BigQuery.

1. Drag and drop ZS Web API Destination.
2. Connect your JSON Generator Transform to Web API destination.
3. Configure general properties:
   

   SSIS Web API Destination –
   Configure for BigQuery Data load

4. Make sure to enter URL in this format:
   

   https://www.googleapis.com/bigquery/v2/projects/MY_PROJECT_ID/datasets/MY_DATASET_ID/tables/MY_TABLE_ID/insertAll

   Make sure to replace 3 parts in above URL (MY_PROJECT_ID, MY_DATASET_ID, MY_TABLE_ID) with actual values from your Google Project and BigQuery dataset/table configuration.
5. Now you can enable Compression and Multiple Threads for higher throughput as below.
   NOTE: Compression Property was added in v3.1.4 so you may not see it if you have older version.
   

   Google BigQuery Data Loading Performance Optimization Options – Enable Multiple Threads and Compression Options

6. If you want to try test insert request from UI without running full package then go back to first tab and edit Body (Use Sample JSON generated by previous JSON Transform – You can grab from JSON Preview Panel on Generator Transform).Click Test Request / Response and confirm Success as below. You can go back to your BigQuery Portal and check one row is inserted after our test click. If everything looking good then run full package to insert all records.Sample JSON for Body:
   

   {
     "kind": "bigquery#tableDataInsertAllRequest",
     "rows": [
        {"json": {"RowId": 1,"CustomerName": "AAA"} }, 
        {"json": {"RowId": 2,"CustomerName": "BBB"} }
      ]
   }

   

   Test Google BigQuery Table Insert – SSIS Web API Destination UI

7. Hit OK to save UI.
8. Run the package and verify data in Google BigQuery Console:
   

   Loading data into Google BigQuery using SSIS

Error Handling for BigQuery Data Load (Bulk Insert API Calls)

There will be a time when some records you insert may not go well in Google BigQuery. In such case you can read output from Web API destination and parse further with JSON Parser Transform. Check for certain values in the output. You must use JSON Parser after Web API destination (Connect Blue Arrow from Web API destination – Since its Soft Error it won’t redirect in Red Arrow ).

For example here is the sample JSON in POST Body for testing which produces error due to bad column name. When bad row found in batch all records will be rejected. Notice that error returns index of record in batch so you can identify which row went bad. It also returns column name in location attribute.

NOTE: Bad column name in 2nd record

Test Body (Bad):

{
  "kind": "bigquery#tableDataInsertAllRequest",
  "rows": [
     {"json": {"RecordID": 1,"CustomerID": "X1"} }, 
     {"json": {"Bad_Column": 2,"CustomerID": "X2"} }
     {"json": {"RecordID": 3,"CustomerID": "X3"} }, 
   ]
}

Response (For Bad Input):

{
  "kind": "bigquery#tableDataInsertAllResponse",
  "insertErrors": [
    {
      "index": 1,
      "errors": [
        {
          "reason": "invalid",
          "location": "bad_column",
          "debugInfo": "",
          "message": "no such field."
        }
      ]
    },
    {
      "index": 0,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    },
    {
      "index": 2,
      "errors": [
        {
          "reason": "stopped",
          "location": "",
          "debugInfo": "",
          "message": ""
        }
      ]
    }
  ]
}

To configure error detection perform following steps.

1. Drag and drop ZS JSON Parser Transform after Web API destination
2. Click on Web API destination. Connect Blue arrow  to JSON Parser Transform
3. Configure JSON Parser Transform like below
4. Connect JSON Parser TRansform to some Destination to save error information (e.g. SQL Table or Trans destination)

Handling BigQuery Insert Errors in SSIS

Other Common Errors in BigQuery API calls

In this section we will talk about many common API errors in BigQuery.

Error: The project XXXXXXX has not enabled BigQuery

Sometimes you might get below error. To fix this error make sure you go to your Project and Enable BigQuery API

Status Code: BadRequest

Response Body: {
  "error": {
    "code": 400,
    "message": "The project bigquerytest-281915 has not enabled BigQuery.",
    "errors": [
      {
        "message": "The project bigquerytest-281915 has not enabled BigQuery.",
        "domain": "global",
        "reason": "invalid"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}

Error: 404 – Not Found: Table / Job xxxxxx

• Make sure you are setting GET request and not POST.
• Also make sure your jobId is valid because it expires after 24 hours.
• Make sure project-id supplied in URL is valid ID  (DO NOT Specify Alias, use internal ID for project)
• Make sure you supplied location parameter if your outside EU / US region. Reading data using this API call might fail if you failed to supply location in URL

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "notFound",
    "message": "Not Found: Table xxxxxx"
   }
  ],
  "code": 404,
  "message": "Not Found: Table xxxxxxx
 }
}

Error: 403 – Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier

If you trying call certain APIs on sandbox mode or free tier then you might get below error. To overcome this error enable billing on google bigquery. As long as your API calls fall under Free Tier  limit you wont be charged but you still need to enable billing if you wish to call Streaming insertAll API call.

Response Url: https://www.googleapis.com/bigquery/v2/projects/bigquerytest-281915/datasets/TestDataset/tables/Table1/insertAll

Status Code: Forbidden

Response Body: {
  "error": {
    "code": 403,
    "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
    "errors": [
      {
        "message": "Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier",
        "domain": "global",
        "reason": "accessDenied"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}

Debugging Web API Requests

If you need to debug actual requests made to Google server then you can use a tool like Fiddler. It’s a very handy tool to troubleshoot JSON format issues. It will allow to see how a request is made to a server.

Using Fiddler to debug Google BigQuery API requests in SSIS

Common Errors

Deployment to Production

Download Sample Package

Click Here to Download SSIS Sample Package – Google BigQuery API Read Write Create Delete (SSIS 2019, 2017, 2012)

Conclusion. What’s next?

In this article we have learned how to load data from Google BigQuery into SQL Server using SSIS (drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from Google BigQuery REST API using OAuth. JSON Source Connector makes it super simple to parse complex/large JSON files or any Web API response into rows and columns so you can load data into a database, e.g. SQL Server database. Download SSIS PowerPack to try many other automation scenarios that were not discussed in this article.

Keywords:

Google BigQuery Integration with SQL Server | How to extract data from google bigquery in SSIS? | How to read data from Google BigQuery API? | Loading BigQuery Data into SQL Server. | BigQuery to SQL Server | SSIS Google Big Query Integration | SSIS Google BigQuery Import  JSON File | SSIS Google BigQuery Export data

The post How to read / write data in Google BigQuery using SSIS appeared first on ZappySys Blog.

Often, REST API documentation uses example syntax for the curl command-line tool because it’s one of the most popular tools for making API calls. However, suppose you are an SSIS developer using ZappySys API Drivers/Connectors from either the SSIS PowerPack or the ODBC PowerPack. In that case, this blog post will help you translate the curl command line syntax to similar options available in the ZappySys Product. For SSIS PowerPack products, you can refer to Connectors/Tasks such as ZappySys REST API Task / JSON Source Connector / XML Source. To make this article short and simple, we will show many screenshots from SSIS PowerPack UI, but the concepts are the same when you use other products, such as ODBC PowerPack JSON Driver / REST  or XML Driver.

If you are using Postman to call the API, then refer to this article

If you are using Fiddler to call an API, then refer to this article

CheatSheet – Curl to SSIS PowerPack / ODBC PowerPack

Before we dig down more. Let’s look at how each command-line parameter from CURL maps to SSIS PowerPack settings.

Convert Curl Syntax to SSIS PowerPack – REST API Task / JSON / XML Connector

Take a look at the following example of CURL syntax

curl POST -u myusername:mypassword -H "Accept: application/json" -H "AppVersion:1.0" "https://mysite/v1/api/customers/" -d "customerid=AAA&OrderDate=2016-01-01"

Where :

-u

--user

-H

--header

-d

--data

For step-by-step information, check Article Learn how to POST data to a URL using SSIS

Here is the screenshot of HTTP POST in JSON format using the ZappySys REST API Task

Convert CURL Command Line Syntax to SSIS – Perform HTTP POST operation using SSIS REST API Task – Send data in JSON content type

If you wish to POST data and parse the JSON response into rows, you can use the SSIS JSON Source. JSON Source has a powerful JSON Parser that parses large JSON (stored in a file, an API URL, or a Direct string) into Rows and Columns. This is useful if you wish to save the response into a target such as SQL Server or a CSV file.

JSON Source example

HTTP Headers – Convert Curl Syntax to SSIS

Set up HTTP headers in ZappySys Task/Connect, or click on Raw Edit or Add/Edit items in Grid.

HTTP Credentials – Convert Curl Syntax to SSIS

Here is how you pass credentials in the curl command line.

curl.exe https://myserver.com/api -u myusername:mypassword

To setup HTTP credentials in SSIS for Basic / NTFS or Kerberos use HTTP Connection Manager (Access Mode must be changed to Url from Connection on ZappySys REST API Task and for JSON Source Connector or XML Source  check Use Connection Option)

If you have OAuth 2.0 Credentials (Application ID, Application Secret) then use SSIS OAuth Connection Manager rather than HTTP Connection

Check this article to learn about how to pass HTTP Basic Credentials in SSIS PowerPack

Proxy Setting – Convert Curl Syntax to SSIS

Here is how you pass Proxy server info in cUrl command line (

-x

curl.exe -x 192.168.2.1:8888 https://myserver.com/api

Convert Curl Syntax to SSIS – Client Certificate Settings

The post Convert CURL to SSIS PowerPack / ODBC for API Call appeared first on ZappySys Blog.

Fiddler is one the most popular tool to inspect your Http Traffic. This tool helps you to test REST API / SOAP Web requests very easily. If you are using SSIS PowerPack or REST API ODBC Drivers you will find this post really useful to debug various REST API integration issues.

When you run Fiddler on your system, it acts as a tiny Web Proxy that sits between your client application and the webserver. Fiddler only starts capturing traffic when you running it. By default, it runs on the local machine (127.0.0.1) Port 8888. You can change the default port from Tools > Options.

How to download Fiddler (it’s FREE)

You can download Fiddler Classic using this link for FREE (If that link doesn’t work then try this one)

Make sure that you have Downloaded the Fiddler Classic and not a Fiddler Everywhere. Both products might have same features listed in this article but screenshots and steps listed in this article is about Fiddler Classic.

How to start/stop Fiddler two ways – Capture HTTP(s) Traffic

After you install Fiddler, Go to Start Menu  >Search for Fiddler

Launch Fiddler exe and it will start capturing HTTP traffic (For HTTPS URL see next section).

Run fiddler to start capturing web requests/responses made by various client applications on your system (e.g. Curl, Chrome, Internet Explorer). To start/stop capture go to File > Check/Uncheck [Capture Traffic] option.

By default when you run Fiddler it behaves like the default proxy server on your system. However, not all web requests appear in fiddler unless the client application using a system Default Proxy.

Start / Stop Fiddler Capture

How to capture HTTPS traffic in Fiddler

By default Fiddler doesn’t show the content of Web requests made to HTTPS Url (Secure site) because it’s encrypted. Perform the following steps if you want to see HTTPS Traffic.

1. Launch Fiddler
2. Go to Tools > Telerik Fiddler Options > HTTPS > Check [Decrypt Https Traffic Option]
   

   Fiddler Option – Decrypt HTTPS Traffic

3. Once prompted Accept certificate generated by Fiddler

How to filter traffic for specific Domain or URL (contains)

If you see lots of unwanted log entries after you enable Capturing then try following setting to Include / Clude certain log entries from the Capture making it easier to analyze.

1. Go to Filters Tab
2. Check Show only if URL contains. Type any words you like to check in URL. e.g. zappysys.com will include only those URL which has “zappysys.com” anywhere in the Full URL. You can add multiple included by adding space after words
3.  If you like to exclude some URLs then check Hide if URL contains option and enter string like previous step
   

How to view Web Request and Response in Fiddler

Once your web requests appear on the left side panel. Simply double click on the request entry to view. On your right side, you will see two panels.

1. Top  Panel – Request View
2. Bottom Panel – Response View

Here is an example of seeing API calls generated by aws command line requests/response in Fiddler.

Capture AWS Command line (CLI) request using Fiddler

How to Test API using Fiddler (Call REST API for without Postman)

Fiddler comes with another cool feature just like Postman so you can test your API call. Basically, there will be a time when you like to edit your request and test with different data (e.g. URL / Body or Headers). If that’s the case you can directly go to Composer Tab or Replay existing request right Right Click.

How to replay existing request / edit / send new request

To reply to the existing request

Test Web Requests in Fiddler Composer – Replay existing REST API requests or send new one (Edit Header, Body, URL)

How to view and format JSON / XML data in Fiddler

By default Fiddler will show you JSON  / XML Tab if request body or response data in that format (see JSON / XML tabs on top and bottom panels). But what if you like to see in Raw Text but in Syntax Highlighted  / formatted way?

Format Fiddler Request / Response (XML or JSON). View text with Syntax Highlighting

How to use fiddler Converters (e.g. Base64 / URL encode – decode)

There will be a time when you like to encode / decode strings appearing in web requests. Fiddler has another very handy feature call Send to Text Wizard. This option appears in most right click Menu in Raw / Syntax View / Headers view and other places.

Encode / Decode / Convert Text in Fiddler (e.g. Base64, URL Encode)

How to add Request Start Time, Overall Elapsed Time Columns in Fiddler

It’s really useful to see how much time each request is taking. For that, you can try the below steps in Fiddler Classic

1. Right-click on any Header Column
2. Select “Customize Columns”
3. Choose “Session Timers“
4.  Choose Overall_Elaspsed and  ClientConnectedTime from the dropdown

How to see request start time, overall elapsed time in Fiddler

How to edit and execute the processed requests in Fiddler

If you want to re-execute existing requests in Fiddler with different parameters then try the below steps

1. Select request from the left-hand side panel (or may be bottom side)
2. Press “E” and you will see a panel like below
3. Edit your request (i.e. URL, Header or Body)
4. Click on Run to Complete

Edit, Execute Processed Requests in Fiddler

How to enable AutoResponder – Fake Response (Rule based)

If you want to test some API calls for testing purposes, then you can use the AutoResponder feature of Fiddler. It allows you to match URL pattern and send response text from a local file. Here is how to enable Fake Response (Rule-based Auto Response)

1. Select URL from the log (Optional) – If you skip this step then in step#3 you will have to enter URL manually.
2.  Go to AutoResponder Tab
3. Click Add Rule (if you didn’t select Log entry then Rule editor will have blank URL)
4. Enter / Change URL Rule in the Rule editor if needed. By default it uses EXACT match rule. Click here to learn more about more rules
5. Enter a file path where you have saved Fake Response you like to serve if URL rule.

Examples:

Exact URL match:  

EXACT:HTTP://abc.com/api/v3/projects/123/milestones

Pattern URL match:

method:GET regex:(.*)/api/v3/projects/([\w\d-])+/milestones(.*)

Fiddler AutoResponder (Test Fake Response from local file – Rule match based on URL Pattern)

How to diff Fiddler requests

Sometimes you like to see difference between two requests. Fiddler comes with very handy feature. Just select two requests and click Diff Option in Right click menu. For diff to work make sure you install some diff utilities like KDiff3 or WinMerge.

Compare requests in Fiddler / Reissue same request with changes (Composer Tab)

How to show SSIS PowerPack requests in Fiddler

In most cases SSIS PowerPack requests will apperar in Fiddler without any extra configurations. However in some cases applications like SSIS PowerPack cannot detect it as System Default Proxy automatically. In that case try to enable Proxy settings in HTTP Connection or OAuth Connection.  If you are not using connection manager then some Components like JSON Source offers its own Proxy tab so just check Enable Proxy.

By default Fiddler register itself as Default Web Proxy on port 8888  (i.e. 

http://127.0.0.1:8888

Using Proxy Settings in SSIS PowerPack

View HTTP Request in Fiddler

To inspect HTTP Web Request using Fiddler perform following steps.

1. Click on the web request entry on left pane
2. Click on the Inspector Tab > Click Raw from top panel

View HTTP Response in Fiddler

To inspect HTTP Web Response using Fiddler perform following steps.

1. Click on the web request entry on left pane
2. Click on the Inspector Tab > Click Raw from bottom panel
3. You can also click on JSON or XML Tab if your want to see response coming in specific format

HTTTP Multipart POST – File Upload Example – Check Fiddler Request

View compressed (i.e. gzip, deflate) Response in Fiddler raw view

When response is compressed then by default you wont be able to see it in Raw tab. When response is compressed you may see header like below in response headers. You can also click on

Content-Encoding: gzip

To view compressed response in fiddler perform following steps.

1. Click on the web request entry on left pane
2. Click on the Inspector Tab > Click Transformer tab from bottom panel
3. Click on transformer tab and select No compression option and then click Raw tab

View uncompressed data in Fiddler (GZip, Deflate Encoding)

How to view Formatted JSON / XML

If you have JSON or XML response and you like to view formatted documents then you can go to Syntax View in the Response Panel and right-click and format

How to format Fiddler Request / Response (JSON/XML document)

Using Fiddler Text Wizard Tools (Encode / Decode – Hex, HTML, URL, Base64)

If you would like to perform some conversions like the one below, then you can use the TextWizard tool found in Fiddler. It’s very handy. Simply right click and click on Send to TextWizard option (found in most tabs / screens)

• Base64 Encode / Decode
• HTML Encode / Decode
• URL Encode / Decode
• Hex Encode / Decode
• and more…

Launching Fiddler TextWizard from Right click menu

Fiddler Send To TextWizard Menu

Launching Fiddler TextWizard from Toolbar

Fiddler TextWizard UI

How to compare requests / re-execute

Sometimes you want to compare two requests and find the difference. Here is how you can compare requests and re-execute same request or edit it.

Compare requests in Fiddler / Reissue same request with changes (Composer Tab)

How to show web request of Curl in Fiddler

By default some application such as Curl doesn’t use system default Proxy. That means if you running Fiddler and trying to analyze requests made from Curl command line then it wont show up. Its easy to tell curl that which proxy server to use. Just use -x command line option with proxy server IP and Port (e.g. Default Fiddler IP and port … 127.0.0.1:8888 )

curl.exe -v -x 127.0.0.1:8888 https://google.com

How to show SoapUI requests in Fiddler

If you are using an application like SoapUI then also it may not use Fiddler as default web proxy. to show your SoapUI web requests in Fiddler try following options

Configure Proxy Settings in SoapUI

How to show aws command line requests in Fiddler

If you are using aws command line application and want to show requests in Fiddler then add –no-verify-ssl like below. This way you can capture RAW API requests and use it like this in SSIS.

aws lambda invoke --function-name HelloWorld c:\temp\outputfile.txt --no-verify-ssl --payload file://c://test/customer.json

How to show Windows Service requests in Fiddler (Local System Account)

By default if Fiddler launched as Admin it will capture most of requests by various apps. But if your like to capture Web requests which are issued by windows service such as ZappySys Data Gateway then you might have to do little extra work (Assuming Service is not setup Run As Your account). By default many service Run under “Local System” account

Here is how to capture those requests which are issued by service account.

1. Go to Folder where Service Exe is located (If you are unsure simply right click on Service > Properties and check path)
2. Create new file in the same folder where Service Exe is located. File name must be <yourexename>.exe.config
3. Enter following XML text in your config file and save
   

   <configuration>
     <system.net>
       <defaultProxy enabled="true">
         <proxy proxyaddress="http://127.0.0.1:8888" bypassonlocal="False"/>
       </defaultProxy>
     </system.net>
   </configuration>

4. Restart Service and check Fiddler now see requests are captured

Capture Windows Service Web Requests in Fiddler (App Config method)

REST API integration using ODBC in BI Apps (e.g. Power BI, Excel, Informatica, Access, C#…)

If you want to consume API data in SQL Server (without ETL) or Live API consumption in Tools like Power BI, Excel, Tableau, SSRS, Talend, Informatica then use ODBC PowerPack from here. See below table for each use case.

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Remove Fiddler Root Certificate (Uninstall / Untrust)

The Fiddler certificate is unique per machine / user, so you do not have to worry about someone else reusing it to intercept traffic, but still, for some reason, if you would like to uninstall the Fiddler root certificate (installed when you checked the Decrypt option), then perform the following steps.

1. Goto Tools > Options > HTTPS
2. Uncheck the Decrypt option first if it’s checked
3. Click on Actions > Remove interception Certificates

How to remove Fiddler Root Certificate

Coding-Free REST API integration using ODBC Drivers (For many Reporting / ETL Tools)

If you ever need to read /write data from REST API without writing a single line of code, check these ODBC Drivers / API Connectors.

ODBC PowerPack is a collection of high-performance drag and drop ODBC / API Drivers for various Modern / Legacy data sources. Connect data in Reporting / ETL Tools / Programming Languages.

Coding-Free REST API integration in SSIS (Microsoft ETL Tool)

If you use SSIS as your primary etl tool and you have API integration usecase then check this product.

The post Tutorial – How to use Fiddler to analyze HTTP Web Requests (Tips) appeared first on ZappySys Blog.

In this post, you will learn how to use the SSIS REST API Task to perform REST API File Upload (i.e., RESTful file upload) to a web server. Typically, to access the REST API, you need to use a programming language, but in this post, we will use a drag-and-drop visual approach.

File upload using HTTP/HTTPS protocol, sometimes also known as multipart/form-data POST. Many REST APIs support file upload using HTTP methods such as POST or PUT.

Real-world scenarios:

• Uploading images/media to Twitter or Facebook
• Import Marketo Lead files using REST API

How to do REST API File Upload using SSIS

First, you will have to download a FREE Trial of SSIS PowerPack (It comes with many REST API Connectors and over 30+ functional tasks/components)

1. Download and install SSIS PowerPack from here
2. From the toolbox of the SSIS designer, drag the ZS SSIS REST API Task
3. Double-click the REST API Task to configure it
4. Specify URL where you wish to POST file/other data
   Example: http://myserver/api/v1/uploaddata
5. Select HTTP Request Method as POST ( make sure your API Documentation for this. Sometimes it’s PUT rather than  POST)
6. Check the File Upload option next to the Body text box (This is most important so don’t forget 🙂 )

Method 1 – Upload file using Raw Mode (Or POST data from a local file)

The first method of uploading a file or posting data to the server from a local file is a RAW upload. To enable this you simply have to make sure 2 things.

1. Check the File upload option next to Body.
2. In the Body, enter the local file path starting with the @ sign (e.g., @D:\Zappysys\CSV files\example1.csv)

When you don’t include a Key/Value pair and only the filepath is found, then the request is treated as a RAW Upload. In raw uploads, the default content type is application/octet-stream. If you want to put a custom Content-Type, then edit the Headers in the Grid.

The above technique can be used to read POST data content from a local file. Using this method, only one file can be uploaded. If the remote server supports multiple files (i.e., Multi-Part/Form-Data POST), use the following process. See the screenshot below of the file content upload using Raw mode. We used Fiddler to inspect the REST API Task’s Web requests.

POST file data using SSIS ( Raw file upload)

Method 2 – Upload file(s) using Multi-Part / Form Data mode (Mixed content, multiple files)

The second method for file upload is Multi-Part Upload. The advantage of using this method is that you can POST upload a key/value pair along with single or multiple files. To use this method, you must make sure the API supports it on the other end. Use the steps below to POST data using the Multi-Part method.  SSIS PowerPack detects RAW mode vs Multi-Part POST based on BODY content. If the Body contains an Equal sign somewhere and the File Upload option is checked, then it’s treated as a Multi-Part upload. If File Upload Option is checked and Body contains only file path which starts with @ sign then its treated as Raw content upload.

1. Check the File upload option next to Body.
2. Now, click the edit button next to the Body text box. You may specify files/fields you wish to submit along with your request in the following format. It’s a key/value pair separated by a new line. An equal sign (=) must separate each key/value pair.NOTE: Any value starting with an at symbol (@) is treated as a File Path. If you don’t want that behavior, then use a slash, also known as an escape sequence (e.g., Somefield3StartsWithAt=\@MyTwitterHandle). You can do the same thing to add a new line character in your value. Use \r\n for a new line. Use \t for tab. Also, notice that the field name can be anything unless your API expects it in a specific way. Example: Twitter Image Upload API may expect your image file as a media field. In that case you would write ….  media=@c:\data\someimage.png …. rather than File1=@c:\data\someimage.png
   

   File1=@c:\data\somedata.txt
   File1.Content-Type=text/plain
   File2=@c:\data\someimage.png
   File2.Content-Type=image/png
   Somefield1=abcdef
   Somefield2=123456
   Extrafield3WithNewLine=This is line1\r\nThis is line2 with tab\t and more \t tab
   Somefield3StartsWithAt=\@MyTwitterHandle

   Click OK to Save above.
   NOTE: If you do not specify Content-Type of file using

   SomeFieldName.Content-Type

   attribute, then the system automatically detects based on the file extension. To exclude content-type altogether, you have to set it as a blank string, like below
   

   file1=@c:\temp\dump.json
   file1.Content-Type=

3. Now, click the Test button in the UI to make sure everything looks good.
   

   Uploading files using REST API in SSIS ( REST API File upload using Multipart form-data POST)

Understanding Multipart Upload / File Upload

Now, let’s take a look at what it looks like in Fiddler when an actual Multipart POST request is made for the REST API File upload. Notice that each field included in our HTTP POST request is separated by some random boundary. The REST API Task automatically generates this boundary. If you post an Image, then you will see binary data rather than text for file content. Here are some known issues you need to be aware of.

HTTP Multipart POST – File Upload Example – Check Fiddler Request

Here is a Sample Multi-Part request.

Notice a few things

• The first line of the body content is “Boundary.”
• Last line in the body is also a Boundary, but it has two extra dashes at the end after the boundary line (i.e. ———–xyzzzzz– )
• Notice name=”something”. This is an essential attribute. This is how the target server knows which item to extract from your response.
• Content-Type is automatically determined for a Multi-Part item when it is a file upload. Based on extension, ZappySys automatically selects the most common content type. See the next section.

POST https://somecompany.com/downloads/files/test/uploadtest.aspx HTTP/1.1
User-Agent: ZappySysApp/1.0.2019.10822
x-custom-hdr: 69c05bf5b6ef48dbb73c55020c50e027
Content-Type: multipart/form-data; boundary=---------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825
Host: somecompany.com
Content-Length: 3619
Expect: 100-continue
Accept-Encoding: gzip, deflate
Connection: Keep-Alive

-----------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825
Content-Disposition: form-data; name="File1"; filename="MultiPartFileUpload1.txt"
Content-Type: text/plain

A000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
-----------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825
Content-Disposition: form-data; name="File2"; filename="MultiPartFileUpload2.txt"
Content-Type: text/plain

B00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
-----------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825
Content-Disposition: form-data; name="Extrafield1"

aaaa
-----------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825
Content-Disposition: form-data; name="Extrafield2"

bbbb
-----------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825
Content-Disposition: form-data; name="Extrafield3WithNewLine"

This is line1
This is line2 with tab	tab	tab
-----------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825
Content-Disposition: form-data; name="Extrafield3WithAt"

@MyTwitterHandle
-----------------------------WebKitFormBoundary55fd218185294c408e9ef21a4ac82825--

Known Issues

Sometimes you may receive the following error if your server is expecting \r\n after the last Multi-Part boundary. Here is the fix for this error.

The remote server returned an error: (500) Internal Server Error.
Response Url: https://your-host/api

Status Code: InternalServerError 

Response Body: {"Message":"An error has occurred."
,"ExceptionMessage":"Unexpected end of MIME multipart stream. MIME multipart message is not complete.","ExceptionType":"System.IO.IOException","StackTrace":" at System.Net.Http.Formatting.Parsers.MimeMultipartBodyPartParser.<ParseBuffer>d__0.MoveNext()\r\n at System.Net.Http.HttpContentMultipartExtensions.MoveNextPart(MultipartAsyncContext context)"
}

Most common extension and Content-Type

In the multi-part upload, you might have noticed that ZappySys adds a content type for some file extensions. Here is a full list of some common extensions and their content-type

Conclusion

If you need to perform REST API file upload or REST API integration without learning complex APIs or programming languages (e.g., Python, Java, C#), SSIS PowerPack provides a perfect way to integrate REST APIs with your existing system. You can uncover endless possibilities with a drag-and-drop approach that is not only simple to use but also offers many options for complex scenarios.

Keywords

How to perform rest api file upload | How to do RESTful file upload in C# | How to do multipart file upload without cURL | How to upload multiple files using HTTP multipart forms | Rest Client Upload File | C# REST API Upload File | Rest API Upload Image | Rest API Upload File

The post REST API File Upload using SSIS (Multi Part POST) appeared first on ZappySys Blog.

In some cases when you call API to http URL it may redirect you to different location (Also known as 301 or 302 redirect). Best example is when you type https://www.zappysys.com it will redirect you to https://zappysys.com. This can cause issue in some case if you are passing credentials (i.e. UserID / Password) because by default your credentials are not passed to redirected URL (This is security feature to avoid sending your credentials to different URL than you anticipated). So how to send Authorization to second URL if second URL expects same credentials too? Well not to worry… it’s just one extra step in that case. Lets see how to do that.

Pass Authorization to 301 / 302 Redirect URL

Assume that you have to call below API URL which requires Basic Authentication (UserID/Password) and it also redirects to some other URL which also needs same credentials. For this purpose we need to mainly perform below important changes.

http://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.com%2F

1. First drag ZS REST API Task from SSIS Control Flow Toolbox
2. Change Url mode to URL from connection. Click New Connection.
3. Configure your connection with API Url, UrserID and Password as below. Click OK to save connection.
   

   SSIS REST API Example – Call with 301 / 302 Redirect

4. Now go to Advanced Tab and Check Prevent 301 redirect option
   

   SSIS REST API Task – Disable automatic redirect option

5. Now go to Response Settings Tab and Map Location Header as below (Assuming you have created SSIS Variable named NewLocation (string type) )
   

   SSIS REST API Task – Map response header to SSIS Variable

6. Click OK to save.

Call API URL from URL

1. Now drag another ZS REST API Task from SSIS Control Flow Toolbox
2. Change Url mode to URL from connection. Select connection created in previous section.
3. Now configure your Url as below so we read it from Variable.
   

   SSIS REST API Task – Calling API URL from SSIS Variable

4. That’s it Save the task and run package.

Debug Web API call using Fiddler

To test how things looking behind the scene we strongly suggest to use tool like  fiddler.

See how Fiddler shows 302 Status code for First API call and see how it passes Authorization (UserID/Password). You can also test Just first task without [Prevent 301/302 redirect] option see what happens]

Debugging Web API call using Fiddler in SSIS

The post Calling Web API in SSIS with Authorization and Redirect appeared first on ZappySys Blog.

In this blog post you will explain how to pass basic credentials (i.e. UserID/Password) along with your web request. You will also learn about setting up Authorization Header for HTTP Web Request in Base64 manually. To learn more about how to consume / call REST API in SSIS check this article.

HTTP Authorization Header basics

As per HTTP Standard you can pass credentials very simple way using basic Authorization header. Below is the sample of Basic Authorization header.

As you can see it consist of HeaderName=Authorization and Value=some base64 encoded string
Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==
As per standard Base64 encoded string is made up with two elements. your_userid:your_password

NOTE: Base64 is encoding and not encryption method. So never hand over your encoded Authorization header string to anyone. Anyone can decode Base64 encoded string easily by using tools like this

Passing Basic Credentials along with HTTP Web Request

You can pass credentials (i.e. UserID/Password) in two different ways.

1. Automatic approach – Using ZS-HTTP Connection Manager (BASE64 Encoded Header automatically generated)
2. Manual approach – Supplying Authorization where UserID/Password encoded as Base64

Automatic approach – Using ZS-HTTP Connection Manager

Now lets look at how to use automatic approach to pass Basic Authorization in following 3 different tasks/components

1. Passing basic credentials in SSIS REST API Task
2. Passing basic credentials in SSIS JSON Source
3. Passing basic credentials in SSIS XML Source

Using SSIS REST API Task (Pass Basic credentials using Connection Manager)

The simplest way to pass credentials is use HTTP Connection Manager. If you are using REST API Task then you can use Url from Connection Access Mode. And on connection manager specify UserId and password to pass Basic Authorization Header

Advantages of this method are..
1) Its very simple
2) Credentials are encrypted in SSIS Package

NOTE: We recommend using https:// Urls whenever possible otherwise your web request can be decoded in plain text by any servers sitting between you and target and possibly they can get your password by decoding base64 header.

Base64 Encoded HTTP Basic Authorization Header – View in Fiddler

Using SSIS JSON/XML Source  (Pass Basic credentials using Connection Manager)

You can use HTTP Connection manager in JSON Source and XML Source too. Only difference is you have to check Use Credentials checkbox and select HTTP connection manager from available connection managers dropdown.

1. Select AccessMode as “File Path or Web URL”
2. Enter API URL
3. Check “Use Credentials” option
4. Click “New” next to Connection dropdown or select New “ZS-HTTP” connection from dropdown
5. Configure HTTP Connection as below (i.e. Enter URL, Select Basic Auth, Enter UserID, Password)
6. Clock OK on HTTP connection manager UI to save and now Test JSON or XML Source UI by clicking Preview (Assuming no other configuration needed)

SSIS JSON Source – Passing Basic Credentials (Supply UserID / Password) using HTTP Connection

Manual approach – Supplying Authorization where UserID/Password encoded as Base64

Our JSON Source, XML Source and REST API Task they all support passing BASE64 Encoded String for Authorization header (see below screenshot). Setting up userid and password in HTTP Connection Manger would do same trick as setting up it manually.

To encode any Header value in BASE64 format you simply have to add BASE64ENC format specifier after variable name.

Example: {{User::varUserAndPassword,BASE64ENC}}

SSIS JSON Source Task – Pass Base64 Encoded HTTP Authorization Header

Reference

Wikipedia Article – HTTP Basic Authorization Header

Download FREE SSIS Components

The post Calling REST API in SSIS with Basic Authentication / Userid-Password (BASE64 Authorization header) appeared first on ZappySys Blog.