If you are looking around to find connectivity options to get Amazon AWS data in Power BI (e.g. Read from Amazon S3 files (CSV, JSON, XML) or get AWS API data such as Billing Data by calling REST API) then unfortunately as of now Power BI doesn’t support it natively.

In this article, you will learn  How to read Amazon S3 data in Power BI (Using S3 CSV Driver) and in later section we will also learn How to import data from AWS API Calls (e.g. Billing, EC2, S3, DynamoDB). We will use ZappySys REST / JSON Driver to access AWS data without any ETL or Programming.

In our previous article, we discussed how to access REST API data in Power BI (Read JSON / XML Web API). Now let’s learn how to connect to AWS in Power BI.

In this article, We will explore interesting scenarios such as below. It will help you to connect Amazon AWS in Power BI and import data from AWS without doing any ETL or programming.

• Import JSON file from S3 bucket in Power BI (Using Amazon S3 Driver for JSON Files).
• Read CSV file from S3 bucket in Power BI (Using Amazon S3 Driver for CSV Files).
• Read XML file from S3 bucket in Power BI (Using Amazon S3 Driver for XML Files).
• Call Amazon AWS REST API (JSON or XML) and get data in Power BI. Some examples of API calls.
  • Get EC2 VM count and their Status in Power BI Dashboard
  • Read data from AWS Athena Service
  • Use AWS Cost / Billing API to display monthly Cost by Service
  • Display summary of Redshift Clusters
  • Extract response from Lambda function and display in Power BI

So let’s get started…

Requirements

1. First, you will need to have Power BI Desktop (FREE) installed
2. Make sure you have AWS Access Key and Secret Key to access AWS data or call API for the desired service
3. On third place, our ZappySys ODBC Power Pack

Import Amazon S3 data in Power BI (CSV, JSON or XML Files)

In this section we will look at step by step approach to load Amazon S3 data in Power BI. Your file can be compressed  (GZip, Zip) or un-compressed. We will use ZappySys Amazon S3 driver for CSV Files for demo but you can also use Amazon S3 JSON Driver or Amazon S3 Driver for XML Files same way for reading different file formats.

Configure ODBC DSN for Amazon S3 Files ( JSON / XML / CSV )

1. Type odbcad32.exe in your search box and launch the DSN Config utility.
2. you want access for yourself then stay of User DSN Tab. If you want grant access other users then go to System DSN tab. For SQL Server Integration go to System Tab and add new System DSN rather than User DSN. Click New button.
   
3. From the Driver list Select ZappySys ODBC Driver. For this example select [ZappySys Amazon S3 CSV Driver].
   
4. Now, we need Amazon S3 Storage Connection. Lets create it.
   
5. Now, When you see DSN Config Editor with zappysys logo first thing you need to do is change default DSN Name at the top and Select your bucket and file from it.
   
6. Here, in the Compression (Zip/GZip) tab set suitable file format.
   
7. Click on Preview Tab, Select Table from Tables Dropdown and select [value] and click Preview.
   
8. If you are using DSN method then Click OK to save our DSN Configuration.

Load Amazon S3 Data in Power BI – using ODBC Driver Connection

In the previous section, we configured ODBC DSN for AWS API call and added the ZappySys drivers in the ODBC Driver Administrator with information to connect to AWS S3 Files. We queried S3 File using ZappySys Amazon S3 Driver for CSV Files. Now we will open Power BI and import the information from ODBC  DSN we defined earlier.

1. Open Power BI Desktop and select the option Get data
   

   Get data in power bi

2. Get Data will allow adding the ZappySys ODBC driver. Go to Other and select ODBC.
   

   Select odbc in Power BI

3. Select ODBC DSN name from the DSN dropdown
4. Now its time to import data. Basically, there are two modes to import data. Table Mode and Query Mode. We will Import using Power BI Query Mode
5. Select your DSN and click Advanced Option to enter custom SQL Query to Import your REST API data. You can use ODBC DSN Data sources Preview tool to generate SQL Query. For example, you can enter a query like below. WITH clause is optional but you can supply many options to override DSN UI settings (e.g. Path, Credentials ).
   

   /*--------- Amazon S3 Driver - Read CSV File Example - Override UI credentials in WITH clause ----------*/  
   SELECT * FROM $  
   WITH (  
     Src='Northwind-Bucket/Orders*.csv' --use wildcard * for pattern search
     -- ,AccessKey='AKIAxxxx' --not needed if you set in DSN connection
     -- ,SecretKey='a1wxxxxxxxxxxxxxx' --not needed if you set in DSN connection
     -- ,RegionName='us-west-1'
     -- ,ColumnDelimiter='|'
     -- ,HasColumnHeaderRow='False'
   )

   
6. On the next screen select Windows Authentication to continue (No userid and password).
   

   Select Windows Credentials to connect to zappysys driver

7. Click OK to import data in Power BI
8. Now you can create custom Dashboard from imported Dataset.

AWS API Example –  Call Amazon S3 API in Power BI

In our previous section we saw how to read Amazon S3 data using native ZappySys S3 Drivers (For CSV , JSON and XML Files), this approach is preferred way to read S3 Files for sure because it gives you UI to browse files, it gives you ability to read from multiple files and many more S3 specific features.

However in some cases you have to call REST API in a manual way using REST API driver rather than S3 Driver. In next section we will see more interesting use case of calling AWS API (e.g. Billing / Cost API).

Now lets look at steps – How to import Amazon S3 data Power BI by calling REST API manually. Steps listed below are almost identical for XML or JSON data format except for selecting Driver Type (i.e. JSON, XML, CSV).

Configure ODBC DSN for REST API ( JSON / XML / CSV )

To consume S3 data in Power BI, first we have to create ODBC DSN using ZappySys Driver (XML or JSON). Perform the following steps.

1. To do this, we will first open the ODBC Data Source (32 bit):
   

   Open odbc data source

2. Use the User DSN page and press Add
   

   Add new data source

3. Add the ZappySys XML Driver if you are accessing XML files from S3 Bucket or calling any AWS API which returns data in XML format. Select JSON driver for JSON data (or API Access in JSON Format)
   

   Create DSN using ZappySys ODBC Driver for XML / SOAP API

4. It is time to connect with Amazon S3 File and read data. In Data Source (URL or File Path), we will use XML file URL as below. Notice that S3 URL has 3 parts (zs-dump1 is bucket name, s3.us-east-1.amazonaws.com is service endpoint for S3 (some service doesn’t require region) and store_001.xml is our file name. After you specify URL select Connection as per the screenshot.Syntax for URL :  BucketName + ServiceURL + FilePath.If you are not sure what can be your service endpoint then check this table.
   

   https://zs-dump1.s3.amazonaws.com/store_001.xml
   --OR--
   https://zs-dump1.s3.amazonaws.com/mysubfolder/store_001.xml
   --OR--
   https://zs-dump1.s3.us-east-1.amazonaws.com/mysubfolder/store_001.xml

   

   Configure Amazon S3 Connection – Read from XML File or JSON File

   NOTE: If a filename contains a space, e.g. “store abc.xml“, make sure to replace space with “%20” (“+” will not work). So “store abc.xml” should become “store%20abc.xml“.
5. Once you specify the URL and Credentials, Next select Filter (This helps to flatten the nested hierarchy of your Data). If you don’t have nested Hierarchy then skip this step.
   

   Select Filter (Flatten the hierarchy to extract data from nested structure)

    

6. Now go to Preview Tab and click Preview button in the toolbar. You can remove attributes in the query to use default attributes from Previous Tab. You can always override DSN level setting inside WITH clause of SQL.
   

   Preview data from AWS S3 File (XML Data) – Use ZappySys XML Driver to Connect to AWS API

7. To learn more about query language click on View Examples
8. Click OK to save your ODBC DSN settings.

Import AWS S3 File or AWS API data in Power BI dataset

Now let’s look at how to import Amazon S3 data in Power BI using ZappySys XML driver.

In the previous section, we configured ODBC DSN for AWS API call and added the ZappySys drivers in the ODBC Driver Administrator with information to connect to AWS REST API. We queried S3 File using ZappySys XML Driver. Now we will open Power BI and import the information from ZappySys JSON Driver via ODBC connectivity option in Power BI.

1. Open Power BI Desktop and select the option Get data
   

   Get data in power bi

2. Get Data will allow adding the ZappySys ODBC driver. Go to Other and select ODBC.
   

   Select odbc in Power BI

3. Select ODBC DSN name from the DSN dropdown
4. Now its time to import data. Basically, there are two modes to import data. Table Mode and Query Mode. We will Import using Power BI Query Mode
5. Select your DSN and click Advanced Option to enter custom SQL Query to Import your REST API data. You can use ODBC DSN Data sources Preview tool to generate SQL Query. For example, you can enter a query like below.
   

   SELECT "author", "title",  "genre", "price",  "publish_date", "description", "tag"
   FROM $
   WITH(
   	 Src='https://zs-dump1.s3.amazonaws.com/store_001.xml'
   	,Filter='$.store.book[*]'
   	,ElementsToTreatAsArray='book'
   )

   

   Import JSON / REST API data in Power BI using SQL Query Mode

    

6. On the next screen select Windows Authentication to continue (No userid and password).
   

   Select Windows Credentials to connect to zappysys driver

7. Click OK to import data in Power BI
8. Now you can create custom Dashboard from imported Dataset.

AWS API Example – Import Amazon AWS Cost / Billing data in Power BI

So far you have seen how to connect to S3 File. Now let’s look at another interesting example to call AWS API and display information in Power BI dashboard. For example purpose, we are going to call this AWS Billing and Cost Analysis API. Perform following steps.

1. Create new DSN using Zappysys JSON Driver
2. Configure ODBC DSN as below
3. Enter URL as we discussed in previous section. For example to call Billing API Service for East-1 Region we need to enter following way.
   

   https://ce.us-east-1.amazonaws.com

4. Select Connection Type as OAuth and click Configure Link.
5. Select Provider as AWS API v4 and enter AWS Key and Secret. Click OK to save the connection.
6. Select Method as POST
7. Enter HTTP Request Headers as below (Note each Billing Cost API action will have different Header value for X-Amz-Target so refer API documentation
   

   Configure AWS API Connection for Cost / Billing API Data – Use ZappySys JSON Driver

8. Enter Request Body (Modify your report parameters). As you see in below request we are requesting billing data for 3 services for Jan-1-2018 till Jun-15-2018. If you dont know service name then simply refer to your Bill and copy Service Name in Values array.
   

   {
     "TimePeriod": {
       "Start":"2018-01-01",
       "End": "2018-06-15"
     },
     "Granularity": "MONTHLY",
     "Filter": {      
       "Dimensions": {
         "Key": "SERVICE",
         "Values": [
           "Amazon Simple Storage Service",
           "Amazon DynamoDB",
           "Amazon Redshift",
           "AWS Key Management Service"
         ]
       }
     },
     "GroupBy":[
       {
         "Type":"DIMENSION",
         "Key":"SERVICE"
       },
       {
         "Type":"TAG",
         "Key":"Environment"
       }
     ],
      "Metrics":["BlendedCost", "UnblendedCost", "UsageQuantity"]
   }

    
9. Select Filter as $.ResultsByTime[*].Groups[*]
10. Click Test Connection button see it shows OK.
11. If the connection is OK then switch to Preview Tab to test query.
12. You can select Table name from the drop-down or type query as below
    Simple-Query to Extract AWS Billing / Cost Information
    

    SELECT  
     "p_ResultsByTime_TimePeriod_Start" as BillStartDate,
     "p_ResultsByTime_TimePeriod_End" as BillEndDate,
     json_array_first("Keys") as ServiceName,
     "Metrics_BlendedCost_Amount" as BillAmount,
     "Metrics_UsageQuantity_Amount" as Quantity
    FROM [ResultsByTime_x_Groups] 
    ORDER BY p_ResultsByTime_TimePeriod_Start

13. You can also override attributes in your query using below syntax
    Custom-Query to Extract AWS Billing / Cost Information (WITH clause)
    

    SELECT 
     "p_ResultsByTime_TimePeriod_Start" as BillStartDate,
     "p_ResultsByTime_TimePeriod_End" as BillEndDate,
     json_array_first("Keys") as ServiceName,
     "Metrics_BlendedCost_Amount" as BillAmount,
     "Metrics_UsageQuantity_Amount" as Quantity
    FROM $
    ORDER BY p_ResultsByTime_TimePeriod_Start
    WITH(
    	 Src='https://ce.us-east-1.amazonaws.com'
    	,DataConnectionType='OAuth'
    	,ScopeSeparator='{space}'
    	,ServiceProvider='AmazonAWS'
    	--,ClientId='AKIAxxxxxxxxxxxxxxx'
    	--,ClientSecret='lPi+XQxxxxxxxxxxxxxxxx'
    	,UseCustomApp='True'
    	,Filter='$.ResultsByTime[*].Groups[*]'
    	,RequestData='{
      "TimePeriod": {
        "Start":"2018-01-01",
        "End": "2018-06-15"
      },
      "Granularity": "MONTHLY",
      "Filter": {      
        "Dimensions": {
          "Key": "SERVICE",
          "Values": [
            "Amazon Simple Storage Service",
            "Amazon DynamoDB",
            "Amazon Redshift",
            "AWS Key Management Service"
          ]
        }
      },
      "GroupBy":[
        {
          "Type":"DIMENSION",
          "Key":"SERVICE"
        },
        {
          "Type":"TAG",
          "Key":"Environment"
        }
      ],
       "Metrics":["BlendedCost", "UnblendedCost", "UsageQuantity"]
    }'
    	,Header='X-Amz-Target: AWSInsightsIndexService.GetCostAndUsage || Content-Type: application/x-amz-json-1.1'
    	,RequestMethod='POST'
    )

14. Click Preview button to see the result.
    

    Query AWS Cost / Billing Data using API in Power BI (Use ZappySys JSON Driver)

15. Click OK to save Your DSN
16. Now you can Import Billing / Cost data in Power BI using same steps as previous section (Except use custom Query and DSN we just created for AWS API call)
    

    Import AWS Billing / Cost Data in Power BI (Call AWS REST API)

17. Repeat steps to import more datasets using different queries

AWS API Example – Import Data From AWS Lambda in Power BI

Now let’s look at another scenario. Assume that someone from other team has written a Lambda Function which exposes certain data in JSON format which you want to display that in your Dashboard. Refer to this Lambda API documentation to learn more about API syntax.

Here is an example query to Call Lambda Function (submit input data in JSON and receive data in JSON format)

SELECT * FROM $
WITH(
	 Src='https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/HelloWorld/invocations'
	,RequestMethod='POST'
	,RequestData='{"id":1}'
	,RequestContentTypeCode='ApplicationJson'
	
	/*** Uncomment to override DSN settings	  
	,DataConnectionType='OAuth'
	,ScopeSeparator='{space}'
	,ServiceProvider='AmazonAWS'
	--,ClientId='AKxxxxxxxxxxx'
	--,ClientSecret='lPi+Xxxxxxxxxxxxxxxxxxxxxxx'
	,UseCustomApp='True'
	***/
)

AWS API Example – Get EC2 VM information in Power BI

Now let’s look at AWS EC2 Service API (Virtual Machine Service). This API is in XML format so you have to use ZappySys XML Driver.  Below query calls this EC2 API to get list of all instances.

SELECT * FROM $
WITH(
	 ElementsToTreatAsArray='item'
	,Src='https://ec2.us-east-1.amazonaws.com/'
	,Filter='$.DescribeInstancesResponse.reservationSet.item[*].instancesSet.item[*]'
	,RequestData='Action=DescribeInstances&Version=2016-11-15'
	,RequestMethod='POST'
	/*
	,DataConnectionType='OAuth'
	,ScopeSeparator='{space}'
	,ServiceProvider='AmazonAWS'
	,ClientId='AKIAxxxxxxxxxxxxxxxxxxx'
	,ClientSecret='lPi+XQvxxxxxxxxxxxxxxxxxxxxxxxxx'
	,UseCustomApp='True'
	*/
)

Get EC2 Instance List in Power BI (Use ZappySys XML Driver)

AWS API Example – Get AWS S3 Buckets and Files Count / Size in Power BI

Now let’s look at how to Query S3 File System in Power BI. For example if you have to prepare dashboard to show Total Data Size and File count in a given Bucket then you can write following Query. For more information check Bucket List API Help

Get File Size and Count for a given S3 Bucket

SELECT count(*) as TotalFiles,sum(Size)/1024/1024 as TotalMB  FROM $
WITH(
	 ElementsToTreatAsArray='Contents'
	,Src='https://zs-dump1.s3.amazonaws.com/?list-type=2&prefix=&max-keys=1000'
	,Filter='$.ListBucketResult.Contents[*]'
	,NextUrlAttributeOrExpr='$.ListBucketResult.NextContinuationToken'
	,NextUrlSuffix='&continuation-token=<%nextlink_encoded%>'
	
	/**** Uncomment below to use override DSN values
	,DataConnectionType='OAuth'
	,ScopeSeparator='{space}'
	,ServiceProvider='AmazonAWS'
	,ClientId='AKIAxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
	,ClientSecret='lPi+Xcxxxxxxxxxxxxxxxxxxxxxxxx'
	,UseCustomApp='True'	
	,RequestMethod='GET'
	***/
)

To list buckets you can use following query

List S3 Buckets

SELECT *  FROM $
WITH(	 
	 Src='https://s3.amazonaws.com'
	,Filter='$.ListAllMyBucketsResult.Buckets.Bucket[*]'
	,ElementsToTreatAsArray='Buckets'

	/**** Uncomment below to use override DSN values
	,DataConnectionType='OAuth'
	,ScopeSeparator='{space}'
	,ServiceProvider='AmazonAWS'
	,ClientId='AKIAxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
	,ClientSecret='lPi+Xcxxxxxxxxxxxxxxxxxxxxxxxx'
	,UseCustomApp='True'	
	,RequestMethod='GET'
	***/
)

Debugging / Crafting AWS API Call Request

Check below article to learn various techniques to debug and craft AWS API Request Body / Headers and URL.

How to call Amazon AWS API using SSIS (EC2, Lambda, API Gateway, SQS)

Understanding AWS REST API Pagination in Power BI

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

Handling AWS Web API Error in Power BI

Sometimes errors occur... they just do and there is nothing you can do! Or can you? Actually, in ODBC PowerPack you can handle them in two ways.

METHOD 1 - Using Error Handling Options

When to use?

You may want to use them when your source is a resource located on the Internet; e.g. a file on a website, a file on an FTP server or just a plain API HTTP response. By default, when a remote server returns an error, data retrieval is stopped, an error is raised and no data is given back to you. This might not be always desirable.

Scenario 1

Imagine a scenario, that there is a web server which each day at 12 AM releases a new JSON file with that day's date as filename, e.g. http://www.some-server.com/data/2018-06-20.json. And, of course, you want to download it and use it daily in your Power BI report. But you have a problem: Power BI report data sources are refreshed each hour and you may get HTTP 404 status code (no file was found) when a file is not released yet. Which consequentially means other data sources won't be updated as well and you will see old and cached data on the report. That's where you could use Continue on any error or Continue when Url is invalid or missing (404 Errors) to avoid an error being raised and let other data sources to be updated.

Scenario 2

Another scenario is when you expect a web server to raise some kind of HTTP error when accessing a URL. You don't want ODBC Data Source to raise an error but instead, you want to get response data. That's where you can use Continue on any error or alike together with  Get response data on error to continue on an error and get the data:

METHOD 2 - Using Connection [Retry Settings]

Another scenario you may run into is a buggy web server. You ask it to give you some file or data and it, like a snotty kid, just doesn't give it to you! You have to ask twice or thrice before it does its job. If that's the case, you have to retry HTTP requests using Connection:

Other Considerations for Calling AWS API in Power BI

Conclusion

In this article, we show how to connect to AWS API and S3 File data in Power BI. We configured ODBC DSN for AWS S3 / REST API connection and finally imported data in Power BI.  We used ZappySys driver for JSON that can be used to extract data from any REST API or a JSON file.  Download ODBC PowerPack to try yourself see how easy it is to consume XML / JSON / REST API and AWS data in Power BI and Say goodbye to ETL or Coding.  If you need any help with your API integration feel free to contact zappysys support you will be amazed for sure how quickly your issue is resolved.

The post Read Amazon S3 data in Power BI or Call AWS REST API (JSON / XML) appeared first on ZappySys Blog.

In this blog post you will learn how to call Amazon AWS API using SSIS (virtually any API) without a single line of code (No more JAVA, C#, Ruby, Python). Yes you heard it right 🙂 . If you are a SSIS / ETL Developer or even coder everyone loves drag & drop interface. SSIS has many advantages over other approaches such as Programming SDKs, Command Lines. Main advantage is ease of use, security and long term maintenance without learning expensive coding approach.

In this post we will use following components to show various possibilities to implement Amazon AWS API integration scenarios inside your ETL workflows.

	JSON / REST API Source (REST API, JSON File or OData Service): Use this dataflow component when you have to fetch data from REST API webservice like a table. This component allows you to extract JSON data from webservice and de-normalize nested structure so you can save to Relational database such as SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local JSON files or direct JSON string (Wildcard pattern supported too e.g. c:\data\file*.json).
	Web API Destination (POST data to API URL) : Use this dataflow component when you have to call API inside Dataflow (see POST to URL). Possible Use case: Call Lambda function with various parameters for each input record found in database.
	REST API Task : Use this task when you don’t want to pull REST API data in tabular format but want to call rest API for POST data to server, DELETE data from server or things like download HTML page, extract Authentication tokens etc where you not necessarily dealing data in tabular format. This task also allows you many other options such as saving RAW response into variable or file.
	XML Source (SOAP, File, REST) : Use this dataflow component when you have to fetch data from XML or SOAP webservice and consume data like a table. This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local XML files or direct XML string.

Prerequisites

Before we do hello world demo for calling Amazon AWS API, you will need to make sure following prerequisites are met.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Access to valid AWS credentials (Access Key, Secret Key for your IAM User). Click here to learn more about IAM users and Access Key/Secret Key
4. Make sure SSIS PowerPack is installed. Click here to download.

Step-By-Step Example-1 (Call AWS API)

Now lets call some simple GET API call using SSIS REST API Task.

1. Install  SSIS PowerPack (Skip this step if you already installed SSIS PowerPack.
2. Open Visual Studio and create new Integration Services Project
3. Open SSIS Package and check your SSIS Toolbox you will see many tasks/components starting with ZS
4. From Control flow SSIS Toolbox Drag & drop ZS REST API Task.
5. Double click the REST API task to configure it.
6. On REST API Task change URL Access mode drop down to URL from Connection
7. Now in the connection dropdown click New ZS-OAUTH connection and configure connection as below
   1. Once you see OAuth connection dialog box change Provider Type from Custom to Amazon AWS API (v4)
   2. In the ClientId enter your AWS Access Key
   3. In the ClientSecret enter your AWS Secret Key
   4. Click OK to save connection
      

      Configure SSIS OAuth Connection – Use Amazon AWS API Provider, Enter Access Key, Secret Key

8. On REST API Task change few more settings as below
   1. Enter API URL you like to call (In our case we will use S3 API (Simple Storage Service). We assume you have ListBucket permission to make this call. If you dont have such permission try to get Full path of File (choose small file) . You have to tweak API url to adjust Service Type, Region,Bucket, Path
      

      https://s3.us-east-1.amazonaws.com
      -- OR -- Use below (list files) if you have single bucket permission---
      https://s3.us-east-1.amazonaws.com/YOUR-BUCKET

   2. Click Test Request. If you have valid Permission and setup looks ok then you will see Response window like below. Using this technique you can call any API to execute AWS operations (E.g. start EC2 VM, Create SQS Queue, Call Lambda Function, Drop or Update resource)
      

      Calling Amazon AWS API Using SSIS REST API Task Example. (Call any AWS API without SDK. such as EC2 API, SQS API, Lambda API, Redshift API, DynamoDB API)

Step-By-Step Example-2 (Loading data from AWS API to SQL Server)

Now lets do more interesting scenario. We will call AWS S3 API to get S3 File list from Bucket. After extract we will save that list to SQL Server Table. Since Amazon S3 API is XML based API we will use ZappySys SSIS XML Source. For JSON based API use JSON Source instead. XML Source / JSON Source both can parse API response into Rows and Columns so you can easily store it into SQL Server. Now lets see how to do this.

1. From Control flow SSIS Toolbox Drag & drop Data Flow Task
2. Double click Data Flow Task. From SSIS Toolbox Drag & drop ZS SSIS XML Source.
3. Double click the XML Source to configure it.
4. In the URL text box enter API URL like below to list S3 Files for specified bucket (Change YOUR-BUCKET to your own name)
   

   https://s3.us-east-1.amazonaws.com/YOUR-BUCKET

5. Now check Use Credentials and Select same Amazon API connection we created in previous example.
6. Click on Select Filter (This step allows us to flatten the XML hierarchy. Select the node which is Array icon like below. If prompted to treat selected node as array click Yes.
   

   Configure SSIS XML Source – Get Amazon S3 File List from specific bucket (AWS API)

7. Click Preview to see your data. Click OK to save.
   

   Preview Data in XML Source – Amazon S3 File List (Calling AWS API)

8. Now attach your XML source to target like OLEDB Destination to load data to SQL Server or other Target (e.g. Oracle, MySQL)
9. Execute SSIS Package to load data from Amazon AWS API to SQL Server.
   

   SSIS Package – Read Amazon S3 File List and load into SQL Server Table (Call AWS API Example)

File Upload Example – Low level API – Call PUT request

There will be a time when you want to take total control of your AWS API calls. One example is if you wish to Upload / Write data to S3 then components like ZappySys Amazon S3 CSV Destination or Amazon Storage Task might need additional permission such as HeadObject . If you have only write permission on bucket then this will fail to execute Task. In such case you can use REST API Task like below way.

Basically we exported data from Relation Database to CSV File using Export CSV File Task and then we uploaded file content using REST API Task.

Upload Text File to S3 (i.e. JSON, CSV, XML …)

Your URL Format can be like this. Region codes can be found here

https://YOUR-BUCKET.s3.YOUR-REGION.amazonaws.com/YOUR-FILE
https://YOUR-BUCKET.s3.YOUR-REGION.amazonaws.com/SOME-FOLDER/SOME-SUB-FOLDER/YOUR-FILE

Upload file to Amazon S3 – Call low level API (PUT request)

Upload Binary File to S3 (i.e. Zip, mp3, gzip, png, jpeg…)

Above method only works for Text Files. If you have Binary files (e.g. Zip file, mp3, png, jpeg) then you can use below workaround.

1. Check Is Multi Part / File Upload Option next to the Body editor (Read this Post for more info)
2. In the Body enter file path like below
   

   @c:\folder\some-file.xyz

3. That’s it, now your binary file can be uploaded to S3 same way as Text file we uploaded in earlier section.

NOTE: Binary file option only works in the latest Build Uploaded after 5/21/2020

Debugging AWS API Command Line Requests using Fiddler (Web Proxy)

Before we see more examples of calling AWS API lets first learn how to capture Request data using aws command line (CLI). We will use Fiddler to capture AWS API Requests.

1. Download and Install Fiddler (Free Tool)
2. Install  aws command line
3. Open command prompt and type aws configure command to set credentials.  For more info see configure aws credentials .. see below example.
   

   c:> aws configure
   AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
   AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
   Default region name [None]: us-west-2
   Default output format [None]: json

4. Once you set credentials launch Fiddler
5. On the Fiddler Tools menu > Click Fiddler Option > HTTPS > Check Decrypt HTTPS Traffic
6. Once you do that you may be asked to Trust Fiddler Certificate click OK
7. Close and Open Fiddler to apply setting
8. Now any command you type in aws command line will show up in fiddler. This command we can use in SSIS REST API Task or XML Source to call virtually Any API AWS supports.
9. For example to call some lambda function (supply input json data from file) use below command and watch Fiddler Trace. Notice we added –no-verify-ssl option so we can see requests in custom web proxy like fiddler (This option will trust Fiddler certificate).
   

   c:\>aws lambda invoke --function-name HelloWorld c:\temp\outputfile.txt --no-verify-ssl --payload file://c://test/customer.json

    
   

   Capture AWS Commandline (CLI) request using Fiddler

10. Once you have this information you can use it inside ZappySys Components which supports API calls (e.g. REST API Task, JSON Source, XML Source). For this example we will use REST API Task to call same Lambda Function. Things to change to call any API is URL, Method, ContentType, Body. If its GET call then you wont have Body.
    

    Calling Amazon AWS Lambda Function using SSIS REST API Task

     

11. That’s it.. You can now take this same concept and call virtually API AWS API right insight SSIS without any SDK or command line tools.

Call API Gateway Endpoint (Default URL)

If you wish to call API hosted on Amazon API Gateway Service then enter direct URL. You have to use OAuth Connection (AWS v4 Provider) as previous section.

Typical URL may look like as below if you calling API gateway Endpoint.

https://c5hhigf5mh.execute-api.us-east-1.amazonaws.com/prod/pets

Call AWS API with Region and Custom Service Name

There will be a time when you will have to call AWS API URL and it doesnt indicate Region / Service but you have to supply part of signature for Authentication.  In this case just enter AWS Service and Region along with ClientID, Secret Fields (Custom Service and  Region attributes were introduced in v3.0 or higher so if you are not seeing on OAuth Connection UI then you probably running older version).

Call Amazon Athena API

Check this article for detailed instructions

Call AWS EC2 API

Here is an example of calling EC2 API. This is example of listing EC2 instances.

Request:

POST https://ec2.us-east-1.amazonaws.com/
Content-Type: application/x-www-form-urlencoded; charset=utf-8

Action=DescribeInstances&Version=2016-11-15

Call AWS Lambda API

Here is some example of calling Lambda function

Request:

POST https://lambda.YOUR-REGION.amazonaws.com/2015-03-31/functions/YOUR-FUNCTION/invocations

{you-json-input-goes-in-body}

Call AWS ElasticSearch

If you are using AWS Hosted Managed ElasticSearch then also you can use OAuth AWS v4 Provider as below. Assuming you have configured correct Policy to allow your IAM User Account / IP address.

Call AWS Hosted ElasticSearch REST API in SSIS (V4 Request Signing)

Call AWS API using Native Task/Components

ZappySys provides many High quality Tasks / Components for AWS integration. See below list.

Amazon AWS Cloud Integration
Amazon S3 Task	Amazon Redshift Data Transfer Task	Amazon Redshift ExecuteSql Task
Amazon Redshift Cluster Management Task	Amazon DynamoDB Source	Amazon DynamoDB Destination
Amazon Redshift Source	Amazon SQS Queue Source	Amazon SQS Queue Destination

Conclusion

Amazon AWS Cloud integration from SSIS packages becoming more and more common scenario. ZappySys provides easy to use no coding connectors to achieve many time consuming scenarios. Having clean drag and drop approach is not only faster but more secure using inbuilt SSIS framework. Try SSIS PowerPack to explore many other scenarios not discussed in this article.

The post How to call Amazon AWS API using SSIS (EC2, Lambda, API Gateway, SQS) appeared first on ZappySys Blog.