aws Archives | ZappySys Blog

Download the Latest File from Amazon S3 Storage using SSIS

ZappySys — Fri, 27 Dec 2019 12:33:04 +0000

Introduction

In our previous article, we saw SSIS Amazon S3 Storage Task examples. Now let’s look at how to Download the Latest File from Amazon S3 Storage using SSIS. Microsoft SSIS includes the Amazon S3 Storage Task that allows us to upload files to Amazon S3 Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Amazon S3 Storage task included in ZappySys SSIS PowerPack.

In nutshell, this post will focus on how to Download the Latest File from Amazon S3 Storage.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from the Microsoft site).
Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
Make sure ZappySys SSIS PowerPack is installed (download it, if you haven't already).
(Optional step). Read this article, if you are planning to deploy packages to a server and schedule their execution later.

What is Amazon S3 Storage

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements. Amazon S3 is designed for 99.999999999% (11 9’s) of durability, and stores data for millions of applications for companies all around the world.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Amazon S3 Storage Task that will allow you to access files/folders from Amazon S3 Storage to the Local machine, Upload files(s) to Amazon S3 Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to Download the Latest File from Amazon S3 Storage.

Step-By-Step to Download the Latest File from Amazon S3 Storage

Let´s start with an example. In this SSIS Amazon S3 Storage Task File task example, we will download the latest file using Amazon S3 Storage Task.

First of All, Drag and Drop Amazon S3 Storage Task from SSIS Toolbox and double click it to edit
Drag and Drop Amazon S3 Storage Task
The following options are available in Amazon S3 Storage Task:
Options to upload, download files in SSIS
You can also overwrite files and folder, skip all, overwrite if the source is older, overwrite if the file size is different, rename files and more:
Overwrite files in SSIS
Let’s select the Download Amazon file(s) to local directory. Now select/add The Source path, Check Recursive Scan(*Optional) to get all the sub-files and subfolders and files, after that add Target Path if you want to, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
SSIS Amazon S3 Storage Task – Download files from Amazon S3 Storage
Now go to Advanced Filter & Sorting and configure it like below screen to download the latest file only.
Storage Task configuration to download the latest file only using sorting
That’s it now execute the package and it will download the latest file from the Amazon S3 Storage Task to Local File Machine.
Sometimes you may need to make advanced filtering on file path, file date, size, etc. for that refer to this article for more details.

Conclusion

To conclude, we can say that working with Amazon S3 Storage is now very simple. In this article, we looked at how to Download the Latest File from Amazon S3 Storage using SSIS. We used Adnavced Filtering & sorting option. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here.

Thanks for your time.

References

The post Download the Latest File from Amazon S3 Storage using SSIS appeared first on ZappySys Blog.

How to Read / Write / Delete Amazon SQS Queue data in SSIS

ZappySys — Mon, 19 Aug 2019 12:17:48 +0000

Introduction

ZappySys provides high-performance drag and drop connectors for Amazon AWS Integration. In this post, we will see How to Read / Write / Delete Amazon SQS Queue data in SSIS using ZS Amazon SQS components. ZappySys developed many AWS related components but in this article, we will look at 2 components for Amazon SQS Queue integration Scenarios (Read and Write). We will discuss how to use SSIS Amazon SQS Source and SSIS Amazon SQS Destination. Amazon SQS Source can be used to read messages from Amazon AWS SQS (Simple Queue Storage) and Amazon SQS Destination can be used to write/delete data from Amazon SQS (Simple Queue Storage) using SSIS.

Prerequisite

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

First, you will need to have SSIS installed.
Secondly, make sure to have SSDT.
Thirdly, you have obtained Amazon SQS Credentials.
Finally, do not forget to install ZappySys SSIS PowerPack.

Component Mention in this article

	Amazon SQS Source Connector
	Amazon SQS Destination Connector

What is Amazon SQS (Simple Queue Service)

Before we talk How to Read / Write / Delete Amazon SQS Queue data in SSIS lets talk What is Amazon SQS (Simple Queue Service) or sometimes referred to as AWS SQS. Amazon SQS offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues and costs allocation tags. It provides a generic web services API and it can be accessed by any programming language that the AWS SDK supports.

SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

Getting Started

In order to start, we will show several examples. ZappySys includes an Amazon SQS Source and Amazon SQS Destination, that will help you How to Read / Write / Delete Amazon SQS Queue data in SSIS. You can connect to your Amazon SQS account by entering credentials.

Amazon SQS Source can be used to read messages from Amazon AWS SQS (Simple Queue Storage) using SSIS. Read Amazon SQS Messages (AWS SQS) in bulk. Amazon SQS Destination can be used to write/delete data from Amazon SQS (Simple Queue Storage) using SSIS. Write messages to Amazon SQS (AWS SQS) in bulk. Support for Visibility Timeout (Hide messages from others for certain time after you read), data preview and max rows, reading message attributes in separate output columns and adjusting UTC DateTime in local time.

Write Data using Amazon SQS Destination in SSIS

Overview

In this tutorial, you will learn how to write data in Amazon SQS Queue storage. You can use ZS DummyData Source for practice it or select suitable ZS Source Adapters. In this case its from ZS DummyData Source. So let’s start with an example, In this section, you will learn how to write messages into Amazon SQS Queue. For this example, we are assuming that you have already SQS Queue Created.

Create Amazon SQS Connection

Firstly, In order to connect to Amazon SQS from SSIS, you will need Access Key and Secret Key (Login to AWS portal and Create Access/Secret Key with SQS service read access). If you are not sure about this ask your SysAdmin or responsible person to provide that to you. Your key will look something like this: (This is just an example key which may differ in your case).
```
AccessKey: AKIAIOSFODNN7EXAMPLE
SecretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
```
Click here to read more about how to get your AccessKey and Secret Key.
Secondly, Once you have Account Key (it’s like UserID) and Secret Key (its like password) you may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
Right-click in Connection Manager Panel and Select [New Connection] menu item.
SSIS Create AWS Storage Connection
Now Select ZS-AWS-STORAGE connection type.
Select ZS-AWS-STORAGE Connection
Let’s configure the Amazon SQS connection. Set Storage Service to SQS and Credential Type to Key-based access control. In the General tab enter your Amazon SQS Credentials and Set Default region as per you have.
SSIS AWS Storage Connection – Configure
In the last, click on Test Connection and OK button to Save connection configure setting UI.

Write Data into Amazon SQS

Once you have created the Amazon SQS Connection successfully, let’s drag and Drop Data Flow Task from SSIS Toolbox on the control flow designer surface.
Drag and Drop SSIS Data Flow Task from SSIS Toolbox
Double click on the Data Flow Task to see Data Flow Designer surface.
Here, from the SSIS Toolbox drag and drop ZS DummyData Source on the data flow designer surface.
SSIS ZS Dummy Data Source – Drag and Drop
Double click on Dummy Data Source to configure it.
SSIS ZS Dummy Data Source – Configure
Click on OK button to save configure setting UI.
From the SSIS toolbox drag and drop Amazon SQS Destination on the Data flow designer surface and single click on the Dummy Data Source, Once you see the blue arrow from the source. connect it to Amazon SQS Destination.
SSIS ZS Amazon SQS Destination
Now double click on Amazon SQS Destination to configure it.
In the [Connection Manager] tab select SQS connection manager we have created before it.
SSIS Amazon SQS Destination – Configure Connection Manager
Click on [Component Properties] tab and Select Target Queue from the dropdown list.
SSIS ZS Amazon SQS Destination – Configure Component Properties
Click on [Mappings] tab and verify mapping. Map document column from upstream to the MessageText target column.
SSIS ZS Amazon SQS Destination – Column Mappings
In the last, click on the OK button to save configure setting UI.
After all, Execute the Package and verify target data by going to your AWS console under SQS service section.
Run or Execute SSIS Package

Verify Amazon SQS Destination

Read Data using Amazon SQS Source in SSIS

Overview

In this tutorial, you will learn how to read data from Amazon SQS storage in SSIS using ZS SSIS Amazon SQS Source Adapter. In this SSIS we will read data. So let’s begin it.

Read Data from Amazon SQS

Firstly, create a new SSIS package. In BIDS/SSDT create a new SSIS package.
From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
SSIS Data Flow Task – Drag and Drop
Double click on the DataFlow task to see DataFlow designer surface.
From the SSIS toolbox drag and drop ZS Amazon SQS Source on the dataflow designer surface.
Double click Amazon SQS Source to configure it.
Select Amazon SQS Connection we have created before and select queue from the dropdown list.
SSIS Amazon SQS Source – Configure
Click on Preview button to see Data Preview and Click on OK button to save settings.
From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface or you can use any of our suitable ZS Destination Adapter.
SSIS ZS Trash Destination – Drag and Drop
Now single click on the Amazon SQS Source once you see the blue arrow from the source … connect it to Trash Destination.
Double click on ZS Trash Destination to Configure it.
SSIS ZS Trash Destination – Configure
Click on OK button to save Trash Destination configure setting UI.
After all, right-click on the path and Add Data Viewer and Execute the package and verify source data in the data viewer.
Run or Execute The SSIS Package
After messages read from SQS Queue, the same message cannot be read again until VisibilityTimeout. By default Queue, visibility timeout is used (-1 means use default) but you can override it on the source UI.

Read all messages from SQS FIFO Queue

If you are using Normal SQS Queue then by setting Max Rows=0 will read all messages from your queue. However reading all messages from FIFO Queue is not that simple. Amazon SQS wont return more messages until you delete previously read Messages from same Message Group (identified by MessageGroupId). Due to this constraint you have to try below technique of Loop to read 10 messages until all messages Read > Processed > Deleted.

Here is note from Amazon Documentation about maximum 10 messages limitation on FIFO Queue.

When you receive a message with a message group ID, no more messages for the same message group ID are returned unless you delete the message or it becomes visible.

Community Reference:

https://stackoverflow.com/questions/51661834/aws-sqs-fifo-how-to-get-more-than-10-messages-at-a-time

If you want to add fault tolerance then you can set Max Retry for NULL Receive like below (e.g. Setting MaxRetry=3 will try max 3 times if no message is read …. It will try every 2 seconds for 3 times before giving up to read more rows).

Read all messages from Amazon SQS FIFO Queue

Read , Process and Delete Amazon SQS FIFO Queue Message in SSIS Data flow

Delete Message from Queue using Amazon SQS Destination in SSIS

Overview

In this tutorial, you will learn how to Delete Messages from the Queue using the ZS SSIS Amazon SQS Destination Adapter. So let’s begin it.

Delete Message from Queue

From the SSIS toolbox drag and drop Amazon SQS Destination on the Data flow designer surface and connect it to Amazon SQS Destination.
Now double click on Amazon SQS Destination to configure it.
In the [Connection Manager] tab select SQS connection manager we have created before it.
Select Amazon SQS Connection
Click on [Component Properties] tab, Set Operation to Delete and Select Target Queue from the dropdown list.
Select Delete Operation and Queue
Click on [Column Mappings] tab and verify mapping. Map document column from upstream to the MessageText target column.
Do Columns Mapping
In the last, click on the OK button to save configure setting UI.
After all, Execute the Package and verify target data by going to your AWS console under SQS service section.

Purge All Messages from SQS Queue (Clear Queue)

There will be a time you like to clear entire queue (i.e. Delete All Messages from Queue). Amazon recently added PurgeQueue API feature so now its possible without doing Delete message technique mentioned in previous section. ZappySys support calling any AWS API using REST API Task along with AWS Provider in OAuth Connection. You can use same technique to call PurgeQueue API like below.

Open SSIS Package and Drag ZS REST API Task from Control Flow Toolbox
Click on URL mode dropdown and change to URL from Connection.
From COnnection Dropdown select NEW ZS-OAuth connection
Configure OAuth connection to use Provider=Amazon AWS v4
1. Enter Access Key and Secret Key and click OK to save connection
Enter below URL on REST API Task (Replace Queue ID and name with your own values) – Refer to your AWS Console / SQS Queue Properties to get this URL
```
https://sqs.ENTER-REGION-HERE.amazonaws.com/ENTER-QUEUE-ID-HERE/ENTER-QUEUE-NAME-HERE/?Action=PurgeQueue
```
Here is an example of Sample Queue from ZappySys Demo
```
https://sqs.us-east-1.amazonaws.com/718883169616/zappysystest.fifo/?Action=PurgeQueue
```
Click ok to Save REST API Task

How to Purge Amazon SQS Queue in SSIS (Clear Queue / Delete All messages using PurgeQueue REST API)

Conclusion

So, In this blog, we learn How to Read / Write / Delete Amazon SQS Queue data in SSIS. We used Amazon SQS Destination for write data and Amazon SQS Source to read data. Additionally, you can download SSIS PowerPack here to try many other scenarios that are not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

Amazon SQS Source
Amazon SQS Destination
Documents of Amazon SQS
Help File:
- Amazon SQS Source
- Amazon SQS Destination

The post How to Read / Write / Delete Amazon SQS Queue data in SSIS appeared first on ZappySys Blog.

SSIS Amazon S3 Storage Task examples (Download, Upload, Delete Files / Folders)

ZappySys — Sat, 06 Apr 2019 10:22:55 +0000

Introduction

In this article, we will show SSIS Amazon S3 Storage task examples. Uploading files using Amazon S3 Storage Task or downloading files are common requirements in SSIS. In order to work with Amazon S3 Storage, Microsoft SSIS includes the Amazon S3 Storage Task that allows to upload files to Amazon S3 Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Amazon S3 Storage task included in ZappySys SSIS PowerPack.

Amazon S3 Storage Task can be used to perform various operations with Amazon S3 Storage objects (files and containers) (e.g. Download, Upload, Copy, Move, Rename, Delete, etc). You can also get single property of any Amazon file or get a list of Amazon files as ADO.net Table and which can be easily looped through using ForEachLoop Task. Most of the operations in this task support recursive search using wild card pattern (e.g. Myfile*.txt).

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
You have obtained Amazon S3 account access key / secret key.
Finally, do not forget to install ZappySys SSIS PowerPack

What is Amazon S3 Storage

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Amazon S3 Storage Task that will allow you to access files / folders from Amazon S3 Storage to the Local machine, Upload files(s) to Amazon S3 Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to upload, download, move and delete files / folders from Amazon S3 Storage.

SSIS Amazon S3 Storage Task Examples

Let´s start with an example. In this SSIS Amazon S3 Storage Task File task example, we will upload, download, move and delete files / folders from Amazon S3 Storage.

First of All, Drag and Drop Amazon S3 Storage Task from SSIS Toolbox and double click it to edit
Drag and Drop Amazon S3 Storage Task
The following options are available in Amazon S3 Storage Task:
Options to upload, download files in SSIS
You can also overwrite files and folder, skip all, overwrite if source is older, overwrite if the file size is different, rename files and more:
Overwrite files in SSIS
Let’s select the Download Amazon file(s) to local directory. Now select/add The Source path, Check Recursive Scan to get all the sub files and sub folders and it’s files and add Target Path, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
SSIS Amazon S3 Storage Task – Download files from Amazon S3 Storage
That’s it now execute the package and it will download all the files and sub folders from the Amazon S3 Storage Task to Local File Machine.
Now Let’s configure the task to upload the local directory files and folder to Amazon S3 Storage like below and execute the package. And also in this example i created the variable and stored target folder path in it and used that variable Target Path Variable.
SSIS Amazon S3 Storage Task – Upload files from Amazon S3 Storage
Let’s configure the task to delete Amazon S3 Storage files like below and execute the package.
SSIS Amazon S3 Storage Task – Delete files from Amazon S3 Storage
Let’s configure the task to move Amazon S3 Storage files to another directory like below and execute the package.
SSIS Amazon S3 Storage Task – Move files from Amazon S3 Storage
That’s it like upper examples you can configure Amazon S3 Storage Task to use it’s other options also.
Let’s see other SSIS Amazon S3 Storage Task examples in below sections

Working with advanced filters

In order to understand filters, we will create an example for you. This time we will create SSIS Amazon S3 Storage task example to show how to exclude some files from the downloading process.

In addition, it is also possible to add advanced filters to the SSIS Amazon S3 Storage task to exclude or include files and folders. The following example shows how to use the Advanced filters. In the Amazon S3 Storage Task, go to the Advanced Filter page and in Exclude RegX we will write \.exe$ to exclude the file with exe extension to be downloaded. You can use RegX (regular expressions) in the advanced filter:
Advanced filters

Working with wildcards in SSIS Amazon S3 Storage task (Multiple files only)

In order to understand wildcards, the new example will show how to work with Wildcards will help you to work with multiple files or folder with common characteristics.

The following example shows how to copy all the files will all the extensions in the source folder. We use the *.*, which means that any file name with any extension will be downloaded:
Working with Wildcards
If we want to download all the files with extension txt, use the following Path:
```
/source/*.txt;
```
Another example would be to download all the files with the following names: file1.txt, file2.txt and file3.txt. We could use the following Path:
```
/source/file?.txt
```

Working with SSIS expressions and variables in Amazon S3 Storage Task

In order to understand expressions, the following example will show how to work with SSIS expressions combined with variables in the SSIS Amazon S3 Storage task. In this example we will delete files in Amazon S3 Storage using variables and expressions.

ZappySys Providers simple and powerful Placeholders. You can still use SSIS expression language. Placeholders allow direct Variable Use easy to see compared to hidden SSIS Expression (C# style). Placeholders have simple embedding use of variables.

First, we will first create a variable in SSIS. In this example the file name is MyFile and the Value file1.
Create an SSIS variable
As a second step, we will write in path the value \destination\{{User.MyFile}}.txt. This will concatenate the folder destination with the variable MyFile. The task will delete the file1.txt from the Amazon S3 Storage folder.
Using SSIS expressions
The next example will show how to create a Amazon S3 Storage file using expressions and variables in SSIS:
Create file
Content to write is the content of the file. In Target, go to path and write the following: For more information on placeholders check this post and also check this post.
```
/destination/{{User::FileName,MM-dd-yyyy}}
```
We are using a variable in SSIS named FileName of type DateTime. We are concatenating the Amazon S3 Storage folder destination with the variable. As you can see, the expressions with ZappySys are very simple.

Example to verify if file exists in SSIS Amazon S3 Storage task

Finally, we will verify if a file in Amazon S3 Storage exists or not using the SSIS Amazon S3 Storage task.

In order to start, we will use the SSIS variables, create a variable of Boolean type named FileExists. In Storage Action, select Get Amazon file exist status. In path, select the path of the file to verify if it exists or not.
SSIS Amazon S3 Storage Task – Check File Exist in Amazon S3 Storage
Secondly, we will use the ZS Logging Task to check the value of the variable File Exists. This task will show the value of the variable in the log. Drag and drop the task and join it with the ZS Amazon S3 Storage Task:
The SSIS Log task
In order to get the value, we will insert it in the Logging Task. To do that, insert the Variable and select the FileExists variable:
Add the variable in the task logging
Finally, run the package and verify the Output (you can use View and Output to see the output) and check the value of the Logging Task. If the file exists, the value will be true. If it does not exist, the value will be false.
Verify the variable in Output

How to loop through Amazon S3 Storage files in SSIS

You can also loop through files to perform custom action file by by. Amazon S3 Storage Task support Action to get File List as ADO.net Recordset

To get File list from Amazon S3 Storage Server perform the following steps.

From Storage Action select Get Amazon files as ADO.net DataTable

Get files as ADO.net DataTable
Create new Amazon S3 Storage Connection by clicking New next to connection dropdown.
Select Source Path (e.g \myroot\folder\*.txt )
Select or create new SSIS Variable (e.g. varFiles ) which will hold file list result (Must be Object datatype variable).
Click OK to close.
Create 3 more SSIS variables to hold File Path (String), Name (String) and Size (Int32). We will use in the next step.
Now drag SSIS ForEach Loop Container Task from SSIS Toolbox
Edit Loop Task and Select Foreach ADO Enumerator from the Enumerator dropdown
Select variable (e.g. varFiles)

Loop through files in SSIS using ForEach Loop Task (use ADO.net Recordset variable)
Click on Variable Mappings Tab and configure like below.
Variable Mappings for ForEach Loop Task – Loop through files in a folder using SSIS
Drag ZS Logging Task and place inside Loop and Display variables
That’s it

For more information to loop through files on local machine using SSIS check this article.

Conclusion

To conclude, we can say that working with Amazon S3 Storage is now very simple. In this article, we looked at SSIS Amazon S3 Storage Task examples on how to upload, download, delete files and folders. We used variables, wildcards and expressions. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here. We saw different SSIS Amazon S3 Storage Task examples to understand how to use the task in different scenarios.

Thanks for your time.

References

The post SSIS Amazon S3 Storage Task examples (Download, Upload, Delete Files / Folders) appeared first on ZappySys Blog.

Read Amazon S3 Storage Files in SSIS (CSV, JSON, XML)

ZappySys — Sat, 16 Mar 2019 09:43:13 +0000

Introduction

In our previous blog we saw how to load data into Amazon S3. Now in this blog, we will see How to read Amazon S3 Storage Files in SSIS (CSV, JSON, XML Format files). To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like perform File operations on Amazon S3 Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on how to Read Amazon S3 Storage CSV, JSON and XML Files using respective SSIS Source tasks.

Components Mentioned in this article

	Amazon S3 CSV File Source
	Amazon S3 CSV File Destination
	Amazon S3 XML File Source
	Amazon S3 JSON File Source

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
You have obtained Amazon S3 account access key / secret key.
Finally, do not forget to install ZappySys SSIS PowerPack

What is Amazon S3 Storage

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Amazon S3 Source for CSV/JSON/XML File that will help you in reading CSV, JSON and XML Files from Amazon S3 to the Local machine, Upload files(s) to Amazon S3 Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to download files from Amazon S3 Storage.

You can connect to your Amazon S3 Account by entering your storage account credentials.

Read Amazon S3 Storage Files in SSIS (CSV, JSON, XML)

Let´s start with an example. In this SSIS Amazon S3 Source for CSV/JSON/XML File task example, we will read CSV/JSON/XML files from Amazon S3 Storage to SQL Server database.

First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
Drag and Drop SSIS Data Flow Task from SSIS Toolbox
Drag and Drop relevant Amazon S3 Source for CSV/JSON/XML File Task from the SSIS Toolbox.
Add Amazon S3 Source Tasks
Create a connection for Amazon S3 Storage Account.
Create Amazon S3 Storage Connection
Select the relevant single file to read from Amazon S3 Storage in their relevant source of CSV/JSON/XML File Task.
Select File From Amazon S3 Storage
We can also read the multiple files stored in Amazon S3 Storage using wildcard pattern supported e.g. dbo.tblNames*.csv / dbo.tblNames*.json / dbo.tblNames*.xml in relevant source task
Use wildcard pattern .* to read multiple files data
We can also read the zip and gzip compressed files also without extracting it in the specific Amazon S3 Source for CSV/JSON/XML File Task.
Reading zip and gzip compressed files (stream mode)
Finally, we are ready to load this file(s) data into the SQL Server.

Load Amazon S3 files data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

Drag and drop the Upsert Destination component from the SSIS Toolbox.
Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

Double-click the Upsert Destination component to open the configuration window.
Under Connection, select an existing target connection or click NEW to create a new connection.
- Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

In the Target Table dropdown, select the table where you want to load data.
Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

Go to the Mappings tab.
Click Auto Map to map source columns to target columns by name.
Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

Above all, in this blog, we learned how to Read Amazon S3 Storage Files in SSIS. We used Amazon S3 Source for CSV file, Amazon S3 Source for JSON file and Amazon S3 Source for XML file to read the file(s) from Amazon S3 Storage and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

The post Read Amazon S3 Storage Files in SSIS (CSV, JSON, XML) appeared first on ZappySys Blog.

Setting Metadata key / value in SSIS Amazon S3 / Azure Storage Task

ZappySys — Thu, 17 Jan 2019 15:21:54 +0000

Introduction

In this post we will see how to set Metadata key/value in Azure Blob Storage Task and Amazon Storage Task (S3 Task). We will also see how to set them dynamically at runtime (Use SSIS Expression).

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from the Microsoft site).
Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
Make sure ZappySys SSIS PowerPack is installed (download it, if you haven't already).
(Optional step). Read this article, if you are planning to deploy packages to a server and schedule their execution later.

Set Amazon S3 Task Metadata (Key / Value Pairs)

Here is how you can set Metadata in Amazon S3 Task.

Setting Amazon S3 Metadata (Key/Value Pairs) in SSIS Amazon Storage Task

Set Azure Blob Storage Task Metadata (Key / Value Pairs)

Just like S3 Task you can use same UI in Azure blob storage task too.

Set Metadata Key/Value Dynamically (SSIS Expression)

So far you have set metadata using static value. What if you have to set Dynamically at runtime?

Basically Metadata is passed as XML Format in the task like below.



	
		x-amz-key1
		AAAAAAA
	
	
		x-amz-key2
		BBBBBBB

Whate we will do is take above format and make it dynamic using SSIS Expression.

Perform the following steps for that.

Select your Amazon Storage Task or Azure Storage Task and right click > Select Properties.
In the Properties Grid check Property name MetadataKeyValue. Copy XML Value from there (We will use it in expression)
Now double click Amazon Storage Task or Azure Storage Task
Click on Expression Button
Select MetadataKeyValue property and for Value enter below expression… Use previously copied XML text and change Value part with your own SSIS variable. In below example we are passing Just one Metadata Key/Value pair ( Name=x-amz-meta-filename , Value=Coming from @[User::MyFileNameVariable] )
Note that we have escaped double quote with
```
\"
```
and replaced static value with
```
" + @[User::MyFileNameVariable] + "
```
.
```
""
+ ""
+ "	"
+ "		x-amz-meta-file-name"
+ "		" + @[User::MyFileNameVariable] + ""
+ "	"
+ "	"
+ "		x-amz-meta-something"
+ "		BBBBBBB"
+ "	"	
+ ""
```
Set Amazon S3 File Metadata Key/Value dynamically
Click OK and that’s it.

The post Setting Metadata key / value in SSIS Amazon S3 / Azure Storage Task appeared first on ZappySys Blog.

Copy Amazon S3 Files in SSIS to different AWS Account

ZappySys — Mon, 29 Oct 2018 21:12:58 +0000

Introduction

In our previous blog we saw how to upload data to Amazon S3 now let’s look at how to Copy Amazon Files from one AWS account to another AWS account (Server Side Copy) using SSIS Amazon Storage Task. (Must be version v2.7.9 or higher)

If you are doing file copy within same account then there is no issue. Just use Copy File feature. However if you want to copy between multiple AWS Accounts (e.g. Your Company AWS account to Vendor AWS Account) then you have to perform extra steps.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from the Microsoft site).
Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
Make sure ZappySys SSIS PowerPack is installed (download it, if you haven't already).
(Optional step). Read this article, if you are planning to deploy packages to a server and schedule their execution later.

Amazon S3 File Copy Scenario

Now let’s think about a scenario where you have to copy some files from Account-A to Account-B. Account-B can be owned by totally different company (may be your Vendor) and you don’t have security control of it.

Before we can copy from Account-A to Account-B we first have to Configure permissions in Account-B. Basically when you use SSIS S3 Task for Copy operation it uses Source account (i.e. Account-A) to push files to destination account (Account-B). So you have Allow write permission for Source Account-A in Destination AWS Bucket (i.e. Account-B).

Find Canonical ID of Source Account

First login to Account-A and findout Canonical ID (Usually 64-digit long)
To get Canonical ID click on Your Account Name > My Security Credentials
When prompted click Continue to Security Credentials option.
Expand Account Identifier and copy Canonical ID
How to find AWS Account ID and Canonical User ID

Configure Target S3 Bucket Permissions to allow Source Account (Write Access)

Now once you have Source account Canonical User ID, you can proceed to next step.

Log in to Target account (i.e. Account B). Where you like to copy files.
Go to S3 Area and Click on the target bucket where you like to copy files
Go to Security Tab
Add Source Account Canonical ID and Check Write Permission (List is Optional for most).
Give Write permission in Target AWS S3 Bucket (Enter Amazon Source Account Canonical User ID)

Copy Amazon S3 Files from one Account Bucket to Another

Now once you configure permission on target bucket. We are ready to execute Amazon S3 File Copy Operation in SSIS

Open SSIS Package in Visual Studio
Drag ZS Amazon Storage Task from SSIS Toolbox
Double click it to configure
Select Copy File Operation from Action Dropdown
Click New for Source Connection (Enter Source Account Credentials). Click Test and OK to save.
Now Click Browse to Select Source Files you like to Copy. You can use Wild card pattern for multiple file copy.
Example: mysourcebucket/subfolder/localsubfile*.csv
Click New for Target Connection (Enter Target Account Credentials). Click Test and OK to save.
Now Click Browse to Select Target Folder Location where you like to Save Copied files.
Example: mytargetbucket/somefolder/
Click OK to Save
Execute SSIS Package to test

Copy Amazon S3 Files using SSIS (Copy Files between multiple Amazon Accounts / S3 Buckets)

Conclusion

In this post you have seen how easy it is to copy S3 Files between multiple AWS accounts in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to Automate Amazon S3 related operations in SSIS.

The post Copy Amazon S3 Files in SSIS to different AWS Account appeared first on ZappySys Blog.

How to delete Amazon S3 File / Folder / Bucket in SSIS

ZappySys — Thu, 20 Sep 2018 13:15:50 +0000

Introduction

In our previous posts we looked at how to use SSIS Amazon Storage Task (Amazon S3 Task) to automate some Amazon S3 related tasks in SSIS. In this blog post will continue discovering more use cases. Let’s learn how to delete Amazon S3 File, delete Amazon S3 Folder and delete Amazon S3 Bucket.

Deleting S3 Files are straight forward using SSIS Amazon Storage Task (Amazon S3 Task but deleting Bucket or Folders require some additional checks and steps, so let’s look at that in depth.

Requirements

Before we look into Step-By-Step section to call Amazon S3 related operations in SSIS let’s make sure you met the following requirements.

SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
You have obtained Amazon S3 account access key / secret key.

Configure SSIS Amazon S3 Connection

Before we perform any S3 related action, you must configure Amazon Storage Connection. This connection manager support connectivity to not only S3 Storage but also other AWS Storage such as DynamoDB, Redshift

There are two ways you can create Amazon Storage Connection Manager in SSIS designer. You can right click in Connection Manager Panel and create new ZS-AMAZON-STORAGE connection type or use Task UI and click New connection.

For now, we will use easier method which is Create S3 Connection from S3 Storage Task UI.

Open SSIS Package
Drag ZS Amazon S3 Storage Task from the toolbox
Double click it to edit. You can now click on New Connection like below.
Create New Amazon S3 Connection from Task UI
On the connection Manager UI. Select S3 as service type. Enter your Access Key, Secret Key and Select Region. We also support IAM Role for EC2 instances (For this package must run inside EC2 VM). Check this article.
Configure SSIS Amazon S3 Connection – Select Region, Enter Access Key / Secret Key
Click Test connection to confirm its working. Click OK to Save Connection Manager UI.
Back to Task UI click OK to close save (We will configure task in later sections.

Using SSIS Amazon S3 Task for common actions

Before we deep dive in delete scenarios let’s first look at what is SSIS Amazon Storage Task (Amazon S3 Task). This task is developed to achieve many S3 related operations. See below screenshot to get idea about many actions you can perform using just one task. In this post we will focus only on delete actions (for file, folder and bucket). In other posts we will cover different operations in depth.

SSIS Amazon S3 Task Operations – Upload, Download, Delete, Create, List, Rename Files, Folders, Buckets

Delete Amazon S3 Files

Once you have configured SSIS S3 Connection Manager, we can perform different S3 related operations using many different Amazon S3 related tasks / components. For now let’s just look at how to delete files from S3 bucket.

Delete Multiple S3 files – Using Simple Pattern Search (wildcard)

Here is how to delete amazon s3 file / files using wildcard pattern search. Using * or ? allowed. (Example SalesOrder_*.??? )

Delete Amazon S3 File (Multiple) using Wildcard pattern

Delete Multiple S3 files – Using Advanced Search – Regex / SQL Expression

Sometimes it’s necessary to filter file based on advanced attributes such as file Age (How old), File Size, Multiple File Patterns (e.g. Delete *.zip, *.gz, *.rar in one action)

You can use Advanced Tab to filter further. First supply primary filter on Source Tab (e.g. mybucket/*.* ) and then supply your advanced filter (e.g. you can exclude *.exe and *.msi file from your delete list as below screenshot, also only delete files between 1000 and 50000 bytes in file size).

Advanced Search in SSIS Amazon S3 Task – Use Regular Expression Patterns, SQL like expression for Sort / Filter

//////////////////////////////
Examples:
//////////////////////////////
AgeInDays < 5  //Get files created in last 5 days
LastEditInDays > 3 //Get files modified before 3 days
AgeInDays > 10 AND LastEditInDays < 3  //Get files created before 10 days and modified in last 3 days
Size < 1024  //Get files where File size is less than 1024 bytes
CreationDate BETWEEN #2015-01-01# AND #2015-01-01#   
Name IN ('File1.txt', 'Download.msi')
FullPath LIKE 'mybucket\somefolder\month_*')
Extension IN ('.txt','.csv')
LastModifiedDate > #2013-01-01T16:00:00#
IsReadOnly = True
/////////////////////////////////////////
//Supported Columns for expression:
/////////////////////////////////////////
Name, FullPath, Extension, FolderPath, AgeInDays, LastEditInDays, Size, CreationDate, CreationDateUtc, LastModifiedDate, LastModifiedDateUtc, IsReadOnly

Delete Amazon S3 files older than X days

If you like to delete files older than certain days then use below technique.

First Specify primary search criteria on Source tab
Go to Advanced Filter Tab and specify Following Pattern in Where Textbox (For example delete Files older than 10 days )
```
AgeInDays > 10
```

Delete Amazon S3 Bucket

Now let’s look at how to delete Amazon S3 Bucket including all its content. There is an option to delete Bucket but it wont delete files and empty folders. So you have to first clear all its content before calling Delete bucket action.

Here is high level steps to delete bucket in SSIS using ZappySys S3 Task.

First call Delete S3 Files Action using Source Path = mybucket/*. Delete file action won’t delete any real folders (i.e. name ends with slash).
Once files are deleted from bucket you can get list of empty folders
1. Select Action = Get Amazon folder list as ADO.net Table
2. In the path enter Path = mybucket/*
3. In the Result Variable Create or Select variable (e.g. User::FolderList)
Use ForEachLoop Task to Delete empty folders (Use Delete File Action because there is no folder concept in S3. File name ends with slash is treated as folder)
1. In the ForEachLoop Task select enumerator as Foreach ADO enumerator
2. In the variable select a variable which holds list of folders (obtained in previous step)
3. On the Variable Mappings Tab. Select Some variable (e.g. User::FolderPath) to map to Index =0
4. Drag new S3 Task inside ForEachLoop and Select Delete File Action
5. In the File Path enter variable as {{User::FolderPath}} (or whatever name you given which holds iteration FolderPath in previous step)
Call Delete Bucket Action once bucket is totally empty.

Delete Amazon S3 Bucket (including files and folders)

Delete Amazon S3 Folders

There is no real folder concept in Amazon S3 file system. S3 File System is just key / value system. Key is basically file name (e.g. somefolder/subfolder/myfile.txt). You can use slash in name to indicate hierarchy which can be used by some client apps like ZappySys S3 File Picker to Mimic Treeview like structure so user feel like real file system but there are no rules if you have to use slash or not.

Most client tools for S3 treats any file name ends with slash as real folder. See below step to delete amazon s3 folder (real folder which was created explicitly – mentioned in next section).

Open S3 Task and select action = Delete Amazon File(s)
In the path enter your Folder Path (e.g. mybucketname/output/sub2/ )

Create Amazon S3 Folder

Some time you may want to create real S3 folder which is not treated as file. In that case use this technique to create s3 folder.

Conclusion

In this post we saw how easy it is to use SSIS Amazon Storage Task (Amazon S3 Task) to delete Amazon S3 file, folder and bucket. You can also perform other common amazon storage related actions (e.g. Upload / Download / Create / List ). Download SSIS PowerPack to try this component and 70+ other tasks / components which are not covered in this article.

The post How to delete Amazon S3 File / Folder / Bucket in SSIS appeared first on ZappySys Blog.

Read Amazon S3 data in Power BI or Call AWS REST API (JSON / XML)

ZappySys — Fri, 15 Jun 2018 23:03:05 +0000

Introduction

If you are looking around to find connectivity options to get Amazon AWS data in Power BI (e.g. Read from Amazon S3 files (CSV, JSON, XML) or get AWS API data such as Billing Data by calling REST API) then unfortunately as of now Power BI doesn’t support it natively.

In this article, you will learn How to read Amazon S3 data in Power BI (Using S3 CSV Driver) and in later section we will also learn How to import data from AWS API Calls (e.g. Billing, EC2, S3, DynamoDB). We will use ZappySys REST / JSON Driver to access AWS data without any ETL or Programming.

In our previous article, we discussed how to access REST API data in Power BI (Read JSON / XML Web API). Now let’s learn how to connect to AWS in Power BI.

In this article, We will explore interesting scenarios such as below. It will help you to connect Amazon AWS in Power BI and import data from AWS without doing any ETL or programming.

Import JSON file from S3 bucket in Power BI (Using Amazon S3 Driver for JSON Files).
Read CSV file from S3 bucket in Power BI (Using Amazon S3 Driver for CSV Files).
Read XML file from S3 bucket in Power BI (Using Amazon S3 Driver for XML Files).
Call Amazon AWS REST API (JSON or XML) and get data in Power BI. Some examples of API calls.
- Get EC2 VM count and their Status in Power BI Dashboard
- Read data from AWS Athena Service
- Use AWS Cost / Billing API to display monthly Cost by Service
- Display summary of Redshift Clusters
- Extract response from Lambda function and display in Power BI

So let’s get started…

Requirements

First, you will need to have Power BI Desktop (FREE) installed
Make sure you have AWS Access Key and Secret Key to access AWS data or call API for the desired service
On third place, our ZappySys ODBC Power Pack

Import Amazon S3 data in Power BI (CSV, JSON or XML Files)

In this section we will look at step by step approach to load Amazon S3 data in Power BI. Your file can be compressed (GZip, Zip) or un-compressed. We will use ZappySys Amazon S3 driver for CSV Files for demo but you can also use Amazon S3 JSON Driver or Amazon S3 Driver for XML Files same way for reading different file formats.

Configure ODBC DSN for Amazon S3 Files ( JSON / XML / CSV )

Type odbcad32.exe in your search box and launch the DSN Config utility.
you want access for yourself then stay of User DSN Tab. If you want grant access other users then go to System DSN tab. For SQL Server Integration go to System Tab and add new System DSN rather than User DSN. Click New button.
From the Driver list Select ZappySys ODBC Driver. For this example select [ZappySys Amazon S3 CSV Driver].
Now, we need Amazon S3 Storage Connection. Lets create it.
Now, When you see DSN Config Editor with zappysys logo first thing you need to do is change default DSN Name at the top and Select your bucket and file from it.
Here, in the Compression (Zip/GZip) tab set suitable file format.
Click on Preview Tab, Select Table from Tables Dropdown and select [value] and click Preview.
If you are using DSN method then Click OK to save our DSN Configuration.

Load Amazon S3 Data in Power BI – using ODBC Driver Connection

In the previous section, we configured ODBC DSN for AWS API call and added the ZappySys drivers in the ODBC Driver Administrator with information to connect to AWS S3 Files. We queried S3 File using ZappySys Amazon S3 Driver for CSV Files. Now we will open Power BI and import the information from ODBC DSN we defined earlier.

Open Power BI Desktop and select the option Get data
Get data in power bi
Get Data will allow adding the ZappySys ODBC driver. Go to Other and select ODBC.
Select odbc in Power BI
Select ODBC DSN name from the DSN dropdown
Now its time to import data. Basically, there are two modes to import data. Table Mode and Query Mode. We will Import using Power BI Query Mode

Select your DSN and click Advanced Option to enter custom SQL Query to Import your REST API data. You can use ODBC DSN Data sources Preview tool to generate SQL Query. For example, you can enter a query like below. WITH clause is optional but you can supply many options to override DSN UI settings (e.g. Path, Credentials ).

/*--------- Amazon S3 Driver - Read CSV File Example - Override UI credentials in WITH clause ----------*/  
SELECT * FROM $  
WITH (  
  Src='Northwind-Bucket/Orders*.csv' --use wildcard * for pattern search
  -- ,AccessKey='AKIAxxxx' --not needed if you set in DSN connection
  -- ,SecretKey='a1wxxxxxxxxxxxxxx' --not needed if you set in DSN connection
  -- ,RegionName='us-west-1'
  -- ,ColumnDelimiter='|'
  -- ,HasColumnHeaderRow='False'
)

On the next screen select Windows Authentication to continue (No userid and password).
Select Windows Credentials to connect to zappysys driver
Click OK to import data in Power BI
Now you can create custom Dashboard from imported Dataset.

AWS API Example – Call Amazon S3 API in Power BI

In our previous section we saw how to read Amazon S3 data using native ZappySys S3 Drivers (For CSV , JSON and XML Files), this approach is preferred way to read S3 Files for sure because it gives you UI to browse files, it gives you ability to read from multiple files and many more S3 specific features.

However in some cases you have to call REST API in a manual way using REST API driver rather than S3 Driver. In next section we will see more interesting use case of calling AWS API (e.g. Billing / Cost API).

Now lets look at steps – How to import Amazon S3 data Power BI by calling REST API manually. Steps listed below are almost identical for XML or JSON data format except for selecting Driver Type (i.e. JSON, XML, CSV).

Configure ODBC DSN for REST API ( JSON / XML / CSV )

To consume S3 data in Power BI, first we have to create ODBC DSN using ZappySys Driver (XML or JSON). Perform the following steps.

To do this, we will first open the ODBC Data Source (32 bit):
Open odbc data source
Use the User DSN page and press Add
Add new data source
Add the ZappySys XML Driver if you are accessing XML files from S3 Bucket or calling any AWS API which returns data in XML format. Select JSON driver for JSON data (or API Access in JSON Format)
Create DSN using ZappySys ODBC Driver for XML / SOAP API
It is time to connect with Amazon S3 File and read data. In Data Source (URL or File Path), we will use XML file URL as below. Notice that S3 URL has 3 parts (zs-dump1 is bucket name, s3.us-east-1.amazonaws.com is service endpoint for S3 (some service doesn’t require region) and store_001.xml is our file name. After you specify URL select Connection as per the screenshot.Syntax for URL : BucketName + ServiceURL + FilePath.If you are not sure what can be your service endpoint then check this table.
```
https://zs-dump1.s3.amazonaws.com/store_001.xml
--OR--
https://zs-dump1.s3.amazonaws.com/mysubfolder/store_001.xml
--OR--
https://zs-dump1.s3.us-east-1.amazonaws.com/mysubfolder/store_001.xml
```
Configure Amazon S3 Connection – Read from XML File or JSON File

NOTE: If a filename contains a space, e.g. “store abc.xml“, make sure to replace space with “%20” (“+” will not work). So “store abc.xml” should become “store%20abc.xml“.
Once you specify the URL and Credentials, Next select Filter (This helps to flatten the nested hierarchy of your Data). If you don’t have nested Hierarchy then skip this step.
Select Filter (Flatten the hierarchy to extract data from nested structure)
Now go to Preview Tab and click Preview button in the toolbar. You can remove attributes in the query to use default attributes from Previous Tab. You can always override DSN level setting inside WITH clause of SQL.
Preview data from AWS S3 File (XML Data) – Use ZappySys XML Driver to Connect to AWS API
To learn more about query language click on View Examples
Click OK to save your ODBC DSN settings.

Import AWS S3 File or AWS API data in Power BI dataset

Now let’s look at how to import Amazon S3 data in Power BI using ZappySys XML driver.

In the previous section, we configured ODBC DSN for AWS API call and added the ZappySys drivers in the ODBC Driver Administrator with information to connect to AWS REST API. We queried S3 File using ZappySys XML Driver. Now we will open Power BI and import the information from ZappySys JSON Driver via ODBC connectivity option in Power BI.

Open Power BI Desktop and select the option Get data
Get data in power bi
Get Data will allow adding the ZappySys ODBC driver. Go to Other and select ODBC.
Select odbc in Power BI
Select ODBC DSN name from the DSN dropdown
Now its time to import data. Basically, there are two modes to import data. Table Mode and Query Mode. We will Import using Power BI Query Mode
Select your DSN and click Advanced Option to enter custom SQL Query to Import your REST API data. You can use ODBC DSN Data sources Preview tool to generate SQL Query. For example, you can enter a query like below.
```
SELECT "author", "title",  "genre", "price",  "publish_date", "description", "tag"
FROM $
WITH(
	 Src='https://zs-dump1.s3.amazonaws.com/store_001.xml'
	,Filter='$.store.book[*]'
	,ElementsToTreatAsArray='book'
)
```
Import JSON / REST API data in Power BI using SQL Query Mode
On the next screen select Windows Authentication to continue (No userid and password).
Select Windows Credentials to connect to zappysys driver
Click OK to import data in Power BI
Now you can create custom Dashboard from imported Dataset.

AWS API Example – Import Amazon AWS Cost / Billing data in Power BI

So far you have seen how to connect to S3 File. Now let’s look at another interesting example to call AWS API and display information in Power BI dashboard. For example purpose, we are going to call this AWS Billing and Cost Analysis API. Perform following steps.

Create new DSN using Zappysys JSON Driver
Configure ODBC DSN as below
Enter URL as we discussed in previous section. For example to call Billing API Service for East-1 Region we need to enter following way.
```
https://ce.us-east-1.amazonaws.com
```
Select Connection Type as OAuth and click Configure Link.
Select Provider as AWS API v4 and enter AWS Key and Secret. Click OK to save the connection.
Select Method as POST
Enter HTTP Request Headers as below (Note each Billing Cost API action will have different Header value for X-Amz-Target so refer API documentation
Configure AWS API Connection for Cost / Billing API Data – Use ZappySys JSON Driver

Enter Request Body (Modify your report parameters). As you see in below request we are requesting billing data for 3 services for Jan-1-2018 till Jun-15-2018. If you dont know service name then simply refer to your Bill and copy Service Name in Values array.

{
  "TimePeriod": {
    "Start":"2018-01-01",
    "End": "2018-06-15"
  },
  "Granularity": "MONTHLY",
  "Filter": {      
    "Dimensions": {
      "Key": "SERVICE",
      "Values": [
        "Amazon Simple Storage Service",
        "Amazon DynamoDB",
        "Amazon Redshift",
        "AWS Key Management Service"
      ]
    }
  },
  "GroupBy":[
    {
      "Type":"DIMENSION",
      "Key":"SERVICE"
    },
    {
      "Type":"TAG",
      "Key":"Environment"
    }
  ],
   "Metrics":["BlendedCost", "UnblendedCost", "UsageQuantity"]
}

Select Filter as $.ResultsByTime[*].Groups[*]
Click Test Connection button see it shows OK.
If the connection is OK then switch to Preview Tab to test query.

You can select Table name from the drop-down or type query as below
Simple-Query to Extract AWS Billing / Cost Information

SELECT  
 "p_ResultsByTime_TimePeriod_Start" as BillStartDate,
 "p_ResultsByTime_TimePeriod_End" as BillEndDate,
 json_array_first("Keys") as ServiceName,
 "Metrics_BlendedCost_Amount" as BillAmount,
 "Metrics_UsageQuantity_Amount" as Quantity
FROM [ResultsByTime_x_Groups] 
ORDER BY p_ResultsByTime_TimePeriod_Start

You can also override attributes in your query using below syntax
Custom-Query to Extract AWS Billing / Cost Information (WITH clause)

SELECT 
 "p_ResultsByTime_TimePeriod_Start" as BillStartDate,
 "p_ResultsByTime_TimePeriod_End" as BillEndDate,
 json_array_first("Keys") as ServiceName,
 "Metrics_BlendedCost_Amount" as BillAmount,
 "Metrics_UsageQuantity_Amount" as Quantity
FROM $
ORDER BY p_ResultsByTime_TimePeriod_Start
WITH(
	 Src='https://ce.us-east-1.amazonaws.com'
	,DataConnectionType='OAuth'
	,ScopeSeparator='{space}'
	,ServiceProvider='AmazonAWS'
	--,ClientId='AKIAxxxxxxxxxxxxxxx'
	--,ClientSecret='lPi+XQxxxxxxxxxxxxxxxx'
	,UseCustomApp='True'
	,Filter='$.ResultsByTime[*].Groups[*]'
	,RequestData='{
  "TimePeriod": {
    "Start":"2018-01-01",
    "End": "2018-06-15"
  },
  "Granularity": "MONTHLY",
  "Filter": {      
    "Dimensions": {
      "Key": "SERVICE",
      "Values": [
        "Amazon Simple Storage Service",
        "Amazon DynamoDB",
        "Amazon Redshift",
        "AWS Key Management Service"
      ]
    }
  },
  "GroupBy":[
    {
      "Type":"DIMENSION",
      "Key":"SERVICE"
    },
    {
      "Type":"TAG",
      "Key":"Environment"
    }
  ],
   "Metrics":["BlendedCost", "UnblendedCost", "UsageQuantity"]
}'
	,Header='X-Amz-Target: AWSInsightsIndexService.GetCostAndUsage || Content-Type: application/x-amz-json-1.1'
	,RequestMethod='POST'
)

Click Preview button to see the result.
Query AWS Cost / Billing Data using API in Power BI (Use ZappySys JSON Driver)
Click OK to save Your DSN
Now you can Import Billing / Cost data in Power BI using same steps as previous section (Except use custom Query and DSN we just created for AWS API call)
Import AWS Billing / Cost Data in Power BI (Call AWS REST API)
Repeat steps to import more datasets using different queries

AWS API Example – Import Data From AWS Lambda in Power BI

Now let’s look at another scenario. Assume that someone from other team has written a Lambda Function which exposes certain data in JSON format which you want to display that in your Dashboard. Refer to this Lambda API documentation to learn more about API syntax.

Here is an example query to Call Lambda Function (submit input data in JSON and receive data in JSON format)

SELECT * FROM $
WITH(
	 Src='https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/HelloWorld/invocations'
	,RequestMethod='POST'
	,RequestData='{"id":1}'
	,RequestContentTypeCode='ApplicationJson'
	
	/*** Uncomment to override DSN settings	  
	,DataConnectionType='OAuth'
	,ScopeSeparator='{space}'
	,ServiceProvider='AmazonAWS'
	--,ClientId='AKxxxxxxxxxxx'
	--,ClientSecret='lPi+Xxxxxxxxxxxxxxxxxxxxxxx'
	,UseCustomApp='True'
	***/
)

AWS API Example – Get EC2 VM information in Power BI

Now let’s look at AWS EC2 Service API (Virtual Machine Service). This API is in XML format so you have to use ZappySys XML Driver. Below query calls this EC2 API to get list of all instances.

SELECT * FROM $
WITH(
	 ElementsToTreatAsArray='item'
	,Src='https://ec2.us-east-1.amazonaws.com/'
	,Filter='$.DescribeInstancesResponse.reservationSet.item[*].instancesSet.item[*]'
	,RequestData='Action=DescribeInstances&Version=2016-11-15'
	,RequestMethod='POST'
	/*
	,DataConnectionType='OAuth'
	,ScopeSeparator='{space}'
	,ServiceProvider='AmazonAWS'
	,ClientId='AKIAxxxxxxxxxxxxxxxxxxx'
	,ClientSecret='lPi+XQvxxxxxxxxxxxxxxxxxxxxxxxxx'
	,UseCustomApp='True'
	*/
)

Get EC2 Instance List in Power BI (Use ZappySys XML Driver)

AWS API Example – Get AWS S3 Buckets and Files Count / Size in Power BI

Now let’s look at how to Query S3 File System in Power BI. For example if you have to prepare dashboard to show Total Data Size and File count in a given Bucket then you can write following Query. For more information check Bucket List API Help

Get File Size and Count for a given S3 Bucket

SELECT count(*) as TotalFiles,sum(Size)/1024/1024 as TotalMB  FROM $
WITH(
	 ElementsToTreatAsArray='Contents'
	,Src='https://zs-dump1.s3.amazonaws.com/?list-type=2&prefix=&max-keys=1000'
	,Filter='$.ListBucketResult.Contents[*]'
	,NextUrlAttributeOrExpr='$.ListBucketResult.NextContinuationToken'
	,NextUrlSuffix='&continuation-token=<%nextlink_encoded%>'
	
	/**** Uncomment below to use override DSN values
	,DataConnectionType='OAuth'
	,ScopeSeparator='{space}'
	,ServiceProvider='AmazonAWS'
	,ClientId='AKIAxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
	,ClientSecret='lPi+Xcxxxxxxxxxxxxxxxxxxxxxxxx'
	,UseCustomApp='True'	
	,RequestMethod='GET'
	***/
)

To list buckets you can use following query

List S3 Buckets

SELECT *  FROM $
WITH(	 
	 Src='https://s3.amazonaws.com'
	,Filter='$.ListAllMyBucketsResult.Buckets.Bucket[*]'
	,ElementsToTreatAsArray='Buckets'

	/**** Uncomment below to use override DSN values
	,DataConnectionType='OAuth'
	,ScopeSeparator='{space}'
	,ServiceProvider='AmazonAWS'
	,ClientId='AKIAxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
	,ClientSecret='lPi+Xcxxxxxxxxxxxxxxxxxxxxxxxx'
	,UseCustomApp='True'	
	,RequestMethod='GET'
	***/
)

Debugging / Crafting AWS API Call Request

Check below article to learn various techniques to debug and craft AWS API Request Body / Headers and URL.

How to call Amazon AWS API using SSIS (EC2, Lambda, API Gateway, SQS)

Understanding AWS REST API Pagination in Power BI

Even we set up ODBC Data Source to get the data, it may not be enough. Usually, if you are getting a huge data set from API provider, it won't give it to you in one HTTP response. Instead, it gives back only a subset of data and provides a mechanism for data pagination. The good news is that ZappySys ODBC Driver includes many options to cover virtually any pagination method.

Below you will find a few examples of API pagination. If you need something more sophisticated check the below link (the article was written for SSIS PowerPack but UI options and concepts apply to ODBC Driver too):

https://zappysys.com/blog/ssis-rest-api-looping-until-no-more-pages-found/

Paginate by Response Attribute

This example shows how to paginate API calls where you need to paginate until the last page detected. In this example, next page is indicated by some attribute called nextlink (found in response). If this attribute is missing or null then it stops fetching the next page.

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

Paginate by URL Parameter (Loop until certain StatusCode)

This example shows how to paginate API calls where you need to pass page number via URL. The driver keeps incrementing page number and calls next URL until the last page detected (401 error). There are few ways to indicate the last page (e.g. By status code, By row count, By response size). If you don't specify end detection then it will use the default (i.e. No records found).

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

Paginate by URL Path (Loop until no record)

This example shows how to paginate API calls where you need to pass page number via URL Path. The driver keeps incrementing page number and calls next URL until the last page is detected. There are few ways to indicate the last page (e.g. By status code, By row count, By response size). If you don't specify end detection then it will use the default (i.e. No records found).

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

Paginate by Header Link (RFC 5988)

API like GitHub / Wordpress use Next link in Headers (RFC 5988)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

Handling AWS Web API Error in Power BI

Sometimes errors occur... they just do and there is nothing you can do! Or can you? Actually, in ODBC PowerPack you can handle them in two ways.

METHOD 1 - Using Error Handling Options

When to use?

You may want to use them when your source is a resource located on the Internet; e.g. a file on a website, a file on an FTP server or just a plain API HTTP response. By default, when a remote server returns an error, data retrieval is stopped, an error is raised and no data is given back to you. This might not be always desirable.

Scenario 1

Imagine a scenario, that there is a web server which each day at 12 AM releases a new JSON file with that day's date as filename, e.g. http://www.some-server.com/data/2018-06-20.json. And, of course, you want to download it and use it daily in your Power BI report. But you have a problem: Power BI report data sources are refreshed each hour and you may get HTTP 404 status code (no file was found) when a file is not released yet. Which consequentially means other data sources won't be updated as well and you will see old and cached data on the report. That's where you could use Continue on any error or Continue when Url is invalid or missing (404 Errors) to avoid an error being raised and let other data sources to be updated.

Scenario 2

Another scenario is when you expect a web server to raise some kind of HTTP error when accessing a URL. You don't want ODBC Data Source to raise an error but instead, you want to get response data. That's where you can use Continue on any error or alike together with Get response data on error to continue on an error and get the data:

METHOD 2 - Using Connection [Retry Settings]

Another scenario you may run into is a buggy web server. You ask it to give you some file or data and it, like a snotty kid, just doesn't give it to you! You have to ask twice or thrice before it does its job. If that's the case, you have to retry HTTP requests using Connection:

Other Considerations for Calling AWS API in Power BI

There are few settings you can coder while calling Web API

API Limit / Throttling

While calling public API or other external web services one important aspect you have to check, how many requests are allowed by your API. Especially when you use API pagination options to pull many records you have to slow down based on API limits. For example, your API may allow you only 5 requests per second. Use Throttling Tab on Driver UI to set delay after each request.

2D Array Transformation

If you are using JSON or XML API Driver then possible you may have to transform your data using 2D array transformation feature. Check this link for more information.

Conclusion

In this article, we show how to connect to AWS API and S3 File data in Power BI. We configured ODBC DSN for AWS S3 / REST API connection and finally imported data in Power BI. We used ZappySys driver for JSON that can be used to extract data from any REST API or a JSON file. Download ODBC PowerPack to try yourself see how easy it is to consume XML / JSON / REST API and AWS data in Power BI and Say goodbye to ETL or Coding. If you need any help with your API integration feel free to contact zappysys support you will be amazed for sure how quickly your issue is resolved.

The post Read Amazon S3 data in Power BI or Call AWS REST API (JSON / XML) appeared first on ZappySys Blog.

How to call Amazon AWS API using SSIS (EC2, Lambda, API Gateway, SQS)

ZappySys — Fri, 03 Nov 2017 15:25:28 +0000

Introduction

In this blog post you will learn how to call Amazon AWS API using SSIS (virtually any API) without a single line of code (No more JAVA, C#, Ruby, Python). Yes you heard it right 🙂 . If you are a SSIS / ETL Developer or even coder everyone loves drag & drop interface. SSIS has many advantages over other approaches such as Programming SDKs, Command Lines. Main advantage is ease of use, security and long term maintenance without learning expensive coding approach.

In this post we will use following components to show various possibilities to implement Amazon AWS API integration scenarios inside your ETL workflows.

	JSON / REST API Source (REST API, JSON File or OData Service): Use this dataflow component when you have to fetch data from REST API webservice like a table. This component allows you to extract JSON data from webservice and de-normalize nested structure so you can save to Relational database such as SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local JSON files or direct JSON string (Wildcard pattern supported too e.g. c:\data\file*.json).
	Web API Destination (POST data to API URL) : Use this dataflow component when you have to call API inside Dataflow (see POST to URL). Possible Use case: Call Lambda function with various parameters for each input record found in database.
	REST API Task : Use this task when you don’t want to pull REST API data in tabular format but want to call rest API for POST data to server, DELETE data from server or things like download HTML page, extract Authentication tokens etc where you not necessarily dealing data in tabular format. This task also allows you many other options such as saving RAW response into variable or file.
	XML Source (SOAP, File, REST) : Use this dataflow component when you have to fetch data from XML or SOAP webservice and consume data like a table. This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local XML files or direct XML string.

Prerequisites

Before we do hello world demo for calling Amazon AWS API, you will need to make sure following prerequisites are met.

SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
Access to valid AWS credentials (Access Key, Secret Key for your IAM User). Click here to learn more about IAM users and Access Key/Secret Key
Make sure SSIS PowerPack is installed. Click here to download.

Step-By-Step Example-1 (Call AWS API)

Now lets call some simple GET API call using SSIS REST API Task.

Install SSIS PowerPack (Skip this step if you already installed SSIS PowerPack.
Open Visual Studio and create new Integration Services Project
Open SSIS Package and check your SSIS Toolbox you will see many tasks/components starting with ZS
From Control flow SSIS Toolbox Drag & drop ZS REST API Task.
Double click the REST API task to configure it.
On REST API Task change URL Access mode drop down to URL from Connection
Now in the connection dropdown click New ZS-OAUTH connection and configure connection as below
1. Once you see OAuth connection dialog box change Provider Type from Custom to Amazon AWS API (v4)
2. In the ClientId enter your AWS Access Key
3. In the ClientSecret enter your AWS Secret Key
4. Click OK to save connection
  Configure SSIS OAuth Connection – Use Amazon AWS API Provider, Enter Access Key, Secret Key
On REST API Task change few more settings as below
1. Enter API URL you like to call (In our case we will use S3 API (Simple Storage Service). We assume you have ListBucket permission to make this call. If you dont have such permission try to get Full path of File (choose small file) . You have to tweak API url to adjust Service Type, Region,Bucket, Path
```
https://s3.us-east-1.amazonaws.com
-- OR -- Use below (list files) if you have single bucket permission---
https://s3.us-east-1.amazonaws.com/YOUR-BUCKET
```
2. Click Test Request. If you have valid Permission and setup looks ok then you will see Response window like below. Using this technique you can call any API to execute AWS operations (E.g. start EC2 VM, Create SQS Queue, Call Lambda Function, Drop or Update resource)
  Calling Amazon AWS API Using SSIS REST API Task Example. (Call any AWS API without SDK. such as EC2 API, SQS API, Lambda API, Redshift API, DynamoDB API)

Step-By-Step Example-2 (Loading data from AWS API to SQL Server)

Now lets do more interesting scenario. We will call AWS S3 API to get S3 File list from Bucket. After extract we will save that list to SQL Server Table. Since Amazon S3 API is XML based API we will use ZappySys SSIS XML Source. For JSON based API use JSON Source instead. XML Source / JSON Source both can parse API response into Rows and Columns so you can easily store it into SQL Server. Now lets see how to do this.

From Control flow SSIS Toolbox Drag & drop Data Flow Task
Double click Data Flow Task. From SSIS Toolbox Drag & drop ZS SSIS XML Source.
Double click the XML Source to configure it.
In the URL text box enter API URL like below to list S3 Files for specified bucket (Change YOUR-BUCKET to your own name)
```
https://s3.us-east-1.amazonaws.com/YOUR-BUCKET
```
Now check Use Credentials and Select same Amazon API connection we created in previous example.
Click on Select Filter (This step allows us to flatten the XML hierarchy. Select the node which is Array icon like below. If prompted to treat selected node as array click Yes.
Configure SSIS XML Source – Get Amazon S3 File List from specific bucket (AWS API)
Click Preview to see your data. Click OK to save.
Preview Data in XML Source – Amazon S3 File List (Calling AWS API)
Now attach your XML source to target like OLEDB Destination to load data to SQL Server or other Target (e.g. Oracle, MySQL)
Execute SSIS Package to load data from Amazon AWS API to SQL Server.
SSIS Package – Read Amazon S3 File List and load into SQL Server Table (Call AWS API Example)

File Upload Example – Low level API – Call PUT request

There will be a time when you want to take total control of your AWS API calls. One example is if you wish to Upload / Write data to S3 then components like ZappySys Amazon S3 CSV Destination or Amazon Storage Task might need additional permission such as HeadObject . If you have only write permission on bucket then this will fail to execute Task. In such case you can use REST API Task like below way.

Basically we exported data from Relation Database to CSV File using Export CSV File Task and then we uploaded file content using REST API Task.

Upload Text File to S3 (i.e. JSON, CSV, XML …)

Your URL Format can be like this. Region codes can be found here

https://YOUR-BUCKET.s3.YOUR-REGION.amazonaws.com/YOUR-FILE
https://YOUR-BUCKET.s3.YOUR-REGION.amazonaws.com/SOME-FOLDER/SOME-SUB-FOLDER/YOUR-FILE

Upload file to Amazon S3 – Call low level API (PUT request)

Upload Binary File to S3 (i.e. Zip, mp3, gzip, png, jpeg…)

Above method only works for Text Files. If you have Binary files (e.g. Zip file, mp3, png, jpeg) then you can use below workaround.

Check Is Multi Part / File Upload Option next to the Body editor (Read this Post for more info)
In the Body enter file path like below
```
@c:\folder\some-file.xyz
```
That’s it, now your binary file can be uploaded to S3 same way as Text file we uploaded in earlier section.

NOTE: Binary file option only works in the latest Build Uploaded after 5/21/2020

Debugging AWS API Command Line Requests using Fiddler (Web Proxy)

Before we see more examples of calling AWS API lets first learn how to capture Request data using aws command line (CLI). We will use Fiddler to capture AWS API Requests.

Download and Install Fiddler (Free Tool)
Install aws command line

Open command prompt and type aws configure command to set credentials. For more info see configure aws credentials .. see below example.

c:> aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

Once you set credentials launch Fiddler
On the Fiddler Tools menu > Click Fiddler Option > HTTPS > Check Decrypt HTTPS Traffic
Once you do that you may be asked to Trust Fiddler Certificate click OK
Close and Open Fiddler to apply setting
Now any command you type in aws command line will show up in fiddler. This command we can use in SSIS REST API Task or XML Source to call virtually Any API AWS supports.
For example to call some lambda function (supply input json data from file) use below command and watch Fiddler Trace. Notice we added –no-verify-ssl option so we can see requests in custom web proxy like fiddler (This option will trust Fiddler certificate).
```
c:\>aws lambda invoke --function-name HelloWorld c:\temp\outputfile.txt --no-verify-ssl --payload file://c://test/customer.json
```
Capture AWS Commandline (CLI) request using Fiddler
Once you have this information you can use it inside ZappySys Components which supports API calls (e.g. REST API Task, JSON Source, XML Source). For this example we will use REST API Task to call same Lambda Function. Things to change to call any API is URL, Method, ContentType, Body. If its GET call then you wont have Body.
Calling Amazon AWS Lambda Function using SSIS REST API Task
That’s it.. You can now take this same concept and call virtually API AWS API right insight SSIS without any SDK or command line tools.

Call API Gateway Endpoint (Default URL)

If you wish to call API hosted on Amazon API Gateway Service then enter direct URL. You have to use OAuth Connection (AWS v4 Provider) as previous section.

Typical URL may look like as below if you calling API gateway Endpoint.

https://c5hhigf5mh.execute-api.us-east-1.amazonaws.com/prod/pets

Call AWS API with Region and Custom Service Name

There will be a time when you will have to call AWS API URL and it doesnt indicate Region / Service but you have to supply part of signature for Authentication. In this case just enter AWS Service and Region along with ClientID, Secret Fields (Custom Service and Region attributes were introduced in v3.0 or higher so if you are not seeing on OAuth Connection UI then you probably running older version).

Call Amazon Athena API

Check this article for detailed instructions

Call AWS EC2 API

Here is an example of calling EC2 API. This is example of listing EC2 instances.

Request:

POST https://ec2.us-east-1.amazonaws.com/
Content-Type: application/x-www-form-urlencoded; charset=utf-8

Action=DescribeInstances&Version=2016-11-15

Call AWS Lambda API

Here is some example of calling Lambda function

Request:

POST https://lambda.YOUR-REGION.amazonaws.com/2015-03-31/functions/YOUR-FUNCTION/invocations

{you-json-input-goes-in-body}

Call AWS ElasticSearch

If you are using AWS Hosted Managed ElasticSearch then also you can use OAuth AWS v4 Provider as below. Assuming you have configured correct Policy to allow your IAM User Account / IP address.

Call AWS Hosted ElasticSearch REST API in SSIS (V4 Request Signing)

Call AWS API using Native Task/Components

ZappySys provides many High quality Tasks / Components for AWS integration. See below list.

Amazon AWS Cloud Integration
Amazon S3 Task	Amazon Redshift Data Transfer Task	Amazon Redshift ExecuteSql Task
Amazon Redshift Cluster Management Task	Amazon DynamoDB Source	Amazon DynamoDB Destination
Amazon Redshift Source	Amazon SQS Queue Source	Amazon SQS Queue Destination

Conclusion

Amazon AWS Cloud integration from SSIS packages becoming more and more common scenario. ZappySys provides easy to use no coding connectors to achieve many time consuming scenarios. Having clean drag and drop approach is not only faster but more secure using inbuilt SSIS framework. Try SSIS PowerPack to explore many other scenarios not discussed in this article.

The post How to call Amazon AWS API using SSIS (EC2, Lambda, API Gateway, SQS) appeared first on ZappySys Blog.

SSIS – Copy Amazon S3 files from AWS to Azure

ZappySys — Wed, 19 Apr 2017 21:40:45 +0000

Introduction

Azure and AWS both are most popular Cloud Platforms. In this blog post we will learn how to copy or move Amazon S3 files to Azure Blob Storage without any coding or scripting (AWS to Azure File Copy / Migration Scenario). To achieve this objective we will use following Drag and Drop SSIS Tasks (i.e. Microsoft SQL Server Integration Services – ETL Platform for SQL Server). Following components are highly optimized for parallel copy/multi threading with secure connection (client side/serverside encryption).

SSIS Azure Blob Storage Task

SSIS Amazon Storage Task (AWS S3)

Concept : Fast Server Side Copy in Azure (Copy files into Azure Blob Storage)

Azure provides unique feature called server side file copy. Using this feature you can load or copy files into Azure Blob Storage without landing data to your local machine. As of now (March 2017) this type of feature is still missing in other Cloud Platform such as Amazon AWS and Google Cloud Platform – GCP.

Using server side copy feature in Azure… you can achieve following scenarios in SSIS without coding

Move / Copy files from Amazon S3 to Azure Blob Storage
Move / Copy files from Google Cloud Platform (GCP) to Azure Blob Storage
Copy files from any public URL to Azure Blob Storage (Assuming URL doesn’t require credentials or its intranet URL)

In first two scenarios you need to have Source File URL using Authentication information inside URL (This is called Pre-Signed URL e.g. http://mycloud.com/myfile.zip?SIGNATURE-GOES-HERE ). All major Cloud Providers Support Pre-Signed URL so you can securely share files with other without sharing your actual Credentials. You can configure Pre-Signed URL to expires in certain time frame (Check your Cloud API documentation). If your file doesn’t need credentials to access then you don’t have to worry about Pre-Signed URL. Your source can be file URL (E.g. https://mysite.com/downloads/fil1.zip)

Pre-Signed URL for Google Cloud (GCP)
Here is more information how to get Pre-Signed URL for Google Cloud Platform (GCP) – Use gsutil command line
Pre-Signed URL for Amazon AWS (S3 file)
To create pre-signed URL for AWS S3 files you can use SSIS Amazon Storage Task and use Get Pre-Signed URL Action (Described in the below section). This action supports creating pre-signed URLs for multiple files using wildcard (e.g /mybkt/*.zip) or you can get single pre-signed URL. If you use pattern search then you will get DataTable back … which can be loop through using ForEach Loop task (Loop ADO Recordset option) . If you do not use wild card then only one URL returned in string format.

Prerequisites

Before we look at next section make sure following prerequisites are met

You have obtained Amazon S3 Access Key and Secret Key to access desired files
You have obtained Azure Storage Account Name and Account Key to access desired Blob Container. If you don’t have access to Azure then you can download Azure Storage Emulator for testing Purpose. Check this.
You have basic knowledge of SSIS. If you dont then search for SSIS tutorial and there are many blogs / tutorials to get started 🙂

Step-By-Step – Create SSIS Package – Copy Amazon S3 Files to Azure Blob Storage

Now lets look at how to copy Amazon S3 files to Azure in few clicks. This approach doesn’t bring any data to your local system so its purely server to server copy and that is why its very fast and secure.

First Download and Install SSIS PowerPack
Once you install SSIS PowerPack. Create a new Sample SSIS Project and open Package
From SSIS Toolbox Drag ZS Amazon Storage Task and drop on Control Flow surface. Rename it to Get S3 File Urls
Double click S3 task to edit. From Action dropdown select (Get Amazon S3 Files Pre-Signed Url Option)
Click New next to the Connection Dropdown to create Amazon Storage Connection. Enter your credentials and bucket region and Click Test. If you are not sure about your bucket region then check this article.
SSIS Amazon Storage Task – Create Pre-Signed URLs for multiple files stored in S3
Now click on Advanced Tab and check Use region specific endpoint option
SSIS Amazon S3 Connection – Region specific endpoint option
Once Test connection is green you can click OK to save connection.
Once you back to Amazon Storage Task UI… Click on browse next to S3 file path. Her you can select one file or enter pattern to create URLs for multiple files.
To save pre-signed URL(s) select Variable. If variable is not there then create new one… If you enter pattern in Source path then Variable must be Object datatype. For multiple URL it will return ADO Recordset (You can use with ForEach Loop Task). If you didn’t enter pattern in source path then variable can be string datatype.
SSIS Amazon Storage Task – Save Pre-Signed URL to Variable (Save Multiple URL as Recordset )
Now drag and drop ForEach Loop Container on the surface and configure like below
SSIS ForEach Loop Container Task – Loop through Amazon S3 file URL

SSIS ForEach Loop Task – Variable Mappings
Once Foreach Loop is configured you can drag ZS Azure Storage Task inside ForEach Loop Container
Double click Azure storage task and select [Copy from external file] option from Action dropdown.
In the source Path/URL we will enter variable name which holds current URL for ForEach Loop iteration. Easy way is click blue variable icon and select <
>. Your placeholder may look like below.
```
{{User::varCurrentS3Url}}
```
For target path … Click New Connection to create and configure new Azure Storage Connection like below. Enter Azure storage credentials and click test. If you don’t have real Azure credentials then you can use Azure Storage Emulator (Check this). Once connection is Configured click Test connection and click OK to save connection.
Azure Storage Task – Configure Blob Storage connection- Serverside Copy S3 files – AWS to Azure
Once everything is configure you can save package and execute. This will copy S3 files to Azure Blob storage.

Here is execution log

SSIS package "C:\SSIS\Amazon-To-Azure-Copy.dtsx" starting.
Information: 0x0 at Get Signed URL for S3 Files, Get Signed URL for S3 Files: You are running TRIAL version. It will expire in 29 day(s)
Information: 0x0 at Get Signed URL for S3 Files, Get Signed URL for S3 Files: Reading [PRESIGNED-URL] property for zs-eu-west-2-london-bkt2/*.*
Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copy started: Source=https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile1.csv, Target=test1/
Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copying https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile1.csv ...
Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copy started: Source=https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile3.txt, Target=test1/
Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copying https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile3.txt ...
SSIS package "C:\SSIS\Amazon-To-Azure-Copy.dtsx" finished: Success.

SSIS Package Execution – Copy Multiple S3 Files from
Amazon to Azure

How to move Amazon S3 files to Azure

In above example you saw how to copy files from Amazon S3 to Azure Blob Storage. But what if you want to Move… so once file is copied you want to delete from source. In that case its simple. Add one more Amazon S3 Task at the end so if all previous steps successful you can Issue delete files command using ZS Amazon Storage Task.

SSIS Delete Amazon S3 Files after Successful S3 to Azure Copy operation (This will mimic Move)

Download Sample SSIS Package

Click here to download sample SSIS file for SQL Server 2012, 2014, 2016

Conclusion

If you want to bring data from Amazon S3 to Azure in fastest way then technique described in this article can help you to cut down your data transfer time by several times. SSIS PowerPack comes with 45+ components and tasks which can give you drag and drop interface for your Cloud connectivity projects. It also comes with many connectors to help with JSON, XML, REST API related integration. Try SSIS PowerPack for FREE without any limitation and find out what else you can do with it.

The post SSIS – Copy Amazon S3 files from AWS to Azure appeared first on ZappySys Blog.