sqs Archives | ZappySys Blog

How to Read / Write / Delete Amazon SQS Queue data in SSIS

ZappySys — Mon, 19 Aug 2019 12:17:48 +0000

Introduction

ZappySys provides high-performance drag and drop connectors for Amazon AWS Integration. In this post, we will see How to Read / Write / Delete Amazon SQS Queue data in SSIS using ZS Amazon SQS components. ZappySys developed many AWS related components but in this article, we will look at 2 components for Amazon SQS Queue integration Scenarios (Read and Write). We will discuss how to use SSIS Amazon SQS Source and SSIS Amazon SQS Destination. Amazon SQS Source can be used to read messages from Amazon AWS SQS (Simple Queue Storage) and Amazon SQS Destination can be used to write/delete data from Amazon SQS (Simple Queue Storage) using SSIS.

Prerequisite

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

First, you will need to have SSIS installed.
Secondly, make sure to have SSDT.
Thirdly, you have obtained Amazon SQS Credentials.
Finally, do not forget to install ZappySys SSIS PowerPack.

Component Mention in this article

	Amazon SQS Source Connector
	Amazon SQS Destination Connector

What is Amazon SQS (Simple Queue Service)

Before we talk How to Read / Write / Delete Amazon SQS Queue data in SSIS lets talk What is Amazon SQS (Simple Queue Service) or sometimes referred to as AWS SQS. Amazon SQS offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues and costs allocation tags. It provides a generic web services API and it can be accessed by any programming language that the AWS SDK supports.

SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

Getting Started

In order to start, we will show several examples. ZappySys includes an Amazon SQS Source and Amazon SQS Destination, that will help you How to Read / Write / Delete Amazon SQS Queue data in SSIS. You can connect to your Amazon SQS account by entering credentials.

Amazon SQS Source can be used to read messages from Amazon AWS SQS (Simple Queue Storage) using SSIS. Read Amazon SQS Messages (AWS SQS) in bulk. Amazon SQS Destination can be used to write/delete data from Amazon SQS (Simple Queue Storage) using SSIS. Write messages to Amazon SQS (AWS SQS) in bulk. Support for Visibility Timeout (Hide messages from others for certain time after you read), data preview and max rows, reading message attributes in separate output columns and adjusting UTC DateTime in local time.

Write Data using Amazon SQS Destination in SSIS

Overview

In this tutorial, you will learn how to write data in Amazon SQS Queue storage. You can use ZS DummyData Source for practice it or select suitable ZS Source Adapters. In this case its from ZS DummyData Source. So let’s start with an example, In this section, you will learn how to write messages into Amazon SQS Queue. For this example, we are assuming that you have already SQS Queue Created.

Create Amazon SQS Connection

Firstly, In order to connect to Amazon SQS from SSIS, you will need Access Key and Secret Key (Login to AWS portal and Create Access/Secret Key with SQS service read access). If you are not sure about this ask your SysAdmin or responsible person to provide that to you. Your key will look something like this: (This is just an example key which may differ in your case).
```
AccessKey: AKIAIOSFODNN7EXAMPLE
SecretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
```
Click here to read more about how to get your AccessKey and Secret Key.
Secondly, Once you have Account Key (it’s like UserID) and Secret Key (its like password) you may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
Right-click in Connection Manager Panel and Select [New Connection] menu item.
SSIS Create AWS Storage Connection
Now Select ZS-AWS-STORAGE connection type.
Select ZS-AWS-STORAGE Connection
Let’s configure the Amazon SQS connection. Set Storage Service to SQS and Credential Type to Key-based access control. In the General tab enter your Amazon SQS Credentials and Set Default region as per you have.
SSIS AWS Storage Connection – Configure
In the last, click on Test Connection and OK button to Save connection configure setting UI.

Write Data into Amazon SQS

Once you have created the Amazon SQS Connection successfully, let’s drag and Drop Data Flow Task from SSIS Toolbox on the control flow designer surface.
Drag and Drop SSIS Data Flow Task from SSIS Toolbox
Double click on the Data Flow Task to see Data Flow Designer surface.
Here, from the SSIS Toolbox drag and drop ZS DummyData Source on the data flow designer surface.
SSIS ZS Dummy Data Source – Drag and Drop
Double click on Dummy Data Source to configure it.
SSIS ZS Dummy Data Source – Configure
Click on OK button to save configure setting UI.
From the SSIS toolbox drag and drop Amazon SQS Destination on the Data flow designer surface and single click on the Dummy Data Source, Once you see the blue arrow from the source. connect it to Amazon SQS Destination.
SSIS ZS Amazon SQS Destination
Now double click on Amazon SQS Destination to configure it.
In the [Connection Manager] tab select SQS connection manager we have created before it.
SSIS Amazon SQS Destination – Configure Connection Manager
Click on [Component Properties] tab and Select Target Queue from the dropdown list.
SSIS ZS Amazon SQS Destination – Configure Component Properties
Click on [Mappings] tab and verify mapping. Map document column from upstream to the MessageText target column.
SSIS ZS Amazon SQS Destination – Column Mappings
In the last, click on the OK button to save configure setting UI.
After all, Execute the Package and verify target data by going to your AWS console under SQS service section.
Run or Execute SSIS Package

Verify Amazon SQS Destination

Read Data using Amazon SQS Source in SSIS

Overview

In this tutorial, you will learn how to read data from Amazon SQS storage in SSIS using ZS SSIS Amazon SQS Source Adapter. In this SSIS we will read data. So let’s begin it.

Read Data from Amazon SQS

Firstly, create a new SSIS package. In BIDS/SSDT create a new SSIS package.
From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
SSIS Data Flow Task – Drag and Drop
Double click on the DataFlow task to see DataFlow designer surface.
From the SSIS toolbox drag and drop ZS Amazon SQS Source on the dataflow designer surface.
Double click Amazon SQS Source to configure it.
Select Amazon SQS Connection we have created before and select queue from the dropdown list.
SSIS Amazon SQS Source – Configure
Click on Preview button to see Data Preview and Click on OK button to save settings.
From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface or you can use any of our suitable ZS Destination Adapter.
SSIS ZS Trash Destination – Drag and Drop
Now single click on the Amazon SQS Source once you see the blue arrow from the source … connect it to Trash Destination.
Double click on ZS Trash Destination to Configure it.
SSIS ZS Trash Destination – Configure
Click on OK button to save Trash Destination configure setting UI.
After all, right-click on the path and Add Data Viewer and Execute the package and verify source data in the data viewer.
Run or Execute The SSIS Package
After messages read from SQS Queue, the same message cannot be read again until VisibilityTimeout. By default Queue, visibility timeout is used (-1 means use default) but you can override it on the source UI.

Read all messages from SQS FIFO Queue

If you are using Normal SQS Queue then by setting Max Rows=0 will read all messages from your queue. However reading all messages from FIFO Queue is not that simple. Amazon SQS wont return more messages until you delete previously read Messages from same Message Group (identified by MessageGroupId). Due to this constraint you have to try below technique of Loop to read 10 messages until all messages Read > Processed > Deleted.

Here is note from Amazon Documentation about maximum 10 messages limitation on FIFO Queue.

When you receive a message with a message group ID, no more messages for the same message group ID are returned unless you delete the message or it becomes visible.

Community Reference:

https://stackoverflow.com/questions/51661834/aws-sqs-fifo-how-to-get-more-than-10-messages-at-a-time

If you want to add fault tolerance then you can set Max Retry for NULL Receive like below (e.g. Setting MaxRetry=3 will try max 3 times if no message is read …. It will try every 2 seconds for 3 times before giving up to read more rows).

Read all messages from Amazon SQS FIFO Queue

Read , Process and Delete Amazon SQS FIFO Queue Message in SSIS Data flow

Delete Message from Queue using Amazon SQS Destination in SSIS

Overview

In this tutorial, you will learn how to Delete Messages from the Queue using the ZS SSIS Amazon SQS Destination Adapter. So let’s begin it.

Delete Message from Queue

From the SSIS toolbox drag and drop Amazon SQS Destination on the Data flow designer surface and connect it to Amazon SQS Destination.
Now double click on Amazon SQS Destination to configure it.
In the [Connection Manager] tab select SQS connection manager we have created before it.
Select Amazon SQS Connection
Click on [Component Properties] tab, Set Operation to Delete and Select Target Queue from the dropdown list.
Select Delete Operation and Queue
Click on [Column Mappings] tab and verify mapping. Map document column from upstream to the MessageText target column.
Do Columns Mapping
In the last, click on the OK button to save configure setting UI.
After all, Execute the Package and verify target data by going to your AWS console under SQS service section.

Purge All Messages from SQS Queue (Clear Queue)

There will be a time you like to clear entire queue (i.e. Delete All Messages from Queue). Amazon recently added PurgeQueue API feature so now its possible without doing Delete message technique mentioned in previous section. ZappySys support calling any AWS API using REST API Task along with AWS Provider in OAuth Connection. You can use same technique to call PurgeQueue API like below.

Open SSIS Package and Drag ZS REST API Task from Control Flow Toolbox
Click on URL mode dropdown and change to URL from Connection.
From COnnection Dropdown select NEW ZS-OAuth connection
Configure OAuth connection to use Provider=Amazon AWS v4
1. Enter Access Key and Secret Key and click OK to save connection
Enter below URL on REST API Task (Replace Queue ID and name with your own values) – Refer to your AWS Console / SQS Queue Properties to get this URL
```
https://sqs.ENTER-REGION-HERE.amazonaws.com/ENTER-QUEUE-ID-HERE/ENTER-QUEUE-NAME-HERE/?Action=PurgeQueue
```
Here is an example of Sample Queue from ZappySys Demo
```
https://sqs.us-east-1.amazonaws.com/718883169616/zappysystest.fifo/?Action=PurgeQueue
```
Click ok to Save REST API Task

How to Purge Amazon SQS Queue in SSIS (Clear Queue / Delete All messages using PurgeQueue REST API)

Conclusion

So, In this blog, we learn How to Read / Write / Delete Amazon SQS Queue data in SSIS. We used Amazon SQS Destination for write data and Amazon SQS Source to read data. Additionally, you can download SSIS PowerPack here to try many other scenarios that are not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

Amazon SQS Source
Amazon SQS Destination
Documents of Amazon SQS
Help File:
- Amazon SQS Source
- Amazon SQS Destination

The post How to Read / Write / Delete Amazon SQS Queue data in SSIS appeared first on ZappySys Blog.

How to call Amazon AWS API using SSIS (EC2, Lambda, API Gateway, SQS)

ZappySys — Fri, 03 Nov 2017 15:25:28 +0000

Introduction

In this blog post you will learn how to call Amazon AWS API using SSIS (virtually any API) without a single line of code (No more JAVA, C#, Ruby, Python). Yes you heard it right 🙂 . If you are a SSIS / ETL Developer or even coder everyone loves drag & drop interface. SSIS has many advantages over other approaches such as Programming SDKs, Command Lines. Main advantage is ease of use, security and long term maintenance without learning expensive coding approach.

In this post we will use following components to show various possibilities to implement Amazon AWS API integration scenarios inside your ETL workflows.

	JSON / REST API Source (REST API, JSON File or OData Service): Use this dataflow component when you have to fetch data from REST API webservice like a table. This component allows you to extract JSON data from webservice and de-normalize nested structure so you can save to Relational database such as SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local JSON files or direct JSON string (Wildcard pattern supported too e.g. c:\data\file*.json).
	Web API Destination (POST data to API URL) : Use this dataflow component when you have to call API inside Dataflow (see POST to URL). Possible Use case: Call Lambda function with various parameters for each input record found in database.
	REST API Task : Use this task when you don’t want to pull REST API data in tabular format but want to call rest API for POST data to server, DELETE data from server or things like download HTML page, extract Authentication tokens etc where you not necessarily dealing data in tabular format. This task also allows you many other options such as saving RAW response into variable or file.
	XML Source (SOAP, File, REST) : Use this dataflow component when you have to fetch data from XML or SOAP webservice and consume data like a table. This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local XML files or direct XML string.

Prerequisites

Before we do hello world demo for calling Amazon AWS API, you will need to make sure following prerequisites are met.

SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
Access to valid AWS credentials (Access Key, Secret Key for your IAM User). Click here to learn more about IAM users and Access Key/Secret Key
Make sure SSIS PowerPack is installed. Click here to download.

Step-By-Step Example-1 (Call AWS API)

Now lets call some simple GET API call using SSIS REST API Task.

Install SSIS PowerPack (Skip this step if you already installed SSIS PowerPack.
Open Visual Studio and create new Integration Services Project
Open SSIS Package and check your SSIS Toolbox you will see many tasks/components starting with ZS
From Control flow SSIS Toolbox Drag & drop ZS REST API Task.
Double click the REST API task to configure it.
On REST API Task change URL Access mode drop down to URL from Connection
Now in the connection dropdown click New ZS-OAUTH connection and configure connection as below
1. Once you see OAuth connection dialog box change Provider Type from Custom to Amazon AWS API (v4)
2. In the ClientId enter your AWS Access Key
3. In the ClientSecret enter your AWS Secret Key
4. Click OK to save connection
  Configure SSIS OAuth Connection – Use Amazon AWS API Provider, Enter Access Key, Secret Key
On REST API Task change few more settings as below
1. Enter API URL you like to call (In our case we will use S3 API (Simple Storage Service). We assume you have ListBucket permission to make this call. If you dont have such permission try to get Full path of File (choose small file) . You have to tweak API url to adjust Service Type, Region,Bucket, Path
```
https://s3.us-east-1.amazonaws.com
-- OR -- Use below (list files) if you have single bucket permission---
https://s3.us-east-1.amazonaws.com/YOUR-BUCKET
```
2. Click Test Request. If you have valid Permission and setup looks ok then you will see Response window like below. Using this technique you can call any API to execute AWS operations (E.g. start EC2 VM, Create SQS Queue, Call Lambda Function, Drop or Update resource)
  Calling Amazon AWS API Using SSIS REST API Task Example. (Call any AWS API without SDK. such as EC2 API, SQS API, Lambda API, Redshift API, DynamoDB API)

Step-By-Step Example-2 (Loading data from AWS API to SQL Server)

Now lets do more interesting scenario. We will call AWS S3 API to get S3 File list from Bucket. After extract we will save that list to SQL Server Table. Since Amazon S3 API is XML based API we will use ZappySys SSIS XML Source. For JSON based API use JSON Source instead. XML Source / JSON Source both can parse API response into Rows and Columns so you can easily store it into SQL Server. Now lets see how to do this.

From Control flow SSIS Toolbox Drag & drop Data Flow Task
Double click Data Flow Task. From SSIS Toolbox Drag & drop ZS SSIS XML Source.
Double click the XML Source to configure it.
In the URL text box enter API URL like below to list S3 Files for specified bucket (Change YOUR-BUCKET to your own name)
```
https://s3.us-east-1.amazonaws.com/YOUR-BUCKET
```
Now check Use Credentials and Select same Amazon API connection we created in previous example.
Click on Select Filter (This step allows us to flatten the XML hierarchy. Select the node which is Array icon like below. If prompted to treat selected node as array click Yes.
Configure SSIS XML Source – Get Amazon S3 File List from specific bucket (AWS API)
Click Preview to see your data. Click OK to save.
Preview Data in XML Source – Amazon S3 File List (Calling AWS API)
Now attach your XML source to target like OLEDB Destination to load data to SQL Server or other Target (e.g. Oracle, MySQL)
Execute SSIS Package to load data from Amazon AWS API to SQL Server.
SSIS Package – Read Amazon S3 File List and load into SQL Server Table (Call AWS API Example)

File Upload Example – Low level API – Call PUT request

There will be a time when you want to take total control of your AWS API calls. One example is if you wish to Upload / Write data to S3 then components like ZappySys Amazon S3 CSV Destination or Amazon Storage Task might need additional permission such as HeadObject . If you have only write permission on bucket then this will fail to execute Task. In such case you can use REST API Task like below way.

Basically we exported data from Relation Database to CSV File using Export CSV File Task and then we uploaded file content using REST API Task.

Upload Text File to S3 (i.e. JSON, CSV, XML …)

Your URL Format can be like this. Region codes can be found here

https://YOUR-BUCKET.s3.YOUR-REGION.amazonaws.com/YOUR-FILE
https://YOUR-BUCKET.s3.YOUR-REGION.amazonaws.com/SOME-FOLDER/SOME-SUB-FOLDER/YOUR-FILE

Upload file to Amazon S3 – Call low level API (PUT request)

Upload Binary File to S3 (i.e. Zip, mp3, gzip, png, jpeg…)

Above method only works for Text Files. If you have Binary files (e.g. Zip file, mp3, png, jpeg) then you can use below workaround.

Check Is Multi Part / File Upload Option next to the Body editor (Read this Post for more info)
In the Body enter file path like below
```
@c:\folder\some-file.xyz
```
That’s it, now your binary file can be uploaded to S3 same way as Text file we uploaded in earlier section.

NOTE: Binary file option only works in the latest Build Uploaded after 5/21/2020

Debugging AWS API Command Line Requests using Fiddler (Web Proxy)

Before we see more examples of calling AWS API lets first learn how to capture Request data using aws command line (CLI). We will use Fiddler to capture AWS API Requests.

Download and Install Fiddler (Free Tool)
Install aws command line

Open command prompt and type aws configure command to set credentials. For more info see configure aws credentials .. see below example.

c:> aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

Once you set credentials launch Fiddler
On the Fiddler Tools menu > Click Fiddler Option > HTTPS > Check Decrypt HTTPS Traffic
Once you do that you may be asked to Trust Fiddler Certificate click OK
Close and Open Fiddler to apply setting
Now any command you type in aws command line will show up in fiddler. This command we can use in SSIS REST API Task or XML Source to call virtually Any API AWS supports.
For example to call some lambda function (supply input json data from file) use below command and watch Fiddler Trace. Notice we added –no-verify-ssl option so we can see requests in custom web proxy like fiddler (This option will trust Fiddler certificate).
```
c:\>aws lambda invoke --function-name HelloWorld c:\temp\outputfile.txt --no-verify-ssl --payload file://c://test/customer.json
```
Capture AWS Commandline (CLI) request using Fiddler
Once you have this information you can use it inside ZappySys Components which supports API calls (e.g. REST API Task, JSON Source, XML Source). For this example we will use REST API Task to call same Lambda Function. Things to change to call any API is URL, Method, ContentType, Body. If its GET call then you wont have Body.
Calling Amazon AWS Lambda Function using SSIS REST API Task
That’s it.. You can now take this same concept and call virtually API AWS API right insight SSIS without any SDK or command line tools.

Call API Gateway Endpoint (Default URL)

If you wish to call API hosted on Amazon API Gateway Service then enter direct URL. You have to use OAuth Connection (AWS v4 Provider) as previous section.

Typical URL may look like as below if you calling API gateway Endpoint.

https://c5hhigf5mh.execute-api.us-east-1.amazonaws.com/prod/pets

Call AWS API with Region and Custom Service Name

There will be a time when you will have to call AWS API URL and it doesnt indicate Region / Service but you have to supply part of signature for Authentication. In this case just enter AWS Service and Region along with ClientID, Secret Fields (Custom Service and Region attributes were introduced in v3.0 or higher so if you are not seeing on OAuth Connection UI then you probably running older version).

Call Amazon Athena API

Check this article for detailed instructions

Call AWS EC2 API

Here is an example of calling EC2 API. This is example of listing EC2 instances.

Request:

POST https://ec2.us-east-1.amazonaws.com/
Content-Type: application/x-www-form-urlencoded; charset=utf-8

Action=DescribeInstances&Version=2016-11-15

Call AWS Lambda API

Here is some example of calling Lambda function

Request:

POST https://lambda.YOUR-REGION.amazonaws.com/2015-03-31/functions/YOUR-FUNCTION/invocations

{you-json-input-goes-in-body}

Call AWS ElasticSearch

If you are using AWS Hosted Managed ElasticSearch then also you can use OAuth AWS v4 Provider as below. Assuming you have configured correct Policy to allow your IAM User Account / IP address.

Call AWS Hosted ElasticSearch REST API in SSIS (V4 Request Signing)

Call AWS API using Native Task/Components

ZappySys provides many High quality Tasks / Components for AWS integration. See below list.

Amazon AWS Cloud Integration
Amazon S3 Task	Amazon Redshift Data Transfer Task	Amazon Redshift ExecuteSql Task
Amazon Redshift Cluster Management Task	Amazon DynamoDB Source	Amazon DynamoDB Destination
Amazon Redshift Source	Amazon SQS Queue Source	Amazon SQS Queue Destination

Conclusion

Amazon AWS Cloud integration from SSIS packages becoming more and more common scenario. ZappySys provides easy to use no coding connectors to achieve many time consuming scenarios. Having clean drag and drop approach is not only faster but more secure using inbuilt SSIS framework. Try SSIS PowerPack to explore many other scenarios not discussed in this article.

The post How to call Amazon AWS API using SSIS (EC2, Lambda, API Gateway, SQS) appeared first on ZappySys Blog.