Introduction
ZappySys provides high-performance drag and drop connectors for Amazon AWS Integration. In this post, we will see How to Read / Write / Delete Amazon SQS Queue data in SSIS using ZS Amazon SQS components. ZappySys developed many AWS related components but in this article, we will look at 2 components for Amazon SQS Queue integration Scenarios (Read and Write). We will discuss how to use SSIS Amazon SQS Source and SSIS Amazon SQS Destination. Amazon SQS Source can be used to read messages from Amazon AWS SQS (Simple Queue Storage) and Amazon SQS Destination can be used to write/delete data from Amazon SQS (Simple Queue Storage) using SSIS.
Prerequisite
Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:
- First, you will need to have SSIS installed.
- Secondly, make sure to have SSDT.
- Thirdly, you have obtained Amazon SQS Credentials.
- Finally, do not forget to install ZappySys SSIS PowerPack.
Component Mention in this article
What is Amazon SQS (Simple Queue Service)
Before we talk How to Read / Write / Delete Amazon SQS Queue data in SSIS lets talk What is Amazon SQS (Simple Queue Service) or sometimes referred to as AWS SQS. Amazon SQS offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues and costs allocation tags. It provides a generic web services API and it can be accessed by any programming language that the AWS SDK supports.
SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.
Getting Started
In order to start, we will show several examples. ZappySys includes an Amazon SQS Source and Amazon SQS Destination, that will help you How to Read / Write / Delete Amazon SQS Queue data in SSIS. You can connect to your Amazon SQS account by entering credentials.
Amazon SQS Source can be used to read messages from Amazon AWS SQS (Simple Queue Storage) using SSIS. Read Amazon SQS Messages (AWS SQS) in bulk. Amazon SQS Destination can be used to write/delete data from Amazon SQS (Simple Queue Storage) using SSIS. Write messages to Amazon SQS (AWS SQS) in bulk. Support for Visibility Timeout (Hide messages from others for certain time after you read), data preview and max rows, reading message attributes in separate output columns and adjusting UTC DateTime in local time.
Write Data using Amazon SQS Destination in SSIS
Overview
In this tutorial, you will learn how to write data in Amazon SQS Queue storage. You can use ZS DummyData Source for practice it or select suitable ZS Source Adapters. In this case its from ZS DummyData Source. So let’s start with an example, In this section, you will learn how to write messages into Amazon SQS Queue. For this example, we are assuming that you have already SQS Queue Created.
Create Amazon SQS Connection
- Firstly, In order to connect to Amazon SQS from SSIS, you will need Access Key and Secret Key (Login to AWS portal and Create Access/Secret Key with SQS service read access). If you are not sure about this ask your SysAdmin or responsible person to provide that to you. Your key will look something like this: (This is just an example key which may differ in your case).
Click here to read more about how to get your AccessKey and Secret Key.12AccessKey: AKIAIOSFODNN7EXAMPLESecretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY - Secondly, Once you have Account Key (it’s like UserID) and Secret Key (its like password) you may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
- Right-click in Connection Manager Panel and Select [New Connection] menu item.
SSIS Create AWS Storage Connection
- Now Select ZS-AWS-STORAGE connection type.
Select ZS-AWS-STORAGE Connection
- Let’s configure the Amazon SQS connection. Set Storage Service to SQS and Credential Type to Key-based access control. In the General tab enter your Amazon SQS Credentials and Set Default region as per you have.
SSIS AWS Storage Connection – Configure
- In the last, click on Test Connection and OK button to Save connection configure setting UI.
Write Data into Amazon SQS
- Once you have created the Amazon SQS Connection successfully, let’s drag and Drop Data Flow Task from SSIS Toolbox on the control flow designer surface.
Drag and Drop SSIS Data Flow Task from SSIS Toolbox
- Double click on the Data Flow Task to see Data Flow Designer surface.
- Here, from the SSIS Toolbox drag and drop ZS DummyData Source on the data flow designer surface.
SSIS ZS Dummy Data Source – Drag and Drop
- Double click on Dummy Data Source to configure it.
SSIS ZS Dummy Data Source – Configure
- Click on OK button to save configure setting UI.
- From the SSIS toolbox drag and drop Amazon SQS Destination on the Data flow designer surface and single click on the Dummy Data Source, Once you see the blue arrow from the source. connect it to Amazon SQS Destination.
SSIS ZS Amazon SQS Destination
- Now double click on Amazon SQS Destination to configure it.
- In the [Connection Manager] tab select SQS connection manager we have created before it.
SSIS Amazon SQS Destination – Configure Connection Manager
- Click on [Component Properties] tab and Select Target Queue from the dropdown list.
SSIS ZS Amazon SQS Destination – Configure Component Properties
- Click on [Mappings] tab and verify mapping. Map document column from upstream to the MessageText target column.
SSIS ZS Amazon SQS Destination – Column Mappings
- In the last, click on the OK button to save configure setting UI.
- After all, Execute the Package and verify target data by going to your AWS console under SQS service section.
Run or Execute SSIS Package
Verify Amazon SQS Destination
Read Data using Amazon SQS Source in SSIS
Overview
In this tutorial, you will learn how to read data from Amazon SQS storage in SSIS using ZS SSIS Amazon SQS Source Adapter. In this SSIS we will read data. So let’s begin it.
Read Data from Amazon SQS
- Firstly, create a new SSIS package. In BIDS/SSDT create a new SSIS package.
- From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
SSIS Data Flow Task – Drag and Drop
- Double click on the DataFlow task to see DataFlow designer surface.
- From the SSIS toolbox drag and drop ZS Amazon SQS Source on the dataflow designer surface.
- Double click Amazon SQS Source to configure it.
- Select Amazon SQS Connection we have created before and select queue from the dropdown list.
SSIS Amazon SQS Source – Configure
- Click on Preview button to see Data Preview and Click on OK button to save settings.
- From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface or you can use any of our suitable ZS Destination Adapter.
SSIS ZS Trash Destination – Drag and Drop
- Now single click on the Amazon SQS Source once you see the blue arrow from the source … connect it to Trash Destination.
- Double click on ZS Trash Destination to Configure it.
SSIS ZS Trash Destination – Configure
- Click on OK button to save Trash Destination configure setting UI.
- After all, right-click on the path and Add Data Viewer and Execute the package and verify source data in the data viewer.
Run or Execute The SSIS Package
- After messages read from SQS Queue, the same message cannot be read again until VisibilityTimeout. By default Queue, visibility timeout is used (-1 means use default) but you can override it on the source UI.
Read all messages from SQS FIFO Queue
If you are using Normal SQS Queue then by setting Max Rows=0 will read all messages from your queue. However reading all messages from FIFO Queue is not that simple. Amazon SQS wont return more messages until you delete previously read Messages from same Message Group (identified by MessageGroupId). Due to this constraint you have to try below technique of Loop to read 10 messages until all messages Read > Processed > Deleted.
Here is note from Amazon Documentation about maximum 10 messages limitation on FIFO Queue.
|
1 |
When you receive a message with a message group ID, no more messages for the same message group ID are returned unless you delete the message or it becomes visible. |
Community Reference:
https://stackoverflow.com/questions/51661834/aws-sqs-fifo-how-to-get-more-than-10-messages-at-a-time
If you want to add fault tolerance then you can set Max Retry for NULL Receive like below (e.g. Setting MaxRetry=3 will try max 3 times if no message is read …. It will try every 2 seconds for 3 times before giving up to read more rows).
Read all messages from Amazon SQS FIFO Queue
Read , Process and Delete Amazon SQS FIFO Queue Message in SSIS Data flow
Delete Message from Queue using Amazon SQS Destination in SSIS
Overview
In this tutorial, you will learn how to Delete Messages from the Queue using the ZS SSIS Amazon SQS Destination Adapter. So let’s begin it.
Delete Message from Queue
- From the SSIS toolbox drag and drop Amazon SQS Destination on the Data flow designer surface and connect it to Amazon SQS Destination.
- Now double click on Amazon SQS Destination to configure it.
- In the [Connection Manager] tab select SQS connection manager we have created before it.
Select Amazon SQS Connection
- Click on [Component Properties] tab, Set Operation to Delete and Select Target Queue from the dropdown list.
Select Delete Operation and Queue
- Click on [Column Mappings] tab and verify mapping. Map document column from upstream to the MessageText target column.
Do Columns Mapping
- In the last, click on the OK button to save configure setting UI.
- After all, Execute the Package and verify target data by going to your AWS console under SQS service section.
Purge All Messages from SQS Queue (Clear Queue)
There will be a time you like to clear entire queue (i.e. Delete All Messages from Queue). Amazon recently added PurgeQueue API feature so now its possible without doing Delete message technique mentioned in previous section. ZappySys support calling any AWS API using REST API Task along with AWS Provider in OAuth Connection. You can use same technique to call PurgeQueue API like below.
- Open SSIS Package and Drag ZS REST API Task from Control Flow Toolbox
- Click on URL mode dropdown and change to URL from Connection.
- From COnnection Dropdown select NEW ZS-OAuth connection
- Configure OAuth connection to use Provider=Amazon AWS v4
- Enter Access Key and Secret Key and click OK to save connection
- Enter below URL on REST API Task (Replace Queue ID and name with your own values) – Refer to your AWS Console / SQS Queue Properties to get this URL
Here is an example of Sample Queue from ZappySys Demo1https://sqs.ENTER-REGION-HERE.amazonaws.com/ENTER-QUEUE-ID-HERE/ENTER-QUEUE-NAME-HERE/?Action=PurgeQueue
1https://sqs.us-east-1.amazonaws.com/718883169616/zappysystest.fifo/?Action=PurgeQueue - Click ok to Save REST API Task
How to Purge Amazon SQS Queue in SSIS (Clear Queue / Delete All messages using PurgeQueue REST API)
Conclusion
So, In this blog, we learn How to Read / Write / Delete Amazon SQS Queue data in SSIS. We used Amazon SQS Destination for write data and Amazon SQS Source to read data. Additionally, you can download SSIS PowerPack here to try many other scenarios that are not discussed in this blog along with 70+ other components.
References
Finally, you can use the following links for more information:
