read Archives | ZappySys Blog

How to Read / Write / Delete Amazon SQS Queue data in SSIS

ZappySys — Mon, 19 Aug 2019 12:17:48 +0000

Introduction

ZappySys provides high-performance drag and drop connectors for Amazon AWS Integration. In this post, we will see How to Read / Write / Delete Amazon SQS Queue data in SSIS using ZS Amazon SQS components. ZappySys developed many AWS related components but in this article, we will look at 2 components for Amazon SQS Queue integration Scenarios (Read and Write). We will discuss how to use SSIS Amazon SQS Source and SSIS Amazon SQS Destination. Amazon SQS Source can be used to read messages from Amazon AWS SQS (Simple Queue Storage) and Amazon SQS Destination can be used to write/delete data from Amazon SQS (Simple Queue Storage) using SSIS.

Prerequisite

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

First, you will need to have SSIS installed.
Secondly, make sure to have SSDT.
Thirdly, you have obtained Amazon SQS Credentials.
Finally, do not forget to install ZappySys SSIS PowerPack.

Component Mention in this article

	Amazon SQS Source Connector
	Amazon SQS Destination Connector

What is Amazon SQS (Simple Queue Service)

Before we talk How to Read / Write / Delete Amazon SQS Queue data in SSIS lets talk What is Amazon SQS (Simple Queue Service) or sometimes referred to as AWS SQS. Amazon SQS offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues and costs allocation tags. It provides a generic web services API and it can be accessed by any programming language that the AWS SDK supports.

SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

Getting Started

In order to start, we will show several examples. ZappySys includes an Amazon SQS Source and Amazon SQS Destination, that will help you How to Read / Write / Delete Amazon SQS Queue data in SSIS. You can connect to your Amazon SQS account by entering credentials.

Amazon SQS Source can be used to read messages from Amazon AWS SQS (Simple Queue Storage) using SSIS. Read Amazon SQS Messages (AWS SQS) in bulk. Amazon SQS Destination can be used to write/delete data from Amazon SQS (Simple Queue Storage) using SSIS. Write messages to Amazon SQS (AWS SQS) in bulk. Support for Visibility Timeout (Hide messages from others for certain time after you read), data preview and max rows, reading message attributes in separate output columns and adjusting UTC DateTime in local time.

Write Data using Amazon SQS Destination in SSIS

Overview

In this tutorial, you will learn how to write data in Amazon SQS Queue storage. You can use ZS DummyData Source for practice it or select suitable ZS Source Adapters. In this case its from ZS DummyData Source. So let’s start with an example, In this section, you will learn how to write messages into Amazon SQS Queue. For this example, we are assuming that you have already SQS Queue Created.

Create Amazon SQS Connection

Firstly, In order to connect to Amazon SQS from SSIS, you will need Access Key and Secret Key (Login to AWS portal and Create Access/Secret Key with SQS service read access). If you are not sure about this ask your SysAdmin or responsible person to provide that to you. Your key will look something like this: (This is just an example key which may differ in your case).
```
AccessKey: AKIAIOSFODNN7EXAMPLE
SecretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
```
Click here to read more about how to get your AccessKey and Secret Key.
Secondly, Once you have Account Key (it’s like UserID) and Secret Key (its like password) you may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
Right-click in Connection Manager Panel and Select [New Connection] menu item.
SSIS Create AWS Storage Connection
Now Select ZS-AWS-STORAGE connection type.
Select ZS-AWS-STORAGE Connection
Let’s configure the Amazon SQS connection. Set Storage Service to SQS and Credential Type to Key-based access control. In the General tab enter your Amazon SQS Credentials and Set Default region as per you have.
SSIS AWS Storage Connection – Configure
In the last, click on Test Connection and OK button to Save connection configure setting UI.

Write Data into Amazon SQS

Once you have created the Amazon SQS Connection successfully, let’s drag and Drop Data Flow Task from SSIS Toolbox on the control flow designer surface.
Drag and Drop SSIS Data Flow Task from SSIS Toolbox
Double click on the Data Flow Task to see Data Flow Designer surface.
Here, from the SSIS Toolbox drag and drop ZS DummyData Source on the data flow designer surface.
SSIS ZS Dummy Data Source – Drag and Drop
Double click on Dummy Data Source to configure it.
SSIS ZS Dummy Data Source – Configure
Click on OK button to save configure setting UI.
From the SSIS toolbox drag and drop Amazon SQS Destination on the Data flow designer surface and single click on the Dummy Data Source, Once you see the blue arrow from the source. connect it to Amazon SQS Destination.
SSIS ZS Amazon SQS Destination
Now double click on Amazon SQS Destination to configure it.
In the [Connection Manager] tab select SQS connection manager we have created before it.
SSIS Amazon SQS Destination – Configure Connection Manager
Click on [Component Properties] tab and Select Target Queue from the dropdown list.
SSIS ZS Amazon SQS Destination – Configure Component Properties
Click on [Mappings] tab and verify mapping. Map document column from upstream to the MessageText target column.
SSIS ZS Amazon SQS Destination – Column Mappings
In the last, click on the OK button to save configure setting UI.
After all, Execute the Package and verify target data by going to your AWS console under SQS service section.
Run or Execute SSIS Package

Verify Amazon SQS Destination

Read Data using Amazon SQS Source in SSIS

Overview

In this tutorial, you will learn how to read data from Amazon SQS storage in SSIS using ZS SSIS Amazon SQS Source Adapter. In this SSIS we will read data. So let’s begin it.

Read Data from Amazon SQS

Firstly, create a new SSIS package. In BIDS/SSDT create a new SSIS package.
From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
SSIS Data Flow Task – Drag and Drop
Double click on the DataFlow task to see DataFlow designer surface.
From the SSIS toolbox drag and drop ZS Amazon SQS Source on the dataflow designer surface.
Double click Amazon SQS Source to configure it.
Select Amazon SQS Connection we have created before and select queue from the dropdown list.
SSIS Amazon SQS Source – Configure
Click on Preview button to see Data Preview and Click on OK button to save settings.
From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface or you can use any of our suitable ZS Destination Adapter.
SSIS ZS Trash Destination – Drag and Drop
Now single click on the Amazon SQS Source once you see the blue arrow from the source … connect it to Trash Destination.
Double click on ZS Trash Destination to Configure it.
SSIS ZS Trash Destination – Configure
Click on OK button to save Trash Destination configure setting UI.
After all, right-click on the path and Add Data Viewer and Execute the package and verify source data in the data viewer.
Run or Execute The SSIS Package
After messages read from SQS Queue, the same message cannot be read again until VisibilityTimeout. By default Queue, visibility timeout is used (-1 means use default) but you can override it on the source UI.

Read all messages from SQS FIFO Queue

If you are using Normal SQS Queue then by setting Max Rows=0 will read all messages from your queue. However reading all messages from FIFO Queue is not that simple. Amazon SQS wont return more messages until you delete previously read Messages from same Message Group (identified by MessageGroupId). Due to this constraint you have to try below technique of Loop to read 10 messages until all messages Read > Processed > Deleted.

Here is note from Amazon Documentation about maximum 10 messages limitation on FIFO Queue.

When you receive a message with a message group ID, no more messages for the same message group ID are returned unless you delete the message or it becomes visible.

Community Reference:

https://stackoverflow.com/questions/51661834/aws-sqs-fifo-how-to-get-more-than-10-messages-at-a-time

If you want to add fault tolerance then you can set Max Retry for NULL Receive like below (e.g. Setting MaxRetry=3 will try max 3 times if no message is read …. It will try every 2 seconds for 3 times before giving up to read more rows).

Read all messages from Amazon SQS FIFO Queue

Read , Process and Delete Amazon SQS FIFO Queue Message in SSIS Data flow

Delete Message from Queue using Amazon SQS Destination in SSIS

Overview

In this tutorial, you will learn how to Delete Messages from the Queue using the ZS SSIS Amazon SQS Destination Adapter. So let’s begin it.

Delete Message from Queue

From the SSIS toolbox drag and drop Amazon SQS Destination on the Data flow designer surface and connect it to Amazon SQS Destination.
Now double click on Amazon SQS Destination to configure it.
In the [Connection Manager] tab select SQS connection manager we have created before it.
Select Amazon SQS Connection
Click on [Component Properties] tab, Set Operation to Delete and Select Target Queue from the dropdown list.
Select Delete Operation and Queue
Click on [Column Mappings] tab and verify mapping. Map document column from upstream to the MessageText target column.
Do Columns Mapping
In the last, click on the OK button to save configure setting UI.
After all, Execute the Package and verify target data by going to your AWS console under SQS service section.

Purge All Messages from SQS Queue (Clear Queue)

There will be a time you like to clear entire queue (i.e. Delete All Messages from Queue). Amazon recently added PurgeQueue API feature so now its possible without doing Delete message technique mentioned in previous section. ZappySys support calling any AWS API using REST API Task along with AWS Provider in OAuth Connection. You can use same technique to call PurgeQueue API like below.

Open SSIS Package and Drag ZS REST API Task from Control Flow Toolbox
Click on URL mode dropdown and change to URL from Connection.
From COnnection Dropdown select NEW ZS-OAuth connection
Configure OAuth connection to use Provider=Amazon AWS v4
1. Enter Access Key and Secret Key and click OK to save connection
Enter below URL on REST API Task (Replace Queue ID and name with your own values) – Refer to your AWS Console / SQS Queue Properties to get this URL
```
https://sqs.ENTER-REGION-HERE.amazonaws.com/ENTER-QUEUE-ID-HERE/ENTER-QUEUE-NAME-HERE/?Action=PurgeQueue
```
Here is an example of Sample Queue from ZappySys Demo
```
https://sqs.us-east-1.amazonaws.com/718883169616/zappysystest.fifo/?Action=PurgeQueue
```
Click ok to Save REST API Task

How to Purge Amazon SQS Queue in SSIS (Clear Queue / Delete All messages using PurgeQueue REST API)

Conclusion

So, In this blog, we learn How to Read / Write / Delete Amazon SQS Queue data in SSIS. We used Amazon SQS Destination for write data and Amazon SQS Source to read data. Additionally, you can download SSIS PowerPack here to try many other scenarios that are not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

Amazon SQS Source
Amazon SQS Destination
Documents of Amazon SQS
Help File:
- Amazon SQS Source
- Amazon SQS Destination

The post How to Read / Write / Delete Amazon SQS Queue data in SSIS appeared first on ZappySys Blog.

How to Read / Write Amazon Redshift Data in SSIS

ZappySys — Fri, 16 Aug 2019 14:05:24 +0000

Introduction

ZappySys provide high-performance drag and drop connectors for Amazon Redshift Integration. In our previous post we saw how to bulk load SQL data into Redshift using S3 staging technique (COPY command). Now in this post, you will see How to Read / Write Amazon Redshift Data in SSIS Data flow without need for S3 Staging. We will use ZS Amazon Redshift components (Source / Destination). If you need to Upsert / delete data from Redshift Table then use Upsert Destination (Not discussed in this article).

ZappySys developed many AWS related components but in this article, we will look at 2 Components for Amazon Redshift Integration Scenarios (Read and Write). We will discuss how to use SSIS Amazon Redshift Source and SSIS Amazon Redshift Destination. Amazon Redshift Source connector supports familiar SQL Like query language so any DBA or SQL user can learn it in no time. ZappySys developed a highly sophisticated engine to turn your SQL Query into native Amazon Redshift query. Load query at runtime from the variables or Direct value.

Prerequisite

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
Thirdly, You have obtained Amazon Redshift Credential.
Finally, do not forget to install ZappySys SSIS PowerPack.

Component Mention in this article

	Amazon Redshift Source
	Amazon Redshift Destination

What is Amazon Redshift

Before we talk How to Read / Write Amazon Redshift Data in SSIS lets talk what is Amazon Redshift (or sometimes referred to as AWS Redshift). Amazon Redshift is a Cloud-based Data warehouse service. This type of system also referred to as MPP (Massively Parallel Processing). Amazon Redshift uses a highly modified version of PostgreSQL Engine behind the scene. Amazon Redshift provides the advantage of Scale as you go, at a very low cost compared to onsite dedicated hardware/software approach. In this article, we will try to learn How to Read / Write Amazon Redshift Data in SSIS.

Getting Started

In order to start, we will show several examples. ZappySys includes an Amazon Redshift Source and Destination, that will help you How to Read / Write Amazon Redshift Data in SSIS. You can connect to your Amazon Redshift Account by entering credentials.

SSIS Amazon Redshift Source Connector can be used to read data from Amazon Redshift. You can use a simple Table mode or write custom SQL Query to extract desired data. You can use multiple modes (i.e. Table or Query) to read data from the Amazon Redshift Database. Support for data preview and max rows and Dynamic query (using SSIS Variable placeholder e.g. select * from mytable where id='{{User:: MyVar}}’).

SSIS Amazon Redshift Destination can be used to bulk insert a large number of records to Amazon Redshift storage from any data source using SSIS. Amazon Redshift is a cloud-based Data Warehouse service from Amazon AWS. Here, you can bulk Insert, Update and Delete Amazon Redshift data and supporting the batch size.

Click here to see Amazon Redshift related articles.

Write Data using Amazon Redshift Destination in SSIS

Overview

In this tutorial, you will learn how to write data in Amazon Redshift Database. You can use ZS DummyData Source for practice it or select suitable ZS Source Adapter (In this case it’s from SQL Server Source). So, Let’s start with an example. In this SSIS we will write data into Amazon Redshift Database using ZS Amazon Redshift Destination.

How to write data into Amazon Redshift Table

In order to connect to the Amazon Redshift Database from SSIS, you will need credentials. Ask your SysAdmin or responsible person to provide that you.

For this sample exercise, we will create a table in Redshift.

CREATE TABLE public."tbl_pg_Customers"
(
    "RecordID" character varying(50) NOT NULL,
    "CustomerID" character varying(50),
    "CustomerName" character varying(50),
    "Address1" character varying(50),
    "City" character varying(50),
    "State" character varying(50),
    "Zip" character varying(50),
    "Country" character varying(50),
    PRIMARY KEY ("RecordID")
);

Once you have created a Table, you may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
In your SSIS toolbox, you should see many new Icons starting with “ZS” (i.e. ZappySys). If you don’t see them then check this.
Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
Drag and Drop SSIS Data Flow Task from SSIS Toolbox
Double click on the Data Flow Task to see Data flow designer surface.
Here, In Visual Studio, Drag and Drop the OLE DB Source and Amazon Redshift Destination in the design surface and join the components with the blue arrow.
SSIS OLE DB Source and Amazon Redshift Destination – Drag and Drop
Now, we need OLE DB Connection and Amazon Redshift Connection. Click here for Create Amazon Redshift Connection.

How to Create an OLE DB Connection.

Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
SSIS OLE DB Connection – Create Connection
Now click on the New button to create a connection.
SSIS OLE DB Connection – Create New Connection
Let’s configure connection Manager, just follow the steps one as we have created in following the image.
SSIS OLE DB Connection – Configure Connection
After all, Click on OK button to save configure setting UI.

Configure SSIS ZS Amazon Redshift Destination

Double click on OLE DB Source for configure it.
Let’s Configure in Connection Manager, just follow below image steps.
SSIS OLE DB Source – Configure Source Editors
Click OK to save OLE DB Source Editor UI settings.
Now double click on Amazon Redshift Destination to configure it. In the [Connection Manager] tab Select Amazon Redshift Connection.
SSIS Amazon Redshift Destination – Connection Manager
Click on [Component Properties] tab and Pick the Destination Table from Tables dropdown list.
SSIS Amazon Redshift Destination – Component Properties
Click on [Column mappings] tab and verify mapping. If the target table is new Table and it’s empty then all input columns will be automatically mapped as shown below. If the table is an existing table with some data then you can manually pick a mapping column by dragging it from source list to target list.
SSIS Amazon Redshift Destination – Column Mappings
Execute the package and verify that the data is saved.
SSIS Amazon Redshift Destination – Run or Execute Package

Read Data using Amazon Redshift Source in SSIS

Overview

In this tutorial, you will learn how to read data from Amazon Redshift in SSIS using ZS Amazon Redshift Source adapter. In this SSIS we will read data with SQL Query and Table Mode. So Let’s begin it.

How to Read data from Amazon Redshift Table

You may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
SSIS Data Flow Task – Drag and Drop
Double click on the Data Flow Task to see the Data Flow design surface.
From the SSIS Toolbox, Drag and Drop Amazon Redshift Source in the data flow designer surface.
SSIS Amazon Redshift Source – Drag and Drop
Now, We need Amazon Redshift Connection. Click here for creating Amazon Redshift Connection.

Configure SSIS ZS Amazon Redshift Source – Query Mode

Here, we are getting data using Query Mode with dynamic expression value.
Let’s Create Variables, In Visual Studio Right Click on Design Panel, Select Variables.
SSIS Create Variables
Make sure correct DataType and Value, use the following image.
SSIS Created Variables with Value
Double click on Amazon Redshift Source to configure it.
Select the connection we have created before, set AccessMode to Query and use the following SQL Query(You can replace table or column name) and insert Variables we have created before.
```
SELECT  * FROM "public"."tbl_pg_Orders"
WHERE "CustomerID" like '%R%' AND "OrderAmount" > '{{User::OrderAmount}}' AND
"{{User::ColumnName}}" = '{{User::Value}}'
```
SSIS Amazon Redshift Source – Configure with SQL Query Mode
Click on Preview button to see Data Preview and OK button to save configure setting UI.
Here, you can use any of our suitable ZS Destination Adapter.
From the SSIS toolbox drag and drop Trash Destination on the Data Flow designer surface.
SSIS Trash Destination – Drag and Drop
Now single click on the Amazon Redshift Source, once you see the blue arrow from the source…connect it to Trash Destination.
Double click on Trash Destination to Configure it.
SSIS Trash Destination – Configure
Click on OK button to save Trash Destination configure setting UI.
That’s all, just Run or Execute the package and verify data.
SSIS Amazon Redshift Source – Run or Execute the Package

Configure SSIS ZS Amazon Redshift Source – Table Mode

Double click on Amazon Redshift Source to configure it.
Select the connection we have created before. Set Access Mode to Table and Select Table from table Drop down list.
SSIS Amazon Redshift Source – Configure with Table Mode
Click on Preview button to see Data Preview and OK button to save Amazon Redshift Source configure setting UI.
From the SSIS toolbox, Drag and Drop Trash Destination on the Data flow designer surface.
SSIS Trash Destination – Drag and Drop
Now single click on the Amazon Redshift Source, once you see the blue arrow from a source… connect it to Trash Destination.
Double click on ZS Trash Destination to configure it.
SSIS Trash Destination – Configure
Click on OK button to save Trash Destination configure setting UI.
That’s all, just run or execute your package and see data.
SSIS Amazon Redshift Source – Run or Execute Package

Conclusion

In this blog, we learn How to Read / Write Amazon Redshift Data in SSIS. We used Amazon Redshift Destination for write data and Amazon Redshift Source to read data. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

Amazon Redshift Source
Amazon Redshift Destination
Document of Amazon Redshift
Help File:
- Amazon Redshift Source
- Amazon Redshift Destination

The post How to Read / Write Amazon Redshift Data in SSIS appeared first on ZappySys Blog.

How to Read/Write Azure Queue Storage data in SSIS

ZappySys — Tue, 06 Aug 2019 12:47:38 +0000

Introduction

In our previous blog, we saw you how to read/write Azure Table Storage data. Now in this blog, we will see How to Read/Write Azure Queue Storage data in SSIS. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like perform other operations on Azure Blob Storage Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on How to Read/Write Azure Queue Storage data in SSIS using the following Source/Destination.

Component Mentioned in this article

	Azure Queue Storage Source
	Azure Queue Storage Destination

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
Download and install Microsoft Azure Storage Emulator
Download and install Microsoft Azure Storage Explorer
Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Storage) then you can skip Step #3

What are Azure Queues?

Azure Queue storage is a service for storing large numbers of messages. You access messages from anywhere in the world via authenticated calls using HTTP or HTTPS. A queue message can be up to 64 KB in size. A queue may contain millions of messages, up to the total capacity limit of a storage account.

Common uses

Microsoft Azure Queue Concept

Common uses of Queue storage include:

Creating a backlog of work to process asynchronously
Passing messages from an Azure web role to an Azure worker role
The Queue service contains the following components:

Getting Started

In order to start, we will show several examples. Zappysys includes an SSIS Azure Queue Storage Source that will help you in reading data of Queue and SSIS Azure Queue Storage Destination that will help you to write data in Queue. Here we are showing you is, how to Read/Write Azure Queue Storage data in SSIS.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of the local Azure Storage Emulator.

Setup Azure Storage client tools and Create Queue

Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
Azure Storage Emulator Physical Location
If you can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
Command Prompt Screen after Microsoft Azure Storage Emulator Started
Now, you have to download and install Microsoft Azure Storage Explorer and then you can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Explorer Location.
If you don’t have Azure Storage account then it’s easy to get FREE Trial Azure Storage account or use your MSDN to get credit each month which will be more than enough for real testing. If you prefer to avoid all this hassle then Microsoft provides another great way to test Azure Storage functionality is totally offline mode on your local machine. You can just download Azure Storage Emulator and start testing.
Click here to learn more about how to configure Azure Storage Emulator for Testing and Development purposes. In order to connect to Azure Storage Service from SSIS, you will need Storage AccountName and an AccessKey. Ask your SysAdmin or responsible person to provide that information to you. Here are sample Account Name and Access Key(this is just example key which may differ in your case)
```
Account Name: mystorageaccount 
Access Key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
```
For Creating a Queue, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Queues. Right-click on Queues and Select Create Queue and give a name you want.
Microsoft Azure Storage Explorer – Create Queue

Write data using ZS Azure Queue Storage Destination

In this tutorial, you will learn how to write data into Azure Queue Storage. We will use ZS DummyData Source for practice it but you can use a suitable Source Adapter. So, Let’s start, In this SSIS we will write data into Azure Queue Storage using ZS Azure Queue Storage Destination.

Once you have created Azure Storage Queue and you got valid Account Name and AccessKey (for the local emulator you don’t need this) you may proceed to create new SSIS package. In BIDS/SSDT create a new SSIS package.
From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface and double click on it too.
Drag and Drop SSIS Data Flow Task from SSIS Toolbox
Click here to create Azure Storage Connection.
From the SSIS toolbox drag and drop Dummy Data Source on the dataflow designer surface.
DummyData Source – Drag and Drop
Double click Dummy Data Source to configure it. From Template pick JsonDocuments and enter row count=100 to generate 100 JSON documents and click OK to save.
DummyData Source – Configure
Let’s drag and drop Azure Queue Storage Destination on the dataflow designer surface from the SSIS toolbox.
Now single click on the Dummy Data Source, once you see the blue arrow from the source connect it to Azure Queue Destination.
Double click Azure Queue Destination to configure it. Select the connection we have created before.
Configure Connection Manager
Click on [Component Properties] tab and select the Target Queue from the dropdown list.
Configure Component Properties
Click on [Mappings] tab and verify mapping. Map Document column from upstream to the MessageText target column.
Configure Column Mappings
Click OK to save settings.
Execute the package and verify target data by going to your Azure Queue. We use Azure Storage Explorer.
SSIS Execute Package

Microsoft Azure Explorer – Verify Queue Data

Read data using ZS Azure Queue Storage Source

Here you may proceed In BIDS/SSDT to create a new SSIS package.
Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
SSIS Data Flow Task – Drag and Drop
Double click on the Data Flow task to see the Data Flow designer surface.
From the SSIS toolbox drag and drop ZS Azure Queue Storage Source on the dataflow designer surface.
SSIS ZS Azure Queue Source – Drag and Drop
Double click Azure Queue Storage Source to configure it.
Select Azure Storage Connection we have created before and select queue from the dropdown and click on preview.
Configure Azure Queue Source
Click on OK button to save configure.
Now, you can use a suitable Destination Adapter from here. In this case, we used Trash Destination.
From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface.
SSIS ZS Trash Destination – Drag and Drop
Now single click on the Azure Queue Storage Source once you see the blue arrow from the source … connect it to Trash Destination.
Double click on ZS Trash Destination to Configure it.
SSIS ZS Trash Destination – Configure
Click on OK button to save Trash Destination configure setting UI.
Execute the package and verify source data in the data viewer.
SSIS ZS Azure Queue Source – Execute the Package
After messages read from Azure Queue, the same message cannot be read again until VisibilityTimeout. By default Queue, visibility timeout is used (-1 means use default) but you can override it on the source UI.

Conclusion

In this blog, we learn How to Read/Write Azure Queue Storage data in SSIS. We used the Azure Queue Destination for write data and Azure Queue Source to read data. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

Azure Queue Source
Azure Queue Destination
Microsoft Azure Queues
Help File: Azure Queue Source, Azure Queue Destination, Trash Destination

The post How to Read/Write Azure Queue Storage data in SSIS appeared first on ZappySys Blog.

How to Read/Write Azure Table Storage data in SSIS

ZappySys — Fri, 02 Aug 2019 13:03:21 +0000

Introduction

In our previous blog, we saw how to read/write Azure Blob Storage Files in SSIS (CSV, JSON, XML Format files). Now in this blog, we will see how to Read/Write Azure Table Storage data in SSIS. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like to perform other operations on Azure Blob Storage Files (e.g. Download, Upload, Create, Delete) then check these articles.

Component Mentioned in this article

	Azure Table Storage Source
	Azure Table Storage Destination

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
Download and install Microsoft Azure Storage Emulator
Download and install Microsoft Azure Storage Explorer
Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Table Storage?

Azure Table storage stores large amounts of structured data. The service is a NoSQL datastore which accepts authenticated calls from inside and outside the Azure cloud. Azure tables are ideal for storing structured, non-relational data. Common uses of Table storage include:

Microsoft Azure Table Storage Introduction

Storing TBs of structured data capable of serving web-scale applications
Storing datasets that don’t require complex joins, foreign keys, or stored procedures and can be denormalized for fast access
Quickly querying data using a clustered index
Accessing data using the OData protocol and LINQ queries with WCF Data Service .NET Libraries

You can use Table storage to store and query huge sets of structured, non-relational data, and your tables will scale as demand increases.

Getting Started

In order to start, we will show several examples. Zappysys includes an SSIS Azure Table Storage Source. Here we are showing you is, how to Read/Write Azure Table Storage data in SSIS.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of the local Azure Storage Emulator.

Setup Azure Storage client tools and Create Table

Once you have downloaded and installed a storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
Azure Storage Emulator Physical Location
If you can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
Command Prompt Screen after Microsoft Azure Storage Emulator Started
Now, you have to download and install Microsoft Azure Storage Explorer and then you can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Explorer Location.
If you don’t have Azure Storage account then it’s easy to get FREE Trial Azure Storage account or use your MSDN to get credit each month which will be more than enough for real testing. If you prefer to avoid all this hassle then Microsoft provides another great way to test Azure Storage functionality is totally offline mode on your local machine. You can just download Azure Storage Emulator and start testing.
Click here to learn more about how to configure Azure Storage Emulator for Testing and Development purposes. In order to connect to Azure Storage Service from SSIS, you will need Storage AccountName and an AccessKey. Ask your SysAdmin or responsible person to provide that information to you. Here are sample Account Name and Access Key(this is just example key which may differ in your case)
```
Account Name: mystorageaccount 
Access Key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
```
For Creating a Tables, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> Emulator – Default Ports(Key) –> Tables. Right-click on Tables and Select Create Table and put your table name you want.
Create Tables in Azure Table Storage Emulator

Write data using ZS Azure Table Storage Destination

In this tutorial, you will learn how to write data into Azure Table Storage. You can use ZS DummyData Source for practice it(In this case it’s from SQL Server Source). So, Let’s start with an example. In this SSIS we will write data into Azure Table Storage using ZS Azure Table Storage Destination.

Once you have created Azure Storage Table and you got valid Account Name and AccessKey (for the local emulator you don’t need this) you may proceed to create a new SSIS package. In BIDS/SSDT create new SSIS package
From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
Drag and Drop SSIS Data Flow Task from SSIS Toolbox
Double click on the DataFlow task to see DataFlow designer surface.
Here, In Visual Studio, drag and drop the OLE DB Source and Azure Table Storage Destination in the design surface and join the components with the blue arrow.
SSIS Azure Table Storage Destination – Drag and Drop
Now, we need two connection, OLE DB Connection, and Azure Storage Connection. Click here to create Azure Storage Connection.

How to create OLE DB Connection

Let’s, Right-click on Connection Managers Panel to Create OLE DB Connection, so you can use Source and Context Menu will appear, Select New OLE DB Connection from the Context Menu.
OLE DB Connection – Create New Connection
Now, click on New Button to create Connection.
OLE DB Connection – Create New Connection
Let’s Configure Connection Manager, just Follow steps one by one as we have created.
OLE DB Connection – Configure Connection
Click on the OK button to save configure Setting UI.

Write data into Azure Storage Table in SSIS

Double click on OLE DB Source for configure it.
Let’s Configure in Connection Manager, just follow the below image steps.
OLE DB Source – Configure
Click OK to Save OLE DB Source Editor UI Settings.
Now double click Azure Table Storage Destination to configure it.
On [Connection Manager] tab select Azure Storage Connection manager.
ZS Azure Table Storage Destination – Connection Manager
Click on [Component Properties] tab and select AccessMode=Table and Pick destination Table from the Tables dropdown list.

If you choose Reload option then target table is first truncated before inserting records so be careful with Reload option.

ZS Azure Table Storage Destination – Component Properties
Click on [Mappings] tab and verify mapping. Drop CustomerID and RecordID mappings and reconnect as shown below. This is just to make sure PrimaryKey columns are mapped.
ZS Azure Table Storage Destination – Column Mappings
Click OK to save settings.
Execute the package and verify the target data.
ZS Azure Table Storage Destination – Execute the Package

ZS Azure Table Storage – Verify Customers Table Data

Read data using ZS Azure Table Storage Source

In this tutorial, you will learn how to read data from Azure Table Storage in SSIS using the ZS Azure Table Storage Source adapter. In this SSIS we will read data with SQL Query and Table Mode. So Let’s begin it.

You may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
SSIS Data Flow Task – Drag and Drop
Double click on the DataFlow task to see DataFlow designer surface.
From the SSIS toolbox drag and drop Azure Table Storage Source on the data flow designer surface.
ZS Azure Table Storage Source – Drag and Drop
Now, we need an Azure Storage connection. Click here to Create Connection.

Extract data with Query Mode

Here, we are getting data using Query Mode with Dynamic expression value.
Lets, create a Variable with correct DataType and Value, use the following image.
SSIS Variables – Create Variable and Store Value
Double click Azure Table Storage Source to configure it.
Select the connection we have created before, set AccessMode to Query. You can use the following SQL Query and select Variable. Click here for more Azure Table Storage Query Examples.
```
SELECT * FROM Customers
WHERE Country = '{{User::varCountry}}'
```
ZS Azure Table Storage Source – Configure Query Mode
Click on the Preview button to see Data Preview and OK button to save configure setting UI.
From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface.
ZS Trash Destination – Drag and Drop
Now single click on the Azure Table Storage Source, once you see the blue arrow from the source … connect it to Trash Destination.
Double click on ZS Trash Destination to Configure it.
ZS Trash Destination – Configure
Click on the OK button to save Trash Destination configure setting UI.
Execute the package and verify source data in the data viewer.
SSIS ZS Azure Table Storage Source – Execute the Package

Extract data with Table Mode

So, Double click Azure Table Storage Source to configure it.
In the Azure Table Storage Source, Select the connection we have created, set AccessMode to Table and Select Table from the Tables dropdown.
SSIS ZS Azure Table Source – Configure Table Mode
Click on the Preview button to see Data Preview and OK button to save configure setting UI.
From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface.
ZS Trash Destination – Drag and Drop
Now single click on the Azure Table Storage Source, once you see the blue arrow from the source … connect it to Trash Destination.
Double click on ZS Trash Destination to Configure it.
ZS Trash Destination – Configure
Click on the OK button to save Trash Destination configure setting UI.
In the last, Execute the package and verify source data in the data viewer.
ZS Azure Table Storage Source – Execute the Package

Conclusion

After all, In this blog, we learned how to write data in Azure Table Storage in SSIS. We used the Azure Table Storage Destination for write and Azure Table Storage Source to read data from Microsoft Azure Table Storage using Query and Table mode. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

There are a few details you should use the following links for more information:

The post How to Read/Write Azure Table Storage data in SSIS appeared first on ZappySys Blog.