In our previous article, we saw SSIS Amazon S3 Storage Task examples. Now let’s look at how to Download the Latest File from Amazon S3 Storage using SSIS. Microsoft SSIS includes the Amazon S3 Storage Task that allows us to upload files to Amazon S3 Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Amazon S3 Storage task included in ZappySys SSIS PowerPack.

In nutshell, this post will focus on how to Download the Latest File from Amazon S3 Storage.

What is Amazon S3 Storage

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements. Amazon S3 is designed for 99.999999999% (11 9’s) of durability, and stores data for millions of applications for companies all around the world.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Amazon S3 Storage Task that will allow you to access files/folders from Amazon S3 Storage to the Local machine, Upload files(s) to Amazon S3 Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to Download the Latest File from Amazon S3 Storage.

Step-By-Step to Download the Latest File from Amazon S3 Storage

Let´s start with an example. In this SSIS Amazon S3 Storage Task File task example, we will download the latest file using Amazon S3 Storage Task.

1. First of All, Drag and Drop Amazon S3 Storage Task from SSIS Toolbox and double click it to edit
   

   Drag and Drop Amazon S3 Storage Task

2. The following options are available in Amazon S3 Storage Task:
   

   Options to upload, download files in SSIS

3. You can also overwrite files and folder, skip all, overwrite if the source is older, overwrite if the file size is different, rename files and more:
   

   Overwrite files in SSIS

4. Let’s select the Download Amazon file(s) to local directory. Now select/add The Source path, Check Recursive Scan(*Optional) to get all the sub-files and subfolders and files, after that add Target Path if you want to, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
   

   SSIS Amazon S3 Storage Task – Download files from Amazon S3 Storage

5. Now go to Advanced Filter & Sorting and configure it like below screen to download the latest file only.
   

   Storage Task configuration to download the latest file only using sorting

6. That’s it now execute the package and it will download the latest file from the Amazon S3 Storage Task to Local File Machine.
7. Sometimes you may need to make advanced filtering on file path, file date, size, etc. for that refer to this article for more details.

Conclusion

To conclude, we can say that working with Amazon S3 Storage is now very simple. In this article, we looked at how to Download the Latest File from Amazon S3 Storage using SSIS. We used Adnavced Filtering & sorting option. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here.

Thanks for your time.

References

• About SSIS PowerPack
• Integration Services (SSIS) Expressions
• Help File: Amazon Storage Task

The post Download the Latest File from Amazon S3 Storage using SSIS appeared first on ZappySys Blog.

In our previous blog we saw how update / insert data into SQL Server using SSIS Upsert Destination. In this post we will look at specific example on Data migration from Access to Amazon Redshift using SSIS Upsert Destination (Insert, Update, Delete), along with few other topics such as how to create table using Upsert Destination. how to read all Customers data from Ms Access Table and Merge it in the Amazon Redshift.

We will go through the steps to read data from Access and Load into Amazon Redshift.

In nutshell, this post will focus on how to read access table data in SSIS.

So let’s get started.

Requirements

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, do not forget to install ZappySys SSIS PowerPack
4. Finally, Make sure that Microsoft Access installed.

How to Read MS Access table data and migrate that data in Amazon Redshift table.

Let´s start with an example. In this article we will see Data migration from Access to Amazon Redshift.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. Furthermore, drag and drop the OLE DB Source.
   

   OLE DB Source – Drag and Drop

3. Double click on OLE DB Source for configure it and click on New Connection and configure connection as below to connect access database and click on OK.
   

   OLE DB : Access Connection

4. Now in OLE DB Source Select the mode as Table or View and select Preview to view the access table data.
   

   OLE DB Source Preview

5. Now drag and drop Upsert Destination (Insert, Update, Delete) and create connection with Amazon Redshift Database.
   

   Upsert Destination : Amazon Redshift Connection

6. Now select Action as Sync and check all the checkboxes Insert, Update and Delete from target if not found in source. Select the table and Map all the columns and select the Key field(s) and click on OK.
   

   Upsert Destination Configuration

7. That’s it we are ready to migrate MS access table data into Amazon Redshift Table. Execute the package and it will migrate the data.
   

   Upsert Destination (Insert, Update, Delete)

Bulk Update data in Amazon Redshift

So in previous example we saw bulk update or insert (Upsert) in Redshift Table. Now let’s look at how to update data in target table if record exists.

Here is how you can perform bulk update in Amazon Redshift using Upsert Destination.

1. Double click on Upsert Destination for configure it.
2. Set Action Bulk Update => based on matching records on target. Select Connection and Target Table. Click on Map All to Mappings all columns and check on Only Primary Key columns.
   
3. Thats all, Click on OK to save Upsert Destination settings UI.

Bulk Delete data in Amazon Redshift

Here is how you can bulk delete data in Amazon Redshift.

1. Double click on Upsert Destination for configure it.
2. Set Action Bulk Delete => based on matching records on target. Select Connection and Target Table. Click on Map All to Mappings all columns and check on Only Primary Key columns.
   
3. Thats all, Click on OK to save Upsert Destination settings UI.

Conclusion

In this article, we show how to read MS Access table data and migrate the data in Amazon Redshift table using SSIS. We show how to do connect access MS Access using OLE DB Source. Also, we show how to write Sync Insert, Update and Delete in target if not found in Source Using ZS Upsert Destination. If you liked this article and you want to try, you can download the SSIS PowerPack from here (includes 70+ Components).

References

• ZappySys SSIS installer.
• Help File: Upsert Destination

The post Merge / Upsert data in Amazon Redshift using SSIS appeared first on ZappySys Blog.

ZappySys provides high-performance drag and drop connectors for Amazon AWS Integration. In this post, we will see How to Read / Write / Delete Amazon SQS Queue data in SSIS using ZS Amazon SQS components. ZappySys developed many AWS related components but in this article, we will look at 2 components for Amazon SQS Queue integration Scenarios (Read and Write). We will discuss how to use SSIS Amazon SQS Source and SSIS Amazon SQS Destination. Amazon SQS Source can be used to read messages from Amazon AWS SQS (Simple Queue Storage) and Amazon SQS Destination can be used to write/delete data from Amazon SQS (Simple Queue Storage) using SSIS.

Prerequisite

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

• First, you will need to have SSIS installed.
• Secondly, make sure to have SSDT.
• Thirdly, you have obtained Amazon SQS Credentials.
• Finally, do not forget to install ZappySys SSIS PowerPack.

Component Mention in this article

	Amazon SQS Source Connector
	Amazon SQS Destination Connector

What is Amazon SQS (Simple Queue Service)

Before we talk How to Read / Write / Delete Amazon SQS Queue data in SSIS lets talk What is Amazon SQS (Simple Queue Service) or sometimes referred to as AWS SQS. Amazon SQS offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues and costs allocation tags. It provides a generic web services API and it can be accessed by any programming language that the AWS SDK supports.

SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

Getting Started

In order to start, we will show several examples. ZappySys includes an Amazon SQS Source and Amazon SQS Destination, that will help you How to Read / Write / Delete Amazon SQS Queue data in SSIS. You can connect to your Amazon SQS account by entering credentials.

Amazon SQS Source can be used to read messages from Amazon AWS SQS (Simple Queue Storage) using SSIS. Read Amazon SQS Messages (AWS SQS) in bulk. Amazon SQS Destination can be used to write/delete data from Amazon SQS (Simple Queue Storage) using SSIS. Write messages to Amazon SQS (AWS SQS) in bulk. Support for Visibility Timeout (Hide messages from others for certain time after you read), data preview and max rows, reading message attributes in separate output columns and adjusting UTC DateTime in local time.

Write Data using Amazon SQS Destination in SSIS

Overview

In this tutorial, you will learn how to write data in Amazon SQS Queue storage. You can use ZS DummyData Source for practice it or select suitable ZS Source Adapters. In this case its from ZS DummyData Source. So let’s start with an example, In this section, you will learn how to write messages into Amazon SQS Queue. For this example, we are assuming that you have already SQS Queue Created.

Create Amazon SQS Connection

1. Firstly, In order to connect to Amazon SQS from SSIS, you will need Access Key and Secret Key (Login to AWS portal and Create Access/Secret Key with SQS service read access). If you are not sure about this ask your SysAdmin or responsible person to provide that to you. Your key will look something like this: (This is just an example key which may differ in your case).
   
   

   AccessKey: AKIAIOSFODNN7EXAMPLE
   SecretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

   Click here to read more about how to get your AccessKey and Secret Key.
2. Secondly, Once you have Account Key (it’s like UserID) and Secret Key (its like password) you may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
3. Right-click in Connection Manager Panel and Select [New Connection] menu item.
   

   SSIS Create AWS Storage Connection

4. Now Select ZS-AWS-STORAGE connection type.
   

   Select ZS-AWS-STORAGE Connection

5. Let’s configure the Amazon SQS connection. Set Storage Service to SQS and Credential Type to Key-based access control. In the General tab enter your Amazon SQS Credentials and Set Default region as per you have.
   

   SSIS AWS Storage Connection – Configure

6. In the last, click on Test Connection and OK button to Save connection configure setting UI.

Write Data into Amazon SQS

1. Once you have created the Amazon SQS Connection successfully, let’s drag and Drop Data Flow Task from SSIS Toolbox on the control flow designer surface.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Double click on the Data Flow Task to see Data Flow Designer surface.
3. Here, from the SSIS Toolbox drag and drop ZS DummyData Source on the data flow designer surface.
   

   SSIS ZS Dummy Data Source – Drag and Drop

4. Double click on Dummy Data Source to configure it.
   

   SSIS ZS Dummy Data Source – Configure

5. Click on OK button to save configure setting UI.
6. From the SSIS toolbox drag and drop Amazon SQS Destination on the Data flow designer surface and single click on the Dummy Data Source, Once you see the blue arrow from the source. connect it to Amazon SQS Destination.
   

   SSIS ZS Amazon SQS Destination

7. Now double click on Amazon SQS Destination to configure it.
8. In the [Connection Manager] tab select SQS connection manager we have created before it.
   

   SSIS Amazon SQS Destination – Configure Connection Manager

9. Click on [Component Properties] tab and Select Target Queue from the dropdown list.
   

   SSIS ZS Amazon SQS Destination – Configure Component Properties

10. Click on [Mappings] tab and verify mapping. Map document column from upstream to the MessageText target column.
    

    SSIS ZS Amazon SQS Destination – Column Mappings

11. In the last, click on the OK button to save configure setting UI.
12. After all, Execute the Package and verify target data by going to your AWS console under SQS service section.
    

    Run or Execute SSIS Package

    

    Verify Amazon SQS Destination

Read Data using Amazon SQS Source in SSIS

Overview

In this tutorial, you will learn how to read data from Amazon SQS storage in SSIS using ZS SSIS Amazon SQS Source Adapter. In this SSIS we will read data. So let’s begin it.

Read Data from Amazon SQS

1. Firstly, create a new SSIS package. In BIDS/SSDT create a new SSIS package.
2. From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
   

   SSIS Data Flow Task – Drag and Drop

3. Double click on the DataFlow task to see DataFlow designer surface.
4. From the SSIS toolbox drag and drop ZS Amazon SQS Source on the dataflow designer surface.
   
5. Double click Amazon SQS Source to configure it.
6. Select Amazon SQS Connection we have created before and select queue from the dropdown list.
   

   SSIS Amazon SQS Source – Configure

7.  Click on Preview button to see Data Preview and Click on  OK button to save settings.
8. From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface or you can use any of our suitable ZS Destination Adapter.
   

   SSIS ZS Trash Destination – Drag and Drop

9. Now single click on the Amazon SQS Source once you see the blue arrow from the source … connect it to Trash Destination.
10. Double click on ZS Trash Destination to Configure it.
    

    SSIS ZS Trash Destination – Configure

11. Click on OK button to save Trash Destination configure setting UI.
12. After all, right-click on the path and Add Data Viewer and Execute the package and verify source data in the data viewer.
    

    Run or Execute The SSIS Package

13. After messages read from SQS Queue, the same message cannot be read again until VisibilityTimeout. By default Queue, visibility timeout is used (-1 means use default) but you can override it on the source UI.

Read all messages from SQS FIFO Queue

If you are using Normal SQS Queue then by setting Max Rows=0 will read all messages from your queue. However reading all messages from FIFO Queue is not that simple. Amazon SQS wont return more messages until you delete previously read Messages from same Message Group (identified by MessageGroupId). Due to this constraint you have to try below technique of Loop to read 10 messages until all messages Read > Processed  > Deleted.

Here is note from Amazon Documentation about maximum 10 messages limitation on FIFO Queue.

When you receive a message with a message group ID, no more messages for the same message group ID are returned unless you delete the message or it becomes visible.

Community Reference:

https://stackoverflow.com/questions/51661834/aws-sqs-fifo-how-to-get-more-than-10-messages-at-a-time

If you want to add fault tolerance then you can set Max Retry for NULL Receive like below (e.g. Setting MaxRetry=3 will try max 3 times if no message is read …. It will try every 2 seconds for 3 times before giving up to read more rows).

Read all messages from Amazon SQS FIFO Queue

Read , Process and Delete Amazon SQS FIFO Queue Message in SSIS Data flow

Delete Message from Queue using Amazon SQS Destination in SSIS

Overview

In this tutorial, you will learn how to Delete Messages from the Queue using the ZS SSIS Amazon SQS Destination Adapter. So let’s begin it.

Delete Message from Queue

1. From the SSIS toolbox drag and drop Amazon SQS Destination on the Data flow designer surface and connect it to Amazon SQS Destination.
2. Now double click on Amazon SQS Destination to configure it.
3. In the [Connection Manager] tab select SQS connection manager we have created before it.
   

   Select Amazon SQS Connection

4. Click on [Component Properties] tab, Set Operation to Delete and Select Target Queue from the dropdown list.
   

   Select Delete Operation and Queue

5. Click on [Column Mappings] tab and verify mapping. Map document column from upstream to the MessageText target column.
   

   Do Columns Mapping

6. In the last, click on the OK button to save configure setting UI.
7. After all, Execute the Package and verify target data by going to your AWS console under SQS service section.

Purge All Messages from SQS Queue (Clear Queue)

There will be a time you like to clear entire queue (i.e. Delete All Messages from Queue). Amazon recently added PurgeQueue API feature so now its possible without doing Delete message technique mentioned in previous section. ZappySys support calling any AWS API using REST API Task along with AWS Provider in OAuth Connection. You can use same technique to call PurgeQueue API like below.

1. Open SSIS Package and Drag ZS REST API Task from Control Flow Toolbox
2. Click on URL mode dropdown and change to URL from Connection.
3. From COnnection Dropdown select NEW ZS-OAuth connection
4. Configure OAuth connection to use Provider=Amazon AWS v4
   1. Enter Access Key and Secret Key and click OK to save connection
5. Enter below URL on REST API Task (Replace Queue ID and name with your own values) – Refer to your AWS Console / SQS Queue Properties to get this URL
   

   https://sqs.ENTER-REGION-HERE.amazonaws.com/ENTER-QUEUE-ID-HERE/ENTER-QUEUE-NAME-HERE/?Action=PurgeQueue

   Here is an example of Sample Queue from ZappySys Demo
   

   https://sqs.us-east-1.amazonaws.com/718883169616/zappysystest.fifo/?Action=PurgeQueue

6. Click ok to Save REST API Task

How to Purge Amazon SQS Queue in SSIS (Clear Queue / Delete All messages using PurgeQueue REST API)

Conclusion

So, In this blog, we learn How to Read / Write / Delete Amazon SQS Queue data in SSIS. We used Amazon SQS Destination for write data and Amazon SQS Source to read data. Additionally, you can download SSIS PowerPack here to try many other scenarios that are not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Amazon SQS Source
• Amazon SQS Destination
• Documents of Amazon SQS
• Help File:
  • Amazon SQS Source
  • Amazon SQS Destination

The post How to Read / Write / Delete Amazon SQS Queue data in SSIS appeared first on ZappySys Blog.

ZappySys provide high-performance drag and drop connectors for Amazon Redshift Integration. In our previous post we saw how to bulk load SQL data into Redshift using S3 staging technique (COPY command). Now in this post, you will see How to Read / Write Amazon Redshift Data in SSIS Data flow without need for S3 Staging. We will use  ZS Amazon Redshift components (Source / Destination). If you need to Upsert / delete data from Redshift Table then use Upsert Destination (Not discussed in this article).

ZappySys developed many AWS related components but in this article, we will look at 2 Components for Amazon Redshift Integration Scenarios (Read and Write). We will discuss how to use SSIS Amazon Redshift Source and SSIS Amazon Redshift Destination. Amazon Redshift Source connector supports familiar SQL Like query language so any DBA or SQL user can learn it in no time. ZappySys developed a highly sophisticated engine to turn your SQL Query into native Amazon Redshift query. Load query at runtime from the variables or Direct value.

Prerequisite

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, You have obtained Amazon Redshift Credential.
4. Finally, do not forget to install ZappySys SSIS PowerPack.

Component Mention in this article

	Amazon Redshift Source
	Amazon Redshift Destination

What is Amazon Redshift

Before we talk How to Read / Write Amazon Redshift Data in SSIS lets talk what is Amazon Redshift (or sometimes referred to as AWS Redshift). Amazon Redshift is a Cloud-based Data warehouse service. This type of system also referred to as MPP (Massively Parallel Processing). Amazon Redshift uses a highly modified version of PostgreSQL Engine behind the scene. Amazon Redshift provides the advantage of Scale as you go, at a very low cost compared to onsite dedicated hardware/software approach. In this article, we will try to learn How to Read / Write Amazon Redshift Data in SSIS.

Getting Started

In order to start, we will show several examples. ZappySys includes an Amazon Redshift Source and Destination, that will help you How to Read / Write Amazon Redshift Data in SSIS. You can connect to your Amazon Redshift Account by entering credentials.

SSIS Amazon Redshift Source Connector can be used to read data from Amazon Redshift. You can use a simple Table mode or write custom SQL Query to extract desired data. You can use multiple modes (i.e. Table or Query) to read data from the Amazon Redshift Database. Support for data preview and max rows and Dynamic query (using SSIS Variable placeholder e.g. select * from mytable where id='{{User:: MyVar}}’).

SSIS Amazon Redshift Destination can be used to bulk insert a large number of records to Amazon Redshift storage from any data source using SSIS. Amazon Redshift is a cloud-based Data Warehouse service from Amazon AWS. Here, you can bulk Insert, Update and Delete Amazon Redshift data and supporting the batch size.

Click here to see Amazon Redshift related articles.

Write Data using Amazon Redshift Destination in SSIS

Overview

In this tutorial, you will learn how to write data in Amazon Redshift Database. You can use ZS DummyData Source for practice it or select suitable ZS Source Adapter (In this case it’s from SQL Server Source). So, Let’s start with an example. In this SSIS we will write data into Amazon Redshift Database using ZS Amazon Redshift Destination.

How to write data into Amazon Redshift Table

1. In order to connect to the Amazon Redshift Database from SSIS, you will need credentials. Ask your SysAdmin or responsible person to provide that you.
2. For this sample exercise, we will create a table in Redshift.
   

   CREATE TABLE public."tbl_pg_Customers"
   (
       "RecordID" character varying(50) NOT NULL,
       "CustomerID" character varying(50),
       "CustomerName" character varying(50),
       "Address1" character varying(50),
       "City" character varying(50),
       "State" character varying(50),
       "Zip" character varying(50),
       "Country" character varying(50),
       PRIMARY KEY ("RecordID")
   );

3. Once you have created a Table, you may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
4. In your SSIS toolbox, you should see many new Icons starting with “ZS” (i.e. ZappySys). If you don’t see them then check this.
5. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

6. Double click on the Data Flow Task to see Data flow designer surface.
7. Here, In Visual Studio, Drag and Drop the OLE DB Source and Amazon Redshift Destination in the design surface and join the components with the blue arrow.
   

   SSIS OLE DB Source and Amazon Redshift Destination – Drag and Drop

8. Now, we need OLE DB Connection and Amazon Redshift Connection. Click here for Create Amazon Redshift Connection.

How to Create an OLE DB Connection.

1. Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
   

   SSIS OLE DB Connection – Create Connection

2. Now click on the New button to create a connection.
   

   SSIS OLE DB Connection – Create New Connection

3. Let’s configure connection Manager, just follow the steps one as we have created in following the image.
   

   SSIS OLE DB Connection – Configure Connection

4. After all, Click on OK button to save configure setting UI.

Configure SSIS ZS Amazon Redshift Destination

1. Double click on OLE DB Source for configure it.
2. Let’s Configure in Connection Manager, just follow below image steps.
   

   SSIS OLE DB Source – Configure Source Editors

3. Click OK to save OLE DB Source Editor UI settings.
4. Now double click on Amazon Redshift Destination to configure it. In the [Connection Manager] tab Select Amazon Redshift Connection.
   

   SSIS Amazon Redshift Destination – Connection Manager

5. Click on [Component Properties] tab and Pick the Destination Table from Tables dropdown list.
   

   SSIS Amazon Redshift Destination – Component Properties

6. Click on [Column mappings] tab and verify mapping. If the target table is new Table and it’s empty then all input columns will be automatically mapped as shown below. If the table is an existing table with some data then you can manually pick a mapping column by dragging it from source list to target list.
   

   SSIS Amazon Redshift Destination – Column Mappings

7. Execute the package and verify that the data is saved.
   

   SSIS Amazon Redshift Destination – Run or Execute Package

Read Data using Amazon Redshift Source in SSIS

Overview

In this tutorial, you will learn how to read data from Amazon Redshift in SSIS using ZS Amazon Redshift Source adapter. In this SSIS we will read data with SQL Query and Table Mode. So Let’s begin it.

How to Read data from Amazon Redshift Table

1. You may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
2. From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
   

   SSIS Data Flow Task – Drag and Drop

3. Double click on the Data Flow Task to see the Data Flow design surface.
4. From the SSIS Toolbox, Drag and Drop Amazon Redshift Source in the data flow designer surface.
   

   SSIS Amazon Redshift Source – Drag and Drop

5. Now, We need Amazon Redshift Connection. Click here for creating Amazon Redshift Connection.

Configure SSIS ZS Amazon Redshift Source – Query Mode

1. Here, we are getting data using Query Mode with dynamic expression value.
2. Let’s Create Variables, In Visual Studio Right Click on Design Panel, Select Variables.
   

   SSIS Create Variables

3. Make sure correct DataType and Value, use the following image.
   

   SSIS Created Variables with Value

4. Double click on Amazon Redshift Source to configure it.
5. Select the connection we have created before, set AccessMode to Query and use the following SQL Query(You can replace table or column name) and insert Variables we have created before.
   

   SELECT  * FROM "public"."tbl_pg_Orders"
   WHERE "CustomerID" like '%R%' AND "OrderAmount" > '{{User::OrderAmount}}' AND
   "{{User::ColumnName}}" = '{{User::Value}}'

   

   SSIS Amazon Redshift Source – Configure with SQL Query Mode

6. Click on Preview button to see Data Preview and OK button to save configure setting UI.
7. Here, you can use any of our suitable ZS Destination Adapter.
8. From the SSIS toolbox drag and drop Trash Destination on the Data Flow designer surface.
   

   SSIS Trash Destination – Drag and Drop

9. Now single click on the Amazon Redshift Source, once you see the blue arrow from the source…connect it to Trash Destination.
10. Double click on Trash Destination to Configure it.
    

    SSIS Trash Destination – Configure

11. Click on OK button to save Trash Destination configure setting UI.
12. That’s all, just Run or Execute the package and verify data.
    

    SSIS Amazon Redshift Source – Run or Execute the Package

Configure SSIS ZS Amazon Redshift Source – Table Mode

1. Double click on Amazon Redshift Source to configure it.
2. Select the connection we have created before. Set Access Mode to Table and Select Table from table Drop down list.
   

   SSIS Amazon Redshift Source – Configure with Table Mode

3. Click on Preview button to see Data Preview and OK button to save Amazon Redshift Source configure setting UI.
4. From the SSIS toolbox, Drag and Drop Trash Destination on the Data flow designer surface.
   

   SSIS Trash Destination – Drag and Drop

5. Now single click on the Amazon Redshift Source, once you see the blue arrow from a source… connect it to Trash Destination.
6. Double click on ZS Trash Destination to configure it.
   

   SSIS Trash Destination – Configure

7. Click on OK button to save Trash Destination configure setting UI.
8. That’s all, just run or execute your package and see data.
   

   SSIS Amazon Redshift Source – Run or Execute Package

Conclusion

In this blog, we learn How to Read / Write Amazon Redshift Data in SSIS. We used Amazon Redshift Destination for write data and Amazon Redshift Source to read data. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Amazon Redshift Source
• Amazon Redshift Destination
• Document of Amazon Redshift
• Help File:
  • Amazon Redshift Source
  • Amazon Redshift Destination

The post How to Read / Write Amazon Redshift Data in SSIS appeared first on ZappySys Blog.

ZappySys provide high-performance drag and drop connectors for Amazon Redshift Integration. In previous post we saw how to read / write Amazon Redshift data in SSIS. Now in this post, you will see how to Execute SQL query against Amazon Redshift in SSIS using ZS Amazon Redshift ExecuteSQL Task. SSIS Redshift Execute SQL Task can be used to send any SQL queries to Redshift Cluster without needing ODBC driver. You can Issue various SQL commands such as DROP, COPY, UNLOAD, CREATE, etc very easily. Load query at runtime from variable, file or Direct value. You can also issue SQL query to fetch resultset to store in variable and then loop records using this technique.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, You have obtained Amazon Redshift Credential.
4. Finally, do not forget to install ZappySys SSIS PowerPack.

Component Mention in this article

Amazon Redshift ExecuteSQL Task

What is Amazon Redshift

Before we talk Execute SQL query against Amazon Redshift in SSIS lets talk what is Amazon Redshift (or sometimes referred to as AWS Redshift). Amazon Redshift is a Cloud-based Data warehouse service. This type of system also referred to as MPP (Massively Parallel Processing). Amazon Redshift uses a highly modified version of PostGrey SQL Engine behind the scene. Amazon Redshift provides the advantage of Scale as you go, at a very low cost compared to onsite dedicated hardware/software approach. In this article, we will try to learn how to Execute SQL query against Amazon Redshift in SSIS.

Getting Started

In order to start, we will show several examples. ZappySys includes an Amazon Redshift ExecuteSQL Task, that will help you how to Create and Drop table by Execute SQL query against Amazon Redshift in SSIS. You can connect to your Amazon Redshift Account by entering credentials.

Execute SQL query against Amazon Redshift in SSIS

Let’s start with an example, In this tutorial, we will learn how to CREATE and DROP table using SQL query against Amazon Redshift ExecuteSQL server using ZS Amazon Redshift ExecuteSQL Task.

1. In order to connect to the Amazon Redshift Database from SSIS, you will need credentials. Ask your SysAdmin or responsible person to provide that you.
2. Once you finished the first step, Open Visual Studio and Create New SSIS Package Project.
3. In your SSIS toolbox, you should see many new Icons starting with “ZS” (i.e. ZappySys). If you don’t see them then check this.
4. Here, In Visual Studio, drag and drop the Amazon Redshift ExecuteSQL Task in the design surface.
   

   SSIS ZS Amazon Redshift ExecuteSQL Task – Drag and Drop

5. Now we need Amazon Redshift Connection. Click here to Create Amazon Redshift Connection.
6. Double click on Amazon Redshift ExecuteSQL Task to configure it.
7. Select connection we have created before, set Sql Source Type to DirectValue, Result Type to FullResultSet and select object Datatype Variable and Use following SQL Query for Amazon Redshift.
   

   DROP TABLE IF EXISTS CustomerData;
   CREATE TABLE CustomerData(
   	Id bigint NULL,
   	CustomerID VARCHAR(5) NOT NULL,
   	CustomerName VARCHAR(50) NOT NULL,
   	ContactName VARCHAR(40) NOT NULL,
   	CustomerCity VARCHAR(50) NOT NULL
   )

   

   SSIS Amazon Redshift ExecuteSQL Task – Configure

8. Click on OK button to save configure setting UI.
9. That’s all, right-click on the task and run or Execute. This should create a new table in Redshift.
   

   Run or Execute SSIS ZS ExecuteSQL Task Package

Conclusion

After all, In this blog, we learned how to Execute SQL query against Amazon Redshift in SSIS using ZS Amazon Redshift ExecuteSQL Task. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Amazon Redshift Document
• Help File: SSIS ZS Amazon Redshift ExecuteSQL Task

The post Execute SQL query against Amazon Redshift in SSIS appeared first on ZappySys Blog.

There might be a case when you have many CSV, JSON or XML files in Amazon S3 bucket and you want them to be imported straight into a SQL Server table. Here come ZappySys ODBC PowerPack and ZappySys Data Gateway (part of ODBC PowerPack) which will enable you to accomplish that. ZappySys ODBC PowerPack includes powerful Amazon S3 CSV, Amazon S3 JSON and Amazon S3 XML drivers that let you connect to an S3 bucket and read the contents from the files in the bucket. Bringing in ZappySys Data Gateway allows doing that right from a SQL Server. Let’s begin and see how to import Amazon S3 files into SQL Server.

These drivers are related to this article:

Amazon S3 Driver (for XML Files)

Amazon S3 Driver (for JSON Files)

Amazon S3 Driver (for CSV Files)

Prerequisites

1. Have an AWS account.
2. Existing Amazon S3 bucket.
3. SQL Server instance installed (can be a SQL Server Express LocalDB instance).
4. SQL Server Management Studio (SSMS) installed.
5. ZappySys ODBC PowerPack installed (must be installed on a Windows machine; can be a different machine than SQL Server is installed on).

Getting started

We will start from a simple example of how to load contacts from a JSON that are located in Amazon S3 bucket, then we will proceed by loading contacts from many XML files that are compressed in ZIP, and finally, we will traverse many folders and subfolders to gather CSVs and load contacts from them.

Step-by-Step – Import JSON file located at Amazon S3 into SQL Server

Open and configure ZappySys Data Gateway

Create an Amazon S3 JSON data source in ZappySys Data Gateway

1. The first thing you will have to do is to create a data source in ZappySys Data Gateway. Just click Add button, give the data source a name, e.g. “MyContactsJSON“, and then select Native – ZappySys Amazon S3 JSON Driver:
   

   Adding a JSON data source in ZappySys Data Gateway to load it from Amazon S3 bucket into SQL Server

2. Then click Edit and add the Data Gateway user you created in Users tab. We will use this user later when adding a Linked Server to the Data Gateway to authenticate:
   

   Adding a user to the newly created Amazon S3 JSON data source

3. Once you do that, then click Edit to configure the data source:
   

   Preparing to configure the Amazon S3 JSON data source to load a JSON from Amazon S3 bucket to SQL Server

4. When a window opens, click Click here to Configure the Connection and enter your Access Key and Secret Key:
   

   Configuring the authentication to Amazon S3 bucket in ZappySys Data Gateway

5. After that, select a JSON file you want to load, and then click Select Filter button to choose data you want to be displayed in SQL Server:
   

   Configuring the JSON data source to load a JSON from Amazon S3 bucket into SQL Server

6. Go to the Preview tab and click the Preview Data button to make sure everything is configured correctly and preview the results:
   

   Previewing JSON data in the data source based on Amazon S3 JSON Driver

Now you are ready to create a Linked Server and connect to the data source you just created. If you used “MyContactsJSON” as the data source name, make sure to use the same name when creating a Linked Server.

Set up a SQL Server Linked Server

Execute the SQL query

Once you created the Linked Server to ZappySys Data Gateway, you are ready to execute the SQL query and load data into SQL Server. Supposedly, you created the Linked Server with the name “GATEWAY”, then open SSMS and execute this query:

SELECT * INTO MyContacts FROM openquery([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyContacts

The results

You should see similar results after you execute the query:

The results of getting JSON data straight from Amazon S3 bucket using ZappySys Amazon S3 JSON driver

Now we are ready to proceed to the next section and import many XML files at once.

Step-by-Step – Import many XML files located at Amazon S3 into SQL Server

Overview

Supposedly, you have many XMLs where each one is zipped and you want to load them all into a SQL Server table:

Zipped XML files located in Amazon S3 bucket to be loaded into SQL Server

The first thing you will have to do is to create a data source, based on ZappySys Amazon S3 XML Driver. Let’s proceed and just do that.

Create an Amazon S3 XML data source in ZappySys Data Gateway

Follow the same steps as in Create an Amazon S3 JSON data source in ZappySys Data Gateway section when adding a new data source, except for these two steps:

1. When adding a data source select Native – ZappySys Amazon S3 XML Driver as Connector Type:
   

   Choosing “ZappySys Amazon S3 XML Driver” to load XMLs to SQL Server

2. Then click Edit and edit the data source similarly like in this window:
   

   Configuring data source based on ZappySys Amazon S3 XML Driver to load XMLs into SQL Server

Now we are ready to set up a Linked Server to this newly created data source.

Set up a SQL Server Linked Server

Follow the very same steps from the section above where we added the Linked Server to a JSON data source, except that change Datasource property of the Linked Server to match the name of the Amazon S3 XML data source you created. Once you do that, you are ready to load XMLs into your SQL Server.

Execute the SQL query

Again, if you created the Linked Server with name “GATEWAY”, execute this SQL query in SSMS:

SELECT * INTO MyContacts FROM openquery([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyContacts

The results

You should see a similar view once you execute the query:

The data of many XMLs loaded from Amazon S3 bucket into SQL Server

Now we are ready to move to the next section and recursively scan CSVs and load them into SQL Server.

Step-by-Step – Import CSV file located at Amazon S3 into SQL Server

Overview

Let’s say you have many CSVs that are located in a folder tree, including subfolders and you want them all in your SQL Server:

Many CSV files located in Amazon S3 bucket folders and subfolders to be loaded into SQL Server

We will follow the same steps to accomplish that as we did for loading JSON and XMLs.

Create an Amazon S3 CSV data source in ZappySys Data Gateway

Follow the steps as in Create an Amazon S3 JSON data source in ZappySys Data Gateway section when adding a new data source, except for these two steps:

1. When adding a CSV data source select Native – ZappySys Amazon S3 CSV Driver as Connector Type:
   

   Choosing “ZappySys Amazon S3 CSV Driver” to load CSVs to SQL Server

2. Then click Edit and edit the data source similarly:
   

   Configuring data source based on ZappySys Amazon S3 CSV Driver to load CSVs into SQL Server

Now we are ready to proceed and add a Linked Server to this data source.

Set up a SQL Server Linked Server

Again, follow the very same steps from the section above where we added the Linked Server to a JSON data source, except that change Datasource property of the Linked Server to match the name of the Amazon S3 CSV data source we created. Once you do that, you are ready to load CSVs into your SQL Server.

Execute the SQL query

Once again, if you created the Linked Server with name “GATEWAY”, execute this SQL query in SSMS:

SELECT * INTO MyContacts FROM openquery([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyContacts

The results

You should see a similar view, once you execute the query:

The results of loading many CSVs from Amazon S3 bucket into SQL Server

Conclusion

We had a goal to import Amazon S3 files into SQL Server from SQL Server itself. We learned how to load JSON, XML and CSV files into SQL Server, which some of them were zipped and were located at different folders and subfolders. It was possible to accomplish that using ODBC PowerPack, ZappySys Data Gateway and ODBC PowerPack drivers – Amazon S3 JSON Driver, Amazon S3 XML Driver, and Amazon S3 CSV Driver. At first, we created data sources in the Data Gateway, then we created Linked Servers in SQL Server and connected to ZappySys Data Gateway, which finally made it possible to load data from Amazon S3 bucket straight from SQL queries in SSMS.

Check Amazon S3 Integration with Other BI Tools (Power BI, Excel, SSRS, MS Access, etc.)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

The post Import Amazon S3 files into SQL Server (CSV/JSON/XML Driver) appeared first on ZappySys Blog.

In this article, we will show SSIS Amazon S3 Storage task examples. Uploading files using Amazon S3 Storage Task or downloading files are common requirements in SSIS. In order to work with Amazon S3 Storage, Microsoft SSIS includes the Amazon S3 Storage Task that allows to upload files to Amazon S3 Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Amazon S3 Storage task included in ZappySys SSIS PowerPack.

Amazon S3 Storage Task can be used to perform various operations with Amazon S3 Storage objects (files and containers) (e.g. Download, Upload, Copy, Move, Rename, Delete, etc). You can also get single property of any Amazon file or get a list of Amazon files as ADO.net Table and which can be easily looped through using ForEachLoop Task. Most of the operations in this task support recursive search using wild card pattern (e.g. Myfile*.txt).

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. You have obtained Amazon S3 account access key / secret key.
4. Finally, do not forget to install ZappySys SSIS PowerPack

What is Amazon S3 Storage

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements. Amazon S3 is designed for 99.999999999% (11 9’s) of durability, and stores data for millions of applications for companies all around the world.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Amazon S3 Storage Task that will allow you to access files / folders from Amazon S3 Storage to the Local machine, Upload files(s) to Amazon S3 Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to upload, download, move and delete files / folders from Amazon S3 Storage.

SSIS Amazon S3 Storage Task Examples

Let´s start with an example. In this SSIS Amazon S3 Storage Task File task example, we will upload, download, move and delete files / folders from Amazon S3 Storage.

1. First of All, Drag and Drop Amazon S3 Storage Task from SSIS Toolbox and double click it to edit
   

   Drag and Drop Amazon S3 Storage Task

2. The following options are available in Amazon S3 Storage Task:
   

   Options to upload, download files in SSIS

3. You can also overwrite files and folder, skip all, overwrite if source is older, overwrite if the file size is different, rename files and more:
   

   Overwrite files in SSIS

4. Let’s select the Download Amazon file(s) to local directory. Now select/add The Source path, Check Recursive Scan to get all the sub files and sub folders and it’s files and add Target Path, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
   

   SSIS Amazon S3 Storage Task – Download files from Amazon S3 Storage

5. That’s it now execute the package and it will download all the files and sub folders from the Amazon S3 Storage Task to Local File Machine.
6. Now Let’s configure the task to upload the local directory files and folder to Amazon S3 Storage like below and execute the package. And also in this example i created the variable and stored target folder path in it and used that variable Target Path Variable.
   

   SSIS Amazon S3 Storage Task – Upload files from Amazon S3 Storage

7. Let’s configure the task to delete Amazon S3 Storage files like below and execute the package.
   

   SSIS Amazon S3 Storage Task – Delete files from Amazon S3 Storage

8. Let’s configure the task to move Amazon S3 Storage files to another directory like below and execute the package.
   

   SSIS Amazon S3 Storage Task – Move files from Amazon S3 Storage

9. That’s it like upper examples you can configure Amazon S3 Storage Task to use it’s other options also.
10. Let’s see other SSIS Amazon S3 Storage Task examples in below sections

Working with advanced filters

In order to understand filters, we will create an example for you. This time we will create SSIS Amazon S3 Storage task example to show how to exclude some files from the downloading process.

1. In addition, it is also possible to add advanced filters to the SSIS Amazon S3 Storage task to exclude or include files and folders. The following example shows how to use the Advanced filters. In the Amazon S3 Storage Task, go to the Advanced Filter page and in Exclude RegX we will write \.exe$ to exclude the file with exe extension to be downloaded. You can use RegX (regular expressions) in the advanced filter:
   

   Advanced filters

Working with wildcards in SSIS Amazon S3 Storage task (Multiple files only)

In order to understand wildcards, the new example will show how to work with Wildcards will help you to work with multiple files or folder with common characteristics.

1. The following example shows how to copy all the files will all the extensions in the source folder. We use the *.*, which means that any file name with any extension will be downloaded:
   

   Working with Wildcards

2. If we want to download all the files with extension txt, use the following Path:
   

   /source/*.txt;

3. Another example would be to download all the files with the following names: file1.txt, file2.txt and file3.txt.  We could use the following Path:
   

   /source/file?.txt

Working with SSIS expressions and variables in Amazon S3 Storage Task

In order to understand expressions, the following example will show how to work with SSIS expressions combined with variables in the SSIS Amazon S3 Storage task. In this example we will delete files in Amazon S3 Storage using variables and expressions.

1.  First, we will first create a variable in SSIS. In this example the file name is MyFile and the Value file1.
   

   Create an SSIS variable

2. As a second step, we will write in path the value \destination\{{User.MyFile}}.txt. This will concatenate the folder destination with the variable MyFile. The task will delete the file1.txt from the Amazon S3 Storage folder.
   

   Using SSIS expressions

3. The next example will show how to create a Amazon S3 Storage file using expressions and variables in SSIS:
   

   Create file

4. Content to write is the content of the file. In Target, go to path and write the following: For more information on placeholders check this post and also check this post.
   

   /destination/{{User::FileName,MM-dd-yyyy}}

5. We are using a variable in SSIS named FileName of type DateTime. We are concatenating the Amazon S3 Storage folder destination with the variable. As you can see, the expressions with ZappySys are very simple.

Example to verify if file exists in SSIS Amazon S3 Storage task

Finally, we will verify if a file in Amazon S3 Storage exists or not using the SSIS Amazon S3 Storage task.

1. In order to start, we will use the SSIS variables, create a variable of Boolean type named FileExists. In Storage Action, select Get Amazon file exist status. In path, select the path of the file to verify if it exists or not.
   

   SSIS Amazon S3 Storage Task – Check File Exist in Amazon S3 Storage

2. Secondly, we will use the ZS Logging Task to check the value of the variable File Exists. This task will show the value of the variable in the log. Drag and drop the task and join it with the ZS Amazon S3 Storage Task:
   

   The SSIS Log task

3. In order to get the value, we will insert it in the Logging Task. To do that, insert the Variable and select the FileExists variable:
   

   Add the variable in the task logging

4. Finally, run the package and verify the Output (you can use View and Output to see the output) and check the value of the Logging Task. If the file exists, the value will be true. If it does not exist, the value will be false.
   

   Verify the variable in Output

How to loop through Amazon S3 Storage files in SSIS

You can also loop through files to perform custom action file by by. Amazon S3 Storage Task support Action to get File List as ADO.net Recordset

To get File list from Amazon S3 Storage Server perform the following steps.

1. From Storage Action select Get Amazon files as ADO.net DataTable
   
   

   Get files as ADO.net DataTable

    

2. Create new Amazon S3 Storage Connection by clicking New next to connection dropdown.
3. Select Source Path (e.g  \myroot\folder\*.txt  )
4. Select or create new SSIS Variable (e.g. varFiles ) which will hold file list result (Must be Object datatype variable).
5. Click OK to close.
6. Create 3 more SSIS variables to hold File Path (String), Name (String) and Size (Int32). We will use in the next step.
7. Now drag SSIS ForEach Loop Container Task from SSIS Toolbox
8. Edit Loop Task and Select Foreach ADO Enumerator from the Enumerator dropdown
9. Select variable (e.g. varFiles)
   
   

   Loop through files in SSIS using ForEach Loop Task (use ADO.net Recordset variable)

10. Click on Variable Mappings Tab and configure like below.
    

    Variable Mappings for ForEach Loop Task – Loop through files in a folder using SSIS

11. Drag ZS Logging Task and place inside Loop and Display variables
12. That’s it

For more information to loop through files on local machine using SSIS check this article.

Conclusion

To conclude, we can say that working with Amazon S3 Storage is now very simple. In this article, we looked at SSIS Amazon S3 Storage Task examples on how to upload, download, delete files and folders. We used variables, wildcards and expressions. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here. We saw different SSIS Amazon S3 Storage Task examples to understand how to use the task in different scenarios.

Thanks for your time.

References

• About SSIS PowerPack
• Integration Services (SSIS) Expressions
• Help File : Amazon Storage Task

The post SSIS Amazon S3 Storage Task examples (Download, Upload, Delete Files / Folders) appeared first on ZappySys Blog.

In our previous blog we saw how to load data into Amazon S3. Now in this blog, we will see How to read Amazon S3 Storage Files in SSIS (CSV, JSON, XML Format files). To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like perform File operations on Amazon S3 Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on how to Read Amazon S3 Storage CSV, JSON and XML Files using respective SSIS Source tasks.

Components Mentioned in this article

	Amazon S3 CSV File Source
	Amazon S3 CSV File Destination
	Amazon S3 XML File Source
	Amazon S3 JSON File Source

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. You have obtained Amazon S3 account access key / secret key.
4. Finally, do not forget to install ZappySys SSIS PowerPack

What is Amazon S3 Storage

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements. Amazon S3 is designed for 99.999999999% (11 9’s) of durability, and stores data for millions of applications for companies all around the world.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Amazon S3 Source for CSV/JSON/XML File that will help you in reading CSV, JSON and XML Files from Amazon S3 to the Local machine, Upload files(s) to Amazon S3 Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to download files from Amazon S3 Storage.

You can connect to your Amazon S3 Account by entering your storage account credentials.

Read Amazon S3 Storage Files in SSIS (CSV, JSON, XML)

Let´s start with an example. In this SSIS Amazon S3 Source for CSV/JSON/XML File task example, we will read CSV/JSON/XML files from Amazon S3 Storage to SQL Server database.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Drag and Drop relevant Amazon S3 Source for CSV/JSON/XML File Task from the SSIS Toolbox.
   

   Add Amazon S3 Source Tasks

3. Create a connection for Amazon S3 Storage Account.
   

   Create Amazon S3 Storage Connection

4. Select the relevant single file to read from Amazon S3 Storage in their relevant source of CSV/JSON/XML File Task.
   

   Select File From Amazon S3 Storage

5. We can also read the multiple files stored in Amazon S3 Storage using wildcard pattern supported e.g. dbo.tblNames*.csv / dbo.tblNames*.json / dbo.tblNames*.xml in relevant source task
   

   Use wildcard pattern .* to read multiple files data

6. We can also read the zip and gzip compressed files also without extracting it in the specific Amazon S3 Source for CSV/JSON/XML File Task.
   

   Reading zip and gzip compressed files (stream mode)

7. Finally, we are ready to load this file(s) data into the SQL Server.

Load Amazon S3 files data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

Above all, in this blog, we learned how to Read Amazon S3 Storage Files in SSIS. We used Amazon S3 Source for CSV file, Amazon S3 Source for JSON file and Amazon S3 Source for XML file to read the file(s) from Amazon S3 Storage and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• SSIS Amazon S3 Source for CSV/JSON/XML File
• Introduction to Amazon S3 Storage Service
• How to Get Your Amazon AWS Access Key & Secret Key

The post Read Amazon S3 Storage Files in SSIS (CSV, JSON, XML) appeared first on ZappySys Blog.

In our previous article we saw how to load data into Redshift using SSIS or load data into Redshift using ZappyShell Redshift Command Line

In this article we will walk through various steps to Extract/UNLOAD Redshift Data into SQL Server using Amazon S3 Storage Task and ExecuteSQL Task for Amazon Redshift. Below is the screenshot of actual SSIS Package to Extract Redshift Data and Load into SQL Server

Extract/Unload Redshift Data using SSIS and Load into SQL Server

Requirements for Extract Redshift Data using SSIS

Before you UNLOAD data from Redshift, you have to make sure few things.

1. Setup your Redshift cluster (Follow these instructions to setup redshift cluster)
2. Load some sample data to Redshift (Red more here: How to load data to Redshift)
3. Make sure you have correct connection settings to connect to Redshift cluster (Host name, Port, UserId, Password, DB name etc). You can get host name from AWS Console.
4. Make sure you have Access to S3 Bucket where files will be dumped from Redshift. You will need AccessKey and SecretKey to fetch files from S3

Step-1: Execute Redshift UNLOAD Command

Very first step would be to unload redshift data as GZip file using ExecuteSQL Task for Amazon Redshift
Below is SQL Command you can use to extract data from Redshift. Notice how we used variable placeholders in SQL Command. These placeholders are replaced at runtime with actual value stored in specified variable.

unload ('select * from (select * from customerdata limit 1000)')
to 's3://bw-rstest/stage/custdata'
credentials 'aws_access_key_id={{User::S3Accesskey}};aws_secret_access_key={{User::S3SecretKey}}'
ALLOWOVERWRITE

Export as GZip files (Compressed files)

If you exporting data as compressed files to save data transfer cost then use GZIP option as below.

NOTE: Make sure there are no spaces before and after AccessKey and SecretKey otherwise you may get error.

unload ('select * from (select * from customerdata limit 1000)')
to 's3://bw-rstest/stage/custdata_file_'
credentials 'aws_access_key_id={{User::S3Accesskey}};aws_secret_access_key={{User::S3SecretKey}}'
ALLOWOVERWRITE
GZIP

Common Errors / Troubleshooting

UNLOAD command issue with Region mismatch (S3 bucket vs Redshift Cluster)

If your S3 bucket is in different region than Redshift cluster then above command may fail with “301 permanent redirect error” in that case you have to change your S3 bucket region. Region can be changed in AWS console (See S3 bucket properties and change location to match region with Redshift cluster region. Both regions must be same.

ERROR: XX000: S3ServiceException:The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.,Status 301,Error PermanentRedirect

UNLOAD command issue with accesskey and secret key

If you specify invalid accesskey or secretkey –or– you have misspelled keywords related to credentials — or — you have spaces before or after accesskey or secret key then you may get following error.

ERROR: XX000: Invalid credentials. Must be of the format: credentials ‘aws_iam_role=…’ or ‘aws_access_key_id=…;aws_secret_access_key=…[;token=…].

Step-2: Download data files from Amazon S3 Bucket to local machine

Once files are exported to S3 bucket we can download then to local machine using Amazon S3 Storage Task

Step-3: Un-compress downloaded files

If you have exported Redshift data as compressed files (using GZIP option) then you can use ZappySys Zip File task to un-compress multiple files.

Or you can write Script to un-compress those files (see below code). You can skip this step if files are not compressed (not used GZIP option in command).

Here is sample C# code to un-compress GZip files

public void Main()
{
	System.IO.DirectoryInfo directorySelected = new System.IO.DirectoryInfo(@"C:\amazon\archive");

	foreach (System.IO.FileInfo fileToDecompress in directorySelected.GetFiles("custdata*_part_*"))
	{
		Decompress(fileToDecompress);
	}

	Dts.TaskResult = (int)ScriptResults.Success;
}
private static void Decompress(System.IO.FileInfo fileToDecompress)
{
	using (System.IO.FileStream originalFileStream = fileToDecompress.OpenRead())
	{
		string currentFileName = fileToDecompress.FullName;
		string newFileName = currentFileName.Remove(currentFileName.Length - fileToDecompress.Extension.Length);

		using (System.IO.FileStream decompressedFileStream = System.IO.File.Create(newFileName))
		{
			using (System.IO.Compression.GZipStream decompressionStream = new System.IO.Compression.GZipStream(originalFileStream, System.IO.Compression.CompressionMode.Decompress))
			{
				decompressionStream.CopyTo(decompressedFileStream);
				//Console.WriteLine("Decompressed: {0}", fileToDecompress.Name);
			}
		}
	}
}

Step-4: Loop through files using ForEachLoop Container

Once files downloaded from S3 bucket we can now loop through files using SSIS ForEach Loop Task and load into SQL Server (One file in each iteration)

Loop through files downloaded from Amazon S3 (Exported using Redshift UNLOAD Command)

Step-5: Data Flow – Load Redshift Data Files to SQL Server

Inside data flow you can use Flat File source and OLEDB Destination for SQL Server. Just map correct File columns to SQL Server fields and you should be good. If needed convert Unicode/Non-unicode columns using Data Conversion Transform (This is not needed if source is DT_STR and target also DT_STR.. or source is DT_WSTR and target is DT_WSTR i.e. Unicode).

Downloads

To download above SSIS Package click on the below links. In order to test below package you first have to download SSIS PowerPack
Download Demo SSIS Package – SSIS 2008
Download Demo SSIS Package – SSIS 2012/2014

Conclusion

amazon Redshift is great way to start your data warehouse projects with very minimum investment in a very simple pay as you go model but loading or unloading data from redshift can be challenging task. Using SSIS PowerPack you can perform Redshift data load or unload in few clicks.

The post Extract / Unload Redshift data into SQL Server using SSIS appeared first on ZappySys Blog.