In our previous blog we saw how update / insert data into SQL Server using SSIS Upsert Destination. In this post we will look at specific example on Data migration from Access to Amazon Redshift using SSIS Upsert Destination (Insert, Update, Delete), along with few other topics such as how to create table using Upsert Destination. how to read all Customers data from Ms Access Table and Merge it in the Amazon Redshift.

We will go through the steps to read data from Access and Load into Amazon Redshift.

In nutshell, this post will focus on how to read access table data in SSIS.

So let’s get started.

Requirements

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, do not forget to install ZappySys SSIS PowerPack
4. Finally, Make sure that Microsoft Access installed.

How to Read MS Access table data and migrate that data in Amazon Redshift table.

Let´s start with an example. In this article we will see Data migration from Access to Amazon Redshift.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. Furthermore, drag and drop the OLE DB Source.
   

   OLE DB Source – Drag and Drop

3. Double click on OLE DB Source for configure it and click on New Connection and configure connection as below to connect access database and click on OK.
   

   OLE DB : Access Connection

4. Now in OLE DB Source Select the mode as Table or View and select Preview to view the access table data.
   

   OLE DB Source Preview

5. Now drag and drop Upsert Destination (Insert, Update, Delete) and create connection with Amazon Redshift Database.
   

   Upsert Destination : Amazon Redshift Connection

6. Now select Action as Sync and check all the checkboxes Insert, Update and Delete from target if not found in source. Select the table and Map all the columns and select the Key field(s) and click on OK.
   

   Upsert Destination Configuration

7. That’s it we are ready to migrate MS access table data into Amazon Redshift Table. Execute the package and it will migrate the data.
   

   Upsert Destination (Insert, Update, Delete)

Bulk Update data in Amazon Redshift

So in previous example we saw bulk update or insert (Upsert) in Redshift Table. Now let’s look at how to update data in target table if record exists.

Here is how you can perform bulk update in Amazon Redshift using Upsert Destination.

1. Double click on Upsert Destination for configure it.
2. Set Action Bulk Update => based on matching records on target. Select Connection and Target Table. Click on Map All to Mappings all columns and check on Only Primary Key columns.
   
3. Thats all, Click on OK to save Upsert Destination settings UI.

Bulk Delete data in Amazon Redshift

Here is how you can bulk delete data in Amazon Redshift.

1. Double click on Upsert Destination for configure it.
2. Set Action Bulk Delete => based on matching records on target. Select Connection and Target Table. Click on Map All to Mappings all columns and check on Only Primary Key columns.
   
3. Thats all, Click on OK to save Upsert Destination settings UI.

Conclusion

In this article, we show how to read MS Access table data and migrate the data in Amazon Redshift table using SSIS. We show how to do connect access MS Access using OLE DB Source. Also, we show how to write Sync Insert, Update and Delete in target if not found in Source Using ZS Upsert Destination. If you liked this article and you want to try, you can download the SSIS PowerPack from here (includes 70+ Components).

References

• ZappySys SSIS installer.
• Help File: Upsert Destination

The post Merge / Upsert data in Amazon Redshift using SSIS appeared first on ZappySys Blog.

ZappySys provide high-performance drag and drop connectors for Amazon Redshift Integration. In our previous post we saw how to bulk load SQL data into Redshift using S3 staging technique (COPY command). Now in this post, you will see How to Read / Write Amazon Redshift Data in SSIS Data flow without need for S3 Staging. We will use  ZS Amazon Redshift components (Source / Destination). If you need to Upsert / delete data from Redshift Table then use Upsert Destination (Not discussed in this article).

ZappySys developed many AWS related components but in this article, we will look at 2 Components for Amazon Redshift Integration Scenarios (Read and Write). We will discuss how to use SSIS Amazon Redshift Source and SSIS Amazon Redshift Destination. Amazon Redshift Source connector supports familiar SQL Like query language so any DBA or SQL user can learn it in no time. ZappySys developed a highly sophisticated engine to turn your SQL Query into native Amazon Redshift query. Load query at runtime from the variables or Direct value.

Prerequisite

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, You have obtained Amazon Redshift Credential.
4. Finally, do not forget to install ZappySys SSIS PowerPack.

Component Mention in this article

	Amazon Redshift Source
	Amazon Redshift Destination

What is Amazon Redshift

Before we talk How to Read / Write Amazon Redshift Data in SSIS lets talk what is Amazon Redshift (or sometimes referred to as AWS Redshift). Amazon Redshift is a Cloud-based Data warehouse service. This type of system also referred to as MPP (Massively Parallel Processing). Amazon Redshift uses a highly modified version of PostgreSQL Engine behind the scene. Amazon Redshift provides the advantage of Scale as you go, at a very low cost compared to onsite dedicated hardware/software approach. In this article, we will try to learn How to Read / Write Amazon Redshift Data in SSIS.

Getting Started

In order to start, we will show several examples. ZappySys includes an Amazon Redshift Source and Destination, that will help you How to Read / Write Amazon Redshift Data in SSIS. You can connect to your Amazon Redshift Account by entering credentials.

SSIS Amazon Redshift Source Connector can be used to read data from Amazon Redshift. You can use a simple Table mode or write custom SQL Query to extract desired data. You can use multiple modes (i.e. Table or Query) to read data from the Amazon Redshift Database. Support for data preview and max rows and Dynamic query (using SSIS Variable placeholder e.g. select * from mytable where id='{{User:: MyVar}}’).

SSIS Amazon Redshift Destination can be used to bulk insert a large number of records to Amazon Redshift storage from any data source using SSIS. Amazon Redshift is a cloud-based Data Warehouse service from Amazon AWS. Here, you can bulk Insert, Update and Delete Amazon Redshift data and supporting the batch size.

Click here to see Amazon Redshift related articles.

Write Data using Amazon Redshift Destination in SSIS

Overview

In this tutorial, you will learn how to write data in Amazon Redshift Database. You can use ZS DummyData Source for practice it or select suitable ZS Source Adapter (In this case it’s from SQL Server Source). So, Let’s start with an example. In this SSIS we will write data into Amazon Redshift Database using ZS Amazon Redshift Destination.

How to write data into Amazon Redshift Table

1. In order to connect to the Amazon Redshift Database from SSIS, you will need credentials. Ask your SysAdmin or responsible person to provide that you.
2. For this sample exercise, we will create a table in Redshift.
   

   CREATE TABLE public."tbl_pg_Customers"
   (
       "RecordID" character varying(50) NOT NULL,
       "CustomerID" character varying(50),
       "CustomerName" character varying(50),
       "Address1" character varying(50),
       "City" character varying(50),
       "State" character varying(50),
       "Zip" character varying(50),
       "Country" character varying(50),
       PRIMARY KEY ("RecordID")
   );

3. Once you have created a Table, you may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
4. In your SSIS toolbox, you should see many new Icons starting with “ZS” (i.e. ZappySys). If you don’t see them then check this.
5. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

6. Double click on the Data Flow Task to see Data flow designer surface.
7. Here, In Visual Studio, Drag and Drop the OLE DB Source and Amazon Redshift Destination in the design surface and join the components with the blue arrow.
   

   SSIS OLE DB Source and Amazon Redshift Destination – Drag and Drop

8. Now, we need OLE DB Connection and Amazon Redshift Connection. Click here for Create Amazon Redshift Connection.

How to Create an OLE DB Connection.

1. Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
   

   SSIS OLE DB Connection – Create Connection

2. Now click on the New button to create a connection.
   

   SSIS OLE DB Connection – Create New Connection

3. Let’s configure connection Manager, just follow the steps one as we have created in following the image.
   

   SSIS OLE DB Connection – Configure Connection

4. After all, Click on OK button to save configure setting UI.

Configure SSIS ZS Amazon Redshift Destination

1. Double click on OLE DB Source for configure it.
2. Let’s Configure in Connection Manager, just follow below image steps.
   

   SSIS OLE DB Source – Configure Source Editors

3. Click OK to save OLE DB Source Editor UI settings.
4. Now double click on Amazon Redshift Destination to configure it. In the [Connection Manager] tab Select Amazon Redshift Connection.
   

   SSIS Amazon Redshift Destination – Connection Manager

5. Click on [Component Properties] tab and Pick the Destination Table from Tables dropdown list.
   

   SSIS Amazon Redshift Destination – Component Properties

6. Click on [Column mappings] tab and verify mapping. If the target table is new Table and it’s empty then all input columns will be automatically mapped as shown below. If the table is an existing table with some data then you can manually pick a mapping column by dragging it from source list to target list.
   

   SSIS Amazon Redshift Destination – Column Mappings

7. Execute the package and verify that the data is saved.
   

   SSIS Amazon Redshift Destination – Run or Execute Package

Read Data using Amazon Redshift Source in SSIS

Overview

In this tutorial, you will learn how to read data from Amazon Redshift in SSIS using ZS Amazon Redshift Source adapter. In this SSIS we will read data with SQL Query and Table Mode. So Let’s begin it.

How to Read data from Amazon Redshift Table

1. You may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
2. From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
   

   SSIS Data Flow Task – Drag and Drop

3. Double click on the Data Flow Task to see the Data Flow design surface.
4. From the SSIS Toolbox, Drag and Drop Amazon Redshift Source in the data flow designer surface.
   

   SSIS Amazon Redshift Source – Drag and Drop

5. Now, We need Amazon Redshift Connection. Click here for creating Amazon Redshift Connection.

Configure SSIS ZS Amazon Redshift Source – Query Mode

1. Here, we are getting data using Query Mode with dynamic expression value.
2. Let’s Create Variables, In Visual Studio Right Click on Design Panel, Select Variables.
   

   SSIS Create Variables

3. Make sure correct DataType and Value, use the following image.
   

   SSIS Created Variables with Value

4. Double click on Amazon Redshift Source to configure it.
5. Select the connection we have created before, set AccessMode to Query and use the following SQL Query(You can replace table or column name) and insert Variables we have created before.
   

   SELECT  * FROM "public"."tbl_pg_Orders"
   WHERE "CustomerID" like '%R%' AND "OrderAmount" > '{{User::OrderAmount}}' AND
   "{{User::ColumnName}}" = '{{User::Value}}'

   

   SSIS Amazon Redshift Source – Configure with SQL Query Mode

6. Click on Preview button to see Data Preview and OK button to save configure setting UI.
7. Here, you can use any of our suitable ZS Destination Adapter.
8. From the SSIS toolbox drag and drop Trash Destination on the Data Flow designer surface.
   

   SSIS Trash Destination – Drag and Drop

9. Now single click on the Amazon Redshift Source, once you see the blue arrow from the source…connect it to Trash Destination.
10. Double click on Trash Destination to Configure it.
    

    SSIS Trash Destination – Configure

11. Click on OK button to save Trash Destination configure setting UI.
12. That’s all, just Run or Execute the package and verify data.
    

    SSIS Amazon Redshift Source – Run or Execute the Package

Configure SSIS ZS Amazon Redshift Source – Table Mode

1. Double click on Amazon Redshift Source to configure it.
2. Select the connection we have created before. Set Access Mode to Table and Select Table from table Drop down list.
   

   SSIS Amazon Redshift Source – Configure with Table Mode

3. Click on Preview button to see Data Preview and OK button to save Amazon Redshift Source configure setting UI.
4. From the SSIS toolbox, Drag and Drop Trash Destination on the Data flow designer surface.
   

   SSIS Trash Destination – Drag and Drop

5. Now single click on the Amazon Redshift Source, once you see the blue arrow from a source… connect it to Trash Destination.
6. Double click on ZS Trash Destination to configure it.
   

   SSIS Trash Destination – Configure

7. Click on OK button to save Trash Destination configure setting UI.
8. That’s all, just run or execute your package and see data.
   

   SSIS Amazon Redshift Source – Run or Execute Package

Conclusion

In this blog, we learn How to Read / Write Amazon Redshift Data in SSIS. We used Amazon Redshift Destination for write data and Amazon Redshift Source to read data. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Amazon Redshift Source
• Amazon Redshift Destination
• Document of Amazon Redshift
• Help File:
  • Amazon Redshift Source
  • Amazon Redshift Destination

The post How to Read / Write Amazon Redshift Data in SSIS appeared first on ZappySys Blog.

ZappySys provide high-performance drag and drop connectors for Amazon Redshift Integration. In previous post we saw how to read / write Amazon Redshift data in SSIS. Now in this post, you will see how to Execute SQL query against Amazon Redshift in SSIS using ZS Amazon Redshift ExecuteSQL Task. SSIS Redshift Execute SQL Task can be used to send any SQL queries to Redshift Cluster without needing ODBC driver. You can Issue various SQL commands such as DROP, COPY, UNLOAD, CREATE, etc very easily. Load query at runtime from variable, file or Direct value. You can also issue SQL query to fetch resultset to store in variable and then loop records using this technique.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, You have obtained Amazon Redshift Credential.
4. Finally, do not forget to install ZappySys SSIS PowerPack.

Component Mention in this article

Amazon Redshift ExecuteSQL Task

What is Amazon Redshift

Before we talk Execute SQL query against Amazon Redshift in SSIS lets talk what is Amazon Redshift (or sometimes referred to as AWS Redshift). Amazon Redshift is a Cloud-based Data warehouse service. This type of system also referred to as MPP (Massively Parallel Processing). Amazon Redshift uses a highly modified version of PostGrey SQL Engine behind the scene. Amazon Redshift provides the advantage of Scale as you go, at a very low cost compared to onsite dedicated hardware/software approach. In this article, we will try to learn how to Execute SQL query against Amazon Redshift in SSIS.

Getting Started

In order to start, we will show several examples. ZappySys includes an Amazon Redshift ExecuteSQL Task, that will help you how to Create and Drop table by Execute SQL query against Amazon Redshift in SSIS. You can connect to your Amazon Redshift Account by entering credentials.

Execute SQL query against Amazon Redshift in SSIS

Let’s start with an example, In this tutorial, we will learn how to CREATE and DROP table using SQL query against Amazon Redshift ExecuteSQL server using ZS Amazon Redshift ExecuteSQL Task.

1. In order to connect to the Amazon Redshift Database from SSIS, you will need credentials. Ask your SysAdmin or responsible person to provide that you.
2. Once you finished the first step, Open Visual Studio and Create New SSIS Package Project.
3. In your SSIS toolbox, you should see many new Icons starting with “ZS” (i.e. ZappySys). If you don’t see them then check this.
4. Here, In Visual Studio, drag and drop the Amazon Redshift ExecuteSQL Task in the design surface.
   

   SSIS ZS Amazon Redshift ExecuteSQL Task – Drag and Drop

5. Now we need Amazon Redshift Connection. Click here to Create Amazon Redshift Connection.
6. Double click on Amazon Redshift ExecuteSQL Task to configure it.
7. Select connection we have created before, set Sql Source Type to DirectValue, Result Type to FullResultSet and select object Datatype Variable and Use following SQL Query for Amazon Redshift.
   

   DROP TABLE IF EXISTS CustomerData;
   CREATE TABLE CustomerData(
   	Id bigint NULL,
   	CustomerID VARCHAR(5) NOT NULL,
   	CustomerName VARCHAR(50) NOT NULL,
   	ContactName VARCHAR(40) NOT NULL,
   	CustomerCity VARCHAR(50) NOT NULL
   )

   

   SSIS Amazon Redshift ExecuteSQL Task – Configure

8. Click on OK button to save configure setting UI.
9. That’s all, right-click on the task and run or Execute. This should create a new table in Redshift.
   

   Run or Execute SSIS ZS ExecuteSQL Task Package

Conclusion

After all, In this blog, we learned how to Execute SQL query against Amazon Redshift in SSIS using ZS Amazon Redshift ExecuteSQL Task. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Amazon Redshift Document
• Help File: SSIS ZS Amazon Redshift ExecuteSQL Task

The post Execute SQL query against Amazon Redshift in SSIS appeared first on ZappySys Blog.

Sometimes if you loading data to Redshift via COPY command or using SSIS Redshift Data Transfer Task to move data to redshift then you will have to make sure your S3 Bucket is in the same region as Redshift Cluster. Many people don’t know how to verify that because its hidden in the console. This post will help you to find it easily.

Find Amazon S3 bucket location (AWS Region endpoint)

Below is step-by-step instructions how to find S3 Bucket region (i.e. location)

1. Open your AWS Console by visiting https://console.aws.amazon.com/
2. From dashboard click on S3 option (or visit https://console.aws.amazon.com/s3/home )
3. You will see all buckets in the left side list
4. Click on desired S3 bucket name
5. Click on Properties Tab at the top
6. Now you will see Region for the selected bucket along with many other properties.

Sometimes You may see US Standard as region name that means your bucket is located in US East (N. Virginia)

If you want to know Region Code rather than name check this link : Region Endpoints for Amazon S3

How to find amazon s3 bucket location (AWS region endpoint)

Download/Upload data to S3 bucket via Command line

If you want to automate S3 file download/upload then check this command line tool

ZappyShell Command line tools for Amazon S3

Upload local files to Amazon S3 from command line

List S3 buckets using command line

List S3 file information using command line

List S3 files using command line

Download/Upload data to S3 bucket from SSIS

Load data to Amazon Redshift using S3 Bucket and COPY command

Below video shows how to use S3 bucket to perform bulk data load into Amazon Redshift using SSIS Redshift Data Transfer Task

Conclusion

Amazon S3 provides most common way to store/load data for various AWS related services (e.g. EMR, Redshift). Zappysys is one of the top leading providers who make tools for AWS, NoSQL and other Cloud connectivity providers. Check SSIS PowerPack (For SSIS) or ZappyShell Command line for AWS if you want command line solution. ZappyShell is just single exe file does all heavy lifting for you 🙂

Cheers!

The post How to check Amazon S3 Bucket location (Region) appeared first on ZappySys Blog.

In our previous post you learned how to load data into Redshift using SSIS. Now in this post you will learn how to load data into Redshift using Informatica PowerCenter. For PowerCenter we will use ZappyShell Command line for Redshift Data Load. This small powerful command line utility can handle load of several millions or billions records in few minutes. It uses parallel execution engine which takes care most tedious steps of loading data into Amazon Redshift.

Using informatica command task to load data into redshift

Informatica PowerCenter has simple task to execute command lines. you can use this task to execute ZappyShell command line. Perform following steps to load data.

1. Download and install ZappyShell for Amazon Redshift from here
2. Now you ready to data into Amazon Redshift
3. Drag new command task into your Informatica workflow
4. Enter below command to load data from SQL Server to Redshift. If you have ODBC connectivity then use ODBC DSN to read your data.
5. You can use script file approach too where each command line parameter can be nicely wrapped in new line.
   Here is sample command you can try to load data from SQL Server to Redshift. if you have ODBC connectivity then use ODBC DSN to read your data.
   

   c:\zappyshell\aws.exe import Db 
   	--source-driver ADONET_MSSQL 
   	--source-query "select ROW_NUMBER()Over(order by a.CustomerID) Id, a.*,b.*,c.OrderID,c.OrderDate,c.Freight  from customers a,products b,orders c" 
   	--source-archivemethod None 
   	--source-stage-archivemethod Delete 
   	--target-stage-archivemethod Delete 
   	--source-stagepath "c:\redshift\stage" 
   	--target-table "customerdata" 
   	--target-stagepath "bw-rstest/cmdstage" 
   	--target-truncate 
   	--logfile "c:\redshift\log.txt" 
   	--maxrows-perfile 100000 
   	--region us-east-1 
   	--accesskey "AKIA*****************" 
   	--secretkey "lPi+XQ************************"  
   	--source-connstr "Data Source=localhost;Initial Catalog=Northwind;Integrated Security=SSPI;"  
   	--target-connstr "Host=mytestcluster-1.csu********.us-east-1.redshift.amazonaws.com;Port=5439;Database=dev;UserName=masteruser;Password=*********;EnableSsl=true;Timeout=30;CommandTimeout=3600;"

6. For complete help on import command check help file //zappysys.com/onlinehelp/zappyshell/scr/aws/aws-redshift-import.htm

Informatica PowerCenter Redshift Data Load – Amazon Redshift Import Command line tools (COPY, S3, AWS)

ZappyShell Command Line for Redshift

Below are few feature for Command line for Amazon Redshift

◾Import data to AWS Redshift database from files or relational source (e.g. MySQL, Oracle, SQL Server)
◾Import huge amount of data (millions of rows) in few minutes with parallel load techniques
◾Load local flat files to Redshift in just single line command (option to compress data files to *.gz to speed up transfer)
◾Support for Client side encryption using AES 256 Key
◾Load data from any data source (ODBC,ADO.net or OLEDB) using sql query
◾Import compressed data files (*.gz) to redshift
◾Archive files, error reporting, file splitting, many other features

The post Amazon Redshift data load in Informatica PowerCenter appeared first on ZappySys Blog.


Error: An error occurred with the following error message: "ERROR: XX000: Mandatory url is not present in manifest file.".

Solution: To fix this error change target folder location (On Target tab -> Staging Area Panel -> Stage folder) such a way so it includes bucket and some folder name after that too.

Invalid Path: mybucket/
Valid Path: mybucket/somefolder

How to fix Redshift Copy Error for SSIS Redshift Data Transfer task: “ERROR: XX000: Mandatory url is not present in manifest file.”.

The post Fix Redshift Copy ERROR: XX000: Mandatory url is not present in manifest file appeared first on ZappySys Blog.

In this article you will learn how to Setup Amazon Redshift Cluster in few clicks. You will also learn how to set Inbound and Outbound Firewall Rules so you can access Redshift Cluster from outside of AWS Network (e.g. from your corporate network or your home). By default Redshift Cluster cannot be access from outside of your AWS Virtual Network (referred as VPC – Virtual Private Cloud)

Once Redshift Cluster is setup you can follow these steps to Load data into Redshift (Using SSIS Redshift Data Transfer Task or Command line for Redshift)

What is Amazon Redshift

Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. You can start with just a few hundred gigabytes of data and scale to a petabyte or more. This enables you to use your data to acquire new insights for your business and customers.

The first step to create a data warehouse is to launch a set of nodes, called an Amazon Redshift cluster. After you provision your cluster, you can upload your data set and then perform data analysis queries. Regardless of the size of the data set, Amazon Redshift offers fast query performance using the same SQL-based tools and business intelligence applications that you use today.

Setup Amazon Redshift Cluster

Add inbound rule for Redshift Cluster

Automate Redshift Cluster Creation

If you have need to automate Redshift Cluster Creation or any of the following things automatically then check Redshift Cluster management Task

• Automate Amazon Redshift Cluster Create Action in few clicks. You can also add Access Security Rule.
• Automate Amazon Redshift Cluster Delete Action
• Fetch Amazon Redshift Cluster Property to SSIS Variable (e.g. Fetch Cluster Status)
• Fetch all cluster and their properties as DataTable (Use ForEach Loop and iterate through all clusters)
• Automate Redshift Cluster Snapshot Creation
• Automate Redshift Cluster Snapshot Delete Action
• Support for Wait until Cluster operation is done

The post How to setup Amazon Redshift Cluster in few clicks appeared first on ZappySys Blog.

In our previous article we saw how to load data into Redshift using SSIS or load data into Redshift using ZappyShell Redshift Command Line

In this article we will walk through various steps to Extract/UNLOAD Redshift Data into SQL Server using Amazon S3 Storage Task and ExecuteSQL Task for Amazon Redshift. Below is the screenshot of actual SSIS Package to Extract Redshift Data and Load into SQL Server

Extract/Unload Redshift Data using SSIS and Load into SQL Server

Requirements for Extract Redshift Data using SSIS

Before you UNLOAD data from Redshift, you have to make sure few things.

1. Setup your Redshift cluster (Follow these instructions to setup redshift cluster)
2. Load some sample data to Redshift (Red more here: How to load data to Redshift)
3. Make sure you have correct connection settings to connect to Redshift cluster (Host name, Port, UserId, Password, DB name etc). You can get host name from AWS Console.
4. Make sure you have Access to S3 Bucket where files will be dumped from Redshift. You will need AccessKey and SecretKey to fetch files from S3

Step-1: Execute Redshift UNLOAD Command

Very first step would be to unload redshift data as GZip file using ExecuteSQL Task for Amazon Redshift
Below is SQL Command you can use to extract data from Redshift. Notice how we used variable placeholders in SQL Command. These placeholders are replaced at runtime with actual value stored in specified variable.

unload ('select * from (select * from customerdata limit 1000)')
to 's3://bw-rstest/stage/custdata'
credentials 'aws_access_key_id={{User::S3Accesskey}};aws_secret_access_key={{User::S3SecretKey}}'
ALLOWOVERWRITE

Export as GZip files (Compressed files)

If you exporting data as compressed files to save data transfer cost then use GZIP option as below.

NOTE: Make sure there are no spaces before and after AccessKey and SecretKey otherwise you may get error.

unload ('select * from (select * from customerdata limit 1000)')
to 's3://bw-rstest/stage/custdata_file_'
credentials 'aws_access_key_id={{User::S3Accesskey}};aws_secret_access_key={{User::S3SecretKey}}'
ALLOWOVERWRITE
GZIP

Common Errors / Troubleshooting

UNLOAD command issue with Region mismatch (S3 bucket vs Redshift Cluster)

If your S3 bucket is in different region than Redshift cluster then above command may fail with “301 permanent redirect error” in that case you have to change your S3 bucket region. Region can be changed in AWS console (See S3 bucket properties and change location to match region with Redshift cluster region. Both regions must be same.

ERROR: XX000: S3ServiceException:The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.,Status 301,Error PermanentRedirect

UNLOAD command issue with accesskey and secret key

If you specify invalid accesskey or secretkey –or– you have misspelled keywords related to credentials — or — you have spaces before or after accesskey or secret key then you may get following error.

ERROR: XX000: Invalid credentials. Must be of the format: credentials ‘aws_iam_role=…’ or ‘aws_access_key_id=…;aws_secret_access_key=…[;token=…].

Step-2: Download data files from Amazon S3 Bucket to local machine

Once files are exported to S3 bucket we can download then to local machine using Amazon S3 Storage Task

Step-3: Un-compress downloaded files

If you have exported Redshift data as compressed files (using GZIP option) then you can use ZappySys Zip File task to un-compress multiple files.

Or you can write Script to un-compress those files (see below code). You can skip this step if files are not compressed (not used GZIP option in command).

Here is sample C# code to un-compress GZip files

public void Main()
{
	System.IO.DirectoryInfo directorySelected = new System.IO.DirectoryInfo(@"C:\amazon\archive");

	foreach (System.IO.FileInfo fileToDecompress in directorySelected.GetFiles("custdata*_part_*"))
	{
		Decompress(fileToDecompress);
	}

	Dts.TaskResult = (int)ScriptResults.Success;
}
private static void Decompress(System.IO.FileInfo fileToDecompress)
{
	using (System.IO.FileStream originalFileStream = fileToDecompress.OpenRead())
	{
		string currentFileName = fileToDecompress.FullName;
		string newFileName = currentFileName.Remove(currentFileName.Length - fileToDecompress.Extension.Length);

		using (System.IO.FileStream decompressedFileStream = System.IO.File.Create(newFileName))
		{
			using (System.IO.Compression.GZipStream decompressionStream = new System.IO.Compression.GZipStream(originalFileStream, System.IO.Compression.CompressionMode.Decompress))
			{
				decompressionStream.CopyTo(decompressedFileStream);
				//Console.WriteLine("Decompressed: {0}", fileToDecompress.Name);
			}
		}
	}
}

Step-4: Loop through files using ForEachLoop Container

Once files downloaded from S3 bucket we can now loop through files using SSIS ForEach Loop Task and load into SQL Server (One file in each iteration)

Loop through files downloaded from Amazon S3 (Exported using Redshift UNLOAD Command)

Step-5: Data Flow – Load Redshift Data Files to SQL Server

Inside data flow you can use Flat File source and OLEDB Destination for SQL Server. Just map correct File columns to SQL Server fields and you should be good. If needed convert Unicode/Non-unicode columns using Data Conversion Transform (This is not needed if source is DT_STR and target also DT_STR.. or source is DT_WSTR and target is DT_WSTR i.e. Unicode).

Downloads

To download above SSIS Package click on the below links. In order to test below package you first have to download SSIS PowerPack
Download Demo SSIS Package – SSIS 2008
Download Demo SSIS Package – SSIS 2012/2014

Conclusion

amazon Redshift is great way to start your data warehouse projects with very minimum investment in a very simple pay as you go model but loading or unloading data from redshift can be challenging task. Using SSIS PowerPack you can perform Redshift data load or unload in few clicks.

The post Extract / Unload Redshift data into SQL Server using SSIS appeared first on ZappySys Blog.