In our previous blog we saw how update / insert data into SQL Server using SSIS Upsert Destination. In this post we will look at specific example on Data migration from Access to Amazon Redshift using SSIS Upsert Destination (Insert, Update, Delete), along with few other topics such as how to create table using Upsert Destination. how to read all Customers data from Ms Access Table and Merge it in the Amazon Redshift.

We will go through the steps to read data from Access and Load into Amazon Redshift.

In nutshell, this post will focus on how to read access table data in SSIS.

So let’s get started.

Requirements

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, do not forget to install ZappySys SSIS PowerPack
4. Finally, Make sure that Microsoft Access installed.

How to Read MS Access table data and migrate that data in Amazon Redshift table.

Let´s start with an example. In this article we will see Data migration from Access to Amazon Redshift.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. Furthermore, drag and drop the OLE DB Source.
   

   OLE DB Source – Drag and Drop

3. Double click on OLE DB Source for configure it and click on New Connection and configure connection as below to connect access database and click on OK.
   

   OLE DB : Access Connection

4. Now in OLE DB Source Select the mode as Table or View and select Preview to view the access table data.
   

   OLE DB Source Preview

5. Now drag and drop Upsert Destination (Insert, Update, Delete) and create connection with Amazon Redshift Database.
   

   Upsert Destination : Amazon Redshift Connection

6. Now select Action as Sync and check all the checkboxes Insert, Update and Delete from target if not found in source. Select the table and Map all the columns and select the Key field(s) and click on OK.
   

   Upsert Destination Configuration

7. That’s it we are ready to migrate MS access table data into Amazon Redshift Table. Execute the package and it will migrate the data.
   

   Upsert Destination (Insert, Update, Delete)

Bulk Update data in Amazon Redshift

So in previous example we saw bulk update or insert (Upsert) in Redshift Table. Now let’s look at how to update data in target table if record exists.

Here is how you can perform bulk update in Amazon Redshift using Upsert Destination.

1. Double click on Upsert Destination for configure it.
2. Set Action Bulk Update => based on matching records on target. Select Connection and Target Table. Click on Map All to Mappings all columns and check on Only Primary Key columns.
   
3. Thats all, Click on OK to save Upsert Destination settings UI.

Bulk Delete data in Amazon Redshift

Here is how you can bulk delete data in Amazon Redshift.

1. Double click on Upsert Destination for configure it.
2. Set Action Bulk Delete => based on matching records on target. Select Connection and Target Table. Click on Map All to Mappings all columns and check on Only Primary Key columns.
   
3. Thats all, Click on OK to save Upsert Destination settings UI.

Conclusion

In this article, we show how to read MS Access table data and migrate the data in Amazon Redshift table using SSIS. We show how to do connect access MS Access using OLE DB Source. Also, we show how to write Sync Insert, Update and Delete in target if not found in Source Using ZS Upsert Destination. If you liked this article and you want to try, you can download the SSIS PowerPack from here (includes 70+ Components).

References

• ZappySys SSIS installer.
• Help File: Upsert Destination

The post Merge / Upsert data in Amazon Redshift using SSIS appeared first on ZappySys Blog.

In our previous post you learned how to load data into Redshift using SSIS. Now in this post you will learn how to load data into Redshift using Informatica PowerCenter. For PowerCenter we will use ZappyShell Command line for Redshift Data Load. This small powerful command line utility can handle load of several millions or billions records in few minutes. It uses parallel execution engine which takes care most tedious steps of loading data into Amazon Redshift.

Using informatica command task to load data into redshift

Informatica PowerCenter has simple task to execute command lines. you can use this task to execute ZappyShell command line. Perform following steps to load data.

1. Download and install ZappyShell for Amazon Redshift from here
2. Now you ready to data into Amazon Redshift
3. Drag new command task into your Informatica workflow
4. Enter below command to load data from SQL Server to Redshift. If you have ODBC connectivity then use ODBC DSN to read your data.
5. You can use script file approach too where each command line parameter can be nicely wrapped in new line.
   Here is sample command you can try to load data from SQL Server to Redshift. if you have ODBC connectivity then use ODBC DSN to read your data.
   

   c:\zappyshell\aws.exe import Db 
   	--source-driver ADONET_MSSQL 
   	--source-query "select ROW_NUMBER()Over(order by a.CustomerID) Id, a.*,b.*,c.OrderID,c.OrderDate,c.Freight  from customers a,products b,orders c" 
   	--source-archivemethod None 
   	--source-stage-archivemethod Delete 
   	--target-stage-archivemethod Delete 
   	--source-stagepath "c:\redshift\stage" 
   	--target-table "customerdata" 
   	--target-stagepath "bw-rstest/cmdstage" 
   	--target-truncate 
   	--logfile "c:\redshift\log.txt" 
   	--maxrows-perfile 100000 
   	--region us-east-1 
   	--accesskey "AKIA*****************" 
   	--secretkey "lPi+XQ************************"  
   	--source-connstr "Data Source=localhost;Initial Catalog=Northwind;Integrated Security=SSPI;"  
   	--target-connstr "Host=mytestcluster-1.csu********.us-east-1.redshift.amazonaws.com;Port=5439;Database=dev;UserName=masteruser;Password=*********;EnableSsl=true;Timeout=30;CommandTimeout=3600;"

6. For complete help on import command check help file //zappysys.com/onlinehelp/zappyshell/scr/aws/aws-redshift-import.htm

Informatica PowerCenter Redshift Data Load – Amazon Redshift Import Command line tools (COPY, S3, AWS)

ZappyShell Command Line for Redshift

Below are few feature for Command line for Amazon Redshift

◾Import data to AWS Redshift database from files or relational source (e.g. MySQL, Oracle, SQL Server)
◾Import huge amount of data (millions of rows) in few minutes with parallel load techniques
◾Load local flat files to Redshift in just single line command (option to compress data files to *.gz to speed up transfer)
◾Support for Client side encryption using AES 256 Key
◾Load data from any data source (ODBC,ADO.net or OLEDB) using sql query
◾Import compressed data files (*.gz) to redshift
◾Archive files, error reporting, file splitting, many other features

The post Amazon Redshift data load in Informatica PowerCenter appeared first on ZappySys Blog.


Error: An error occurred with the following error message: "ERROR: XX000: Mandatory url is not present in manifest file.".

Solution: To fix this error change target folder location (On Target tab -> Staging Area Panel -> Stage folder) such a way so it includes bucket and some folder name after that too.

Invalid Path: mybucket/
Valid Path: mybucket/somefolder

How to fix Redshift Copy Error for SSIS Redshift Data Transfer task: “ERROR: XX000: Mandatory url is not present in manifest file.”.

The post Fix Redshift Copy ERROR: XX000: Mandatory url is not present in manifest file appeared first on ZappySys Blog.

In this article you will learn how to Setup Amazon Redshift Cluster in few clicks. You will also learn how to set Inbound and Outbound Firewall Rules so you can access Redshift Cluster from outside of AWS Network (e.g. from your corporate network or your home). By default Redshift Cluster cannot be access from outside of your AWS Virtual Network (referred as VPC – Virtual Private Cloud)

Once Redshift Cluster is setup you can follow these steps to Load data into Redshift (Using SSIS Redshift Data Transfer Task or Command line for Redshift)

What is Amazon Redshift

Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. You can start with just a few hundred gigabytes of data and scale to a petabyte or more. This enables you to use your data to acquire new insights for your business and customers.

The first step to create a data warehouse is to launch a set of nodes, called an Amazon Redshift cluster. After you provision your cluster, you can upload your data set and then perform data analysis queries. Regardless of the size of the data set, Amazon Redshift offers fast query performance using the same SQL-based tools and business intelligence applications that you use today.

Setup Amazon Redshift Cluster

Add inbound rule for Redshift Cluster

Automate Redshift Cluster Creation

If you have need to automate Redshift Cluster Creation or any of the following things automatically then check Redshift Cluster management Task

• Automate Amazon Redshift Cluster Create Action in few clicks. You can also add Access Security Rule.
• Automate Amazon Redshift Cluster Delete Action
• Fetch Amazon Redshift Cluster Property to SSIS Variable (e.g. Fetch Cluster Status)
• Fetch all cluster and their properties as DataTable (Use ForEach Loop and iterate through all clusters)
• Automate Redshift Cluster Snapshot Creation
• Automate Redshift Cluster Snapshot Delete Action
• Support for Wait until Cluster operation is done

The post How to setup Amazon Redshift Cluster in few clicks appeared first on ZappySys Blog.

In our previous article we saw how to load data into Redshift using SSIS or load data into Redshift using ZappyShell Redshift Command Line

In this article we will walk through various steps to Extract/UNLOAD Redshift Data into SQL Server using Amazon S3 Storage Task and ExecuteSQL Task for Amazon Redshift. Below is the screenshot of actual SSIS Package to Extract Redshift Data and Load into SQL Server

Extract/Unload Redshift Data using SSIS and Load into SQL Server

Requirements for Extract Redshift Data using SSIS

Before you UNLOAD data from Redshift, you have to make sure few things.

1. Setup your Redshift cluster (Follow these instructions to setup redshift cluster)
2. Load some sample data to Redshift (Red more here: How to load data to Redshift)
3. Make sure you have correct connection settings to connect to Redshift cluster (Host name, Port, UserId, Password, DB name etc). You can get host name from AWS Console.
4. Make sure you have Access to S3 Bucket where files will be dumped from Redshift. You will need AccessKey and SecretKey to fetch files from S3

Step-1: Execute Redshift UNLOAD Command

Very first step would be to unload redshift data as GZip file using ExecuteSQL Task for Amazon Redshift
Below is SQL Command you can use to extract data from Redshift. Notice how we used variable placeholders in SQL Command. These placeholders are replaced at runtime with actual value stored in specified variable.

unload ('select * from (select * from customerdata limit 1000)')
to 's3://bw-rstest/stage/custdata'
credentials 'aws_access_key_id={{User::S3Accesskey}};aws_secret_access_key={{User::S3SecretKey}}'
ALLOWOVERWRITE

Export as GZip files (Compressed files)

If you exporting data as compressed files to save data transfer cost then use GZIP option as below.

NOTE: Make sure there are no spaces before and after AccessKey and SecretKey otherwise you may get error.

unload ('select * from (select * from customerdata limit 1000)')
to 's3://bw-rstest/stage/custdata_file_'
credentials 'aws_access_key_id={{User::S3Accesskey}};aws_secret_access_key={{User::S3SecretKey}}'
ALLOWOVERWRITE
GZIP

Common Errors / Troubleshooting

UNLOAD command issue with Region mismatch (S3 bucket vs Redshift Cluster)

If your S3 bucket is in different region than Redshift cluster then above command may fail with “301 permanent redirect error” in that case you have to change your S3 bucket region. Region can be changed in AWS console (See S3 bucket properties and change location to match region with Redshift cluster region. Both regions must be same.

ERROR: XX000: S3ServiceException:The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.,Status 301,Error PermanentRedirect

UNLOAD command issue with accesskey and secret key

If you specify invalid accesskey or secretkey –or– you have misspelled keywords related to credentials — or — you have spaces before or after accesskey or secret key then you may get following error.

ERROR: XX000: Invalid credentials. Must be of the format: credentials ‘aws_iam_role=…’ or ‘aws_access_key_id=…;aws_secret_access_key=…[;token=…].

Step-2: Download data files from Amazon S3 Bucket to local machine

Once files are exported to S3 bucket we can download then to local machine using Amazon S3 Storage Task

Step-3: Un-compress downloaded files

If you have exported Redshift data as compressed files (using GZIP option) then you can use ZappySys Zip File task to un-compress multiple files.

Or you can write Script to un-compress those files (see below code). You can skip this step if files are not compressed (not used GZIP option in command).

Here is sample C# code to un-compress GZip files

public void Main()
{
	System.IO.DirectoryInfo directorySelected = new System.IO.DirectoryInfo(@"C:\amazon\archive");

	foreach (System.IO.FileInfo fileToDecompress in directorySelected.GetFiles("custdata*_part_*"))
	{
		Decompress(fileToDecompress);
	}

	Dts.TaskResult = (int)ScriptResults.Success;
}
private static void Decompress(System.IO.FileInfo fileToDecompress)
{
	using (System.IO.FileStream originalFileStream = fileToDecompress.OpenRead())
	{
		string currentFileName = fileToDecompress.FullName;
		string newFileName = currentFileName.Remove(currentFileName.Length - fileToDecompress.Extension.Length);

		using (System.IO.FileStream decompressedFileStream = System.IO.File.Create(newFileName))
		{
			using (System.IO.Compression.GZipStream decompressionStream = new System.IO.Compression.GZipStream(originalFileStream, System.IO.Compression.CompressionMode.Decompress))
			{
				decompressionStream.CopyTo(decompressedFileStream);
				//Console.WriteLine("Decompressed: {0}", fileToDecompress.Name);
			}
		}
	}
}

Step-4: Loop through files using ForEachLoop Container

Once files downloaded from S3 bucket we can now loop through files using SSIS ForEach Loop Task and load into SQL Server (One file in each iteration)

Loop through files downloaded from Amazon S3 (Exported using Redshift UNLOAD Command)

Step-5: Data Flow – Load Redshift Data Files to SQL Server

Inside data flow you can use Flat File source and OLEDB Destination for SQL Server. Just map correct File columns to SQL Server fields and you should be good. If needed convert Unicode/Non-unicode columns using Data Conversion Transform (This is not needed if source is DT_STR and target also DT_STR.. or source is DT_WSTR and target is DT_WSTR i.e. Unicode).

Downloads

To download above SSIS Package click on the below links. In order to test below package you first have to download SSIS PowerPack
Download Demo SSIS Package – SSIS 2008
Download Demo SSIS Package – SSIS 2012/2014

Conclusion

amazon Redshift is great way to start your data warehouse projects with very minimum investment in a very simple pay as you go model but loading or unloading data from redshift can be challenging task. Using SSIS PowerPack you can perform Redshift data load or unload in few clicks.

The post Extract / Unload Redshift data into SQL Server using SSIS appeared first on ZappySys Blog.