Sometimes if you loading data to Redshift via COPY command or using SSIS Redshift Data Transfer Task to move data to redshift then you will have to make sure your S3 Bucket is in the same region as Redshift Cluster. Many people don’t know how to verify that because its hidden in the console. This post will help you to find it easily.

Find Amazon S3 bucket location (AWS Region endpoint)

Below is step-by-step instructions how to find S3 Bucket region (i.e. location)

1. Open your AWS Console by visiting https://console.aws.amazon.com/
2. From dashboard click on S3 option (or visit https://console.aws.amazon.com/s3/home )
3. You will see all buckets in the left side list
4. Click on desired S3 bucket name
5. Click on Properties Tab at the top
6. Now you will see Region for the selected bucket along with many other properties.

Sometimes You may see US Standard as region name that means your bucket is located in US East (N. Virginia)

If you want to know Region Code rather than name check this link : Region Endpoints for Amazon S3

How to find amazon s3 bucket location (AWS region endpoint)

Download/Upload data to S3 bucket via Command line

If you want to automate S3 file download/upload then check this command line tool

ZappyShell Command line tools for Amazon S3

Upload local files to Amazon S3 from command line

List S3 buckets using command line

List S3 file information using command line

List S3 files using command line

Download/Upload data to S3 bucket from SSIS

Load data to Amazon Redshift using S3 Bucket and COPY command

Below video shows how to use S3 bucket to perform bulk data load into Amazon Redshift using SSIS Redshift Data Transfer Task

Conclusion

Amazon S3 provides most common way to store/load data for various AWS related services (e.g. EMR, Redshift). Zappysys is one of the top leading providers who make tools for AWS, NoSQL and other Cloud connectivity providers. Check SSIS PowerPack (For SSIS) or ZappyShell Command line for AWS if you want command line solution. ZappyShell is just single exe file does all heavy lifting for you 🙂

Cheers!

The post How to check Amazon S3 Bucket location (Region) appeared first on ZappySys Blog.

In our previous post you learned how to load data into Redshift using SSIS. Now in this post you will learn how to load data into Redshift using Informatica PowerCenter. For PowerCenter we will use ZappyShell Command line for Redshift Data Load. This small powerful command line utility can handle load of several millions or billions records in few minutes. It uses parallel execution engine which takes care most tedious steps of loading data into Amazon Redshift.

Using informatica command task to load data into redshift

Informatica PowerCenter has simple task to execute command lines. you can use this task to execute ZappyShell command line. Perform following steps to load data.

1. Download and install ZappyShell for Amazon Redshift from here
2. Now you ready to data into Amazon Redshift
3. Drag new command task into your Informatica workflow
4. Enter below command to load data from SQL Server to Redshift. If you have ODBC connectivity then use ODBC DSN to read your data.
5. You can use script file approach too where each command line parameter can be nicely wrapped in new line.
   Here is sample command you can try to load data from SQL Server to Redshift. if you have ODBC connectivity then use ODBC DSN to read your data.
   

   c:\zappyshell\aws.exe import Db 
   	--source-driver ADONET_MSSQL 
   	--source-query "select ROW_NUMBER()Over(order by a.CustomerID) Id, a.*,b.*,c.OrderID,c.OrderDate,c.Freight  from customers a,products b,orders c" 
   	--source-archivemethod None 
   	--source-stage-archivemethod Delete 
   	--target-stage-archivemethod Delete 
   	--source-stagepath "c:\redshift\stage" 
   	--target-table "customerdata" 
   	--target-stagepath "bw-rstest/cmdstage" 
   	--target-truncate 
   	--logfile "c:\redshift\log.txt" 
   	--maxrows-perfile 100000 
   	--region us-east-1 
   	--accesskey "AKIA*****************" 
   	--secretkey "lPi+XQ************************"  
   	--source-connstr "Data Source=localhost;Initial Catalog=Northwind;Integrated Security=SSPI;"  
   	--target-connstr "Host=mytestcluster-1.csu********.us-east-1.redshift.amazonaws.com;Port=5439;Database=dev;UserName=masteruser;Password=*********;EnableSsl=true;Timeout=30;CommandTimeout=3600;"

6. For complete help on import command check help file //zappysys.com/onlinehelp/zappyshell/scr/aws/aws-redshift-import.htm

Informatica PowerCenter Redshift Data Load – Amazon Redshift Import Command line tools (COPY, S3, AWS)

ZappyShell Command Line for Redshift

Below are few feature for Command line for Amazon Redshift

◾Import data to AWS Redshift database from files or relational source (e.g. MySQL, Oracle, SQL Server)
◾Import huge amount of data (millions of rows) in few minutes with parallel load techniques
◾Load local flat files to Redshift in just single line command (option to compress data files to *.gz to speed up transfer)
◾Support for Client side encryption using AES 256 Key
◾Load data from any data source (ODBC,ADO.net or OLEDB) using sql query
◾Import compressed data files (*.gz) to redshift
◾Archive files, error reporting, file splitting, many other features

The post Amazon Redshift data load in Informatica PowerCenter appeared first on ZappySys Blog.