Redshift Data Transfer Task Archives | ZappySys Blog

How to check Amazon S3 Bucket location (Region)

ZappySys — Tue, 15 Mar 2016 14:17:39 +0000

Problem

Sometimes if you loading data to Redshift via COPY command or using SSIS Redshift Data Transfer Task to move data to redshift then you will have to make sure your S3 Bucket is in the same region as Redshift Cluster. Many people don’t know how to verify that because its hidden in the console. This post will help you to find it easily.

Find Amazon S3 bucket location (AWS Region endpoint)

Below is step-by-step instructions how to find S3 Bucket region (i.e. location)

Open your AWS Console by visiting https://console.aws.amazon.com/
From dashboard click on S3 option (or visit https://console.aws.amazon.com/s3/home )
You will see all buckets in the left side list
Click on desired S3 bucket name
Click on Properties Tab at the top
Now you will see Region for the selected bucket along with many other properties.

Sometimes You may see US Standard as region name that means your bucket is located in US East (N. Virginia)

If you want to know Region Code rather than name check this link : Region Endpoints for Amazon S3

How to find amazon s3 bucket location (AWS region endpoint)

Download/Upload data to S3 bucket via Command line

If you want to automate S3 file download/upload then check this command line tool

ZappyShell Command line tools for Amazon S3

Upload local files to Amazon S3 from command line

List S3 buckets using command line

List S3 file information using command line

List S3 files using command line

Download/Upload data to S3 bucket from SSIS

Load data to Amazon Redshift using S3 Bucket and COPY command

Below video shows how to use S3 bucket to perform bulk data load into Amazon Redshift using SSIS Redshift Data Transfer Task

Conclusion

Amazon S3 provides most common way to store/load data for various AWS related services (e.g. EMR, Redshift). Zappysys is one of the top leading providers who make tools for AWS, NoSQL and other Cloud connectivity providers. Check SSIS PowerPack (For SSIS) or ZappyShell Command line for AWS if you want command line solution. ZappyShell is just single exe file does all heavy lifting for you 🙂

Cheers!

The post How to check Amazon S3 Bucket location (Region) appeared first on ZappySys Blog.

Fix Redshift Copy ERROR: XX000: Mandatory url is not present in manifest file

ZappySys — Fri, 22 Jan 2016 19:56:55 +0000

If you are doing data transfer using SSIS Redshift Data Transfer task
and if you receive following error then most likely you have not specified folder in your target location.

Error: An error occurred with the following error message: "ERROR: XX000: Mandatory url is not present in manifest file.".
Solution: To fix this error change target folder location (On Target tab -> Staging Area Panel -> Stage folder) such a way so it includes bucket and some folder name after that too.

Invalid Path: mybucket/
Valid Path: mybucket/somefolder

How to fix Redshift Copy Error for SSIS Redshift Data Transfer task: “ERROR: XX000: Mandatory url is not present in manifest file.”.

The post Fix Redshift Copy ERROR: XX000: Mandatory url is not present in manifest file appeared first on ZappySys Blog.

How to setup Amazon Redshift Cluster in few clicks

ZappySys — Tue, 17 Nov 2015 14:47:38 +0000

Introduction

In this article you will learn how to Setup Amazon Redshift Cluster in few clicks. You will also learn how to set Inbound and Outbound Firewall Rules so you can access Redshift Cluster from outside of AWS Network (e.g. from your corporate network or your home). By default Redshift Cluster cannot be access from outside of your AWS Virtual Network (referred as VPC – Virtual Private Cloud)

Once Redshift Cluster is setup you can follow these steps to Load data into Redshift (Using SSIS Redshift Data Transfer Task or Command line for Redshift)

What is Amazon Redshift

Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. You can start with just a few hundred gigabytes of data and scale to a petabyte or more. This enables you to use your data to acquire new insights for your business and customers.

The first step to create a data warehouse is to launch a set of nodes, called an Amazon Redshift cluster. After you provision your cluster, you can upload your data set and then perform data analysis queries. Regardless of the size of the data set, Amazon Redshift offers fast query performance using the same SQL-based tools and business intelligence applications that you use today.

Setup Amazon Redshift Cluster

NOTE: Skip this step if you already setup you Redshift Cluster

Login to your AWS Console and Click on Redshift icon. Or click here to land directly to redshift
Click on Launch Cluster
On Cluster Detail Page specify Cluster Identifier, Database Name, Port, Master User and Password. Click Continue to go to next page

Configure Redshift Cluster Identifier, Database Name, Port , UserID and Password
On Node Configuration Page specify Node Type (This is VM Type), Cluster Type and Number of Node. If you are trying under Free Tire then select smallest Node possible (in this case it was dw2.large). Click Continue to go to next page

Configure Redshift Node Type and Cluster Type
On Additional Configuration Page you can pick VPC (virtual private connection), Security group for Cluster and other options for Encryption. For demo purpose select as below screenshot . Click Continue to review your settings and click Create Cluster

Configure Redshift Cluster Encryption, VPC and Additional Detail
Give it few mins while your cluster is being created. After few minutes (5-10 mins) you can go back to same page and review cluster Status and other properties as below. Copy Cluster Endpoint to somewhere because we will need it later.

Check Redshift Cluster Status , Endpoint and Other Properties

Add inbound rule for Redshift Cluster

NOTE: Skip this step if you have already added your IP to inbound exclusion rule.
By default you cannot connect to Amazon Redshift cluster from outside AWS Network (e.g. from your On-Premises Machine). If you wish to connect then you must add inbound exception rule to allow your request to redshift cluster on specific port.

To add create new inbound rule perform following steps

Under Redshift home page click [Security] tab. You may see following Notice depending on which region you are. Click on [Go to the EC2 Console] link or you can direct go to EC2 by clicking Services -> EC2 menu at the top

Configure Security Group and Inbound Filter Firewall Rule to allow Local Connection
On EC2 Security Groups Page select Security group attached with your Redshift Cluster and then in the bottom pane click on Inbound Tab

Security Group Screen – Add or Edit Inbound Firewall Rule to allow Local Connection
On Inbound Tab click Edit option to modify default entry or you can add new Rule. Notice how IP Range is specified.. 0.0.0.0/0 means all IP. If you wish to add range then you have to set something like this… 50.34.234.10/250 .. this will cover 50.34.234.10 to 50.34.234.250 IP range. Make sure your port range covers Port you specified for Redshift cluster.
Click on Add rule if you wish to add new entry else edit as below and click save

Automate Redshift Cluster Creation

If you have need to automate Redshift Cluster Creation or any of the following things automatically then check Redshift Cluster management Task

Automate Amazon Redshift Cluster Create Action in few clicks. You can also add Access Security Rule.
Automate Amazon Redshift Cluster Delete Action
Fetch Amazon Redshift Cluster Property to SSIS Variable (e.g. Fetch Cluster Status)
Fetch all cluster and their properties as DataTable (Use ForEach Loop and iterate through all clusters)
Automate Redshift Cluster Snapshot Creation
Automate Redshift Cluster Snapshot Delete Action
Support for Wait until Cluster operation is done

The post How to setup Amazon Redshift Cluster in few clicks appeared first on ZappySys Blog.