Introduction
In our previous blog we saw how to upload data to Amazon S3 now let’s look at how to Copy Amazon Files from one AWS account to another AWS account (Server Side Copy) using SSIS Amazon Storage Task. (Must be version v2.7.9 or higher)
If you are doing file copy within same account then there is no issue. Just use Copy File feature. However if you want to copy between multiple AWS Accounts (e.g. Your Company AWS account to Vendor AWS Account) then you have to perform extra steps.
Prerequisites
Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:- SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from the Microsoft site).
- Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
- Make sure ZappySys SSIS PowerPack is installed (download it, if you haven't already).
- (Optional step). Read this article, if you are planning to deploy packages to a server and schedule their execution later.
Amazon S3 File Copy Scenario
Now let’s think about a scenario where you have to copy some files from Account-A to Account-B. Account-B can be owned by totally different company (may be your Vendor) and you don’t have security control of it.
Before we can copy from Account-A to Account-B we first have to Configure permissions in Account-B. Basically when you use SSIS S3 Task for Copy operation it uses Source account (i.e. Account-A) to push files to destination account (Account-B). So you have Allow write permission for Source Account-A in Destination AWS Bucket (i.e. Account-B).
Find Canonical ID of Source Account
- First login to Account-A and findout Canonical ID (Usually 64-digit long)
- To get Canonical ID click on Your Account Name > My Security Credentials
- When prompted click Continue to Security Credentials option.
- Expand Account Identifier and copy Canonical ID
How to find AWS Account ID and Canonical User ID
Configure Target S3 Bucket Permissions to allow Source Account (Write Access)
Now once you have Source account Canonical User ID, you can proceed to next step.
- Log in to Target account (i.e. Account B). Where you like to copy files.
- Go to S3 Area and Click on the target bucket where you like to copy files
- Go to Security Tab
- Add Source Account Canonical ID and Check Write Permission (List is Optional for most).
Give Write permission in Target AWS S3 Bucket (Enter Amazon Source Account Canonical User ID)
Copy Amazon S3 Files from one Account Bucket to Another
Now once you configure permission on target bucket. We are ready to execute Amazon S3 File Copy Operation in SSIS
- Open SSIS Package in Visual Studio
- Drag ZS Amazon Storage Task from SSIS Toolbox
- Double click it to configure
- Select Copy File Operation from Action Dropdown
- Click New for Source Connection (Enter Source Account Credentials). Click Test and OK to save.
- Now Click Browse to Select Source Files you like to Copy. You can use Wild card pattern for multiple file copy.
Example: mysourcebucket/subfolder/localsubfile*.csv - Click New for Target Connection (Enter Target Account Credentials). Click Test and OK to save.
- Now Click Browse to Select Target Folder Location where you like to Save Copied files.
Example: mytargetbucket/somefolder/ - Click OK to Save
- Execute SSIS Package to test
Copy Amazon S3 Files using SSIS (Copy Files between multiple Amazon Accounts / S3 Buckets)
Conclusion
In this post you have seen how easy it is to copy S3 Files between multiple AWS accounts in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to Automate Amazon S3 related operations in SSIS.
