In our previous article, we saw SSIS Amazon S3 Storage Task examples. Now let’s look at how to Download the Latest File from Amazon S3 Storage using SSIS. Microsoft SSIS includes the Amazon S3 Storage Task that allows us to upload files to Amazon S3 Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Amazon S3 Storage task included in ZappySys SSIS PowerPack.

In nutshell, this post will focus on how to Download the Latest File from Amazon S3 Storage.

What is Amazon S3 Storage

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements. Amazon S3 is designed for 99.999999999% (11 9’s) of durability, and stores data for millions of applications for companies all around the world.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Amazon S3 Storage Task that will allow you to access files/folders from Amazon S3 Storage to the Local machine, Upload files(s) to Amazon S3 Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to Download the Latest File from Amazon S3 Storage.

Step-By-Step to Download the Latest File from Amazon S3 Storage

Let´s start with an example. In this SSIS Amazon S3 Storage Task File task example, we will download the latest file using Amazon S3 Storage Task.

1. First of All, Drag and Drop Amazon S3 Storage Task from SSIS Toolbox and double click it to edit
   

   Drag and Drop Amazon S3 Storage Task

2. The following options are available in Amazon S3 Storage Task:
   

   Options to upload, download files in SSIS

3. You can also overwrite files and folder, skip all, overwrite if the source is older, overwrite if the file size is different, rename files and more:
   

   Overwrite files in SSIS

4. Let’s select the Download Amazon file(s) to local directory. Now select/add The Source path, Check Recursive Scan(*Optional) to get all the sub-files and subfolders and files, after that add Target Path if you want to, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
   

   SSIS Amazon S3 Storage Task – Download files from Amazon S3 Storage

5. Now go to Advanced Filter & Sorting and configure it like below screen to download the latest file only.
   

   Storage Task configuration to download the latest file only using sorting

6. That’s it now execute the package and it will download the latest file from the Amazon S3 Storage Task to Local File Machine.
7. Sometimes you may need to make advanced filtering on file path, file date, size, etc. for that refer to this article for more details.

Conclusion

To conclude, we can say that working with Amazon S3 Storage is now very simple. In this article, we looked at how to Download the Latest File from Amazon S3 Storage using SSIS. We used Adnavced Filtering & sorting option. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here.

Thanks for your time.

References

• About SSIS PowerPack
• Integration Services (SSIS) Expressions
• Help File: Amazon Storage Task

The post Download the Latest File from Amazon S3 Storage using SSIS appeared first on ZappySys Blog.

In this article, we will show SSIS Amazon S3 Storage task examples. Uploading files using Amazon S3 Storage Task or downloading files are common requirements in SSIS. In order to work with Amazon S3 Storage, Microsoft SSIS includes the Amazon S3 Storage Task that allows to upload files to Amazon S3 Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Amazon S3 Storage task included in ZappySys SSIS PowerPack.

Amazon S3 Storage Task can be used to perform various operations with Amazon S3 Storage objects (files and containers) (e.g. Download, Upload, Copy, Move, Rename, Delete, etc). You can also get single property of any Amazon file or get a list of Amazon files as ADO.net Table and which can be easily looped through using ForEachLoop Task. Most of the operations in this task support recursive search using wild card pattern (e.g. Myfile*.txt).

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. You have obtained Amazon S3 account access key / secret key.
4. Finally, do not forget to install ZappySys SSIS PowerPack

What is Amazon S3 Storage

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements. Amazon S3 is designed for 99.999999999% (11 9’s) of durability, and stores data for millions of applications for companies all around the world.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Amazon S3 Storage Task that will allow you to access files / folders from Amazon S3 Storage to the Local machine, Upload files(s) to Amazon S3 Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to upload, download, move and delete files / folders from Amazon S3 Storage.

SSIS Amazon S3 Storage Task Examples

Let´s start with an example. In this SSIS Amazon S3 Storage Task File task example, we will upload, download, move and delete files / folders from Amazon S3 Storage.

1. First of All, Drag and Drop Amazon S3 Storage Task from SSIS Toolbox and double click it to edit
   

   Drag and Drop Amazon S3 Storage Task

2. The following options are available in Amazon S3 Storage Task:
   

   Options to upload, download files in SSIS

3. You can also overwrite files and folder, skip all, overwrite if source is older, overwrite if the file size is different, rename files and more:
   

   Overwrite files in SSIS

4. Let’s select the Download Amazon file(s) to local directory. Now select/add The Source path, Check Recursive Scan to get all the sub files and sub folders and it’s files and add Target Path, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
   

   SSIS Amazon S3 Storage Task – Download files from Amazon S3 Storage

5. That’s it now execute the package and it will download all the files and sub folders from the Amazon S3 Storage Task to Local File Machine.
6. Now Let’s configure the task to upload the local directory files and folder to Amazon S3 Storage like below and execute the package. And also in this example i created the variable and stored target folder path in it and used that variable Target Path Variable.
   

   SSIS Amazon S3 Storage Task – Upload files from Amazon S3 Storage

7. Let’s configure the task to delete Amazon S3 Storage files like below and execute the package.
   

   SSIS Amazon S3 Storage Task – Delete files from Amazon S3 Storage

8. Let’s configure the task to move Amazon S3 Storage files to another directory like below and execute the package.
   

   SSIS Amazon S3 Storage Task – Move files from Amazon S3 Storage

9. That’s it like upper examples you can configure Amazon S3 Storage Task to use it’s other options also.
10. Let’s see other SSIS Amazon S3 Storage Task examples in below sections

Working with advanced filters

In order to understand filters, we will create an example for you. This time we will create SSIS Amazon S3 Storage task example to show how to exclude some files from the downloading process.

1. In addition, it is also possible to add advanced filters to the SSIS Amazon S3 Storage task to exclude or include files and folders. The following example shows how to use the Advanced filters. In the Amazon S3 Storage Task, go to the Advanced Filter page and in Exclude RegX we will write \.exe$ to exclude the file with exe extension to be downloaded. You can use RegX (regular expressions) in the advanced filter:
   

   Advanced filters

Working with wildcards in SSIS Amazon S3 Storage task (Multiple files only)

In order to understand wildcards, the new example will show how to work with Wildcards will help you to work with multiple files or folder with common characteristics.

1. The following example shows how to copy all the files will all the extensions in the source folder. We use the *.*, which means that any file name with any extension will be downloaded:
   

   Working with Wildcards

2. If we want to download all the files with extension txt, use the following Path:
   

   /source/*.txt;

3. Another example would be to download all the files with the following names: file1.txt, file2.txt and file3.txt.  We could use the following Path:
   

   /source/file?.txt

Working with SSIS expressions and variables in Amazon S3 Storage Task

In order to understand expressions, the following example will show how to work with SSIS expressions combined with variables in the SSIS Amazon S3 Storage task. In this example we will delete files in Amazon S3 Storage using variables and expressions.

1.  First, we will first create a variable in SSIS. In this example the file name is MyFile and the Value file1.
   

   Create an SSIS variable

2. As a second step, we will write in path the value \destination\{{User.MyFile}}.txt. This will concatenate the folder destination with the variable MyFile. The task will delete the file1.txt from the Amazon S3 Storage folder.
   

   Using SSIS expressions

3. The next example will show how to create a Amazon S3 Storage file using expressions and variables in SSIS:
   

   Create file

4. Content to write is the content of the file. In Target, go to path and write the following: For more information on placeholders check this post and also check this post.
   

   /destination/{{User::FileName,MM-dd-yyyy}}

5. We are using a variable in SSIS named FileName of type DateTime. We are concatenating the Amazon S3 Storage folder destination with the variable. As you can see, the expressions with ZappySys are very simple.

Example to verify if file exists in SSIS Amazon S3 Storage task

Finally, we will verify if a file in Amazon S3 Storage exists or not using the SSIS Amazon S3 Storage task.

1. In order to start, we will use the SSIS variables, create a variable of Boolean type named FileExists. In Storage Action, select Get Amazon file exist status. In path, select the path of the file to verify if it exists or not.
   

   SSIS Amazon S3 Storage Task – Check File Exist in Amazon S3 Storage

2. Secondly, we will use the ZS Logging Task to check the value of the variable File Exists. This task will show the value of the variable in the log. Drag and drop the task and join it with the ZS Amazon S3 Storage Task:
   

   The SSIS Log task

3. In order to get the value, we will insert it in the Logging Task. To do that, insert the Variable and select the FileExists variable:
   

   Add the variable in the task logging

4. Finally, run the package and verify the Output (you can use View and Output to see the output) and check the value of the Logging Task. If the file exists, the value will be true. If it does not exist, the value will be false.
   

   Verify the variable in Output

How to loop through Amazon S3 Storage files in SSIS

You can also loop through files to perform custom action file by by. Amazon S3 Storage Task support Action to get File List as ADO.net Recordset

To get File list from Amazon S3 Storage Server perform the following steps.

1. From Storage Action select Get Amazon files as ADO.net DataTable
   
   

   Get files as ADO.net DataTable

    

2. Create new Amazon S3 Storage Connection by clicking New next to connection dropdown.
3. Select Source Path (e.g  \myroot\folder\*.txt  )
4. Select or create new SSIS Variable (e.g. varFiles ) which will hold file list result (Must be Object datatype variable).
5. Click OK to close.
6. Create 3 more SSIS variables to hold File Path (String), Name (String) and Size (Int32). We will use in the next step.
7. Now drag SSIS ForEach Loop Container Task from SSIS Toolbox
8. Edit Loop Task and Select Foreach ADO Enumerator from the Enumerator dropdown
9. Select variable (e.g. varFiles)
   
   

   Loop through files in SSIS using ForEach Loop Task (use ADO.net Recordset variable)

10. Click on Variable Mappings Tab and configure like below.
    

    Variable Mappings for ForEach Loop Task – Loop through files in a folder using SSIS

11. Drag ZS Logging Task and place inside Loop and Display variables
12. That’s it

For more information to loop through files on local machine using SSIS check this article.

Conclusion

To conclude, we can say that working with Amazon S3 Storage is now very simple. In this article, we looked at SSIS Amazon S3 Storage Task examples on how to upload, download, delete files and folders. We used variables, wildcards and expressions. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here. We saw different SSIS Amazon S3 Storage Task examples to understand how to use the task in different scenarios.

Thanks for your time.

References

• About SSIS PowerPack
• Integration Services (SSIS) Expressions
• Help File : Amazon Storage Task

The post SSIS Amazon S3 Storage Task examples (Download, Upload, Delete Files / Folders) appeared first on ZappySys Blog.

In this post we will see how to set Metadata key/value in Azure Blob Storage Task and Amazon Storage Task (S3 Task). We will also see how to set them dynamically at runtime (Use SSIS Expression).

Set Amazon S3 Task Metadata (Key / Value Pairs)

Here is how you can set Metadata in Amazon S3 Task.

Setting Amazon S3 Metadata (Key/Value Pairs) in SSIS Amazon Storage Task

Set Azure Blob Storage Task Metadata (Key / Value Pairs)

Just like S3 Task you can use same UI in Azure blob storage task too.

Set Metadata Key/Value Dynamically (SSIS Expression)

So far you have set metadata using static value. What if you have to set Dynamically at runtime?

Basically Metadata is passed as XML Format in the task like below.

<?xml version="1.0" encoding="utf-8"?>
<ArrayOfStringKeyValue xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<StringKeyValue>
		<Key>x-amz-key1</Key>
		<Value>AAAAAAA</Value>
	</StringKeyValue>
	<StringKeyValue>
		<Key>x-amz-key2</Key>
		<Value>BBBBBBB</Value>
	</StringKeyValue>	
</ArrayOfStringKeyValue>

Whate we will do is take above format and make it dynamic using SSIS Expression.

Perform the following steps for that.

1. Select your Amazon Storage Task or Azure Storage Task and right click > Select Properties.
2. In the Properties Grid check Property name MetadataKeyValue. Copy XML Value from there (We will use it in expression)
3. Now double click Amazon Storage Task or Azure Storage Task
4. Click on Expression Button
5. Select MetadataKeyValue property and for Value enter below expression… Use previously copied XML text  and change Value part with your own SSIS variable. In below example we are passing Just one Metadata Key/Value pair ( Name=x-amz-meta-filename  , Value=Coming from @[User::MyFileNameVariable] )
   Note that we have escaped double quote with

   \"

     and replaced static value with

   <Value>" + @[User::MyFileNameVariable] + "</Value>

    .
   

   "<?xml version=\"1.0\" encoding=\"utf-8\"?>"
   + "<ArrayOfStringKeyValue xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\">"
   + "	<StringKeyValue>"
   + "		<Key>x-amz-meta-file-name</Key>"
   + "		<Value>" + @[User::MyFileNameVariable] + "</Value>"
   + "	</StringKeyValue>"
   + "	<StringKeyValue>"
   + "		<Key>x-amz-meta-something</Key>"
   + "		<Value>BBBBBBB</Value>"
   + "	</StringKeyValue>"	
   + "</ArrayOfStringKeyValue>"

   

   Set Amazon S3 File Metadata Key/Value dynamically

6.  Click OK and that’s it.

The post Setting Metadata key / value in SSIS Amazon S3 / Azure Storage Task appeared first on ZappySys Blog.

In our previous blog we saw how to upload data to Amazon S3 now let’s look at how to Copy Amazon Files from one AWS account to another AWS account (Server Side Copy) using SSIS Amazon Storage Task. (Must be version v2.7.9 or higher)

If you are doing file copy within same account then there is no issue. Just use Copy File feature. However if you want to copy between multiple AWS Accounts (e.g. Your Company AWS account to Vendor AWS Account) then you have to perform extra steps.

Amazon S3 File Copy Scenario

Now let’s think about a scenario where you have to copy some files from Account-A to Account-B. Account-B can be owned by totally different company (may be your Vendor) and you don’t have security control of it.

Before we can copy from Account-A to Account-B we first have to Configure permissions in Account-B. Basically when you use SSIS S3 Task for Copy operation it uses Source account (i.e. Account-A) to push files to destination account (Account-B). So you have Allow write permission for Source Account-A in Destination AWS Bucket (i.e. Account-B).

Find Canonical ID of Source Account

1. First login to Account-A and findout Canonical ID (Usually 64-digit long)
2. To get Canonical ID click on Your Account Name > My Security Credentials
3. When prompted click Continue to Security Credentials option.
4. Expand Account Identifier and copy Canonical ID
   

   How to find AWS Account ID and Canonical User ID

Configure Target S3 Bucket Permissions to allow Source Account (Write Access)

Now once you have Source account Canonical User ID, you can proceed to next step.

1. Log in to Target account (i.e. Account B). Where you like to copy files.
2. Go to S3 Area and Click on the target bucket where you like to copy files
3. Go to Security Tab
4. Add Source Account Canonical ID and Check Write Permission (List is Optional for most).
   

   Give Write permission in Target AWS S3 Bucket (Enter Amazon Source Account Canonical User ID)

Copy Amazon S3 Files from one Account Bucket to Another

Now once you configure permission on target bucket. We are ready to execute Amazon S3 File Copy Operation in SSIS

1. Open SSIS Package in Visual Studio
2. Drag ZS Amazon Storage Task from SSIS Toolbox
3. Double click it to configure
4. Select Copy File Operation from Action Dropdown
5. Click New for Source Connection (Enter Source Account Credentials). Click Test and OK to save.
6. Now Click Browse to Select Source Files you like to Copy. You can use Wild card pattern for multiple file copy.
   Example:  mysourcebucket/subfolder/localsubfile*.csv
7. Click New for Target Connection (Enter Target Account Credentials). Click Test and OK to save.
8. Now Click Browse to Select Target Folder Location where you like to Save Copied files.
   Example:  mytargetbucket/somefolder/
9. Click OK to Save
10. Execute SSIS Package to test

Copy Amazon S3 Files using SSIS (Copy Files between multiple Amazon Accounts / S3 Buckets)

Conclusion

In this post you have seen how easy it is to copy S3 Files between multiple AWS accounts in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to Automate Amazon S3 related operations in SSIS.

The post Copy Amazon S3 Files in SSIS to different AWS Account appeared first on ZappySys Blog.

In our previous posts we looked at how to use SSIS Amazon Storage Task (Amazon S3 Task) to automate some Amazon S3 related tasks in SSIS. In this blog post will continue discovering more use cases. Let’s learn how to delete Amazon S3 File, delete Amazon S3 Folder and delete Amazon S3 Bucket.

Deleting S3 Files are straight forward using SSIS Amazon Storage Task (Amazon S3 Task but deleting Bucket or Folders require some additional checks and steps, so let’s look at that in depth.

Requirements

Before we look into Step-By-Step section to call Amazon S3 related operations in SSIS let’s make sure you met the following requirements.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have obtained Amazon S3 account access key / secret key.

Configure SSIS Amazon S3 Connection

Before we perform any S3 related action, you must configure Amazon Storage Connection. This connection manager support connectivity to not only S3 Storage but also other AWS Storage such as DynamoDB, Redshift

There are two ways you can create Amazon Storage Connection Manager in SSIS designer. You can right click in Connection Manager Panel and create new ZS-AMAZON-STORAGE connection type or use Task UI and click New connection.

For now, we will use easier method which is Create S3 Connection from S3 Storage Task UI.

1. Open SSIS Package
2. Drag ZS Amazon S3 Storage Task from the toolbox
3. Double click it to edit. You can now click on New Connection like below.
   

   Create New Amazon S3 Connection from Task UI

4. On the connection Manager UI. Select S3 as service type. Enter your Access Key, Secret Key and Select Region. We also support IAM Role for EC2 instances (For this package must run inside EC2 VM). Check this article.
   

   Configure SSIS Amazon S3 Connection – Select Region, Enter Access Key / Secret Key

5. Click Test connection to confirm its working. Click OK to Save Connection Manager UI.
6. Back to Task UI click OK to close save (We will configure task in later sections.

Using SSIS Amazon S3 Task for common actions

Before we deep dive in delete scenarios let’s first look at what is  SSIS Amazon Storage Task (Amazon S3 Task). This task is developed to achieve many S3 related operations. See below screenshot to get idea about many actions you can perform using just one task. In this post we will focus only on delete actions (for file, folder and bucket). In other posts we will cover different operations in depth.

SSIS Amazon S3 Task Operations – Upload, Download, Delete, Create, List, Rename Files, Folders, Buckets

Delete Amazon S3 Files

Once you have configured SSIS S3 Connection Manager, we can perform different S3 related operations using many different Amazon S3 related tasks / components. For now let’s just look at how to delete files from S3 bucket.

Delete Multiple S3 files – Using Simple Pattern Search (wildcard)

Here is how to delete amazon s3 file / files using wildcard pattern search. Using * or ? allowed. (Example  SalesOrder_*.??? )

Delete Amazon S3 File (Multiple) using Wildcard pattern

Delete Multiple S3 files – Using Advanced Search – Regex / SQL Expression

Sometimes it’s necessary to filter file based on advanced attributes such as file Age (How old), File Size, Multiple File Patterns (e.g. Delete *.zip, *.gz, *.rar in one action)

You can use Advanced Tab to filter further. First supply primary filter on Source Tab (e.g. mybucket/*.* ) and then supply your advanced filter (e.g. you can exclude *.exe and *.msi file from your delete list as below screenshot, also only delete files between 1000 and 50000 bytes in file size).

Advanced Search in SSIS Amazon S3 Task – Use Regular Expression Patterns, SQL like expression for Sort / Filter

//////////////////////////////
Examples:
//////////////////////////////
AgeInDays < 5  //Get files created in last 5 days
LastEditInDays > 3 //Get files modified before 3 days
AgeInDays > 10 AND LastEditInDays < 3  //Get files created before 10 days and modified in last 3 days
Size < 1024  //Get files where File size is less than 1024 bytes
CreationDate BETWEEN #2015-01-01# AND #2015-01-01#   
Name IN ('File1.txt', 'Download.msi')
FullPath LIKE 'mybucket\somefolder\month_*')
Extension IN ('.txt','.csv')
LastModifiedDate > #2013-01-01T16:00:00#
IsReadOnly = True
/////////////////////////////////////////
//Supported Columns for expression:
/////////////////////////////////////////
Name, FullPath, Extension, FolderPath, AgeInDays, LastEditInDays, Size, CreationDate, CreationDateUtc, LastModifiedDate, LastModifiedDateUtc, IsReadOnly

Delete Amazon S3 files older than X days

If you like to delete files older than certain days then use below technique.

1. First Specify primary search criteria on Source tab
2. Go to Advanced Filter Tab and specify Following Pattern in Where Textbox (For example delete Files older than 10 days )
   

   AgeInDays > 10

Delete Amazon S3 Bucket

Now let’s look at how to delete Amazon S3 Bucket including all its content. There is an option to delete Bucket but it wont delete files and empty folders. So you have to first clear all its content before calling Delete bucket action.

Here is high level steps to delete bucket in SSIS using ZappySys S3 Task.

1. First call Delete S3 Files Action using Source Path = mybucket/*. Delete file action won’t delete any real folders (i.e. name ends with slash).
2. Once files are deleted from bucket you can get list of empty folders
   1. Select Action = Get Amazon folder list as ADO.net Table
   2. In the path enter Path = mybucket/*
   3. In the Result Variable Create or Select variable (e.g. User::FolderList)
3. Use ForEachLoop Task to Delete empty folders (Use Delete File Action because there is no folder concept in S3. File name ends with slash is treated as folder)
   1. In the ForEachLoop Task select enumerator as  Foreach ADO enumerator
   2. In the variable select a variable which holds list of folders (obtained in previous step)
   3. On the Variable Mappings Tab. Select Some variable (e.g. User::FolderPath) to map to Index =0
   4. Drag new S3 Task inside ForEachLoop and Select Delete File Action
   5. In the File Path enter variable as {{User::FolderPath}}  (or whatever name you given which holds iteration FolderPath in previous step)
4. Call Delete Bucket Action once bucket is totally empty.

Delete Amazon S3 Bucket (including files and folders)

Delete Amazon S3 Folders

There is no real folder concept in Amazon S3 file system. S3 File System is just key / value system. Key is basically file name (e.g. somefolder/subfolder/myfile.txt). You can use slash in name to indicate hierarchy which can be used by some client apps like ZappySys S3 File Picker to Mimic Treeview like structure so user feel like real file system but there are no rules if you have to use slash or not.

Most client tools for S3 treats any file name ends with slash as real folder.  See below step to delete amazon s3 folder (real folder which was created explicitly – mentioned in next section).

1. Open S3 Task and select action = Delete Amazon File(s)
2. In the path enter your Folder Path (e.g. mybucketname/output/sub2/ )

Create Amazon S3 Folder

Some time you may want to create real S3 folder which is not treated as file. In that case use this technique to create s3 folder.

Conclusion

In this post we saw how easy it is to use SSIS Amazon Storage Task (Amazon S3 Task) to delete Amazon S3 file, folder and bucket. You can also perform other common amazon storage related actions (e.g. Upload / Download / Create / List ). Download SSIS PowerPack to try this component and 70+ other tasks / components which are not covered in this article.

The post How to delete Amazon S3 File / Folder / Bucket in SSIS appeared first on ZappySys Blog.

In this short article, you will learn how to write Regular expressions in SSIS (i.e. Regex) and what tool to use to test them. You will also find helpful resources on how to write more sophisticated expressions and learn more about them. For demo purposes, we will use FREE SSIS Regex Parser Task to parse and extract the text using Regex.

You can use Regular expressions in several SSIS PowerPack connectors:

• SSIS Regex Parser Task (FREE),
• Azure Blob Storage Task,
• Advanced File System Task,
• Amazon S3 Storage Task and others.

Writing Regular Expressions in SSIS

Based on the SSIS component you use it will include the filename in filtering options or match the text: See below Syntax to write Regex in ZappySys tools. We support additional construct {{X, Y}} at the end of Regex to control two parameters. This additional construct is useful for data extracted from matching regex. If you want to Test patterns and not worry about data extraction then no need to use last {{

Occurance_Index

Group_Index_Or_Name

Syntax:

<your Regular Expression>[{{Occurance_Index|*[,Group_Index_Or_Name]}}]

Where:

Occurance_Index

Group_Index_Or_Name

How to name a group?

E.g. (\w+)@(?<domain>\w+.com)

How to use the group name in the match extract?

E.g. (\w+)@(?<domain>\w+.com){{0,domain}}

Example Input:

Let’s assume we have the following input text. We will test various Expressions.

Customer => AAA
Email => aaa@google.com
Phone => 101-222-3333
========
Customer => BBB
Email => bbb@yahoo.com
Phone => 102-222-3333
========
Customer => CCC
Email => ccc@hotmail.com
Phone => 103-222-3333
========
Customer => DDD
Email => ddd@outlook.com
Phone => 104-222-3333

Sample Regex Expressions

(.*)

More Regular Expression Examples

Regex Examples (Using SSIS Regular Expression Parser Task)

Here is an example how Regex

(.+)_(.+)_(.+){{1,2}}

Using Regular Expressions in SSIS Regex Parser Task (Extract Groups)

Using Groups / Occurrence Index

Some tasks like SSIS Regex Parser Task (FREE) supports extracting value from specific occurrence and specific part of matched pattern using special syntax at the end of your pattern (see below).

Your Regex Pattern Here{{OccuranceIndex,GroupIndex}}

Where :
OccuranceIndex is 0 based (0=extract first occurrence)
GroupIndex is 0 based (0=extract first matching group from pattern. First group is always entire text. )

See above screenshot in previous section for example.

Tools

The best tool we’ve found to write and test Regex is Regex Hero (will require IE with Silverlight if you want to use it online, in the browser):

Another great site for Regex testing is https://regex101.com (Works in any browser unlike previous one)

and few more sites as below

http://www.regexr.com/
http://www.regexlib.com/
http://www.regular-expressions.info/

Resources

Regular Expressions cheat-sheet to hang on the wall

Regular Expressions quick reference

The post Using Regular Expressions in SSIS appeared first on ZappySys Blog.

In our previous post we explored unlimited possibilities to call Amazon AWS API using SSIS. Now in this post we will learn how to import / export data from Amazon Athena using SSIS. Amazon Athena is very similar service like Google BigQuery which we documented already.  To read data from Amazon Athena we will use ZappySys JSON / REST API Source which supports excellent JSON parsing capability like this.

Prerequisites

Before we do hello world demo for calling Amazon AWS API, you will need to make sure following prerequisites are met.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Access to valid AWS credentials (Access Key, Secret Key for your IAM User). Click here to learn more about IAM users and Access Key/Secret Key
4. Make sure SSIS PowerPack is installed. Click here to download.

Steps needed to read data from Amazon Athena

To read data from Amazon Athena you need to do atleast 3 steps at very high level. Later on this article we will see each step in depth.

1. Call StartQueryExecution API to send SQL Query (Its Job Style API so this call starts Job)
2. Keep checking Job Status by calling  GetQueryExecution API (In the response check State field for Status – We need to look for word SUCCESS)
3. Once Job is finished it produces CSV file in S3 Bucket (Output Location is supplied in Step-1)

Read Amazon Athena Query output in SSIS (S3 CSV files)

AWS Console – Athena Query Editor / Configure Output Settings

Few things you might have to setup before you can call Athena API in SSIS

1. Make sure your IAM user has correct permission (We used existing policies for ease of use but you can use custom policy for precise permission set)
2. Configure default output location for your Athena queries

Configure IAM user permission to call Amazon Athena API

Set Athena Query Output Location

Testing Athena Query inside Amazon AWS Console (Query Editor)

Creating Connection to call Athena API in SSIS

Any API calls made to Amazon Athena service using ZappySys JSON / REST API Source  or  ZS REST API Task needs OAuth connection.

For REST API Task configure as below (More details about using REST API Task is in the next section).

1. Drag ZS REST API Task from Control Flow Toolbox
2. Double click to edit.
3. Change Request URL Access Mode to [Url from Connection]
4. Now select <<New OAuth Connection>> from connection drop down
5. On OAuth connection select Amazon AWS API v4 as Provider
6. Enter your Access Key and Secret Key

Create SSIS Connection for Amazon AWS API Call (Athena REST API Example)

Creating Table in Amazon Athena using API call

For this demo we assume you have already created sample table in Amazon Athena. If you wish to automate creating amazon athena table using SSIS then you need to call CREATE TABLE DDL command using ZS REST API Task.

1. In the previous ZS REST API Task select OAuth connection (See previous section)
2. Enter Request Url as below
   
   

   https://athena.us-east-1.amazonaws.com/

3. In the HTTP Headers enter below 2 headers
   

   X-Amz-Target: AmazonAthena.StartQueryExecution
   Content-Type: application/x-amz-json-1.1

4. Change Request method to POST
5. In the Body enter below
   

   {
    "ResultConfiguration": {
    "OutputLocation": "s3://my-bucket/output-files/"
    },
    "QueryString": "create external table tbl01 (CustomerID STRING, CompanyName STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION 's3://my-bucket/input-files/';",
    "ClientRequestToken": "{{System::ExecutionInstanceGUID}}"
   }

   Notice: Rather than using hard coded  DDL SQL Statement we can use variable placeholder (e.g.

   "QueryString": "{{User::Query,FUN_JSONENCODE}}"

      ). Use 

   FUN_JSONENCODE

    to replace new line with \r\n else JSON becomes invalid.
6. Now click Test Request / Response. This will create a new Athena table which we will query in later section.

Notice that in above Body ClientRequestToken is a unique number each time you have to call StartQueryExecution API to make it simple we used SSIS system variable System::ExecutionInstanceGUID. Its unique for each execution. You can also use newid() function from SQL Server and save result to variable if you like.

Here is the screenshot of above Request

RESt API Task – Create table in Amazon Athena using SSIS (StartQueryExecution API Call)

Uploading data for Amazon Athena (Source files)

Once you have athena table created, we will need some data to query. Loading data to amazon athena table is nothing but upload files to S3. If you noticed we added below line in CREATE TABLE statement in the previous section. This means all source files are located at this S3 location.

LOCATION 's3://my-bucket/input-files/'

We have written blog post to explain this process (Click Here – How to load data from SQL server to S3 files).

Extract data from Amazon Athena using SSIS (Query Athena)

Now its time to read some data by writing SQL query for Athena data (i.e. S3 files). As we mentioned earlier, reading data from Athena can be done using following steps.

1. Call  StartQueryExecution API  using REST API Task to execute your Query (It returns you QueryExecutionID). Use same technique as Create Table example. Replace CREATE Table command in body with your SQL Query (New line must be replaced by \r\n in JSON).
2. Use REST API Task to call GetQueryExecution API and configure periodic status check loop (Task keeps waiting until Success or Failure Value not found in Response)
3. Once data is available read data using S3 CSV File Source
4. At last you can clean up query output files or keep it if some other process wants to read it (You can also clean up S3 files using bucket retention policy automatically at scheduled interval).

Now lets see each step in depth

Step1-Start Amazon Athena Query Execution

Now first thing is to execute Athena Query by calling StartQueryExecution API . Below steps are almost same steps as we saw in section Creating Table in Amazon Athena using API call

Only part is different here is SQL query is SELECT query rather than CREATE TABLE.

1. Drag and drop ZS REST API Task from SSIS toolbox
   
2. Enter Request Url as below
   
   

   https://athena.us-east-1.amazonaws.com/

3. In the HTTP Headers enter below 2 headers
   

   X-Amz-Target: AmazonAthena.StartQueryExecution
   Content-Type: application/x-amz-json-1.1

4. Change Request method to POST
5. In the Body enter below
   

   {
     "ResultConfiguration": {
       "OutputLocation": "s3://my-bucket/output-files/"
     },
     "QueryString": "select * from sampledb.elb_logs\r\nlimit 100000",
     "ClientRequestToken": "{{System::ExecutionInstanceGUID}}"
   }

   Notice: Rather than using hard coded  SQL Query we can use variable placeholder (e.g.

   "QueryString": "{{User::Query,FUN_JSONENCODE}}"

      ). Use 

   FUN_JSONENCODE

    to replace new line with \r\n else JSON becomes invalid.

   

6. Now go to Response Settings Tab and configure below way to extract QueryExecutionId (we will need it later).

   Set Extract Type as Json.
   Enter Expression as

   $.QueryExecutionId

   Check Save output to Variable and create new Variable (we can name as QueryExecutionId )
   

   Call Athena SQL, Save Amazon Athena QueryExecutionId in SSIS Variable

7. That’s it now we can move to next step

Step2 – Wait until Athena Query Execution is done

New version of ZS REST API Task includes Status check feature. This feature avoids complex looping logic. Here is how to configure to check Job Status (Wait until Query execution is finished). Steps are almost same as previous section except three things ( Headers, Body and Status Check Tab)

For Status check we will call GetQueryExecution API .

1. Drag and drop ZS REST API Task from SSIS toolbox
   
2. Enter Request Url as below
   
   

   https://athena.us-east-1.amazonaws.com/

3. In the HTTP Headers enter below 2 headers
   

   X-Amz-Target: AmazonAthena.GetQueryExecution
   Content-Type: application/x-amz-json-1.1

4. Change Request method to POST
5. In the Body enter below
   

   {"QueryExecutionId": "{{User::QueryExecutionId}}"}

   Notice: Here we have used Variable in the body. This variable we extracted in the previous step. It contains QueryExecution ID for the Job we submitted.
   

   SSIS Amazon Athena – Call GetQueryExecution API (Status Check)

6. Now go to Status Check Tab and configure below way to implement Status check loop. We want to stop check once status is SUCCESS or FAILED or CANCELLED. If we find FAILED or CANCELLED in State field then we must fail the step with error.
   1. Check Enable Status Check
   2. Enter Success Value as

      SUCCEEDED

   3. Check Fail Task option and enter this regex in the text box 

      FAILED|CANCELLED

   4. In the Interval enter 5 seconds delay after each iteration.
7. That’s it. Status check Tab will look like below. Now we can move to next step
   

   REST API Status check Loop – Keep checking until JOB is done

Step3-Read data from Athena Query output files (CSV / JSON stored in S3 bucket)

When you create Athena table you have to specify query output folder and data input location and file format (e.g. CSV, JSON, Avro, ORC, Parquet …) they can be GZip, Snappy Compressed. Once you execute query it generates CSV file. Zappysys can read CSV, TSV or JSON files using S3 CSV File Source or S3 JSON File Source connectors. In this section we will use CSV connector to read Athena output files.

Here is how to Athena output data

1. Drag and drop Data flow task from SSIS Toolbox
   
2. Double click data flow and drag and drop  ZS Amazon S3 CSV File Source
   
3. Double click to configure it.
4.  Click New Connection and configure S3 Connection
5. Once connection is created, browse to S3 bucket / folder location where Athena outputs files.
6. Replace file name with variable like below
   

   my-bucket/output-files/{{User::QueryExecutionId}}.csv

    

Reading data from Amazon Athena CSV files (Stored in S3)

Putting all together – Athena Data Extract Example

Here is final package with all pieces we talked.

Conclusion

You have seen that using ZappySys SSIS PowerPack how quickly you can integrate with Amazon Athena and other AWS Cloud Services.  Download SSIS PowerPack and try it out by yourself.

Keywords: Extract data from Amazon Athena | Read from Amazon Athena Query file | Import / Export data from Amazon Athena | Fetch data from Amazon Athena

The post Import / Export data from Amazon Athena using SSIS appeared first on ZappySys Blog.

Introduction

Sometimes you may need to make an advanced filtering on file path, file date, size etc in SSIS Amazon S3 Task or SSIS Azure Blob Storage Task . For example, you have a directory “Customers“, which contains client directories. Each of them contains a directory named “Invoices“, which is partitioned by year. E.g. “Customers/Microsoft/Invoices/2015”. So if you want to get all invoices of all customers of all years, use Advanced Filter & Sorting option and configure Where clause to use wildcards (*) like this:

SSIS Amazon/Azure Storage Task – To get all invoices of all customers of all years, use this Where clause: FullPath LIKE ‘Customers/*/Invoices/*’

Using Advanced Search – Regex / SQL Expression (Filter by file date, size, name)

Sometimes it’s necessary to filter file based on advanced attributes such as file Age (How old), File Size, Multiple File Patterns (e.g. Delete *.zip, *.gz, *.rar in one action)

You can check this link or this link for more information. For Azure Blob Storage Task its almost same UI.

You can use Advanced Tab to filter further. First supply primary filter on Source Tab (e.g. mybucket/*.* ) and then supply your advanced filter (e.g. you can exclude *.exe and *.msi file from your delete list as below screenshot, also only delete files between 1000 and 50000 bytes in file size).

Select / Delete Amazon S3 File (Multiple) using Wildcard pattern

Advanced Search in SSIS Amazon S3 Task – Use Regular Expression Patterns, SQL like expression for Sort / Filter

//////////////////////////////
Examples:
//////////////////////////////
AgeInDays < 5  //Get files created in last 5 days
LastEditInDays > 3 //Get files modified before 3 days
AgeInDays > 10 AND LastEditInDays < 3  //Get files created before 10 days and modified in last 3 days
Size < 1024  //Get files where File size is less than 1024 bytes
CreationDate BETWEEN #2015-01-01# AND #2015-01-01#  
Name IN ('File1.txt', 'Download.msi')
FullPath LIKE 'mybucket\somefolder\month_*')
Extension IN ('.txt','.csv')
LastModifiedDate > #2013-01-01T16:00:00#
IsReadOnly = True
/////////////////////////////////////////
//Supported Columns for expression:
/////////////////////////////////////////
Name, FullPath, Extension, FolderPath, AgeInDays, LastEditInDays, Size, CreationDate, CreationDateUtc, LastModifiedDate, LastModifiedDateUtc, IsReadOnly

Select Amazon S3 files older than X days

If you like to select / delete files older than certain days then use below technique.

AgeInDays > 10

1. First Specify primary search criteria on Source tab
2. Go to Advanced Filter Tab and specify Following Pattern in Where Textbox (For example delete Files older than 10 days )

The post Filter by file size, name, date for Amazon / Azure Storage SSIS Task appeared first on ZappySys Blog.

Azure and AWS both are most popular Cloud Platforms. In this blog post we will learn how to copy or move Amazon S3 files to Azure Blob Storage without any coding or scripting  (AWS to Azure File Copy / Migration Scenario). To achieve this objective we will use following Drag and Drop SSIS  Tasks (i.e. Microsoft SQL Server Integration Services – ETL Platform for SQL Server). Following components are highly optimized for parallel copy/multi threading with secure connection (client side/serverside encryption).

SSIS Azure Blob Storage Task

SSIS Amazon Storage Task (AWS S3)

Concept : Fast Server Side Copy in Azure (Copy files into Azure Blob Storage)

Azure provides unique feature called server side file copy. Using this feature you can load or copy files into Azure Blob Storage without landing data to your local machine. As of now (March 2017) this type of feature is still missing in other Cloud Platform such as Amazon AWS and Google Cloud Platform – GCP.

Using server side copy feature in Azure… you can achieve following scenarios in SSIS without coding

• Move / Copy files from Amazon S3 to Azure Blob Storage
• Move / Copy files from Google Cloud Platform (GCP) to Azure Blob Storage
• Copy files from any public URL to Azure Blob Storage (Assuming URL doesn’t require credentials or its intranet URL)

In first two scenarios you need to have Source File URL using Authentication information inside URL (This is called Pre-Signed URL e.g. http://mycloud.com/myfile.zip?SIGNATURE-GOES-HERE ). All major Cloud Providers Support Pre-Signed URL so you can securely share files  with other without sharing your actual Credentials. You can configure Pre-Signed URL to expires in certain time frame (Check your Cloud API documentation). If your file doesn’t need credentials to access then you don’t have to worry about Pre-Signed URL. Your source can be file URL  (E.g. https://mysite.com/downloads/fil1.zip)

Pre-Signed URL for Google Cloud (GCP)
Here is more information how to get Pre-Signed URL for Google Cloud Platform (GCP) – Use gsutil command line
Pre-Signed URL for Amazon AWS (S3 file)
To create pre-signed URL for AWS S3 files you can use SSIS Amazon Storage Task and use Get Pre-Signed URL Action (Described in the below section). This action supports creating pre-signed URLs for multiple files using wildcard (e.g /mybkt/*.zip)  or you can get single pre-signed URL. If you use pattern search then you will get DataTable back … which can be loop through using ForEach Loop task (Loop ADO Recordset option) . If you do not use wild card then only one URL returned in string format.

Prerequisites

Before we look at next section make sure following prerequisites are met

1. You have obtained Amazon S3 Access Key and Secret Key to access desired files
2. You have obtained Azure Storage Account Name and Account Key to access desired Blob Container. If you don’t have access to Azure then you can download Azure Storage Emulator for testing Purpose. Check this.
3. You have basic knowledge of SSIS. If you dont then search for SSIS tutorial and there are many blogs / tutorials to get started 🙂

Step-By-Step – Create SSIS Package – Copy Amazon S3 Files to Azure Blob Storage

Now lets look at how to copy Amazon S3 files to Azure in few clicks. This approach doesn’t bring any data to your local system so its purely server to server copy and that is why its very fast and secure.

1. First Download and Install SSIS PowerPack
2. Once you install SSIS PowerPack. Create a new Sample SSIS Project and open Package
3. From SSIS Toolbox Drag ZS Amazon Storage Task and drop on Control Flow surface. Rename it to Get S3 File Urls
4. Double click S3 task to edit. From Action dropdown select (Get Amazon S3 Files Pre-Signed Url Option)
5. Click New next to the Connection Dropdown to create Amazon Storage Connection. Enter your credentials and bucket region and Click Test. If you are not sure about your bucket region then check this article.
   

   SSIS Amazon Storage Task – Create Pre-Signed URLs for multiple files stored in S3

6. Now click on Advanced Tab and check Use region specific endpoint option
   

   SSIS Amazon S3 Connection – Region specific endpoint option

7. Once Test connection is green you can click OK to save connection.
8. Once you back to Amazon Storage Task UI… Click on browse next to S3 file path. Her you can select one file or enter pattern to create URLs for multiple files.
9. To save pre-signed URL(s) select Variable. If variable is not there then create new one… If you enter pattern in Source path then Variable must be Object datatype. For multiple URL it will return ADO Recordset (You can use with ForEach Loop Task). If you didn’t enter pattern in source path then variable can be string datatype.
   

   SSIS Amazon Storage Task – Save Pre-Signed URL to Variable (Save Multiple URL as Recordset )

10. Now drag and drop ForEach Loop Container on the surface and configure like below
    

    SSIS ForEach Loop Container Task – Loop through Amazon S3 file URL

    

    SSIS ForEach Loop Task – Variable Mappings

11. Once Foreach Loop is configured you can drag ZS Azure Storage Task inside ForEach Loop Container
12. Double click Azure storage task and select [Copy from external file] option from Action dropdown.
13. In the source Path/URL we will enter variable name which holds current URL for ForEach Loop iteration. Easy way is click blue variable icon and select <<Insert Variable>>. Your placeholder may look like below.
    

    {{User::varCurrentS3Url}}

14. For target path … Click New Connection to create and configure new Azure Storage Connection like below. Enter Azure storage credentials and click test. If you don’t have real Azure credentials then you can use Azure Storage Emulator (Check this). Once connection is Configured click Test connection and click OK to save connection.
    

    Azure Storage Task – Configure Blob Storage connection- Serverside Copy S3 files – AWS to Azure

15. Once everything is configure you can save package and execute. This will copy S3 files to Azure Blob storage.
16. Here is execution log
    

    SSIS package "C:\SSIS\Amazon-To-Azure-Copy.dtsx" starting.
    Information: 0x0 at Get Signed URL for S3 Files, Get Signed URL for S3 Files: You are running TRIAL version. It will expire in 29 day(s)
    Information: 0x0 at Get Signed URL for S3 Files, Get Signed URL for S3 Files: Reading [PRESIGNED-URL] property for zs-eu-west-2-london-bkt2/*.*
    Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copy started: Source=https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile1.csv, Target=test1/
    Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copying https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile1.csv ...
    Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copy started: Source=https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile3.txt, Target=test1/
    Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copying https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile3.txt ...
    SSIS package "C:\SSIS\Amazon-To-Azure-Copy.dtsx" finished: Success.

    

    SSIS Package Execution – Copy Multiple S3 Files from
    Amazon to Azure

How to move Amazon S3 files to Azure

In above example you saw how to copy files from Amazon S3 to Azure Blob Storage. But what if you want to Move… so once file is copied you want to delete from source. In that case its simple. Add one more Amazon S3 Task at the end so if all previous steps successful you can Issue delete files command using ZS Amazon Storage Task.

SSIS Delete Amazon S3 Files after Successful S3 to Azure Copy operation (This will mimic Move)

Download Sample SSIS Package

Click here to download sample SSIS file for SQL Server 2012, 2014, 2016

Conclusion

If you want to bring data from Amazon S3 to Azure in fastest way then technique described in this article can help you to cut down your data transfer time by several times. SSIS PowerPack comes with 45+ components and tasks which can give you drag and drop interface for your Cloud connectivity projects. It also comes with many connectors to help with JSON, XML, REST API related integration. Try SSIS PowerPack for FREE without any limitation and find out what else you can do with it.

The post SSIS – Copy Amazon S3 files from AWS to Azure appeared first on ZappySys Blog.

In this blog post you will see how easy it is to load large amount of data from SQL Server to Amazon S3 Storage. For demo purpose we will use SQL Server as relational source but you can use same steps for any database engine such as Oracle, MySQL, DB2. In this post we will use Export CSV Task and Amazon S3 Storage Task  to achieve desired integration with Amazon S3 with drag and drop approach. You can also export JSON or XML data to Amazon S3 using same techniques (Use Export JSON Task  or Export XML Task ).

Our goal is to achieve following things

• Extract large amount of data from SQL Server Table or Query and export to CSV files
• Generate CSV files in compressed format (*.gz) to speedup upload and save data transfer cost to S3
• Split CSV files by row count
• Upload data to Amazon S3 using highly parallel manner for maximum speed

There are three different ways you can achieve data export to Amazon S3 using SSIS.

1. Method-1 (Fastest): Use two step process (First export SQL Server data to local files using Export Task and then upload files to S3 using  Amazon S3 Storage Task  )
2. Method-2 (Slower): Use Export Task with Amazon S3 Connection as Target rather than save to Local files.
3. Method-3 (Slower): Use Data flow components like Amazon S3 Destination for CSV  (for JSON / XML  use Method1 or Method2)

Each method has its own advantage / disadvantage. If you prefer to upload / compress / split large amount of data then we recommend Method#1 (Two steps). If you have not very huge dataset then you can use Method#2 or Method#3. For Last method you can only use CSV export option (we don’t have JSON/ XML Destination for Amazon S3 yet – we may add in future)

Screenshot of SSIS Package

Extract SQL Server Data to CSV files in SSIS (Bulk export) and Split / GZip Compress / upload files to Amazon S3 (AWS Cloud)

Method-1 : Upload SQL data to Amazon S3 in Two steps

In this section we will see first method (recommended) to upload SQL data to Amazon S3. This is the fastest approach if you have lots of data to upload.  In this approach we first create CSV files from SQL Server data on local disk using SSIS Export CSV Task. After that in second step we upload all files to Amazon S3 using SSIS Amazon Storage Task.

Step-1: Configure Source Connection in Export CSV Task

To extract data from SQL Server you can use Export CSV Task. It has many options which makes it possible to split large amount of data into multiple files. You can specify single table or multiple tables as your data source.

For multiple table use vertical bar. e.g.  dbo.Customers|dbo.Products|dbo.Orders. When you export this it will create 3 files ( dbo.Customers.csv , dbo.Products.csv, dbo.Orders.csv )

Steps:

1. Drag ZS Export CSV Task from Toolbox
2. Double click task to configure
3. From connection drop down select New connection option (OLEDB or ADO.net)
4. Once connection is configured for Source database specify SQL Query to extract data as below
   

   Export SQL Server Table or Query as CSV file (Bulk export in SSIS)

5. Now go to target tab. Here you can specify full path for file. e.g. c:\ssis\temp\s3dump\cust.csv

Step-2: Compress CSV Files in SSIS ( GZIP format – *.gz )

Above steps will export file as CSV format without splitting or compression. But to compress file once exported you can go to Target tab of Export CSV Task and check [Compress file to *.gz format] option.

Compress exported SQL Server data files to GZip ( *.gz) in SSIS Export CSV Task

Step-3: Split CSV files by row count or data size in SSIS

Now lets look at how to split exported CSV files into multiple files so we can upload many files in parallel. Goto Split Options and check [Enable Split by Size/Rows]

Using SSIS Split Exported CSV files (Split by row count or size)

Step-4: Upload CSV files to Amazon S3 – Using multi threaded option

Now final thing is use Amzon S3 Task to upload files to S3.

Things to remember

Sometimes times due to high network activity you may get timeout errors during upload. In that case you can adjust few settings described here. Also try to reduce total parallel threads on S3 Connection see that helps.

Steps:

1. Drag ZS Amazon Storage Task from SSIS toolbox
2. Double click Amazon Storage Task to configure it
3. Specify Action = UploadFilesToAmazon
4. Specify Source file path (or pattern) e.g. c:\SSIS\temp\s3dump\*.*
5. Now in the Target connection dropdown click [New]
6. When Connection UI opens select Service Type = S3
7. Enter your Access Key, Secret Key and Region (Leave all other parameters default if you not sure)
8. Click Test and close connection UI
9. On the Target path on S3 Storage Task enter your bucket and folder path where you want to upload local files. For example your bucket name is bw-east-1 and folder is sqldata then enter as below
   bw-east-1/sqldata/
10. Click ok and Run package to test full package

Upload local files to Amazon S3 using SSIS AWS Storage Task

Method-2 : Upload SQL data to Amazon S3 without local stage (One step)

Now let’s change previous approach little bit to send SQL server data directly to Amazon S3 without any Landing area on local disk.  Export CSV Task , Export JSON Task  and Export XML Task all of them supports Amazon S3 / Azure Blob and Secure FTP (SFTP) connection as target (Only available in Pro Edition). We will use this feature in following section.

This approach helps to avoid any local disk need and it may be useful for security reason for some users. However drawback of this approach is, it wont use parallel threads to upload large amount of data like previous method.

Following change will be needed on Export task to upload SQL data directly to S3 / FTP or Azure storage.

Export SQL data to multiple files to Amazon S3, Azure, Secure FTP (SFTP) in Stream Mode using SSIS. Configure Compress GZip, Overwrite, Split Options

Method-3 : Using Amazon S3 destination – Generate Amazon S3 file from any source

Now let’s look at third approach to save data from any SSIS Source to Amazon S3 file. Advantage of this approach is you are not limited to few source options provided by Export CSV Task. If you have complex data transformation needed in Data Flow before sending data to S3 then use this approach.  We will use Amazon S3 Destination for CSV as below

1. Drag SSIS Data flow task from toolbox
2. Create necessary source connection (e.g. OLEDB connection)
3. Create Amazon S3 Connection (Right click in Connection Managers panel in bottom and click New connection and select ZS-AMAZON-STORAGE type )
4. Once connection managers are created Go to data flow designer and Drag OLEDB Source
5. Configure OLEDB Source to read desired data from source system (e.g. SQL Server / Oracle)
6. Once source is configured drag ZS Amazon S3 CSV File Destination from SSIS toolbox
7. Double click S3 Destination and configure as below
   1. On Connection Managers tab select S3 Connection (We created in earlier section).
   2. Properties tab configure like below screenshot
   3. On Input Columns tab select desired column you like to write in the target file. Your name from upstream will be taken as is for target file. So make sure to name upstream columns correctly.
   4. Click OK to save UI
8. Execute package and check your S3 Bucket to see files got created.

Loading SQL Server data into S3 Bucket Files (Split, Compress Gzip Options) – SSIS Amazon S3 CSV File Destination

Conclusion

In this post you have seen how easy it is to upload / archive your SQL Server data (or any other RDBMS data) to Amazon S3 Storage in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to integrate SQL Server and Amazon S3 using SSIS.

The post Loading data from SQL Server to Amazon S3 in SSIS (Split Files, GZip) appeared first on ZappySys Blog.