In our previous blog we saw how to load data into Amazon S3. Now in this blog, we will see How to read Amazon S3 Storage Files in SSIS (CSV, JSON, XML Format files). To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like perform File operations on Amazon S3 Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on how to Read Amazon S3 Storage CSV, JSON and XML Files using respective SSIS Source tasks.

Components Mentioned in this article

	Amazon S3 CSV File Source
	Amazon S3 CSV File Destination
	Amazon S3 XML File Source
	Amazon S3 JSON File Source

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. You have obtained Amazon S3 account access key / secret key.
4. Finally, do not forget to install ZappySys SSIS PowerPack

What is Amazon S3 Storage

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements. Amazon S3 is designed for 99.999999999% (11 9’s) of durability, and stores data for millions of applications for companies all around the world.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Amazon S3 Source for CSV/JSON/XML File that will help you in reading CSV, JSON and XML Files from Amazon S3 to the Local machine, Upload files(s) to Amazon S3 Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to download files from Amazon S3 Storage.

You can connect to your Amazon S3 Account by entering your storage account credentials.

Read Amazon S3 Storage Files in SSIS (CSV, JSON, XML)

Let´s start with an example. In this SSIS Amazon S3 Source for CSV/JSON/XML File task example, we will read CSV/JSON/XML files from Amazon S3 Storage to SQL Server database.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Drag and Drop relevant Amazon S3 Source for CSV/JSON/XML File Task from the SSIS Toolbox.
   

   Add Amazon S3 Source Tasks

3. Create a connection for Amazon S3 Storage Account.
   

   Create Amazon S3 Storage Connection

4. Select the relevant single file to read from Amazon S3 Storage in their relevant source of CSV/JSON/XML File Task.
   

   Select File From Amazon S3 Storage

5. We can also read the multiple files stored in Amazon S3 Storage using wildcard pattern supported e.g. dbo.tblNames*.csv / dbo.tblNames*.json / dbo.tblNames*.xml in relevant source task
   

   Use wildcard pattern .* to read multiple files data

6. We can also read the zip and gzip compressed files also without extracting it in the specific Amazon S3 Source for CSV/JSON/XML File Task.
   

   Reading zip and gzip compressed files (stream mode)

7. Finally, we are ready to load this file(s) data into the SQL Server.

Load Amazon S3 files data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

Above all, in this blog, we learned how to Read Amazon S3 Storage Files in SSIS. We used Amazon S3 Source for CSV file, Amazon S3 Source for JSON file and Amazon S3 Source for XML file to read the file(s) from Amazon S3 Storage and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• SSIS Amazon S3 Source for CSV/JSON/XML File
• Introduction to Amazon S3 Storage Service
• How to Get Your Amazon AWS Access Key & Secret Key

The post Read Amazon S3 Storage Files in SSIS (CSV, JSON, XML) appeared first on ZappySys Blog.

Azure and AWS both are most popular Cloud Platforms. In this blog post we will learn how to copy or move Amazon S3 files to Azure Blob Storage without any coding or scripting  (AWS to Azure File Copy / Migration Scenario). To achieve this objective we will use following Drag and Drop SSIS  Tasks (i.e. Microsoft SQL Server Integration Services – ETL Platform for SQL Server). Following components are highly optimized for parallel copy/multi threading with secure connection (client side/serverside encryption).

SSIS Azure Blob Storage Task

SSIS Amazon Storage Task (AWS S3)

Concept : Fast Server Side Copy in Azure (Copy files into Azure Blob Storage)

Azure provides unique feature called server side file copy. Using this feature you can load or copy files into Azure Blob Storage without landing data to your local machine. As of now (March 2017) this type of feature is still missing in other Cloud Platform such as Amazon AWS and Google Cloud Platform – GCP.

Using server side copy feature in Azure… you can achieve following scenarios in SSIS without coding

• Move / Copy files from Amazon S3 to Azure Blob Storage
• Move / Copy files from Google Cloud Platform (GCP) to Azure Blob Storage
• Copy files from any public URL to Azure Blob Storage (Assuming URL doesn’t require credentials or its intranet URL)

In first two scenarios you need to have Source File URL using Authentication information inside URL (This is called Pre-Signed URL e.g. http://mycloud.com/myfile.zip?SIGNATURE-GOES-HERE ). All major Cloud Providers Support Pre-Signed URL so you can securely share files  with other without sharing your actual Credentials. You can configure Pre-Signed URL to expires in certain time frame (Check your Cloud API documentation). If your file doesn’t need credentials to access then you don’t have to worry about Pre-Signed URL. Your source can be file URL  (E.g. https://mysite.com/downloads/fil1.zip)

Pre-Signed URL for Google Cloud (GCP)
Here is more information how to get Pre-Signed URL for Google Cloud Platform (GCP) – Use gsutil command line
Pre-Signed URL for Amazon AWS (S3 file)
To create pre-signed URL for AWS S3 files you can use SSIS Amazon Storage Task and use Get Pre-Signed URL Action (Described in the below section). This action supports creating pre-signed URLs for multiple files using wildcard (e.g /mybkt/*.zip)  or you can get single pre-signed URL. If you use pattern search then you will get DataTable back … which can be loop through using ForEach Loop task (Loop ADO Recordset option) . If you do not use wild card then only one URL returned in string format.

Prerequisites

Before we look at next section make sure following prerequisites are met

1. You have obtained Amazon S3 Access Key and Secret Key to access desired files
2. You have obtained Azure Storage Account Name and Account Key to access desired Blob Container. If you don’t have access to Azure then you can download Azure Storage Emulator for testing Purpose. Check this.
3. You have basic knowledge of SSIS. If you dont then search for SSIS tutorial and there are many blogs / tutorials to get started 🙂

Step-By-Step – Create SSIS Package – Copy Amazon S3 Files to Azure Blob Storage

Now lets look at how to copy Amazon S3 files to Azure in few clicks. This approach doesn’t bring any data to your local system so its purely server to server copy and that is why its very fast and secure.

1. First Download and Install SSIS PowerPack
2. Once you install SSIS PowerPack. Create a new Sample SSIS Project and open Package
3. From SSIS Toolbox Drag ZS Amazon Storage Task and drop on Control Flow surface. Rename it to Get S3 File Urls
4. Double click S3 task to edit. From Action dropdown select (Get Amazon S3 Files Pre-Signed Url Option)
5. Click New next to the Connection Dropdown to create Amazon Storage Connection. Enter your credentials and bucket region and Click Test. If you are not sure about your bucket region then check this article.
   

   SSIS Amazon Storage Task – Create Pre-Signed URLs for multiple files stored in S3

6. Now click on Advanced Tab and check Use region specific endpoint option
   

   SSIS Amazon S3 Connection – Region specific endpoint option

7. Once Test connection is green you can click OK to save connection.
8. Once you back to Amazon Storage Task UI… Click on browse next to S3 file path. Her you can select one file or enter pattern to create URLs for multiple files.
9. To save pre-signed URL(s) select Variable. If variable is not there then create new one… If you enter pattern in Source path then Variable must be Object datatype. For multiple URL it will return ADO Recordset (You can use with ForEach Loop Task). If you didn’t enter pattern in source path then variable can be string datatype.
   

   SSIS Amazon Storage Task – Save Pre-Signed URL to Variable (Save Multiple URL as Recordset )

10. Now drag and drop ForEach Loop Container on the surface and configure like below
    

    SSIS ForEach Loop Container Task – Loop through Amazon S3 file URL

    

    SSIS ForEach Loop Task – Variable Mappings

11. Once Foreach Loop is configured you can drag ZS Azure Storage Task inside ForEach Loop Container
12. Double click Azure storage task and select [Copy from external file] option from Action dropdown.
13. In the source Path/URL we will enter variable name which holds current URL for ForEach Loop iteration. Easy way is click blue variable icon and select <<Insert Variable>>. Your placeholder may look like below.
    

    {{User::varCurrentS3Url}}

14. For target path … Click New Connection to create and configure new Azure Storage Connection like below. Enter Azure storage credentials and click test. If you don’t have real Azure credentials then you can use Azure Storage Emulator (Check this). Once connection is Configured click Test connection and click OK to save connection.
    

    Azure Storage Task – Configure Blob Storage connection- Serverside Copy S3 files – AWS to Azure

15. Once everything is configure you can save package and execute. This will copy S3 files to Azure Blob storage.
16. Here is execution log
    

    SSIS package "C:\SSIS\Amazon-To-Azure-Copy.dtsx" starting.
    Information: 0x0 at Get Signed URL for S3 Files, Get Signed URL for S3 Files: You are running TRIAL version. It will expire in 29 day(s)
    Information: 0x0 at Get Signed URL for S3 Files, Get Signed URL for S3 Files: Reading [PRESIGNED-URL] property for zs-eu-west-2-london-bkt2/*.*
    Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copy started: Source=https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile1.csv, Target=test1/
    Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copying https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile1.csv ...
    Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copy started: Source=https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile3.txt, Target=test1/
    Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copying https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile3.txt ...
    SSIS package "C:\SSIS\Amazon-To-Azure-Copy.dtsx" finished: Success.

    

    SSIS Package Execution – Copy Multiple S3 Files from
    Amazon to Azure

How to move Amazon S3 files to Azure

In above example you saw how to copy files from Amazon S3 to Azure Blob Storage. But what if you want to Move… so once file is copied you want to delete from source. In that case its simple. Add one more Amazon S3 Task at the end so if all previous steps successful you can Issue delete files command using ZS Amazon Storage Task.

SSIS Delete Amazon S3 Files after Successful S3 to Azure Copy operation (This will mimic Move)

Download Sample SSIS Package

Click here to download sample SSIS file for SQL Server 2012, 2014, 2016

Conclusion

If you want to bring data from Amazon S3 to Azure in fastest way then technique described in this article can help you to cut down your data transfer time by several times. SSIS PowerPack comes with 45+ components and tasks which can give you drag and drop interface for your Cloud connectivity projects. It also comes with many connectors to help with JSON, XML, REST API related integration. Try SSIS PowerPack for FREE without any limitation and find out what else you can do with it.

The post SSIS – Copy Amazon S3 files from AWS to Azure appeared first on ZappySys Blog.

In this blog post you will see how easy it is to load large amount of data from SQL Server to Amazon S3 Storage. For demo purpose we will use SQL Server as relational source but you can use same steps for any database engine such as Oracle, MySQL, DB2. In this post we will use Export CSV Task and Amazon S3 Storage Task  to achieve desired integration with Amazon S3 with drag and drop approach. You can also export JSON or XML data to Amazon S3 using same techniques (Use Export JSON Task  or Export XML Task ).

Our goal is to achieve following things

• Extract large amount of data from SQL Server Table or Query and export to CSV files
• Generate CSV files in compressed format (*.gz) to speedup upload and save data transfer cost to S3
• Split CSV files by row count
• Upload data to Amazon S3 using highly parallel manner for maximum speed

There are three different ways you can achieve data export to Amazon S3 using SSIS.

1. Method-1 (Fastest): Use two step process (First export SQL Server data to local files using Export Task and then upload files to S3 using  Amazon S3 Storage Task  )
2. Method-2 (Slower): Use Export Task with Amazon S3 Connection as Target rather than save to Local files.
3. Method-3 (Slower): Use Data flow components like Amazon S3 Destination for CSV  (for JSON / XML  use Method1 or Method2)

Each method has its own advantage / disadvantage. If you prefer to upload / compress / split large amount of data then we recommend Method#1 (Two steps). If you have not very huge dataset then you can use Method#2 or Method#3. For Last method you can only use CSV export option (we don’t have JSON/ XML Destination for Amazon S3 yet – we may add in future)

Screenshot of SSIS Package

Extract SQL Server Data to CSV files in SSIS (Bulk export) and Split / GZip Compress / upload files to Amazon S3 (AWS Cloud)

Method-1 : Upload SQL data to Amazon S3 in Two steps

In this section we will see first method (recommended) to upload SQL data to Amazon S3. This is the fastest approach if you have lots of data to upload.  In this approach we first create CSV files from SQL Server data on local disk using SSIS Export CSV Task. After that in second step we upload all files to Amazon S3 using SSIS Amazon Storage Task.

Step-1: Configure Source Connection in Export CSV Task

To extract data from SQL Server you can use Export CSV Task. It has many options which makes it possible to split large amount of data into multiple files. You can specify single table or multiple tables as your data source.

For multiple table use vertical bar. e.g.  dbo.Customers|dbo.Products|dbo.Orders. When you export this it will create 3 files ( dbo.Customers.csv , dbo.Products.csv, dbo.Orders.csv )

Steps:

1. Drag ZS Export CSV Task from Toolbox
2. Double click task to configure
3. From connection drop down select New connection option (OLEDB or ADO.net)
4. Once connection is configured for Source database specify SQL Query to extract data as below
   

   Export SQL Server Table or Query as CSV file (Bulk export in SSIS)

5. Now go to target tab. Here you can specify full path for file. e.g. c:\ssis\temp\s3dump\cust.csv

Step-2: Compress CSV Files in SSIS ( GZIP format – *.gz )

Above steps will export file as CSV format without splitting or compression. But to compress file once exported you can go to Target tab of Export CSV Task and check [Compress file to *.gz format] option.

Compress exported SQL Server data files to GZip ( *.gz) in SSIS Export CSV Task

Step-3: Split CSV files by row count or data size in SSIS

Now lets look at how to split exported CSV files into multiple files so we can upload many files in parallel. Goto Split Options and check [Enable Split by Size/Rows]

Using SSIS Split Exported CSV files (Split by row count or size)

Step-4: Upload CSV files to Amazon S3 – Using multi threaded option

Now final thing is use Amzon S3 Task to upload files to S3.

Things to remember

Sometimes times due to high network activity you may get timeout errors during upload. In that case you can adjust few settings described here. Also try to reduce total parallel threads on S3 Connection see that helps.

Steps:

1. Drag ZS Amazon Storage Task from SSIS toolbox
2. Double click Amazon Storage Task to configure it
3. Specify Action = UploadFilesToAmazon
4. Specify Source file path (or pattern) e.g. c:\SSIS\temp\s3dump\*.*
5. Now in the Target connection dropdown click [New]
6. When Connection UI opens select Service Type = S3
7. Enter your Access Key, Secret Key and Region (Leave all other parameters default if you not sure)
8. Click Test and close connection UI
9. On the Target path on S3 Storage Task enter your bucket and folder path where you want to upload local files. For example your bucket name is bw-east-1 and folder is sqldata then enter as below
   bw-east-1/sqldata/
10. Click ok and Run package to test full package

Upload local files to Amazon S3 using SSIS AWS Storage Task

Method-2 : Upload SQL data to Amazon S3 without local stage (One step)

Now let’s change previous approach little bit to send SQL server data directly to Amazon S3 without any Landing area on local disk.  Export CSV Task , Export JSON Task  and Export XML Task all of them supports Amazon S3 / Azure Blob and Secure FTP (SFTP) connection as target (Only available in Pro Edition). We will use this feature in following section.

This approach helps to avoid any local disk need and it may be useful for security reason for some users. However drawback of this approach is, it wont use parallel threads to upload large amount of data like previous method.

Following change will be needed on Export task to upload SQL data directly to S3 / FTP or Azure storage.

Export SQL data to multiple files to Amazon S3, Azure, Secure FTP (SFTP) in Stream Mode using SSIS. Configure Compress GZip, Overwrite, Split Options

Method-3 : Using Amazon S3 destination – Generate Amazon S3 file from any source

Now let’s look at third approach to save data from any SSIS Source to Amazon S3 file. Advantage of this approach is you are not limited to few source options provided by Export CSV Task. If you have complex data transformation needed in Data Flow before sending data to S3 then use this approach.  We will use Amazon S3 Destination for CSV as below

1. Drag SSIS Data flow task from toolbox
2. Create necessary source connection (e.g. OLEDB connection)
3. Create Amazon S3 Connection (Right click in Connection Managers panel in bottom and click New connection and select ZS-AMAZON-STORAGE type )
4. Once connection managers are created Go to data flow designer and Drag OLEDB Source
5. Configure OLEDB Source to read desired data from source system (e.g. SQL Server / Oracle)
6. Once source is configured drag ZS Amazon S3 CSV File Destination from SSIS toolbox
7. Double click S3 Destination and configure as below
   1. On Connection Managers tab select S3 Connection (We created in earlier section).
   2. Properties tab configure like below screenshot
   3. On Input Columns tab select desired column you like to write in the target file. Your name from upstream will be taken as is for target file. So make sure to name upstream columns correctly.
   4. Click OK to save UI
8. Execute package and check your S3 Bucket to see files got created.

Loading SQL Server data into S3 Bucket Files (Split, Compress Gzip Options) – SSIS Amazon S3 CSV File Destination

Conclusion

In this post you have seen how easy it is to upload / archive your SQL Server data (or any other RDBMS data) to Amazon S3 Storage in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to integrate SQL Server and Amazon S3 using SSIS.

The post Loading data from SQL Server to Amazon S3 in SSIS (Split Files, GZip) appeared first on ZappySys Blog.

In our previous post you learned how to load data into Redshift using SSIS. Now in this post you will learn how to load data into Redshift using Informatica PowerCenter. For PowerCenter we will use ZappyShell Command line for Redshift Data Load. This small powerful command line utility can handle load of several millions or billions records in few minutes. It uses parallel execution engine which takes care most tedious steps of loading data into Amazon Redshift.

Using informatica command task to load data into redshift

Informatica PowerCenter has simple task to execute command lines. you can use this task to execute ZappyShell command line. Perform following steps to load data.

1. Download and install ZappyShell for Amazon Redshift from here
2. Now you ready to data into Amazon Redshift
3. Drag new command task into your Informatica workflow
4. Enter below command to load data from SQL Server to Redshift. If you have ODBC connectivity then use ODBC DSN to read your data.
5. You can use script file approach too where each command line parameter can be nicely wrapped in new line.
   Here is sample command you can try to load data from SQL Server to Redshift. if you have ODBC connectivity then use ODBC DSN to read your data.
   

   c:\zappyshell\aws.exe import Db 
   	--source-driver ADONET_MSSQL 
   	--source-query "select ROW_NUMBER()Over(order by a.CustomerID) Id, a.*,b.*,c.OrderID,c.OrderDate,c.Freight  from customers a,products b,orders c" 
   	--source-archivemethod None 
   	--source-stage-archivemethod Delete 
   	--target-stage-archivemethod Delete 
   	--source-stagepath "c:\redshift\stage" 
   	--target-table "customerdata" 
   	--target-stagepath "bw-rstest/cmdstage" 
   	--target-truncate 
   	--logfile "c:\redshift\log.txt" 
   	--maxrows-perfile 100000 
   	--region us-east-1 
   	--accesskey "AKIA*****************" 
   	--secretkey "lPi+XQ************************"  
   	--source-connstr "Data Source=localhost;Initial Catalog=Northwind;Integrated Security=SSPI;"  
   	--target-connstr "Host=mytestcluster-1.csu********.us-east-1.redshift.amazonaws.com;Port=5439;Database=dev;UserName=masteruser;Password=*********;EnableSsl=true;Timeout=30;CommandTimeout=3600;"

6. For complete help on import command check help file //zappysys.com/onlinehelp/zappyshell/scr/aws/aws-redshift-import.htm

Informatica PowerCenter Redshift Data Load – Amazon Redshift Import Command line tools (COPY, S3, AWS)

ZappyShell Command Line for Redshift

Below are few feature for Command line for Amazon Redshift

◾Import data to AWS Redshift database from files or relational source (e.g. MySQL, Oracle, SQL Server)
◾Import huge amount of data (millions of rows) in few minutes with parallel load techniques
◾Load local flat files to Redshift in just single line command (option to compress data files to *.gz to speed up transfer)
◾Support for Client side encryption using AES 256 Key
◾Load data from any data source (ODBC,ADO.net or OLEDB) using sql query
◾Import compressed data files (*.gz) to redshift
◾Archive files, error reporting, file splitting, many other features

The post Amazon Redshift data load in Informatica PowerCenter appeared first on ZappySys Blog.