In our previous article, we saw SSIS Azure Blob Storage task examples. Now let’s look at how to Download the Latest File from Azure Blob Storage using SSIS. Microsoft SSIS includes the Azure Blob Storage Storage Task that allows us to upload files to Azure Blob Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Azure Blob Storage Task included in ZappySys SSIS PowerPack.

In nutshell, this post will focus on how to Download the Latest File from Azure Blob Storage.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Download and install Microsoft Azure Storage Emulator
4. Download and install Microsoft Azure Storage Explorer
5. Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

• Storage account (You can access data objects in Azure Storage through a storage account.
  For more information, click here.)
• the containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
  All blobs lie within a container, Note: Container name must be lowercase)
• the blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
  See the below-attached diagram. It shows the relationship between these resources.
       You can also use Azure Storage Explorer on your local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Blob Storage task that will allow you to access files/folders from Azure Blob to the Local machine, Upload files(s) to Azure Blob Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you Download the Latest File from Azure Blob Storage.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of the local Azure Storage Emulator.

Setup Azure Storage client tools

1. Once you have downloaded and installed a storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Emulator Physical Location

2. If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Explorer Physical Location

Create an Azure Blob Storage Container

For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).

Microsoft Azure Storage Explorer: Create a Blob Container

you can also create a Virtual Directory under it. A Virtual Directory does not actually exist in Azure until you paste, drag or upload blobs into it.

Creating the new Virtual Directory under Blob Container

Step-By-Step to Download the Latest File from Azure Blob Storage

Let´s start with an example. In this SSIS Azure Blob Storage Task File task example, we will download the latest file from Azure Blob Storage.

1. First of All, Drag and Drop Azure Blob Storage Task from SSIS Toolbox and double click it to edit
   

   Drag and Drop Azure Blob Storage Task

2. The following options are available in Azure Blob Storage Task:
   

   Options to upload, download files in SSIS

3. You can also overwrite files and folder, skip all, overwrite if the source is older, overwrite if the file size is different, rename files and more:
   

   Overwrite files in SSIS

4. Let’s select the Download Azure file(s) to local directory. Now select/add The Source path, Check Recursive Scan to get all the sub-files and subfolders and it’s files and add Target Path, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
   

   SSIS Azure Blob Storage Task – Download files from Azure Blob Storage

5. Now go to Advanced Filter & Sorting and configure it like below screen to download the latest file only
   

   Storage Task configuration to download the latest file only using sorting

6. That’s it now execute the package and it will download the latest file from the Azure Blob Storage Task to Local File Machine.
7. Sometimes you may need to make advanced filtering on file path, file date, size, etc. for that refer to this article for more details.

Conclusion

To conclude, we can say that working with Azure Blob Storage is now very simple. In this article, we looked at how to Download the Latest File from Azure Blob Storage using SSIS. We used Adnavced Filtering & sorting option. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here.

Thanks for your time.

References

• About SSIS PowerPack
• Integration Services (SSIS) Expressions
• Help File: Azure Storage Task

The post Download the Latest File from Azure Blob Storage using SSIS appeared first on ZappySys Blog.

In this article, we will show SSIS Azure Blob Storage task examples. Uploading files using Azure Blob Storage Task or downloading files are common requirements in SSIS. In order to work with Azure Blob Storage, Microsoft SSIS includes the Azure Blob Storage Task that allows to upload files to Azure Blob Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Azure Blob Storage task included in ZappySys SSIS PowerPack.

Azure Blob Storage Task can be used to perform various operations with Azure Storage objects (blobs and containers) (e.g. Download, Upload, Copy, Move, Rename, Delete, etc). You can also get single property of any Azure blob or get a list of blobs as ADO.net Table and which can be easily looped through using ForEachLoop Task. Most of the operations in this task support recursive search using wild card pattern (e.g. Myfile*.txt).

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Download and install Microsoft Azure Storage Emulator
4. Download and install Microsoft Azure Storage Explorer
5. Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

• Storage account (You can access data objects in Azure Storage through a storage account.
  For more information, click here.)
• the containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
  All blobs lie within a container, Note: Container name must be lowercase)
• the blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
  See the below-attached diagram. It shows the relationship between these resources.
       You can also use Azure Storage Explorer on your Local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Blob Storage Task that will allow you to access files / folders from Azure Blob to the Local machine, Upload files(s) to Azure Blob Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to upload, download, move and delete files / folders from Azure Blob Storage.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of local Azure Storage Emulator.

Setup Azure Storage client tools

1. Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Emulator Physical Location

2. If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Explorer Physical Location

Create an Azure Blob Storage Container

For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).

Microsoft Azure Storage Explorer: Create a Blob Container

you can also create a Virtual Directory under it. A Virtual Directory does not actually exist in Azure until you paste, drag or upload blobs into it.

Creating the new Virtual Directory under Blob Container

SSIS Azure Blob Storage Task Examples

Let´s start with an example. In this SSIS Azure Blob Storage Task File task example, we will upload, download, move and delete files / folders from Azure Blob Storage.

1. 1. First of All, Drag and Drop Azure Blob Storage Task from SSIS Toolbox and double click it to edit
      

      Drag and Drop Azure Blob Storage Task

   2. The following options are available in Azure Blob Storage Task:
      

      Options to upload, download files in SSIS

   3. You can also overwrite files and folder, skip all, overwrite if source is older, overwrite if the file size is different, rename files and more:
      

      Overwrite files in SSIS

   4. Let’s select the Download Azure file(s) to local directory. Now select/add The Source path, Check Recursive Scan to get all the sub files and sub folders and it’s files and add Target Path, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
      

      SSIS Azure Blob Storage Task – Download files from Azure Blob Storage

   5. That’s it now execute the package and it will download all the files and sub folders from the Azure Blob Storage Task to Local File Machine.

1. Now Let’s configure the task to upload the local directory files and folder to Azure Blob Storage like below and execute the package. And also in this example i created the variable and stored target folder path in it and used that variable Target Path Variable.
   

   SSIS Azure Blob Storage Task – Upload files from Azure Blob Storage

2. Let’s configure the task to delete Azure Blob Storage files like below and execute the package.
   

   SSIS Azure Blob Storage Task – Delete files from Azure Blob Storage

3. Let’s configure the task to move Azure Blob Storage files to another directory like below and execute the package.
   

   SSIS Azure Blob Storage Task – Move files from Azure Blob Storage

4. That’s it like upper examples you can configure Azure Blob Storage Task to use it’s other options also.
5. Let’s see other SSIS Azure Blob Storage Task examples in below sections

Working with advanced filters

In order to understand filters, we will create an example for you. This time we will create SSIS Azure Blob Storage task example to show how to exclude some files from the downloading process.

1. In addition, it is also possible to add advanced filters to the SSIS Azure Blob Storage task to exclude or include files and folders. The following example shows how to use the Advanced filters. In the Azure Blob Storage Task, go to the Advanced Filter page and in Exclude RegX we will write \.exe$ to exclude the file with exe extension to be downloaded. You can use RegX (regular expressions) in the advanced filter:
   

   Advanced filters

Working with wildcards in SSIS Azure Blob Storage task (Multiple files only)

In order to understand wildcards, the new example will show how to work with Wildcards will help you to work with multiple files or folder with common characteristics.

1. The following example shows how to copy all the files will all the extensions in the source folder. We use the *.*, which means that any file name with any extension will be downloaded:
   

   Working with Wildcards

2. If we want to download all the files with extension txt, use the following Path:
   

   /source/*.txt;

3. Another example would be to download all the files with the following names: file1.txt, file2.txt and file3.txt.  We could use the following Path:
   

   /source/file?.txt

Working with SSIS expressions and variables in Azure Blob Storage Task

In order to understand expressions, the following example will show how to work with SSIS expressions combined with variables in the SSIS Azure Blob Storage task. In this example we will delete files in Azure Blob Storage using variables and expressions.

1.  First, we will first create a variable in SSIS. In this example the file name is MyFile and the Value file1.
   

   Create an SSIS variable

2. As a second step, we will write in path the value \destination\{{User.MyFile}}.txt. This will concatenate the folder destination with the variable MyFile. The task will delete the file1.txt from the Azure Blob Storage folder.
   

   Using SSIS expressions

3. The next example will show how to create a Azure Blob Storage file using expressions and variables in SSIS:
   

   Create file

4. Content to write is the content of the file. In Target, go to path and write the following: For more information on placeholders check this post and also check this post.
   

   /destination/{{User::FileName,MM-dd-yyyy}}

5. We are using a variable in SSIS named FileName of type DateTime. We are concatenating the Azure Blob Storage folder destination with the variable. As you can see, the expressions with ZappySys are very simple.

Example to verify if file exists in SSIS Azure Blob Storage task

Finally, we will verify if a file in Azure Blob Storage exists or not using the SSIS Azure Blob Storage task.

1. In order to start, we will use the SSIS variables, create a variable of Boolean type named FileExists. In Storage Action, select Get Azure file exist status. In path, select the path of the file to verify if it exists or not.
   

   SSIS Azure Blob Storage Task – Check File Exist in Azure Blob Storage

2. Secondly, we will use the ZS Logging Task to check the value of the variable File Exists. This task will show the value of the variable in the log. Drag and drop the task and join it with the ZS Azure Blob Storage Task:
   

   The SSIS Log task

3. In order to get the value, we will insert it in the Logging Task. To do that, insert the Variable and select the FileExists variable:
   

   Add the variable in the task logging

4. Finally, run the package and verify the Output (you can use View and Output to see the output) and check the value of the Logging Task. If the file exists, the value will be true. If it does not exist, the value will be false.
   

   Verify the variable in Output

How to loop through Azure Blob Storage files in SSIS

You can also loop through files to perform custom action file by by. Azure Blob Storage Task support Action to get File List as ADO.net Recordset

To get File list from Azure Blob Storage Server perform the following steps.

1. From Storage Action select Get Azure files as ADO.net DataTable
   
   

   Get files as ADO.net DataTable

    

2. Create new Azure Blob Storage Connection by clicking New next to connection dropdown.
3. Select Source Path (e.g  \myroot\folder\*.txt  )
4. Select or create new SSIS Variable (e.g. varFiles ) which will hold file list result (Must be Object datatype variable).
5. Click OK to close.
6. Create 3 more SSIS variables to hold File Path (String), Name (String) and Size (Int32). We will use in the next step.
7. Now drag SSIS ForEach Loop Container Task from SSIS Toolbox
8. Edit Loop Task and Select Foreach ADO Enumerator from the Enumerator dropdown
9. Select variable (e.g. varFiles)
   
   

   Loop through files in SSIS using ForEach Loop Task (use ADO.net Recordset variable)

10. Click on Variable Mappings Tab and configure like below.
    

    Variable Mappings for ForEach Loop Task – Loop through files in a folder using SSIS

11. Drag ZS Logging Task and place inside Loop and Display variables
12. That’s it

For more information to loop through files on local machine using SSIS check this article.

Conclusion

To conclude, we can say that working with Azure Blob Storage is now very simple. In this article, we looked at SSIS Azure Blob Storage Task examples on how to upload, download, delete files and folders. We used variables, wildcards and expressions. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here. We saw different SSIS Azure Blob Storage Task examples to understand how to use the task in different scenarios.

Thanks for your time.

References

• About SSIS PowerPack
• Integration Services (SSIS) Expressions
• Help File : Azure Storage Task

The post SSIS Azure Blob Storage Task Examples (Download, Upload, Delete files / folders) appeared first on ZappySys Blog.

In our previous blog we saw how to load data into Azure Blob Storage. Now in this blog, we will see How to read Azure Blob Storage Files in SSIS (CSV, JSON, XML Format files). To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like perform other operations on Azure Blob Storage Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on how to Read Azure Blob Storage files in SSIS using following tasks / sources.

Components Mentioned in this article

	Azure Blob CSV File Source
	Azure Blob CSV File Destination
	Azure Blob XML File Source
	Azure Blob JSON File Source

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Download and install Microsoft Azure Storage Emulator
4. Download and install Microsoft Azure Storage Explorer
5. Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

• Storage account (You can access data objects in Azure Storage through a storage account.
  For more information, click here.)
• the containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
  All blobs lie within a container, Note: Container name must be lowercase)
• the blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
  See the below-attached diagram. It shows the relationship between these resources.
       You can also use Azure Storage Explorer on your Local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Blob Source for CSV/JSON/XML File that will help you in reading CSV, JSON and XML Files from Azure Blob to the Local machine, Upload files(s) to Azure Blob Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to download files from Azure Blob Storage.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of local Azure Storage Emulator.

Setup Azure Storage client tools

1. Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Emulator Physical Location

2. If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Explorer Physical Location

Create an Azure Blob Storage Container

For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).

Microsoft Azure Storage Explorer: Create a Blob Container

you can also create a Virtual Directory under it. A Virtual Directory does not actually exist in Azure until you paste, drag or upload blobs into it.

Creating the new Virtual Directory under Blob Container

Read Azure Blob Storage Files in SSIS (CSV, JSON, XML)

Let´s start with an example. In this SSIS Azure Blob Source for CSV/JSON/XML File task example, we will read CSV/JSON/XML files from Azure Blob Storage to SQL Server database.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Drag and Drop relevant Azure Blob Source for CSV/JSON/XML File Task from the SSIS Toolbox.
   

   Add Azure Blob Source Tasks

3. Create a connection for Azure Blob Storage Account.
   

   Create Azure Storage Connection

4. We can also connect the Microsoft Azure Storage emulator also like this.
   

   Connection Form of Azure Blob Storage Account

5. Select the relevant single file to read from Azure Blob Storage in their relevant source of CSV/JSON/XML File Task.
   

   Select File From Azure Blob Storage

6. We can also read the multiple files stored in Azure Blob Storage using wildcard pattern supported e.g. dbo.tblNames*.csv / dbo.tblNames*.json / dbo.tblNames*.xml in relevant source task
   

   Use wildcard pattern .* to read multiple files data

7. We can also read the zip and gzip compressed files also without extracting it in the specific Azure Blob Source for CSV/JSON/XML File Task.
   

   Reading zip and gzip compressed files (stream mode)

8. That’s it, we are ready to load this file(s) data into the SQL Server.

Load Azure Blob Storage Files data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Read / Write data to Azure Data Lake Storage Gen 2 / Gen 1 (CSV / XML / JSON)

Check the below articles if you like to know more about how to write to Azure Blob Storage.

Article#1

https://community.zappysys.com/t/how-to-read-write-from-azure-data-lake-storage-gen2-in-ssis/125

Article#2

SSIS Data Load – SQL Server to Azure Blob (Split Files, GZip)

Conclusion

Above all, in this blog, we learned how to Read Azure Blob Storage Files in SSIS. We used Azure Blob Source for CSV File, Azure Blob Source for JSON File and Azure Blob Source for XML File to read the file(s) from Microsoft Azure Blob Storage and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• • Azure Blob Source for CSV File.
  • Blob Source for JSON File
  • Azure Blob Source for XML File
  • Microsoft Azure Blob Storage
  • Help File: Documentation of Azure Blob Source for CSV File, Azure Blob Source for JSON File and Azure Blob Source for XML File

The post Read Azure Blob Storage Files in SSIS (CSV, JSON, XML) – Gen2 / Gen1 appeared first on ZappySys Blog.

In this post we will see how to set Metadata key/value in Azure Blob Storage Task and Amazon Storage Task (S3 Task). We will also see how to set them dynamically at runtime (Use SSIS Expression).

Set Amazon S3 Task Metadata (Key / Value Pairs)

Here is how you can set Metadata in Amazon S3 Task.

Setting Amazon S3 Metadata (Key/Value Pairs) in SSIS Amazon Storage Task

Set Azure Blob Storage Task Metadata (Key / Value Pairs)

Just like S3 Task you can use same UI in Azure blob storage task too.

Set Metadata Key/Value Dynamically (SSIS Expression)

So far you have set metadata using static value. What if you have to set Dynamically at runtime?

Basically Metadata is passed as XML Format in the task like below.

<?xml version="1.0" encoding="utf-8"?>
<ArrayOfStringKeyValue xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<StringKeyValue>
		<Key>x-amz-key1</Key>
		<Value>AAAAAAA</Value>
	</StringKeyValue>
	<StringKeyValue>
		<Key>x-amz-key2</Key>
		<Value>BBBBBBB</Value>
	</StringKeyValue>	
</ArrayOfStringKeyValue>

Whate we will do is take above format and make it dynamic using SSIS Expression.

Perform the following steps for that.

1. Select your Amazon Storage Task or Azure Storage Task and right click > Select Properties.
2. In the Properties Grid check Property name MetadataKeyValue. Copy XML Value from there (We will use it in expression)
3. Now double click Amazon Storage Task or Azure Storage Task
4. Click on Expression Button
5. Select MetadataKeyValue property and for Value enter below expression… Use previously copied XML text  and change Value part with your own SSIS variable. In below example we are passing Just one Metadata Key/Value pair ( Name=x-amz-meta-filename  , Value=Coming from @[User::MyFileNameVariable] )
   Note that we have escaped double quote with

   \"

     and replaced static value with

   <Value>" + @[User::MyFileNameVariable] + "</Value>

    .
   

   "<?xml version=\"1.0\" encoding=\"utf-8\"?>"
   + "<ArrayOfStringKeyValue xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\">"
   + "	<StringKeyValue>"
   + "		<Key>x-amz-meta-file-name</Key>"
   + "		<Value>" + @[User::MyFileNameVariable] + "</Value>"
   + "	</StringKeyValue>"
   + "	<StringKeyValue>"
   + "		<Key>x-amz-meta-something</Key>"
   + "		<Value>BBBBBBB</Value>"
   + "	</StringKeyValue>"	
   + "</ArrayOfStringKeyValue>"

   

   Set Amazon S3 File Metadata Key/Value dynamically

6.  Click OK and that’s it.

The post Setting Metadata key / value in SSIS Amazon S3 / Azure Storage Task appeared first on ZappySys Blog.

In this blog post you will see how easy it is to load large amount of data from SQL Server to Azure Blob Storage using SSIS. We will export / compress data to multiple files.

For demo purpose we will use SQL Server as relational source but you can use same steps for any database engine such as Oracle, MySQL, DB2. In this post we will use Export CSV Task and Azure Blob Storage Task  to achieve desired integration with Azure Blob with drag and drop approach. You can also export JSON or XML data to Azure Blob using same techniques (Use Export JSON Task  or Export XML Task ).

Our goal is to achieve following things

• Extract large amount of data from SQL Server Table or Query and export to CSV files
• Generate CSV files in compressed format (*.gz) to speedup upload and save data transfer cost to Azure
• Split CSV files by row count
• Upload data to Azure Blob using highly parallel manner for maximum speed

There are three different ways you can achieve data export to Azure Blob using SSIS.

1. Method-1 (Fastest): Use two step process (First export SQL Server data to local files using Export Task and then upload files to Azure using  Azure Blob Storage Task  )
2. Method-2 (Slower): Use Export Task with Azure Blob Connection as Target rather than save to Local files.
3. Method-3 (Slower): Use Data flow components like Azure Blob Destination for CSV  (for JSON / XML  use Method1 or Method2)

Each method has its own advantage / disadvantage. If you prefer to upload / compress / split large amount of data then we recommend Method#1 (Two steps). If you have not very huge dataset then you can use Method#2 or Method#3. For Last method you can only use CSV export option (we don’t have JSON/ XML Destination for Azure Blob yet – we may add in future)

Screenshot of SSIS Package

Extract SQL Server Data to CSV files in SSIS (Bulk export) Split / GZip Compress / upload files to Azure Blob Storage

Method-1 : Upload SQL data to Azure Blob in Two steps

In this section we will see first method (recommended) to upload SQL data to Azure Blob. This is the fastest approach if you have lots of data to upload.  In this approach we first create CSV files from SQL Server data on local disk using SSIS Export CSV Task. After that in second step we upload all files to Azure Blob using SSIS Azure Storage Task.

Step-1: Configure Source Connection in Export CSV Task

To extract data from SQL Server you can use Export CSV Task. It has many options which makes it possible to split large amount of data into multiple files. You can specify single table or multiple tables as your data source.

For multiple table use vertical bar. e.g.  dbo.Customers|dbo.Products|dbo.Orders. When you export this it will create 3 files ( dbo.Customers.csv , dbo.Products.csv, dbo.Orders.csv )

Steps:

1. Drag ZS Export CSV Task from Toolbox
2. Double click task to configure
3. From connection drop down select New connection option (OLEDB or ADO.net)
4. Once connection is configured for Source database specify SQL Query to extract data as below
   

   Export SQL Server Table or Query as CSV file (Bulk export in SSIS)

5. Now go to target tab. Here you can specify full path for file. e.g. c:\ssis\temp\azure\cust.csv

Step-2: Compress CSV Files in SSIS ( GZIP format – *.gz )

Above steps will export file as CSV format without splitting or compression. But to compress file once exported you can go to Target tab of Export CSV Task and check [Compress file to *.gz format] option.

Compress exported SQL Server data files to GZip ( *.gz) in SSIS Export CSV Task

Step-3: Split CSV files by row count or data size in SSIS

Now lets look at how to split exported CSV files into multiple files so we can upload many files in parallel. Goto Split Options and check [Enable Split by Size/Rows]

Using SSIS Split Exported CSV files (Split by row count or size)

Step-4: Upload CSV files to Azure Blob – Using multi threaded option

Now final thing is use Azure Storage Task to upload files to Azure.

Steps:

1. Drag ZS Azure Storage Task from SSIS toolbox
2. Double click Azure Storage Task to configure it
3. Specify Action = UploadFilesToAzure
4. Specify Source file path (or pattern) e.g. c:\SSIS\temp\azure\*.*
5. Now in the Target connection dropdown click [New]
6. When Connection UI opens Enter your Account, Secret Key (Leave all other parameters default if you not sure)
7. Click Test and close connection UI
8. On the Target path on Azure Storage Task enter your bucket and folder path where you want to upload local files. For example your container name is bw-east-1 and folder is sqldata then enter as below
   bw-east-1/sqldata/
9. Click ok and Run package to test full package

Method-2 : Upload SQL data to Azure Blob without local stage (One step)

Now let’s change previous approach little bit to send SQL server data directly to Azure Blob without any Landing area on local disk.  Export CSV Task , Export JSON Task  and Export XML Task all of them supports Azure Blob / Azure Blob and Secure FTP (SFTP) connection as target (Only available in Pro Edition). We will use this feature in following section.

This approach helps to avoid any local disk need and it may be useful for security reason for some users. However drawback of this approach is, it wont use parallel threads to upload large amount of data like previous method.

Following change will be needed on Export task to upload SQL data directly to Azure / FTP or Azure storage.

Export SQL data to multiple files to Azure Blob, Amazon S3, Secure FTP (SFTP) in Stream Mode using SSIS. Configure Compress GZip, Overwrite, Split Options

Method-3 : Using Azure Blob destination – Generate Azure Blob file from any source

Now let’s look at third approach to save data from any SSIS Source to Azure Blob file. Advantage of this approach is you are not limited to few source options provided by Export CSV Task. If you have complex data transformation needed in Data Flow before sending data to Azure then use this approach.  We will use Azure Blob Destination for CSV as below

1. Drag SSIS Data flow task from toolbox
2. Create necessary source connection (e.g. OLEDB connection)
3. Create Azure Blob Connection (Right click in Connection Managers panel in bottom and click New connection and select ZS-Azure-STORAGE type )
4. Once connection managers are created Go to data flow designer and Drag OLEDB Source
5. Configure OLEDB Source to read desired data from source system (e.g. SQL Server / Oracle)
6. Once source is configured drag ZS Azure Blob CSV File Destination from SSIS toolbox
7. Double click Azure Destination and configure as below
   1. On Connection Managers tab select Azure Connection (We created in earlier section).
   2. Properties tab configure like below screenshot
   3. On Input Columns tab select desired column you like to write in the target file. Your name from upstream will be taken as is for target file. So make sure to name upstream columns correctly.
   4. Click OK to save UI
8. Execute package and check your Azure Bucket to see files got created.

Loading SQL Server data into Azure Container Files (Split, Compress Gzip Options) – SSIS Azure Blob CSV File Destination

Conclusion

In this post you have seen how easy it is to upload / archive your SQL Server data (or any other RDBMS data) to Azure Blob Storage in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to integrate SQL Server and Azure Blob using SSIS.

The post SSIS Data Load – SQL Server to Azure Blob (Split Files, GZip) appeared first on ZappySys Blog.

In this blog, we will show how to download files from Azure Blob Storage using SSIS. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Download and install Microsoft Azure Storage Emulator
4. Download and install Microsoft Azure Storage Explorer
5. Finally, do not forget to install ZappySys SSIS PowerPack

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

• storage account (You can access data objects in Azure Storage through a storage account.
  For more information, click here.)
• the containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
  All blobs lie within a container, Note: Container name must be lowercase)
• the blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
  See the below-attached diagram. It shows the relationship between these resources.
       You can also use Azure Storage Explorer on your Local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Blob Storage Task that will help you in Download File from Azure Blob to the Local machine, Upload files(s) to Azure Blob Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to download files from Azure Blob Storage.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of local Azure Storage Emulator.

Setup necessary client tools

1. Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Emulator Physical Location

2. If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Explorer Physical Location

Create Blob Container

For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).

Microsoft Azure Storage Explorer: Create Blob Container

you can also create a Virtual Directory under it. A Virtual Directory does not actually exist in Azure until you paste, drag or upload blobs into it.

Creating the new Virtual Directory under Blob Container

Download Files from Azure Blob Storage

Let´s start with an example. In this SSIS Azure Blob Storage task example, we will show how to download files from Azure Blob Storage to local folders.

1. First of All, Goto Control Flow Section, Drag and Drop ZS Azure Blob Storage Task from the SSIS Toolbox.
   

   Drag and Drop Azure Blob Storage Task

2. Double click on that Task to Edit the Configurations. You can see the Storage Action DropDown. It has many actions listed. you can do all that things with the use of this Task also, Create a connection
   

   Azure Blob Storage Task with Listing of All Storage Action

   

   How to Set Connection: Create New Connection

3. Connection Window: Configure the necessary settings and hit Test Connection button
   

   Connection Form of Azure Blob Storage Account

4. Now, Set the Path Variable by Selecting Existing Variable OR Creating a New Variable.
   

   Select Path Variable Option

5. Set Value in Variable: Specify FileName along with the Path. Select that Variable in Source Path Variable.
   

   Set Variable by giving File Name with File Path

6. Now configure the Target Settings: Select Path AccessMode.
   

   Select Path Access Mode in Target Section

7. We are selecting Connection as Path AccessMode. Creating new Connection:
   

   Specify Usage Type in file Connection Manager

8. After that just Specify the Target File with File Name and Path.
9. Then just select the OverWrite Action in the Target Section:
   

   Select Overwrite Action in Target Section

10. you can mark checked the checkbox “Delete source file after successful transfer” If you want to delete the file which you are downloading from Blob Container after the successful downloading of that file.
11. You can also specify the Advanced Filter & Sorting :
    

    Specify Advanced Filter and Sorting If you have requirements

12. After all the configurations settings are done, you can execute the Package.
    

    A Package was Executed and the File Downloaded successfully

Conclusion

In this blog, we learned how to Download files from Azure Blob Storage using SSIS PowerPack ZS Azure Blob Storage Task in a very simple way. You can achieve many more functionalities with the use of this Task. Check our Blog articles on Azure Blob Storage to find out what Azure Blob Storage task is capable of more.

References

Finally, you can use the following links for more information about Azure Blob Storage Task:

• Introduction to Microsoft Azure Storage
• How to: View, copy, and regenerate storage access keys
• Using the Azure Storage Emulator for Development and Testing
• Lending Page: SSIS Azure Blob Storage Task you can also find Tutorial Video here
• Help File: Documentation of SSIS Azure Blob Storage Task

The post How to download files from Azure Blob Storage in SSIS appeared first on ZappySys Blog.

In this short article, you will learn how to write Regular expressions in SSIS (i.e. Regex) and what tool to use to test them. You will also find helpful resources on how to write more sophisticated expressions and learn more about them. For demo purposes, we will use FREE SSIS Regex Parser Task to parse and extract the text using Regex.

You can use Regular expressions in several SSIS PowerPack connectors:

• SSIS Regex Parser Task (FREE),
• Azure Blob Storage Task,
• Advanced File System Task,
• Amazon S3 Storage Task and others.

Writing Regular Expressions in SSIS

Based on the SSIS component you use it will include the filename in filtering options or match the text: See below Syntax to write Regex in ZappySys tools. We support additional construct {{X, Y}} at the end of Regex to control two parameters. This additional construct is useful for data extracted from matching regex. If you want to Test patterns and not worry about data extraction then no need to use last {{

Occurance_Index

Group_Index_Or_Name

Syntax:

<your Regular Expression>[{{Occurance_Index|*[,Group_Index_Or_Name]}}]

Where:

Occurance_Index

Group_Index_Or_Name

How to name a group?

E.g. (\w+)@(?<domain>\w+.com)

How to use the group name in the match extract?

E.g. (\w+)@(?<domain>\w+.com){{0,domain}}

Example Input:

Let’s assume we have the following input text. We will test various Expressions.

Customer => AAA
Email => aaa@google.com
Phone => 101-222-3333
========
Customer => BBB
Email => bbb@yahoo.com
Phone => 102-222-3333
========
Customer => CCC
Email => ccc@hotmail.com
Phone => 103-222-3333
========
Customer => DDD
Email => ddd@outlook.com
Phone => 104-222-3333

Sample Regex Expressions

(.*)

More Regular Expression Examples

Regex Examples (Using SSIS Regular Expression Parser Task)

Here is an example how Regex

(.+)_(.+)_(.+){{1,2}}

Using Regular Expressions in SSIS Regex Parser Task (Extract Groups)

Using Groups / Occurrence Index

Some tasks like SSIS Regex Parser Task (FREE) supports extracting value from specific occurrence and specific part of matched pattern using special syntax at the end of your pattern (see below).

Your Regex Pattern Here{{OccuranceIndex,GroupIndex}}

Where :
OccuranceIndex is 0 based (0=extract first occurrence)
GroupIndex is 0 based (0=extract first matching group from pattern. First group is always entire text. )

See above screenshot in previous section for example.

Tools

The best tool we’ve found to write and test Regex is Regex Hero (will require IE with Silverlight if you want to use it online, in the browser):

Another great site for Regex testing is https://regex101.com (Works in any browser unlike previous one)

and few more sites as below

http://www.regexr.com/
http://www.regexlib.com/
http://www.regular-expressions.info/

Resources

Regular Expressions cheat-sheet to hang on the wall

Regular Expressions quick reference

The post Using Regular Expressions in SSIS appeared first on ZappySys Blog.

Introduction

Sometimes you may need to make an advanced filtering on file path, file date, size etc in SSIS Amazon S3 Task or SSIS Azure Blob Storage Task . For example, you have a directory “Customers“, which contains client directories. Each of them contains a directory named “Invoices“, which is partitioned by year. E.g. “Customers/Microsoft/Invoices/2015”. So if you want to get all invoices of all customers of all years, use Advanced Filter & Sorting option and configure Where clause to use wildcards (*) like this:

SSIS Amazon/Azure Storage Task – To get all invoices of all customers of all years, use this Where clause: FullPath LIKE ‘Customers/*/Invoices/*’

Using Advanced Search – Regex / SQL Expression (Filter by file date, size, name)

Sometimes it’s necessary to filter file based on advanced attributes such as file Age (How old), File Size, Multiple File Patterns (e.g. Delete *.zip, *.gz, *.rar in one action)

You can check this link or this link for more information. For Azure Blob Storage Task its almost same UI.

You can use Advanced Tab to filter further. First supply primary filter on Source Tab (e.g. mybucket/*.* ) and then supply your advanced filter (e.g. you can exclude *.exe and *.msi file from your delete list as below screenshot, also only delete files between 1000 and 50000 bytes in file size).

Select / Delete Amazon S3 File (Multiple) using Wildcard pattern

Advanced Search in SSIS Amazon S3 Task – Use Regular Expression Patterns, SQL like expression for Sort / Filter

//////////////////////////////
Examples:
//////////////////////////////
AgeInDays < 5  //Get files created in last 5 days
LastEditInDays > 3 //Get files modified before 3 days
AgeInDays > 10 AND LastEditInDays < 3  //Get files created before 10 days and modified in last 3 days
Size < 1024  //Get files where File size is less than 1024 bytes
CreationDate BETWEEN #2015-01-01# AND #2015-01-01#  
Name IN ('File1.txt', 'Download.msi')
FullPath LIKE 'mybucket\somefolder\month_*')
Extension IN ('.txt','.csv')
LastModifiedDate > #2013-01-01T16:00:00#
IsReadOnly = True
/////////////////////////////////////////
//Supported Columns for expression:
/////////////////////////////////////////
Name, FullPath, Extension, FolderPath, AgeInDays, LastEditInDays, Size, CreationDate, CreationDateUtc, LastModifiedDate, LastModifiedDateUtc, IsReadOnly

Select Amazon S3 files older than X days

If you like to select / delete files older than certain days then use below technique.

AgeInDays > 10

1. First Specify primary search criteria on Source tab
2. Go to Advanced Filter Tab and specify Following Pattern in Where Textbox (For example delete Files older than 10 days )

The post Filter by file size, name, date for Amazon / Azure Storage SSIS Task appeared first on ZappySys Blog.

Azure and AWS both are most popular Cloud Platforms. In this blog post we will learn how to copy or move Amazon S3 files to Azure Blob Storage without any coding or scripting  (AWS to Azure File Copy / Migration Scenario). To achieve this objective we will use following Drag and Drop SSIS  Tasks (i.e. Microsoft SQL Server Integration Services – ETL Platform for SQL Server). Following components are highly optimized for parallel copy/multi threading with secure connection (client side/serverside encryption).

SSIS Azure Blob Storage Task

SSIS Amazon Storage Task (AWS S3)

Concept : Fast Server Side Copy in Azure (Copy files into Azure Blob Storage)

Azure provides unique feature called server side file copy. Using this feature you can load or copy files into Azure Blob Storage without landing data to your local machine. As of now (March 2017) this type of feature is still missing in other Cloud Platform such as Amazon AWS and Google Cloud Platform – GCP.

Using server side copy feature in Azure… you can achieve following scenarios in SSIS without coding

• Move / Copy files from Amazon S3 to Azure Blob Storage
• Move / Copy files from Google Cloud Platform (GCP) to Azure Blob Storage
• Copy files from any public URL to Azure Blob Storage (Assuming URL doesn’t require credentials or its intranet URL)

In first two scenarios you need to have Source File URL using Authentication information inside URL (This is called Pre-Signed URL e.g. http://mycloud.com/myfile.zip?SIGNATURE-GOES-HERE ). All major Cloud Providers Support Pre-Signed URL so you can securely share files  with other without sharing your actual Credentials. You can configure Pre-Signed URL to expires in certain time frame (Check your Cloud API documentation). If your file doesn’t need credentials to access then you don’t have to worry about Pre-Signed URL. Your source can be file URL  (E.g. https://mysite.com/downloads/fil1.zip)

Pre-Signed URL for Google Cloud (GCP)
Here is more information how to get Pre-Signed URL for Google Cloud Platform (GCP) – Use gsutil command line
Pre-Signed URL for Amazon AWS (S3 file)
To create pre-signed URL for AWS S3 files you can use SSIS Amazon Storage Task and use Get Pre-Signed URL Action (Described in the below section). This action supports creating pre-signed URLs for multiple files using wildcard (e.g /mybkt/*.zip)  or you can get single pre-signed URL. If you use pattern search then you will get DataTable back … which can be loop through using ForEach Loop task (Loop ADO Recordset option) . If you do not use wild card then only one URL returned in string format.

Prerequisites

Before we look at next section make sure following prerequisites are met

1. You have obtained Amazon S3 Access Key and Secret Key to access desired files
2. You have obtained Azure Storage Account Name and Account Key to access desired Blob Container. If you don’t have access to Azure then you can download Azure Storage Emulator for testing Purpose. Check this.
3. You have basic knowledge of SSIS. If you dont then search for SSIS tutorial and there are many blogs / tutorials to get started 🙂

Step-By-Step – Create SSIS Package – Copy Amazon S3 Files to Azure Blob Storage

Now lets look at how to copy Amazon S3 files to Azure in few clicks. This approach doesn’t bring any data to your local system so its purely server to server copy and that is why its very fast and secure.

1. First Download and Install SSIS PowerPack
2. Once you install SSIS PowerPack. Create a new Sample SSIS Project and open Package
3. From SSIS Toolbox Drag ZS Amazon Storage Task and drop on Control Flow surface. Rename it to Get S3 File Urls
4. Double click S3 task to edit. From Action dropdown select (Get Amazon S3 Files Pre-Signed Url Option)
5. Click New next to the Connection Dropdown to create Amazon Storage Connection. Enter your credentials and bucket region and Click Test. If you are not sure about your bucket region then check this article.
   

   SSIS Amazon Storage Task – Create Pre-Signed URLs for multiple files stored in S3

6. Now click on Advanced Tab and check Use region specific endpoint option
   

   SSIS Amazon S3 Connection – Region specific endpoint option

7. Once Test connection is green you can click OK to save connection.
8. Once you back to Amazon Storage Task UI… Click on browse next to S3 file path. Her you can select one file or enter pattern to create URLs for multiple files.
9. To save pre-signed URL(s) select Variable. If variable is not there then create new one… If you enter pattern in Source path then Variable must be Object datatype. For multiple URL it will return ADO Recordset (You can use with ForEach Loop Task). If you didn’t enter pattern in source path then variable can be string datatype.
   

   SSIS Amazon Storage Task – Save Pre-Signed URL to Variable (Save Multiple URL as Recordset )

10. Now drag and drop ForEach Loop Container on the surface and configure like below
    

    SSIS ForEach Loop Container Task – Loop through Amazon S3 file URL

    

    SSIS ForEach Loop Task – Variable Mappings

11. Once Foreach Loop is configured you can drag ZS Azure Storage Task inside ForEach Loop Container
12. Double click Azure storage task and select [Copy from external file] option from Action dropdown.
13. In the source Path/URL we will enter variable name which holds current URL for ForEach Loop iteration. Easy way is click blue variable icon and select <<Insert Variable>>. Your placeholder may look like below.
    

    {{User::varCurrentS3Url}}

14. For target path … Click New Connection to create and configure new Azure Storage Connection like below. Enter Azure storage credentials and click test. If you don’t have real Azure credentials then you can use Azure Storage Emulator (Check this). Once connection is Configured click Test connection and click OK to save connection.
    

    Azure Storage Task – Configure Blob Storage connection- Serverside Copy S3 files – AWS to Azure

15. Once everything is configure you can save package and execute. This will copy S3 files to Azure Blob storage.
16. Here is execution log
    

    SSIS package "C:\SSIS\Amazon-To-Azure-Copy.dtsx" starting.
    Information: 0x0 at Get Signed URL for S3 Files, Get Signed URL for S3 Files: You are running TRIAL version. It will expire in 29 day(s)
    Information: 0x0 at Get Signed URL for S3 Files, Get Signed URL for S3 Files: Reading [PRESIGNED-URL] property for zs-eu-west-2-london-bkt2/*.*
    Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copy started: Source=https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile1.csv, Target=test1/
    Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copying https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile1.csv ...
    Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copy started: Source=https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile3.txt, Target=test1/
    Information: 0x0 at Copy S3 File to Azure, Copy S3 File to Azure: Copying https://zs-eu-west-2-london-bkt2.s3.eu-west-2.amazonaws.com/cloudfile3.txt ...
    SSIS package "C:\SSIS\Amazon-To-Azure-Copy.dtsx" finished: Success.

    

    SSIS Package Execution – Copy Multiple S3 Files from
    Amazon to Azure

How to move Amazon S3 files to Azure

In above example you saw how to copy files from Amazon S3 to Azure Blob Storage. But what if you want to Move… so once file is copied you want to delete from source. In that case its simple. Add one more Amazon S3 Task at the end so if all previous steps successful you can Issue delete files command using ZS Amazon Storage Task.

SSIS Delete Amazon S3 Files after Successful S3 to Azure Copy operation (This will mimic Move)

Download Sample SSIS Package

Click here to download sample SSIS file for SQL Server 2012, 2014, 2016

Conclusion

If you want to bring data from Amazon S3 to Azure in fastest way then technique described in this article can help you to cut down your data transfer time by several times. SSIS PowerPack comes with 45+ components and tasks which can give you drag and drop interface for your Cloud connectivity projects. It also comes with many connectors to help with JSON, XML, REST API related integration. Try SSIS PowerPack for FREE without any limitation and find out what else you can do with it.

The post SSIS – Copy Amazon S3 files from AWS to Azure appeared first on ZappySys Blog.