storage Archives | ZappySys Blog

Download the Latest File from Azure Blob Storage using SSIS

ZappySys — Sat, 28 Dec 2019 06:53:57 +0000

Introduction

In our previous article, we saw SSIS Azure Blob Storage task examples. Now let’s look at how to Download the Latest File from Azure Blob Storage using SSIS. Microsoft SSIS includes the Azure Blob Storage Storage Task that allows us to upload files to Azure Blob Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Azure Blob Storage Task included in ZappySys SSIS PowerPack.

In nutshell, this post will focus on how to Download the Latest File from Azure Blob Storage.

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
Download and install Microsoft Azure Storage Emulator
Download and install Microsoft Azure Storage Explorer
Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

Storage account (You can access data objects in Azure Storage through a storage account.
For more information, click here.)
the containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
All blobs lie within a container, Note: Container name must be lowercase)
the blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
See the below-attached diagram. It shows the relationship between these resources.
You can also use Azure Storage Explorer on your local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Blob Storage task that will allow you to access files/folders from Azure Blob to the Local machine, Upload files(s) to Azure Blob Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you Download the Latest File from Azure Blob Storage.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of the local Azure Storage Emulator.

Setup Azure Storage client tools

Once you have downloaded and installed a storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Emulator Physical Location
If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
Command Prompt Screen after Microsoft Azure Storage Emulator Started
Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Explorer Physical Location

Create an Azure Blob Storage Container

For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).

Microsoft Azure Storage Explorer: Create a Blob Container

you can also create a Virtual Directory under it. A Virtual Directory does not actually exist in Azure until you paste, drag or upload blobs into it.

Creating the new Virtual Directory under Blob Container

Step-By-Step to Download the Latest File from Azure Blob Storage

Let´s start with an example. In this SSIS Azure Blob Storage Task File task example, we will download the latest file from Azure Blob Storage.

First of All, Drag and Drop Azure Blob Storage Task from SSIS Toolbox and double click it to edit
Drag and Drop Azure Blob Storage Task
The following options are available in Azure Blob Storage Task:
Options to upload, download files in SSIS
You can also overwrite files and folder, skip all, overwrite if the source is older, overwrite if the file size is different, rename files and more:
Overwrite files in SSIS
Let’s select the Download Azure file(s) to local directory. Now select/add The Source path, Check Recursive Scan to get all the sub-files and subfolders and it’s files and add Target Path, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
SSIS Azure Blob Storage Task – Download files from Azure Blob Storage
Now go to Advanced Filter & Sorting and configure it like below screen to download the latest file only
Storage Task configuration to download the latest file only using sorting
That’s it now execute the package and it will download the latest file from the Azure Blob Storage Task to Local File Machine.
Sometimes you may need to make advanced filtering on file path, file date, size, etc. for that refer to this article for more details.

Conclusion

To conclude, we can say that working with Azure Blob Storage is now very simple. In this article, we looked at how to Download the Latest File from Azure Blob Storage using SSIS. We used Adnavced Filtering & sorting option. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here.

Thanks for your time.

References

The post Download the Latest File from Azure Blob Storage using SSIS appeared first on ZappySys Blog.

Download the Latest File from Amazon S3 Storage using SSIS

ZappySys — Fri, 27 Dec 2019 12:33:04 +0000

Introduction

In our previous article, we saw SSIS Amazon S3 Storage Task examples. Now let’s look at how to Download the Latest File from Amazon S3 Storage using SSIS. Microsoft SSIS includes the Amazon S3 Storage Task that allows us to upload files to Amazon S3 Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Amazon S3 Storage task included in ZappySys SSIS PowerPack.

In nutshell, this post will focus on how to Download the Latest File from Amazon S3 Storage.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from the Microsoft site).
Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
Make sure ZappySys SSIS PowerPack is installed (download it, if you haven't already).
(Optional step). Read this article, if you are planning to deploy packages to a server and schedule their execution later.

What is Amazon S3 Storage

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements. Amazon S3 is designed for 99.999999999% (11 9’s) of durability, and stores data for millions of applications for companies all around the world.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Amazon S3 Storage Task that will allow you to access files/folders from Amazon S3 Storage to the Local machine, Upload files(s) to Amazon S3 Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to Download the Latest File from Amazon S3 Storage.

Step-By-Step to Download the Latest File from Amazon S3 Storage

Let´s start with an example. In this SSIS Amazon S3 Storage Task File task example, we will download the latest file using Amazon S3 Storage Task.

First of All, Drag and Drop Amazon S3 Storage Task from SSIS Toolbox and double click it to edit
Drag and Drop Amazon S3 Storage Task
The following options are available in Amazon S3 Storage Task:
Options to upload, download files in SSIS
You can also overwrite files and folder, skip all, overwrite if the source is older, overwrite if the file size is different, rename files and more:
Overwrite files in SSIS
Let’s select the Download Amazon file(s) to local directory. Now select/add The Source path, Check Recursive Scan(*Optional) to get all the sub-files and subfolders and files, after that add Target Path if you want to, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
SSIS Amazon S3 Storage Task – Download files from Amazon S3 Storage
Now go to Advanced Filter & Sorting and configure it like below screen to download the latest file only.
Storage Task configuration to download the latest file only using sorting
That’s it now execute the package and it will download the latest file from the Amazon S3 Storage Task to Local File Machine.
Sometimes you may need to make advanced filtering on file path, file date, size, etc. for that refer to this article for more details.

Conclusion

To conclude, we can say that working with Amazon S3 Storage is now very simple. In this article, we looked at how to Download the Latest File from Amazon S3 Storage using SSIS. We used Adnavced Filtering & sorting option. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here.

Thanks for your time.

References

The post Download the Latest File from Amazon S3 Storage using SSIS appeared first on ZappySys Blog.

How to Read/Write Azure Queue Storage data in SSIS

ZappySys — Tue, 06 Aug 2019 12:47:38 +0000

Introduction

In our previous blog, we saw you how to read/write Azure Table Storage data. Now in this blog, we will see How to Read/Write Azure Queue Storage data in SSIS. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like perform other operations on Azure Blob Storage Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on How to Read/Write Azure Queue Storage data in SSIS using the following Source/Destination.

Component Mentioned in this article

	Azure Queue Storage Source
	Azure Queue Storage Destination

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
Download and install Microsoft Azure Storage Emulator
Download and install Microsoft Azure Storage Explorer
Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Storage) then you can skip Step #3

What are Azure Queues?

Azure Queue storage is a service for storing large numbers of messages. You access messages from anywhere in the world via authenticated calls using HTTP or HTTPS. A queue message can be up to 64 KB in size. A queue may contain millions of messages, up to the total capacity limit of a storage account.

Common uses

Microsoft Azure Queue Concept

Common uses of Queue storage include:

Creating a backlog of work to process asynchronously
Passing messages from an Azure web role to an Azure worker role
The Queue service contains the following components:

Getting Started

In order to start, we will show several examples. Zappysys includes an SSIS Azure Queue Storage Source that will help you in reading data of Queue and SSIS Azure Queue Storage Destination that will help you to write data in Queue. Here we are showing you is, how to Read/Write Azure Queue Storage data in SSIS.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of the local Azure Storage Emulator.

Setup Azure Storage client tools and Create Queue

Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
Azure Storage Emulator Physical Location
If you can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
Command Prompt Screen after Microsoft Azure Storage Emulator Started
Now, you have to download and install Microsoft Azure Storage Explorer and then you can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Explorer Location.
If you don’t have Azure Storage account then it’s easy to get FREE Trial Azure Storage account or use your MSDN to get credit each month which will be more than enough for real testing. If you prefer to avoid all this hassle then Microsoft provides another great way to test Azure Storage functionality is totally offline mode on your local machine. You can just download Azure Storage Emulator and start testing.
Click here to learn more about how to configure Azure Storage Emulator for Testing and Development purposes. In order to connect to Azure Storage Service from SSIS, you will need Storage AccountName and an AccessKey. Ask your SysAdmin or responsible person to provide that information to you. Here are sample Account Name and Access Key(this is just example key which may differ in your case)
```
Account Name: mystorageaccount 
Access Key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
```
For Creating a Queue, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Queues. Right-click on Queues and Select Create Queue and give a name you want.
Microsoft Azure Storage Explorer – Create Queue

Write data using ZS Azure Queue Storage Destination

In this tutorial, you will learn how to write data into Azure Queue Storage. We will use ZS DummyData Source for practice it but you can use a suitable Source Adapter. So, Let’s start, In this SSIS we will write data into Azure Queue Storage using ZS Azure Queue Storage Destination.

Once you have created Azure Storage Queue and you got valid Account Name and AccessKey (for the local emulator you don’t need this) you may proceed to create new SSIS package. In BIDS/SSDT create a new SSIS package.
From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface and double click on it too.
Drag and Drop SSIS Data Flow Task from SSIS Toolbox
Click here to create Azure Storage Connection.
From the SSIS toolbox drag and drop Dummy Data Source on the dataflow designer surface.
DummyData Source – Drag and Drop
Double click Dummy Data Source to configure it. From Template pick JsonDocuments and enter row count=100 to generate 100 JSON documents and click OK to save.
DummyData Source – Configure
Let’s drag and drop Azure Queue Storage Destination on the dataflow designer surface from the SSIS toolbox.
Now single click on the Dummy Data Source, once you see the blue arrow from the source connect it to Azure Queue Destination.
Double click Azure Queue Destination to configure it. Select the connection we have created before.
Configure Connection Manager
Click on [Component Properties] tab and select the Target Queue from the dropdown list.
Configure Component Properties
Click on [Mappings] tab and verify mapping. Map Document column from upstream to the MessageText target column.
Configure Column Mappings
Click OK to save settings.
Execute the package and verify target data by going to your Azure Queue. We use Azure Storage Explorer.
SSIS Execute Package

Microsoft Azure Explorer – Verify Queue Data

Read data using ZS Azure Queue Storage Source

Here you may proceed In BIDS/SSDT to create a new SSIS package.
Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
SSIS Data Flow Task – Drag and Drop
Double click on the Data Flow task to see the Data Flow designer surface.
From the SSIS toolbox drag and drop ZS Azure Queue Storage Source on the dataflow designer surface.
SSIS ZS Azure Queue Source – Drag and Drop
Double click Azure Queue Storage Source to configure it.
Select Azure Storage Connection we have created before and select queue from the dropdown and click on preview.
Configure Azure Queue Source
Click on OK button to save configure.
Now, you can use a suitable Destination Adapter from here. In this case, we used Trash Destination.
From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface.
SSIS ZS Trash Destination – Drag and Drop
Now single click on the Azure Queue Storage Source once you see the blue arrow from the source … connect it to Trash Destination.
Double click on ZS Trash Destination to Configure it.
SSIS ZS Trash Destination – Configure
Click on OK button to save Trash Destination configure setting UI.
Execute the package and verify source data in the data viewer.
SSIS ZS Azure Queue Source – Execute the Package
After messages read from Azure Queue, the same message cannot be read again until VisibilityTimeout. By default Queue, visibility timeout is used (-1 means use default) but you can override it on the source UI.

Conclusion

In this blog, we learn How to Read/Write Azure Queue Storage data in SSIS. We used the Azure Queue Destination for write data and Azure Queue Source to read data. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

Azure Queue Source
Azure Queue Destination
Microsoft Azure Queues
Help File: Azure Queue Source, Azure Queue Destination, Trash Destination

The post How to Read/Write Azure Queue Storage data in SSIS appeared first on ZappySys Blog.

How to Read/Write Azure Table Storage data in SSIS

ZappySys — Fri, 02 Aug 2019 13:03:21 +0000

Introduction

In our previous blog, we saw how to read/write Azure Blob Storage Files in SSIS (CSV, JSON, XML Format files). Now in this blog, we will see how to Read/Write Azure Table Storage data in SSIS. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like to perform other operations on Azure Blob Storage Files (e.g. Download, Upload, Create, Delete) then check these articles.

Component Mentioned in this article

	Azure Table Storage Source
	Azure Table Storage Destination

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
Download and install Microsoft Azure Storage Emulator
Download and install Microsoft Azure Storage Explorer
Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Table Storage?

Azure Table storage stores large amounts of structured data. The service is a NoSQL datastore which accepts authenticated calls from inside and outside the Azure cloud. Azure tables are ideal for storing structured, non-relational data. Common uses of Table storage include:

Microsoft Azure Table Storage Introduction

Storing TBs of structured data capable of serving web-scale applications
Storing datasets that don’t require complex joins, foreign keys, or stored procedures and can be denormalized for fast access
Quickly querying data using a clustered index
Accessing data using the OData protocol and LINQ queries with WCF Data Service .NET Libraries

You can use Table storage to store and query huge sets of structured, non-relational data, and your tables will scale as demand increases.

Getting Started

In order to start, we will show several examples. Zappysys includes an SSIS Azure Table Storage Source. Here we are showing you is, how to Read/Write Azure Table Storage data in SSIS.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of the local Azure Storage Emulator.

Setup Azure Storage client tools and Create Table

Once you have downloaded and installed a storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
Azure Storage Emulator Physical Location
If you can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
Command Prompt Screen after Microsoft Azure Storage Emulator Started
Now, you have to download and install Microsoft Azure Storage Explorer and then you can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Explorer Location.
If you don’t have Azure Storage account then it’s easy to get FREE Trial Azure Storage account or use your MSDN to get credit each month which will be more than enough for real testing. If you prefer to avoid all this hassle then Microsoft provides another great way to test Azure Storage functionality is totally offline mode on your local machine. You can just download Azure Storage Emulator and start testing.
Click here to learn more about how to configure Azure Storage Emulator for Testing and Development purposes. In order to connect to Azure Storage Service from SSIS, you will need Storage AccountName and an AccessKey. Ask your SysAdmin or responsible person to provide that information to you. Here are sample Account Name and Access Key(this is just example key which may differ in your case)
```
Account Name: mystorageaccount 
Access Key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
```
For Creating a Tables, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> Emulator – Default Ports(Key) –> Tables. Right-click on Tables and Select Create Table and put your table name you want.
Create Tables in Azure Table Storage Emulator

Write data using ZS Azure Table Storage Destination

In this tutorial, you will learn how to write data into Azure Table Storage. You can use ZS DummyData Source for practice it(In this case it’s from SQL Server Source). So, Let’s start with an example. In this SSIS we will write data into Azure Table Storage using ZS Azure Table Storage Destination.

Once you have created Azure Storage Table and you got valid Account Name and AccessKey (for the local emulator you don’t need this) you may proceed to create a new SSIS package. In BIDS/SSDT create new SSIS package
From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
Drag and Drop SSIS Data Flow Task from SSIS Toolbox
Double click on the DataFlow task to see DataFlow designer surface.
Here, In Visual Studio, drag and drop the OLE DB Source and Azure Table Storage Destination in the design surface and join the components with the blue arrow.
SSIS Azure Table Storage Destination – Drag and Drop
Now, we need two connection, OLE DB Connection, and Azure Storage Connection. Click here to create Azure Storage Connection.

How to create OLE DB Connection

Let’s, Right-click on Connection Managers Panel to Create OLE DB Connection, so you can use Source and Context Menu will appear, Select New OLE DB Connection from the Context Menu.
OLE DB Connection – Create New Connection
Now, click on New Button to create Connection.
OLE DB Connection – Create New Connection
Let’s Configure Connection Manager, just Follow steps one by one as we have created.
OLE DB Connection – Configure Connection
Click on the OK button to save configure Setting UI.

Write data into Azure Storage Table in SSIS

Double click on OLE DB Source for configure it.
Let’s Configure in Connection Manager, just follow the below image steps.
OLE DB Source – Configure
Click OK to Save OLE DB Source Editor UI Settings.
Now double click Azure Table Storage Destination to configure it.
On [Connection Manager] tab select Azure Storage Connection manager.
ZS Azure Table Storage Destination – Connection Manager
Click on [Component Properties] tab and select AccessMode=Table and Pick destination Table from the Tables dropdown list.

If you choose Reload option then target table is first truncated before inserting records so be careful with Reload option.

ZS Azure Table Storage Destination – Component Properties
Click on [Mappings] tab and verify mapping. Drop CustomerID and RecordID mappings and reconnect as shown below. This is just to make sure PrimaryKey columns are mapped.
ZS Azure Table Storage Destination – Column Mappings
Click OK to save settings.
Execute the package and verify the target data.
ZS Azure Table Storage Destination – Execute the Package

ZS Azure Table Storage – Verify Customers Table Data

Read data using ZS Azure Table Storage Source

In this tutorial, you will learn how to read data from Azure Table Storage in SSIS using the ZS Azure Table Storage Source adapter. In this SSIS we will read data with SQL Query and Table Mode. So Let’s begin it.

You may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
SSIS Data Flow Task – Drag and Drop
Double click on the DataFlow task to see DataFlow designer surface.
From the SSIS toolbox drag and drop Azure Table Storage Source on the data flow designer surface.
ZS Azure Table Storage Source – Drag and Drop
Now, we need an Azure Storage connection. Click here to Create Connection.

Extract data with Query Mode

Here, we are getting data using Query Mode with Dynamic expression value.
Lets, create a Variable with correct DataType and Value, use the following image.
SSIS Variables – Create Variable and Store Value
Double click Azure Table Storage Source to configure it.
Select the connection we have created before, set AccessMode to Query. You can use the following SQL Query and select Variable. Click here for more Azure Table Storage Query Examples.
```
SELECT * FROM Customers
WHERE Country = '{{User::varCountry}}'
```
ZS Azure Table Storage Source – Configure Query Mode
Click on the Preview button to see Data Preview and OK button to save configure setting UI.
From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface.
ZS Trash Destination – Drag and Drop
Now single click on the Azure Table Storage Source, once you see the blue arrow from the source … connect it to Trash Destination.
Double click on ZS Trash Destination to Configure it.
ZS Trash Destination – Configure
Click on the OK button to save Trash Destination configure setting UI.
Execute the package and verify source data in the data viewer.
SSIS ZS Azure Table Storage Source – Execute the Package

Extract data with Table Mode

So, Double click Azure Table Storage Source to configure it.
In the Azure Table Storage Source, Select the connection we have created, set AccessMode to Table and Select Table from the Tables dropdown.
SSIS ZS Azure Table Source – Configure Table Mode
Click on the Preview button to see Data Preview and OK button to save configure setting UI.
From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface.
ZS Trash Destination – Drag and Drop
Now single click on the Azure Table Storage Source, once you see the blue arrow from the source … connect it to Trash Destination.
Double click on ZS Trash Destination to Configure it.
ZS Trash Destination – Configure
Click on the OK button to save Trash Destination configure setting UI.
In the last, Execute the package and verify source data in the data viewer.
ZS Azure Table Storage Source – Execute the Package

Conclusion

After all, In this blog, we learned how to write data in Azure Table Storage in SSIS. We used the Azure Table Storage Destination for write and Azure Table Storage Source to read data from Microsoft Azure Table Storage using Query and Table mode. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

There are a few details you should use the following links for more information:

The post How to Read/Write Azure Table Storage data in SSIS appeared first on ZappySys Blog.

SSIS Amazon S3 Storage Task examples (Download, Upload, Delete Files / Folders)

ZappySys — Sat, 06 Apr 2019 10:22:55 +0000

Introduction

In this article, we will show SSIS Amazon S3 Storage task examples. Uploading files using Amazon S3 Storage Task or downloading files are common requirements in SSIS. In order to work with Amazon S3 Storage, Microsoft SSIS includes the Amazon S3 Storage Task that allows to upload files to Amazon S3 Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Amazon S3 Storage task included in ZappySys SSIS PowerPack.

Amazon S3 Storage Task can be used to perform various operations with Amazon S3 Storage objects (files and containers) (e.g. Download, Upload, Copy, Move, Rename, Delete, etc). You can also get single property of any Amazon file or get a list of Amazon files as ADO.net Table and which can be easily looped through using ForEachLoop Task. Most of the operations in this task support recursive search using wild card pattern (e.g. Myfile*.txt).

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
You have obtained Amazon S3 account access key / secret key.
Finally, do not forget to install ZappySys SSIS PowerPack

What is Amazon S3 Storage

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Amazon S3 Storage Task that will allow you to access files / folders from Amazon S3 Storage to the Local machine, Upload files(s) to Amazon S3 Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to upload, download, move and delete files / folders from Amazon S3 Storage.

SSIS Amazon S3 Storage Task Examples

Let´s start with an example. In this SSIS Amazon S3 Storage Task File task example, we will upload, download, move and delete files / folders from Amazon S3 Storage.

First of All, Drag and Drop Amazon S3 Storage Task from SSIS Toolbox and double click it to edit
Drag and Drop Amazon S3 Storage Task
The following options are available in Amazon S3 Storage Task:
Options to upload, download files in SSIS
You can also overwrite files and folder, skip all, overwrite if source is older, overwrite if the file size is different, rename files and more:
Overwrite files in SSIS
Let’s select the Download Amazon file(s) to local directory. Now select/add The Source path, Check Recursive Scan to get all the sub files and sub folders and it’s files and add Target Path, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
SSIS Amazon S3 Storage Task – Download files from Amazon S3 Storage
That’s it now execute the package and it will download all the files and sub folders from the Amazon S3 Storage Task to Local File Machine.
Now Let’s configure the task to upload the local directory files and folder to Amazon S3 Storage like below and execute the package. And also in this example i created the variable and stored target folder path in it and used that variable Target Path Variable.
SSIS Amazon S3 Storage Task – Upload files from Amazon S3 Storage
Let’s configure the task to delete Amazon S3 Storage files like below and execute the package.
SSIS Amazon S3 Storage Task – Delete files from Amazon S3 Storage
Let’s configure the task to move Amazon S3 Storage files to another directory like below and execute the package.
SSIS Amazon S3 Storage Task – Move files from Amazon S3 Storage
That’s it like upper examples you can configure Amazon S3 Storage Task to use it’s other options also.
Let’s see other SSIS Amazon S3 Storage Task examples in below sections

Working with advanced filters

In order to understand filters, we will create an example for you. This time we will create SSIS Amazon S3 Storage task example to show how to exclude some files from the downloading process.

In addition, it is also possible to add advanced filters to the SSIS Amazon S3 Storage task to exclude or include files and folders. The following example shows how to use the Advanced filters. In the Amazon S3 Storage Task, go to the Advanced Filter page and in Exclude RegX we will write \.exe$ to exclude the file with exe extension to be downloaded. You can use RegX (regular expressions) in the advanced filter:
Advanced filters

Working with wildcards in SSIS Amazon S3 Storage task (Multiple files only)

In order to understand wildcards, the new example will show how to work with Wildcards will help you to work with multiple files or folder with common characteristics.

The following example shows how to copy all the files will all the extensions in the source folder. We use the *.*, which means that any file name with any extension will be downloaded:
Working with Wildcards
If we want to download all the files with extension txt, use the following Path:
```
/source/*.txt;
```
Another example would be to download all the files with the following names: file1.txt, file2.txt and file3.txt. We could use the following Path:
```
/source/file?.txt
```

Working with SSIS expressions and variables in Amazon S3 Storage Task

In order to understand expressions, the following example will show how to work with SSIS expressions combined with variables in the SSIS Amazon S3 Storage task. In this example we will delete files in Amazon S3 Storage using variables and expressions.

ZappySys Providers simple and powerful Placeholders. You can still use SSIS expression language. Placeholders allow direct Variable Use easy to see compared to hidden SSIS Expression (C# style). Placeholders have simple embedding use of variables.

First, we will first create a variable in SSIS. In this example the file name is MyFile and the Value file1.
Create an SSIS variable
As a second step, we will write in path the value \destination\{{User.MyFile}}.txt. This will concatenate the folder destination with the variable MyFile. The task will delete the file1.txt from the Amazon S3 Storage folder.
Using SSIS expressions
The next example will show how to create a Amazon S3 Storage file using expressions and variables in SSIS:
Create file
Content to write is the content of the file. In Target, go to path and write the following: For more information on placeholders check this post and also check this post.
```
/destination/{{User::FileName,MM-dd-yyyy}}
```
We are using a variable in SSIS named FileName of type DateTime. We are concatenating the Amazon S3 Storage folder destination with the variable. As you can see, the expressions with ZappySys are very simple.

Example to verify if file exists in SSIS Amazon S3 Storage task

Finally, we will verify if a file in Amazon S3 Storage exists or not using the SSIS Amazon S3 Storage task.

In order to start, we will use the SSIS variables, create a variable of Boolean type named FileExists. In Storage Action, select Get Amazon file exist status. In path, select the path of the file to verify if it exists or not.
SSIS Amazon S3 Storage Task – Check File Exist in Amazon S3 Storage
Secondly, we will use the ZS Logging Task to check the value of the variable File Exists. This task will show the value of the variable in the log. Drag and drop the task and join it with the ZS Amazon S3 Storage Task:
The SSIS Log task
In order to get the value, we will insert it in the Logging Task. To do that, insert the Variable and select the FileExists variable:
Add the variable in the task logging
Finally, run the package and verify the Output (you can use View and Output to see the output) and check the value of the Logging Task. If the file exists, the value will be true. If it does not exist, the value will be false.
Verify the variable in Output

How to loop through Amazon S3 Storage files in SSIS

You can also loop through files to perform custom action file by by. Amazon S3 Storage Task support Action to get File List as ADO.net Recordset

To get File list from Amazon S3 Storage Server perform the following steps.

From Storage Action select Get Amazon files as ADO.net DataTable

Get files as ADO.net DataTable
Create new Amazon S3 Storage Connection by clicking New next to connection dropdown.
Select Source Path (e.g \myroot\folder\*.txt )
Select or create new SSIS Variable (e.g. varFiles ) which will hold file list result (Must be Object datatype variable).
Click OK to close.
Create 3 more SSIS variables to hold File Path (String), Name (String) and Size (Int32). We will use in the next step.
Now drag SSIS ForEach Loop Container Task from SSIS Toolbox
Edit Loop Task and Select Foreach ADO Enumerator from the Enumerator dropdown
Select variable (e.g. varFiles)

Loop through files in SSIS using ForEach Loop Task (use ADO.net Recordset variable)
Click on Variable Mappings Tab and configure like below.
Variable Mappings for ForEach Loop Task – Loop through files in a folder using SSIS
Drag ZS Logging Task and place inside Loop and Display variables
That’s it

For more information to loop through files on local machine using SSIS check this article.

Conclusion

To conclude, we can say that working with Amazon S3 Storage is now very simple. In this article, we looked at SSIS Amazon S3 Storage Task examples on how to upload, download, delete files and folders. We used variables, wildcards and expressions. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here. We saw different SSIS Amazon S3 Storage Task examples to understand how to use the task in different scenarios.

Thanks for your time.

References

The post SSIS Amazon S3 Storage Task examples (Download, Upload, Delete Files / Folders) appeared first on ZappySys Blog.

SSIS Azure Blob Storage Task Examples (Download, Upload, Delete files / folders)

ZappySys — Sat, 30 Mar 2019 09:48:44 +0000

Introduction

In this article, we will show SSIS Azure Blob Storage task examples. Uploading files using Azure Blob Storage Task or downloading files are common requirements in SSIS. In order to work with Azure Blob Storage, Microsoft SSIS includes the Azure Blob Storage Task that allows to upload files to Azure Blob Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Azure Blob Storage task included in ZappySys SSIS PowerPack.

Azure Blob Storage Task can be used to perform various operations with Azure Storage objects (blobs and containers) (e.g. Download, Upload, Copy, Move, Rename, Delete, etc). You can also get single property of any Azure blob or get a list of blobs as ADO.net Table and which can be easily looped through using ForEachLoop Task. Most of the operations in this task support recursive search using wild card pattern (e.g. Myfile*.txt).

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
Download and install Microsoft Azure Storage Emulator
Download and install Microsoft Azure Storage Explorer
Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

Storage account (You can access data objects in Azure Storage through a storage account.
For more information, click here.)
the containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
All blobs lie within a container, Note: Container name must be lowercase)
the blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
See the below-attached diagram. It shows the relationship between these resources.
You can also use Azure Storage Explorer on your Local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Blob Storage Task that will allow you to access files / folders from Azure Blob to the Local machine, Upload files(s) to Azure Blob Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to upload, download, move and delete files / folders from Azure Blob Storage.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of local Azure Storage Emulator.

Setup Azure Storage client tools

Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Emulator Physical Location
If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
Command Prompt Screen after Microsoft Azure Storage Emulator Started
Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Explorer Physical Location

Create an Azure Blob Storage Container

For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).

Microsoft Azure Storage Explorer: Create a Blob Container

you can also create a Virtual Directory under it. A Virtual Directory does not actually exist in Azure until you paste, drag or upload blobs into it.

Creating the new Virtual Directory under Blob Container

SSIS Azure Blob Storage Task Examples

Let´s start with an example. In this SSIS Azure Blob Storage Task File task example, we will upload, download, move and delete files / folders from Azure Blob Storage.

1. First of All, Drag and Drop Azure Blob Storage Task from SSIS Toolbox and double click it to edit
  Drag and Drop Azure Blob Storage Task
2. The following options are available in Azure Blob Storage Task:
  Options to upload, download files in SSIS
3. You can also overwrite files and folder, skip all, overwrite if source is older, overwrite if the file size is different, rename files and more:
  Overwrite files in SSIS
4. Let’s select the Download Azure file(s) to local directory. Now select/add The Source path, Check Recursive Scan to get all the sub files and sub folders and it’s files and add Target Path, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
  SSIS Azure Blob Storage Task – Download files from Azure Blob Storage
5. That’s it now execute the package and it will download all the files and sub folders from the Azure Blob Storage Task to Local File Machine.

Now Let’s configure the task to upload the local directory files and folder to Azure Blob Storage like below and execute the package. And also in this example i created the variable and stored target folder path in it and used that variable Target Path Variable.
SSIS Azure Blob Storage Task – Upload files from Azure Blob Storage
Let’s configure the task to delete Azure Blob Storage files like below and execute the package.
SSIS Azure Blob Storage Task – Delete files from Azure Blob Storage
Let’s configure the task to move Azure Blob Storage files to another directory like below and execute the package.
SSIS Azure Blob Storage Task – Move files from Azure Blob Storage
That’s it like upper examples you can configure Azure Blob Storage Task to use it’s other options also.
Let’s see other SSIS Azure Blob Storage Task examples in below sections

Working with advanced filters

In order to understand filters, we will create an example for you. This time we will create SSIS Azure Blob Storage task example to show how to exclude some files from the downloading process.

In addition, it is also possible to add advanced filters to the SSIS Azure Blob Storage task to exclude or include files and folders. The following example shows how to use the Advanced filters. In the Azure Blob Storage Task, go to the Advanced Filter page and in Exclude RegX we will write \.exe$ to exclude the file with exe extension to be downloaded. You can use RegX (regular expressions) in the advanced filter:
Advanced filters

Working with wildcards in SSIS Azure Blob Storage task (Multiple files only)

In order to understand wildcards, the new example will show how to work with Wildcards will help you to work with multiple files or folder with common characteristics.

The following example shows how to copy all the files will all the extensions in the source folder. We use the *.*, which means that any file name with any extension will be downloaded:
Working with Wildcards
If we want to download all the files with extension txt, use the following Path:
```
/source/*.txt;
```
Another example would be to download all the files with the following names: file1.txt, file2.txt and file3.txt. We could use the following Path:
```
/source/file?.txt
```

Working with SSIS expressions and variables in Azure Blob Storage Task

In order to understand expressions, the following example will show how to work with SSIS expressions combined with variables in the SSIS Azure Blob Storage task. In this example we will delete files in Azure Blob Storage using variables and expressions.

First, we will first create a variable in SSIS. In this example the file name is MyFile and the Value file1.
Create an SSIS variable
As a second step, we will write in path the value \destination\{{User.MyFile}}.txt. This will concatenate the folder destination with the variable MyFile. The task will delete the file1.txt from the Azure Blob Storage folder.
Using SSIS expressions
The next example will show how to create a Azure Blob Storage file using expressions and variables in SSIS:
Create file
Content to write is the content of the file. In Target, go to path and write the following: For more information on placeholders check this post and also check this post.
```
/destination/{{User::FileName,MM-dd-yyyy}}
```
We are using a variable in SSIS named FileName of type DateTime. We are concatenating the Azure Blob Storage folder destination with the variable. As you can see, the expressions with ZappySys are very simple.

Example to verify if file exists in SSIS Azure Blob Storage task

Finally, we will verify if a file in Azure Blob Storage exists or not using the SSIS Azure Blob Storage task.

In order to start, we will use the SSIS variables, create a variable of Boolean type named FileExists. In Storage Action, select Get Azure file exist status. In path, select the path of the file to verify if it exists or not.
SSIS Azure Blob Storage Task – Check File Exist in Azure Blob Storage
Secondly, we will use the ZS Logging Task to check the value of the variable File Exists. This task will show the value of the variable in the log. Drag and drop the task and join it with the ZS Azure Blob Storage Task:
The SSIS Log task
In order to get the value, we will insert it in the Logging Task. To do that, insert the Variable and select the FileExists variable:
Add the variable in the task logging
Finally, run the package and verify the Output (you can use View and Output to see the output) and check the value of the Logging Task. If the file exists, the value will be true. If it does not exist, the value will be false.
Verify the variable in Output

How to loop through Azure Blob Storage files in SSIS

You can also loop through files to perform custom action file by by. Azure Blob Storage Task support Action to get File List as ADO.net Recordset

To get File list from Azure Blob Storage Server perform the following steps.

From Storage Action select Get Azure files as ADO.net DataTable

Get files as ADO.net DataTable
Create new Azure Blob Storage Connection by clicking New next to connection dropdown.
Select Source Path (e.g \myroot\folder\*.txt )
Select or create new SSIS Variable (e.g. varFiles ) which will hold file list result (Must be Object datatype variable).
Click OK to close.
Create 3 more SSIS variables to hold File Path (String), Name (String) and Size (Int32). We will use in the next step.
Now drag SSIS ForEach Loop Container Task from SSIS Toolbox
Edit Loop Task and Select Foreach ADO Enumerator from the Enumerator dropdown
Select variable (e.g. varFiles)

Loop through files in SSIS using ForEach Loop Task (use ADO.net Recordset variable)
Click on Variable Mappings Tab and configure like below.
Variable Mappings for ForEach Loop Task – Loop through files in a folder using SSIS
Drag ZS Logging Task and place inside Loop and Display variables
That’s it

For more information to loop through files on local machine using SSIS check this article.

Conclusion

To conclude, we can say that working with Azure Blob Storage is now very simple. In this article, we looked at SSIS Azure Blob Storage Task examples on how to upload, download, delete files and folders. We used variables, wildcards and expressions. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here. We saw different SSIS Azure Blob Storage Task examples to understand how to use the task in different scenarios.

Thanks for your time.

References

The post SSIS Azure Blob Storage Task Examples (Download, Upload, Delete files / folders) appeared first on ZappySys Blog.