Download the Latest File from Azure Blob Storage using SSIS

Introduction

In our previous article, we saw SSIS Azure Blob Storage task examples. Now let’s look at how to Download the Latest File from Azure Blob Storage using SSIS. Microsoft SSIS includes the Azure Blob Storage Storage Task that allows us to upload files to Azure Blob Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Azure Blob Storage Task included in ZappySys SSIS PowerPack.

In nutshell, this post will focus on how to Download the Latest File from Azure Blob Storage.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Download and install Microsoft Azure Storage Emulator
4. Download and install Microsoft Azure Storage Explorer
5. Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

• Storage account (You can access data objects in Azure Storage through a storage account.
  For more information, click here.)
• the containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
  All blobs lie within a container, Note: Container name must be lowercase)
• the blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
  See the below-attached diagram. It shows the relationship between these resources.
       You can also use Azure Storage Explorer on your local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Blob Storage task that will allow you to access files/folders from Azure Blob to the Local machine, Upload files(s) to Azure Blob Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you Download the Latest File from Azure Blob Storage.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of the local Azure Storage Emulator.

Setup Azure Storage client tools

1. Once you have downloaded and installed a storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Emulator Physical Location

2. If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Explorer Physical Location

Create an Azure Blob Storage Container

For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).

Microsoft Azure Storage Explorer: Create a Blob Container

you can also create a Virtual Directory under it. A Virtual Directory does not actually exist in Azure until you paste, drag or upload blobs into it.

Creating the new Virtual Directory under Blob Container

Step-By-Step to Download the Latest File from Azure Blob Storage

Let´s start with an example. In this SSIS Azure Blob Storage Task File task example, we will download the latest file from Azure Blob Storage.

1. First of All, Drag and Drop Azure Blob Storage Task from SSIS Toolbox and double click it to edit
   

   Drag and Drop Azure Blob Storage Task

2. The following options are available in Azure Blob Storage Task:
   

   Options to upload, download files in SSIS

3. You can also overwrite files and folder, skip all, overwrite if the source is older, overwrite if the file size is different, rename files and more:
   

   Overwrite files in SSIS

4. Let’s select the Download Azure file(s) to local directory. Now select/add The Source path, Check Recursive Scan to get all the sub-files and subfolders and it’s files and add Target Path, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
   

   SSIS Azure Blob Storage Task – Download files from Azure Blob Storage

5. Now go to Advanced Filter & Sorting and configure it like below screen to download the latest file only
   

   Storage Task configuration to download the latest file only using sorting

6. That’s it now execute the package and it will download the latest file from the Azure Blob Storage Task to Local File Machine.
7. Sometimes you may need to make advanced filtering on file path, file date, size, etc. for that refer to this article for more details.

Conclusion

To conclude, we can say that working with Azure Blob Storage is now very simple. In this article, we looked at how to Download the Latest File from Azure Blob Storage using SSIS. We used Adnavced Filtering & sorting option. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here.

Thanks for your time.

References

• About SSIS PowerPack
• Integration Services (SSIS) Expressions
• Help File: Azure Storage Task