SSIS Azure Blob Connection Archives | ZappySys Blog

Download the Latest File from Azure Blob Storage using SSIS

ZappySys — Sat, 28 Dec 2019 06:53:57 +0000

Introduction

In our previous article, we saw SSIS Azure Blob Storage task examples. Now let’s look at how to Download the Latest File from Azure Blob Storage using SSIS. Microsoft SSIS includes the Azure Blob Storage Storage Task that allows us to upload files to Azure Blob Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Azure Blob Storage Task included in ZappySys SSIS PowerPack.

In nutshell, this post will focus on how to Download the Latest File from Azure Blob Storage.

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
Download and install Microsoft Azure Storage Emulator
Download and install Microsoft Azure Storage Explorer
Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

Storage account (You can access data objects in Azure Storage through a storage account.
For more information, click here.)
the containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
All blobs lie within a container, Note: Container name must be lowercase)
the blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
See the below-attached diagram. It shows the relationship between these resources.
You can also use Azure Storage Explorer on your local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Blob Storage task that will allow you to access files/folders from Azure Blob to the Local machine, Upload files(s) to Azure Blob Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you Download the Latest File from Azure Blob Storage.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of the local Azure Storage Emulator.

Setup Azure Storage client tools

Once you have downloaded and installed a storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Emulator Physical Location
If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
Command Prompt Screen after Microsoft Azure Storage Emulator Started
Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Explorer Physical Location

Create an Azure Blob Storage Container

For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).

Microsoft Azure Storage Explorer: Create a Blob Container

you can also create a Virtual Directory under it. A Virtual Directory does not actually exist in Azure until you paste, drag or upload blobs into it.

Creating the new Virtual Directory under Blob Container

Step-By-Step to Download the Latest File from Azure Blob Storage

Let´s start with an example. In this SSIS Azure Blob Storage Task File task example, we will download the latest file from Azure Blob Storage.

First of All, Drag and Drop Azure Blob Storage Task from SSIS Toolbox and double click it to edit
Drag and Drop Azure Blob Storage Task
The following options are available in Azure Blob Storage Task:
Options to upload, download files in SSIS
You can also overwrite files and folder, skip all, overwrite if the source is older, overwrite if the file size is different, rename files and more:
Overwrite files in SSIS
Let’s select the Download Azure file(s) to local directory. Now select/add The Source path, Check Recursive Scan to get all the sub-files and subfolders and it’s files and add Target Path, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
SSIS Azure Blob Storage Task – Download files from Azure Blob Storage
Now go to Advanced Filter & Sorting and configure it like below screen to download the latest file only
Storage Task configuration to download the latest file only using sorting
That’s it now execute the package and it will download the latest file from the Azure Blob Storage Task to Local File Machine.
Sometimes you may need to make advanced filtering on file path, file date, size, etc. for that refer to this article for more details.

Conclusion

To conclude, we can say that working with Azure Blob Storage is now very simple. In this article, we looked at how to Download the Latest File from Azure Blob Storage using SSIS. We used Adnavced Filtering & sorting option. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here.

Thanks for your time.

References

The post Download the Latest File from Azure Blob Storage using SSIS appeared first on ZappySys Blog.

Read and Write in Azure Queue Storage in SSIS

ZappySys — Sat, 13 Jul 2019 08:01:01 +0000

Introduction

In our previous blog we saw How to load data into Azure Blob Storage. Now in this blog, we will How to Read and Write in Azure Queue Storage in SSIS. To illustrate, we will use ZappySys SSIS PowerPack, Which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like perform other operations on Azure Blob Storage Files (e.g. Download, Upload, Create, Delete) then check these articles.

In a nutshell, In this post will focus on how to Read and Write in Azure Storage Queue in SSIS using following tasks/sources.

Components Mentioned in this article

	SSIS Azure Queue Storage Source
	SSIS Azure Queue Storage Destination

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
Download and install Microsoft Azure Storage Emulator
Download and install Microsoft Azure Storage Explorer
Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

Storage account (You can access data objects in Azure Storage through a storage account. For more information, click here.)
The containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
All blobs lie within a container, Note: Container name must be lowercase)
The blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
See the below-attached diagram. It shows the relationship between these resources.
You can also use Azure Storage Explorer on your Local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

What is Azure Queue Storage?

Azure Queue storage is a service for storing large numbers of messages. You access messages from anywhere in the world via authenticated calls using HTTP or HTTPS. A queue message can be up to 64 KB in size. A queue may contain millions of messages, up to the total capacity limit of a storage account.

Common uses

Common uses of Queue storage include:

Creating a backlog of work to process asynchronously
Passing messages from an Azure web role to an Azure worker role

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Queue Storage Source that will help you in reading messages from Azure Queue Storage. And SSIS Azure Queue Storage Destination that will help you in insert and delete messages from Azure Queue Storage. Here we are showing you is, How to Read and Write in Azure Storage Queue.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of local Azure Storage Emulator.

Setup Azure Storage client tools

Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Emulator Physical Location
If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
Command Prompt Screen after Microsoft Azure Storage Emulator Started
Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Explorer Physical Location

Create an Azure Blob Storage Queue

For Creating a Blob Queue, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Queues).

Microsoft Azure Storage Explorer : Create Blob Queue

And enter the appropriate name to that queue as Queue names can contain only lowercase letters, numbers, and hyphens, and must begin and end with a letter or a number. The name can’t contain two consecutive hyphens..

Microsoft Azure Storage Explorer : Create Blob Queue

Video Tutorial – SSIS Azure Queue Storage – Bulk Read, Write, Update, Delete Queue Messages

[youtube https://www.youtube.com/watch?v=3ZQcNiG_jeI]

Read Azure Queue Messages in SSIS

Let’s start with an example. We use SSIS Azure Queue Storage Source to read Azure Queue Messages.

First of all, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
Drag and Drop SSIS Data Flow Task from SSIS Toolbox
Furthermore, Drag and Drop ZS Azure Queue Source from the SSIS Toolbox.
Drag and Drop ZS Azure Queue Source
Now, double click on that Component to Edit the Configurations. Create New Connection and test it.
Azure Queue Source Connection using Microsoft Azure Storage emulator
Now select the desire Queue from where you wants to read the messages and click on preview.
Preview Azure Queue Source Messages
That’s it now we are ready to load this queue messages to another Azure Storage Queue

Write Azure Queue Messages in SSIS

Let’s start with an example. We use SSIS Azure Queue Storage Destination to insert Azure Queue Messages.

First, Drag and Drop ZS Azure Queue Destination from the SSIS Toolbox.
Drag and Drop ZS Azure Queue Destination
Double click on it to Edit the Configurations. And select the desire Azure Storage Connection.
Azure Queue Destination : Select Connection
Go to Component Properties and select the operation as insert and desire queue from the drop down where you wants to insert messages.
Azure Queue Destination : Select Queue to Insert Messages
Now on Column Mapping tab map the desire columns and click on OK button.
Azure Queue Destination : Column Mapping
That’s it now execute the package and it will read the Azure Queue messages and insert the messages to that selected queue.
Azure Queue Source and Destination Execution
Furthermore, go to MS Azure Storage Exporter and open/refresh that queue and you will find that 15 messages there.
Microsoft Azure Storage : Queues

Performance Tips

Disable Nagle in SSIS

Nagle’s algorithm is widely implemented across TCP/IP networks as a means to improve network performance. However, it is not optimal in all circumstances (such as highly interactive environments). For Azure Storage, Nagle’s algorithm has a negative impact on the performance of requests to the table and queue services, and you should disable it if possible.

Please refer to this link for more information: Click Here

To disable Nagle algorithm in SSIS we need to use Script Task at very First step of the Package.

In the Package File at the very first step add the Scrip Task and click on Edit Script.
Edit Script Task
So that it opens ScriptMain.cs Script class in another visual studio instance. Now add below Line in the Main() function and save it.
```
System.Net.ServicePointManager.UseNagleAlgorithm = false;
```
Script Task : Disable Nagle Algorithm
Now go to Script Task and click on OK , so that it’s save the script and close the class window.
Save and close Script Task
That’s it we have disabled the Nagle Algorithm now you can go to next steps. In upper case we need to add data flow task and follow the steps from the 2# step.
Add Data Flow Task

Increase Max Parallel Threads size

We can increase the thread size to improve performance further more, in the azure connection manager we can increase Max Parallel Threads size like 20 or more.

Azure Storage Connection : Max Parallel Threads

Conclusion

Above all, in this blog, we learned how to Read and Write in Azure Storage Queues. We used Azure Queue Storage to read messages from Azure Queue Storage. And Azure Queue Storage Destination to insert messages from Azure Queue Storage. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

The post Read and Write in Azure Queue Storage in SSIS appeared first on ZappySys Blog.

SSIS Azure Blob Storage Task Examples (Download, Upload, Delete files / folders)

ZappySys — Sat, 30 Mar 2019 09:48:44 +0000

Introduction

In this article, we will show SSIS Azure Blob Storage task examples. Uploading files using Azure Blob Storage Task or downloading files are common requirements in SSIS. In order to work with Azure Blob Storage, Microsoft SSIS includes the Azure Blob Storage Task that allows to upload files to Azure Blob Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Azure Blob Storage task included in ZappySys SSIS PowerPack.

Azure Blob Storage Task can be used to perform various operations with Azure Storage objects (blobs and containers) (e.g. Download, Upload, Copy, Move, Rename, Delete, etc). You can also get single property of any Azure blob or get a list of blobs as ADO.net Table and which can be easily looped through using ForEachLoop Task. Most of the operations in this task support recursive search using wild card pattern (e.g. Myfile*.txt).

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
Download and install Microsoft Azure Storage Emulator
Download and install Microsoft Azure Storage Explorer
Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

Storage account (You can access data objects in Azure Storage through a storage account.
For more information, click here.)
the containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
All blobs lie within a container, Note: Container name must be lowercase)
the blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
See the below-attached diagram. It shows the relationship between these resources.
You can also use Azure Storage Explorer on your Local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Blob Storage Task that will allow you to access files / folders from Azure Blob to the Local machine, Upload files(s) to Azure Blob Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to upload, download, move and delete files / folders from Azure Blob Storage.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of local Azure Storage Emulator.

Setup Azure Storage client tools

Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Emulator Physical Location
If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
Command Prompt Screen after Microsoft Azure Storage Emulator Started
Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Explorer Physical Location

Create an Azure Blob Storage Container

For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).

Microsoft Azure Storage Explorer: Create a Blob Container

you can also create a Virtual Directory under it. A Virtual Directory does not actually exist in Azure until you paste, drag or upload blobs into it.

Creating the new Virtual Directory under Blob Container

SSIS Azure Blob Storage Task Examples

Let´s start with an example. In this SSIS Azure Blob Storage Task File task example, we will upload, download, move and delete files / folders from Azure Blob Storage.

1. First of All, Drag and Drop Azure Blob Storage Task from SSIS Toolbox and double click it to edit
  Drag and Drop Azure Blob Storage Task
2. The following options are available in Azure Blob Storage Task:
  Options to upload, download files in SSIS
3. You can also overwrite files and folder, skip all, overwrite if source is older, overwrite if the file size is different, rename files and more:
  Overwrite files in SSIS
4. Let’s select the Download Azure file(s) to local directory. Now select/add The Source path, Check Recursive Scan to get all the sub files and sub folders and it’s files and add Target Path, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
  SSIS Azure Blob Storage Task – Download files from Azure Blob Storage
5. That’s it now execute the package and it will download all the files and sub folders from the Azure Blob Storage Task to Local File Machine.

Now Let’s configure the task to upload the local directory files and folder to Azure Blob Storage like below and execute the package. And also in this example i created the variable and stored target folder path in it and used that variable Target Path Variable.
SSIS Azure Blob Storage Task – Upload files from Azure Blob Storage
Let’s configure the task to delete Azure Blob Storage files like below and execute the package.
SSIS Azure Blob Storage Task – Delete files from Azure Blob Storage
Let’s configure the task to move Azure Blob Storage files to another directory like below and execute the package.
SSIS Azure Blob Storage Task – Move files from Azure Blob Storage
That’s it like upper examples you can configure Azure Blob Storage Task to use it’s other options also.
Let’s see other SSIS Azure Blob Storage Task examples in below sections

Working with advanced filters

In order to understand filters, we will create an example for you. This time we will create SSIS Azure Blob Storage task example to show how to exclude some files from the downloading process.

In addition, it is also possible to add advanced filters to the SSIS Azure Blob Storage task to exclude or include files and folders. The following example shows how to use the Advanced filters. In the Azure Blob Storage Task, go to the Advanced Filter page and in Exclude RegX we will write \.exe$ to exclude the file with exe extension to be downloaded. You can use RegX (regular expressions) in the advanced filter:
Advanced filters

Working with wildcards in SSIS Azure Blob Storage task (Multiple files only)

In order to understand wildcards, the new example will show how to work with Wildcards will help you to work with multiple files or folder with common characteristics.

The following example shows how to copy all the files will all the extensions in the source folder. We use the *.*, which means that any file name with any extension will be downloaded:
Working with Wildcards
If we want to download all the files with extension txt, use the following Path:
```
/source/*.txt;
```
Another example would be to download all the files with the following names: file1.txt, file2.txt and file3.txt. We could use the following Path:
```
/source/file?.txt
```

Working with SSIS expressions and variables in Azure Blob Storage Task

In order to understand expressions, the following example will show how to work with SSIS expressions combined with variables in the SSIS Azure Blob Storage task. In this example we will delete files in Azure Blob Storage using variables and expressions.

ZappySys Providers simple and powerful Placeholders. You can still use SSIS expression language. Placeholders allow direct Variable Use easy to see compared to hidden SSIS Expression (C# style). Placeholders have simple embedding use of variables.

First, we will first create a variable in SSIS. In this example the file name is MyFile and the Value file1.
Create an SSIS variable
As a second step, we will write in path the value \destination\{{User.MyFile}}.txt. This will concatenate the folder destination with the variable MyFile. The task will delete the file1.txt from the Azure Blob Storage folder.
Using SSIS expressions
The next example will show how to create a Azure Blob Storage file using expressions and variables in SSIS:
Create file
Content to write is the content of the file. In Target, go to path and write the following: For more information on placeholders check this post and also check this post.
```
/destination/{{User::FileName,MM-dd-yyyy}}
```
We are using a variable in SSIS named FileName of type DateTime. We are concatenating the Azure Blob Storage folder destination with the variable. As you can see, the expressions with ZappySys are very simple.

Example to verify if file exists in SSIS Azure Blob Storage task

Finally, we will verify if a file in Azure Blob Storage exists or not using the SSIS Azure Blob Storage task.

In order to start, we will use the SSIS variables, create a variable of Boolean type named FileExists. In Storage Action, select Get Azure file exist status. In path, select the path of the file to verify if it exists or not.
SSIS Azure Blob Storage Task – Check File Exist in Azure Blob Storage
Secondly, we will use the ZS Logging Task to check the value of the variable File Exists. This task will show the value of the variable in the log. Drag and drop the task and join it with the ZS Azure Blob Storage Task:
The SSIS Log task
In order to get the value, we will insert it in the Logging Task. To do that, insert the Variable and select the FileExists variable:
Add the variable in the task logging
Finally, run the package and verify the Output (you can use View and Output to see the output) and check the value of the Logging Task. If the file exists, the value will be true. If it does not exist, the value will be false.
Verify the variable in Output

How to loop through Azure Blob Storage files in SSIS

You can also loop through files to perform custom action file by by. Azure Blob Storage Task support Action to get File List as ADO.net Recordset

To get File list from Azure Blob Storage Server perform the following steps.

From Storage Action select Get Azure files as ADO.net DataTable

Get files as ADO.net DataTable
Create new Azure Blob Storage Connection by clicking New next to connection dropdown.
Select Source Path (e.g \myroot\folder\*.txt )
Select or create new SSIS Variable (e.g. varFiles ) which will hold file list result (Must be Object datatype variable).
Click OK to close.
Create 3 more SSIS variables to hold File Path (String), Name (String) and Size (Int32). We will use in the next step.
Now drag SSIS ForEach Loop Container Task from SSIS Toolbox
Edit Loop Task and Select Foreach ADO Enumerator from the Enumerator dropdown
Select variable (e.g. varFiles)

Loop through files in SSIS using ForEach Loop Task (use ADO.net Recordset variable)
Click on Variable Mappings Tab and configure like below.
Variable Mappings for ForEach Loop Task – Loop through files in a folder using SSIS
Drag ZS Logging Task and place inside Loop and Display variables
That’s it

For more information to loop through files on local machine using SSIS check this article.

Conclusion

To conclude, we can say that working with Azure Blob Storage is now very simple. In this article, we looked at SSIS Azure Blob Storage Task examples on how to upload, download, delete files and folders. We used variables, wildcards and expressions. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here. We saw different SSIS Azure Blob Storage Task examples to understand how to use the task in different scenarios.

Thanks for your time.

References

The post SSIS Azure Blob Storage Task Examples (Download, Upload, Delete files / folders) appeared first on ZappySys Blog.

Read Azure Blob Storage Files in SSIS (CSV, JSON, XML) – Gen2 / Gen1

ZappySys — Sat, 16 Mar 2019 06:03:00 +0000

Introduction

In our previous blog we saw how to load data into Azure Blob Storage. Now in this blog, we will see How to read Azure Blob Storage Files in SSIS (CSV, JSON, XML Format files). To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like perform other operations on Azure Blob Storage Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on how to Read Azure Blob Storage files in SSIS using following tasks / sources.

Components Mentioned in this article

	Azure Blob CSV File Source
	Azure Blob CSV File Destination
	Azure Blob XML File Source
	Azure Blob JSON File Source

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
Download and install Microsoft Azure Storage Emulator
Download and install Microsoft Azure Storage Explorer
Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

Storage account (You can access data objects in Azure Storage through a storage account.
For more information, click here.)
the containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
All blobs lie within a container, Note: Container name must be lowercase)
the blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
See the below-attached diagram. It shows the relationship between these resources.
You can also use Azure Storage Explorer on your Local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Blob Source for CSV/JSON/XML File that will help you in reading CSV, JSON and XML Files from Azure Blob to the Local machine, Upload files(s) to Azure Blob Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to download files from Azure Blob Storage.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of local Azure Storage Emulator.

Setup Azure Storage client tools

Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Emulator Physical Location
If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
Command Prompt Screen after Microsoft Azure Storage Emulator Started
Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
Microsoft Azure Storage Explorer Physical Location

Create an Azure Blob Storage Container

For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).

Microsoft Azure Storage Explorer: Create a Blob Container

you can also create a Virtual Directory under it. A Virtual Directory does not actually exist in Azure until you paste, drag or upload blobs into it.

Creating the new Virtual Directory under Blob Container

Read Azure Blob Storage Files in SSIS (CSV, JSON, XML)

Let´s start with an example. In this SSIS Azure Blob Source for CSV/JSON/XML File task example, we will read CSV/JSON/XML files from Azure Blob Storage to SQL Server database.

First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
Drag and Drop SSIS Data Flow Task from SSIS Toolbox
Drag and Drop relevant Azure Blob Source for CSV/JSON/XML File Task from the SSIS Toolbox.
Add Azure Blob Source Tasks
Create a connection for Azure Blob Storage Account.
Create Azure Storage Connection
We can also connect the Microsoft Azure Storage emulator also like this.
Connection Form of Azure Blob Storage Account
Select the relevant single file to read from Azure Blob Storage in their relevant source of CSV/JSON/XML File Task.
Select File From Azure Blob Storage
We can also read the multiple files stored in Azure Blob Storage using wildcard pattern supported e.g. dbo.tblNames*.csv / dbo.tblNames*.json / dbo.tblNames*.xml in relevant source task
Use wildcard pattern .* to read multiple files data
We can also read the zip and gzip compressed files also without extracting it in the specific Azure Blob Source for CSV/JSON/XML File Task.
Reading zip and gzip compressed files (stream mode)
That’s it, we are ready to load this file(s) data into the SQL Server.

Load Azure Blob Storage Files data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

Drag and drop the Upsert Destination component from the SSIS Toolbox.
Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

Double-click the Upsert Destination component to open the configuration window.
Under Connection, select an existing target connection or click NEW to create a new connection.
- Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

In the Target Table dropdown, select the table where you want to load data.
Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

Go to the Mappings tab.
Click Auto Map to map source columns to target columns by name.
Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Read / Write data to Azure Data Lake Storage Gen 2 / Gen 1 (CSV / XML / JSON)

Check the below articles if you like to know more about how to write to Azure Blob Storage.

Article#1

https://community.zappysys.com/t/how-to-read-write-from-azure-data-lake-storage-gen2-in-ssis/125

Article#2

SSIS Data Load – SQL Server to Azure Blob (Split Files, GZip)

Conclusion

Above all, in this blog, we learned how to Read Azure Blob Storage Files in SSIS. We used Azure Blob Source for CSV File, Azure Blob Source for JSON File and Azure Blob Source for XML File to read the file(s) from Microsoft Azure Blob Storage and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

- Azure Blob Source for CSV File.
- Blob Source for JSON File
- Azure Blob Source for XML File
- Microsoft Azure Blob Storage
- Help File: Documentation of Azure Blob Source for CSV File, Azure Blob Source for JSON File and Azure Blob Source for XML File

The post Read Azure Blob Storage Files in SSIS (CSV, JSON, XML) – Gen2 / Gen1 appeared first on ZappySys Blog.