In our previous article, we saw SSIS Azure Blob Storage task examples. Now let’s look at how to Download the Latest File from Azure Blob Storage using SSIS. Microsoft SSIS includes the Azure Blob Storage Storage Task that allows us to upload files to Azure Blob Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Azure Blob Storage Task included in ZappySys SSIS PowerPack.

In nutshell, this post will focus on how to Download the Latest File from Azure Blob Storage.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Download and install Microsoft Azure Storage Emulator
4. Download and install Microsoft Azure Storage Explorer
5. Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

• Storage account (You can access data objects in Azure Storage through a storage account.
  For more information, click here.)
• the containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
  All blobs lie within a container, Note: Container name must be lowercase)
• the blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
  See the below-attached diagram. It shows the relationship between these resources.
       You can also use Azure Storage Explorer on your local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Blob Storage task that will allow you to access files/folders from Azure Blob to the Local machine, Upload files(s) to Azure Blob Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you Download the Latest File from Azure Blob Storage.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of the local Azure Storage Emulator.

Setup Azure Storage client tools

1. Once you have downloaded and installed a storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Emulator Physical Location

2. If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Explorer Physical Location

Create an Azure Blob Storage Container

For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).

Microsoft Azure Storage Explorer: Create a Blob Container

you can also create a Virtual Directory under it. A Virtual Directory does not actually exist in Azure until you paste, drag or upload blobs into it.

Creating the new Virtual Directory under Blob Container

Step-By-Step to Download the Latest File from Azure Blob Storage

Let´s start with an example. In this SSIS Azure Blob Storage Task File task example, we will download the latest file from Azure Blob Storage.

1. First of All, Drag and Drop Azure Blob Storage Task from SSIS Toolbox and double click it to edit
   

   Drag and Drop Azure Blob Storage Task

2. The following options are available in Azure Blob Storage Task:
   

   Options to upload, download files in SSIS

3. You can also overwrite files and folder, skip all, overwrite if the source is older, overwrite if the file size is different, rename files and more:
   

   Overwrite files in SSIS

4. Let’s select the Download Azure file(s) to local directory. Now select/add The Source path, Check Recursive Scan to get all the sub-files and subfolders and it’s files and add Target Path, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
   

   SSIS Azure Blob Storage Task – Download files from Azure Blob Storage

5. Now go to Advanced Filter & Sorting and configure it like below screen to download the latest file only
   

   Storage Task configuration to download the latest file only using sorting

6. That’s it now execute the package and it will download the latest file from the Azure Blob Storage Task to Local File Machine.
7. Sometimes you may need to make advanced filtering on file path, file date, size, etc. for that refer to this article for more details.

Conclusion

To conclude, we can say that working with Azure Blob Storage is now very simple. In this article, we looked at how to Download the Latest File from Azure Blob Storage using SSIS. We used Adnavced Filtering & sorting option. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here.

Thanks for your time.

References

• About SSIS PowerPack
• Integration Services (SSIS) Expressions
• Help File: Azure Storage Task

The post Download the Latest File from Azure Blob Storage using SSIS appeared first on ZappySys Blog.

There might be a case when you have many CSV, JSON or XML files in Azure Blob and you want them to be imported straight into a SQL Server table. Here come ZappySys ODBC PowerPack and ZappySys Data Gateway (part of ODBC PowerPack) which will enable you to accomplish that. ZappySys ODBC PowerPack includes powerful Azure Blob CSV, Azure Blob JSON and Azure Blob XML drivers that let you connect to an Azure Storage Explorer and read the contents from the files in the Container. Bringing in ZappySys Data Gateway allows doing that right from a SQL Server. Let’s begin and see how to Import Azure Blob files into SQL Server (CSV/JSON/XML Driver).

These drivers are related to this article:

Azure Blob Driver (for CSV Files)

Azure Blob Driver (for JSON Files)

Azure Blob Driver (for XML Files)

Prerequisites

1. Firstly, Download and install Microsoft Azure Storage Emulator and Microsoft Azure Storage Explorer.
2. SQL Server instance installed (can be a SQL Server Express LocalDB instance).
3. SQL Server Management Studio (SSMS) installed.
4. ZappySys ODBC PowerPack installed (must be installed on a Windows machine; can be a different machine than SQL Server is installed on).

NOTE: If you want to use Live account (Azure Storage) then you can skip Step #3

Getting started

In order to start, we will show several examples. ZappySys includes Data Gateway that will help you in reading data of Azure Blob files. Here we are showing you is, how to Import Azure Blob files into SQL Server (CSV/JSON/XML Driver).

You can connect to Azure Storage Service from SSIS, you will need Storage Account Name and Access Key. Ask your SysAdmin or responsible person to provide that information to you. Click here to read more about how to get your Storage Account Name and Access Key. Here are sample Credentials.

Account Name: mystorageaccount
Access Key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

If you don’t have Azure Storage account then you can try offline mode on your local machine. You can just download Azure Storage Emulator and start testing.

Setup Azure Storage Explorer

Setup Azure Storage client tools we will start from a simple example of how to load Invoice Data from a CSV that is located in Azure Blob, then we will load JSON contacts data from them and finally, we will proceed by loading contacts from many XML files those are compressed in ZIP.

1. Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.

   

   Azure Storage Emulator Physical Location

2. If you can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.

   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, you have to download and install Microsoft Azure Storage Explorer and then you can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.

   

   Microsoft Azure Storage Explorer Location.

4. For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).
   

   Microsoft Azure Storage Explorer: Create a Blob Container

Open and Configure ZappySys Data Gateway

Import CSV file located at Azure Blob into SQL Server

Overview

Let’s say you have CSV file that is located in Azure Blob Container, and you want them all data into your SQL Server. We will follow the same steps for that we will do for import JSON and XML file into SQL Server.

Azure Blob Container – CSV File Located

Create an Azure Blob CSV data source in ZappySys Data Gateway

1. The first thing you will have to do is to create a data source in ZappySys Data Gateway. In the Data Source tab, Just click on Add button, give the data source a name, e.g. “MyInvoiceCSV“, and then select Native – ZappySys Azure Blob CSV Driver.
   

   Create Data Source – Azure Blob CSV Driver

2. Then click on Edit and add the Data Gateway user you created in the Users tab. We will use this user later when adding a Linked Server to the Data Gateway to authenticate:
3. Now, click on the Edit button to configure “MyInvoiceCSV” Azure Blob CSV Data Source.
   

   Configure Data Gateway

4. When a window open, click on Click here to Configure the Connection and select Use the Microsoft Azure Storage Emulator. If you have Online Storage Account and Account Key then Select Enter Storage Account Credentials and enter it.
   
   

   Create Azure Storage Connection

5. Now, Select Azure Blob Container and file from it and In the Data Format / Compression (Zip / GZip) tab set suitable file Compression Format (Zip or GZip).
   

   ZappySys Azure Blob CSV – Configure Driver

6. Go to the Preview tab and click on the Preview Data button to make sure everything is configured correctly and preview the results.
   

   ZappySys Azure Blob CSV – Preview Data

7. Now, Click on OK button also Save and Restart Service.
   

   ZappySys Data Gateway Service – Save and Restart

8. That’s all, now we are ready to proceed and add a Linked Server to this data source.

Set up a SQL Server Linked Server

Execute the SQL query

Once again, if you created the Linked Server with name “GATEWAY”, execute this SQL query in SSMS:

SELECT * INTO MyInvoices FROM OPENQUERY([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyInvoices

The results

You should see a similar view, once you execute the query:

The results of loading many CSVs from Azure Blob into SQL Server

Import JSON file located at Azure Blob into SQL Server

Overview

Let’s say you have JSON file that is located in Azure Blob Container, and you want them all data into your SQL Server. We will follow the same steps of the import CSV file into SQL Server and we will do for XML file into SQL Server.

Create an Azure Blob JSON data source in ZappySys Data Gateway

Follow the steps as in Create an Azure Blob CSV data source in ZappySys Data Gateway section when adding a new data source, except for these two steps:

1. The first thing you will have to do is to create a data source in ZappySys Data Gateway. Just click Add button, give the data source a name, e.g. “MyContactsJSON“, and then select Native – ZappySys Azure Blob JSON Driver.
   

   ZappySys Data Gateway – Create New Data Source
2. After that, select a JSON file from container you want to load, and then click Select Filter button to choose data you want to be displayed in SQL Server.
   

   Configuring the JSON data source to load a JSON from Azure Blob container into SQL Server

Now you are ready to create a Linked Server and connect to the data source you just created. If you used “MyContactsJSON” as the data source name, make sure to use the same name when creating a Linked Server.

Set up a SQL Server Linked Server

Again, follow the very same steps from the section above where we added the Linked Server to a CSV data source, except that change Catalog property of the Linked Server to match the name of the Azure Blob JSON data source we created. Once you do that, you are ready to load JSON into your SQL Server.

Execute the SQL query

Once you created the Linked Server to ZappySys Data Gateway, you are ready to execute the SQL query and load data into SQL Server. Supposedly, you created the Linked Server with the name “GATEWAY”, then open SSMS and execute the following query:

SELECT * INTO MyContacts FROM openquery([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyContacts

The results

You should see similar results after you execute the query:

The results of getting JSON data straight from Azure Blob JSON

Now we are ready to proceed to the next section and import many XML files at once.

Import many XML files located at Azure Blob into SQL Server

Overview

Supposedly, you have many XMLs where each one is zipped and you want to load them all into a SQL Server table.

Zipped XML files located in Azure Blob container to be loaded into SQL Server

The first thing you will have to do is to create a data source, based on ZappySys Azure Blob XML Driver. Let’s proceed and just do that.

Create an Azure Blob XML data source in ZappySys Data Gateway

Follow the same steps as in Create an Azure Blob JSON data source in ZappySys Data Gateway section when adding a new data source, except for these two steps:

1. When adding a data source select Native – ZappySys Azure Blob XML Driver as Connector Type.
   

   Choosing “ZappySys Azure Blob XML Driver” to load XMLs to SQL Server
2. Then click on Edit and select Azure Blob Container and file from it and In the Data Format / Compression (Zip / GZip) tab set suitable file Compression Format (Zip or GZip).
   

   Configuring data source based on ZappySys Azure Blob XML Driver to load XMLs into SQL Server

3. Now we are ready to set up a Linked Server to this newly created data source.

Set up a SQL Server Linked Server

Follow the very same steps from the section above where we added the Linked Server to a CSV data source, except that change Catalog property of the Linked Server to match the name of the Azure Blob XML data source you created. Once you do that, you are ready to load XMLs into your SQL Server.

Execute the SQL query

Again, if you created the Linked Server with name “GATEWAY”, execute the following SQL query in SSMS:

SELECT * INTO MyContacts FROM openquery([GATEWAY], 'SELECT * FROM $')
SELECT * FROM MyContacts

The results

You should see a similar view once you execute the query:

The data of many XMLs loaded from Azure Blob Container into SQL Server

Now we are ready to move to the next section and recursively scan CSVs and load them into SQL Server.

Conclusion

We had a goal to import Azure Blob files into SQL Server from SQL Server itself. We learned how to Import Azure Blob files into SQL Server (CSV/JSON/XML Driver), which some of them were zipped and were located at different folders and subfolders. It was possible to accomplish that using ODBC PowerPack, ZappySys Data Gateway, and ODBC PowerPack drivers – Azure Blob JSON Driver, Azure Blob XML Driver, and Azure Blob CSV Driver. At first, we created data sources in the Data Gateway, then we created Linked Servers in SQL Server and connected to ZappySys Data Gateway, which finally made it possible to load data from Azure Blob straight from SQL queries in SSMS.

References

Finally, you can use the following links for more information:

• Azure Blob CSV Driver
• Azure Blob JSON Driver
• Azure Blob XML Driver

The post Import Azure Blob files into SQL Server (CSV/JSON/XML Driver) appeared first on ZappySys Blog.

In our previous blog, we saw you how to read/write Azure Table Storage data. Now in this blog, we will see How to Read/Write Azure Queue Storage data in SSIS. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like perform other operations on Azure Blob Storage Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on How to Read/Write Azure Queue Storage data in SSIS using the following Source/Destination.

Component Mentioned in this article

	Azure Queue Storage Source
	Azure Queue Storage Destination

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Download and install Microsoft Azure Storage Emulator
4. Download and install Microsoft Azure Storage Explorer
5. Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Storage) then you can skip Step #3

What are Azure Queues?

Azure Queue storage is a service for storing large numbers of messages. You access messages from anywhere in the world via authenticated calls using HTTP or HTTPS. A queue message can be up to 64 KB in size. A queue may contain millions of messages, up to the total capacity limit of a storage account.

Common uses

Microsoft Azure Queue Concept

Common uses of Queue storage include:

• Creating a backlog of work to process asynchronously
• Passing messages from an Azure web role to an Azure worker role
• The Queue service contains the following components:

Getting Started

In order to start, we will show several examples. Zappysys includes an SSIS Azure Queue Storage Source that will help you in reading data of Queue and SSIS Azure Queue Storage Destination that will help you to write data in Queue. Here we are showing you is, how to Read/Write Azure Queue Storage data in SSIS.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of the local Azure Storage Emulator.

Setup Azure Storage client tools and Create Queue

1. Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
   

   Azure Storage Emulator Physical Location

2. If you can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, you have to download and install Microsoft Azure Storage Explorer and then you can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Explorer Location.

4. If you don’t have Azure Storage account then it’s easy to get FREE Trial Azure Storage account or use your MSDN to get credit each month which will be more than enough for real testing. If you prefer to avoid all this hassle then Microsoft provides another great way to test Azure Storage functionality is totally offline mode on your local machine. You can just download Azure Storage Emulator and start testing.
5. Click here to learn more about how to configure Azure Storage Emulator for Testing and Development purposes. In order to connect to Azure Storage Service from SSIS, you will need Storage AccountName and an AccessKey. Ask your SysAdmin or responsible person to provide that information to you. Here are sample Account Name and Access Key(this is just example key which may differ in your case)
   

   Account Name: mystorageaccount 
   Access Key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

6. For Creating a Queue, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Queues. Right-click on Queues and Select Create Queue and give a name you want.
   

   Microsoft Azure Storage Explorer – Create Queue

Write data using ZS Azure Queue Storage Destination

In this tutorial, you will learn how to write data into Azure Queue Storage. We will use ZS DummyData Source for practice it but you can use a suitable Source Adapter. So, Let’s start, In this SSIS we will write data into Azure Queue Storage using ZS Azure Queue Storage Destination.

1. Once you have created Azure Storage Queue and you got valid Account Name and AccessKey (for the local emulator you don’t need this) you may proceed to create new SSIS package. In BIDS/SSDT create a new SSIS package.
2. From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface and double click on it too.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

3. Click here to create Azure Storage Connection.
4. From the SSIS toolbox drag and drop Dummy Data Source on the dataflow designer surface.
   

   DummyData Source – Drag and Drop

5. Double click Dummy Data Source to configure it. From Template pick JsonDocuments and enter row count=100 to generate 100 JSON documents and click OK to save.
   

   DummyData Source – Configure

6. Let’s drag and drop Azure Queue Storage Destination on the dataflow designer surface from the SSIS toolbox.
7. Now single click on the Dummy Data Source, once you see the blue arrow from the source connect it to Azure Queue Destination.
8. Double click Azure Queue Destination to configure it. Select the connection we have created before.
   

   Configure Connection Manager

9. Click on [Component Properties] tab and select the Target Queue from the dropdown list.
   

   Configure Component Properties

10. Click on [Mappings] tab and verify mapping. Map Document column from upstream to the MessageText target column.
    

    Configure Column Mappings

11. Click OK to save settings.
12. Execute the package and verify target data by going to your Azure Queue. We use Azure Storage Explorer.
    

    SSIS Execute Package

    

    Microsoft Azure Explorer – Verify Queue Data

Read data using ZS Azure Queue Storage Source

1. Here you may proceed In BIDS/SSDT to create a new SSIS package.
2. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   SSIS Data Flow Task – Drag and Drop

3. Double click on the Data Flow task to see the Data Flow designer surface.
4. From the SSIS toolbox drag and drop ZS Azure Queue Storage Source on the dataflow designer surface.
   

   SSIS ZS Azure Queue Source – Drag and Drop

5. Double click Azure Queue Storage Source to configure it.
6. Select Azure Storage Connection we have created before and select queue from the dropdown and click on preview.
   

   Configure Azure Queue Source

7. Click on OK button to save configure.
8. Now, you can use a suitable Destination Adapter from here. In this case, we used Trash Destination.
9. From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface.
   

   SSIS ZS Trash Destination – Drag and Drop

10. Now single click on the Azure Queue Storage Source once you see the blue arrow from the source … connect it to Trash Destination.
11. Double click on ZS Trash Destination to Configure it.
    

    SSIS ZS Trash Destination – Configure

12. Click on OK button to save Trash Destination configure setting UI.
13. Execute the package and verify source data in the data viewer.
    

    SSIS ZS Azure Queue Source – Execute the Package

14. After messages read from Azure Queue, the same message cannot be read again until VisibilityTimeout. By default Queue, visibility timeout is used (-1 means use default) but you can override it on the source UI.

Conclusion

In this blog, we learn How to Read/Write Azure Queue Storage data in SSIS. We used the Azure Queue Destination for write data and Azure Queue Source to read data. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Azure Queue Source
• Azure Queue Destination
• Microsoft Azure Queues
• Help File: Azure Queue Source, Azure Queue Destination, Trash Destination

The post How to Read/Write Azure Queue Storage data in SSIS appeared first on ZappySys Blog.

In our previous blog, we saw how to read/write Azure Blob Storage Files in SSIS (CSV, JSON, XML Format files). Now in this blog, we will see how to Read/Write Azure Table Storage data in SSIS. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like to perform other operations on Azure Blob Storage Files (e.g. Download, Upload, Create, Delete) then check these articles.

Component Mentioned in this article

	Azure Table Storage Source
	Azure Table Storage Destination

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Download and install Microsoft Azure Storage Emulator
4. Download and install Microsoft Azure Storage Explorer
5. Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Table Storage?

Azure Table storage stores large amounts of structured data. The service is a NoSQL datastore which accepts authenticated calls from inside and outside the Azure cloud. Azure tables are ideal for storing structured, non-relational data. Common uses of Table storage include:

Microsoft Azure Table Storage Introduction

• Storing TBs of structured data capable of serving web-scale applications
• Storing datasets that don’t require complex joins, foreign keys, or stored procedures and can be denormalized for fast access
• Quickly querying data using a clustered index
• Accessing data using the OData protocol and LINQ queries with WCF Data Service .NET Libraries

You can use Table storage to store and query huge sets of structured, non-relational data, and your tables will scale as demand increases.

Getting Started

In order to start, we will show several examples. Zappysys includes an SSIS Azure Table Storage Source. Here we are showing you is, how to Read/Write Azure Table Storage data in SSIS.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of the local Azure Storage Emulator.

Setup Azure Storage client tools and Create Table

1. Once you have downloaded and installed a storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
   

   Azure Storage Emulator Physical Location

2. If you can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, you have to download and install Microsoft Azure Storage Explorer and then you can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Explorer Location.

4. If you don’t have Azure Storage account then it’s easy to get FREE Trial Azure Storage account or use your MSDN to get credit each month which will be more than enough for real testing. If you prefer to avoid all this hassle then Microsoft provides another great way to test Azure Storage functionality is totally offline mode on your local machine. You can just download Azure Storage Emulator and start testing.
5.  Click here to learn more about how to configure Azure Storage Emulator for Testing and Development purposes. In order to connect to Azure Storage Service from SSIS, you will need Storage AccountName and an AccessKey. Ask your SysAdmin or responsible person to provide that information to you. Here are sample Account Name and Access Key(this is just example key which may differ in your case)
   

   Account Name: mystorageaccount 
   Access Key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

6. For Creating a Tables, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> Emulator – Default Ports(Key) –> Tables. Right-click on Tables and Select Create Table and put your table name you want.
   

   Create Tables in Azure Table Storage Emulator

Write data using ZS Azure Table Storage Destination

In this tutorial, you will learn how to write data into Azure Table Storage. You can use ZS DummyData Source for practice it(In this case it’s from SQL Server Source). So, Let’s start with an example. In this SSIS we will write data into Azure Table Storage using ZS Azure Table Storage Destination.

1. Once you have created Azure Storage Table and you got valid Account Name and AccessKey (for the local emulator you don’t need this) you may proceed to create a new SSIS package. In BIDS/SSDT create new SSIS package
2. From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

3. Double click on the DataFlow task to see DataFlow designer surface.
4. Here, In Visual Studio, drag and drop the OLE DB Source and Azure Table Storage Destination in the design surface and join the components with the blue arrow.
   

   SSIS Azure Table Storage Destination – Drag and Drop

5. Now, we need two connection, OLE DB Connection, and Azure Storage Connection. Click here to create Azure Storage Connection.

How to create OLE DB Connection

1. Let’s, Right-click on Connection Managers Panel to Create OLE DB Connection, so you can use Source and Context Menu will appear, Select New OLE DB Connection from the Context Menu.
   

   OLE DB Connection – Create New Connection

2. Now, click on New Button to create Connection.
   

   OLE DB Connection – Create New Connection

3. Let’s Configure Connection Manager, just Follow steps one by one as we have created.
   

   OLE DB Connection – Configure Connection

4. Click on the OK button to save configure Setting UI.

Write data into Azure Storage Table in SSIS

1. Double click on OLE DB Source for configure it.
2. Let’s Configure in Connection Manager, just follow the below image steps.
   

   OLE DB Source – Configure

3. Click OK to Save OLE DB Source Editor UI Settings.
4. Now double click Azure Table Storage Destination to configure it.
5. On [Connection Manager] tab select Azure Storage Connection manager.
   

   ZS Azure Table Storage Destination – Connection Manager

6. Click on [Component Properties] tab and select AccessMode=Table and Pick destination Table from the Tables dropdown list.
   
   If you choose Reload option then target table is first truncated before inserting records so be careful with Reload option.
   

   ZS Azure Table Storage Destination – Component Properties

7. Click on [Mappings] tab and verify mapping. Drop CustomerID and RecordID mappings and reconnect as shown below. This is just to make sure PrimaryKey columns are mapped.
   

   ZS Azure Table Storage Destination – Column Mappings

8. Click OK to save settings.
9. Execute the package and verify the target data.
   

   ZS Azure Table Storage Destination – Execute the Package

   

   ZS Azure Table Storage – Verify Customers Table Data

Read data using ZS Azure Table Storage Source

In this tutorial, you will learn how to read data from Azure Table Storage in SSIS using the ZS Azure Table Storage Source adapter. In this SSIS we will read data with SQL Query and Table Mode. So Let’s begin it.

1. You may proceed to create a new SSIS package. In BIDS/SSDT create a new SSIS package.
2. From the SSIS toolbox drag and drop Data Flow Task on the control flow designer surface.
   

   SSIS Data Flow Task – Drag and Drop

3. Double click on the DataFlow task to see DataFlow designer surface.
4. From the SSIS toolbox drag and drop Azure Table Storage Source on the data flow designer surface.
   

   ZS Azure Table Storage Source – Drag and Drop

5. Now, we need an Azure Storage connection. Click here to Create Connection.

Extract data with Query Mode

1. Here, we are getting data using Query Mode with Dynamic expression value.
2. Lets, create a Variable with correct DataType and Value, use the following image.
   

   SSIS Variables – Create Variable and Store Value

3. Double click Azure Table Storage Source to configure it.
4. Select the connection we have created before, set AccessMode to Query. You can use the following SQL Query and select Variable. Click here for more Azure Table Storage Query Examples.
   

   SELECT * FROM Customers
   WHERE Country = '{{User::varCountry}}'

   

   ZS Azure Table Storage Source – Configure Query Mode

5. Click on the Preview button to see Data Preview and OK button to save configure setting UI.
6. From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface.
   

   ZS Trash Destination – Drag and Drop

7. Now single click on the Azure Table Storage Source, once you see the blue arrow from the source … connect it to Trash Destination.
8. Double click on ZS Trash Destination to Configure it.
   

   ZS Trash Destination – Configure

9. Click on the OK button to save Trash Destination configure setting UI.
10. Execute the package and verify source data in the data viewer.
    

    SSIS ZS Azure Table Storage Source – Execute the Package

Extract data with Table Mode

1. So, Double click Azure Table Storage Source to configure it.
2. In the Azure Table Storage Source, Select the connection we have created, set AccessMode to Table and Select Table from the Tables dropdown.
   

   SSIS ZS Azure Table Source – Configure Table Mode

3. Click on the Preview button to see Data Preview and OK button to save configure setting UI.
4. From the SSIS toolbox drag and drop Trash Destination on the data flow designer surface.
   

   ZS Trash Destination – Drag and Drop

5. Now single click on the Azure Table Storage Source, once you see the blue arrow from the source … connect it to Trash Destination.
6. Double click on ZS Trash Destination to Configure it.
   

   ZS Trash Destination – Configure

7. Click on the OK button to save Trash Destination configure setting UI.
8. In the last, Execute the package and verify source data in the data viewer.
   

   ZS Azure Table Storage Source – Execute the Package

Conclusion

After all, In this blog, we learned how to write data in Azure Table Storage in SSIS. We used the Azure Table Storage Destination for write and Azure Table Storage Source to read data from Microsoft Azure Table Storage using Query and Table mode. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

There are a few details you should use the following links for more information:

• Azure Table Storage Destination
• Azure Table Storage Source
• ZS Trash Destination
• Microsoft Azure Table Storage
• Help File:
  • Azure Table Storage Destination
  • Azure Table Storage Source
  • ZS Trash Destination

The post How to Read/Write Azure Table Storage data in SSIS appeared first on ZappySys Blog.

In our previous blog we saw How to load data into Azure Blob Storage. Now in this blog, we will How to Read and Write in Azure Queue Storage in SSIS. To illustrate, we will use ZappySys SSIS PowerPack, Which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like perform other operations on Azure Blob Storage Files (e.g. Download, Upload, Create, Delete) then check these articles.

In a nutshell, In this post will focus on how to Read and Write in Azure Storage Queue in SSIS using following tasks/sources.

Components Mentioned in this article

	SSIS Azure Queue Storage Source
	SSIS Azure Queue Storage Destination

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Download and install Microsoft Azure Storage Emulator
4. Download and install Microsoft Azure Storage Explorer
5. Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

• Storage account (You can access data objects in Azure Storage through a storage account. For more information, click here.)
• The containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
  All blobs lie within a container, Note: Container name must be lowercase)
• The blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
  See the below-attached diagram. It shows the relationship between these resources.
  You can also use Azure Storage Explorer on your Local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

What is Azure Queue Storage?

Azure Queue storage is a service for storing large numbers of messages. You access messages from anywhere in the world via authenticated calls using HTTP or HTTPS. A queue message can be up to 64 KB in size. A queue may contain millions of messages, up to the total capacity limit of a storage account.

Common uses

Common uses of Queue storage include:

• Creating a backlog of work to process asynchronously
• Passing messages from an Azure web role to an Azure worker role

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Queue Storage Source that will help you in reading messages from Azure Queue Storage. And SSIS Azure Queue Storage Destination that will help you in insert and delete messages from Azure Queue Storage. Here we are showing you is, How to Read and Write in Azure Storage Queue.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of local Azure Storage Emulator.

Setup Azure Storage client tools

1. Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Emulator Physical Location

2. If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Explorer Physical Location

Create an Azure Blob Storage Queue

For Creating a Blob Queue, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Queues).

Microsoft Azure Storage Explorer : Create Blob Queue

And enter the appropriate name to that queue as Queue names can contain only lowercase letters, numbers, and hyphens, and must begin and end with a letter or a number. The name can’t contain two consecutive hyphens..

Microsoft Azure Storage Explorer : Create Blob Queue

Video Tutorial – SSIS Azure Queue Storage – Bulk Read, Write, Update, Delete Queue Messages

Read Azure Queue Messages in SSIS

Let’s start with an example. We use SSIS Azure Queue Storage Source to read Azure Queue Messages.

1. First of all, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Furthermore, Drag and Drop ZS Azure Queue Source from the SSIS Toolbox.
   

   Drag and Drop ZS Azure Queue Source

3. Now, double click on that Component to Edit the Configurations. Create New Connection and test it.
   

   Azure Queue Source Connection using Microsoft Azure Storage emulator

4. Now select the desire Queue from where you wants to read the messages and click on preview.
   

   Preview Azure Queue Source Messages

5. That’s it now we are ready to load this queue messages to another Azure Storage Queue

Write Azure Queue Messages in SSIS

Let’s start with an example. We use SSIS Azure Queue Storage Destination to insert Azure Queue Messages.

1. First, Drag and Drop ZS Azure Queue Destination from the SSIS Toolbox.
   

   Drag and Drop ZS Azure Queue Destination

2. Double click on it to Edit the Configurations. And select the desire Azure Storage Connection.
   

   Azure Queue Destination : Select Connection

3. Go to Component Properties and select the operation as insert and desire queue from the drop down where you wants to insert messages.
   

   Azure Queue Destination : Select Queue to Insert Messages

4. Now on Column Mapping tab map the desire columns and click on OK button.
   

   Azure Queue Destination : Column Mapping

5. That’s it now execute the package and it will read the Azure Queue messages and insert the messages to that selected queue.
   

   Azure Queue Source and Destination Execution

6. Furthermore, go to MS Azure Storage Exporter and open/refresh that queue and you will find that 15 messages there.
   

   Microsoft Azure Storage : Queues

Performance Tips

Disable Nagle in SSIS

Nagle’s algorithm is widely implemented across TCP/IP networks as a means to improve network performance. However, it is not optimal in all circumstances (such as highly interactive environments). For Azure Storage, Nagle’s algorithm has a negative impact on the performance of requests to the table and queue services, and you should disable it if possible.

Please refer to this link for more information: Click Here

To disable Nagle algorithm in SSIS we need to use Script Task at very First step of the Package.

1. In the Package File at the very first step add the Scrip Task and click on Edit Script.
   

   Edit Script Task

2. So that it opens ScriptMain.cs Script class in another visual studio instance. Now add below Line in the Main() function and save it.
   

   System.Net.ServicePointManager.UseNagleAlgorithm = false;

   

   Script Task : Disable Nagle Algorithm

3. Now go to Script Task and click on OK , so that it’s save the script and close the class window.
   

   Save and close Script Task

4. That’s it we have disabled the Nagle Algorithm now you can go to next steps. In upper case we need to add data flow task and follow the steps from the 2# step.
   

   Add Data Flow Task

Increase Max Parallel Threads size

We can increase the thread size to improve performance further more, in the azure connection manager we can increase Max Parallel Threads size like 20 or more.

Azure Storage Connection : Max Parallel Threads

Conclusion

Above all, in this blog, we learned how to Read and Write in Azure Storage Queues. We used Azure Queue Storage to read messages from Azure Queue Storage. And Azure Queue Storage Destination to insert messages from Azure Queue Storage. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Help file : Azure Queue Source
• Help file : Azure Queue Destination

The post Read and Write in Azure Queue Storage in SSIS appeared first on ZappySys Blog.

In this article, we will show SSIS Azure Blob Storage task examples. Uploading files using Azure Blob Storage Task or downloading files are common requirements in SSIS. In order to work with Azure Blob Storage, Microsoft SSIS includes the Azure Blob Storage Task that allows to upload files to Azure Blob Storage, download files, creating local and remote directories an more. In this article, we will show how to use the SSIS Azure Blob Storage task included in ZappySys SSIS PowerPack.

Azure Blob Storage Task can be used to perform various operations with Azure Storage objects (blobs and containers) (e.g. Download, Upload, Copy, Move, Rename, Delete, etc). You can also get single property of any Azure blob or get a list of blobs as ADO.net Table and which can be easily looped through using ForEachLoop Task. Most of the operations in this task support recursive search using wild card pattern (e.g. Myfile*.txt).

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Download and install Microsoft Azure Storage Emulator
4. Download and install Microsoft Azure Storage Explorer
5. Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

• Storage account (You can access data objects in Azure Storage through a storage account.
  For more information, click here.)
• the containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
  All blobs lie within a container, Note: Container name must be lowercase)
• the blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
  See the below-attached diagram. It shows the relationship between these resources.
       You can also use Azure Storage Explorer on your Local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Blob Storage Task that will allow you to access files / folders from Azure Blob to the Local machine, Upload files(s) to Azure Blob Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to upload, download, move and delete files / folders from Azure Blob Storage.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of local Azure Storage Emulator.

Setup Azure Storage client tools

1. Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Emulator Physical Location

2. If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Explorer Physical Location

Create an Azure Blob Storage Container

For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).

Microsoft Azure Storage Explorer: Create a Blob Container

you can also create a Virtual Directory under it. A Virtual Directory does not actually exist in Azure until you paste, drag or upload blobs into it.

Creating the new Virtual Directory under Blob Container

SSIS Azure Blob Storage Task Examples

Let´s start with an example. In this SSIS Azure Blob Storage Task File task example, we will upload, download, move and delete files / folders from Azure Blob Storage.

1. 1. First of All, Drag and Drop Azure Blob Storage Task from SSIS Toolbox and double click it to edit
      

      Drag and Drop Azure Blob Storage Task

   2. The following options are available in Azure Blob Storage Task:
      

      Options to upload, download files in SSIS

   3. You can also overwrite files and folder, skip all, overwrite if source is older, overwrite if the file size is different, rename files and more:
      

      Overwrite files in SSIS

   4. Let’s select the Download Azure file(s) to local directory. Now select/add The Source path, Check Recursive Scan to get all the sub files and sub folders and it’s files and add Target Path, This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable.
      

      SSIS Azure Blob Storage Task – Download files from Azure Blob Storage

   5. That’s it now execute the package and it will download all the files and sub folders from the Azure Blob Storage Task to Local File Machine.

1. Now Let’s configure the task to upload the local directory files and folder to Azure Blob Storage like below and execute the package. And also in this example i created the variable and stored target folder path in it and used that variable Target Path Variable.
   

   SSIS Azure Blob Storage Task – Upload files from Azure Blob Storage

2. Let’s configure the task to delete Azure Blob Storage files like below and execute the package.
   

   SSIS Azure Blob Storage Task – Delete files from Azure Blob Storage

3. Let’s configure the task to move Azure Blob Storage files to another directory like below and execute the package.
   

   SSIS Azure Blob Storage Task – Move files from Azure Blob Storage

4. That’s it like upper examples you can configure Azure Blob Storage Task to use it’s other options also.
5. Let’s see other SSIS Azure Blob Storage Task examples in below sections

Working with advanced filters

In order to understand filters, we will create an example for you. This time we will create SSIS Azure Blob Storage task example to show how to exclude some files from the downloading process.

1. In addition, it is also possible to add advanced filters to the SSIS Azure Blob Storage task to exclude or include files and folders. The following example shows how to use the Advanced filters. In the Azure Blob Storage Task, go to the Advanced Filter page and in Exclude RegX we will write \.exe$ to exclude the file with exe extension to be downloaded. You can use RegX (regular expressions) in the advanced filter:
   

   Advanced filters

Working with wildcards in SSIS Azure Blob Storage task (Multiple files only)

In order to understand wildcards, the new example will show how to work with Wildcards will help you to work with multiple files or folder with common characteristics.

1. The following example shows how to copy all the files will all the extensions in the source folder. We use the *.*, which means that any file name with any extension will be downloaded:
   

   Working with Wildcards

2. If we want to download all the files with extension txt, use the following Path:
   

   /source/*.txt;

3. Another example would be to download all the files with the following names: file1.txt, file2.txt and file3.txt.  We could use the following Path:
   

   /source/file?.txt

Working with SSIS expressions and variables in Azure Blob Storage Task

In order to understand expressions, the following example will show how to work with SSIS expressions combined with variables in the SSIS Azure Blob Storage task. In this example we will delete files in Azure Blob Storage using variables and expressions.

1.  First, we will first create a variable in SSIS. In this example the file name is MyFile and the Value file1.
   

   Create an SSIS variable

2. As a second step, we will write in path the value \destination\{{User.MyFile}}.txt. This will concatenate the folder destination with the variable MyFile. The task will delete the file1.txt from the Azure Blob Storage folder.
   

   Using SSIS expressions

3. The next example will show how to create a Azure Blob Storage file using expressions and variables in SSIS:
   

   Create file

4. Content to write is the content of the file. In Target, go to path and write the following: For more information on placeholders check this post and also check this post.
   

   /destination/{{User::FileName,MM-dd-yyyy}}

5. We are using a variable in SSIS named FileName of type DateTime. We are concatenating the Azure Blob Storage folder destination with the variable. As you can see, the expressions with ZappySys are very simple.

Example to verify if file exists in SSIS Azure Blob Storage task

Finally, we will verify if a file in Azure Blob Storage exists or not using the SSIS Azure Blob Storage task.

1. In order to start, we will use the SSIS variables, create a variable of Boolean type named FileExists. In Storage Action, select Get Azure file exist status. In path, select the path of the file to verify if it exists or not.
   

   SSIS Azure Blob Storage Task – Check File Exist in Azure Blob Storage

2. Secondly, we will use the ZS Logging Task to check the value of the variable File Exists. This task will show the value of the variable in the log. Drag and drop the task and join it with the ZS Azure Blob Storage Task:
   

   The SSIS Log task

3. In order to get the value, we will insert it in the Logging Task. To do that, insert the Variable and select the FileExists variable:
   

   Add the variable in the task logging

4. Finally, run the package and verify the Output (you can use View and Output to see the output) and check the value of the Logging Task. If the file exists, the value will be true. If it does not exist, the value will be false.
   

   Verify the variable in Output

How to loop through Azure Blob Storage files in SSIS

You can also loop through files to perform custom action file by by. Azure Blob Storage Task support Action to get File List as ADO.net Recordset

To get File list from Azure Blob Storage Server perform the following steps.

1. From Storage Action select Get Azure files as ADO.net DataTable
   
   

   Get files as ADO.net DataTable

    

2. Create new Azure Blob Storage Connection by clicking New next to connection dropdown.
3. Select Source Path (e.g  \myroot\folder\*.txt  )
4. Select or create new SSIS Variable (e.g. varFiles ) which will hold file list result (Must be Object datatype variable).
5. Click OK to close.
6. Create 3 more SSIS variables to hold File Path (String), Name (String) and Size (Int32). We will use in the next step.
7. Now drag SSIS ForEach Loop Container Task from SSIS Toolbox
8. Edit Loop Task and Select Foreach ADO Enumerator from the Enumerator dropdown
9. Select variable (e.g. varFiles)
   
   

   Loop through files in SSIS using ForEach Loop Task (use ADO.net Recordset variable)

10. Click on Variable Mappings Tab and configure like below.
    

    Variable Mappings for ForEach Loop Task – Loop through files in a folder using SSIS

11. Drag ZS Logging Task and place inside Loop and Display variables
12. That’s it

For more information to loop through files on local machine using SSIS check this article.

Conclusion

To conclude, we can say that working with Azure Blob Storage is now very simple. In this article, we looked at SSIS Azure Blob Storage Task examples on how to upload, download, delete files and folders. We used variables, wildcards and expressions. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here. We saw different SSIS Azure Blob Storage Task examples to understand how to use the task in different scenarios.

Thanks for your time.

References

• About SSIS PowerPack
• Integration Services (SSIS) Expressions
• Help File : Azure Storage Task

The post SSIS Azure Blob Storage Task Examples (Download, Upload, Delete files / folders) appeared first on ZappySys Blog.

In our previous blog we saw how to load data into Azure Blob Storage. Now in this blog, we will see How to read Azure Blob Storage Files in SSIS (CSV, JSON, XML Format files). To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like perform other operations on Azure Blob Storage Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on how to Read Azure Blob Storage files in SSIS using following tasks / sources.

Components Mentioned in this article

	Azure Blob CSV File Source
	Azure Blob CSV File Destination
	Azure Blob XML File Source
	Azure Blob JSON File Source

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Download and install Microsoft Azure Storage Emulator
4. Download and install Microsoft Azure Storage Explorer
5. Finally, do not forget to install ZappySys SSIS PowerPack

NOTE: If you want to use Live account (Azure Blob Storage) then you can skip Step #3

What is Azure Blob Storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. you can store large amounts of unstructured data, such as text or binary data. Blob storage discloses three resources:

• Storage account (You can access data objects in Azure Storage through a storage account.
  For more information, click here.)
• the containers in the account(constructs a set of blobs, it is similar to a folder in a file system.
  All blobs lie within a container, Note: Container name must be lowercase)
• the blobs in a container. (Azure Storage offers three types of blobs: block blobs, append blobs, and page blobs)
  See the below-attached diagram. It shows the relationship between these resources.
       You can also use Azure Storage Explorer on your Local machine. Azure Storage Explorer is a standalone app that enables you to easily work with Azure Storage data on Windows, macOS, and Linux. You can use Blob storage to expose data publicly to the world or to store application data privately. Connect to an Azure storage account or service

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Azure Blob Source for CSV/JSON/XML File that will help you in reading CSV, JSON and XML Files from Azure Blob to the Local machine, Upload files(s) to Azure Blob Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to download files from Azure Blob Storage.

You can connect to your Azure Storage Account by entering your storage account credentials. Here I am showing an example of the use of local Azure Storage Emulator.

Setup Azure Storage client tools

1. Once you have downloaded and installed storage emulator You can launch Microsoft Azure Storage Emulator from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Emulator Physical Location

2. If You can see the below-attached Command Prompt screen after Emulator started. Then you can proceed to start Microsoft Azure Storage Explorer as the Azure Storage Emulator is started successfully.
   

   Command Prompt Screen after Microsoft Azure Storage Emulator Started

3. Now, You have to download and install Microsoft Azure Storage Explorer and then You can launch Microsoft Azure Storage Explorer from its Physical location or from the desktop or start menu shortcut.
   

   Microsoft Azure Storage Explorer Physical Location

Create an Azure Blob Storage Container

For Creating a Blob Container, First of all, you need to go to Microsoft Storage Explorer Window. Then you can go through like this way (Storage Accounts –> (Development) –> Blob Containers).

Microsoft Azure Storage Explorer: Create a Blob Container

you can also create a Virtual Directory under it. A Virtual Directory does not actually exist in Azure until you paste, drag or upload blobs into it.

Creating the new Virtual Directory under Blob Container

Read Azure Blob Storage Files in SSIS (CSV, JSON, XML)

Let´s start with an example. In this SSIS Azure Blob Source for CSV/JSON/XML File task example, we will read CSV/JSON/XML files from Azure Blob Storage to SQL Server database.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Drag and Drop relevant Azure Blob Source for CSV/JSON/XML File Task from the SSIS Toolbox.
   

   Add Azure Blob Source Tasks

3. Create a connection for Azure Blob Storage Account.
   

   Create Azure Storage Connection

4. We can also connect the Microsoft Azure Storage emulator also like this.
   

   Connection Form of Azure Blob Storage Account

5. Select the relevant single file to read from Azure Blob Storage in their relevant source of CSV/JSON/XML File Task.
   

   Select File From Azure Blob Storage

6. We can also read the multiple files stored in Azure Blob Storage using wildcard pattern supported e.g. dbo.tblNames*.csv / dbo.tblNames*.json / dbo.tblNames*.xml in relevant source task
   

   Use wildcard pattern .* to read multiple files data

7. We can also read the zip and gzip compressed files also without extracting it in the specific Azure Blob Source for CSV/JSON/XML File Task.
   

   Reading zip and gzip compressed files (stream mode)

8. That’s it, we are ready to load this file(s) data into the SQL Server.

Load Azure Blob Storage Files data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Read / Write data to Azure Data Lake Storage Gen 2 / Gen 1 (CSV / XML / JSON)

Check the below articles if you like to know more about how to write to Azure Blob Storage.

Article#1

https://community.zappysys.com/t/how-to-read-write-from-azure-data-lake-storage-gen2-in-ssis/125

Article#2

SSIS Data Load – SQL Server to Azure Blob (Split Files, GZip)

Conclusion

Above all, in this blog, we learned how to Read Azure Blob Storage Files in SSIS. We used Azure Blob Source for CSV File, Azure Blob Source for JSON File and Azure Blob Source for XML File to read the file(s) from Microsoft Azure Blob Storage and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• • Azure Blob Source for CSV File.
  • Blob Source for JSON File
  • Azure Blob Source for XML File
  • Microsoft Azure Blob Storage
  • Help File: Documentation of Azure Blob Source for CSV File, Azure Blob Source for JSON File and Azure Blob Source for XML File

The post Read Azure Blob Storage Files in SSIS (CSV, JSON, XML) – Gen2 / Gen1 appeared first on ZappySys Blog.

In this post we will see how to set Metadata key/value in Azure Blob Storage Task and Amazon Storage Task (S3 Task). We will also see how to set them dynamically at runtime (Use SSIS Expression).

Set Amazon S3 Task Metadata (Key / Value Pairs)

Here is how you can set Metadata in Amazon S3 Task.

Setting Amazon S3 Metadata (Key/Value Pairs) in SSIS Amazon Storage Task

Set Azure Blob Storage Task Metadata (Key / Value Pairs)

Just like S3 Task you can use same UI in Azure blob storage task too.

Set Metadata Key/Value Dynamically (SSIS Expression)

So far you have set metadata using static value. What if you have to set Dynamically at runtime?

Basically Metadata is passed as XML Format in the task like below.

<?xml version="1.0" encoding="utf-8"?>
<ArrayOfStringKeyValue xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<StringKeyValue>
		<Key>x-amz-key1</Key>
		<Value>AAAAAAA</Value>
	</StringKeyValue>
	<StringKeyValue>
		<Key>x-amz-key2</Key>
		<Value>BBBBBBB</Value>
	</StringKeyValue>	
</ArrayOfStringKeyValue>

Whate we will do is take above format and make it dynamic using SSIS Expression.

Perform the following steps for that.

1. Select your Amazon Storage Task or Azure Storage Task and right click > Select Properties.
2. In the Properties Grid check Property name MetadataKeyValue. Copy XML Value from there (We will use it in expression)
3. Now double click Amazon Storage Task or Azure Storage Task
4. Click on Expression Button
5. Select MetadataKeyValue property and for Value enter below expression… Use previously copied XML text  and change Value part with your own SSIS variable. In below example we are passing Just one Metadata Key/Value pair ( Name=x-amz-meta-filename  , Value=Coming from @[User::MyFileNameVariable] )
   Note that we have escaped double quote with

   \"

     and replaced static value with

   <Value>" + @[User::MyFileNameVariable] + "</Value>

    .
   

   "<?xml version=\"1.0\" encoding=\"utf-8\"?>"
   + "<ArrayOfStringKeyValue xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\">"
   + "	<StringKeyValue>"
   + "		<Key>x-amz-meta-file-name</Key>"
   + "		<Value>" + @[User::MyFileNameVariable] + "</Value>"
   + "	</StringKeyValue>"
   + "	<StringKeyValue>"
   + "		<Key>x-amz-meta-something</Key>"
   + "		<Value>BBBBBBB</Value>"
   + "	</StringKeyValue>"	
   + "</ArrayOfStringKeyValue>"

   

   Set Amazon S3 File Metadata Key/Value dynamically

6.  Click OK and that’s it.

The post Setting Metadata key / value in SSIS Amazon S3 / Azure Storage Task appeared first on ZappySys Blog.

In this blog post you will see how easy it is to load large amount of data from SQL Server to Azure Blob Storage using SSIS. We will export / compress data to multiple files.

For demo purpose we will use SQL Server as relational source but you can use same steps for any database engine such as Oracle, MySQL, DB2. In this post we will use Export CSV Task and Azure Blob Storage Task  to achieve desired integration with Azure Blob with drag and drop approach. You can also export JSON or XML data to Azure Blob using same techniques (Use Export JSON Task  or Export XML Task ).

Our goal is to achieve following things

• Extract large amount of data from SQL Server Table or Query and export to CSV files
• Generate CSV files in compressed format (*.gz) to speedup upload and save data transfer cost to Azure
• Split CSV files by row count
• Upload data to Azure Blob using highly parallel manner for maximum speed

There are three different ways you can achieve data export to Azure Blob using SSIS.

1. Method-1 (Fastest): Use two step process (First export SQL Server data to local files using Export Task and then upload files to Azure using  Azure Blob Storage Task  )
2. Method-2 (Slower): Use Export Task with Azure Blob Connection as Target rather than save to Local files.
3. Method-3 (Slower): Use Data flow components like Azure Blob Destination for CSV  (for JSON / XML  use Method1 or Method2)

Each method has its own advantage / disadvantage. If you prefer to upload / compress / split large amount of data then we recommend Method#1 (Two steps). If you have not very huge dataset then you can use Method#2 or Method#3. For Last method you can only use CSV export option (we don’t have JSON/ XML Destination for Azure Blob yet – we may add in future)

Screenshot of SSIS Package

Extract SQL Server Data to CSV files in SSIS (Bulk export) Split / GZip Compress / upload files to Azure Blob Storage

Method-1 : Upload SQL data to Azure Blob in Two steps

In this section we will see first method (recommended) to upload SQL data to Azure Blob. This is the fastest approach if you have lots of data to upload.  In this approach we first create CSV files from SQL Server data on local disk using SSIS Export CSV Task. After that in second step we upload all files to Azure Blob using SSIS Azure Storage Task.

Step-1: Configure Source Connection in Export CSV Task

To extract data from SQL Server you can use Export CSV Task. It has many options which makes it possible to split large amount of data into multiple files. You can specify single table or multiple tables as your data source.

For multiple table use vertical bar. e.g.  dbo.Customers|dbo.Products|dbo.Orders. When you export this it will create 3 files ( dbo.Customers.csv , dbo.Products.csv, dbo.Orders.csv )

Steps:

1. Drag ZS Export CSV Task from Toolbox
2. Double click task to configure
3. From connection drop down select New connection option (OLEDB or ADO.net)
4. Once connection is configured for Source database specify SQL Query to extract data as below
   

   Export SQL Server Table or Query as CSV file (Bulk export in SSIS)

5. Now go to target tab. Here you can specify full path for file. e.g. c:\ssis\temp\azure\cust.csv

Step-2: Compress CSV Files in SSIS ( GZIP format – *.gz )

Above steps will export file as CSV format without splitting or compression. But to compress file once exported you can go to Target tab of Export CSV Task and check [Compress file to *.gz format] option.

Compress exported SQL Server data files to GZip ( *.gz) in SSIS Export CSV Task

Step-3: Split CSV files by row count or data size in SSIS

Now lets look at how to split exported CSV files into multiple files so we can upload many files in parallel. Goto Split Options and check [Enable Split by Size/Rows]

Using SSIS Split Exported CSV files (Split by row count or size)

Step-4: Upload CSV files to Azure Blob – Using multi threaded option

Now final thing is use Azure Storage Task to upload files to Azure.

Steps:

1. Drag ZS Azure Storage Task from SSIS toolbox
2. Double click Azure Storage Task to configure it
3. Specify Action = UploadFilesToAzure
4. Specify Source file path (or pattern) e.g. c:\SSIS\temp\azure\*.*
5. Now in the Target connection dropdown click [New]
6. When Connection UI opens Enter your Account, Secret Key (Leave all other parameters default if you not sure)
7. Click Test and close connection UI
8. On the Target path on Azure Storage Task enter your bucket and folder path where you want to upload local files. For example your container name is bw-east-1 and folder is sqldata then enter as below
   bw-east-1/sqldata/
9. Click ok and Run package to test full package

Method-2 : Upload SQL data to Azure Blob without local stage (One step)

Now let’s change previous approach little bit to send SQL server data directly to Azure Blob without any Landing area on local disk.  Export CSV Task , Export JSON Task  and Export XML Task all of them supports Azure Blob / Azure Blob and Secure FTP (SFTP) connection as target (Only available in Pro Edition). We will use this feature in following section.

This approach helps to avoid any local disk need and it may be useful for security reason for some users. However drawback of this approach is, it wont use parallel threads to upload large amount of data like previous method.

Following change will be needed on Export task to upload SQL data directly to Azure / FTP or Azure storage.

Export SQL data to multiple files to Azure Blob, Amazon S3, Secure FTP (SFTP) in Stream Mode using SSIS. Configure Compress GZip, Overwrite, Split Options

Method-3 : Using Azure Blob destination – Generate Azure Blob file from any source

Now let’s look at third approach to save data from any SSIS Source to Azure Blob file. Advantage of this approach is you are not limited to few source options provided by Export CSV Task. If you have complex data transformation needed in Data Flow before sending data to Azure then use this approach.  We will use Azure Blob Destination for CSV as below

1. Drag SSIS Data flow task from toolbox
2. Create necessary source connection (e.g. OLEDB connection)
3. Create Azure Blob Connection (Right click in Connection Managers panel in bottom and click New connection and select ZS-Azure-STORAGE type )
4. Once connection managers are created Go to data flow designer and Drag OLEDB Source
5. Configure OLEDB Source to read desired data from source system (e.g. SQL Server / Oracle)
6. Once source is configured drag ZS Azure Blob CSV File Destination from SSIS toolbox
7. Double click Azure Destination and configure as below
   1. On Connection Managers tab select Azure Connection (We created in earlier section).
   2. Properties tab configure like below screenshot
   3. On Input Columns tab select desired column you like to write in the target file. Your name from upstream will be taken as is for target file. So make sure to name upstream columns correctly.
   4. Click OK to save UI
8. Execute package and check your Azure Bucket to see files got created.

Loading SQL Server data into Azure Container Files (Split, Compress Gzip Options) – SSIS Azure Blob CSV File Destination

Conclusion

In this post you have seen how easy it is to upload / archive your SQL Server data (or any other RDBMS data) to Azure Blob Storage in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to integrate SQL Server and Azure Blob using SSIS.

The post SSIS Data Load – SQL Server to Azure Blob (Split Files, GZip) appeared first on ZappySys Blog.

In this article, we will show how to do sentiment Analysis in SSIS of text using artificial intelligence. It is very important for a company to measure customer satisfaction. Today, it is possible to do sentiment and use artificial intelligence and detect if a customer liked or not our products based in on a comment?
Yes, it is. Now it is possible to analyze text and detect customer feelings. In this article, we will show how to detect the feelings of a comment, how to export the results in a Database and how to display some information based on the feelings.

In this article we will be using ZappySys SSIS PowerPack components and Microsoft Azure Service for Text Analytics (i.e. Cognitive Services) to make things work:

	SSIS REST API Web Service Task
	SSIS JSON Source (File, REST API, OData Connector)
	SSIS JSON File Destination (Create JSON File)
	SSIS Export JSON File Task
	SSIS Logging Task (FREE)

Requirements

• In order to start, we will use SSDT with SQL Server Integration Services(SSIS). SSIS is a Microsoft Powerful tool to import, export data and automate tasks.
• Secondly, ZappySys SSIS PowerPack, which is a powerful tool to make REST API calls and other tools that we will use to get data and analyze data and export the data.
• Finally, an Azure Account will be required to access the Text Analytics Services.

Getting started

In order to start with the Sentiment Analysis in SSIS, we will show first how to analyze text and detect the customer feelings. If it is a positive feeling, the value will be 1 or close to 1. If the feeling is negative, the value will be 0 or close to 0. Let’s start with a simple example of 2 real customer reviews:

First review:

I looked everywhere for a product that could easily integrate to a specific web portal backend API via 
JSON, and after many attempts, the ZappySys product was the only solution that could give me all the 
options I needed in a quick & clean interface within SSIS. Coupled with some great tech support and 
customer support, I would definitely recommend ZappySys SSIS PowerPack...

Second review

The responsiveness of your support staff during the evaluation period really has been excellent. 
I’ve contacted them a few times with queries specific to the scenario I am working with and they have 
answered straight away via chat and provide exactly the detailed technical input required to progress...

In order to detect the feelings, we will use the Microsoft Cognitive Services. Microsoft included in Azure pretty nice services to Analyze text, translate text, detect sentiments, language understanding, and several other services. In this example, we will use the sentiment detector service.

Create a service for Sentiment Analysis in SSIS using Azure

1. First, log in to the Azure Portal.
2. Secondly, in the Portal, press the + green icon and select AI+ Machine Learning and select Text Analysis:
   Also, enter a name, a location and select a resource group and a pricing Tier.
   

   Add text Analytics

3. Finally, check the endpoints and the keys. We will use that information to connect with REST API in SSIS.
   

   Endpoint in Azure

Using REST API for Sentiment Analysis in SSIS

1. First, we want to detect the feelings. To do it, we will use the REST API task included with the ZappySys SSIS PowerPack.
   

   SSIS Rest API Task

2. Secondly, we will specify the URL and in the Raw Edit we will add the key to enter. The URL is the Endpoint that we had in the Azure Portal. It should be something like this:
   

   https://eastus2.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment

3. Also, enter the key that can be obtained in the Azure Portal. It will be something like this in the HTTP Header:
   

   Ocp-Apim-Subscription-Key: 0e99402669b54beaa2af705d6c

4. In addition, in the Body Request, we will create something like the following:
   

   {
     "documents": [
       {
         "language": "en",
         "id": "1",
         "text": "I looked everywhere for a product that could easily integrate to a specific web portal backend api via JSON, and after many attempts, the ZappySys product was the only solution that could give me all the options I needed in a quick & clean interface within SSIS."
       },
       {
         "language": " en ",
         "id": " 2 ",
         "text": "The responsiveness of your support staff during the evaluation period really has been excellent.I’ve contacted them a few times with queries specific to the scenario I am working with and they have answered straight away via chat and provide exactly the detailed technical input required to progress."
       }
     ]
   }

5. In addition, the JSON text includes the ID (1 and 2), which is just an identifier. The language is to specify the language of the text. Currently it supports English (en), Spanish (es), Portuguese from Portugal (pt-PT) and other languages. For a complete list, you can check this link:
   

   https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/text-analytics-supported-languages

6. The REST API tasks should look like this:
   

   SSIS REST API connection to Azure Cognitive services

7. Also, when we press the Test Request/Response button, we can see the score of the comments:
   

   Score detecting feelings

8. Finally, the scores are 0.84 and 0.789. They are closer to 1, so they are 2 positive comments. So, we could read comments and detect if the feelings are positive or negative. Also, it is possible to store the results in a file or in a variable:
   

   SSIS save JSON result to a variable

How to detect Sentiment Analysis in SSIS using a database

In the previous example, we show how to detect the sentiments of text. The text must be in JSON format, so it is necessary to convert the data from JSON. In this new example, we will show how to convert comments from an SQL Server database to JSON and then we will learn how to export the scores of the sentiments to SQL Server and count how many users are happy and unhappy. The package will have the following tasks:

SSIS package to count positive and negative sentiments using AI

We will first show how to export reviews stored in an SQL Server database to a JSON format necessary to be read by REST API.

Export SQL Server data to JSON format

1. First of all, we will use a table named comments. The code to create the table with data is the following:
   

   CREATE TABLE [dbo].[comment](
   [comments] [nchar](2000) NULL,
   [id] [smallint] IDENTITY(1,1) NOT NULL,
   CONSTRAINT [PK_comment] PRIMARY KEY CLUSTERED
   (
   [id] ASC
   )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
   ) ON [PRIMARY]
   GO
   SET IDENTITY_INSERT [dbo].[comment] ON
   
   INSERT [dbo].[comment] ([comments], [id]) VALUES (N'Great product. I love it. ', 1)
   INSERT [dbo].[comment] ([comments], [id]) VALUES (N'It is not bad. I will use it. ', 2)
   INSERT [dbo].[comment] ([comments], [id]) VALUES (N'The product is wonderful ', 3)
   INSERT [dbo].[comment] ([comments], [id]) VALUES (N'I hate this product. I do not know why is it online ', 4)
   SET IDENTITY_INSERT [dbo].[comment] OFF

   The table is this one:
   

   Sample table to detect feelings

2. Secondly, we want to convert this SQL table to JSON like this to send the data to our Azure REST API text analytics service:
   

   {
   "documents": [
   {
   "id": 1,
   "text": "Great product. I love it. "
   },
   {
   "id": 2,
   "text": "It is not bad. I will use it. "
   },
   {
   "id": 3,
   "text": "The product is wonderful "
   },
   {
   "id": 4,
   "text": "I hate this product. I do not know why is it online "
   }
   ]
   }

3. Also, will use the Export JSON Task include in the ZappySys SSIS PowerPack.
   First, in Source, enter your server name, connection, and SQL Server Database and check the Use Custom Layout option:
   

   SSIS export to JSON task

4. Check the Single Dataset Array option and on Dataset, right click and select Add Dataset:
   

   Add JSON data set

5. Also, write a DataSet name, select the SQL Server database connection and select the table comment that contains the comments to export to JSON. You can export data from Tables, Views, queries and stored procedures with parameters using our Export JSON Task:
   

   SSIS get tables views to convert to JSON

6. In addition, right-click Mapping and select Add Document Array:
   Next, add an Element name and press OK:
   

   SSIS add array document

7. Next, right-click documents and select Add Elements:
   

   Add JSON element in SSIS

8. Also, add the comments and the id:
   

   JSON elements added in SSIS

9. The format displayed will be like this:
   

   SSIS SQL Server data in JSON Format

10. Another thing that you have to do is go to the Target tab, the target location will be a variable and we will store the JSON in a variable named jsonvar:
    

    Save JSON into variable

How to get the feelings using REST API

In the previous section, we exported data from SQL Server to a JSON format and stored the data in an SSIS variable named jsonvar. In this new section, we will call the API to get Analyze the text and detect if the comments are positive or negative.

REST API detect feelings SSIS

1. First, we will use the REST API to invoke the Azure Analytics Text services. We will use the Web URL of the Create a service to Analyze text using Azure section at the beginning of the article and in the HTTP Headers, go Raw Edit and specify the Ocp-Apim-Subscription key. You can also see this information in the Analyze test using Azure section.
2. Secondly, in the body (Request Data), we will use the variable with JSON results generated by the Export to JSON task in the previous section:
   

   Header information REST API

3. Finally, you will store the results in a file named sentiment.json using the Response Settings tab:
   

   Save JSON results into a file in SSIS

Export JSON results to SQL Server

1. We have a file with the Sentiments in a file names sentiments.json and we want to export the JSON file to SQL Server to count the values and do some queries. To do that, we will use the Data Flow task:
   

   Export from JSON to SQL in SSIS

2. In the Data Flow task, use the JSON Source and join to the OLE DB Destination. The JSON Source will read our JSON file and the OLE DB Destination will connect to SQL Server:
   

   The JSON Source in SSIS

3. Also, in JSON Source task, enter the path of the JSON file. In Filter write the following filter:
   

   $.documents[*]

   

   Add filters to JSON data

4. Also, we will create the following table named feeling to store the JSON results to SQL Server and use the OLEDB Destination to store JSON values into the SQL table. In this example, the scores of the feelings:
   

   CREATE TABLE [dbo].[Feeling]([score] 
   [float] NULL,
   [id] [int] NULL ) 
   ON [PRIMARY] GO

   

   SQL Server SSIS Oledb

Count positive and negative feelings in SSIS

We stored the feelings in an SQL Server table. Now, we need to count them and then publish the values into variables.

Store T-SQL results into

1. In order to start, we will create two SSIS variables. One named positive and another negative of type int16. We will store the positive feelings in one variable and negative feelings in another.
2. Secondly, we will use the Execute SQL Task. In connection, you must provide your SQL Server name, database, credentials. Go to Result Set and select single row.
3. In SQLStatement, write the following query to count the positive feelings:
   

   select count(score) as positive 
   from dbo.feeling 
   where score>0.5

   

   Save T-SQL count result in a variable

4. Also, in result set specify 0 in Result Name and select the Positive variable.
   

   Handle SSIS Results Set

5. In addition, to count the negative feeling we will use the same Execute SQL Task with the following query and store the results in the negative SSIS variable:
   

   select count(score) as negative 
   from dbo.feeling 
   where score<0.5

   

   Store number of negative sentiments into SSIS variable

6. Finally, in Result Set, Assign the values to the negative variable:

Show positive and negative feelings using an SSIS variable

Finally, we will show the number of positive and negative feelings using the ZS Logging Task. This task allows storing variable values in a File, the Log file a messagebox. The article will show the value in the Output log.

1. In order to start, in the ZS Logging task use the ExecutionLog Log mode. We will send the following text to show the variables in the output:
   

   Positive feelings: {{User::Positive}}.
   Negative feelings: {{User::Negative}}

    

    

   

   SSIS variables in output

2. Finally, if you run the package in the Visual Studio output, you will be able to see the that the positive feelings are 3 and negative feelings 1:
   

   Show variables in output

Conclusion about Sentiment Analysis in SSIS

In this article, we learned how to use text analytics using Microsoft Azure Services. Specifically, we use a sentiment detector to detect if sentiments are positive or negative using Artificial Intelligence.
Also, to detect that, we need to convert the text to JSON format. We learn how to convert data from an SQL Server database to JSON using our SSIS Export to JSON task. We also used our REST API to invoke the Microsoft Text Analytic services in Azure to detect the sentiments of our data.
Finally, we use the JSON Source to export the information to SQL Server and count the positive and negative sentiments.
To conclude, If you want to try yourself, you can download the ZappySys SSIS PowerPack here.

References to Sentiment Analysis in SSIS

Finally, for more information about Text Analytics, JSON conversion, refer to these links:

• Microsoft Azure Text Analytics
• Call REST API using SSIS Web Service Task / JSON / XML Source
• How to detect sentiment in Text Analytics

The post Sentiment Analysis in SSIS using Azure AI – Machine Learning REST API appeared first on ZappySys Blog.