In our last post (Regex Cheat Sheet) we explained use cases of SSIS Regular Expression Parser Task . Now lets look at some real world usecase. Basically in this article we are going to Extract data from audit log using SSIS. For demo purpose we will use log generated by SSIS Export CSV File Task output. When you export many tables dynamically using this task you want to know how many Row count exported for each Table. This information is logged in SSIS Output Log but no easy way to capture in SSIS variable. So what we will do we will use few tricks to capture that data using Regular Expression usecase.

So let’s get started.

Setup Export CSV Task (Output Multiple Tables to CSV Files Dynamically)

For example purpose we will use Export CSV File Task but you can use this technique for any other Tasks.

1. Drag and Drop ZS Export CSV Task from SSIS Toolbox and configure as below.
   
2.   Configure Export CSV Task to output multiple tables as explained here
3. Now create 2 SSIS User Variables
   1. FilePath (String Type)
   2. RowCount (String Type)

That’s it… now in the next section we will setup event handler to capture output from Log.

Setup EventHandle / Extract Audit Data using Regex

1. Now select task and click Event Handler tab
2. Select Executable Name from the dropdown, Select Event Name OnInformation from Handler Dropdown
3. Click Create Hyper Link to Create Event Handler like below
   

   Create Task Level Event Handler in SSIS

4. Now Drag below 3 tasks and connect like this
   1. Script Task (We will use as dummy start (No configuration needed)
   2. ZS Regular Expression Parser Task
   3. ZS Logging Task (Optional) but we will use to Output Extracted Values
      
5. Connect Dummy Script Task to ZS Regular Expression Parser Task and Right click on Green connected Arrow and Use Expression as below. Click OK to save
6. Configure ZS Regular Expression Parser Task as below
   1. Enter

      {{System::ErrorDescription}}

        as Direct Value
   2. Enter Below two mappings
      1. For RowCount set
         

         Total (\d+) records written to : (.*){{1,1}}

      2. For FilePath set
         

         Total (\d+) records written to : (.*){{1,2}}

7. Here is how it will look like
   

   Extract data from audit log using SSIS Regular Expression Parser Task

8. Thats it… Now connect 2nd step to last Logging Task…. in that Task you can set Text like below just to log extracted data. You can also use Execute SQL Task to log into Database Table
   

   {{User::FilePath}} ===> {{User::RowCount}}

9. Now run Package and you will see audit data we extracted for each file. Once File is exported we capture OnInformation event… and if Message contains substring like “records written to” then we extract data into variable for Row Count and Path then you can save to desired place for auditing purpose

The post Extract audit data from SSIS Execution Log (EventHandler and Regular Expression) appeared first on ZappySys Blog.

There are several tricks related to converting SQL to CSV. For example, dates, working with variables, converting text, working with stored procedures. In this article, we will work with several tips to convert SQL Server data to the CSV format.

For this article, we will use the following ZappySys component:

SSIS Export CSV File Task

How to convert SQL to CSV using specific dates

1. First, to convert the date to a specific format, you can use the Export CSV task.
2. This task allows you to convert SQL Server tables into CSV files. We will show how to set the date to a specific format:
   

   Export from SSIS SQL Server to CSV in a specific date format
3. The query used for this purpose is the following:
   

   SELECT [OrderID] ,FORMAT([OrderDate], 'dd/MM/yyyy') as mydateformat FROM [dbo].[Orders]

4. As you can see, you can send custom queries in the Export CSV task.
   In this example, we utilize the T-SQL Format function, which enables us to convert the date to a custom format. In this example, we are converting the OrderDate column into the format dd/MM/yyyy.
5. Also, specify the destination path:
   

   Export path csv
6. Finally, run the package. In this example, we converted the date to the format dd/MM/yyyy.

How to convert SQL Server to CSV using dynamic column names

1. A common question is to have column names based on SSIS variables. This time, we will show how to do this.
2. First of all, we will need an SSIS variable that will contain the column name:
   

   SSIS variable with a column name
3. Secondly, we need to use the value of the variable as the column name. We will use the alias (AS in SQL) to create a custom name.
   

   SSIS column name

4. The SQL query will look like this:
   

   SELECT [ShipperID] as {{User::column}} FROM [Northwind].[dbo].[Shippers]

5. The previous example used the SSIS variable named column to store the name of the column. If the Variable value is ‘column1’, the name of the column will be ‘column1’.

How to convert SQL Server to CSV to change the decimal point to a decimal comma

1. Most of the countries in the world use the decimal point or the decimal comma. For decimals, some countries use the comma and others the period. In this new example, we will show how to convert decimal data into a comma.
2. In order to convert, we will convert and work with a text data type. We will use the following script to create a table with data:
   

   create table decimalpoint
   (
   decimal nvarchar(50)
   )
   
   insert into decimalpoint values 
   ('3,45'),
   ('3,44'),
   ('6,56')

3. Secondly, we will use the following T-SQL Sentence: select replace(decimal,’,’,’.’) as mydecimal from decimalpoint.
   

   select replace(decimal,',','.') from decimalpoint

4. In addition, we will create a path to export.
5. If everything is fine, the CSV file created will convert the period decimal to a coma decimal.

How to export data from a parameterized SQL Server stored procedure into a
CSV file

1. In this new example, we will use a stored procedure with parameters and store the results into a CSV file.
2.  First of all, we will use the following parameterized stored procedure:
   

   create procedure [dbo].[Sales by Year]
   @Beginning_Date DateTime, @Ending_Date DateTime AS
   SELECT Orders.ShippedDate, Orders.OrderID, "Order Subtotals".Subtotal, 
   DATENAME(yy,ShippedDate) AS Year
   FROM Orders INNER JOIN "Order Subtotals" 
   ON Orders.OrderID = "Order Subtotals".OrderID
   WHERE Orders.ShippedDate Between @Beginning_Date And @Ending_Date
   GO

3. Secondly, we will invoke the stored procedure like this:
   

   Invoice stored procedure and export to csv

4. Finally, if everything is fine, the data will be exported to the CSV file after running the package.

Conclusion

In this article, we demonstrate how to export data to CSV from SQL Server using various date formats. We also learned how to change the types of decimals used. For more fun, please download our components.

The post How to convert SQL to CSV in SSIS appeared first on ZappySys Blog.

In this article, we will cover the points on how to load 10 million rows from SQL Server to Snowflake in just 3 minutes.

Snowflake is a data warehousing platform that resides in a cloud. Basically, it is a data warehouse software exposed as a service. It allows integrating many data sources via internal Snowflake Partner apps and loading them into the Snowflake storage engine. Another part of Snowflake is a computing engine that is responsible for serving your SQL queries. Both engines can work independently thus users that are querying a data warehouse are not affected by a data load that is happening at the same time. Snowflake is an elastic service, which means you pay only for the resources used. Specifically, you pay only for each second of the processing time you use.

To show how data loading works in Snowflake, we will take 10 million rows from SQL Server and load them into Snowflake using SSIS and ZappySys SSIS PowerPack. These SSIS PowerPack connectors will be used to achieve the task:

Export CSV File Task

Step-by-step – How to load 10 million rows from SQL Server to Snowflake in 3 minutes

Getting started

To achieve the goal, we will use a slightly modified Northwind database (example database from Microsoft) and SnowSQL – a command-line tool provided by Snowflake. So in the first steps, you will need to install the Northwind database and SnowSQL. After that, we will proceed to create a table in Snowflake – a table we will load the data into – and a file format which will be used to load data from the Snowflake staging area into the destination table. Once that’s complete, we will proceed in creating an SSIS package, adding and configuring the connectors, and finally running it to get the results. Let’s proceed!

Install a Northwind database

Download and run the creation script of a modified Northwind database. The only thing that was modified is a “CustomersForSnowflake” view added which returns 10M rows.

Install and configure SnowSQL command-line tool

Download and install SnowSQL; you will find the instructions on how to do that in https://docs.snowflake.net/manuals/user-guide/snowsql-install-config.html.

Once installed, you will need to configure the default Snowflake account name, user name, and password:

1. Go to Windows File Explorer and enter this path:
   %USERPROFILE%\.snowsql\
   Usually, when entered the path looks similar to this one:
   C:\Users\myUserName\.snowsql
2. Find a file named config and open it.
3. Then configure your credentials:

Create a table in Snowflake

Login to your Snowflake account, open a Worksheet and execute this query:

create or replace table Customers(
ID number,
CustomerID nchar(5),
CompanyName nvarchar(40),
ContactName nvarchar(30),
ContactTitle nvarchar(30),
Address nvarchar(60),
City nvarchar(15),
Region nvarchar(15),
PostalCode nvarchar(10),
Country nvarchar(15),
Phone nvarchar(24),
Fax nvarchar(24)
)

We will load 10M customers from the Northwind database and load them into this table.

Create a file format in Snowflake

Then in the same Worksheet create a file format for zipped CSV files by executing this query:

create or replace file format GzipCsvFormat
type = csv
field_delimiter = ','
null_if = ('NULL', 'null')
empty_field_as_null = true
compression = gzip
field_optionally_enclosed_by = '"'

We will use this file format when loading data from a Snowflake stage to the Snowflake destination table. This file format defines CSV format which is used by ZappySys Export CSV Task in SSIS.

Create an SSIS package

We are ready to create a new SSIS package and load some data into Snowflake. We are going to use two methods of how we load data into Snowflake. In one method we will load data into Snowflake’s local storage, while in the second one we will stage data in Amazon S3 bucket. Decide how you want to stage files, and then choose the appropriate workflow when creating a new package:

Loading 10 million rows from SQL Server to Snowflake

Add & configure ZappySys Export CSV Task

Firstly, drag and drop ZappySys Export CSV Task from SSIS toolbox and follow the instructions on how to configure it:

Configure source

1. Proceed with configuring the data source, from which you will be exporting data. We will use the Northwind database as an example:
   

   Export CSV Task: configuring OLE DB Connection to export data from SQL Server to Snowflake

2. Then move on and select it and enter the query or table name you want to export data from:
   

   Export CSV Task: configuring source SQL query for data loading from SQL Server to Snowflake

   We will be using this query:
   select top 10000000 * from CustomersForSnowflake

3. Then in the Split Options tab split the exported CSV into many files, e.g. into 50 MB chunks:
   

   Using data split options in the Export CSV Task

   NOTE: Snowflake recommends having files that are 10-100 MB in size when compressed. So it is perfectly safe to configure the value to 100 MB and above.

Configure target

Depending on which staging approach you chose to use, set the export target to a local path or S3 bucket:

Using local Snowflake storage

Just set the appropriate Save Mode and a file path:

Saving files locally to upload them to Snowflake local stage later

Using S3 storage

For storing staging files in S3, follow these steps:

1. Set Save Mode to Save to Connection and select <New ZS-AWS-STORAGE Connection>:
   

   Export CSV Task: configuring Amazon S3 endpoint as the target for data export

2. Then select S3 as Storage Service and fill in Access and Secret Key, select your region (optional):
   

   Configuring Amazon Connection Manager to store staging data in S3

3. Then in the Export CSV Task window, in Target tab configure it similarly:
   

   Export CSV Task: configuring target to store staging data in S3

   Another option you may consider is to use Amazon Storage Task or Azure Storage Task to upload files to S3 in a separate step after the Export CSV Step. For this perform the following steps.

   1. On the Target tab, set Save Mode to Save to Path (Local Disk) instead of Connection to S3 or Azure.
   2. Drag & Drop Amazon Storage Task or Azure Storage Task  and connect with the previous Export CSV Step
   3. Configure Storage Task to upload Local files to S3 or Azure Blob Storage.
   4. Continue to the next section for more instructions.

Add Execute Process Task to create a staging area

We are ready to add Execute Process Task to create a staging area in Snowflake. Again, depending on where you will store staging files, Snowflake’s local storage or Amazon S3, use one of the approaches below:

Create Snowflake local staging

Creating a local stage in Snowflake using SnowSQL command-line tool and SSIS

File path:

C:\Program Files\Snowflake SnowSQL\snowsql.exe

Arguments:

-q "CREATE OR REPLACE STAGE CustomersStaging" -d DEMO_DB -s Public

Create Amazon S3 staging

Creating an Amazon S3 stage in Snowflake using SnowSQL command-line tool and SSISFile path:

C:\Program Files\Snowflake SnowSQL\snowsql.exe

Arguments:

CREATE OR REPLACE STAGE CustomersStaging url='s3://your-bucket-name/destinationFolder/' credentials=(aws_key_id='AKIAXXXXXXXXXXXXXXXX' aws_secret_key='6p1ayaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaLXz88')

Add Execute Process Task to upload files to the staging area (local staging approach only)

We still need to upload CSV files that were saved locally for local Snowflake staging. So add another Execute Process Task and configure it similarly:

Uploading local CSV files to Snowflake stage using SnowSQL command-line tool and SSISFile path:

C:\Program Files\Snowflake SnowSQL\snowsql.exe

Arguments:

-q "PUT file://e:/temp/*.csv @DEMO_DB.PUBLIC.CustomersStaging"

Add Execute Process Task to copy files from the Snowflake/Amazon S3 stage to Snowflake table

Finally, add the final Execute Process Task to issue a command to load files from the stage into a real Snowflake table:

Copying data from a staging area to a Snowflake table using SSIS

File path:

C:\Program Files\Snowflake SnowSQL\snowsql.exe

Arguments:

-q "COPY INTO CUSTOMERS FROM @CustomersStaging file_format = (format_name = 'GZIPCSVFORMAT')" -d DEMO_DB -s Public

Execute the package

We are ready to execute the package, so just run it. In this example you see the workflow when using Amazon S3 as stage:

Executing an SSIS package to load data from Snowflake to SQL Server

The Results

Once the package executes, we can check the results:

Successful execution of SSIS package when loading data from Snowflake to SQL Server

If we go to Snowflake and execute the SELECT query, we see all 10M customers loaded:

Successful loading of 10 million rows from SQL Server to Snowflake using SSIS and ZappySys SSIS PowerPack

All done in less than 3 mins:

Conclusion

We achieved just what we aimed for – we loaded 10 million rows from SQL Server to Snowflake; all done in less than 3 minutes! Two approaches were taken into consideration: in one we staged CSV files in the local Snowflake stage and in the other we staged files in the Amazon S3 bucket. To accomplish this we used ZappySys Export CSV Task, which allowed us to export data from the SQL Server view to CSV format, split files into smaller ones, zip them and store them locally or upload to the Amazon S3 bucket. Finally, we used a standard SSIS Task “Execute Process Task” to issue Snowflake commands such as creating the stage area in Snowflake, uploading CSVs into the local stage and loading data into the Snowflake table.

Download a sample package

Load 10M rows from SQL Server into Snowflake.dtsx.zip

References

https://docs.snowflake.net/manuals/user-guide/data-load-bulk.html

https://docs.snowflake.net/manuals/user-guide/snowsql.html

https://docs.snowflake.net/manuals/user-guide/data-load-s3-create-stage.html

https://docs.snowflake.net/manuals/sql-reference/sql/create-file-format.html

https://docs.snowflake.net/manuals/sql-reference/sql/put.html

https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-table.html

The post Load 10M rows from SQL Server to Snowflake in 3 minutes appeared first on ZappySys Blog.

In this blog post you will see how easy it is to load large amount of data from SQL Server to FTP/SFTP. SSIS Data Load – SQL Server to FTP/SFTP (Split Files, GZip).

For demo purpose we will use SQL Server as relational source but you can use same steps for any database engine such as Oracle, MySQL, DB2. In this post we will use Export CSV Task and ZS Secure FTP Task (SFTP, FTP, FTPS) to achieve desired integration with FTP/SFTP/FTPS with drag and drop approach. You can also export data to JSON or XML or Excel file to FTP/SFTP/FTPS using same techniques (Use Export JSON Task  or Export XML Task or Export Excel Task ).

Our goal is to achieve following things

• Extract large amount of data from SQL Server Table or Query and export to CSV files
• Generate CSV files in compressed format (*.gz) to speedup upload
• Split CSV files by row count
• Upload data to FTP/SFTP/FTPS using highly parallel manner for maximum speed

There are three different ways you can achieve data export to Secure FTP using SSIS.

1. Method-1 (Fastest): Use two step process (First export SQL Server data to local files using Export Task and then upload files to Secure FTP using  ZS Secure FTP Task (SFTP, FTP, FTPS) )
2. Method-2 (Slower): Use Export Task with Secure FTP Connection as Target rather than save to Local files.
3. Method-3 (Slower): Use Data flow components like Secure FTP CSV File Destination  (for JSON / XML / EXCEL   use Method1 or Method2)

Each method has its own advantage / disadvantage. If you prefer to upload / compress / split large amount of data then we recommend Method#1 (Two steps). If you have not very huge dataset then you can use Method#2 or Method#3. For Last method you can only use CSV export option (we don’t have JSON/ XML / EXCEL Destination for Secure FTP yet – we may add in future)

Screenshot of SSIS Package

Extract SQL Server Data to CSV files in SSIS (Bulk export) Split / GZip Compress / upload files to FTP/SFTP/FTPS

Method-1 : Upload SQL data to Secure FTP in Two steps

In this section we will see first method (recommended) to upload SQL data to Secure FTP. This is the fastest approach if you have lots of data to upload.  In this approach we first create CSV files from SQL Server data on local disk using SSIS Export CSV Task. After that in second step we upload all files to Secure FTP Storage using ZS Secure FTP Task (SFTP, FTP, FTPS).

Step-1: Configure Source Connection in Export CSV Task

To extract data from SQL Server you can use Export CSV Task. It has many options which makes it possible to split large amount of data into multiple files. You can specify single table or multiple tables as your data source.

For multiple table use vertical bar. e.g.  dbo.Customers|dbo.Products|dbo.Orders. When you export this it will create 3 files ( dbo.Customers.csv , dbo.Products.csv, dbo.Orders.csv )

Steps:

1. Drag ZS Export CSV Task from Toolbox
2. Double click task to configure
3. From connection drop down select New connection option (OLEDB or ADO.net)
4. Once connection is configured for Source database specify SQL Query to extract data as below
   

   Export SQL Server Table or Query as CSV file (Bulk export in SSIS)

5. Now go to target tab. Here you can specify full path for file. e.g. c:\ssis\temp\cust.csv

Step-2: Compress CSV Files in SSIS ( GZIP format – *.gz )

Above steps will export file as CSV format without splitting or compression. But to compress file once exported you can go to Target tab of Export CSV Task and check [Compress file to *.gz format] option.

Compress exported SQL Server data files to GZip ( *.gz) in SSIS Export CSV Task

Step-3: Split CSV files by row count or data size in SSIS

Now lets look at how to split exported CSV files into multiple files so we can upload many files in parallel. Goto Split Options and check [Enable Split by Size/Rows]

Using SSIS Split Exported CSV files (Split by row count or size)

Step-4: Upload CSV files to Secure FTP – Using multi threaded option

Now final thing is use ZS Secure FTP Task (SFTP, FTP, FTPS) to upload files to Secure FTP.

Steps:

1. Drag ZS Secure FTP Task (SFTP, FTP, FTPS) from SSIS toolbox
2. Double click ZS Secure FTP Task (SFTP, FTP, FTPS) to configure it
3. Specify Action = Upload Files To FTP Server
4. Specify Source file path (or pattern) e.g. c:\SSIS\temp\*.*
5. Now in the Target connection dropdown click [New]
6. When Connection UI opens Enter your Account, Secret Key (Leave all other parameters default if you not sure)
7. Click Test and close connection UI
8. On the Target path on ZS Secure FTP Task (SFTP, FTP, FTPS) enter your bucket and folder path where you want to upload local files. (For example: /datafolder/sqldata/)
9. Click ok and Run package to test full package

Method-2 : Upload SQL data to Secure FTP without local stage (One step)

Now let’s change previous approach little bit to send SQL server data directly to FTP/SFTP/FTPS without any Landing area on local disk.  Export CSV Task , Export JSON Task , Export Excel Task and Export XML Task all of them supports Azure Blob / Amazon S3 and Secure FTP (SFTP) connection as target (Only available in Pro Edition). We will use this feature in following section.

This approach helps to avoid any local disk need and it may be useful for security reason for some users. However drawback of this approach is, it wont use parallel threads to upload large amount of data like previous method.

Following change will be needed on Export task to upload SQL data directly to Azure / FTP or Amazon storage.

Export SQL data to multiple files to Amazon S3, Azure Blob, Secure FTP (SFTP) in Stream Mode. Compress GZip, Overwrite, Split Options

Method-3 : Using Secure FTP CSV File Destination – Generate Secure FTP file from any source

Now let’s look at third approach to save data from any SSIS Source to FTP/SFTP/FTPS file. Advantage of this approach is you are not limited to few source options provided by Export CSV Task. If you have complex data transformation needed in Data Flow before sending data to FTP/SFTP/FTPS then use this approach.  We will use Secure FTP CSV File Destination as below

1. Drag SSIS Data flow task from toolbox
2. Create necessary source connection (e.g. OLEDB connection)
3. Create Secure FTP Connection (Right click in Connection Managers panel in bottom and click New connection and select ZS-SFTP type )
4. Once connection managers are created Go to data flow designer and Drag OLEDB Source
5. Configure OLEDB Source to read desired data from source system (e.g. SQL Server / Oracle)
6. Once source is configured drag ZS Secure FTP CSV File Destination from SSIS toolbox
7. Double click Secure FTP CSV File Destination and configure as below
   1. On Connection Managers tab select Secure FTP Connection (We created in earlier section).
   2. Properties tab configure like below screenshot
   3. On Input Columns tab select desired column you like to write in the target file. Your name from upstream will be taken as is for target file. So make sure to name upstream columns correctly.
   4. Click OK to save UI
8. Execute package and check your Secure FTP to see files got created.
   

   Loading SQL Server data into Secure FTP Files (Split, Compress Gzip Options) – SSIS Secure Ftp CSV File Destination

Conclusion

In this post you have seen how easy it is to upload / archive your SQL Server data (or any other RDBMS data) to FTP/SFTP/FTPS in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to integrate SQL Server and Secure FTP using SSIS.

The post SSIS Data Load – SQL Server to FTP/SFTP (Split Files, GZip) appeared first on ZappySys Blog.

In this blog post you will see how easy it is to load large amount of data from SQL Server to Azure Blob Storage using SSIS. We will export / compress data to multiple files.

For demo purpose we will use SQL Server as relational source but you can use same steps for any database engine such as Oracle, MySQL, DB2. In this post we will use Export CSV Task and Azure Blob Storage Task  to achieve desired integration with Azure Blob with drag and drop approach. You can also export JSON or XML data to Azure Blob using same techniques (Use Export JSON Task  or Export XML Task ).

Our goal is to achieve following things

• Extract large amount of data from SQL Server Table or Query and export to CSV files
• Generate CSV files in compressed format (*.gz) to speedup upload and save data transfer cost to Azure
• Split CSV files by row count
• Upload data to Azure Blob using highly parallel manner for maximum speed

There are three different ways you can achieve data export to Azure Blob using SSIS.

1. Method-1 (Fastest): Use two step process (First export SQL Server data to local files using Export Task and then upload files to Azure using  Azure Blob Storage Task  )
2. Method-2 (Slower): Use Export Task with Azure Blob Connection as Target rather than save to Local files.
3. Method-3 (Slower): Use Data flow components like Azure Blob Destination for CSV  (for JSON / XML  use Method1 or Method2)

Each method has its own advantage / disadvantage. If you prefer to upload / compress / split large amount of data then we recommend Method#1 (Two steps). If you have not very huge dataset then you can use Method#2 or Method#3. For Last method you can only use CSV export option (we don’t have JSON/ XML Destination for Azure Blob yet – we may add in future)

Screenshot of SSIS Package

Extract SQL Server Data to CSV files in SSIS (Bulk export) Split / GZip Compress / upload files to Azure Blob Storage

Method-1 : Upload SQL data to Azure Blob in Two steps

In this section we will see first method (recommended) to upload SQL data to Azure Blob. This is the fastest approach if you have lots of data to upload.  In this approach we first create CSV files from SQL Server data on local disk using SSIS Export CSV Task. After that in second step we upload all files to Azure Blob using SSIS Azure Storage Task.

Step-1: Configure Source Connection in Export CSV Task

To extract data from SQL Server you can use Export CSV Task. It has many options which makes it possible to split large amount of data into multiple files. You can specify single table or multiple tables as your data source.

For multiple table use vertical bar. e.g.  dbo.Customers|dbo.Products|dbo.Orders. When you export this it will create 3 files ( dbo.Customers.csv , dbo.Products.csv, dbo.Orders.csv )

Steps:

1. Drag ZS Export CSV Task from Toolbox
2. Double click task to configure
3. From connection drop down select New connection option (OLEDB or ADO.net)
4. Once connection is configured for Source database specify SQL Query to extract data as below
   

   Export SQL Server Table or Query as CSV file (Bulk export in SSIS)

5. Now go to target tab. Here you can specify full path for file. e.g. c:\ssis\temp\azure\cust.csv

Step-2: Compress CSV Files in SSIS ( GZIP format – *.gz )

Above steps will export file as CSV format without splitting or compression. But to compress file once exported you can go to Target tab of Export CSV Task and check [Compress file to *.gz format] option.

Compress exported SQL Server data files to GZip ( *.gz) in SSIS Export CSV Task

Step-3: Split CSV files by row count or data size in SSIS

Now lets look at how to split exported CSV files into multiple files so we can upload many files in parallel. Goto Split Options and check [Enable Split by Size/Rows]

Using SSIS Split Exported CSV files (Split by row count or size)

Step-4: Upload CSV files to Azure Blob – Using multi threaded option

Now final thing is use Azure Storage Task to upload files to Azure.

Steps:

1. Drag ZS Azure Storage Task from SSIS toolbox
2. Double click Azure Storage Task to configure it
3. Specify Action = UploadFilesToAzure
4. Specify Source file path (or pattern) e.g. c:\SSIS\temp\azure\*.*
5. Now in the Target connection dropdown click [New]
6. When Connection UI opens Enter your Account, Secret Key (Leave all other parameters default if you not sure)
7. Click Test and close connection UI
8. On the Target path on Azure Storage Task enter your bucket and folder path where you want to upload local files. For example your container name is bw-east-1 and folder is sqldata then enter as below
   bw-east-1/sqldata/
9. Click ok and Run package to test full package

Method-2 : Upload SQL data to Azure Blob without local stage (One step)

Now let’s change previous approach little bit to send SQL server data directly to Azure Blob without any Landing area on local disk.  Export CSV Task , Export JSON Task  and Export XML Task all of them supports Azure Blob / Azure Blob and Secure FTP (SFTP) connection as target (Only available in Pro Edition). We will use this feature in following section.

This approach helps to avoid any local disk need and it may be useful for security reason for some users. However drawback of this approach is, it wont use parallel threads to upload large amount of data like previous method.

Following change will be needed on Export task to upload SQL data directly to Azure / FTP or Azure storage.

Export SQL data to multiple files to Azure Blob, Amazon S3, Secure FTP (SFTP) in Stream Mode using SSIS. Configure Compress GZip, Overwrite, Split Options

Method-3 : Using Azure Blob destination – Generate Azure Blob file from any source

Now let’s look at third approach to save data from any SSIS Source to Azure Blob file. Advantage of this approach is you are not limited to few source options provided by Export CSV Task. If you have complex data transformation needed in Data Flow before sending data to Azure then use this approach.  We will use Azure Blob Destination for CSV as below

1. Drag SSIS Data flow task from toolbox
2. Create necessary source connection (e.g. OLEDB connection)
3. Create Azure Blob Connection (Right click in Connection Managers panel in bottom and click New connection and select ZS-Azure-STORAGE type )
4. Once connection managers are created Go to data flow designer and Drag OLEDB Source
5. Configure OLEDB Source to read desired data from source system (e.g. SQL Server / Oracle)
6. Once source is configured drag ZS Azure Blob CSV File Destination from SSIS toolbox
7. Double click Azure Destination and configure as below
   1. On Connection Managers tab select Azure Connection (We created in earlier section).
   2. Properties tab configure like below screenshot
   3. On Input Columns tab select desired column you like to write in the target file. Your name from upstream will be taken as is for target file. So make sure to name upstream columns correctly.
   4. Click OK to save UI
8. Execute package and check your Azure Bucket to see files got created.

Loading SQL Server data into Azure Container Files (Split, Compress Gzip Options) – SSIS Azure Blob CSV File Destination

Conclusion

In this post you have seen how easy it is to upload / archive your SQL Server data (or any other RDBMS data) to Azure Blob Storage in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to integrate SQL Server and Azure Blob using SSIS.

The post SSIS Data Load – SQL Server to Azure Blob (Split Files, GZip) appeared first on ZappySys Blog.

In this blog post you will see how easy it is to load large amount of data from SQL Server to Amazon S3 Storage. For demo purpose we will use SQL Server as relational source but you can use same steps for any database engine such as Oracle, MySQL, DB2. In this post we will use Export CSV Task and Amazon S3 Storage Task  to achieve desired integration with Amazon S3 with drag and drop approach. You can also export JSON or XML data to Amazon S3 using same techniques (Use Export JSON Task  or Export XML Task ).

Our goal is to achieve following things

• Extract large amount of data from SQL Server Table or Query and export to CSV files
• Generate CSV files in compressed format (*.gz) to speedup upload and save data transfer cost to S3
• Split CSV files by row count
• Upload data to Amazon S3 using highly parallel manner for maximum speed

There are three different ways you can achieve data export to Amazon S3 using SSIS.

1. Method-1 (Fastest): Use two step process (First export SQL Server data to local files using Export Task and then upload files to S3 using  Amazon S3 Storage Task  )
2. Method-2 (Slower): Use Export Task with Amazon S3 Connection as Target rather than save to Local files.
3. Method-3 (Slower): Use Data flow components like Amazon S3 Destination for CSV  (for JSON / XML  use Method1 or Method2)

Each method has its own advantage / disadvantage. If you prefer to upload / compress / split large amount of data then we recommend Method#1 (Two steps). If you have not very huge dataset then you can use Method#2 or Method#3. For Last method you can only use CSV export option (we don’t have JSON/ XML Destination for Amazon S3 yet – we may add in future)

Screenshot of SSIS Package

Extract SQL Server Data to CSV files in SSIS (Bulk export) and Split / GZip Compress / upload files to Amazon S3 (AWS Cloud)

Method-1 : Upload SQL data to Amazon S3 in Two steps

In this section we will see first method (recommended) to upload SQL data to Amazon S3. This is the fastest approach if you have lots of data to upload.  In this approach we first create CSV files from SQL Server data on local disk using SSIS Export CSV Task. After that in second step we upload all files to Amazon S3 using SSIS Amazon Storage Task.

Step-1: Configure Source Connection in Export CSV Task

To extract data from SQL Server you can use Export CSV Task. It has many options which makes it possible to split large amount of data into multiple files. You can specify single table or multiple tables as your data source.

For multiple table use vertical bar. e.g.  dbo.Customers|dbo.Products|dbo.Orders. When you export this it will create 3 files ( dbo.Customers.csv , dbo.Products.csv, dbo.Orders.csv )

Steps:

1. Drag ZS Export CSV Task from Toolbox
2. Double click task to configure
3. From connection drop down select New connection option (OLEDB or ADO.net)
4. Once connection is configured for Source database specify SQL Query to extract data as below
   

   Export SQL Server Table or Query as CSV file (Bulk export in SSIS)

5. Now go to target tab. Here you can specify full path for file. e.g. c:\ssis\temp\s3dump\cust.csv

Step-2: Compress CSV Files in SSIS ( GZIP format – *.gz )

Above steps will export file as CSV format without splitting or compression. But to compress file once exported you can go to Target tab of Export CSV Task and check [Compress file to *.gz format] option.

Compress exported SQL Server data files to GZip ( *.gz) in SSIS Export CSV Task

Step-3: Split CSV files by row count or data size in SSIS

Now lets look at how to split exported CSV files into multiple files so we can upload many files in parallel. Goto Split Options and check [Enable Split by Size/Rows]

Using SSIS Split Exported CSV files (Split by row count or size)

Step-4: Upload CSV files to Amazon S3 – Using multi threaded option

Now final thing is use Amzon S3 Task to upload files to S3.

Things to remember

Sometimes times due to high network activity you may get timeout errors during upload. In that case you can adjust few settings described here. Also try to reduce total parallel threads on S3 Connection see that helps.

Steps:

1. Drag ZS Amazon Storage Task from SSIS toolbox
2. Double click Amazon Storage Task to configure it
3. Specify Action = UploadFilesToAmazon
4. Specify Source file path (or pattern) e.g. c:\SSIS\temp\s3dump\*.*
5. Now in the Target connection dropdown click [New]
6. When Connection UI opens select Service Type = S3
7. Enter your Access Key, Secret Key and Region (Leave all other parameters default if you not sure)
8. Click Test and close connection UI
9. On the Target path on S3 Storage Task enter your bucket and folder path where you want to upload local files. For example your bucket name is bw-east-1 and folder is sqldata then enter as below
   bw-east-1/sqldata/
10. Click ok and Run package to test full package

Upload local files to Amazon S3 using SSIS AWS Storage Task

Method-2 : Upload SQL data to Amazon S3 without local stage (One step)

Now let’s change previous approach little bit to send SQL server data directly to Amazon S3 without any Landing area on local disk.  Export CSV Task , Export JSON Task  and Export XML Task all of them supports Amazon S3 / Azure Blob and Secure FTP (SFTP) connection as target (Only available in Pro Edition). We will use this feature in following section.

This approach helps to avoid any local disk need and it may be useful for security reason for some users. However drawback of this approach is, it wont use parallel threads to upload large amount of data like previous method.

Following change will be needed on Export task to upload SQL data directly to S3 / FTP or Azure storage.

Export SQL data to multiple files to Amazon S3, Azure, Secure FTP (SFTP) in Stream Mode using SSIS. Configure Compress GZip, Overwrite, Split Options

Method-3 : Using Amazon S3 destination – Generate Amazon S3 file from any source

Now let’s look at third approach to save data from any SSIS Source to Amazon S3 file. Advantage of this approach is you are not limited to few source options provided by Export CSV Task. If you have complex data transformation needed in Data Flow before sending data to S3 then use this approach.  We will use Amazon S3 Destination for CSV as below

1. Drag SSIS Data flow task from toolbox
2. Create necessary source connection (e.g. OLEDB connection)
3. Create Amazon S3 Connection (Right click in Connection Managers panel in bottom and click New connection and select ZS-AMAZON-STORAGE type )
4. Once connection managers are created Go to data flow designer and Drag OLEDB Source
5. Configure OLEDB Source to read desired data from source system (e.g. SQL Server / Oracle)
6. Once source is configured drag ZS Amazon S3 CSV File Destination from SSIS toolbox
7. Double click S3 Destination and configure as below
   1. On Connection Managers tab select S3 Connection (We created in earlier section).
   2. Properties tab configure like below screenshot
   3. On Input Columns tab select desired column you like to write in the target file. Your name from upstream will be taken as is for target file. So make sure to name upstream columns correctly.
   4. Click OK to save UI
8. Execute package and check your S3 Bucket to see files got created.

Loading SQL Server data into S3 Bucket Files (Split, Compress Gzip Options) – SSIS Amazon S3 CSV File Destination

Conclusion

In this post you have seen how easy it is to upload / archive your SQL Server data (or any other RDBMS data) to Amazon S3 Storage in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to integrate SQL Server and Amazon S3 using SSIS.

The post Loading data from SQL Server to Amazon S3 in SSIS (Split Files, GZip) appeared first on ZappySys Blog.