In this blog post you will see how easy it is to load large amount of data from SQL Server to FTP/SFTP. SSIS Data Load – SQL Server to FTP/SFTP (Split Files, GZip).

For demo purpose we will use SQL Server as relational source but you can use same steps for any database engine such as Oracle, MySQL, DB2. In this post we will use Export CSV Task and ZS Secure FTP Task (SFTP, FTP, FTPS) to achieve desired integration with FTP/SFTP/FTPS with drag and drop approach. You can also export data to JSON or XML or Excel file to FTP/SFTP/FTPS using same techniques (Use Export JSON Task  or Export XML Task or Export Excel Task ).

Our goal is to achieve following things

• Extract large amount of data from SQL Server Table or Query and export to CSV files
• Generate CSV files in compressed format (*.gz) to speedup upload
• Split CSV files by row count
• Upload data to FTP/SFTP/FTPS using highly parallel manner for maximum speed

There are three different ways you can achieve data export to Secure FTP using SSIS.

1. Method-1 (Fastest): Use two step process (First export SQL Server data to local files using Export Task and then upload files to Secure FTP using  ZS Secure FTP Task (SFTP, FTP, FTPS) )
2. Method-2 (Slower): Use Export Task with Secure FTP Connection as Target rather than save to Local files.
3. Method-3 (Slower): Use Data flow components like Secure FTP CSV File Destination  (for JSON / XML / EXCEL   use Method1 or Method2)

Each method has its own advantage / disadvantage. If you prefer to upload / compress / split large amount of data then we recommend Method#1 (Two steps). If you have not very huge dataset then you can use Method#2 or Method#3. For Last method you can only use CSV export option (we don’t have JSON/ XML / EXCEL Destination for Secure FTP yet – we may add in future)

Screenshot of SSIS Package

Extract SQL Server Data to CSV files in SSIS (Bulk export) Split / GZip Compress / upload files to FTP/SFTP/FTPS

Method-1 : Upload SQL data to Secure FTP in Two steps

In this section we will see first method (recommended) to upload SQL data to Secure FTP. This is the fastest approach if you have lots of data to upload.  In this approach we first create CSV files from SQL Server data on local disk using SSIS Export CSV Task. After that in second step we upload all files to Secure FTP Storage using ZS Secure FTP Task (SFTP, FTP, FTPS).

Step-1: Configure Source Connection in Export CSV Task

To extract data from SQL Server you can use Export CSV Task. It has many options which makes it possible to split large amount of data into multiple files. You can specify single table or multiple tables as your data source.

For multiple table use vertical bar. e.g.  dbo.Customers|dbo.Products|dbo.Orders. When you export this it will create 3 files ( dbo.Customers.csv , dbo.Products.csv, dbo.Orders.csv )

Steps:

1. Drag ZS Export CSV Task from Toolbox
2. Double click task to configure
3. From connection drop down select New connection option (OLEDB or ADO.net)
4. Once connection is configured for Source database specify SQL Query to extract data as below
   

   Export SQL Server Table or Query as CSV file (Bulk export in SSIS)

5. Now go to target tab. Here you can specify full path for file. e.g. c:\ssis\temp\cust.csv

Step-2: Compress CSV Files in SSIS ( GZIP format – *.gz )

Above steps will export file as CSV format without splitting or compression. But to compress file once exported you can go to Target tab of Export CSV Task and check [Compress file to *.gz format] option.

Compress exported SQL Server data files to GZip ( *.gz) in SSIS Export CSV Task

Step-3: Split CSV files by row count or data size in SSIS

Now lets look at how to split exported CSV files into multiple files so we can upload many files in parallel. Goto Split Options and check [Enable Split by Size/Rows]

Using SSIS Split Exported CSV files (Split by row count or size)

Step-4: Upload CSV files to Secure FTP – Using multi threaded option

Now final thing is use ZS Secure FTP Task (SFTP, FTP, FTPS) to upload files to Secure FTP.

Steps:

1. Drag ZS Secure FTP Task (SFTP, FTP, FTPS) from SSIS toolbox
2. Double click ZS Secure FTP Task (SFTP, FTP, FTPS) to configure it
3. Specify Action = Upload Files To FTP Server
4. Specify Source file path (or pattern) e.g. c:\SSIS\temp\*.*
5. Now in the Target connection dropdown click [New]
6. When Connection UI opens Enter your Account, Secret Key (Leave all other parameters default if you not sure)
7. Click Test and close connection UI
8. On the Target path on ZS Secure FTP Task (SFTP, FTP, FTPS) enter your bucket and folder path where you want to upload local files. (For example: /datafolder/sqldata/)
9. Click ok and Run package to test full package

Method-2 : Upload SQL data to Secure FTP without local stage (One step)

Now let’s change previous approach little bit to send SQL server data directly to FTP/SFTP/FTPS without any Landing area on local disk.  Export CSV Task , Export JSON Task , Export Excel Task and Export XML Task all of them supports Azure Blob / Amazon S3 and Secure FTP (SFTP) connection as target (Only available in Pro Edition). We will use this feature in following section.

This approach helps to avoid any local disk need and it may be useful for security reason for some users. However drawback of this approach is, it wont use parallel threads to upload large amount of data like previous method.

Following change will be needed on Export task to upload SQL data directly to Azure / FTP or Amazon storage.

Export SQL data to multiple files to Amazon S3, Azure Blob, Secure FTP (SFTP) in Stream Mode. Compress GZip, Overwrite, Split Options

Method-3 : Using Secure FTP CSV File Destination – Generate Secure FTP file from any source

Now let’s look at third approach to save data from any SSIS Source to FTP/SFTP/FTPS file. Advantage of this approach is you are not limited to few source options provided by Export CSV Task. If you have complex data transformation needed in Data Flow before sending data to FTP/SFTP/FTPS then use this approach.  We will use Secure FTP CSV File Destination as below

1. Drag SSIS Data flow task from toolbox
2. Create necessary source connection (e.g. OLEDB connection)
3. Create Secure FTP Connection (Right click in Connection Managers panel in bottom and click New connection and select ZS-SFTP type )
4. Once connection managers are created Go to data flow designer and Drag OLEDB Source
5. Configure OLEDB Source to read desired data from source system (e.g. SQL Server / Oracle)
6. Once source is configured drag ZS Secure FTP CSV File Destination from SSIS toolbox
7. Double click Secure FTP CSV File Destination and configure as below
   1. On Connection Managers tab select Secure FTP Connection (We created in earlier section).
   2. Properties tab configure like below screenshot
   3. On Input Columns tab select desired column you like to write in the target file. Your name from upstream will be taken as is for target file. So make sure to name upstream columns correctly.
   4. Click OK to save UI
8. Execute package and check your Secure FTP to see files got created.
   

   Loading SQL Server data into Secure FTP Files (Split, Compress Gzip Options) – SSIS Secure Ftp CSV File Destination

Conclusion

In this post you have seen how easy it is to upload / archive your SQL Server data (or any other RDBMS data) to FTP/SFTP/FTPS in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to integrate SQL Server and Secure FTP using SSIS.

The post SSIS Data Load – SQL Server to FTP/SFTP (Split Files, GZip) appeared first on ZappySys Blog.

In our previous post we saw how to create excel dynamically in SSIS. Now let’s advance further and find out how to create Excel reports in SSIS using Powerful Template engine (introduced in PowerPack 2.7.4)  In this post you will learn how to use ZappySys Export Excel Task to export data from Multiple SQL Tables to Single or Multiple Excel Sheets, we will use Formatting, Expressions and Filtering to export parent child datasets (Master-Detail) using Template approach which supports full customization as per your need.

We will be using following ZappySys SSIS PowerPack component to make things work in this tutorial:

SSIS Export Excel File Task (Dynamically generate)

You can check the following video to learn basic things about Export Excel Task. In future we will post video tutorial to show how Template engine works.

Requirements

1. First of all, MS Excel installed.
2. Secondly, an Excel template that you can download here.
3. In addition, you will need SQL Server installed.
4. Aso, the SQL Server Northwind sample database that you can download here.
5. Additionally, SSDT for Business Intelligence.
6. Finally, use the SSIS Excel package example that you can download here.

Concept about Excel Template Engine

ZappySys Export Excel Template engine is compatible with FlexCel engine described here. So read it carefully to get most details about how template engine works and what elements supported in Template engine. Basically you have to use Template file which will be used to generate final excel report in SSIS. Advantage of Template engine approach compared to Simple export is you can control every aspect of your report. Also you can create nested reports use multiple datasets in a single sheet (Master-Detail JOINS) which is not possible in default export mode. The only down side of this approach is… learning curve. However once you understand concepts of Template Engine it wont take you too much time to master it.

In Template file you can define bands (e.g. Customer Info, Order Info…). Each band can grow in Vertical or Horizontal direction. Band must be created by Named Range in Excel. You must name your range in a very specific way. Based on how you suffix and prefix your named range it will decide how to grow your band (i.e. Expand Vertical or Horizontal).

Few things to remember with Template Engine.

• Each Template file must have at least one Band which can be used to attach Dataset.
• Each Template file must have Config Sheet somewhere.
• You can use many functions in your Template using Tag system
• You can define Charts, Graphs any elements you like in Excel Template and link to Ranges you define.

Getting started

In next few section we will see step by step instructions on how to use export SQL server data to Excel using Template File approach.

Prepare Sample Tables (Source data)

In this example, we will create an Excel report based in two SQL Server tables from the Northwind database mentioned in the requirements. We are going to use Northwind dataset. You can download SQL Script file from here. Just download / extract and run it in SSMS to create necessary Tables with Sample Data.

1. First of all, we will see the data displayed in the Customer table:
   

   Northwind table in SQL Server

2. Secondly, we will check the data of the Orders table. Note that the CustomerID is included in this table, so we can create a master-detail report based on these 2 tables:
   

   SQL Server Orders

Prepare Excel Template

Now let’s create a  template file which will be used to share our exported file. To make demo simple we will use this already prepared template file. Lets understand how template file was created.

We assume you have read concepts of template engine by reading this link.

Basically few things to check in Template file

1. Defining correct Config Tab (Only needed if you are using special features such as reusable expression, or filters (i.e. subset of data).
2. Defining bands which is bound to dataset

Define Reusable Configuration Settings ( <#Config> tab )

In our example we are creating 2 extra tabs (UK, US) using filter expression (subset of dataset) so we need to use Config setting for that purpose. Notice that Config tab must be names as <#Config> and Reusable sections for Data, Format and Expression must be defined as below Cell A10 , Cell H10, Cell L10.

Define Config Settings for Excel Report

Create Bands

Now let’s look at how we define bands in template file. Read more about different types of bands in Excel Template Engine. Band are defined using Named ranges. In our example we use two bands in first Sheet1.

• For customer data __cust__  (start / end with double underscore)
• For orders data __ord__X (Read more on “X” ranges )
• For US tab  __cust_us__
• For UK tab __cust_uk__

Designing Excel Report Template (Define Ranges for Master-Detail Report)

Using Filter (Subset of data) in Excel Report

Create SSIS Package

Now let’s use Our sample package to get started. You can create your own Package but we have already created one to make demo simple.

1. In an SSIS project add the Demo SSIS package from the step 5 of the requirements:
   

   SSIS add package

2. In addition, in the SSIS project, drag and drop the ZS Export Excel task and Create a connection to the Northwind Database by clicking New for Source Connection.
3. and enter the source tables/queries you like to use in your report. In our example, we will use two tables.
   Syntax for Source data is

   alias1=Table1_Or_Query1 | alias2=Table2_Or_Query2 .... 

   In this example, we will use  cust= select * from dbo.Customers | ord=dbo.orders.
   You can also use just the table names for both e.g. cust=dbo.Customers | ord=dbo.orders
   
   

   Source table/query for Excel Report

    

4. Check the option to Use Template Engine To get the access to Excel Report Template Tab.
   

   Use Template Engine

5. Now go to Excel Report Template tab, enter the path of the template file which we downloaded in step 2 of the requirements and write the relationship between the customer and order tables (cust.CustomerID=ord.CustomerID):
   

   Specify template file for Excel report, define relationship for master-detail report

    

6. Finally, in the Target tab specify the path of the Excel Target and run the package:
   

   SSIS target in Excel

Checking the results of the package to create an Excel Report in SSIS using a template

1. Once that you run the package, you can check the results. First of all, we will check the Sheet1. It will contain the orders grouped by customerID:
   

   Excel report exported in SSIS based on a template

    

2. Secondly, in the US sheet in Excel, it will show the customers filtered by the country where the country is US:
   

   filter data by country in Excel

    

3. Also, in the UK sheet, it will show all the customers where the country is equal to UK:
   

   Show data where the country is the United Kingdom

Add Dynamic Comments

Sometimes you might want to add dynamic comments. Several cells may need comments, similar to tooltips, which come from another data column in the database.

To achieve this, add the comment to the cell in the template file and place a placeholder column under the specified column.

Place the placeholder column under the specified column.

This will allow you to achieve the desired output. The Excel file will be generated with comments in this format.

Excel File with Comments

Other Examples

There are many other examples of template engine use case found here. Refer to related Github link and download xlsx files from Github. You can use those template file path to run demo in SSIS.

Pivot Table Example

For example to run Pivot Table demo using Template Engine download below files

Pivot Tables.template.xlsx

ReportHeader.xls

Found under below path

https://github.com/tmssoftware/TMS-FlexCel.NET-demos/tree/master/csharp/VS2019/netframework/Modules/20.Reports/80.Pivot%20Tables

Conclusion

To conclude, we learned how to create a report from data in SQL Server and export based on a template. The template not only included a nice format in Excel, but also filter and relationships between two tables. If you liked our products, enjoy them HERE.

References

For more information about creating an Excel Report in SSIS using a template, refer to the following links:

• Getting Started with FlexCel Studio for the .NET Framework
• SSIS Export Excel File Task (Dynamically generate)
• SSIS Export Excel Task – Generate Excel Dynamically

The post Create Excel Report in SSIS using a template (Export Formatted xlsx File) appeared first on ZappySys Blog.

SSIS comes with out-of-the-box support for reading and writing to Excel. However, it’s very restrictive if you want to make things dynamic, as any metadata inside DataFlow cannot be changed at runtime. Here are a few problems with using the native Excel Source or Destination.

• Metadata cannot be changed at runtime.
• Datatype guessing nightmare – only the first few rows are scanned, and the wrong datatype is selected.
• Cannot export multiple tables to Excel.
• Cannot split data into multiple tabs based on specific grouping criteria
• Cannot specify the location where data has to be written

SSIS export to Excel dynamically using Excel Export Task

If you want to generate Excel dynamically without any of the above hassle, then you can check the SSIS Excel Export Task. It comes with many advantages not found in the native Microsoft SSIS Excel Destination

Here is the list of a few features for the SSIS Excel Export Task

• Export multiple tables/views to Excel files (e.g., Sales% or do SalesJan|SalesFeb).
• Support for all major office versions (e.g, Office 2003, 2007, 2013+).
• Export SQL query output to an Excel file.
• A completely dynamic approach without limitations on DataFlow Strict Metadata.
• Option to export data to a specific starting location in an Excel Sheet (e.g, A5 will start writing to the  5th row, 1st column).
• Support for a password-protected Excel file.
• Option to clear a range of cells before writing data to an existing Excel workbook (e.g., A1:D5000 will clear A to D columns from 1st to 5000 rows).
• Automatically Split exported Excel data into multiple files by Size or Number of records.
• Automatically Split exported Excel data into multiple files or tabs when Split By Column (e.g., SplitBy=Country will create a new file for each country). If the SplitOnTab option is specified, then data is written to a new Excel tab rather than a file for each new group.
• Support for Compression and Append.
• Fully managed means it works out of the box on both 32-bit and 64-bit systems without any changes (Note that Native SSIS Excel functionality is 32-bit only).
• Support for SQL Server 2005, 2008, 2012, 2014 (32-bit and 64-bit).

Step-By-Step

SSIS Export to Excel File – Generate Excel files for selected tables/views

SSIS Export to Excel File – Generate Excel files for selected tables/views

SSIS Export to Excel File – Split Excel Data into Multiple Files

SSIS Export to Excel File – Split Excel Data into Multiple sheets

SSIS Export to Excel File – Date Formatting, Encoding Options

SSIS Export to Excel File – Execution Log

Conclusion

In this post, we have seen how Excel generation can be a headache when you have dynamic metadata. You can use the SSIS Excel Export Task to solve many of these challenges.

The post SSIS export to excel dynamically (supports multiple tables) appeared first on ZappySys Blog.