In this blog post you will see how easy it is to load large amount of data from SQL Server to FTP/SFTP. SSIS Data Load – SQL Server to FTP/SFTP (Split Files, GZip).

For demo purpose we will use SQL Server as relational source but you can use same steps for any database engine such as Oracle, MySQL, DB2. In this post we will use Export CSV Task and ZS Secure FTP Task (SFTP, FTP, FTPS) to achieve desired integration with FTP/SFTP/FTPS with drag and drop approach. You can also export data to JSON or XML or Excel file to FTP/SFTP/FTPS using same techniques (Use Export JSON Task  or Export XML Task or Export Excel Task ).

Our goal is to achieve following things

• Extract large amount of data from SQL Server Table or Query and export to CSV files
• Generate CSV files in compressed format (*.gz) to speedup upload
• Split CSV files by row count
• Upload data to FTP/SFTP/FTPS using highly parallel manner for maximum speed

There are three different ways you can achieve data export to Secure FTP using SSIS.

1. Method-1 (Fastest): Use two step process (First export SQL Server data to local files using Export Task and then upload files to Secure FTP using  ZS Secure FTP Task (SFTP, FTP, FTPS) )
2. Method-2 (Slower): Use Export Task with Secure FTP Connection as Target rather than save to Local files.
3. Method-3 (Slower): Use Data flow components like Secure FTP CSV File Destination  (for JSON / XML / EXCEL   use Method1 or Method2)

Each method has its own advantage / disadvantage. If you prefer to upload / compress / split large amount of data then we recommend Method#1 (Two steps). If you have not very huge dataset then you can use Method#2 or Method#3. For Last method you can only use CSV export option (we don’t have JSON/ XML / EXCEL Destination for Secure FTP yet – we may add in future)

Screenshot of SSIS Package

Extract SQL Server Data to CSV files in SSIS (Bulk export) Split / GZip Compress / upload files to FTP/SFTP/FTPS

Method-1 : Upload SQL data to Secure FTP in Two steps

In this section we will see first method (recommended) to upload SQL data to Secure FTP. This is the fastest approach if you have lots of data to upload.  In this approach we first create CSV files from SQL Server data on local disk using SSIS Export CSV Task. After that in second step we upload all files to Secure FTP Storage using ZS Secure FTP Task (SFTP, FTP, FTPS).

Step-1: Configure Source Connection in Export CSV Task

To extract data from SQL Server you can use Export CSV Task. It has many options which makes it possible to split large amount of data into multiple files. You can specify single table or multiple tables as your data source.

For multiple table use vertical bar. e.g.  dbo.Customers|dbo.Products|dbo.Orders. When you export this it will create 3 files ( dbo.Customers.csv , dbo.Products.csv, dbo.Orders.csv )

Steps:

1. Drag ZS Export CSV Task from Toolbox
2. Double click task to configure
3. From connection drop down select New connection option (OLEDB or ADO.net)
4. Once connection is configured for Source database specify SQL Query to extract data as below
   

   Export SQL Server Table or Query as CSV file (Bulk export in SSIS)

5. Now go to target tab. Here you can specify full path for file. e.g. c:\ssis\temp\cust.csv

Step-2: Compress CSV Files in SSIS ( GZIP format – *.gz )

Above steps will export file as CSV format without splitting or compression. But to compress file once exported you can go to Target tab of Export CSV Task and check [Compress file to *.gz format] option.

Compress exported SQL Server data files to GZip ( *.gz) in SSIS Export CSV Task

Step-3: Split CSV files by row count or data size in SSIS

Now lets look at how to split exported CSV files into multiple files so we can upload many files in parallel. Goto Split Options and check [Enable Split by Size/Rows]

Using SSIS Split Exported CSV files (Split by row count or size)

Step-4: Upload CSV files to Secure FTP – Using multi threaded option

Now final thing is use ZS Secure FTP Task (SFTP, FTP, FTPS) to upload files to Secure FTP.

Steps:

1. Drag ZS Secure FTP Task (SFTP, FTP, FTPS) from SSIS toolbox
2. Double click ZS Secure FTP Task (SFTP, FTP, FTPS) to configure it
3. Specify Action = Upload Files To FTP Server
4. Specify Source file path (or pattern) e.g. c:\SSIS\temp\*.*
5. Now in the Target connection dropdown click [New]
6. When Connection UI opens Enter your Account, Secret Key (Leave all other parameters default if you not sure)
7. Click Test and close connection UI
8. On the Target path on ZS Secure FTP Task (SFTP, FTP, FTPS) enter your bucket and folder path where you want to upload local files. (For example: /datafolder/sqldata/)
9. Click ok and Run package to test full package

Method-2 : Upload SQL data to Secure FTP without local stage (One step)

Now let’s change previous approach little bit to send SQL server data directly to FTP/SFTP/FTPS without any Landing area on local disk.  Export CSV Task , Export JSON Task , Export Excel Task and Export XML Task all of them supports Azure Blob / Amazon S3 and Secure FTP (SFTP) connection as target (Only available in Pro Edition). We will use this feature in following section.

This approach helps to avoid any local disk need and it may be useful for security reason for some users. However drawback of this approach is, it wont use parallel threads to upload large amount of data like previous method.

Following change will be needed on Export task to upload SQL data directly to Azure / FTP or Amazon storage.

Export SQL data to multiple files to Amazon S3, Azure Blob, Secure FTP (SFTP) in Stream Mode. Compress GZip, Overwrite, Split Options

Method-3 : Using Secure FTP CSV File Destination – Generate Secure FTP file from any source

Now let’s look at third approach to save data from any SSIS Source to FTP/SFTP/FTPS file. Advantage of this approach is you are not limited to few source options provided by Export CSV Task. If you have complex data transformation needed in Data Flow before sending data to FTP/SFTP/FTPS then use this approach.  We will use Secure FTP CSV File Destination as below

1. Drag SSIS Data flow task from toolbox
2. Create necessary source connection (e.g. OLEDB connection)
3. Create Secure FTP Connection (Right click in Connection Managers panel in bottom and click New connection and select ZS-SFTP type )
4. Once connection managers are created Go to data flow designer and Drag OLEDB Source
5. Configure OLEDB Source to read desired data from source system (e.g. SQL Server / Oracle)
6. Once source is configured drag ZS Secure FTP CSV File Destination from SSIS toolbox
7. Double click Secure FTP CSV File Destination and configure as below
   1. On Connection Managers tab select Secure FTP Connection (We created in earlier section).
   2. Properties tab configure like below screenshot
   3. On Input Columns tab select desired column you like to write in the target file. Your name from upstream will be taken as is for target file. So make sure to name upstream columns correctly.
   4. Click OK to save UI
8. Execute package and check your Secure FTP to see files got created.
   

   Loading SQL Server data into Secure FTP Files (Split, Compress Gzip Options) – SSIS Secure Ftp CSV File Destination

Conclusion

In this post you have seen how easy it is to upload / archive your SQL Server data (or any other RDBMS data) to FTP/SFTP/FTPS in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to integrate SQL Server and Secure FTP using SSIS.

The post SSIS Data Load – SQL Server to FTP/SFTP (Split Files, GZip) appeared first on ZappySys Blog.

In our previous blog, we saw how to perform SFTP / FTP File Operations in SSIS. Now in this blog, we will see How to Read SFTP / FTP Files in SSIS (CSV, JSON, XML Format files). To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like to perform file operations on FTP Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on how to Read files from secure FTP Storage (CSV, JSON and XML Files) and load into SQL Server Table in few clicks.

Components Mentioned in this article

	Secure FTP Source for JSON File
	Secure FTP Source for XML File
	Secure FTP Source for CSV File
	Secure FTP Destination for CSV File

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. You have obtained FTP/SFTP Credential.
4. Finally, do not forget to install ZappySys SSIS PowerPack

What is FTP, FTPS, SFTP, FTP/SSL?

Many people still have confusion over various protocols around FTP terminology. Most common terms you hear are as below.

• FTP
• Classic FTP
• Secure FTP
• SFTP
• FTPS
• FTP/SSL
• FTP over SSL
• FTP Over SSH

You must be overwhelmed by now? So are they all same or different? See below for quick clarification in nutshell.

Mainly there are 2 protocols for FTP and yes they are very different but still used to do Files Transfer to/from the remote server.

1. Classic FTP – sometimes referred as just FTP (Typically server runs on Port 21)
2. SFTP – Sometimes referred as Secure FTP or FTP over SSH (Typically server runs on Port 22)

So in short SFTP is not the same as the Classic FTP protocol. SFTP is more secure because traffic is always encrypted. On the other hand, Classic FTP is not encrypted by default but you can use FTP over SSL to request encrypted traffic in classic FTP. FTP over SSL also referred to as FTPS or FTP/SSL.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Secure FTP Source for CSV/JSON/XML File that will help you in reading CSV, JSON and XML Files from FTP to the Local machine, Upload files(s) to FTP Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to download files from FTP Storage.

You can connect to your FTP by entering your FTP credentials.

Read SFTP / FTP Files in SSIS (CSV, JSON, XML)

Let´s start with an example. First of all, In this SSIS FTP Source for CSV/JSON/XML File task example, we will read CSV/JSON/XML files from FTP to SQL Server database.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Now drag and drop relevant Secure FTP for CSV/JSON/XML File Task from the SSIS Toolbox.
   

   Drag and Drop FTP

3. Create a connection for Secure FTP Account.
   

   Create FTP Connection

4. Likewise, select the relevant single file to read from FTP in their relevant source of CSV/JSON/XML File Task.
   

   Select File From FTP Storage

5. Similarly, we can also read the multiple files stored in FTP Storage using wildcard pattern supported e.g. dbo.tblNames*.csv / dbo.tblNames*.json / dbo.tblNames*.xml in relevant source task
   

   Use wildcard pattern .* to read multiple files data

6. We can also read the zip and gzip compressed files also without extracting it in the specific FTP Storage for CSV/JSON/XML File Task.
   

   Reading zip and gzip compressed files (stream mode)

7. Finally, we are ready to load this file(s) data into the SQL Server.

Load Secure FTP files data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

Above all, in this blog, we learned how to Read Secure FTP Storage Files in SSIS. Even more, we used Secure FTP Source for CSV File, Secure FTP Source for JSON File and Secure FTP Source for XML File to read the file(s) from FTP Storage and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, You can use the following links for more information:

• SSIS Secure FTP Source for CSV/JSON/XML File
• About SSIS PowerPack
• About Secure FTP Task

The post Read SFTP / FTP Files in SSIS (CSV, JSON, XML) appeared first on ZappySys Blog.