Read SFTP / FTP Files in SSIS (CSV, JSON, XML)

In our previous blog, we saw how to perform SFTP / FTP File Operations in SSIS. Now in this blog, we will see How to Read SFTP / FTP Files in SSIS (CSV, JSON, XML Format files). To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like to perform file operations on FTP Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on how to Read files from secure FTP Storage (CSV, JSON and XML Files) and load into SQL Server Table in few clicks.

Components Mentioned in this article

	Secure FTP Source for JSON File
	Secure FTP Source for XML File
	Secure FTP Source for CSV File
	Secure FTP Destination for CSV File

Prerequisite

First, you will need to have SSIS installed
Secondly, make sure to have SSDT
You have obtained FTP/SFTP Credential.
Finally, do not forget to install ZappySys SSIS PowerPack

What is FTP, FTPS, SFTP, FTP/SSL?

Many people still have confusion over various protocols around FTP terminology. Most common terms you hear are as below.

FTP
Classic FTP
Secure FTP
SFTP
FTPS
FTP/SSL
FTP over SSL
FTP Over SSH

You must be overwhelmed by now? So are they all same or different? See below for quick clarification in nutshell.

Mainly there are 2 protocols for FTP and yes they are very different but still used to do Files Transfer to/from the remote server.

Classic FTP – sometimes referred as just FTP (Typically server runs on Port 21)
SFTP – Sometimes referred as Secure FTP or FTP over SSH (Typically server runs on Port 22)

So in short SFTP is not the same as the Classic FTP protocol. SFTP is more secure because traffic is always encrypted. On the other hand, Classic FTP is not encrypted by default but you can use FTP over SSL to request encrypted traffic in classic FTP. FTP over SSL also referred to as FTPS or FTP/SSL.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Secure FTP Source for CSV/JSON/XML File that will help you in reading CSV, JSON and XML Files from FTP to the Local machine, Upload files(s) to FTP Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to download files from FTP Storage.

You can connect to your FTP by entering your FTP credentials.

Read SFTP / FTP Files in SSIS (CSV, JSON, XML)

Let´s start with an example. First of all, In this SSIS FTP Source for CSV/JSON/XML File task example, we will read CSV/JSON/XML files from FTP to SQL Server database.

First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit
Drag and Drop SSIS Data Flow Task from SSIS Toolbox
Now drag and drop relevant Secure FTP for CSV/JSON/XML File Task from the SSIS Toolbox.
Drag and Drop FTP
Create a connection for Secure FTP Account.
Create FTP Connection
Likewise, select the relevant single file to read from FTP in their relevant source of CSV/JSON/XML File Task.
Select File From FTP Storage
Similarly, we can also read the multiple files stored in FTP Storage using wildcard pattern supported e.g. dbo.tblNames*.csv / dbo.tblNames*.json / dbo.tblNames*.xml in relevant source task
Use wildcard pattern .* to read multiple files data
We can also read the zip and gzip compressed files also without extracting it in the specific FTP Storage for CSV/JSON/XML File Task.
Reading zip and gzip compressed files (stream mode)
Finally, we are ready to load this file(s) data into the SQL Server.

Load Secure FTP files data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

Drag and drop the Upsert Destination component from the SSIS Toolbox.
Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

Double-click the Upsert Destination component to open the configuration window.
Under Connection, select an existing target connection or click NEW to create a new connection.
- Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

In the Target Table dropdown, select the table where you want to load data.
Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

Go to the Mappings tab.
Click Auto Map to map source columns to target columns by name.
Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

Above all, in this blog, we learned how to Read Secure FTP Storage Files in SSIS. Even more, we used Secure FTP Source for CSV File, Secure FTP Source for JSON File and Secure FTP Source for XML File to read the file(s) from FTP Storage and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, You can use the following links for more information: