In this post, we will see how to create SSIS Package Programmatically to use ZappySys SSIS PowerPack. This blog post assumes you have basic knowledge of Programming language like C# and you have installed Visual Studio Express or a similar product (VS Code not supported).

Add SSIS Dll References

The very first thing is to create a C# Project (i.e. Console or WinForm). Add SSIS dll reference.

You will need the following references in your project. The below example is for SSIS 2017 (Version 14.x.x)

• For SSIS 2012 change to Version=11.0.0.0
• For SSIS 2014 change to Version=12.0.0.0
• For SSIS 2016 change to Version=13.0.0.0
• For SSIS 2017 change to Version=14.0.0.0
• For SSIS 2019 change to Version=15.0.0.0
• And may be future versions same way ….

<Reference Include="Microsoft.SqlServer.Dts.Design"/>
<Reference Include="Microsoft.SqlServer.DTSPipelineWrap" />
<Reference Include="Microsoft.SQLServer.DTSRuntimeWrap" />
<Reference Include="Microsoft.SQLServer.ManagedDTS" />
<Reference Include="Microsoft.SqlServer.PipelineHost" />

Step-By-Step Example (Create SSIS Package Programmatically – C#)

Now let’s see each piece to create SSIS Package in code and later we will show the full code.

Import Namespaces

First step is at the top you need to import some namespaces.

using Microsoft.SqlServer.Dts.Runtime;
using Microsoft.SqlServer.Dts.Pipeline;
//using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
//using Microsoft.SqlServer.Dts.Runtime.Wrapper;

Create SSIS Package

Here how you can create SSIS Package instance in code

var p = new Package();

Add SSIS Connection

Now lets add connection manager to our package instance. To add connection you will need two pieces. Creation Name (i.e. ZS-SFTP) and ConnectionString. You can find creation name from exsing Dtsx file by opening in text editor.

var conn1=p.Connections.Add("ZS-SFTP");
conn1.Name = "MySftpConnection";
conn1.ConnectionString= "Protocol=Sftp;ServerName=127.0.0.1;Port=0;LogonType=Normal;UserName=zstest;Password=zstest123";

How to find CreationName of SSIS Connection manager

To find the creation name of your Connection Type open an existing DTSX file where you had similar connection type in the package. Locate node like below and check CrewationName. It can be a friendly name or can be fully qualified assembly name. Both ways is fine but a friendly name is easy to remember.

<DTS:ConnectionManagers>
  <DTS:ConnectionManager
    DTS:refId="Package.ConnectionManagers[RedshiftConn1]"
    DTS:CreationName="ZS-POSTGRESQL"
    .........
    .........

How to find ConnectionString

Now another thing you need to know is the exact connectionstring. You can go to SSIS Designer and right-click on Connection Icon and check Properties Grid. It will show you ConnectionString or you can use the above technique > Open DTSX file in notepad and find connectinstring there.

ConnectionString doesnt include any passwords / secrets. So manually append those properties in your connectionstring when you set in code.

Add SSIS Task

Now let’s add task in our SSIS Package instance. To create SSIS Task programmatically in code, you need to know the exact type name (i.e. creation name). As we discussed earlier check your dtsx file to know the creation name for the task you like to create. In the below code we used a friendly name which is not supported in SSIS 2012 so for that you must use full name. For SSIS 2014 or higher you can use friendly names.

You can also find it in file like this (example of SSIS 2017 and 2019 folders)
C:\Program Files (x86)\Microsoft SQL Server\140\DTS\UpgradeMappings\ZappySys.PowerPack140.Extensions.xml
C:\Program Files (x86)\Microsoft SQL Server\150\DTS\UpgradeMappings\ZappySys.PowerPack150.Extensions.xml

//var SSIS_VERSION = "110"; //110=SQL2012, 120=SQL2014, 130=SQL2016, 140=SQL2017, 150=SQL2019
//For SSIS 2012
var SFTP_TASK_2012 = "ZappySys.PowerPack.Task.SecureFtpTask.SecureFtpTask, ZappySys.PowerPack110, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502";
//For SSIS 2014 or higher
//Goto folder like this and find zappysys extension file 
//(e.g. 140) => C:\Program Files (x86)\Microsoft SQL Server\140\DTS\UpgradeMappings\ZappySys.PowerPack140.Extensions.xml
//Find Task creation name there
var SFTP_TASK = "ZappySys.SecureFtpTask";

var tsk1 = p.Executables.Add(SFTP_TASK) as TaskHost;
tsk1.Properties["SourceStorageConnection"].SetValue(tsk1, "MySftpConnection");
tsk1.Properties["StorageAction"].SetValue(tsk1, 10); //DownloadFileFromFtp [10] find numeric value for this enum from help file> properties https://zappysys.com/onlinehelp/ssis-powerpack/scr/ssis-sftp-ftp-ftps-task.htm
tsk1.Properties["SourcePathValue"].SetValue(tsk1, "subfolder/localsubfile1.csv");
tsk1.Properties["TargetPathValue"].SetValue(tsk1, @"c:\temp\");
tsk1.Properties["FileOverwriteAction"].SetValue(tsk1, 2); //OverwriteAll [2]

Execute SSIS Package

Now let’s execute the package using the below line

var status =p.Execute();

Check Execution Errors

At the end you need to check execution errors if any.

if (status != DTSExecResult.Success)
{
	string errs = p.Errors.Cast<DtsError>().Aggregate("",
           (current, err) => current + string.Format("Source={0}, Err={1}", err.Source, err.Description));
	throw new Exception("Package status was >> " + status + " >> " + errs);
}

Full Code

Here is the full code

using System;
using System.Linq;
using System.Runtime.InteropServices;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime;
/*
For SSIS 2012 - Compile with .net 4.0
For SSIS 2014-2017 - Compile with .net 4.5 
For SSIS 2019 - Compile with .net 4.6.2
 */

//current one is SSIS 2017 (140 dlls)
/*
 * References needed for SSIS 2017 , for 2019 change 14.0.0.0 to 15.0.0.0
 * 
    <Reference Include="Microsoft.SqlServer.DtsMsg, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL" />
    <Reference Include="Microsoft.SqlServer.Dts.Design, Version=14.0.0.0, PublicKeyToken=89845dcd8080cc91" />
    <Reference Include="Microsoft.SqlServer.DTSPipelineWrap, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL" />
    <Reference Include="Microsoft.SQLServer.DTSRuntimeWrap, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91" />
    <Reference Include="Microsoft.SQLServer.ManagedDTS, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL" />
    <Reference Include="Microsoft.SqlServer.PipelineHost, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL" /> 
 */
namespace SSISCodeDemo
{
    class Program
    {
        static void Main(string[] args)
        {
            
            //Create_SSIS_Controlflow_Task_Demo();
            Create_SSIS_Dataflow_Source_Component_Demo();

            //Create_SSIS_Connection_Demo();
            //Create_SSIS_Variable_Demo();
            //Create_SSIS_Connect_Tasks_Demo();

        }

        private static void Create_SSIS_Controlflow_Task_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();

            var conn1 = p.Connections.Add("ZS-SFTP");
            conn1.Name = "MySftpConnection";
            conn1.ConnectionString = "Protocol=Sftp;ServerName=127.0.0.1;Port=0;LogonType=Normal;UserName=zstest;Password=zstest123";

            //var SSIS_VERSION = "110"; //110=SQL2012, 120=SQL2014, 130=SQL2016, 140=SQL2017, 150=SQL2019
            //For SSIS 2012
            var SFTP_TASK_2012 = "ZappySys.PowerPack.Task.SecureFtpTask.SecureFtpTask, ZappySys.PowerPack110, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502";
            //For SSIS 2014 or higher
            //Goto folder like this and find zappysys extension file (e.g. 140) => C:\Program Files (x86)\Microsoft SQL Server\140\DTS\UpgradeMappings\ZappySys.PowerPack140.Extensions.xml
            //Find Task creation name there
            var SFTP_TASK = "ZappySys.SecureFtpTask";

            var tsk1 = p.Executables.Add(SFTP_TASK) as TaskHost;
            tsk1.Name = "SFTP_Task1";
            tsk1.Properties["SourceStorageConnection"].SetValue(tsk1, "MySftpConnection");
            tsk1.Properties["StorageAction"].SetValue(tsk1, 10); //DownloadFileFromFtp [10] find numeric value for this enum from help file> properties https://zappysys.com/onlinehelp/ssis-powerpack/scr/ssis-sftp-ftp-ftps-task.htm
            tsk1.Properties["SourcePathValue"].SetValue(tsk1, "subfolder/localsubfile1.csv");
            tsk1.Properties["TargetPathValue"].SetValue(tsk1, @"c:\temp\");
            tsk1.Properties["FileOverwriteAction"].SetValue(tsk1, 2); //OverwriteAll [2]


            /*
             Go to package file (*.dtsx) > Open in Notepad > and findout which properties set. Do same way (ignore some default proprties)
             Some enum properties needs numeric values > Check Task / Component help file to findout number for that

      <DTS:ObjectData>
        <SecureFtpTask
          StorageAction="DeleteFtpFile"
          FileOverwriteAction="OverwriteAll"
          TreatRegXForFullPath="False"
          ContinueOnError="False"
...................
...................
...................            
            PrefixTimestamp="False" />
      </DTS:ObjectData>
    </DTS:Executable>             
             */

            //Application.

            /* use below code to save to disk file 
            var app = new Microsoft.SqlServer.Dts.Runtime.Application();
            var filepath = @"c:\temp\demo.dtsx";
            app.SaveToXml(filepath, p, null);

            --load from disk to memory
            p = app.LoadPackage(filepath, null);

            */
            var status = p.Execute();

            if (status != DTSExecResult.Success)
            {
                var errs = "";
                foreach (var e in p.Errors)
                {
                    errs = errs + string.Format("Source={0}, Err={1}", e.Source, e.Description) + "\r\n";
                }

                throw new Exception("Package status was >> " + status + " >> " + errs);
            }

            Console.WriteLine("Execution Completed... Press any key to exit!!!");
            Console.ReadKey();
        }
        private static void Create_SSIS_Dataflow_Source_Component_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();


            /*============ STEP-1: Create Dataflow Task =========== */

            var tsk1 = p.Executables.Add("STOCK:PipelineTask") as TaskHost;
            var dataFlowTask = tsk1.InnerObject as MainPipe;

            /*============ STEP-2: Create Connection =========== */
            var conn1 = p.Connections.Add("ZS-HTTP");
            conn1.Name = "MyConnection";
            conn1.ConnectionString = "Url=https://httpbin.org/get;CredentialType=Basic;UserName=test;Password=test123";


            /*============ STEP-3: Create Source =========== */
            var sourceComponent = dataFlowTask.ComponentMetaDataCollection.New();

            try
            {
                sourceComponent.ComponentClassID = "ZappySys.JsonSource";
                var sourceInstance = sourceComponent.Instantiate();
                sourceInstance.ProvideComponentProperties();

                sourceComponent.Name = "MySource";
                /**** Special Case for JSON Source - if user wants connection to use then we create Runtime connection. For all other components this is always created when ProvideComponentProperties() is called*/
                if (sourceComponent.RuntimeConnectionCollection.Count <= 0)
                {
                    var dtsRuntimeConnection = sourceComponent.RuntimeConnectionCollection.New();
                    dtsRuntimeConnection.Name = "Connection"; 
                    //dtsRuntimeConnection.Description = "This can be ZS-HTTP, ZS-OAUTH or ZS-SALESFORCE";
                }
                sourceComponent.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.GetExtendedInterface(conn1);
                sourceComponent.RuntimeConnectionCollection[0].ConnectionManagerID = conn1.ID;

                sourceInstance.SetComponentProperty("UseConnection", true);//JsonSource specific -- for all other components connection always used
                sourceInstance.SetComponentProperty("DirectPath", "https://httpbin.org/basic-auth/test/test123");
                //sourceInstance.SetComponentProperty("Filter", "$.value[*]");

                //make connection active before next step
                sourceInstance.AcquireConnections(null);

                //after connection + props set ... call ReInitialize meta... it creates columns / output / input etc
                sourceInstance.ReinitializeMetaData(); 
            }
            catch (COMException ex)
            {
                throw new Exception("Error Happened ==> " + sourceComponent.GetErrorDescription(ex.ErrorCode));
            }




            /*============ STEP-4: Create Target (without Reinitialize) =========== */

            var targetComponent = dataFlowTask.ComponentMetaDataCollection.New();
            CManagedComponentWrapper targetInstance=null;
            try
            {
                targetComponent.ComponentClassID = "ZappySys.TrashDestination";
                targetInstance = targetComponent.Instantiate();
                targetInstance.ProvideComponentProperties();

                targetComponent.Name = "MyDestination";

                targetInstance.SetComponentProperty("SaveDataToFile", true);
                targetInstance.SetComponentProperty("FilePath", @"c:\temp\dump.txt");
                targetInstance.SetComponentProperty("OverwriteFileIfExists", true);

                //--- Trash destination doesnt have connection but incase destination uses connection... set below way
                //if you have connection then set this way
                //var conn2 = p.Connections.Add("....."); //Creation name i.e. ZS-HTTP
                //conn2.Name = "MyConnection2";
                //conn3.ConnectionString = "Url=https://httpbin.org/get;CredentialType=Basic;UserName=test;Password=test123";
                //if (targetComponent.RuntimeConnectionCollection.Count <= 0)
                //{
                //    var dtsRuntimeConnection = targetComponent.RuntimeConnectionCollection.New();
                //    dtsRuntimeConnection.Name = "Connection"; 
                //}
                //sourceComponent.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.GetExtendedInterface(conn2);
                //sourceComponent.RuntimeConnectionCollection[0].ConnectionManagerID = conn2.ID;
                //make connection active before next step
                //targetInstance.AcquireConnections(null);
            }
            catch (COMException ex)
            {
                throw new Exception("Error Happened ==> " + targetComponent.GetErrorDescription(ex.ErrorCode));
            }

            /*============ STEP-5: Connect Source to Target =========== */

            //before we call reintializemetadata for dest .. lets attach src and dest so .. dest can use virtual inputs if needed
            var output = sourceComponent.OutputCollection[0]; //to make it simple lets pick first output but possible component can have more outputs
            var input = targetComponent.InputCollection[0];

            dataFlowTask.PathCollection.New().AttachPathAndPropagateNotifications(output, input);

            /*============ STEP-6: Configure Target =========== */
            try
            {
                //after connection + props set and source to target attached... call ReInitialize meta... it creates columns / output / input etc
                targetInstance.ReinitializeMetaData(); 
            }
            catch (COMException ex)
            {
                throw new Exception("Error Happened ==> " + targetComponent.GetErrorDescription(ex.ErrorCode));
            }


            /*============ STEP-6: Save Package =========== */

            //* use below code to save to disk file 
            var app = new Microsoft.SqlServer.Dts.Runtime.Application();
            var filepath = @"c:\temp\demo.dtsx";
            app.SaveToXml(filepath, p, null);

            /*============ STEP-7: Load Package =========== */
            //--load from disk to memory
            p = app.LoadPackage(filepath, null);

            //*/

            /*============ STEP-8: Execute Package =========== */
            var status = p.Execute();

            /*============ STEP-9: Print execution status / log =========== */
            if (status != DTSExecResult.Success)
            {
                var errs = "";
                foreach (var e in p.Errors)
                {
                    errs = errs + string.Format("Source={0}, Err={1}", e.Source, e.Description) + "\r\n";
                }

                throw new Exception("Package status was >> " + status + " >> " + errs);
            }

            Console.WriteLine("Execution Completed... Press any key to exit!!!");
            Console.ReadKey();
        }

        private static void Create_SSIS_Variable_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();
            
            p.Variables.Add("MyVariableName1", false, "User", "Some value");
        }

        private static void Create_SSIS_Connection_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();

            var conn1 = p.Connections.Add("ZS-SFTP");
            conn1.Name = "MySftpConnection";
            conn1.ConnectionString = "Protocol=Sftp;ServerName=127.0.0.1;Port=0;LogonType=Normal;UserName=zstest;Password=zstest123";

        }

        private static void Create_SSIS_Connect_Tasks_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();

            var SFTP_TASK = "ZappySys.SecureFtpTask";

            var tsk1 = p.Executables.Add(SFTP_TASK) as TaskHost;
            tsk1.Name = "SFTP_Task1";
            /*set some props - see Create_SSIS_Controlflow_Task_Demo */

            var tsk2 = p.Executables.Add(SFTP_TASK) as TaskHost;
            tsk2.Name = "SFTP_Task2";
            /*set some props - see Create_SSIS_Controlflow_Task_Demo */
            
            //add link between task1-->task2
            var pc = p.PrecedenceConstraints.Add(tsk1, tsk2);
            pc.EvalOp = DTSPrecedenceEvalOp.Constraint;
            pc.Value = DTSExecResult.Success;
            pc.LogicalAnd = true; //only useful if multiple inputs coming and you want all previous steps to be Green vs Any One Green .. if its any use LogicalAnd=False

            //If you need to connect with expression use below
            //pc.EvalOp = DTSPrecedenceEvalOp.ExpressionAndConstraint;
            //pc.Expression = "@SomeVariable == true";
        }

       

     
    }
}

The post How to create SSIS Package Programmatically (Add / Execute Tasks) appeared first on ZappySys Blog.

In this blog post you will see how easy it is to load large amount of data from SQL Server to FTP/SFTP. SSIS Data Load – SQL Server to FTP/SFTP (Split Files, GZip).

For demo purpose we will use SQL Server as relational source but you can use same steps for any database engine such as Oracle, MySQL, DB2. In this post we will use Export CSV Task and ZS Secure FTP Task (SFTP, FTP, FTPS) to achieve desired integration with FTP/SFTP/FTPS with drag and drop approach. You can also export data to JSON or XML or Excel file to FTP/SFTP/FTPS using same techniques (Use Export JSON Task  or Export XML Task or Export Excel Task ).

Our goal is to achieve following things

• Extract large amount of data from SQL Server Table or Query and export to CSV files
• Generate CSV files in compressed format (*.gz) to speedup upload
• Split CSV files by row count
• Upload data to FTP/SFTP/FTPS using highly parallel manner for maximum speed

There are three different ways you can achieve data export to Secure FTP using SSIS.

1. Method-1 (Fastest): Use two step process (First export SQL Server data to local files using Export Task and then upload files to Secure FTP using  ZS Secure FTP Task (SFTP, FTP, FTPS) )
2. Method-2 (Slower): Use Export Task with Secure FTP Connection as Target rather than save to Local files.
3. Method-3 (Slower): Use Data flow components like Secure FTP CSV File Destination  (for JSON / XML / EXCEL   use Method1 or Method2)

Each method has its own advantage / disadvantage. If you prefer to upload / compress / split large amount of data then we recommend Method#1 (Two steps). If you have not very huge dataset then you can use Method#2 or Method#3. For Last method you can only use CSV export option (we don’t have JSON/ XML / EXCEL Destination for Secure FTP yet – we may add in future)

Screenshot of SSIS Package

Extract SQL Server Data to CSV files in SSIS (Bulk export) Split / GZip Compress / upload files to FTP/SFTP/FTPS

Method-1 : Upload SQL data to Secure FTP in Two steps

In this section we will see first method (recommended) to upload SQL data to Secure FTP. This is the fastest approach if you have lots of data to upload.  In this approach we first create CSV files from SQL Server data on local disk using SSIS Export CSV Task. After that in second step we upload all files to Secure FTP Storage using ZS Secure FTP Task (SFTP, FTP, FTPS).

Step-1: Configure Source Connection in Export CSV Task

To extract data from SQL Server you can use Export CSV Task. It has many options which makes it possible to split large amount of data into multiple files. You can specify single table or multiple tables as your data source.

For multiple table use vertical bar. e.g.  dbo.Customers|dbo.Products|dbo.Orders. When you export this it will create 3 files ( dbo.Customers.csv , dbo.Products.csv, dbo.Orders.csv )

Steps:

1. Drag ZS Export CSV Task from Toolbox
2. Double click task to configure
3. From connection drop down select New connection option (OLEDB or ADO.net)
4. Once connection is configured for Source database specify SQL Query to extract data as below
   

   Export SQL Server Table or Query as CSV file (Bulk export in SSIS)

5. Now go to target tab. Here you can specify full path for file. e.g. c:\ssis\temp\cust.csv

Step-2: Compress CSV Files in SSIS ( GZIP format – *.gz )

Above steps will export file as CSV format without splitting or compression. But to compress file once exported you can go to Target tab of Export CSV Task and check [Compress file to *.gz format] option.

Compress exported SQL Server data files to GZip ( *.gz) in SSIS Export CSV Task

Step-3: Split CSV files by row count or data size in SSIS

Now lets look at how to split exported CSV files into multiple files so we can upload many files in parallel. Goto Split Options and check [Enable Split by Size/Rows]

Using SSIS Split Exported CSV files (Split by row count or size)

Step-4: Upload CSV files to Secure FTP – Using multi threaded option

Now final thing is use ZS Secure FTP Task (SFTP, FTP, FTPS) to upload files to Secure FTP.

Steps:

1. Drag ZS Secure FTP Task (SFTP, FTP, FTPS) from SSIS toolbox
2. Double click ZS Secure FTP Task (SFTP, FTP, FTPS) to configure it
3. Specify Action = Upload Files To FTP Server
4. Specify Source file path (or pattern) e.g. c:\SSIS\temp\*.*
5. Now in the Target connection dropdown click [New]
6. When Connection UI opens Enter your Account, Secret Key (Leave all other parameters default if you not sure)
7. Click Test and close connection UI
8. On the Target path on ZS Secure FTP Task (SFTP, FTP, FTPS) enter your bucket and folder path where you want to upload local files. (For example: /datafolder/sqldata/)
9. Click ok and Run package to test full package

Method-2 : Upload SQL data to Secure FTP without local stage (One step)

Now let’s change previous approach little bit to send SQL server data directly to FTP/SFTP/FTPS without any Landing area on local disk.  Export CSV Task , Export JSON Task , Export Excel Task and Export XML Task all of them supports Azure Blob / Amazon S3 and Secure FTP (SFTP) connection as target (Only available in Pro Edition). We will use this feature in following section.

This approach helps to avoid any local disk need and it may be useful for security reason for some users. However drawback of this approach is, it wont use parallel threads to upload large amount of data like previous method.

Following change will be needed on Export task to upload SQL data directly to Azure / FTP or Amazon storage.

Export SQL data to multiple files to Amazon S3, Azure Blob, Secure FTP (SFTP) in Stream Mode. Compress GZip, Overwrite, Split Options

Method-3 : Using Secure FTP CSV File Destination – Generate Secure FTP file from any source

Now let’s look at third approach to save data from any SSIS Source to FTP/SFTP/FTPS file. Advantage of this approach is you are not limited to few source options provided by Export CSV Task. If you have complex data transformation needed in Data Flow before sending data to FTP/SFTP/FTPS then use this approach.  We will use Secure FTP CSV File Destination as below

1. Drag SSIS Data flow task from toolbox
2. Create necessary source connection (e.g. OLEDB connection)
3. Create Secure FTP Connection (Right click in Connection Managers panel in bottom and click New connection and select ZS-SFTP type )
4. Once connection managers are created Go to data flow designer and Drag OLEDB Source
5. Configure OLEDB Source to read desired data from source system (e.g. SQL Server / Oracle)
6. Once source is configured drag ZS Secure FTP CSV File Destination from SSIS toolbox
7. Double click Secure FTP CSV File Destination and configure as below
   1. On Connection Managers tab select Secure FTP Connection (We created in earlier section).
   2. Properties tab configure like below screenshot
   3. On Input Columns tab select desired column you like to write in the target file. Your name from upstream will be taken as is for target file. So make sure to name upstream columns correctly.
   4. Click OK to save UI
8. Execute package and check your Secure FTP to see files got created.
   

   Loading SQL Server data into Secure FTP Files (Split, Compress Gzip Options) – SSIS Secure Ftp CSV File Destination

Conclusion

In this post you have seen how easy it is to upload / archive your SQL Server data (or any other RDBMS data) to FTP/SFTP/FTPS in few clicks. Try SSIS PowerPack for free and find out yourself how easy it is to integrate SQL Server and Secure FTP using SSIS.

The post SSIS Data Load – SQL Server to FTP/SFTP (Split Files, GZip) appeared first on ZappySys Blog.

In our previous blog, we saw how to perform SFTP / FTP File Operations in SSIS. Now in this blog, we will see How to Read SFTP / FTP Files in SSIS (CSV, JSON, XML Format files). To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more. They are Coding free, drag and drop high-performance suite of Custom SSIS Components and SSIS Tasks. If you like to perform file operations on FTP Files (e.g. Download, Upload, Create, Delete) then check these articles.

In nutshell, this post will focus on how to Read files from secure FTP Storage (CSV, JSON and XML Files) and load into SQL Server Table in few clicks.

Components Mentioned in this article

	Secure FTP Source for JSON File
	Secure FTP Source for XML File
	Secure FTP Source for CSV File
	Secure FTP Destination for CSV File

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. You have obtained FTP/SFTP Credential.
4. Finally, do not forget to install ZappySys SSIS PowerPack

What is FTP, FTPS, SFTP, FTP/SSL?

Many people still have confusion over various protocols around FTP terminology. Most common terms you hear are as below.

• FTP
• Classic FTP
• Secure FTP
• SFTP
• FTPS
• FTP/SSL
• FTP over SSL
• FTP Over SSH

You must be overwhelmed by now? So are they all same or different? See below for quick clarification in nutshell.

Mainly there are 2 protocols for FTP and yes they are very different but still used to do Files Transfer to/from the remote server.

1. Classic FTP – sometimes referred as just FTP (Typically server runs on Port 21)
2. SFTP – Sometimes referred as Secure FTP or FTP over SSH (Typically server runs on Port 22)

So in short SFTP is not the same as the Classic FTP protocol. SFTP is more secure because traffic is always encrypted. On the other hand, Classic FTP is not encrypted by default but you can use FTP over SSL to request encrypted traffic in classic FTP. FTP over SSL also referred to as FTPS or FTP/SSL.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS Secure FTP Source for CSV/JSON/XML File that will help you in reading CSV, JSON and XML Files from FTP to the Local machine, Upload files(s) to FTP Storage. It will also support Delete, Rename, List, Get Property, Copy, Move, Create, Set Permission … and many more operations. Here we are showing you is, How to download files from FTP Storage.

You can connect to your FTP by entering your FTP credentials.

Read SFTP / FTP Files in SSIS (CSV, JSON, XML)

Let´s start with an example. First of all, In this SSIS FTP Source for CSV/JSON/XML File task example, we will read CSV/JSON/XML files from FTP to SQL Server database.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Now drag and drop relevant Secure FTP for CSV/JSON/XML File Task from the SSIS Toolbox.
   

   Drag and Drop FTP

3. Create a connection for Secure FTP Account.
   

   Create FTP Connection

4. Likewise, select the relevant single file to read from FTP in their relevant source of CSV/JSON/XML File Task.
   

   Select File From FTP Storage

5. Similarly, we can also read the multiple files stored in FTP Storage using wildcard pattern supported e.g. dbo.tblNames*.csv / dbo.tblNames*.json / dbo.tblNames*.xml in relevant source task
   

   Use wildcard pattern .* to read multiple files data

6. We can also read the zip and gzip compressed files also without extracting it in the specific FTP Storage for CSV/JSON/XML File Task.
   

   Reading zip and gzip compressed files (stream mode)

7. Finally, we are ready to load this file(s) data into the SQL Server.

Load Secure FTP files data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

Above all, in this blog, we learned how to Read Secure FTP Storage Files in SSIS. Even more, we used Secure FTP Source for CSV File, Secure FTP Source for JSON File and Secure FTP Source for XML File to read the file(s) from FTP Storage and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, You can use the following links for more information:

• SSIS Secure FTP Source for CSV/JSON/XML File
• About SSIS PowerPack
• About Secure FTP Task

The post Read SFTP / FTP Files in SSIS (CSV, JSON, XML) appeared first on ZappySys Blog.

In this blog, we will learn how to Download Latest File from FTP using SSIS Secure FTP Task, We also learn how to connect with FTP/SFTP and also see how to Get Latest File list in ADO.net DataTable variable and How to sort ADO.net DataTable and get and set latest FTP File Path in a variable using Script Task.

We also learn how to get and set variables and how to use it in SSIS Script Task and Secure FTP Task.

Requirements

1. First, you will need to have SSIS installed
2. Also, make sure to have SSDT
3. Make sure to have installed ZappySys SSIS PowerPack.

What is FTP, FTPS, SFTP, FTP/SSL?

Many people still have confusion over various protocols around FTP terminology. Most common terms you hear are as below.

• FTP
• Classic FTP
• Secure FTP
• SFTP
• FTPS
• FTP/SSL
• FTP over SSL
• FTP Over SSH

You must be overwhelmed by now? So are they all same or different? See below for quick clarification in nutshell.

Mainly there are 2 protocols for FTP and yes they are very different but still used to do Files Transfer to/from the remote server.

1. Classic FTP – sometimes referred as just FTP (Typically server runs on Port 21)
2. SFTP – Sometimes referred as Secure FTP or FTP over SSH (Typically server runs on Port 22)

So in short SFTP is not the same as the Classic FTP protocol. SFTP is more secure because traffic is always encrypted. On the other hand, Classic FTP is not encrypted by default but you can use FTP over SSL to request encrypted traffic in classic FTP. FTP over SSL also referred as FTPS or FTP/SSL.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS – Secure FTP Task that will help you Download Latest File From FTP/SFTP, get file list in ADO.net DataTable variable and do many more actions using SSIS Script Task.

In this example, we will show 3 examples.

1. First, we will show how to get FTP File List as ADO.net DataTable in a variable using SSIS – Secure FTP Task.
2. Secondly, we will show how to sort ADO.net DataTable and get latest FTP File Path in a variable using Script Task.
3. Third, we will Download Latest File from FTP using SSIS Secure FTP Task using that latest file path variable.

Get FTP File List as ADO.net DataTable in a variable using SSIS Secure FTP Task

Follow these steps to accomplish the task:

1. Drag and drop Secure FTP Task the design panel and rename it as “Get File List as ADOnet DataTable” if you want to.

ADD SSIS – ZS SECURE FTP TASK

Connect with FTP/SFTP

Get Latest file list from FTP to ADO.net table in a variable

Set Latest ADO.net DataTable Variable

Get latest FTP File Path in a variable using Script Task

Follow these steps to accomplish the task:

1. Drag and drop SSIS Script Task the design panel and link it with the ZS SFTP Task.

Add SSIS – Script Task

 

2. Before proceeding with Script Task let’s add one new SSIS variables first that will store the Latest FTP File Path.

Add Latest File Variable

3. Now edit Script Task and add User::v_dtFileLists and User::v_LatestFilePath to ReadOnly/ReadWrite variable lists:

4. Then proceed and press “Edit Script” button to edit the code the Script Task — it will open the “ScriptMain.cs” C# file for you in new VS Project. And Write The Following Code in it.

DataTable dt = (DataTable)Dts.Variables["User::v_dtFileLists"].Value;

DataView dv = dt.DefaultView;
dv.Sort = "CreationTime desc";
DataTable sortedDT = dv.ToTable();
if (sortedDT.Rows.Count > 0)
{
	Dts.Variables["User::v_LatestFilePath"].Value = sortedDT.Rows[0][0].ToString();
}
dt=null;
dv = null;
sortedDT = null;

Get Latest File Path in Variable

5. Click Ok In “Script Task” And it will save and close the “ScriptMain.cs” Task.

6. That’s it we have successfully completed our second task. Let’s move on next step.

Download Latest File from FTP using the latest file path variable

Follow these steps to accomplish the task:

1. Drag and drop ZS Secure FTP Task the design panel and link it with Script Task.

Add SSIS – ZS Secure FTP Task

2. Let’s use that latest file variable and FTP Connection to download the latest file in the local folder. Follow the mentioned steps of the below screen.

Download Latest File using variable using SSIS -Secure FTP Task

3. Click OK to close Secure FTP TASK UI.

Finally, run the package and you will be able to see the latest file downloaded on your local folder from the FTP/SFTP.

Conclusion

To conclude, we can say that working with FTP/SFTP is now very simple. In this article, we looked at SSIS SFTP task examples on how to Download latest File from FTP using SSIS in the local folder. We also learned how to Sort ADO.net DataTable and Set latest file path in the variable using SSIS Script Task.  We used variables, expressions, and SSIS Script Task. If you liked the tasks you can start using them by downloading SSIS PowerPack from our website here.

References

• About SSIS PowerPack
• About Secure FTP Task
• SISS Script Task
• About SSIS Variables

The post Download latest File from FTP using SSIS (SFTP / FTPS) appeared first on ZappySys Blog.

In this article, we will show SSIS SFTP task examples. This task supports multiple protocols (Classic FTP, SFTP, FTPS).

Uploading files using SFTP or downloading files are common requirements in SSIS. In order to work with FTP, Microsoft SSIS includes the FTP Task that allows to upload files to FTP, download files, creating local and remote directories an more. However, there is no native task to work with SFTP. Also Native FTP task doesn’t support many advanced options such as FTP over SSL (i.e. FTPS).  In this article, we will show how to use the SSIS SFTP task included in ZappySys SSIS PowerPack.

ZappySys SFTP Task will let you send files to SFTP/FTP/FTPS server, receive files from SFTP/FTP/FTPS server. With ZappySys SFTP Task, you can not only create/delete files and folders on ftp server but also get file properties (creation date, last modified, last read dates, size and more), verify if the files exist, rename and move files and more.

SSIS SFTP Task Video Tutorial

The following video shows a nice tutorial about a ZappySys SSIS SFTP Task with examples:

Requirements

1. First, you will need to have SSIS installed
2. Also, make sure to have SSDT
3. Finally, do not forget to install ZappySys SSIS PowerPack

Whats is FTP, FTPS, SFTP, FTP/SSL?

Many people still have confusion over various protocols around FTP terminology. Most common terms you hear are as below.

• FTP
• Classic FTP
• Secure FTP
• SFTP
• FTPS
• FTP/SSL
• FTP over SSL
• FTP Over SSH

You must be overwhelmed by now 🙂 … So are they all same or different? See below for quick clarification in nutshell.

Mainly there are 2 protocols for FTP and yes they are very different but still used to do Files Transfer to / from remote server.

1. Classic FTP – sometimes referred as just FTP (Typically server runs on Port 21)
2. SFTP – Sometimes referred as Secure FTP or FTP over SSH (Typically server runs on Port 22)

So in short SFTP is not same as Classic FTP protocol. SFTP is more secure because traffic is always encrypted. On the other hand Classic FTP is not encrypted by default but you can use FTP over SSL to request encrypted traffic in classic FTP. FTP over SSL also referred as FTPS or FTP/SSL.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS SFTP task that will allow to work with SFTP, FTP or even FTPS. Let´s start with an example. In this SSIS SFTP task example, we will show how to download files from SFTP to local folders.

1. First, in a new SSIS project, drag and drop the ZS Secure FTP Task. This task will let you send and receive files in FTP:
   

   Using SSIS SFTP Task in the control Flow Designer

2. Secondly, double click the ZS Secure FTP Task and select the Download FTP server file(s) to local directory. This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable. In path, you can use “/ ” to specify the root folder /source would be a folder named source in the root. If there were another folder inside source named child1, the path would be /source/child1.  There is a nice option that you can check to include folders and child folders. If you need to include them, check the option Recursive scan. In target, the tab can be written directly (direct) and using SSIS variables. The variables will be explained later. In path you specify the path of the target. You can throw an exception if there is a need to overwrite files and folders. Finally, in Connection, press New button to create a new connection:
   

   SSIS Secure FTP Task – Download files from FTP / SFTP Server

3. The following options are available in SSIS Secure FTP Task:
   

   Option to upload, download files in SSIS

4. You can also overwrite files and folder, skip all, overwrite if source is older, overwrite if the file size is different, rename files and more:
   

   Overwrite files in SSIS

Configuring the SSIS SFTP Connection Manager

In this new section, we will show how to configure the Connection Manager. We configure before the Secure FTP task, and we pressed the New button to create a new connection.

1. In the connection manager for SFTP, we will work with the SFTP Connection Manager. We will choose if the protocol is FTP, FTPS or SFTP. Also, we need to specify the host. In this example, we are using a localhost, but in other cases, you will need to specify the IP. By default, the SFTP port is 22. If it is a different port in your case, specify the port, otherwise leave blank. The FTP default port is 21 and FTPS 990.  If you have multiple files, make sure to check the option Retain the same connection.
   

   SSIS SFTP Connection manager (Support for Classic FTP, SFTP, FTPS (i.e. FTP / SSL))

2. There are different types of Logon Types. A username and a password is the most common way to login, but you can also use other types like the Public/private key authentication, X509 certificate authentication, KerversosV5 user/password/domain, NTLM single sign-on (GSSAPI) and NTLM user/password/domain (GSSAPI).
   

   SFTP Connection Manager – Logon types (i.e. Username/password, Public / Private Key, NTLM )

3. If everything is OK, it will copy all the files from the SFTP in the source folder in the C:\destination folder.

Working with advanced filters

In order to understand filters, we will create an example for you. This time we will create SSIS SFTP task example to show how to exclude some files from the downloading process.

1. In addition, it is also possible to add advanced filters to the SSIS SFTP task to exclude or include files and folders. The following example shows how to use the Advanced filters. In the Secure FTP Task, go to the Advanced Filter page and in Exclude RegX we will write \.exe$ to exclude the file with exe extension to be downloaded. You can use RegX (regular expressions) in the advanced filter:
   

   Advanced filters FTP

Working with wildcards in SSIS SFTP task (Multiple files)

In order to understand wildcards, the new example will show how to work with Wildcards will help you to work with multiple files or folder with common characteristics.

1. The following example shows how to copy all the files will all the extensions in the source folder. We use the *.*, which means that any file name with any extension will be downloaded:
   

   SSIS SFTP connection manager

2. If we want to download all the files with extension txt, use the following Path:
   

   /source/*.txt;

3. Another example would be to download all the files with the following names: file1.txt, file2.txt and file3.txt.  We could use the following Path:
   

   /source/file?.txt

Working with SSIS expressions and variables in SFTP / FTPS to delete FTP files

In order to understand expressions, the following example will show how to work with SSIS expressions combined with variables in the SSIS SFTP task. In this example we will delete files in SFTP using variables and expressions.

1.  First, we will first create a variable in SSIS. In the SSIS Project, go to the menu and select SSIS and variables. Create a variable with the Data type string and specify a Value. In this example the file name is MyFile and the Value file1.
   

   Create an SSIS variable

2. As a second step, we will write in path the value \destination\{{User.MyFile}}.txt. This will concatenate the folder destination with the variable MyFile. The task will delete the file1.txt from the SFTP folder.
   

   Create SSIS expressions

3. The next example will show how to create a SFTP file using expressions and variables in SSIS:
   

   Create file ftp

4. Content to write is the content of the file. In Target, go to path and write the following: For more information on placeholders check this post and also check this post.
   

   /destination/{{User::FileName,MM-dd-yyyy}}

5. We are using a variable in SSIS named FileName of type DateTime. We are concatenating the SFTP folder destination with the variable. As you can see, the expressions with ZappySys are very simple.

Example working with variables in the SSIS FTP Task to upload files

In order to understand variables, this example about SSIS TASKS FOR SFTP will show how to work with variables. We will specify the path in a variable.

1. In this example, we will create a variable with a path. The Data type will be string and the Value will contain the path:
   

   SSIS variable values

2.  Secondly, we will go to Path AccessMode, we will select Variable and select the variable in Path variable:
   

   ssis upload files SFTP

Working with variables to create a file in FTP

In order to understand the use of variables, will show how to create a file in SFTP. In this new SSIS SFTP task example, we will create a new file using variables.

1. First, we will create an SSIS variable named Content that will store the file content. Create a variable of type String and write some text in it.
2. Secondly, in AccessMode, select Content Variable and select Create new FTP file.  In target path, you need to specify the file name that will be created. Next, select the User::Content variable to add content to the file:
   

   Example for SSIS SFTP task

Example to verify if file exists in SSIS SFTP task

Finally, we will verify if a file in SFTP exists or not using the SSIS SFTP task. The new SSIS SFTP task example will show how to work with variables and another useful ZappySys task used to get values of variables.

1. In order to start, we will use the SSIS variables, create a variable of type string named FileExists. In Storage Action, select Get FTP file exist status. In path, select the path of the file to verify if it exists or not.
   

   Verify if file exists in SFTP

2. Secondly, we will use the ZS Logging Task to check the value of the variable File Exists. This task will show the value of the variable in the log. Drag and drop the task and join it with the ZS Secure FTP Task:
   

   The SSIS Log task

3. In order to get the value, we will insert it in the Logging Task. To do that, insert the Variable and select the FileExists variable:
   

   Add the variable in the task logging

4. Finally, run the package and verify the Output (you can use View and Output to see the output) and check the value of the Logging Task. If the file exists, the value will be true. If it does not exist, the value will be false.
   

   Verify the variable in Output

How to loop through SFTP / FTP files in SSIS

You can also loop through files to perform custom action file by by. SFTP Task support Action to get File List as ADO.net Recordset

To get File list from FTP Server perform the following steps.

1. Download and Install SSIS PowerPack from here
2. Open Visual Studio and Create New SSIS Project.
3. In the SSIS control flow designer drag & drop ZS Secure FTP Task
4. Double click to configure SSIS SFTP Task.
5. From Storage Action select Get FTP files as ADO.net DataTable
   
   

   SSIS SFTP Task – Loop through Files / Folders stored on FTP Server

    

    

6. Create new FTP Connection by clicking New next to connection dropdown.
7. Select Source Path (e.g  \myroot\folder\*.txt  )
8. Select or create new SSIS Variable (e.g. varFiles ) which will hold file list result (Must be Object datatype variable).
9. Click OK to close.
10. Create 3 more SSIS variables to hold File Path (String), Name (String) and Size (Int32). We will use in the next step.
11. Now drag SSIS ForEach Loop Container Task from SSIS Toolbox
12. Edit Loop Task and Select Foreach ADO Enumerator from the Enumerator dropdown
13. Select variable (e.g. varFiles)
    
    

    Loop through files in SSIS using ForEach Loop Task (use ADO.net Recordset variable)

14. Click on Variable Mappings Tab and configure like below.
    

    Variable Mappings for ForEach Loop Task – Loop through files in a folder using SSIS

15. Drag ZS Logging Task and place inside Loop and Display variables
16. Thats

For more information to loop through files on local machine using SSIS check this article.

Algorithms / Ciphers supported by Secure FTP Task

Secure FTP Task comes with many option to tweak many aspects about security so you can turn on and off algorithms used by components.

Click here to learn more about supported Ciphers / Macs and Encryption Algorithms used by SSIS SFTP Task.

Conclusion

To conclude, we can say that working with SFTP is now very simple. In this article, we looked at SSIS SFTP task examples on how to upload, download, delete files and folders. We used variables, wildcards and expressions. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here. We saw different SSIS SFTP task examples to understand how to use the task in different scenarios.

Thanks for your time.

References

• About SSIS PowerPack
• Integration Services (SSIS) Expressions

Keywords: SSIS SFTP task example,upload SFTP files, download SFTP files.

The post SSIS SFTP task examples to upload, download, move and delete files / folders appeared first on ZappySys Blog.