ZappySys provide high-performance drag and drop connectors for Amazon Redshift Integration. In previous post we saw how to read / write Amazon Redshift data in SSIS. Now in this post, you will see how to Execute SQL query against Amazon Redshift in SSIS using ZS Amazon Redshift ExecuteSQL Task. SSIS Redshift Execute SQL Task can be used to send any SQL queries to Redshift Cluster without needing ODBC driver. You can Issue various SQL commands such as DROP, COPY, UNLOAD, CREATE, etc very easily. Load query at runtime from variable, file or Direct value. You can also issue SQL query to fetch resultset to store in variable and then loop records using this technique.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, You have obtained Amazon Redshift Credential.
4. Finally, do not forget to install ZappySys SSIS PowerPack.

Component Mention in this article

Amazon Redshift ExecuteSQL Task

What is Amazon Redshift

Before we talk Execute SQL query against Amazon Redshift in SSIS lets talk what is Amazon Redshift (or sometimes referred to as AWS Redshift). Amazon Redshift is a Cloud-based Data warehouse service. This type of system also referred to as MPP (Massively Parallel Processing). Amazon Redshift uses a highly modified version of PostGrey SQL Engine behind the scene. Amazon Redshift provides the advantage of Scale as you go, at a very low cost compared to onsite dedicated hardware/software approach. In this article, we will try to learn how to Execute SQL query against Amazon Redshift in SSIS.

Getting Started

In order to start, we will show several examples. ZappySys includes an Amazon Redshift ExecuteSQL Task, that will help you how to Create and Drop table by Execute SQL query against Amazon Redshift in SSIS. You can connect to your Amazon Redshift Account by entering credentials.

Execute SQL query against Amazon Redshift in SSIS

Let’s start with an example, In this tutorial, we will learn how to CREATE and DROP table using SQL query against Amazon Redshift ExecuteSQL server using ZS Amazon Redshift ExecuteSQL Task.

1. In order to connect to the Amazon Redshift Database from SSIS, you will need credentials. Ask your SysAdmin or responsible person to provide that you.
2. Once you finished the first step, Open Visual Studio and Create New SSIS Package Project.
3. In your SSIS toolbox, you should see many new Icons starting with “ZS” (i.e. ZappySys). If you don’t see them then check this.
4. Here, In Visual Studio, drag and drop the Amazon Redshift ExecuteSQL Task in the design surface.
   

   SSIS ZS Amazon Redshift ExecuteSQL Task – Drag and Drop

5. Now we need Amazon Redshift Connection. Click here to Create Amazon Redshift Connection.
6. Double click on Amazon Redshift ExecuteSQL Task to configure it.
7. Select connection we have created before, set Sql Source Type to DirectValue, Result Type to FullResultSet and select object Datatype Variable and Use following SQL Query for Amazon Redshift.
   

   DROP TABLE IF EXISTS CustomerData;
   CREATE TABLE CustomerData(
   	Id bigint NULL,
   	CustomerID VARCHAR(5) NOT NULL,
   	CustomerName VARCHAR(50) NOT NULL,
   	ContactName VARCHAR(40) NOT NULL,
   	CustomerCity VARCHAR(50) NOT NULL
   )

   

   SSIS Amazon Redshift ExecuteSQL Task – Configure

8. Click on OK button to save configure setting UI.
9. That’s all, right-click on the task and run or Execute. This should create a new table in Redshift.
   

   Run or Execute SSIS ZS ExecuteSQL Task Package

Conclusion

After all, In this blog, we learned how to Execute SQL query against Amazon Redshift in SSIS using ZS Amazon Redshift ExecuteSQL Task. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• Amazon Redshift Document
• Help File: SSIS ZS Amazon Redshift ExecuteSQL Task

The post Execute SQL query against Amazon Redshift in SSIS appeared first on ZappySys Blog.

ZappySys provides high-performance drag and drop connectors for PostgreSQL Integration. In this post you will see how to Execute SQL query against PostgreSQL in SSIS, in various SQL commands such as SELECT, INSERT, DROP, COPY, CREATE, etc using PostgreSQL Execute SQL Task can be used to execute DDL and DML statements for PostgreSQL without needing any extra driver. PostgreSQL is a powerful, open-source relational database with a strong reputation for reliability, feature robustness, and performance.

In nutshell, this post will focus on how to Execute SQL query against PostgreSQL in just a few clicks.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, You have obtained PostgreSQL Credential.
4. Finally, do not forget to install ZappySys SSIS PowerPack

Component Mentioned in this article

PostgreSQL Execute SQL Task

What is PostgreSQL?

PostgreSQL is a powerful, open source object-relational database system that uses and extends the SQL language combined with many features that safely store and scale the most complicated data workloads. The origins of PostgreSQL date back to 1986 as part of the POSTGRES project at the University of California at Berkeley and has more than 30 years of active development on the core platform.

PostgreSQL has earned a strong reputation for its proven architecture, reliability, data integrity, robust feature set, extensibility, and the dedication of the open source community behind the software to consistently deliver performant and innovative solutions. PostgreSQL runs on all major operating systems, has been ACID-compliant since 2001, and has powerful add-ons such as the popular PostGIS geospatial database extender. It is no surprise that PostgreSQL has become the open source relational database of choice for many people and organisations.

Getting started with using PostgreSQL has never been easier – pick a project you want to build, and let PostgreSQL safely and robustly store your data.

Download and Install PostgreSQL

Please go to below download link and download the Windows version Binary package and install it and follow the steps from the installer.

https://www.postgresql.org/download/

Download PostgreSQL

Create a PostgreSQL Database

1. To do this, go to Start Menu and search for pgAdmin and we will first open the pgAdmin Panel.
   

   Open pgAdmin from Start Menu

2. It will open the pgAdmin Panel in the system default browser. Select Server and Right-click on Databases to create a database.
   

   PostgreSQL Create Database

3. Now, Enter the Database name and click on Save to create it.
   

   Create a PostgreSQL Database

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS PostgreSQL ExecuteSQL Task that will help you how to Create and Drop table by Execute SQL query against PostgreSQL in SSIS. ZS PostgreSQL ExecuteSQL Task support execute queries against PostgreSQL without ODBC driver, load query at runtime from variable, file or Direct value, 100% Managed code means its 32bit and 64bit compatible, you can issue COPY, UNLOAD commands very easily, Support for SQL Server 2017, 2016, 2014, 2012, 2008 (32/64 bit) and Click here to see articles related to PostgreSQL.

You can connect to your PostgreSQL database by entering your PostgreSQL credentials.

Execute SQL query against PostgreSQL in SSIS

Let’s start with an example. In this tutorial, we will learn how to Create and Drop table using SQL query against PostgreSQL server using ZS PostgreSQL ExecuteSQL Task.

1. In order to connect to the PostgreSQL Database from SSIS, you will need Credentials. Ask your SysAdmin or responsible person to provide that to you.
2. Once you finished the above steps, Open Visual Studio and Create New SSIS Package Project.
3. In Visual Studio, drag and drop the ZS PostgreSQL ExecuteSql Task in the design surface.
   

   SSIS ZS PostgreSQL ExecuteSQL Task – Drag and Drop

4. We need a PostgreSQL Database connection. Click here to create a connection.
5. Now, double click on ZS PostgreSQL ExecuteSql Task to configure it.
6. In the PostgreSQL ExecuteSql Task, add the following SQL code to DROP a table if it exists and CREATE a table named pg_equipment in the SQL Statement section.
   You can replace the table name from the following code.
   

   DROP TABLE IF EXISTS playground_equip;
   CREATE TABLE pg_equipment (
   	equip_id serial PRIMARY KEY,
   	type varchar (50) NOT NULL,
   	color varchar (25) NOT NULL,
   	location varchar(25) check (location in ('north', 'south', 'west', 'east', 'northeast', 'southeast', 'southwest', 'northwest')),
   	install_date date
   );

   

   SSIS ZS PostgreSQL ExecuteSQL Task – Configure

7. In the above step, you can Select Variable as Sql Source Type.
8. Click OK to save ZS PostgreSQL ExecuteSql Task UI settings.
9. Finally, Right Click on Task and Select Execute Task to run it.

Conclusion

After all, In this blog, we learned Execute SQL query against PostgreSQL in SSIS using ZS PostgreSQL ExecuteSQL Task. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information:

• About PostgreSQL
• PostgreSQL Documentation
• PostgreSQL Query
• Help File: PostgreSQL ExecuteSQL Task

The post Execute SQL query against PostgreSQL in SSIS appeared first on ZappySys Blog.

In this article, we will see how to read Walmart API data in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call Walmart APIs can be useful for any developer regardless of which programming language or toolset you use.

Are you a Walmart.com Supplier or Marketplace Seller? Then you may need to click here to visit the site that is focused on these profiles.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Walmart API. Click here to create API.

Components Mentioned in this article

	ZS Logging Task
	JSON Source(REST API or File)

What is Walmart?

Walmart is the world’s largest retailer, and the Walmart Open API provides access to our extensive product catalog, thus enabling digital distribution partners to earn substantial affiliate revenues from customer referrals.

Read Walmart API Data using SSIS JSON Source

Let’s start with an example. We use SSIS JSON Source component to make the call to Walmart API, we will read Walmart data and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

Get Dynamic Token (Configure ZS-HTTP Connection)

To get API response data using SOAP / REST API call, you need to get an API Token from your login request URL. Follow the steps mentioned below to get API Token:

1. Now, Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
   

   Create a New HTTP Connection.

2. Select ZS HTTP Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
   

   Select ZS-HTTP Connection from SSIS Connection Manager.

3. In the SSIS ZS HTTP Connection Manager, Go to General Tab and Configure HTTP Connection Manager. Enter API URL, Set Credentials Type to Dynamic Token – Two-Step Auth, and configure some basic settings for the same. We used Login URL as just example but you can enter any valid API URL endpoint here. Actual Login URL goes on Dynamic Token Tab (see next step).
   

   https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20

   WM_SEC.ACCESS_TOKEN

   

   ZS HTTP Connection – Configure General Tab

4. Now, In the Dynamic Token Tab do configure like below image. Please click here to refer about Walmart API Token.
   

   https://marketplace.walmartapis.com/v3/token

   grant_type=client_credentials

   Accept: application/json
   Cache-Control: no-cache
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>
   Authorization: Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>

   

   ZS HTTP Connection – Dynamic Token Tab

5. On the Response Settings tab configure as below
   

   Walmart API – Extract Token – Response Settings

   1. Extract Mode = Extract from Body
   2. Expression Type = Json
   3. Response Content Filter =

      $.access_token

6. Click on OK button to save HTTP Connection configure settings.

That’s it we have successfully configured Connection for Walmart API in SSIS. In the next section, we will see how to use this connection and read various data from Walmart API.

Read Walmart API Data in SSIS

Once we have done creating HTTP Connection Manager we can move forward to read Walmart API data inside Data Flow. So lets Configure SSIS JSON / REST API Source.

1. Now, Drag and drop Data Flow Task from SSIS Toolbox in the currently open project and double click it to edit.
   
   

   Drag and Drop Data Flow Task.

2. Double click on the Data Flow task to see the Data Flow designer surface.
3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   

   Drag and Drop JSON Source(REST API or File).

4. Double click JSON Source and enter the following URL as below.
   

   https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20

5. Check Use Credentials and select existing HTTP connection we have created already it.
6. Enter Following HTTP Headers. If you prefer Raw edit .. Just click on Raw Edit (#6 in screenshot) and paste below headers. Make sure to replace YourClientIdGoesHere and YourSecretGoesHere
   

   Content-Type: application/xml
   Accept: application/xml
   Authorization: Basic <<YourClientIdGoesHere:YourSecretGoesHere,FUN_BASE64ENC>>
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>

    
7. Select Array Filter (ItemResponse node) or type $.ItemResponse[*]  as below.
   

   ZS JSON Source – Configure

8. That’s it, you are ready to load Wallmart API Data into SQL Server and you can use more ZappySys Destination components.
9. For pagination call please click here for reference to this link.

Load Walmart API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Walmart API integration using ODBC Driver (Power BI, Excel, MS Access)

So far we have seen how to access Walmart API using SSIS. However there will be a time when you like to access API data inside other Apps using ODBC Driver. ZappySys Provides JSON/XML/CSV drivers for virtually any API including Walmart API. Here is how to access Walmart API using XML ODBC Driver.

1. Type “ODBC” in Start menu. Open ODBC Data sources (64 bit)
   
2. Click Add on User DSN tab (if you need to use DSN under different Account …i.e. Service Account then click on System DSN Tab)
3. Select ZappySys XML Driver
4. When New DSN UI shows up, Change DSN name to anything you like (e.g. WalmartAPI)
5. Click on Load ConnectionString button and enter below connection. Make sure to replace few things
   First ######YourClientId####### with your own clientid
   Second ######YourSecret####### with your own secret
   Third ######Base64OfClientIdAndSecret####### with base64 encoded string of

   clientId:Secret

       (both separated with colon). You can use any site like this one to generate base64 string of your clientid and secret (seperated by “:”)
   

   DRIVER={ZappySys XML Driver};
   ElementsToTreatAsArray='ns2:ItemResponse';
   DataPath='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20';
   DataConnectionType=HTTP;
   AuthScheme='{none}';
   Url='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20';
   TokenUrl='https://marketplace.walmartapis.com/v3/token';
   TokenRequestData='grant_type=client_credentials';
   TokenRequestMethod='POST';
   TokenResponseContentFilter='$.access_token';
   TokenRequestHeaders='Accept:application/json||Cache-Control:no-cache||WM_SVC.NAME:Walmart Marketplace||WM_QOS.CORRELATION_ID:TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>||Authorization:Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>';
   TokenAuthHeader='WM_SEC.ACCESS_TOKEN';
   TokenResponseContentType=Json;
   UserName='######YourClientId#######';
   CredentialType=TokenDynamic;
   Password='######YourSecret#######';
   Filter='$.ns2:ItemResponses.ns2:ItemResponse[*]';
   RequestMethod='GET';
   RequestHeaders='Content-Type: application/xml || Accept: application/xml || Authorization: Basic ######Base64OfClientIdAndSecret####### || WM_SVC.NAME: Walmart Marketplace || WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>'

   

   ZappySys Driver – Load Connection String

    

6. Now go to preview tab and enter below query and click Run below SQL.

   select * from $

   –OR–
   NOTE: Change YourClientId and YourSecret in below sql before running

   SELECT * FROM $
   WITH(
   Src='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20'
   ,Header='Content-Type: application/xml
   Accept: application/xml
   Authorization: Basic <<YourClientid:YourSecret,FUN_BASE64ENC>>
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>'
   )

    

Walmart API Integration with Other BI Tools (Power BI, Excel, SSRS, MS Access…)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

How to encrypt Authorization Token as Base64Encode

1. If you want to directly pass Authorization header Token as Base64Encode(clientId:clientSecret) encrypted in the get token call, you can encrypted it using ZS Logging task and used it. Please refer to this link for the same.
2. In visual studio just Drag and Drop ZS Logging Task in the design panel.
   

   Drag and Drop ZS Logging Task

3. Double click on ZS Logging Task to configure it.
4. In the ZS Logging Task configure like the following image.
   

   Logging Task : Base64Enocde

5. That’s all, Save Base64Encoded token for use in HTTP Connection.

Conclusion

After all, we saw you how to extract information from REST API such as Walmart REST API using HTTP Connection and load into SQL Server. We also learned techniques like How to get Dynamic Token using HTTP Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File), ZS Logging Task
• Walmart API: HTTP Connection

The post How to read Walmart API data in SSIS / ODBC appeared first on ZappySys Blog.

How to delay SSIS tasks using SSIS Timer Task

1. In order to solve this problem, you need to download this free tool and use it. ZS Timer task does not require too many explanations to use it. It is simple. You just specify the time to wait.
2. Secondly, we will have a package like this one:
   

   Package with SSIS timer task

3. Also, we need to wait 1 minute in order to have everything ready for the
   execution of the next package:
   

   SSIS delay task 1 minute

4. Finally, it is possible to wait days, hours, minutes and seconds.

Delay SSIS tasks to a specific date

1. It is also possible to run the package using the ZS Timer task and execute the timer task to specific data.
2. In order to do this we will use the following configuration:
   

   SSIS delay task until a date

3. In this example, the configuration provided will run on June 11, 2019.
4. Also, remember to uncheck the option Ignore date part. This option is checked by default and disables the date.

Delay SSIS tasks to a specific time

1. The next example will be used to run the task in a customized time.
2. First of all, we will configure the time to 13:02:01. This means that the task will run at that time:
   

   Run SSIS task at specific time

3. Also, note that by default the ignore date part is checked.

Delay SSIS tasks from a variable

1. We will show how to get the date from a variable. We will store the time from a variable and delay the task until the specified date of the variable.
2. First of all, we will create a variable of type dateTime:
   

   SSIS datetime variable

3. If the ignore date part is checked, it will run today at the time specified.
   

   Timer ssis task with variable

4. Otherwise, it will run the date and time specified.

Conclusion

In this article, we show how to delay SSIS tasks a custom number of days or at a specific date or time. We also learned how to set the date and time manually or using a variable.

References

• ZS SSIS Timer task.

The post How to delay SSIS execution using Timer Task appeared first on ZappySys Blog.

In this article, we will show SSIS SFTP task examples. This task supports multiple protocols (Classic FTP, SFTP, FTPS).

Uploading files using SFTP or downloading files are common requirements in SSIS. In order to work with FTP, Microsoft SSIS includes the FTP Task that allows to upload files to FTP, download files, creating local and remote directories an more. However, there is no native task to work with SFTP. Also Native FTP task doesn’t support many advanced options such as FTP over SSL (i.e. FTPS).  In this article, we will show how to use the SSIS SFTP task included in ZappySys SSIS PowerPack.

ZappySys SFTP Task will let you send files to SFTP/FTP/FTPS server, receive files from SFTP/FTP/FTPS server. With ZappySys SFTP Task, you can not only create/delete files and folders on ftp server but also get file properties (creation date, last modified, last read dates, size and more), verify if the files exist, rename and move files and more.

SSIS SFTP Task Video Tutorial

The following video shows a nice tutorial about a ZappySys SSIS SFTP Task with examples:

Requirements

1. First, you will need to have SSIS installed
2. Also, make sure to have SSDT
3. Finally, do not forget to install ZappySys SSIS PowerPack

Whats is FTP, FTPS, SFTP, FTP/SSL?

Many people still have confusion over various protocols around FTP terminology. Most common terms you hear are as below.

• FTP
• Classic FTP
• Secure FTP
• SFTP
• FTPS
• FTP/SSL
• FTP over SSL
• FTP Over SSH

You must be overwhelmed by now 🙂 … So are they all same or different? See below for quick clarification in nutshell.

Mainly there are 2 protocols for FTP and yes they are very different but still used to do Files Transfer to / from remote server.

1. Classic FTP – sometimes referred as just FTP (Typically server runs on Port 21)
2. SFTP – Sometimes referred as Secure FTP or FTP over SSH (Typically server runs on Port 22)

So in short SFTP is not same as Classic FTP protocol. SFTP is more secure because traffic is always encrypted. On the other hand Classic FTP is not encrypted by default but you can use FTP over SSL to request encrypted traffic in classic FTP. FTP over SSL also referred as FTPS or FTP/SSL.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS SFTP task that will allow to work with SFTP, FTP or even FTPS. Let´s start with an example. In this SSIS SFTP task example, we will show how to download files from SFTP to local folders.

1. First, in a new SSIS project, drag and drop the ZS Secure FTP Task. This task will let you send and receive files in FTP:
   

   Using SSIS SFTP Task in the control Flow Designer

2. Secondly, double click the ZS Secure FTP Task and select the Download FTP server file(s) to local directory. This option will download files. In Path AccessMode we can use Direct to write the path directly or to use an SSIS variable. In path, you can use “/ ” to specify the root folder /source would be a folder named source in the root. If there were another folder inside source named child1, the path would be /source/child1.  There is a nice option that you can check to include folders and child folders. If you need to include them, check the option Recursive scan. In target, the tab can be written directly (direct) and using SSIS variables. The variables will be explained later. In path you specify the path of the target. You can throw an exception if there is a need to overwrite files and folders. Finally, in Connection, press New button to create a new connection:
   

   SSIS Secure FTP Task – Download files from FTP / SFTP Server

3. The following options are available in SSIS Secure FTP Task:
   

   Option to upload, download files in SSIS

4. You can also overwrite files and folder, skip all, overwrite if source is older, overwrite if the file size is different, rename files and more:
   

   Overwrite files in SSIS

Configuring the SSIS SFTP Connection Manager

In this new section, we will show how to configure the Connection Manager. We configure before the Secure FTP task, and we pressed the New button to create a new connection.

1. In the connection manager for SFTP, we will work with the SFTP Connection Manager. We will choose if the protocol is FTP, FTPS or SFTP. Also, we need to specify the host. In this example, we are using a localhost, but in other cases, you will need to specify the IP. By default, the SFTP port is 22. If it is a different port in your case, specify the port, otherwise leave blank. The FTP default port is 21 and FTPS 990.  If you have multiple files, make sure to check the option Retain the same connection.
   

   SSIS SFTP Connection manager (Support for Classic FTP, SFTP, FTPS (i.e. FTP / SSL))

2. There are different types of Logon Types. A username and a password is the most common way to login, but you can also use other types like the Public/private key authentication, X509 certificate authentication, KerversosV5 user/password/domain, NTLM single sign-on (GSSAPI) and NTLM user/password/domain (GSSAPI).
   

   SFTP Connection Manager – Logon types (i.e. Username/password, Public / Private Key, NTLM )

3. If everything is OK, it will copy all the files from the SFTP in the source folder in the C:\destination folder.

Working with advanced filters

In order to understand filters, we will create an example for you. This time we will create SSIS SFTP task example to show how to exclude some files from the downloading process.

1. In addition, it is also possible to add advanced filters to the SSIS SFTP task to exclude or include files and folders. The following example shows how to use the Advanced filters. In the Secure FTP Task, go to the Advanced Filter page and in Exclude RegX we will write \.exe$ to exclude the file with exe extension to be downloaded. You can use RegX (regular expressions) in the advanced filter:
   

   Advanced filters FTP

Working with wildcards in SSIS SFTP task (Multiple files)

In order to understand wildcards, the new example will show how to work with Wildcards will help you to work with multiple files or folder with common characteristics.

1. The following example shows how to copy all the files will all the extensions in the source folder. We use the *.*, which means that any file name with any extension will be downloaded:
   

   SSIS SFTP connection manager

2. If we want to download all the files with extension txt, use the following Path:
   

   /source/*.txt;

3. Another example would be to download all the files with the following names: file1.txt, file2.txt and file3.txt.  We could use the following Path:
   

   /source/file?.txt

Working with SSIS expressions and variables in SFTP / FTPS to delete FTP files

In order to understand expressions, the following example will show how to work with SSIS expressions combined with variables in the SSIS SFTP task. In this example we will delete files in SFTP using variables and expressions.

1.  First, we will first create a variable in SSIS. In the SSIS Project, go to the menu and select SSIS and variables. Create a variable with the Data type string and specify a Value. In this example the file name is MyFile and the Value file1.
   

   Create an SSIS variable

2. As a second step, we will write in path the value \destination\{{User.MyFile}}.txt. This will concatenate the folder destination with the variable MyFile. The task will delete the file1.txt from the SFTP folder.
   

   Create SSIS expressions

3. The next example will show how to create a SFTP file using expressions and variables in SSIS:
   

   Create file ftp

4. Content to write is the content of the file. In Target, go to path and write the following: For more information on placeholders check this post and also check this post.
   

   /destination/{{User::FileName,MM-dd-yyyy}}

5. We are using a variable in SSIS named FileName of type DateTime. We are concatenating the SFTP folder destination with the variable. As you can see, the expressions with ZappySys are very simple.

Example working with variables in the SSIS FTP Task to upload files

In order to understand variables, this example about SSIS TASKS FOR SFTP will show how to work with variables. We will specify the path in a variable.

1. In this example, we will create a variable with a path. The Data type will be string and the Value will contain the path:
   

   SSIS variable values

2.  Secondly, we will go to Path AccessMode, we will select Variable and select the variable in Path variable:
   

   ssis upload files SFTP

Working with variables to create a file in FTP

In order to understand the use of variables, will show how to create a file in SFTP. In this new SSIS SFTP task example, we will create a new file using variables.

1. First, we will create an SSIS variable named Content that will store the file content. Create a variable of type String and write some text in it.
2. Secondly, in AccessMode, select Content Variable and select Create new FTP file.  In target path, you need to specify the file name that will be created. Next, select the User::Content variable to add content to the file:
   

   Example for SSIS SFTP task

Example to verify if file exists in SSIS SFTP task

Finally, we will verify if a file in SFTP exists or not using the SSIS SFTP task. The new SSIS SFTP task example will show how to work with variables and another useful ZappySys task used to get values of variables.

1. In order to start, we will use the SSIS variables, create a variable of type string named FileExists. In Storage Action, select Get FTP file exist status. In path, select the path of the file to verify if it exists or not.
   

   Verify if file exists in SFTP

2. Secondly, we will use the ZS Logging Task to check the value of the variable File Exists. This task will show the value of the variable in the log. Drag and drop the task and join it with the ZS Secure FTP Task:
   

   The SSIS Log task

3. In order to get the value, we will insert it in the Logging Task. To do that, insert the Variable and select the FileExists variable:
   

   Add the variable in the task logging

4. Finally, run the package and verify the Output (you can use View and Output to see the output) and check the value of the Logging Task. If the file exists, the value will be true. If it does not exist, the value will be false.
   

   Verify the variable in Output

How to loop through SFTP / FTP files in SSIS

You can also loop through files to perform custom action file by by. SFTP Task support Action to get File List as ADO.net Recordset

To get File list from FTP Server perform the following steps.

1. Download and Install SSIS PowerPack from here
2. Open Visual Studio and Create New SSIS Project.
3. In the SSIS control flow designer drag & drop ZS Secure FTP Task
4. Double click to configure SSIS SFTP Task.
5. From Storage Action select Get FTP files as ADO.net DataTable
   
   

   SSIS SFTP Task – Loop through Files / Folders stored on FTP Server

    

    

6. Create new FTP Connection by clicking New next to connection dropdown.
7. Select Source Path (e.g  \myroot\folder\*.txt  )
8. Select or create new SSIS Variable (e.g. varFiles ) which will hold file list result (Must be Object datatype variable).
9. Click OK to close.
10. Create 3 more SSIS variables to hold File Path (String), Name (String) and Size (Int32). We will use in the next step.
11. Now drag SSIS ForEach Loop Container Task from SSIS Toolbox
12. Edit Loop Task and Select Foreach ADO Enumerator from the Enumerator dropdown
13. Select variable (e.g. varFiles)
    
    

    Loop through files in SSIS using ForEach Loop Task (use ADO.net Recordset variable)

14. Click on Variable Mappings Tab and configure like below.
    

    Variable Mappings for ForEach Loop Task – Loop through files in a folder using SSIS

15. Drag ZS Logging Task and place inside Loop and Display variables
16. Thats

For more information to loop through files on local machine using SSIS check this article.

Algorithms / Ciphers supported by Secure FTP Task

Secure FTP Task comes with many option to tweak many aspects about security so you can turn on and off algorithms used by components.

Click here to learn more about supported Ciphers / Macs and Encryption Algorithms used by SSIS SFTP Task.

Conclusion

To conclude, we can say that working with SFTP is now very simple. In this article, we looked at SSIS SFTP task examples on how to upload, download, delete files and folders. We used variables, wildcards and expressions. If you liked the tasks you can start using them by downloading SSIS PowerPack from our web site here. We saw different SSIS SFTP task examples to understand how to use the task in different scenarios.

Thanks for your time.

References

• About SSIS PowerPack
• Integration Services (SSIS) Expressions

Keywords: SSIS SFTP task example,upload SFTP files, download SFTP files.

The post SSIS SFTP task examples to upload, download, move and delete files / folders appeared first on ZappySys Blog.