If you need to upload files from SSIS to SharePoint Online, the ZappySys SharePoint Online Connector for SSIS gives you a direct way to configure the connection, authenticate, and send files without building the API calls from scratch.

This guide shows how to upload one file or multiple files to SharePoint Online using SSIS, API Source, and supporting SSIS components.

Upload files to SharePoint Online with SSIS

In this section, you will configure the SharePoint Online Connector in API Source and use it to upload files to SharePoint Online.

1. Begin by opening Visual Studio and creating a new project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom
   

   SSIS Toolbox – ZappySys Tasks

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Drag and drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS Toolbox, drag and drop API Source (Predefined Templates) onto the Data Flow Designer surface, then double-click it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the SharePoint Online Connector

1. Use a preinstalled SharePoint Online Connector from the Popular Connector List or press the Search Online radio button to download SharePoint Online Connector. Once downloaded simply use it in the configuration:
   

   SSIS ZS API Connection Manager – Select SharePoint Online Connector

2. Select the desired authentication type. Then choose the API Base URL. In most cases, the default value is correct. Fill in the required parameters and add optional parameters only when needed.
3. After configuring the required properties in API Connection Manager, click Test Connection to confirm that the connection works.
   

   Method 1: Connect to SharePoint Online using user credentials [OAuth]

   Use the Steps to Configure link to help set the required parameters.

   

   SSIS ZS API Connection Manager – Generate Token – SharePoint Online with User Credentials

   Method 2: Connect to SharePoint Online using application credentials [OAuth]

   Use the Steps to Configure link to help set the required parameters.

   

   SSIS API Source – Generate Token – SharePoint Online with Application Credentials

How to upload a local file to SharePoint Online

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to upload a local file to SharePoint Online.

1. Choose the Upload file endpoint from the “Select Table/Endpoint” dropdown menu.
2. Select the desired Drive ID from the DriveId drop-down.
3. Specify the desired source file path for the file being uploaded by setting the DiskFilePath parameter.
4. Specify the desired Target Path by setting the SharePointFilePath parameter.

Please refer to the sample screenshot below.

SSIS API Source – SharePoint Online – Upload Single File

After configuring the properties above, click Preview Data to test the API call. At this point, the SharePoint Online Connector is ready to upload a local file from SSIS.

How to upload multiple files to SharePoint Online

In the API Source Component, we’ve selected the API Connection Manager. Next, we need to configure the remaining properties of the component to upload multiple files to SharePoint Online. Additionally, we should utilize a Foreach Loop Container to iterate through each file one by one in a loop.

1. Create 3 SSIS variables as described below.
   

   Variable Name	Datatype	Value
   varFileLists	Object	System.Object
   varFileName	String	fake.extension
   varFullFilePath	String	C:/fake/fake.extension

2. Go to Control Flow, From the SSIS toolbox drag and drop ZS Advanced File System Task on the control flow designer surface, and double click on it to edit it:
   
3. Select the “Get file list as ADO.NET Data Table” action from the Action drop-down menu. Set the file path in the Path textbox. Choose the variable (Object type) where you want to store the file list, and then click the OK button to save the settings.
   

   SSIS Advanced File System Task – Get file list as ADO.net Data Table

4. Now, drag and drop the “Foreach Loop Container” from the SSIS Toolbox and connect it with the “ZS Advanced File System Task.” Double-click on the “Foreach Loop Container” to edit its properties.
   

   SSIS Foreach Loop Container – Drag & Drop

5. In the Foreach Loop Editor’s “Collection” tab, choose the Enumerator as “Foreach ADO Enumerator.” In the “ADO object source variable” field, select your SSIS object-typed variable. Select the “Rows in the first table” radio option.
   

   SSIS Foreach Loop Editor – Collection – Select Variable & Enumeration mode

6. In the Foreach Loop Editor’s “Variable Mappings” tab, select the variables that you want to map to the values from the collection, and then click the “OK” button to save your selections.
   

   SSIS Foreach Loop Editor – Variable Mappings

7. Inside the Foreach Loop Container, drag and drop the Data Flow Task, and then rename it according to your preference.
   

   SSIS Data Flow Task – Drag & Drop inside Foreach Loop Container

8. Double-click on the Data Flow Task to see the Data Flow Designer, From the SSIS Toolbox, drag and drop API Source (Predefined Templates) onto the Data Flow Designer surface, then double-click it to edit it:
   
9. Double-click on the API Source component to configure it for uploading multiple files to SharePoint Online.
   1. Choose the Upload file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. Set the SharePointFilePath parameter to the SSIS variable that contains the file name, such as varFileName.
   3. Set the DiskFilePath parameter to the SSIS variable that contains the local file path, such as varFullFilePath.
      When configuring the SSIS variable directly in the Raw Edit, JSON may become invalid due to the requirement of slashes within the local file path. To address this, we use a JSONENC Placeholder Function for proper escaping.
      

      <<{{User::varUserFullFilePath}},FUN_JSONENC>>

   4. If you wish to upload the local file to a specific folder in SharePoint Online, please set the folderName in the SharePointFilePath parameter.

   Please refer to the sample screenshot below.

   

   SSIS API Source – SharePoint Online – Upload Multiple files configuration

   To configure dynamic filenames and file paths, you can set variables. Click on “Raw Edit,” and in the “Edit Parameters” popup, you can use SSIS variables, as shown in the above image.

   Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call.
   After that, execute the SSIS package and confirm that the files are uploaded to SharePoint Online. The screenshot below shows a sample result.

   

   Multiple files uploaded on SharePoint Online using SSIS

   You have now configured the SharePoint Online Connector to upload multiple files from SSIS.

Conclusion

In this guide, you learned how to connect to SharePoint Online using SSIS API Connection Manager and upload one or many files using the SSIS API Source component, SSIS ZS Advanced File System Task, Foreach Loop Container, and SSIS variables.

Frequently Asked Questions

Can I upload more than one file to SharePoint Online from SSIS

Yes. You can use a Foreach Loop Container with SSIS variables and ZS Advanced File System Task to iterate through a file list and upload files one by one.

What is the easiest way to upload files to SharePoint Online from SSIS

The simplest approach is to use the SharePoint Online Connector for SSIS, because it provides ready-made authentication and upload endpoints inside the ZappySys connector interface.

Which component uploads the files

The upload action is configured in API Source by selecting the Upload file endpoint and setting values such as DriveId, DiskFilePath, and SharePointFilePath.

Next step

If you want to build this workflow in your own SSIS package, download SSIS PowerPack and review the SharePoint Online Connector guide.

The post How to upload files on SharePoint Online using SSIS appeared first on ZappySys Blog.

In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate SharePoint Online into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful SharePoint Online Connector that makes downloading files from SharePoint Online a breeze. In this guide, we’ll walk you through the process of downloading files from SharePoint Online using SSIS and the ZappySys SSIS PowerPack SharePoint Online Connector.

Download files from SharePoint Online with SSIS:

In this section, we will learn how to configure and use SharePoint Online Connector in the API Source & API Destination to download files from SharePoint Online.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom
   

   SSIS Toolbox – ZappySys Tasks

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the SharePoint Online Connector

1. Use a preinstalled SharePoint Online Connector from the Popular Connector List or press the Search Online radio button to download SharePoint Online Connector. Once downloaded simply use it in the configuration:
   

   SSIS ZS API Connection Manager – Select SharePoint Online Connector

2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to SharePoint Online using User Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS ZS API Connection Manager – Generate Token – SharePoint Online with User Credentials

   Method2: Connect to SharePoint Online using Application Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – SharePoint Online with Application Credentials

How to download a single file from SharePoint Online

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to download the single file from SharePoint Online.

1. Choose the Download file endpoint from the “Select Table/Endpoint” dropdown menu.
2. Please choose DriveId from the drop-down menu.
3. Please set the FileId that indicates which file you wish to download. This is an essential parameter for the download process.
4. Please set the local folder’s physical path which indicates the desired file being downloaded by setting the TargetFilePath parameter.
5. If you intend to replace the local file with the one being downloaded, please select the ‘AlwaysOverwrite‘ option from the dropdown menu labeled ‘FileOverwriteMode‘. Alternatively, you can choose ‘FileIfExists’ or ‘SkipIfExists’ based on your specific needs.

Please refer to the sample screenshot below.

SSIS API Source – SharePoint Online – Download Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to SharePoint Online using the ZappySys SharePoint Online Connector for downloading a Single file from SharePoint Online.

How to download multiple files from SharePoint Online

In the API Destination Component, we have chosen the API Connection Manager. Now, we must proceed to configure the remaining properties of the component to facilitate the download of multiple files from SharePoint Online. Additionally, we should make use of an OLE DB Source and Derived Column component to map the files for the download process.

1. Drag and drop the OLE DB Source Component from the SSIS Toolbox. Double-click on the OLE D B Source Component to initiate its configuration.
   

   OLE DB Source – Drag and Drop

2. Create a new OLE DB Connection Manager and configure your SQL Server settings for the connection. Next, select the Data access mode, specify the name of the table or view, or write an SQL Command. Afterward, click on the ‘Preview’ button to view a preview and check the ‘Columns’ tab for the source columns.
   

   SSIS OLE DB Source – Configure SQL Connection and fetch the table data

3. Next, drag and drop the Derived Column Component from the SSIS Toolbox. Double-click on the Derived Column Component to begin its configuration.
   

   Drag and drop the Derived Column

4. Now, add a new Derived Column by setting the expression to concatenate FilePath and FileName. Click ‘OK’ to save the configuration.
   

   SSIS Derived Column – Add New Column and Set Expression

5. Now, drag and drop the API Destination Component from the SSIS Toolbox. Double-click on the API Destination Component to begin its configuration.
   

   Drag and Drop the SSIS API Destination Component

6. Double-click on the API Destination component to configure it for downloading multiple files from SharePoint Online.
   1. Choose the Download file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. Please specify the ‘FileOverwriteMode’ option as ‘AlwaysOverwrite,’ or you can choose ‘FailIfExists’ or ‘SkipIfExists’ options based on your requirements.
      Please refer to the sample screenshot below.
      

      SSIS API Destination Component – Configure Settings Tab to download Files from SharePoint Online

   3. Now, go to the Mappings tab and specify the source columns with the corresponding target properties.
      Note: Here We’re mapping FileID with $$Id and FileNameWithPath with $$ResponseDataFile. You can create mappings based on your Source columns
      Please refer to the sample screenshot below
      

      SSIS API Destination Component – Configure the Mappings Tab to download multiple files from SharePoint Online

7. Once you have configured all the mentioned properties above, please click on the “Preview Data” button from the Settings tab to initiate the API call. With just a few clicks, we have successfully configured the call to SharePoint Online using the ZappySys SharePoint Online Connector for downloading multiple files from SharePoint Online.
   Please refer to the sample screenshot below.
   

   Download Multiple files from SharePoint Online

Conclusion

In this blog, we learned how to connect with SharePoint Online using SSIS API Connection Manager and download single or multiple files from SharePoint Online using a combination of SSIS components, including the SSIS API Source component, SSIS API Destination component, OLE DB Source component, and Derived Column component.

The post How to download files from SharePoint Online using SSIS appeared first on ZappySys Blog.

In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate OneDrive into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful OneDrive Connector that makes downloading files from OneDrive a breeze. In this guide, we’ll walk you through the process of downloading files from OneDrive using SSIS and the ZappySys SSIS PowerPack OneDrive Connector.

Download files from OneDrive with SSIS

In this section, we will learn how to configure and use OneDrive Connector in the API Source & API Destination to download files from OneDrive.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom
   

   SSIS Toolbox – ZappySys Tasks

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the OneDrive Connector

1. Use a preinstalled OneDrive Connector from the Popular Connector List or press the Search Online radio button to download OneDrive Connector. Once downloaded simply use it in the configuration:
   
2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to OneDrive using User Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – OneDrive with User Credentials

   Method2: Connect to OneDrive using Application Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – OneDrive with Application Credentials

How to download a single file from OneDrive

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to download the single file from OneDrive.

1. Choose the Download file endpoint from the “Select Table/Endpoint” dropdown menu.
2. Please choose DriveId from the drop-down menu.
3. Please set the FileId that indicates which file you wish to download. This is an essential parameter for the download process.
4. Please set the local folder’s physical path which indicates the desired file being downloaded by setting the TargetFilePath parameter.
5. If you intend to replace the local file with the one being downloaded, please select the ‘AlwaysOverwrite‘ option from the dropdown menu labeled ‘FileOverwriteMode‘. Alternatively, you can choose ‘FileIfExists’ or ‘SkipIfExists’ based on your specific needs.

Please refer to the sample screenshot below.

SSIS API Source – OneDrive – Download Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to OneDrive using the ZappySys OneDrive Connector for downloading a Single file from OneDrive.

How to download multiple files from OneDrive

In the API Destination Component, we have chosen the API Connection Manager. Now, we must proceed to configure the remaining properties of the component to facilitate the download of multiple files from OneDrive. Additionally, we should make use of an OLE DB Source and Derived Column component to map the files for the download process.

1. Drag and drop the OLE DB Source Component from the SSIS Toolbox. Double-click on the OLE DB Source Component to initiate its configuration.
   

   OLE DB Source – Drag and Drop

2. Create a new OLE DB Connection Manager and configure your SQL Server settings for the connection. Next, select the Data access mode, specify the name of the table or view, or write an SQL Command. Afterward, click on the ‘Preview’ button to view a preview and check the ‘Columns’ tab for the source columns.
   

   SSIS OLE DB Source – Configure SQL Connection and fetch the table data

3. Next, drag and drop the Derived Column Component from the SSIS Toolbox. Double-click on the Derived Column Component to begin its configuration.
   

   Drag and drop the Derived Column

4. Now, add a new Derived Column by setting the expression to concatenate FilePath and FileName. Click ‘OK’ to save the configuration.
   

   SSIS Derived Column – Add New Column and Set Expression

5. Now, drag and drop the API Destination Component from the SSIS Toolbox. Double-click on the API Destination Component to begin its configuration.
   

   Drag and Drop the SSIS API Destination Component

6. Double-click on the API Destination component to configure it for downloading multiple files from OneDrive.
   1. Choose the Download file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. Please specify the ‘FileOverwriteMode’ option as ‘AlwaysOverwrite,’ or you can choose ‘FailIfExists’ or ‘SkipIfExists’ options based on your requirements.
      Please refer to the sample screenshot below.
      

      SSIS API Destination Component – Configure Settings Tab

   3. Now, go to the Mappings tab and specify the source columns with the corresponding target properties.
      Note: Here We’re mapping FileID with $$Id and FileNameWithPath with $$ResponseDataFile. You can create mappings based on your Source columns
      Please refer to the sample screenshot below
      

      SSIS API Destination Component – Configure Mappings Tab

7. Once you have configured all the mentioned properties above, please click on the “Preview Data” button from the Settings tab to initiate the API call. With just a few clicks, we have successfully configured the call to OneDrive using the ZappySys OneDrive Connector for downloading multiple files from OneDrive.
   Please refer to the sample screenshot below.
   

   Download Multiple files from OneDrive

Conclusion

In this blog, we learned how to connect with OneDrive using SSIS API Connection Manager and download single or multiple files from OneDrive using a combination of SSIS components, including the SSIS API Source component, SSIS API Destination component, OLE DB Source component, and Derived Column component.

The post How to download files from OneDrive using SSIS appeared first on ZappySys Blog.

In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate OneDrive into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful OneDrive Connector that makes uploading files to OneDrive a breeze. In this guide, we’ll walk you through the process of uploading files to OneDrive using SSIS and the ZappySys SSIS PowerPack OneDrive Connector.

Upload files to OneDrive with SSIS:

In this section, we will learn how to configure and use OneDrive Connector in the API Source to upload files on OneDrive.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom
   

   SSIS Toolbox – ZappySys Tasks

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the OneDrive Connector

1. Use a preinstalled OneDrive Connector from the Popular Connector List or press the Search Online radio button to download OneDrive Connector. Once downloaded simply use it in the configuration:
   
2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to OneDrive using User Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – OneDrive with User Credentials

   Method2: Connect to OneDrive using Application Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – OneDrive with Application Credentials

How to upload a local file to OneDrive

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to upload a local file to OneDrive.

1. Choose the Upload file endpoint from the “Select Table/Endpoint” dropdown menu.
2. Select the desired Drive ID from the DriveId drop-down.
3. Specify the desired source file path for the file being uploaded by setting the SourceFilePath parameter.
4. Specify the desired Target Path by setting the TargetFilePath parameter.

Please refer to the sample screenshot below.

SSIS API Source – OneDrive – Upload Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to OneDrive using the ZappySys OneDrive Connector for uploading a local file to OneDrive.

How to upload multiple files to OneDrive

In the API Source Component, we’ve selected the API Connection Manager. Next, we need to configure the remaining properties of the component to upload multiple files to OneDrive. Additionally, we should utilize a Foreach Loop Container to iterate through each file one by one in a loop.

1. Create 3 SSIS variables as described below.
   

   Variable Name	Datatype	Value
   varFileLists	Object	System.Object
   varFileName	String	fake.extension
   varFullFilePath	String	C:/fake/fake.extension

2. Go to Control Flow, From the SSIS toolbox drag and drop ZS Advanced File System Task on the control flow designer surface, and double click on it to edit it:
   
3. Select the “Get file list as ADO.NET Data Table” action from the Action drop-down menu. Set the file path in the Path textbox. Choose the variable (Object type) where you want to store the file list, and then click the OK button to save the settings.
   

   SSIS Advanced File System Task – Get file list as ADO.net Data Table

4. Now, drag and drop the “Foreach Loop Container” from the SSIS Toolbox and connect it with the “ZS Advanced File System Task.” Double-click on the “Foreach Loop Container” to edit its properties.
   

   SSIS Foreach Loop Container – Drag & Drop

5. In the Foreach Loop Editor’s “Collection” tab, choose the Enumerator as “Foreach ADO Enumerator.” In the “ADO object source variable” field, select your SSIS object-typed variable. Select the “Rows in the first table” radio option.
   

   SSIS Foreach Loop Editor – Collection – Select Variable & Enumeration mode

6. In the Foreach Loop Editor’s “Variable Mappings” tab, select the variables that you want to map to the values from the collection, and then click the “OK” button to save your selections.
   

   SSIS Foreach Loop Editor – Variable Mappings

7. Inside the Foreach Loop Container, drag and drop the Data Flow Task, and then rename it according to your preference.
   

   SSIS Data Flow Task – Drag & Drop inside Foreach Loop Container

8. Double-click on the Data Flow Task to see the Data Flow Designer, From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
9. Double-click on the API Source component to configure it for uploading multiple files to OneDrive.
   1. Choose the Upload file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. Please specify the desired SSIS Variable [varFileName] containing the uploaded file name by setting the TargetFilePath parameter.
   3. To specify the desired local SSIS variable [varFullFilePath], which holds the file path, set the SourceFilePath parameter.
      When configuring the SSIS variable directly in the Raw Edit, JSON may become invalid due to the requirement of slashes within the local file path. To address this, we use a JSONENC Placeholder Function for proper escaping.
      

      <<{{User::varUserFullFilePath}},FUN_JSONENC>>

   4. If you wish to upload the local file to a specific folder in OneDrive, please set the folderName in the TargetFilePath parameter.

   Please refer to the sample screenshot below.

   

   SSIS API Source – OneDrive – Upload Multiple files configuration

   To configure dynamic filenames and file paths, you can set variables. Click on “Raw Edit,” and in the “Edit Parameters” popup, you can use SSIS variables, as shown in the above image.

   Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call.
   After that, you can execute the SSIS Package and check the execution to upload multiple files on OneDrive. Below is the sample screenshot attached.
   
   Multiple files uploaded on OneDrive using SSIS

   With just a few clicks, we have successfully configured the call to OneDrive using the ZappySys OneDrive Connector for uploading multiple files to OneDrive.

Conclusion

In this blog, we learned how to connect with OneDrive using SSIS API Connection Manager and upload single or multiple files to OneDrive using a combination of SSIS components, including the SSIS API Source component, SSIS ZS Advanced File System Task, Foreach Loop Container, and SSIS variables.

The post How to upload files on OneDrive using SSIS appeared first on ZappySys Blog.

In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate Google Drive into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful Google Drive Connector that makes downloading files from Google Drive a breeze. In this guide, we’ll walk you through the process of downloading files from Google Drive using SSIS and the ZappySys SSIS PowerPack Google Drive Connector.

Download files from Google Drive with SSIS:

In this section, we will learn how to configure and use Google Drive Connector in the API Source & API Destination to download files from Google Drive.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom

   -`
   

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the Google Drive Connector

1. Use a preinstalled Google Drive Connector from the Popular Connector List or press the Search Online radio button to download Google Drive Connector. Once downloaded simply use it in the configuration:
   
2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to Google Drive using User Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with User Account

   Method2: Connect to Google Drive using Service Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with Service Account

How to download a single file from Google Drive

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to download the single file from Google Drive.

1. Choose the Download a file endpoint from the “Select Table/Endpoint” dropdown menu.
2. To access detailed API documentation for downloading files, you can click on the following link: Help for: Download a File.
3. Please choose a File ID from the drop-down menu to indicate which file you wish to download. This is an essential parameter for the download process.
4. Please indicate the desired filename for the file being downloaded by setting the FileName parameter.
5. If you intend to replace the local file with the one being downloaded, please select the ‘AlwaysOverwrite‘ option from the dropdown menu labeled ‘FileOverwriteMode‘. Alternatively, you can choose ‘FileIfExists’ or ‘SkipIfExists’ based on your specific needs.

Please refer to the sample screenshot below.

SSIS API Source – Google Drive – Download Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for downloading a Single file from Google Drive.

How to download multiple files from Google Drive

In the API Destination Component, we have chosen the API Connection Manager. Now, we must proceed to configure the remaining properties of the component to facilitate the download of multiple files from Google Drive. Additionally, we should make use of an OLE DB Source and Derived Column component to map the files for the download process.

1. Drag and drop the OLE DB Source Component from the SSIS Toolbox. Double-click on the OLE DB Source Component to initiate its configuration.
   

   OLE DB Source – Drag and Drop

2. Create a new OLE DB Connection Manager and configure your SQL Server settings for the connection. Next, select the Data access mode, specify the name of the table or view, or write an SQL Command. Afterward, click on the ‘Preview’ button to view a preview and check the ‘Columns’ tab for the source columns.
   

   SSIS OLE DB Source – Configure SQL Connection and fetch the table data

3. Next, drag and drop the Derived Column Component from the SSIS Toolbox. Double-click on the Derived Column Component to begin its configuration.
   

   Drag and drop the Derived Column

4. Now, add a new Derived Column by setting the expression to concatenate FilePath and FileName. Click ‘OK’ to save the configuration.
   

   SSIS Derived Column – Add New Column & Set Expression

5. Now, drag and drop the API Destination Component from the SSIS Toolbox. Double-click on the API Destination Component to begin its configuration.
   

   Drag and Drop the SSIS API Destination Component

6. Double-click on the API Destination component to configure it for uploading multiple files to Google Drive.
   1. Choose the Download a file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. To access detailed API documentation for downloading files, you can click on the following link: Help for: Download File.
   3. Please specify the ‘FileOverwriteMode’ option as ‘AlwaysOverwrite,’ or you can choose ‘FailIfExists’ or ‘SkipIfExists’ options based on your requirements.
      Please refer to the sample screenshot below.
      

      SSIS API Destination Component – Configure Settings Tab

   4. Now, go to the Mappings tab and specify the source columns with the corresponding target properties.
      Note: Here We’re mapping FileID with $$Id and FileNameWithPath with $$ResponseDataFile. You can create mappings based on your Source columns
      Please refer to the sample screenshot below
      

      SSIS API Destination Component – Configure Mappings Tab

7. Once you have configured all the mentioned properties above, please click on the “Preview Data” button from the Settings tab to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for downloading multiple files from Google Drive.
   Please refer to the sample screenshot below.
   

   Download Multiple files from Google Drive

Conclusion

In this blog, we learned how to connect with Google Drive using SSIS API Connection Manager and download single or multiple files from Google Drive using a combination of SSIS components, including the SSIS API Source component, SSIS API Destination component, OLE DB Source component, and Derived Column component.

The post How to download files from Google Drive using SSIS appeared first on ZappySys Blog.

In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate Google Drive into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful Google Drive Connector that makes uploading files to Google Drive a breeze. In this guide, we’ll walk you through the process of uploading files to Google Drive using SSIS and the ZappySys SSIS PowerPack Google Drive Connector.

Upload files to Google Drive with SSIS:

In this section, we will learn how to configure and use Google Drive Connector in the API Source to upload files on Google Drive.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom
   

   SSIS Toolbox – ZappySys Tasks

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the Google Drive Connector

1. Use a preinstalled Google Drive Connector from the Popular Connector List or press the Search Online radio button to download Google Drive Connector. Once downloaded simply use it in the configuration:
   
2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to Google Drive using User Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with User Account

   Method2: Connect to Google Drive using Service Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with Service Account

How to upload a local file to Google Drive

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to upload a local file to Google Drive.

1. Choose the Upload a file endpoint from the “Select Table/Endpoint” dropdown menu.
2. To access detailed API documentation for uploading files, you can click on the following link: Help for: Upload File.
3. Please specify the desired filename for the uploaded file by setting the FileName parameter.
4. Specify the desired local file path by setting the DiskFilePath parameter.
5. If you wish to upload the local file to a specific folder in Google Drive, please choose the folder ID from the dropdown menu labeled ParentFolderId. Otherwise, you can skip this parameter as it’s optional.

Please refer to the sample screenshot below.

SSIS API Source – Google Drive – Upload Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for uploading a local file to Google Drive.

How to upload multiple files to Google Drive

In the API Source Component, we’ve selected the API Connection Manager. Next, we need to configure the remaining properties of the component to upload multiple files to Google Drive. Additionally, we should utilize a Foreach Loop Container to iterate through each file one by one in a loop.

1. Create 3 SSIS variables as described below.
   

   Variable Name	Datatype	Value
   varFileLists	Object	System.Object
   varFileName	String	fake.extension
   varFullFilePath	String	C:/fake/fake.extension

2. Go to Control Flow, From the SSIS toolbox drag and drop ZS Advanced File System Task on the control flow designer surface, and double click on it to edit it:
   
3. Select the “Get file list as ADO.NET Data Table” action from the Action drop-down menu. Set the file path in the Path textbox. Choose the variable (Object type) where you want to store the file list, and then click the OK button to save the settings.
   

   SSIS Advanced File System Task – Get file list as ADO.net Data Table

4. Now, drag and drop the “Foreach Loop Container” from the SSIS Toolbox and connect it with the “ZS Advanced File System Task.” Double-click on the “Foreach Loop Container” to edit its properties.
   

   SSIS Foreach Loop Container – Drag & Drop

5. In the Foreach Loop Editor’s “Collection” tab, choose the Enumerator as “Foreach ADO Enumerator.” In the “ADO object source variable” field, select your SSIS object-typed variable. Select the “Rows in the first table” radio option.
   

   SSIS Foreach Loop Editor – Collection – Select Variable & Enumeration mode

6. In the Foreach Loop Editor’s “Variable Mappings” tab, select the variables that you want to map to the values from the collection, and then click the “OK” button to save your selections.
   

   SSIS Foreach Loop Editor – Variable Mappings

7. Inside the Foreach Loop Container, drag and drop the Data Flow Task, and then rename it according to your preference.
   

   SSIS Data Flow Task – Drag & Drop inside Foreach Loop Container

8. Double-click on the Data Flow Task to see the Data Flow Designer, From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double click on it to edit it:
   
9. Double-click on the API Source component to configure it for uploading multiple files to Google Drive.
   1. Choose the Upload a file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. To access detailed API documentation for uploading files, you can click on the following link: Help for: Upload File.
   3. Please specify the desired SSIS Variable [varFileName] which contains file name for the uploaded file by setting the FileName parameter.
   4. To specify the desired local SSIS variable [varFullFilePath], which holds the file path, set the DiskFilePath parameter.
      When configuring the SSIS variable directly in the Raw Edit, JSON may become invalid due to the requirement of slashes within the local file path. To address this, we use a JSONENC Placeholder Function for proper escaping.
      

      <<{{User::varUserFullFilePath}},FUN_JSONENC>>

   5. If you wish to upload the local file to a specific folder in Google Drive, please choose the folder ID from the dropdown menu labeled ParentFolderId. Otherwise, you can skip this parameter as it’s optional.

   Please refer to the sample screenshot below.

   

   SSIS API Source – Google Drive – Upload Multiple files configuration

   To configure dynamic filenames and file paths, you can set variables. Click on “Raw Edit,” and in the “Edit Parameters” popup, you can use SSIS variables, as shown in the image below.

   Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for uploading multiple files to Google Drive.

Conclusion

In this blog, we learned how to connect with Google Drive using SSIS API Connection Manager and upload single or multiple files to Google Drive using a combination of SSIS components, including the SSIS API Source component, SSIS ZS Advanced File System Task, Foreach Loop Container, and SSIS variables.

The post How to upload files on Google Drive using SSIS appeared first on ZappySys Blog.

In this blog,  we are going to learn to how to compare records in SSIS using hash method. We will use FREE Template Transform component along with Sort Transform, Merge Join Transform and Conditional Split Transform. We will see how to produce hash value from multiple columns and later compare source\target hash values to detect changes\new rows.

Data Hashing

Here is sample design how to compare records using hash method

1. In Step 1, we need two data tables for record comparison from source and destination.Source Data:
   Target Data:
   
2. In Step 2, both the source and target data table records would be hash using the ZS Template Transform component as below. You can use FUN_MD5 to produce hash from multiple columns and. Please make sure that you change the Output Column name as below e.g. HashCrm or HashSQL
3. In Step 3, both the source and target data table need to be sorted by primary key using SSIS “Sort Transformation Editor” component as shown in above sample design screenshot.
4. In Step 4, both the source and target data table need to be merge using SSIS “Merge Join” component providing appropriate join type and key mapping
   as below
5. In Step 4, we adding the conditional split with appropriate condition based on your need e.g. we want to check similar records as below
6. From conditional split, we can feed the modified or same records into destination component based on our needs.
7. This is how we can compare the data for source and target tables for updated records and also we can set up different Merge Join and conditional split for desired set of output records set.

Detect Update/Insert

1. In previous section, we have chosen the updated records using hash. Here are going to see the Updated/New records using Left Outer Join as belowLeft Outer Join
   
2. Here is conditional split criteria for Update/New records as below

Conclusion

In this tutorial we learned how to compare the records in SSIS using hash method. If you want take a look it then download the ZappySys SSIS PowerPack here.

The post How to compare records in SSIS using hash appeared first on ZappySys Blog.

In this post, we will cover how to connect ADP API using HTTP connection manager over OAuth connection manager with JSON Source. ADP API works a bit differently with the OAuth connection manager. Let’s take a look at it…

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Optional (If you want to Deploy and Schedule ) – Deploy and Schedule SSIS Packages
5. Make sure you have ADP Client Id & Secret.

What things are needed to connect ADP API

1. You need to have the Client ID and Client Secret before making your ADP API call.
2. If you don’t have this information, contact your client representative.
3. Certificate Signing Request (CSR): For more detailed information, see the Certificate Signing Request: https://developers.adp.com/articles/general/generate-a-certificate-signing-request
4. If you already have the .csr file from your API Settings as below: –
   

   ADP CSR Certificate

    

5. Then you need to convert those *.cer + *.key files to *.PFX
6. We do not support cert+key files but converting to PFX is fairly simple.
7. You can find many blog posts on how to convert to PFX. Here are some helpful links on how to convert *.pem to *.PFX (i.e. PKCS#12 OR p12 file format)
   -https://www.sslshopper.com/ssl-converter.html
   -https://stackoverflow.com/questions/808669/convert-a-cert-pem-certificate-to-a-pfx-certificate
8. Once you have .PFX certificate, Client ID, and Client Secret ready then you can provide these details to the  HTTP Connection manager as below: –
   General Settings: –
   

   ADP HTTP Connection Manager – General Tab

   Client Certificate: –

   

   ADP CSR Client Certificate Tab
9. After you provide all the required details in HTTP Connection Manager for ADP API.
10. You can select HTTP Connection manager as the connection provider in the JSON Source/REST API and preview the data.
11. Here you can connect ADP API successfully using the HTTP connection manager.

Note: By default, when a PFX file expires, the ADFP API throws a generic SSL/TLS channel error, which can be misleading. To verify the certificate’s expiration date, we have to:

1. Import the PFX file into the local certificate store (using the same blank password).
2. Use the mmc command from the Start menu.
3. Go to Add/Remove Snap-ins → Certificate → User Store → View Entry to check the expiration date.

Steps to Convert .cer and .key Files into a .PFX File

If you have separate .cer and .key files, you’ll need to convert them into a single .PFX file (PKCS#12 format) to use with our tool, as it does not support separate certificate and key files.
Converting to PFX is straightforward, and you can find many detailed guides online. Here are some helpful links:

• SSL Converter – sslshopper.com
• Convert .pem to .pfx – Stack Overflow

Conclusion:

This is how you can successfully connect ADP API using HTTP Connection over OAuth connection using ZappySys JSON Source/REST API.

The post How To Connect ADP API using HTTP Connection over OAuth Connection appeared first on ZappySys Blog.

In this post, we will see how to create SSIS Package Programmatically to use ZappySys SSIS PowerPack. This blog post assumes you have basic knowledge of Programming language like C# and you have installed Visual Studio Express or a similar product (VS Code not supported).

Add SSIS Dll References

The very first thing is to create a C# Project (i.e. Console or WinForm). Add SSIS dll reference.

You will need the following references in your project. The below example is for SSIS 2017 (Version 14.x.x)

• For SSIS 2012 change to Version=11.0.0.0
• For SSIS 2014 change to Version=12.0.0.0
• For SSIS 2016 change to Version=13.0.0.0
• For SSIS 2017 change to Version=14.0.0.0
• For SSIS 2019 change to Version=15.0.0.0
• And may be future versions same way ….

<Reference Include="Microsoft.SqlServer.Dts.Design"/>
<Reference Include="Microsoft.SqlServer.DTSPipelineWrap" />
<Reference Include="Microsoft.SQLServer.DTSRuntimeWrap" />
<Reference Include="Microsoft.SQLServer.ManagedDTS" />
<Reference Include="Microsoft.SqlServer.PipelineHost" />

Step-By-Step Example (Create SSIS Package Programmatically – C#)

Now let’s see each piece to create SSIS Package in code and later we will show the full code.

Import Namespaces

First step is at the top you need to import some namespaces.

using Microsoft.SqlServer.Dts.Runtime;
using Microsoft.SqlServer.Dts.Pipeline;
//using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
//using Microsoft.SqlServer.Dts.Runtime.Wrapper;

Create SSIS Package

Here how you can create SSIS Package instance in code

var p = new Package();

Add SSIS Connection

Now lets add connection manager to our package instance. To add connection you will need two pieces. Creation Name (i.e. ZS-SFTP) and ConnectionString. You can find creation name from exsing Dtsx file by opening in text editor.

var conn1=p.Connections.Add("ZS-SFTP");
conn1.Name = "MySftpConnection";
conn1.ConnectionString= "Protocol=Sftp;ServerName=127.0.0.1;Port=0;LogonType=Normal;UserName=zstest;Password=zstest123";

How to find CreationName of SSIS Connection manager

To find the creation name of your Connection Type open an existing DTSX file where you had similar connection type in the package. Locate node like below and check CrewationName. It can be a friendly name or can be fully qualified assembly name. Both ways is fine but a friendly name is easy to remember.

<DTS:ConnectionManagers>
  <DTS:ConnectionManager
    DTS:refId="Package.ConnectionManagers[RedshiftConn1]"
    DTS:CreationName="ZS-POSTGRESQL"
    .........
    .........

How to find ConnectionString

Now another thing you need to know is the exact connectionstring. You can go to SSIS Designer and right-click on Connection Icon and check Properties Grid. It will show you ConnectionString or you can use the above technique > Open DTSX file in notepad and find connectinstring there.

ConnectionString doesnt include any passwords / secrets. So manually append those properties in your connectionstring when you set in code.

Add SSIS Task

Now let’s add task in our SSIS Package instance. To create SSIS Task programmatically in code, you need to know the exact type name (i.e. creation name). As we discussed earlier check your dtsx file to know the creation name for the task you like to create. In the below code we used a friendly name which is not supported in SSIS 2012 so for that you must use full name. For SSIS 2014 or higher you can use friendly names.

You can also find it in file like this (example of SSIS 2017 and 2019 folders)
C:\Program Files (x86)\Microsoft SQL Server\140\DTS\UpgradeMappings\ZappySys.PowerPack140.Extensions.xml
C:\Program Files (x86)\Microsoft SQL Server\150\DTS\UpgradeMappings\ZappySys.PowerPack150.Extensions.xml

//var SSIS_VERSION = "110"; //110=SQL2012, 120=SQL2014, 130=SQL2016, 140=SQL2017, 150=SQL2019
//For SSIS 2012
var SFTP_TASK_2012 = "ZappySys.PowerPack.Task.SecureFtpTask.SecureFtpTask, ZappySys.PowerPack110, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502";
//For SSIS 2014 or higher
//Goto folder like this and find zappysys extension file 
//(e.g. 140) => C:\Program Files (x86)\Microsoft SQL Server\140\DTS\UpgradeMappings\ZappySys.PowerPack140.Extensions.xml
//Find Task creation name there
var SFTP_TASK = "ZappySys.SecureFtpTask";

var tsk1 = p.Executables.Add(SFTP_TASK) as TaskHost;
tsk1.Properties["SourceStorageConnection"].SetValue(tsk1, "MySftpConnection");
tsk1.Properties["StorageAction"].SetValue(tsk1, 10); //DownloadFileFromFtp [10] find numeric value for this enum from help file> properties https://zappysys.com/onlinehelp/ssis-powerpack/scr/ssis-sftp-ftp-ftps-task.htm
tsk1.Properties["SourcePathValue"].SetValue(tsk1, "subfolder/localsubfile1.csv");
tsk1.Properties["TargetPathValue"].SetValue(tsk1, @"c:\temp\");
tsk1.Properties["FileOverwriteAction"].SetValue(tsk1, 2); //OverwriteAll [2]

Execute SSIS Package

Now let’s execute the package using the below line

var status =p.Execute();

Check Execution Errors

At the end you need to check execution errors if any.

if (status != DTSExecResult.Success)
{
	string errs = p.Errors.Cast<DtsError>().Aggregate("",
           (current, err) => current + string.Format("Source={0}, Err={1}", err.Source, err.Description));
	throw new Exception("Package status was >> " + status + " >> " + errs);
}

Full Code

Here is the full code

using System;
using System.Linq;
using System.Runtime.InteropServices;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime;
/*
For SSIS 2012 - Compile with .net 4.0
For SSIS 2014-2017 - Compile with .net 4.5 
For SSIS 2019 - Compile with .net 4.6.2
 */

//current one is SSIS 2017 (140 dlls)
/*
 * References needed for SSIS 2017 , for 2019 change 14.0.0.0 to 15.0.0.0
 * 
    <Reference Include="Microsoft.SqlServer.DtsMsg, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL" />
    <Reference Include="Microsoft.SqlServer.Dts.Design, Version=14.0.0.0, PublicKeyToken=89845dcd8080cc91" />
    <Reference Include="Microsoft.SqlServer.DTSPipelineWrap, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL" />
    <Reference Include="Microsoft.SQLServer.DTSRuntimeWrap, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91" />
    <Reference Include="Microsoft.SQLServer.ManagedDTS, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL" />
    <Reference Include="Microsoft.SqlServer.PipelineHost, Version=14.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL" /> 
 */
namespace SSISCodeDemo
{
    class Program
    {
        static void Main(string[] args)
        {
            
            //Create_SSIS_Controlflow_Task_Demo();
            Create_SSIS_Dataflow_Source_Component_Demo();

            //Create_SSIS_Connection_Demo();
            //Create_SSIS_Variable_Demo();
            //Create_SSIS_Connect_Tasks_Demo();

        }

        private static void Create_SSIS_Controlflow_Task_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();

            var conn1 = p.Connections.Add("ZS-SFTP");
            conn1.Name = "MySftpConnection";
            conn1.ConnectionString = "Protocol=Sftp;ServerName=127.0.0.1;Port=0;LogonType=Normal;UserName=zstest;Password=zstest123";

            //var SSIS_VERSION = "110"; //110=SQL2012, 120=SQL2014, 130=SQL2016, 140=SQL2017, 150=SQL2019
            //For SSIS 2012
            var SFTP_TASK_2012 = "ZappySys.PowerPack.Task.SecureFtpTask.SecureFtpTask, ZappySys.PowerPack110, Version=1.0.0.0, Culture=neutral, PublicKeyToken=1903265a93eae502";
            //For SSIS 2014 or higher
            //Goto folder like this and find zappysys extension file (e.g. 140) => C:\Program Files (x86)\Microsoft SQL Server\140\DTS\UpgradeMappings\ZappySys.PowerPack140.Extensions.xml
            //Find Task creation name there
            var SFTP_TASK = "ZappySys.SecureFtpTask";

            var tsk1 = p.Executables.Add(SFTP_TASK) as TaskHost;
            tsk1.Name = "SFTP_Task1";
            tsk1.Properties["SourceStorageConnection"].SetValue(tsk1, "MySftpConnection");
            tsk1.Properties["StorageAction"].SetValue(tsk1, 10); //DownloadFileFromFtp [10] find numeric value for this enum from help file> properties https://zappysys.com/onlinehelp/ssis-powerpack/scr/ssis-sftp-ftp-ftps-task.htm
            tsk1.Properties["SourcePathValue"].SetValue(tsk1, "subfolder/localsubfile1.csv");
            tsk1.Properties["TargetPathValue"].SetValue(tsk1, @"c:\temp\");
            tsk1.Properties["FileOverwriteAction"].SetValue(tsk1, 2); //OverwriteAll [2]


            /*
             Go to package file (*.dtsx) > Open in Notepad > and findout which properties set. Do same way (ignore some default proprties)
             Some enum properties needs numeric values > Check Task / Component help file to findout number for that

      <DTS:ObjectData>
        <SecureFtpTask
          StorageAction="DeleteFtpFile"
          FileOverwriteAction="OverwriteAll"
          TreatRegXForFullPath="False"
          ContinueOnError="False"
...................
...................
...................            
            PrefixTimestamp="False" />
      </DTS:ObjectData>
    </DTS:Executable>             
             */

            //Application.

            /* use below code to save to disk file 
            var app = new Microsoft.SqlServer.Dts.Runtime.Application();
            var filepath = @"c:\temp\demo.dtsx";
            app.SaveToXml(filepath, p, null);

            --load from disk to memory
            p = app.LoadPackage(filepath, null);

            */
            var status = p.Execute();

            if (status != DTSExecResult.Success)
            {
                var errs = "";
                foreach (var e in p.Errors)
                {
                    errs = errs + string.Format("Source={0}, Err={1}", e.Source, e.Description) + "\r\n";
                }

                throw new Exception("Package status was >> " + status + " >> " + errs);
            }

            Console.WriteLine("Execution Completed... Press any key to exit!!!");
            Console.ReadKey();
        }
        private static void Create_SSIS_Dataflow_Source_Component_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();


            /*============ STEP-1: Create Dataflow Task =========== */

            var tsk1 = p.Executables.Add("STOCK:PipelineTask") as TaskHost;
            var dataFlowTask = tsk1.InnerObject as MainPipe;

            /*============ STEP-2: Create Connection =========== */
            var conn1 = p.Connections.Add("ZS-HTTP");
            conn1.Name = "MyConnection";
            conn1.ConnectionString = "Url=https://httpbin.org/get;CredentialType=Basic;UserName=test;Password=test123";


            /*============ STEP-3: Create Source =========== */
            var sourceComponent = dataFlowTask.ComponentMetaDataCollection.New();

            try
            {
                sourceComponent.ComponentClassID = "ZappySys.JsonSource";
                var sourceInstance = sourceComponent.Instantiate();
                sourceInstance.ProvideComponentProperties();

                sourceComponent.Name = "MySource";
                /**** Special Case for JSON Source - if user wants connection to use then we create Runtime connection. For all other components this is always created when ProvideComponentProperties() is called*/
                if (sourceComponent.RuntimeConnectionCollection.Count <= 0)
                {
                    var dtsRuntimeConnection = sourceComponent.RuntimeConnectionCollection.New();
                    dtsRuntimeConnection.Name = "Connection"; 
                    //dtsRuntimeConnection.Description = "This can be ZS-HTTP, ZS-OAUTH or ZS-SALESFORCE";
                }
                sourceComponent.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.GetExtendedInterface(conn1);
                sourceComponent.RuntimeConnectionCollection[0].ConnectionManagerID = conn1.ID;

                sourceInstance.SetComponentProperty("UseConnection", true);//JsonSource specific -- for all other components connection always used
                sourceInstance.SetComponentProperty("DirectPath", "https://httpbin.org/basic-auth/test/test123");
                //sourceInstance.SetComponentProperty("Filter", "$.value[*]");

                //make connection active before next step
                sourceInstance.AcquireConnections(null);

                //after connection + props set ... call ReInitialize meta... it creates columns / output / input etc
                sourceInstance.ReinitializeMetaData(); 
            }
            catch (COMException ex)
            {
                throw new Exception("Error Happened ==> " + sourceComponent.GetErrorDescription(ex.ErrorCode));
            }




            /*============ STEP-4: Create Target (without Reinitialize) =========== */

            var targetComponent = dataFlowTask.ComponentMetaDataCollection.New();
            CManagedComponentWrapper targetInstance=null;
            try
            {
                targetComponent.ComponentClassID = "ZappySys.TrashDestination";
                targetInstance = targetComponent.Instantiate();
                targetInstance.ProvideComponentProperties();

                targetComponent.Name = "MyDestination";

                targetInstance.SetComponentProperty("SaveDataToFile", true);
                targetInstance.SetComponentProperty("FilePath", @"c:\temp\dump.txt");
                targetInstance.SetComponentProperty("OverwriteFileIfExists", true);

                //--- Trash destination doesnt have connection but incase destination uses connection... set below way
                //if you have connection then set this way
                //var conn2 = p.Connections.Add("....."); //Creation name i.e. ZS-HTTP
                //conn2.Name = "MyConnection2";
                //conn3.ConnectionString = "Url=https://httpbin.org/get;CredentialType=Basic;UserName=test;Password=test123";
                //if (targetComponent.RuntimeConnectionCollection.Count <= 0)
                //{
                //    var dtsRuntimeConnection = targetComponent.RuntimeConnectionCollection.New();
                //    dtsRuntimeConnection.Name = "Connection"; 
                //}
                //sourceComponent.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.GetExtendedInterface(conn2);
                //sourceComponent.RuntimeConnectionCollection[0].ConnectionManagerID = conn2.ID;
                //make connection active before next step
                //targetInstance.AcquireConnections(null);
            }
            catch (COMException ex)
            {
                throw new Exception("Error Happened ==> " + targetComponent.GetErrorDescription(ex.ErrorCode));
            }

            /*============ STEP-5: Connect Source to Target =========== */

            //before we call reintializemetadata for dest .. lets attach src and dest so .. dest can use virtual inputs if needed
            var output = sourceComponent.OutputCollection[0]; //to make it simple lets pick first output but possible component can have more outputs
            var input = targetComponent.InputCollection[0];

            dataFlowTask.PathCollection.New().AttachPathAndPropagateNotifications(output, input);

            /*============ STEP-6: Configure Target =========== */
            try
            {
                //after connection + props set and source to target attached... call ReInitialize meta... it creates columns / output / input etc
                targetInstance.ReinitializeMetaData(); 
            }
            catch (COMException ex)
            {
                throw new Exception("Error Happened ==> " + targetComponent.GetErrorDescription(ex.ErrorCode));
            }


            /*============ STEP-6: Save Package =========== */

            //* use below code to save to disk file 
            var app = new Microsoft.SqlServer.Dts.Runtime.Application();
            var filepath = @"c:\temp\demo.dtsx";
            app.SaveToXml(filepath, p, null);

            /*============ STEP-7: Load Package =========== */
            //--load from disk to memory
            p = app.LoadPackage(filepath, null);

            //*/

            /*============ STEP-8: Execute Package =========== */
            var status = p.Execute();

            /*============ STEP-9: Print execution status / log =========== */
            if (status != DTSExecResult.Success)
            {
                var errs = "";
                foreach (var e in p.Errors)
                {
                    errs = errs + string.Format("Source={0}, Err={1}", e.Source, e.Description) + "\r\n";
                }

                throw new Exception("Package status was >> " + status + " >> " + errs);
            }

            Console.WriteLine("Execution Completed... Press any key to exit!!!");
            Console.ReadKey();
        }

        private static void Create_SSIS_Variable_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();
            
            p.Variables.Add("MyVariableName1", false, "User", "Some value");
        }

        private static void Create_SSIS_Connection_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();

            var conn1 = p.Connections.Add("ZS-SFTP");
            conn1.Name = "MySftpConnection";
            conn1.ConnectionString = "Protocol=Sftp;ServerName=127.0.0.1;Port=0;LogonType=Normal;UserName=zstest;Password=zstest123";

        }

        private static void Create_SSIS_Connect_Tasks_Demo()
        {
            var p = new Microsoft.SqlServer.Dts.Runtime.Package();

            var SFTP_TASK = "ZappySys.SecureFtpTask";

            var tsk1 = p.Executables.Add(SFTP_TASK) as TaskHost;
            tsk1.Name = "SFTP_Task1";
            /*set some props - see Create_SSIS_Controlflow_Task_Demo */

            var tsk2 = p.Executables.Add(SFTP_TASK) as TaskHost;
            tsk2.Name = "SFTP_Task2";
            /*set some props - see Create_SSIS_Controlflow_Task_Demo */
            
            //add link between task1-->task2
            var pc = p.PrecedenceConstraints.Add(tsk1, tsk2);
            pc.EvalOp = DTSPrecedenceEvalOp.Constraint;
            pc.Value = DTSExecResult.Success;
            pc.LogicalAnd = true; //only useful if multiple inputs coming and you want all previous steps to be Green vs Any One Green .. if its any use LogicalAnd=False

            //If you need to connect with expression use below
            //pc.EvalOp = DTSPrecedenceEvalOp.ExpressionAndConstraint;
            //pc.Expression = "@SomeVariable == true";
        }

       

     
    }
}

The post How to create SSIS Package Programmatically (Add / Execute Tasks) appeared first on ZappySys Blog.

In this section, we will discuss how to register OAuth App to call Salesforce API using Client ID / Secret rather than using your User Id / Password. Once you follow these steps below to register your Salesforce App (OAuth App), at the end you will get a Client ID (sometimes referred to as App Id) and Client Secret (or App Secret). These two pieces can be used later on to call Salesforce API using OAuth. So let's get started.

Register Salesforce App (OAuth)

1. Login into your account and go to Setup as shown below (click on gear icon at the top right corner):
2. Then in the search bar input "app" and open App Manager:
3. Then click New Connected App to create a new OAuth App: NOTE: In some version, it may say New External Client App

   Look for one of the following buttons:

   • New Connected App (Classic name)

   • New External Client App (Lightning name)

   
4. Give your App a name and: - Enable OAuth Settings - Set callback URL to following two URLs (one in each line) https://login.salesforce.com/services/oauth2/success https://zappysys.com/oauth - Add an OAuth scope you want or simply add "Full access". - Add "Perform requests on your behalf at any time (offline_token)" scope to get a refresh token so that you don't have to re-authenticate each time requesting an access token: IMPORTANT: You can use "Access and manage your data (api)" if you do not like to grant full permission to this app but make sure you request same permission from the caller app too  (Example on ZappySys Salesforce Connection UI -> Advanced Tab -> Set scope as "api offline_access" rather than "full offline_access"
5. Click "Save".

Get Client Id and Client Secret for Salesforce Connected App

Once the App is created, we need to find out the App Id / Secret so we can use it for API calls.

1. Go to App Manager
2. Find out the recently created App and click View (click on the small arrow to the right)
3. Once App page is loaded you can Copy Consumer Key and Consumer Secret (click on reveal to see). Also copy anyone Call back URL (We will need this for OAuth Connection > Advanced Tab > Call Back URL or Redirect URL property depending on which ZappySys Product / Driver you are using)
4. That's it, you can now use these 3 pieces to ZappySys OAuth connection UI or on Salesforce Connection UI (new version with OAuth support). The above screenshot shows where to enter these 3 pieces in the New Salesforce Connection. For the OAuth Connection, the redirect URL goes to the Advanced Tab > Callback URL textbox.

Get Client Id and Secret for Salesforce External Client App

If you created a new App by clicking on the New External Client App button, then your App may not be visible in the App Manager Screen. In that case, find it below way. [caption id="attachment_11637" align="alignnone" width="1222"] Get Client Id, Client Secret - For New External Client App[/caption]

Enable Client Credentials Grant (Non-Interactive Run-As)

If you like to enable Non-Interactive mode, just like UserId / Password Authentication, but without actually passing a real UserId or Password on the Connection UI, then you can enable Client Credentials Flow as below. NOTE: Only the New version of ZappySys Salesforce Connection supports Client Credentials Grant. Version released after Nov 1, 2025 supports this grant type. [caption id="attachment_11639" align="alignnone" width="419"] Enable Salesforce Client Credentials Grant (Run As)[/caption]

(OPTIONAL) Configure Salesforce OAuth refresh token validity

By default, Salesforce Refresh Token doesn't expire once you generate. If you want to change the default behavior, follow these steps.

1. Go to "Manage Connected Apps" menu item and press on your App name:
2. Click "Edit Policies" to configure refresh token validity:
3. Set "All users may self-authorize" and "Refresh token is valid until revoked" options:
4. Click "Save".

Using Salesforce Connection with OAuth App in SSIS / ODBC

Once you have Client ID / Secret, you can use it in the Salesforce Connection in SSIS Connector / ODBC Driver for Salesforce like below (Only the new version will have OAuth option)

Troubleshooting Known Issues

Let’s look at some common issues with Salesforce Connection and how to fix them.

OAuth Error: Missing Required Code Challenge

When you click Generate Token button, you may get an error like the one below.

Salesforce OAuth Generate Token Error – Missing Required Code Challenge

Solution

Here is how to fix it.

For Salesforce Connection UI: Go to Advanced Tab => Check Enable PKCE Code Verifier option
–OR–
For OAuth Connection UI: Go to OAuth2 Grant Options Tab => Check Enable Code Verifier Option

Enable PKCE code verifier option – OAuth Connection for Salesforce.

The post How to register Salesforce App and obtain Client ID / Secret (for API Call / OAuth) appeared first on ZappySys Blog.