In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate SharePoint Online into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful SharePoint Online Connector that makes uploading files to SharePoint Online a breeze. In this guide, we’ll walk you through the process of uploading files to SharePoint Online using SSIS and the ZappySys SSIS PowerPack SharePoint Online Connector.

Upload files to SharePoint Online with SSIS:

In this section, we will learn how to configure and use SharePoint Online Connector in the API Source to upload files on SharePoint Online.

1. Begin by opening Visual Studio and creating a new project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom
   

   SSIS Toolbox – ZappySys Tasks

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the SharePoint Online Connector

1. Use a preinstalled SharePoint Online Connector from the Popular Connector List or press the Search Online radio button to download SharePoint Online Connector. Once downloaded simply use it in the configuration:
   

   SSIS ZS API Connection Manager – Select SharePoint Online Connector

2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to SharePoint Online using User Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS ZS API Connection Manager – Generate Token – SharePoint Online with User Credentials

   Method2: Connect to SharePoint Online using Application Credentials [OAuth]

   You may press a link, Steps to Configure, to help set specific parameters.

   

   SSIS API Source – Generate Token – SharePoint Online with Application Credentials

How to upload a local file to SharePoint Online

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to upload a local file to SharePoint Online.

1. Choose the Upload file endpoint from the “Select Table/Endpoint” dropdown menu.
2. Select the desired Drive ID from the DriveId drop-down.
3. Specify the desired source file path for the file being uploaded by setting the DiskFilePath parameter.
4. Specify the desired Target Path by setting the SharePointFilePath parameter.

Please refer to the sample screenshot below.

SSIS API Source – SharePoint Online – Upload Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to SharePoint Online using the ZappySys SharePoint Online Connector for uploading a local file to SharePoint Online.

How to upload multiple files to SharePoint Online

In the API Source Component, we’ve selected the API Connection Manager. Next, we need to configure the remaining properties of the component to upload multiple files to SharePoint Online. Additionally, we should utilize a Foreach Loop Container to iterate through each file one by one in a loop.

1. Create 3 SSIS variables as described below.
   

   Variable Name	Datatype	Value
   varFileLists	Object	System.Object
   varFileName	String	fake.extension
   varFullFilePath	String	C:/fake/fake.extension

2. Go to Control Flow, From the SSIS toolbox drag and drop ZS Advanced File System Task on the control flow designer surface, and double click on it to edit it:
   
3. Select the “Get file list as ADO.NET Data Table” action from the Action drop-down menu. Set the file path in the Path textbox. Choose the variable (Object type) where you want to store the file list, and then click the OK button to save the settings.
   

   SSIS Advanced File System Task – Get file list as ADO.net Data Table

4. Now, drag and drop the “Foreach Loop Container” from the SSIS Toolbox and connect it with the “ZS Advanced File System Task.” Double-click on the “Foreach Loop Container” to edit its properties.
   

   SSIS Foreach Loop Container – Drag & Drop

5. In the Foreach Loop Editor’s “Collection” tab, choose the Enumerator as “Foreach ADO Enumerator.” In the “ADO object source variable” field, select your SSIS object-typed variable. Select the “Rows in the first table” radio option.
   

   SSIS Foreach Loop Editor – Collection – Select Variable & Enumeration mode

6. In the Foreach Loop Editor’s “Variable Mappings” tab, select the variables that you want to map to the values from the collection, and then click the “OK” button to save your selections.
   

   SSIS Foreach Loop Editor – Variable Mappings

7. Inside the Foreach Loop Container, drag and drop the Data Flow Task, and then rename it according to your preference.
   

   SSIS Data Flow Task – Drag & Drop inside Foreach Loop Container

8. Double-click on the Data Flow Task to see the Data Flow Designer, From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
9. Double-click on the API Source component to configure it for uploading multiple files to SharePoint Online.
   1. Choose the Upload file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. Please specify the desired SSIS Variable [varFileName] containing the uploaded file name by setting the SharePointFilePath parameter.
   3. To specify the desired local SSIS variable [varFullFilePath], which holds the file path, set the DiskFilePath parameter.
      When configuring the SSIS variable directly in the Raw Edit, JSON may become invalid due to the requirement of slashes within the local file path. To address this, we use a JSONENC Placeholder Function for proper escaping.
      

      <<{{User::varUserFullFilePath}},FUN_JSONENC>>

   4. If you wish to upload the local file to a specific folder in SharePoint Online, please set the folderName in the SharePointFilePath parameter.

   Please refer to the sample screenshot below.

   

   SSIS API Source – SharePoint Online – Upload Multiple files configuration

   To configure dynamic filenames and file paths, you can set variables. Click on “Raw Edit,” and in the “Edit Parameters” popup, you can use SSIS variables, as shown in the above image.

   Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call.
   After that, you can execute the SSIS Package and check the execution to upload multiple files on SharePoint Online. Below is the sample screenshot attached.

   

   Multiple files uploaded on SharePoint Online using SSIS

   With just a few clicks, we have successfully configured the call to SharePoint Online using the ZappySys SharePoint Online Connector for uploading multiple files to SharePoint Online.

Conclusion

In this blog, we learned how to connect with SharePoint Online using SSIS API Connection Manager and upload single or multiple files to SharePoint Online using a combination of SSIS components, including the SSIS API Source component, SSIS ZS Advanced File System Task, Foreach Loop Container, and SSIS variables.

The post How to upload files on SharePoint Online using SSIS appeared first on ZappySys Blog.

In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate OneDrive into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful OneDrive Connector that makes uploading files to OneDrive a breeze. In this guide, we’ll walk you through the process of uploading files to OneDrive using SSIS and the ZappySys SSIS PowerPack OneDrive Connector.

Upload files to OneDrive with SSIS:

In this section, we will learn how to configure and use OneDrive Connector in the API Source to upload files on OneDrive.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom
   

   SSIS Toolbox – ZappySys Tasks

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the OneDrive Connector

1. Use a preinstalled OneDrive Connector from the Popular Connector List or press the Search Online radio button to download OneDrive Connector. Once downloaded simply use it in the configuration:
   
2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to OneDrive using User Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – OneDrive with User Credentials

   Method2: Connect to OneDrive using Application Credentials [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – OneDrive with Application Credentials

How to upload a local file to OneDrive

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to upload a local file to OneDrive.

1. Choose the Upload file endpoint from the “Select Table/Endpoint” dropdown menu.
2. Select the desired Drive ID from the DriveId drop-down.
3. Specify the desired source file path for the file being uploaded by setting the SourceFilePath parameter.
4. Specify the desired Target Path by setting the TargetFilePath parameter.

Please refer to the sample screenshot below.

SSIS API Source – OneDrive – Upload Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to OneDrive using the ZappySys OneDrive Connector for uploading a local file to OneDrive.

How to upload multiple files to OneDrive

In the API Source Component, we’ve selected the API Connection Manager. Next, we need to configure the remaining properties of the component to upload multiple files to OneDrive. Additionally, we should utilize a Foreach Loop Container to iterate through each file one by one in a loop.

1. Create 3 SSIS variables as described below.
   

   Variable Name	Datatype	Value
   varFileLists	Object	System.Object
   varFileName	String	fake.extension
   varFullFilePath	String	C:/fake/fake.extension

2. Go to Control Flow, From the SSIS toolbox drag and drop ZS Advanced File System Task on the control flow designer surface, and double click on it to edit it:
   
3. Select the “Get file list as ADO.NET Data Table” action from the Action drop-down menu. Set the file path in the Path textbox. Choose the variable (Object type) where you want to store the file list, and then click the OK button to save the settings.
   

   SSIS Advanced File System Task – Get file list as ADO.net Data Table

4. Now, drag and drop the “Foreach Loop Container” from the SSIS Toolbox and connect it with the “ZS Advanced File System Task.” Double-click on the “Foreach Loop Container” to edit its properties.
   

   SSIS Foreach Loop Container – Drag & Drop

5. In the Foreach Loop Editor’s “Collection” tab, choose the Enumerator as “Foreach ADO Enumerator.” In the “ADO object source variable” field, select your SSIS object-typed variable. Select the “Rows in the first table” radio option.
   

   SSIS Foreach Loop Editor – Collection – Select Variable & Enumeration mode

6. In the Foreach Loop Editor’s “Variable Mappings” tab, select the variables that you want to map to the values from the collection, and then click the “OK” button to save your selections.
   

   SSIS Foreach Loop Editor – Variable Mappings

7. Inside the Foreach Loop Container, drag and drop the Data Flow Task, and then rename it according to your preference.
   

   SSIS Data Flow Task – Drag & Drop inside Foreach Loop Container

8. Double-click on the Data Flow Task to see the Data Flow Designer, From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
9. Double-click on the API Source component to configure it for uploading multiple files to OneDrive.
   1. Choose the Upload file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. Please specify the desired SSIS Variable [varFileName] containing the uploaded file name by setting the TargetFilePath parameter.
   3. To specify the desired local SSIS variable [varFullFilePath], which holds the file path, set the SourceFilePath parameter.
      When configuring the SSIS variable directly in the Raw Edit, JSON may become invalid due to the requirement of slashes within the local file path. To address this, we use a JSONENC Placeholder Function for proper escaping.
      

      <<{{User::varUserFullFilePath}},FUN_JSONENC>>

   4. If you wish to upload the local file to a specific folder in OneDrive, please set the folderName in the TargetFilePath parameter.

   Please refer to the sample screenshot below.

   

   SSIS API Source – OneDrive – Upload Multiple files configuration

   To configure dynamic filenames and file paths, you can set variables. Click on “Raw Edit,” and in the “Edit Parameters” popup, you can use SSIS variables, as shown in the above image.

   Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call.
   After that, you can execute the SSIS Package and check the execution to upload multiple files on OneDrive. Below is the sample screenshot attached.
   
   Multiple files uploaded on OneDrive using SSIS

   With just a few clicks, we have successfully configured the call to OneDrive using the ZappySys OneDrive Connector for uploading multiple files to OneDrive.

Conclusion

In this blog, we learned how to connect with OneDrive using SSIS API Connection Manager and upload single or multiple files to OneDrive using a combination of SSIS components, including the SSIS API Source component, SSIS ZS Advanced File System Task, Foreach Loop Container, and SSIS variables.

The post How to upload files on OneDrive using SSIS appeared first on ZappySys Blog.

In today’s data-driven world, the ability to seamlessly integrate and transfer files between different platforms is crucial for efficient business operations. If you’re a user of SQL Server Integration Services (SSIS) and looking to integrate Google Drive into your data workflow, you’re in luck. ZappySys SSIS PowerPack offers a powerful Google Drive Connector that makes uploading files to Google Drive a breeze. In this guide, we’ll walk you through the process of uploading files to Google Drive using SSIS and the ZappySys SSIS PowerPack Google Drive Connector.

Upload files to Google Drive with SSIS:

In this section, we will learn how to configure and use Google Drive Connector in the API Source to upload files on Google Drive.

1. Begin with opening Visual Studio and Creating a New Project.
2. Select Integration Service Project and in the new project window set the appropriate name and location for the project. Click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on the left sidebar
   2. Solution Explorer and Property Window on the right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window at the bottom
   

   SSIS Toolbox – ZappySys Tasks

   NOTE: If you don’t see ZappySys SSIS PowerPack Task or Components in the SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop the SSIS Data Flow Task from the SSIS Toolbox. Double-click on the Data Flow Task to see the Data Flow Designer.
   
5. From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double-click on it to edit it:
   
6. Select New Connection to create a new connection:
   

Configuring the Google Drive Connector

1. Use a preinstalled Google Drive Connector from the Popular Connector List or press the Search Online radio button to download Google Drive Connector. Once downloaded simply use it in the configuration:
   
2. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed.
3. After configuring all the required properties in the API Connection Manager, please click on the “Test Connection” button to confirm that the connection is established successfully.
   

   Method1: Connect to Google Drive using User Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with User Account

   Method2: Connect to Google Drive using Service Account [OAuth]

   You may press a link Steps to Configure which will help set certain parameters.

   

   SSIS API Source – Generate Token – Google Drive with Service Account

How to upload a local file to Google Drive

In the API Source Component, we’ve chosen the API Connection Manager. Next, we need to configure the remaining properties of the component to upload a local file to Google Drive.

1. Choose the Upload a file endpoint from the “Select Table/Endpoint” dropdown menu.
2. To access detailed API documentation for uploading files, you can click on the following link: Help for: Upload File.
3. Please specify the desired filename for the uploaded file by setting the FileName parameter.
4. Specify the desired local file path by setting the DiskFilePath parameter.
5. If you wish to upload the local file to a specific folder in Google Drive, please choose the folder ID from the dropdown menu labeled ParentFolderId. Otherwise, you can skip this parameter as it’s optional.

Please refer to the sample screenshot below.

SSIS API Source – Google Drive – Upload Single File

Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for uploading a local file to Google Drive.

How to upload multiple files to Google Drive

In the API Source Component, we’ve selected the API Connection Manager. Next, we need to configure the remaining properties of the component to upload multiple files to Google Drive. Additionally, we should utilize a Foreach Loop Container to iterate through each file one by one in a loop.

1. Create 3 SSIS variables as described below.
   

   Variable Name	Datatype	Value
   varFileLists	Object	System.Object
   varFileName	String	fake.extension
   varFullFilePath	String	C:/fake/fake.extension

2. Go to Control Flow, From the SSIS toolbox drag and drop ZS Advanced File System Task on the control flow designer surface, and double click on it to edit it:
   
3. Select the “Get file list as ADO.NET Data Table” action from the Action drop-down menu. Set the file path in the Path textbox. Choose the variable (Object type) where you want to store the file list, and then click the OK button to save the settings.
   

   SSIS Advanced File System Task – Get file list as ADO.net Data Table

4. Now, drag and drop the “Foreach Loop Container” from the SSIS Toolbox and connect it with the “ZS Advanced File System Task.” Double-click on the “Foreach Loop Container” to edit its properties.
   

   SSIS Foreach Loop Container – Drag & Drop

5. In the Foreach Loop Editor’s “Collection” tab, choose the Enumerator as “Foreach ADO Enumerator.” In the “ADO object source variable” field, select your SSIS object-typed variable. Select the “Rows in the first table” radio option.
   

   SSIS Foreach Loop Editor – Collection – Select Variable & Enumeration mode

6. In the Foreach Loop Editor’s “Variable Mappings” tab, select the variables that you want to map to the values from the collection, and then click the “OK” button to save your selections.
   

   SSIS Foreach Loop Editor – Variable Mappings

7. Inside the Foreach Loop Container, drag and drop the Data Flow Task, and then rename it according to your preference.
   

   SSIS Data Flow Task – Drag & Drop inside Foreach Loop Container

8. Double-click on the Data Flow Task to see the Data Flow Designer, From the SSIS toolbox drag and drop API Source (Predefined Templates) on the data flow designer surface, and double click on it to edit it:
   
9. Double-click on the API Source component to configure it for uploading multiple files to Google Drive.
   1. Choose the Upload a file endpoint from the “Select Table/Endpoint” dropdown menu.
   2. To access detailed API documentation for uploading files, you can click on the following link: Help for: Upload File.
   3. Please specify the desired SSIS Variable [varFileName] which contains file name for the uploaded file by setting the FileName parameter.
   4. To specify the desired local SSIS variable [varFullFilePath], which holds the file path, set the DiskFilePath parameter.
      When configuring the SSIS variable directly in the Raw Edit, JSON may become invalid due to the requirement of slashes within the local file path. To address this, we use a JSONENC Placeholder Function for proper escaping.
      

      <<{{User::varUserFullFilePath}},FUN_JSONENC>>

   5. If you wish to upload the local file to a specific folder in Google Drive, please choose the folder ID from the dropdown menu labeled ParentFolderId. Otherwise, you can skip this parameter as it’s optional.

   Please refer to the sample screenshot below.

   

   SSIS API Source – Google Drive – Upload Multiple files configuration

   To configure dynamic filenames and file paths, you can set variables. Click on “Raw Edit,” and in the “Edit Parameters” popup, you can use SSIS variables, as shown in the image below.

   Once you have configured all the mentioned properties above, please click on the “Preview Data” button to initiate the API call. With just a few clicks, we have successfully configured the call to Google Drive using the ZappySys Google Drive Connector for uploading multiple files to Google Drive.

Conclusion

In this blog, we learned how to connect with Google Drive using SSIS API Connection Manager and upload single or multiple files to Google Drive using a combination of SSIS components, including the SSIS API Source component, SSIS ZS Advanced File System Task, Foreach Loop Container, and SSIS variables.

The post How to upload files on Google Drive using SSIS appeared first on ZappySys Blog.

In this post, we will show how to save a list of files into a table a table using SSIS. The article will show how to get a list of files and then store the list of files into an Excel file.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from the Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Optional (If you want to Deploy and Schedule ) – Deploy and Schedule SSIS Packages

Step-by-step process to save the list of files and folders into an article

Save the information in a variable with the Advanced file system task

1. Drag and drop the Advance File System Task from the SSIS toolbox, chose to Get file list as ADO.net Data table and select an object variable type

Getting the file list in a variable

Save the result in a file with Logging task in order to save a list of files into a table

2. The next step is to save the result in a HTML file: select the file destination, uncheck append and prefix datetime. Select the file in log mode and none in message type, and insert the variable in the message as follows:

Save the value from the variable in a file without a prefix

Read the table and save it on a new file in order to save a list of files into a table

3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox

4. Double click on the DataFlow task to see DataFlow designer surface.

5. From the SSIS toolbox drag and drop HTML source and insert the path for the new file, you can preview the file if it was previously created.

Preview the HTML file

7. Then you can use a destination like the Excel destination, JSON destination or CSV destination

Save the path list into an excel file

If everything is OK, the information will be stored in Excel. However, you can select any destination.

Conclusion

In this article, we show how to save a list of files in a table. Basically, we use the Advanced file system to save the list in a file. We use the Logging Task

The post How to save a list of files into a table using SSIS. appeared first on ZappySys Blog.

In this blog post, you will see how to remove invalid characters from XML using SSIS. We will use the search and replace feature of the Advanced File System Task.

Remove Invalid characters from XML.

The XML file specification specifies which characters can be part of XML data and which should be avoided. If you use an invalid character, the XML Parser can throw an error indicating that it found an invalid character. Here is a very useful link that explains which characters are allowed in XML and which are not.

Using SSIS and Regex to clean invalid XML Characters

To clean invalid characters, we can use the following regular expression. This basically removes any characters outside the allowed character range.

[^\x09\x0A\x0D\x20-\xD7FF\xE000-\xFFFD\x10000-x10FFFF]

The above expression is sufficient for most cases, but it is not the best solution. If you really want the most accurate expression per the XML Specs, use the expression below. It can be SLOW for large XML files.

[^\u0009\u000a\u000d\u0020-\ud7ff\ue000-\ufffd]|([\ud800-\udbff](?![\udc00-\udfff]))|((?<![\ud800-\udbff])[\udc00-\udfff])

Remove any characters

If you don’t have a range of valid characters but you know a list of invalid characters, use the expression below (without the ^). For example, if you want to remove A, C, and E; then use the below

[ACE]

--OR-- Use hex codes

[\x41\x43\x45]

You can also use a range

Step-By-Step Example

1. Download and install SSIS PowerPack from here.
2. Create a new SSIS Project.
3. Drag the ZS Advanced File System Task from the SSIS Toolbox to the control flow designer.
4. Double-click the task to edit its properties, as shown below. Enter the following expression for a Regular Expression.
   

   [^\x09\x0A\x0D\x20-\xD7FF\xE000-\xFFFD\x10000-x10FFFF]

5. Save the package and execute. After you run the task, it will remove (replace with blank) any invalid character in your XML file.

Here is the screenshot of the Task configuration.

SSIS Advanced File System Task – Search and Replace Option -Remove invalid XML characters using Regex (Regular Expression)

The post How to remove invalid characters from XML using SSIS and Regex appeared first on ZappySys Blog.

In this post you will learn how to use FREE SSIS Logging Task to perform ssis base64 encode decode and save to Variable or File  (i.e. convert Base64 Encoded value to plain text or encode plain text to base64) . You can also use many other format specifiers found here

How to invoke Placeholder Functions

There are 3 ways you can use placeholder functions in many tasks with Variable Icon next to the property.

1. User Variable Placeholder
2. Static String Placeholder
3. Mixed Placeholder (Static String with Variable)

Examples: 

{{User::var1,FUN_BASE64ENC}} 

<<some direct string,FUN_BASE64ENC>>

–OR–

<<{{User::var1}} and {{User::var2}} ,FUN_BASE64ENC>>

Encode Plain text to Base64 string (SSIS base64 encode decode)

Many times you have requirement to convert plain text into Base64 encoded string (This is very common when you have to transfer data over the web – set query string / http headers / body ). Typically to perform this you may have to write C# Script if you are SSIS developer but below is very simple way to do without writing any code. Here is the most common use case of Base64 encoded string

1. Download FREE Tasks/Components of SSIS PowerPack
2. Create new SSIS Package
3. Drag ZS Logging Task from Toolbox
4. Create SSIS variable which holds plain text value (Lets call it varPlainText. Set some initial value which you want to encode)
5. Configure SSIS Logging Task as below
   

   SSIS Base64 Encode, Decode using SSIS Logging Task (Without coding convert plain text to base 64 string)

6. Run it and you will see new file created with encoded value. You can also save encoded value to Variable if you change Logging Mode = Variable rather than File

Convert Base64 Encoded value to plain text (Decode)

To convert Base64 encoded string into plaintext you can perform similar steps described in the above section except you need to use BASE64DEC format specifier rather than BASE64ENC

Example:

<<QUJDREVGMTIzIyM=,FUN_BASE64DEC>>
--OR--
{{User::MySSISVar1,FUN_BASE64DEC}}

How to decode Base64 file in SSIS and save to new file

If you have a file coming as a base64 encoded string and you want to decode it and save it to a new file then it’s simple too. Basically, you will need SSIS Advanced File System Task  and SSIS Logging Task to achieve this (see package below).

SSIS decode base64 file – Read content convert to plain text and save to new file

Here are your steps.

1. Use SSIS Advanced File System Task and use read file content option to save file content into SSIS Variable
2. Configure Advanced File System as below.
   

   Read file content into SSIS variable using SSIS Advanced File System task

3. Next step connect SSIS Logging Task
4. Configure SSIS Logging Task as per below screenshot.
   

   Decode base64 file content in SSIS – Use Logging Task

Convert Base64 Encoded string of local file content (Binary or Text)

Using FUN_FILE_BASE64ENC we can encode the image or any other local file(Binary or Text) to the Base64 string.

FUN_FILE_BASE64ENC

Syntax:

<<file_path,FUN_FILE_BASE64ENC>>

Returns BASE64 encoded string of local file content (binary or text).

Parameters:

Examples:

<<c:\someimage.png,FUN_FILE_BASE64ENC>>
--OR--
{{User::MySSISVar1,FUN_FILE_BASE64ENC}}

Sample Output:

This returns base64 encoded string of binary file (png)

bXl1c2V..........Ac3NXb3Jk

The post SSIS Base64 Encode Decode without Coding appeared first on ZappySys Blog.

In this small blog post you will learn How to move files using SSIS Advanced File System Task and How to wait until file is unlocked using C# Script Task.

How to check if file is locked (SSIS C# Script Task)

If you want to check if file is locked in C# then below code will do the trick. But no worry if you dont know C#. ZappySys SSIS PowerPack comes with two tasks which can solve this issue without coding. You can use SSIS Advanced File System Task with Get file lock status action or Use SSIS Validation Task which has option to throw error on lock condition or you can continue by saving lock status into variable and continue without throwing error.

// Attempt to open the file exclusively. -- If you get erro means file is locked
using (FileStream fs = new FileStream(fullPath,
	FileMode.Open, FileAccess.ReadWrite,
	FileShare.None, 100))
{
	fs.ReadByte();

	// If we got this far the file is ready
	break;
}

How to wait until file is unlocked (SSIS C# Script Task)

Now lets check real example which will first check for locked file using SSIS Validation Task and then if file is locked then wait until file is unlocked (with some timeout hardcoded in script).

If specified wait time is reached then script will throw error. Once file is unlocked SSIS Advanced File System Task will copy file to target.

SSIS – How to check if file is locked. Wait until file is unlocked (C# Code – SSIS Script Task).

SSIS Validation Task -Store file lock status into variable

SSIS Validation Task -Check if file is locked. Save status to variable or throw error.

SSIS Advanced File System Task – Copy, Move, Rename, Delete multiple files

SSIS Advanced File System Task. Copy, Move, Rename, Delete multiple files

SSIS C# Script Task – Check file is locked, wait until file is unlocked

#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.IO;
#endregion

namespace ST_334a75922c6a47a5b4ac21ee8e4e5fff
{
	[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
	public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
	{
		public void Main()
		{
			// TODO: Add your code here
            try
            {
                var file = Dts.Variables["User::filePath"].Value.ToString();

                bool continueIfFileMissing = false;
                if (!System.IO.File.Exists(file))
                {
                    if (continueIfFileMissing)
                    {
                        Dts.TaskResult = (int)ScriptResults.Success;
                        return;
                    }
                    LogError("File not found: " + file);
                    Dts.TaskResult = (int)ScriptResults.Failure;
                }
                    
                WaitForFile(file, maxWaitInSec:30);
                Dts.TaskResult = (int)ScriptResults.Success;
            }
            catch (Exception ex)
            {
                LogError(ex.Message);
                Dts.TaskResult = (int)ScriptResults.Failure;
            }           
			
		}
        /// <summary>
        /// Blocks until the file is not locked any more.
        /// </summary>
        /// <param name="fullPath"></param>
        bool WaitForFile(string fullPath,int maxWaitInSec)
        {
            int numTries = 0;
            while (true)
            {
                ++numTries;
                try
                {
                    // Attempt to open the file exclusively.
                    using (FileStream fs = new FileStream(fullPath,
                        FileMode.Open, FileAccess.ReadWrite,
                        FileShare.None, 100))
                    {
                        fs.ReadByte();

                        // If we got this far the file is ready
                        break;
                    }
                }
                catch (Exception ex)
                {
                   if(numTries==1 ||  numTries % 20 ==0 )
                      LogWarning(string.Format("WaitForFile {0} failed to get an exclusive lock: {1}",fullPath, ex.Message));

                    //if (numTries > 10)
                    //{
                    //    LogWarning(string.Format(
                    //        "WaitForFile {0} giving up after 10 tries",
                    //        fullPath));
                    //    return false;
                    //}

                   if (numTries >= maxWaitInSec * 2)
                   {
                       throw new Exception("Max wait time reached for file : " + fullPath + ". Waited for " + maxWaitInSec + " seconds but lock not released");
                   }

                    // Wait for the lock to be released
                    System.Threading.Thread.Sleep(500);
                }

                
            }

            LogInformation( string.Format( "WaitForFile {0} returning true after {1} tries",fullPath, numTries));
            return true;
        }

        private void LogInformation(string msg)
        {
            bool again = false;
            Dts.Events.FireInformation(0, "ScriptTask", msg, "", 0,ref again);
        }
        
        private void LogError(string msg)
        {
            Dts.Events.FireError(0, "ScriptTask", msg, "", 0);
        }
        private void LogWarning(string msg)
        {
            Dts.Events.FireWarning(0, "ScriptTask", msg, "", 0);
        }
        #region ScriptResults declaration
        /// <summary>
        /// This enum provides a convenient shorthand within the scope of this class for setting the
        /// result of the script.
        /// 
        /// This code was generated automatically.
        /// </summary>
        enum ScriptResults
        {
            Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
            Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
        };
        #endregion

	}
}

Download Sample Package

Below sample will work only if you have SSIS PowerPack Installed. Download it from here it will take only 1 minute to install
Download SSIS 2012 – Sample Package (Process Locked file)

Conclusion

Processing and detecting locked files in SSIS can be tricky but using small C# script it can save you some headache. Download Advanced File System Task to try many options not available in native File System Task.

Other Keywords:
Check locked files in SSIS
How to check whether file is locked or not in SSIS
Detect locked file in SSIS
Wait until file is unlocked using C# script
How to handle file locking issue in SSIS using C# script

The post SSIS check file is locked and wait until file is unlocked (C# Script) appeared first on ZappySys Blog.