In this post, we will show how to save a list of files into a table a table using SSIS. The article will show how to get a list of files and then store the list of files into an Excel file.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from the Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Optional (If you want to Deploy and Schedule ) – Deploy and Schedule SSIS Packages

Step-by-step process to save the list of files and folders into an article

Save the information in a variable with the Advanced file system task

1. Drag and drop the Advance File System Task from the SSIS toolbox, chose to Get file list as ADO.net Data table and select an object variable type

Getting the file list in a variable

Save the result in a file with Logging task in order to save a list of files into a table

2. The next step is to save the result in a HTML file: select the file destination, uncheck append and prefix datetime. Select the file in log mode and none in message type, and insert the variable in the message as follows:

Save the value from the variable in a file without a prefix

Read the table and save it on a new file in order to save a list of files into a table

3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox

4. Double click on the DataFlow task to see DataFlow designer surface.

5. From the SSIS toolbox drag and drop HTML source and insert the path for the new file, you can preview the file if it was previously created.

Preview the HTML file

7. Then you can use a destination like the Excel destination, JSON destination or CSV destination

Save the path list into an excel file

If everything is OK, the information will be stored in Excel. However, you can select any destination.

Conclusion

In this article, we show how to save a list of files in a table. Basically, we use the Advanced file system to save the list in a file. We use the Logging Task

The post How to save a list of files into a table using SSIS. appeared first on ZappySys Blog.

In this new article, we will show you how to perform PGP encryption using SSIS (encrypt / decrypt files using public / private key). Our previous article was about SFTP using our SFTP task for SSIS. Now we will show how to encrypt the information.

Requirements

1. First of all, you will need SSDT for Business Intelligence for SSIS projects.
2. Download and install FREE PGP Command line gnupg for Windows.
3. Optionally you can install SSIS PowerPack.

What is PGP / GPG (GnuPG) / OpenPGP ?

PGP stands for Pretty Good Privacy. It was created by Phil Zimmermann. PGP is the most popular email encryption in the world. It is an Encryption that requires a public key, a private key. You can encrypt, decrypt and sign documents, emails. PGP Tools now acquired by Symantec. In sort PGP Tools developed by Phil are not Open Source any more so use other on if you really need FREE one.

GPG (i.e. GnuPG)) stands for GNU Privacy Guard. This software is for multiple platforms like Linux and Windows. In this article, we will focus on Windows because we are using SSIS that is mainly installed in Windows (at least by the moment). This tool is free, so we will focus on this one instead of trying other licensed versions.

OpenPGP is standard around Public / Private Key Cryptography (RFC4880) . Check this page. Both GPG and PGP follow OpenPGP Standard however there will be a case when they are not 100% compatible with each other.

For more information read this link and this link.

Getting started

In this article we will use GPG only which is essentially same features as PGP but GPG is Open Source around OpenPGP Standard and PGP is not open source.

So let’s get started. First of all, we will start with a simple the command line.

GPG command line example

After you install gpg command line we can open it to get started. There are many more examples explained here and here.

1. First of all, check the requirements and open the command line.
2. Secondly, in the command line write gpg –help this will list all the commands available:
   

   gpg help command

3. Also, in order to generate your personal key pair, write the following command:
   

   gpg --gen-key

4. Finally, the command previously provided will ask for a name and email, write an O to press OK and a Passphrase. You will be able to create your key pair with that information.

PGP encryption using SSIS introduction

It is possible to perform PGP encryption using SSIS with the GPG command line which is free. We can use  native execute process task that comes with SSIS. In this article, we will show some commands of gpg.

1. Open a new SSIS Package.
2. Drag Execute Process Task from SSIS Toolbox and double click to configure (see next step).
3. Using SSIS Execute Process Task, we will invoke the gpg commands. There are different ways to invoke the gpg commands. One way is to execute command stored in a .bat file or .cmd file
4. To invoke command from a file create a file called gpg.cmd and type

   gpg --version

      , save file to path like c:\sql\gpg.cmd for this demo.
5. Configure Execute Process Task something like this if you invoke command from batch file:
   

   SSIS invoke gpg

6. Also, note that the windows style is hidden. That will hide the command line when running the package.
7. In addition, the StandardOutputVariable will create an object variable in the output.
8. To verify the output, we will use the ZS Logging task
   

   SSIS output results

9. Click OK to save Execute Process Task UI.
10. Now drag ZS Logging task from toolbox and configure like below. This will Log output captured by gpg command. This step is optional but will help you a lot if you want to know what happened in previous step (e.g. capture error message)
    

    SSIS Logging task

Execute GPG / PGP commands directly in SSIS Execute Process Task (Pass Arguments)

1. In the previous example, we used a cmd file with the commands. Sometimes we need to use SSIS variables and pass information dynamically. In this new example, we will invoke the gpg command line directly and send the arguments in the arguments property like this:

SSIS execute gpg encryption

Generate public key in PGP encryption using SSIS

Now lets look at common task with PGP cryptography which is generate Public / Private Key pair.

1. In order to generate a public key, you need to run the following arguments in SSIS:
   

   SSIS generation of public key

   --armor --output "c:\sql\ssiskey.txt" --export "Name"

2. In addition, the key name is the name of the pair key created and the
   output contains the path of the stored key.

How to encrypt a file in PGP encryption using SSIS

1. First of all, we will encrypt a file named ssiskey.txt:
2. The argument used is the following:
   

   --recipient "Name" --output "c:\sql\ssiskey.txt.gpg" --encrypt "c:\sql\ssiskey.txt"

   

   Encrypt file gpg in SSIS

3. We are creating an encrypted file named ssiskey.txt.pgp

How to decrypt a file in PGP encryption using SSIS

1. In order to start, we will decrypt the file created previously:
   

   Decrypt file using PGP

2. Also, we will use the following argument:
   

   --decrypt-files "c:\sql\ssiskey.txt.gpg"

    

How to import a public key in PGP encryption using SSIS

1. In order to send a message and decrypt it, you need a public key. If you publish a public key, the recipient needs to import the public key. To import a key use the following argument:
   

   gpg --import "c:\sql\publickey.txt".

   You can download a public key here:
   • Public key
2. Secondly, in the Execute Process Task, will look like this:
   

   How to import a public key in SSIS

How to encrypt multiple files in PGP encryption using SSIS

1. If you want to encrypt multiple files, you can use the ZS Advanced File System combined with the Foreach loop container to filter and select the files to encrypt.
2. In addition, the package will look like this:
   

   Send a list of files to the foreach loop

3. Also, in the Advanced File System, we will list the files as ADO.net DataTable:
   

   Store list of files in SSIS

4. In the next step, we will use the Foreach ADO Enumerator in the Foreach Loop container:
   

   Foreach loop from the Advanced Files system

5. Next, we will select the first column (column 0) to get the file path from the list.
   

   Send column 0 that contains the full path

6. Finally, we will use the following expression for the argument property to encrypt all the txt files of the ADO list:
   

   "--recipient \"Daniel Calbimonte\" --output \""+ @[User::filepath]+".gpg\""+" 
   --encrypt \""+@[User::filepath]+"\""

   

   SSIS expression to encrypt multiple files

Troubleshooting PGP encryption using SSIS

1. First of all, if your package works in SSDT or VS and fails in the agent, verify that you are using a proxy account with permissions to run the gpg command line. For more information about SQL Agent proxies, refer to this link.

References

For more information, refer to these links:

• Practical Introduction to GNU Privacy Guard in Windows
• The GNU Privacy Handbook

The post SSIS PGP Encryption / Decryption (Using FREE GPG Tool) appeared first on ZappySys Blog.

Calling SSRS Reports in SSIS is straightforward using the ZappySys Reporting Services task. Also, we will show how to export files SSRS files using SSIS, how to send reports in emails and how to send parameters.

In addition, we will be using this ZappySys SSIS PowerPack component to make things work:

	SSIS Report Generator Task (Export SSRS Reports)
	Advanced SSIS File System Task

In order to understand the functionality, the following video tutorial can be useful to understand the functionality of this useful Reporting Services task:
[youtube https://www.youtube.com/watch?v=ajnSgbXls7I&w=560&h=315]

Requirements

1. First of all, we will need ZappySys SSIS PowerPack. This tool contains the Reporting Services Task.
2. Secondly, you will require SSDT for Business Intelligence to create SSIS projects.
3. Also, make sure to have an SSRS report deployed. For more information
   about creating reports and deploying, refer to these links:

• • Create a Basic Table Report (SSRS Tutorial)
  • Publishing Reports to a Report Server

Getting started

We will first show how to export an SSRS Report to Word format (docx). Also, the ZappySys SSIS PowerPack, allows converting your SSRS reports deployed into Word (docx and doc), Excel (xlsx and xls), PDF, TIFF, CSV and XML.

Calling SSRS Reports in SSIS

1. First of all drag and drop the ZS Reporting Task that is included with our ZappySys SSIS PowerPack:
   

   SSRS task in SSIS

2. Secondly, in general tab, make sure that the Export server report is enabled.
3. Additionally, specify the report service URL. If you are not sure what is your report service URL, check the step 3 in requirements.
4. In addition, in report path, select the report that you want to export to a file.
5. Also, check the output as a file option:
   

   Export report to file

6. Additionally, in the export settings tab, go to Save as and specify the path and extension of the file to export. You can export to Excel, Word, PDF, Tiff, CSV and XML:
   

   Export SSRS report using SSIS to Excel, Word, PDF

7. Finally, run the package and the file will be created:
   

   SSRS report exported to docx

Changing the names of the reports with the date format

In order to explain how to add the prefix or suffix of a file with the current date. For example, I want my SSRS reports with this format:
Filename20180323.xlsx where 20180323 means march 23, 2018. To understand how to do it, In this new example, we will show how to do it.

1. First of all, we will use the ZappySys Advanced File System Task:
2. Secondly, we will use the option Rename files and rename using the fun_getdate function like this: Filename<<yyyy-MM-dd,FUN_GETDATE>>.xlsx
   

   Rename the file with a date in SSIS

3. Also, you can watch all the available examples of functions by pressing the Expressions (x) button:
   

   Special SSIS functions

4. Finally, press insert variable and function examples and you will be able to see all the examples of special functions to simplify your life:
   

   Samples of functions

Moving your report files to a UNC path

Once that you have your files exported from reporting services to a file, you can move, copy them to another path. In this example, we will show how to move files to a Shared folder using a UNC path.

1.  First of all, specify your file name to move. You can use *.xml for example to move all the files.
2. Secondly, specify the UNC path in destination:
   

   SSIS move files to UNC path

3. Also, It is possible to sort files by name, size, modified date or other properties:
   

   Sort file in SSIS by name, size, modified date

4. Finally, press the Examples link, you can see some custom queries that you can create:
   

   Sample SSIS queries to files

Sending SSRS Report as an attachment in an email

The task also allows calling SSRS reports in SSIS and send the reports to an email. To do it, it is necessary to have an SMTP Server. Let’s take a look at the steps:

1. First of all, in the general tab, check the email report file and uncheck the Output as file option:
   

   Output SSRS report in SSIS as attachment

2. Also, in Email setting tab, press New in SMTP connections to create a new connection. You will need to provide an SMTP server name, an email and a password. It is also necessary to enter the port. You can find the list of some SMTP servers and ports here.
3. In addition, you have an option to specify the sender, receiver, subject, and body. It is possible to write the body in HTML format. You can also specify SSIS variables. The following HTML code shows how to write in bold,
   italic, with bullets and more:
   

   <h1>This is an example using headers</h1>
   <p><strong>This is an example to write using bold</strong> <strong><br />
   </strong><em>This is an example to write using italic <br /></em>
   This is an example using underline<br /><br />
   </p>
   <ol>
     <li>This is an example with lists</li>
     <li>Sedond of the list</li>
     <li>Third of the list</li>
   </ol>
   <ul style="list-style-type:circle">
     <li>This is an example with bullets</li>
     <li>Bullet sample </li>
     <li>Working with bullets</li>
   </ul>

    
   

   Dynamic HTML in SSIS

4. Additionally, in Visual Studio go to the menu and SSIS, Variables and create two variables that will be used in the report named Title and ReportName of type string and specify values for them:
   

   Using SSIS variables in emails

5. Finally, run the script and check your email received:
   

   Email received SSIS Html format

Calling SSRS Reports in SSIS with parameters

Another key point is the use of parameters. Sometimes your SSRS includes parameters. With our Reporting Service Task, it is possible to call an SSRS report and pass SSIS parameters. You will require a Report with a parameter to test it. If you do not know how to create a parameter in SSRS, click here.

1. First of all, you will need an SSRS Report with a parameter:
   

   Reporting Service Parameter

2. Secondly, in SSDT or Visual Studio, in the menu, go to SSIS and variables. Create an SSIS variable:
   

   Variables in SSIS

3.  Finally, in Report Services Task in Visual Studio, go to Report Parameters and map the SSRS parameter with the SSIS variable:
   

   Map SSIS variable with SSRS parameters

Calling SSRS Reports in SSIS using data-driven configuration

Data-driven configuration allows configuring different reports with different configurations. In addition, the option allows handling multiple reports using a table with all the configurations for each report. Also, this task creates a table in SQL Server with the configurations for each report. Let’s take a look to see how this works.

1. First of all, select the data-driving report export using config table.
2.  Also, you will need to specify the connection to the SQL Server
   database.
3. In addition, go to config table, press new to create a new table:
   

   SSRS Data-driven in SSIS

4. Additionally, ZappySys allows creating a sample table that can help you to understand how to configure local reports, Calling SSRS Reports in SSIS and save to files, specify paths, ports, email options and more:
   

   CREATE TABLE [{{SchemaName}}].[{{TableName}}](
   	Id int identity(1,1) NOT NULL  PRIMARY KEY, 
   	GroupName varchar(100) NULL,  /* Used to filter reports */
   	ReportType varchar(10) NOT NULL, /*Local or Server*/
   	ReportServiceUrl  varchar(500) null, 
   	ReportPath varchar(500) NOT NULL, 
   	IsEnabled bit NOT NULL default(1),
   	ExportAsFile bit NOT NULL default(1),
   	ExportAsEmail bit NOT NULL default(0),
   	Parameters varchar(max) NULL, /* ParameterName1;IsMultiValue;Value1|....ParameterNameN;IsMultiValue;ValueN */
   	ExportPath varchar(500) NULL,
   	AddFileSuffix bit NULL,
   	ExportFileSuffix varchar(500) NULL, /* e.g. _ddMMyyyy, _ddMMyyyyhhmmss, _ddMMyyyyhhmmss.fff (see this URL http://msdn.microsoft.com/en-us/library/az4se3k1(v=vs.110).aspx) */
   	Description varchar(500) NULL,
   	EmailFrom varchar(255) NULL,
   	EmailTo varchar(1000) NULL,
   	EmailCc varchar(1000) NULL,
   	EmailBcc varchar(1000) NULL,
   	EmailSubject varchar(500) NULL,	
   	EmailIsHtml bit NULL default(1),	
   	EmailBodyText varchar(max) NULL,
   	EmailAttachmentFile varchar(500) NULL, /*Unused: For future use only*/
   	LastExportStartDate datetime NULL,
   	LastExportEndDate datetime NULL,
   	LastStatus varchar(20) NULL,
   	LastMessage varchar(max) NULL,
   	Datasets varchar(max) NULL, /*if ReportType=local and report file is using dataset(s) then supply data in this column (multiple values must be pipe delimited). e.g. Dataset1|Dataset2)*/
   	DatasetConnectionStrings varchar(max) NULL, /*if ReportType=local and report file is using dataset(s) then supply data in this column (connection string must be same order as datasets (multiple values must be pipe delimited) enter just one connectionstring if all datasets using same connection. ConnectionString must be ADO.NET for SQLServer, ODBC or OLEDB connection string and correct driver must be installed on the system)*/
   	DatasetQueries varchar(max) NULL /*if ReportType=local and report file is using dataset(s) then supply data in this column (queries must be in the same order as dataset(s) listed in Datasets column. enter multiple queries in this format <xml><query>{query for 1st dataset}</query><query>{query for 2nd dataset}</query></xml>)*/
   );

    
   

   Sample Database in SSIS for SSRS

5. Finally, you can check the table with the configuration:
   

   Data-driven table created in SSIS

Calling SSRS Reports in SSIS to Export local report option

With local reports, you can work with rdl files. The rdl (Report Definition Language) files are part of the SSRS. They are physical files that contain the report format and other components. Using the export local report you can customize queries and see the results as reports. Here we will explain how to do it.

1. First of all, in the Reporting Services task in Visual Studio, make sure to select the option Export local report.
2. Secondly, in report path, you need to specify your rdl file path.
3. In addition, check the dataset. The dataset is usually detected by default. It is included in the SSRS.
4. Also, in Dataset Constr, you will need to specify the connection information. You need to specify the SQL Server name, Database name, Authentication type. For more information about connection string, click here.
5. Finally, in dataset queries, you need to specify your query using the XML and query tags:
   

   SSIS local SSRS report

Conclusion

In summary, calling SSRS Reports in SSIS is a straightforward task using the ZappySys Reporting Service task. In this article, we learned how to export a Reporting services Report to a file. Also, we learned how to send emails using SSIS with the report attached and we also learned how to rename, copy files and finally how to use Data-Driven configuration and create queries to local reports. If you liked the tool, you can download it here.

References

Finally, refer to these links for more information:

• Reporting Services Tutorial
  
• SSRS Task in SSIS

The post Calling SSRS Reports in SSIS (Export / Email) appeared first on ZappySys Blog.

In this article, we will show how to send values from an SSIS Variable to an HTML file. We will use the SSIS Advanced File System Task to store the list of system files of a folder into a variable and then we will use the SSIS Logging Task to store the variable in an HTML file.

In the second part, we will read values from an HTML table and export the data into SQL Server. We will use the HTML file created in the first example. We will also use the ZS HTML Table Source to read the data from the HTML table and export to SQL Server using the OLEDB Destination.

We will be using this ZappySys SSIS PowerPack component to make things work:

	SSIS Logging Task (Free)
	SSIS HTML Table Source
	Advanced SSIS File System Task

Finally, we will show how to read data from a SQL Server query and store into an HTML file. We will use the Execute SQL Task combined with the ZS SSIS Logging Task.

Requirements

1. First, you will need SSDT installed.
2. Secondly, ZappySys PowerPack for SSIS.
3. Finally, SQL Server installed.

Getting started

How to save the list of system files into SQL an HTML table

In the first example, we will use our ZappySys Advanced File System to copy a list of the system files in a folder. This list of files will be stored in an object variable and then we will save to a file.

1. In the first place, we will use the Advanced File System Task and select the Get file list ADO.net DataTable action. We will list the files of the c:\sql folder and save the results into a variable:
   

   SSIS list files

2. In addition, we will create a variable of type object named listfiles that will store the files information:
   

   SSIS object variable

3. Also, we will use the ZS Logging Tasks and join with the ZS Advanced File. The ZS Logging Task will be used to store the variable into a file:
   

   SSIS save variable to HTML

4. In addition, it is possible to send the variable values into the Execution Log (SSIS output), to a file, to a MessageBox:
   

   SSIS output variable to log, message box and file variable to read and write data to HTML in SSIS

5. In this example, we will show the values into an HTML file named listfiles.html
   

   SSIS save variable iHTMLhtml

6. After executing the package, you will be able to see the HTML with a list of files, names, size, modification and creation information:
   

   ssis list files

How to copy data from an HTML table to SQL Server

The next example will save data from a file named listfiles.html created in the previous example with an HTML table. The example will export the table to SQL Server. However, with our HTML Table Source, you can export to any destination.

1. In order to start, we will Drag and drop a data flow task and double click it:
   

   SSIS data flow

2. Next, in the Data Flow task, use the HTML Table Source and join to the OLEDB Destination:
   

   SSIS HTML table SQL server

3. Also, in URLs enter the path of the HTML file created in the first example:
   

   SSIS HTML table read

4. In addition, go to the OLEDB Source, specify the SQL Server name, database, and log in to it.
   In the name of table or view press new to create a new table:
   

   CREATE TABLE [dbo.listfiles] (
   [_rowNumber] int,
   [_groupName] nvarchar(255),
   [FullPath] nvarchar(255),
   [Name] nvarchar(255),
   [Size] int,
   [AgeInDays] int,
   [LastEditInDays] int,
   [LastModifiedDate] datetime,
   [CreationDate] datetime,
   [LastModifiedDateUtc] datetime,
   [CreationDateUtc] datetime,
   [Extension] nvarchar(255),
   [IsReadOnly] nvarchar(255),
   [FolderPath] nvarchar(255)
   )

   

   SSIS OLEDB create table

5. Also, go to Mappings page to map all the columns.
   

   SSIS map columns

6. Finally, run the package and you will have all the data exported.

How to export SQL Server results to an HTML in SSIS

The following example will show how to export the results of an SQL query into an HTML file.

1. In order to start, we will use the Execute SQL Task. Specify the option Full result set and create an OLEDB connection specifying the Server Name, database name and the select statement to a table:
   

   SSIS SQL query results

2. Secondly, go to the Result Set page and in the Result Name, set 0 and in Variable name, create a new variable:
   

   ssis sql result to variable

3. Also, create a variable of type object:
   

   SSIS object data type

4. Finally, use the ZS Logging Task to save the results in a File. Specify the Log file path and insert the variable:
   

   SSIS save to HTML file

Conclusion about read and write data to HTML in SSIS

To conclude, in this article, we learned how to export a list of files system properties and names to SQL Server. How to read an HTML table to SQL Server and finally how to export SQL Server results to an HTML in SSIS. There are nice tools to read and write data to HTML in SSIS

If you liked the tools, you can download them HERE.

References

• SSIS PowerPack ZappySys
• Integration Services (SSIS) Variables
• Execute SQL Task

The post How to read and write data to HTML in SSIS appeared first on ZappySys Blog.

In this short article, you will learn how to write Regular expressions in SSIS (i.e. Regex) and what tool to use to test them. You will also find helpful resources on how to write more sophisticated expressions and learn more about them. For demo purposes, we will use FREE SSIS Regex Parser Task to parse and extract the text using Regex.

You can use Regular expressions in several SSIS PowerPack connectors:

• SSIS Regex Parser Task (FREE),
• Azure Blob Storage Task,
• Advanced File System Task,
• Amazon S3 Storage Task and others.

Writing Regular Expressions in SSIS

Based on the SSIS component you use it will include the filename in filtering options or match the text: See below Syntax to write Regex in ZappySys tools. We support additional construct {{X, Y}} at the end of Regex to control two parameters. This additional construct is useful for data extracted from matching regex. If you want to Test patterns and not worry about data extraction then no need to use last {{

Occurance_Index

Group_Index_Or_Name

Syntax:

<your Regular Expression>[{{Occurance_Index|*[,Group_Index_Or_Name]}}]

Where:

Occurance_Index

Group_Index_Or_Name

How to name a group?

E.g. (\w+)@(?<domain>\w+.com)

How to use the group name in the match extract?

E.g. (\w+)@(?<domain>\w+.com){{0,domain}}

Example Input:

Let’s assume we have the following input text. We will test various Expressions.

Customer => AAA
Email => aaa@google.com
Phone => 101-222-3333
========
Customer => BBB
Email => bbb@yahoo.com
Phone => 102-222-3333
========
Customer => CCC
Email => ccc@hotmail.com
Phone => 103-222-3333
========
Customer => DDD
Email => ddd@outlook.com
Phone => 104-222-3333

Sample Regex Expressions

(.*)

More Regular Expression Examples

Regex Examples (Using SSIS Regular Expression Parser Task)

Here is an example how Regex

(.+)_(.+)_(.+){{1,2}}

Using Regular Expressions in SSIS Regex Parser Task (Extract Groups)

Using Groups / Occurrence Index

Some tasks like SSIS Regex Parser Task (FREE) supports extracting value from specific occurrence and specific part of matched pattern using special syntax at the end of your pattern (see below).

Your Regex Pattern Here{{OccuranceIndex,GroupIndex}}

Where :
OccuranceIndex is 0 based (0=extract first occurrence)
GroupIndex is 0 based (0=extract first matching group from pattern. First group is always entire text. )

See above screenshot in previous section for example.

Tools

The best tool we’ve found to write and test Regex is Regex Hero (will require IE with Silverlight if you want to use it online, in the browser):

Another great site for Regex testing is https://regex101.com (Works in any browser unlike previous one)

and few more sites as below

http://www.regexr.com/
http://www.regexlib.com/
http://www.regular-expressions.info/

Resources

Regular Expressions cheat-sheet to hang on the wall

Regular Expressions quick reference

The post Using Regular Expressions in SSIS appeared first on ZappySys Blog.

In this blog post, you will see how to remove invalid characters from XML using SSIS. We will use the search and replace feature of the Advanced File System Task.

Remove Invalid characters from XML.

The XML file specification specifies which characters can be part of XML data and which should be avoided. If you use an invalid character, the XML Parser can throw an error indicating that it found an invalid character. Here is a very useful link that explains which characters are allowed in XML and which are not.

Using SSIS and Regex to clean invalid XML Characters

To clean invalid characters, we can use the following regular expression. This basically removes any characters outside the allowed character range.

[^\x09\x0A\x0D\x20-\xD7FF\xE000-\xFFFD\x10000-x10FFFF]

The above expression is sufficient for most cases, but it is not the best solution. If you really want the most accurate expression per the XML Specs, use the expression below. It can be SLOW for large XML files.

[^\u0009\u000a\u000d\u0020-\ud7ff\ue000-\ufffd]|([\ud800-\udbff](?![\udc00-\udfff]))|((?<![\ud800-\udbff])[\udc00-\udfff])

Remove any characters

If you don’t have a range of valid characters but you know a list of invalid characters, use the expression below (without the ^). For example, if you want to remove A, C, and E; then use the below

[ACE]

--OR-- Use hex codes

[\x41\x43\x45]

You can also use a range

Step-By-Step Example

1. Download and install SSIS PowerPack from here.
2. Create a new SSIS Project.
3. Drag the ZS Advanced File System Task from the SSIS Toolbox to the control flow designer.
4. Double-click the task to edit its properties, as shown below. Enter the following expression for a Regular Expression.
   

   [^\x09\x0A\x0D\x20-\xD7FF\xE000-\xFFFD\x10000-x10FFFF]

5. Save the package and execute. After you run the task, it will remove (replace with blank) any invalid character in your XML file.

Here is the screenshot of the Task configuration.

SSIS Advanced File System Task – Search and Replace Option -Remove invalid XML characters using Regex (Regular Expression)

The post How to remove invalid characters from XML using SSIS and Regex appeared first on ZappySys Blog.

In this small blog post you will learn How to move files using SSIS Advanced File System Task and How to wait until file is unlocked using C# Script Task.

How to check if file is locked (SSIS C# Script Task)

If you want to check if file is locked in C# then below code will do the trick. But no worry if you dont know C#. ZappySys SSIS PowerPack comes with two tasks which can solve this issue without coding. You can use SSIS Advanced File System Task with Get file lock status action or Use SSIS Validation Task which has option to throw error on lock condition or you can continue by saving lock status into variable and continue without throwing error.

// Attempt to open the file exclusively. -- If you get erro means file is locked
using (FileStream fs = new FileStream(fullPath,
	FileMode.Open, FileAccess.ReadWrite,
	FileShare.None, 100))
{
	fs.ReadByte();

	// If we got this far the file is ready
	break;
}

How to wait until file is unlocked (SSIS C# Script Task)

Now lets check real example which will first check for locked file using SSIS Validation Task and then if file is locked then wait until file is unlocked (with some timeout hardcoded in script).

If specified wait time is reached then script will throw error. Once file is unlocked SSIS Advanced File System Task will copy file to target.

SSIS – How to check if file is locked. Wait until file is unlocked (C# Code – SSIS Script Task).

SSIS Validation Task -Store file lock status into variable

SSIS Validation Task -Check if file is locked. Save status to variable or throw error.

SSIS Advanced File System Task – Copy, Move, Rename, Delete multiple files

SSIS Advanced File System Task. Copy, Move, Rename, Delete multiple files

SSIS C# Script Task – Check file is locked, wait until file is unlocked

#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.IO;
#endregion

namespace ST_334a75922c6a47a5b4ac21ee8e4e5fff
{
	[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
	public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
	{
		public void Main()
		{
			// TODO: Add your code here
            try
            {
                var file = Dts.Variables["User::filePath"].Value.ToString();

                bool continueIfFileMissing = false;
                if (!System.IO.File.Exists(file))
                {
                    if (continueIfFileMissing)
                    {
                        Dts.TaskResult = (int)ScriptResults.Success;
                        return;
                    }
                    LogError("File not found: " + file);
                    Dts.TaskResult = (int)ScriptResults.Failure;
                }
                    
                WaitForFile(file, maxWaitInSec:30);
                Dts.TaskResult = (int)ScriptResults.Success;
            }
            catch (Exception ex)
            {
                LogError(ex.Message);
                Dts.TaskResult = (int)ScriptResults.Failure;
            }           
			
		}
        /// <summary>
        /// Blocks until the file is not locked any more.
        /// </summary>
        /// <param name="fullPath"></param>
        bool WaitForFile(string fullPath,int maxWaitInSec)
        {
            int numTries = 0;
            while (true)
            {
                ++numTries;
                try
                {
                    // Attempt to open the file exclusively.
                    using (FileStream fs = new FileStream(fullPath,
                        FileMode.Open, FileAccess.ReadWrite,
                        FileShare.None, 100))
                    {
                        fs.ReadByte();

                        // If we got this far the file is ready
                        break;
                    }
                }
                catch (Exception ex)
                {
                   if(numTries==1 ||  numTries % 20 ==0 )
                      LogWarning(string.Format("WaitForFile {0} failed to get an exclusive lock: {1}",fullPath, ex.Message));

                    //if (numTries > 10)
                    //{
                    //    LogWarning(string.Format(
                    //        "WaitForFile {0} giving up after 10 tries",
                    //        fullPath));
                    //    return false;
                    //}

                   if (numTries >= maxWaitInSec * 2)
                   {
                       throw new Exception("Max wait time reached for file : " + fullPath + ". Waited for " + maxWaitInSec + " seconds but lock not released");
                   }

                    // Wait for the lock to be released
                    System.Threading.Thread.Sleep(500);
                }

                
            }

            LogInformation( string.Format( "WaitForFile {0} returning true after {1} tries",fullPath, numTries));
            return true;
        }

        private void LogInformation(string msg)
        {
            bool again = false;
            Dts.Events.FireInformation(0, "ScriptTask", msg, "", 0,ref again);
        }
        
        private void LogError(string msg)
        {
            Dts.Events.FireError(0, "ScriptTask", msg, "", 0);
        }
        private void LogWarning(string msg)
        {
            Dts.Events.FireWarning(0, "ScriptTask", msg, "", 0);
        }
        #region ScriptResults declaration
        /// <summary>
        /// This enum provides a convenient shorthand within the scope of this class for setting the
        /// result of the script.
        /// 
        /// This code was generated automatically.
        /// </summary>
        enum ScriptResults
        {
            Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
            Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
        };
        #endregion

	}
}

Download Sample Package

Below sample will work only if you have SSIS PowerPack Installed. Download it from here it will take only 1 minute to install
Download SSIS 2012 – Sample Package (Process Locked file)

Conclusion

Processing and detecting locked files in SSIS can be tricky but using small C# script it can save you some headache. Download Advanced File System Task to try many options not available in native File System Task.

Other Keywords:
Check locked files in SSIS
How to check whether file is locked or not in SSIS
Detect locked file in SSIS
Wait until file is unlocked using C# script
How to handle file locking issue in SSIS using C# script

The post SSIS check file is locked and wait until file is unlocked (C# Script) appeared first on ZappySys Blog.

In this post you will learn how to use Advanced File System Task to get list of files and folders into variable. Advanced File System Task is significantly better than native SSIS File System Task

How to get list of files in SSIS

Most simplest way to get list of files in SSIS is use wildcard pattern as below screenshot. You can also use Scan Recursive option to include files inside child folders. For advanced filtering options use Filter and Sorting Tab

1. Download and install SSIS PowerPack from here
2. From toolbox of SSIS designer drag ZS Advanced File System Task
3. Double click Advanced File System task to configure it
4. Select Action as [Get file list as ADO.net DataTable]
5. Now select files you want to get using wildcard pattern (e.g. c:\ssis\file*.txt)
6. In the result variable select variable or create new variable with Object datatype. This variable will host file list in the form of ADO.net DataTable

SSIS Advanced File System Task – get file list as ADO.net Recordset into variable

Loop through files in SSIS using ForEach Loop Task (use ADO.net Recordset variable)

Variable Mappings for ForEach Loop Task – Loop through files in a folder using SSIS

Get files modified in last N days

To get files modified in last N days you can use Advanced File System Task.

1. Download and install SSIS PowerPack from here
2. From toolbox of SSIS designer drag ZS Advanced File System Task
3. Double click Advanced File System task to configure it
4. Select Action as [Get file list as ADO.net DataTable]
5. Now select files you want to get using wildcard pattern (e.g. c:\ssis\file*.txt)
6. Go to Filter and Sorting tab. In the Where Field enter expression LastEditInDays < 5 This will give you list of files modified in last 5 days. Here are some more examples.
   

   //////////////////////////////
   Examples:
   //////////////////////////////
   AgeInDays < 5  //Get files created in last 5 days
   LastEditInDays > 3 //Get files modified before 3 days
   AgeInDays > 10 AND LastEditInDays < 3  //Get files created before 10 days and modified in last 3 days
   Size < 1024  //Get files where File size is less than 1024 bytes
   CreationDate BETWEEN #2015-01-01# AND #2015-01-01#   
   Name IN ('File1.txt', 'Download.msi')
   FullPath LIKE 'c:\temp\da*')
   Extension IN ('.txt','.csv')
   LastModifiedDate > #2013-01-01T16:00:00#
   IsReadOnly = True
   /////////////////////////////////////////
   //Supported Columns for expression:
   /////////////////////////////////////////
   Name, FullPath, Extension, FolderPath, AgeInDays, LastEditInDays, Size, CreationDate, CreationDateUtc, LastModifiedDate, LastModifiedDateUtc, IsReadOnly

   

   SSIS Advanced File System Task – Get files modified in last N days

Get files with specific size

To get files with specific size you can use Where Expression on Size attribute (Size is in Bytes)

1. Download and install SSIS PowerPack from here
2. From toolbox of SSIS designer drag ZS Advanced File System Task
3. Double click Advanced File System task to configure it
4. Select Action as [Get file list as ADO.net DataTable]
5. Now select files you want to get using wildcard pattern (e.g. c:\ssis\file*.txt)
6. Go to Filter and Sorting tab. In the Where Field enter expression Size < 1024 This will give you list of files which are less than 1024 bytes (e.g 1KB).
   

   Size < 1024

Get files with Include/Exclude pattern (Use Regular Expression)

Sometimes you have need to list files with specific name patterns which is hard to get using simple wildcard then use Advanced Regular Expression option. For this use wildcard in Path (e.g. c:\ssis\*.*) and then in the Filter and Sorting tab use Include / Exclude Regular Expressions (e.g. RegX) for file name pattern.

For Example if you want to include all files containing *MMDD.* pattern but not *.zip file or *.gz then use below expressions

Include ==>  ^w+_\d\d\d\d.\w+$   
Exclude ==>  \.zip|\.gz

How to get list of folders in SSIS (list directories)

Coming soon

Loop through files or folders in SSIS using ForEachLoop Task

Coming soon

Read Recordset and load into database (e.g. SQL Server Table)

Coming soon

Conclusion

Microsoft native file system task has very limited features and many scenarios require you to write C# / VB.net Script. But using ZS Advanced File System Task you can eliminate need for scripting and use simple drag and drop approach. To use this task Download SSIS PowerPack

The post Get list of files in SSIS for Looping appeared first on ZappySys Blog.

Many times you have requirement to search and replace in files (single or multiple file) content using SSIS. If you are not C# or VB.net programmer then you may find yourself at dead end struggling how to achieve this in SSIS. If you have this need then you can use Advanced File System Task

Search and replace in files using Simple text search

In this option you can specify text you want to search and text you want to replace with. You can check ignore text if you dont want case-sensitive search/replace.

Extended text search (special characters allowed)

For extended search mode you may use any of the following in your search/replace string

• \n – for line feed
• \r – for return character
• \0 – for null character
• \t – for tab character
• \v – for vertical tab character

Example:
Search: “Hello\r\nWorld”
Replace: “Hello World”

Regular expression search/replace

Search and Replace File Content in SSIS (Find sub string, Search for special character)

Advanced File System Task

If you want to perform some other operations then here is the list of few features supported by Advanced File System Task.

• File operations for single or multiple files using wildcard(e.g. copy, move, rename, delete, write to file, get properties)
• Folder operations such as delete folder and create folder
• Support for bulk rename (e.g. by specifying {%name%}.{%ext%}.old as new name will rename all selected files to *.old)
• Search and Replace in files (Regular expression search supported)
• Change file encoding
• Get file properties as data table which you can loop easily (Using ForEach Loop – ADO Enumerator)
• Option to continue on error (e.g. Copy file only when source file found)
• Auto create missing target directory
• Use placeholders anywhere in the path, makes it easy to timestamp folders or files (e.g. c:\Archive\{{System::StartTime,yyyMMdd}})
• Regular Expression based Include Pattern (e.g. ^(file\d\d).(\w+)$ will include file01.exe,file22.csv …)
• Regular Expression based Exclude Pattern (e.g. (.exe|.msi) will exclude files with extension exe or msi)
• Sort based on specific property (e.g. Order Size DESC – extract top 10 files by size)
• Advanced SQL like filter support (e.g. Size> 1024 AND AgeInDays > 10 AND Extension IN(‘.txt’,’.csv’) will include file greater than 1024 bytes and extension is txt or csv)
• Support for Regular Expression based Include Pattern (e.g. ^(Customer\d\d).(\w+)$ )
• Support for SQL Server 2005, 2008, 2012, 2014, 2016 (32 bit and 64 bit)

Download FREE SSIS Components

Conclusion

Search and replace in many files can be tricky but using Advanced File System Task it can be few click. Click here  to download.

The post Search and replace in files using SSIS – No Coding appeared first on ZappySys Blog.

Many times during your ETL process you receive files which are in different encoding than you expect. Example all files are in UTF-16 format but your application expect them to be in UTF-8. Or Sometimes files are in ASCII format and you want to convert to UTF-8.

In this post you will see how to use Advanced File System Task to convert multiple files into desired encoding (e.g. ASCII, UTF-8, UTF-16).

Steps to Convert file encoding for Multiple Files in SSIS

1. Download and Install SSIS PowerPack
2. From you SSIS Toolbox drag Advanced File System Task
3. Double click task to configure it
4. Select “Change Encoding of files” action from dropdown as below screenshot
5. Enter path or pattern for source file you want to convert to different encoding
6. Select desired encoding from Encoding dropdown (e.g. if you files are in ASCII and you want to convert them to UTF-8 .. just select UTF-8)
7. Execute package and observe SSIS execution log to find out which files are converted to new encoding.

Change file encoding using SSIS Advanced File System Task

Change file encoding to UTF-8 from ASCII using SSIS Advanced File System Task

SSIS Execution log – Convert files to UTF-8 using SSIS Advanced File System Task

The post How to change file encoding in SSIS (UTF8, ASCII or UTF16) appeared first on ZappySys Blog.