Introduction

Sometimes you may need to make an advanced filtering on file path, file date, size etc in SSIS Amazon S3 Task or SSIS Azure Blob Storage Task . For example, you have a directory “Customers“, which contains client directories. Each of them contains a directory named “Invoices“, which is partitioned by year. E.g. “Customers/Microsoft/Invoices/2015”. So if you want to get all invoices of all customers of all years, use Advanced Filter & Sorting option and configure Where clause to use wildcards (*) like this:

SSIS Amazon/Azure Storage Task – To get all invoices of all customers of all years, use this Where clause: FullPath LIKE ‘Customers/*/Invoices/*’

Using Advanced Search – Regex / SQL Expression (Filter by file date, size, name)

Sometimes it’s necessary to filter file based on advanced attributes such as file Age (How old), File Size, Multiple File Patterns (e.g. Delete *.zip, *.gz, *.rar in one action)

You can check this link or this link for more information. For Azure Blob Storage Task its almost same UI.

You can use Advanced Tab to filter further. First supply primary filter on Source Tab (e.g. mybucket/*.* ) and then supply your advanced filter (e.g. you can exclude *.exe and *.msi file from your delete list as below screenshot, also only delete files between 1000 and 50000 bytes in file size).

Select / Delete Amazon S3 File (Multiple) using Wildcard pattern

Advanced Search in SSIS Amazon S3 Task – Use Regular Expression Patterns, SQL like expression for Sort / Filter

//////////////////////////////
Examples:
//////////////////////////////
AgeInDays < 5  //Get files created in last 5 days
LastEditInDays > 3 //Get files modified before 3 days
AgeInDays > 10 AND LastEditInDays < 3  //Get files created before 10 days and modified in last 3 days
Size < 1024  //Get files where File size is less than 1024 bytes
CreationDate BETWEEN #2015-01-01# AND #2015-01-01#  
Name IN ('File1.txt', 'Download.msi')
FullPath LIKE 'mybucket\somefolder\month_*')
Extension IN ('.txt','.csv')
LastModifiedDate > #2013-01-01T16:00:00#
IsReadOnly = True
/////////////////////////////////////////
//Supported Columns for expression:
/////////////////////////////////////////
Name, FullPath, Extension, FolderPath, AgeInDays, LastEditInDays, Size, CreationDate, CreationDateUtc, LastModifiedDate, LastModifiedDateUtc, IsReadOnly

Select Amazon S3 files older than X days

If you like to select / delete files older than certain days then use below technique.

AgeInDays > 10

1. First Specify primary search criteria on Source tab
2. Go to Advanced Filter Tab and specify Following Pattern in Where Textbox (For example delete Files older than 10 days )

The post Filter by file size, name, date for Amazon / Azure Storage SSIS Task appeared first on ZappySys Blog.