This time we will explain how to get all URLS from emails using MS Outlook. Sometimes we need get all URLs from emails. In this post, we will show how to do this. Step by step using SSIS.

Microsoft Graph API is a unified way to access many Microsoft services API including Office 365 API

Prerequisites

Before we perform steps listed in this article, you will need to make sure following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Optional (If you want to Deploy and Schedule ) – Deploy and Schedule SSIS Packages

Steps-by-step process to download images from HTML using SSIS

Register Application (OAuth2 App for Graph API)

First, check our article about how to get the OAuth2 connection, click here for the article

Get the information for the body content from the emails

1. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.

Drag and drop Data flow task

2. Double click on the DataFlow task to see the DataFlow designer surface.

3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.

Drag and drop a JSON sourcec

4. Select the OAuth connection you created, then use this URL to get the body content

https://graph.microsoft.com/v1.0/me/messages?$select=subject,body,bodyPreview,uniqueBody

Getting the information from the emails

5. Drag and drop the trash destination and save the result in a file with the following configuration. Make sure to check the Overwrite target file if exists.

Save the body from the emails in a file

Now we have saved the email body in the file, to extract the links we need to go control flow and drag and drop the Regular Expression Parser Task and follow the steps from below section.

Getting all URLs from emails inside a variable

6. The next step is to save the URLs’. We will use the Regular Expression Parser Task for this.
Also, you need to use Regex and here is one example you can use, in this page Regex101 you can check more details about the expressions we are using:

Getting all URL from the emails

Expression:

href="(.*?)"{{*}}

Save the result in a file of the URLs from emails

7. Finally, use the Logging task to save the URLs in a file, use the following configuration

Uncheck all options and select message type as none

Conclusion

If everything is OK, you will be able to download the URLS from your emails. To do that, we read the list from the body of the emails. Then we get the URLs of the emails using expressions. Finally, we store them in a local file.

The post How to get all URLs from emails from Outlook appeared first on ZappySys Blog.

In our last post (Regex Cheat Sheet) we explained use cases of SSIS Regular Expression Parser Task . Now lets look at some real world usecase. Basically in this article we are going to Extract data from audit log using SSIS. For demo purpose we will use log generated by SSIS Export CSV File Task output. When you export many tables dynamically using this task you want to know how many Row count exported for each Table. This information is logged in SSIS Output Log but no easy way to capture in SSIS variable. So what we will do we will use few tricks to capture that data using Regular Expression usecase.

So let’s get started.

Setup Export CSV Task (Output Multiple Tables to CSV Files Dynamically)

For example purpose we will use Export CSV File Task but you can use this technique for any other Tasks.

1. Drag and Drop ZS Export CSV Task from SSIS Toolbox and configure as below.
   
2.   Configure Export CSV Task to output multiple tables as explained here
3. Now create 2 SSIS User Variables
   1. FilePath (String Type)
   2. RowCount (String Type)

That’s it… now in the next section we will setup event handler to capture output from Log.

Setup EventHandle / Extract Audit Data using Regex

1. Now select task and click Event Handler tab
2. Select Executable Name from the dropdown, Select Event Name OnInformation from Handler Dropdown
3. Click Create Hyper Link to Create Event Handler like below
   

   Create Task Level Event Handler in SSIS

4. Now Drag below 3 tasks and connect like this
   1. Script Task (We will use as dummy start (No configuration needed)
   2. ZS Regular Expression Parser Task
   3. ZS Logging Task (Optional) but we will use to Output Extracted Values
      
5. Connect Dummy Script Task to ZS Regular Expression Parser Task and Right click on Green connected Arrow and Use Expression as below. Click OK to save
6. Configure ZS Regular Expression Parser Task as below
   1. Enter

      {{System::ErrorDescription}}

        as Direct Value
   2. Enter Below two mappings
      1. For RowCount set
         

         Total (\d+) records written to : (.*){{1,1}}

      2. For FilePath set
         

         Total (\d+) records written to : (.*){{1,2}}

7. Here is how it will look like
   

   Extract data from audit log using SSIS Regular Expression Parser Task

8. Thats it… Now connect 2nd step to last Logging Task…. in that Task you can set Text like below just to log extracted data. You can also use Execute SQL Task to log into Database Table
   

   {{User::FilePath}} ===> {{User::RowCount}}

9. Now run Package and you will see audit data we extracted for each file. Once File is exported we capture OnInformation event… and if Message contains substring like “records written to” then we extract data into variable for Row Count and Path then you can save to desired place for auditing purpose

The post Extract audit data from SSIS Execution Log (EventHandler and Regular Expression) appeared first on ZappySys Blog.

In this short article, you will learn how to write Regular expressions in SSIS (i.e. Regex) and what tool to use to test them. You will also find helpful resources on how to write more sophisticated expressions and learn more about them. For demo purposes, we will use FREE SSIS Regex Parser Task to parse and extract the text using Regex.

You can use Regular expressions in several SSIS PowerPack connectors:

• SSIS Regex Parser Task (FREE),
• Azure Blob Storage Task,
• Advanced File System Task,
• Amazon S3 Storage Task and others.

Writing Regular Expressions in SSIS

Based on the SSIS component you use it will include the filename in filtering options or match the text: See below Syntax to write Regex in ZappySys tools. We support additional construct {{X, Y}} at the end of Regex to control two parameters. This additional construct is useful for data extracted from matching regex. If you want to Test patterns and not worry about data extraction then no need to use last {{

Occurance_Index

Group_Index_Or_Name

Syntax:

<your Regular Expression>[{{Occurance_Index|*[,Group_Index_Or_Name]}}]

Where:

Occurance_Index

Group_Index_Or_Name

How to name a group?

E.g. (\w+)@(?<domain>\w+.com)

How to use the group name in the match extract?

E.g. (\w+)@(?<domain>\w+.com){{0,domain}}

Example Input:

Let’s assume we have the following input text. We will test various Expressions.

Customer => AAA
Email => aaa@google.com
Phone => 101-222-3333
========
Customer => BBB
Email => bbb@yahoo.com
Phone => 102-222-3333
========
Customer => CCC
Email => ccc@hotmail.com
Phone => 103-222-3333
========
Customer => DDD
Email => ddd@outlook.com
Phone => 104-222-3333

Sample Regex Expressions

(.*)

More Regular Expression Examples

Regex Examples (Using SSIS Regular Expression Parser Task)

Here is an example how Regex

(.+)_(.+)_(.+){{1,2}}

Using Regular Expressions in SSIS Regex Parser Task (Extract Groups)

Using Groups / Occurrence Index

Some tasks like SSIS Regex Parser Task (FREE) supports extracting value from specific occurrence and specific part of matched pattern using special syntax at the end of your pattern (see below).

Your Regex Pattern Here{{OccuranceIndex,GroupIndex}}

Where :
OccuranceIndex is 0 based (0=extract first occurrence)
GroupIndex is 0 based (0=extract first matching group from pattern. First group is always entire text. )

See above screenshot in previous section for example.

Tools

The best tool we’ve found to write and test Regex is Regex Hero (will require IE with Silverlight if you want to use it online, in the browser):

Another great site for Regex testing is https://regex101.com (Works in any browser unlike previous one)

and few more sites as below

http://www.regexr.com/
http://www.regexlib.com/
http://www.regular-expressions.info/

Resources

Regular Expressions cheat-sheet to hang on the wall

Regular Expressions quick reference

The post Using Regular Expressions in SSIS appeared first on ZappySys Blog.

In this post you will learn how to use FREE SSIS Regex Parser Task along with REST API Task to extract HTML content in few clicks.

Scenario

Assume that you want to search certain keywords from Bing or google and want to know how many pages found for that keyword. Url for search would be something like http://www.bing.com/search?q=regex where regex is our search word.

When page is returned view source code of that page and you will find tag like below.

<span class="sb_count" data-bm="4">21,00,000 results</span>

What we want is number 21,00,000 using Regular expression pattern search.

Step-By-Step : Extract HTML Tag value using Regex Expression

1. Download and Install SSIS PowerPack (It includes FREE SSIS Regex Parser Task )
2. Create new SSIS Package
3. Drag ZS REST API Task on Control flow designer from SSIS Toolbox
4. Double click to configure the task. Enter URL you like to fetch e.g. http://www.bing.com/search?q=regex
5. Click on Response Tab and check Save response option. Select Save to Variable. If needed create new variable.
6. Click Test (Scroll at the bottom to see html content)
7. Now drag ZS Regex Parser Task and connect with REST API Task
8. Select Variable which will hold HTML text you like to parse.
9. Enter following expression and map target to some Variable if you like to save extracted value. Below expression ends with {{0,1}} which means extract first match and 2nd group of that match (0 based Index). 2nd group of match will hold actual count of search result. If you omit {{x,y}} at the end then {{0,0}} is used.
   

   \<span\s*\w*\s*class="sb_count"\s*>\s*(?<p2>[0-9,.]*){{0,1}}

   See below screenshot
   

   SSIS Regex Parser Task – Extract HTML Tag Value using Regular Expression

10. In the above step you can select Variable as Input or use placeholder in Direct string (e.g  {{Use::varHtml}} )
11.  You can also connect ZS Logging task to show extracted value

Here is final flow.

SSIS Regular expression parsing example

The post Using SSIS Regex Parser Task for Extracting HTML Content appeared first on ZappySys Blog.