In this post you will learn how to extract data from web pages using SSIS. In other words, read HTML Table in SSIS, then Loop through extracted links and finally download files. To achieve this scenario we will use SSIS HTML Source and REST API Task.

Step-By-Step

To achieve desired extraction from web pages / download files we have to use few different components. Let’s get started.

Read HTML Table from Web Page

First, let’s read desired content from web URL using   SSIS HTML Source. This component can read data from URL(s), Local HTML File(s) or Direct HTML String stored in a SSIS variable.

1. Create a new SSIS Project
2. In the control flow, Right click anywhere in the designer and click Variables option.
3. Create following 3 variables which we will use later on.
   FileList (Object Type)
   ZipFileName (String Type)
   ZipFileUrl (String Type) – Default Value must not be blank (Enter anything like  http://a.com ).
4. Drag Data Flow Task from SSIS Toolbox to designer
5. Go to data flow designer by clicking on the data flow task.
6. Drag and drop ZS HTML Table Source and double click to edit it. Enter the HTML Web Page URL from where you like to read information. For example purpose, we will use below URL. On that page we have a table with files we need to download.
   

   http://www.patentsview.org/download

   

   Configure – SSIS HTML Table Source URL / Other Options

7. Now let’s go to Extract Links and Images tab and check Output Links for each column. This will give us link for Zip File URL which we want to download.
   

   Configure SSIS HTML Table Source (Extract Text / Images/ Links from Web Pages / URL)

8. Click Preview to see data.
   

   Preview HTML Table Data

9. Now click on OK to save the UI.
10. In next section we will configure Recordset Destination

Save Records to SSIS Variable

Now let’s look at how to save HTML Table into in memory recordset which we will loop through later on. You can save data directly to relational destination such as SQL Server or Oracle but in our scenario we are extracting links and downloading them one by one.

1. In continuation of previous step, Once done with HTML Table Source configuration, Drag Recordset Destination from Data flow toolbox
2. Connect HTML Table Source to Recordset Destination
3. Select Variable to hold recordset (Must be Object Datatype – in our case we have User::FileList)
4. Select Input columns you like to use. In our case we will Pick two columns as below (Name and Link).
   

   Select Input Columns for SSIS Recordset Destination

5. Click OK to save Recordset destination.
6. In our next step we will look at how to loop through these records.

Loop through records in SSIS

Once you done with previous step, switch to control flow tab and follow below steps.

1. Drag ForEach Loop Task from SSIS Toolbox.
2. Connect Data Flow Task to ForEach Loop Task
3. Double click ForEach Loop task to configure
4. On the Collection Pane, select Enumerator as Foreach ADO Enumerator as below.
   

   Configure ForEach Loop Task – Download Files

5. On the Variable Mappings tab you can map columns you care about. For example in the previous section we selected two input columns for recordset destination. We can map same columns here in same the same order.
   

   SSIS ForEach Loop Task – Variable Mappings

6. Click OK to save our settings.

Download File from URL in SSIS

Now, after Loop configuration we can define what we want to do in each iteration. For example we can simply drag Logging Task and display Variables we extracted in ForEach Loop Variable Mappings tab. For downloading links we extracted in previous step you can use REST API Task as below.

1. Drag ZS REST API Task from SSIS Toolbox and drop inside ForEach Loop.
2. You can now set Dynamic URL using placeholder 

   {{User::ZipFileUrl}}

3. Now one more thing we need to do is define Target path where we will save downloaded file.
4. Go to, Response Settings Tab, Check Save Response option and Select Save to File.
5. Also, check Treat Response As Binary
   

   Configure SSIS REST API Task to download Zip File from URL

6. Click OK to save REST API Task.

That’s it. You can now run SSIS Package and see things in action.

Conclusion

In this article, we saw how easy it is to extract unstructured from web in few clicks. Using components such as HTML Table SOurce and REST API Task we can read / parse HTML Pages. Download SSIS PowerPack to explore many other options including 70+ tasks / components we have not discussed in this article.

The post Read HTML Table in SSIS – Extract / Download Links / Images appeared first on ZappySys Blog.

In this article, we will show how to send values from an SSIS Variable to an HTML file. We will use the SSIS Advanced File System Task to store the list of system files of a folder into a variable and then we will use the SSIS Logging Task to store the variable in an HTML file.

In the second part, we will read values from an HTML table and export the data into SQL Server. We will use the HTML file created in the first example. We will also use the ZS HTML Table Source to read the data from the HTML table and export to SQL Server using the OLEDB Destination.

We will be using this ZappySys SSIS PowerPack component to make things work:

	SSIS Logging Task (Free)
	SSIS HTML Table Source
	Advanced SSIS File System Task

Finally, we will show how to read data from a SQL Server query and store into an HTML file. We will use the Execute SQL Task combined with the ZS SSIS Logging Task.

Requirements

1. First, you will need SSDT installed.
2. Secondly, ZappySys PowerPack for SSIS.
3. Finally, SQL Server installed.

Getting started

How to save the list of system files into SQL an HTML table

In the first example, we will use our ZappySys Advanced File System to copy a list of the system files in a folder. This list of files will be stored in an object variable and then we will save to a file.

1. In the first place, we will use the Advanced File System Task and select the Get file list ADO.net DataTable action. We will list the files of the c:\sql folder and save the results into a variable:
   

   SSIS list files

2. In addition, we will create a variable of type object named listfiles that will store the files information:
   

   SSIS object variable

3. Also, we will use the ZS Logging Tasks and join with the ZS Advanced File. The ZS Logging Task will be used to store the variable into a file:
   

   SSIS save variable to HTML

4. In addition, it is possible to send the variable values into the Execution Log (SSIS output), to a file, to a MessageBox:
   

   SSIS output variable to log, message box and file variable to read and write data to HTML in SSIS

5. In this example, we will show the values into an HTML file named listfiles.html
   

   SSIS save variable iHTMLhtml

6. After executing the package, you will be able to see the HTML with a list of files, names, size, modification and creation information:
   

   ssis list files

How to copy data from an HTML table to SQL Server

The next example will save data from a file named listfiles.html created in the previous example with an HTML table. The example will export the table to SQL Server. However, with our HTML Table Source, you can export to any destination.

1. In order to start, we will Drag and drop a data flow task and double click it:
   

   SSIS data flow

2. Next, in the Data Flow task, use the HTML Table Source and join to the OLEDB Destination:
   

   SSIS HTML table SQL server

3. Also, in URLs enter the path of the HTML file created in the first example:
   

   SSIS HTML table read

4. In addition, go to the OLEDB Source, specify the SQL Server name, database, and log in to it.
   In the name of table or view press new to create a new table:
   

   CREATE TABLE [dbo.listfiles] (
   [_rowNumber] int,
   [_groupName] nvarchar(255),
   [FullPath] nvarchar(255),
   [Name] nvarchar(255),
   [Size] int,
   [AgeInDays] int,
   [LastEditInDays] int,
   [LastModifiedDate] datetime,
   [CreationDate] datetime,
   [LastModifiedDateUtc] datetime,
   [CreationDateUtc] datetime,
   [Extension] nvarchar(255),
   [IsReadOnly] nvarchar(255),
   [FolderPath] nvarchar(255)
   )

   

   SSIS OLEDB create table

5. Also, go to Mappings page to map all the columns.
   

   SSIS map columns

6. Finally, run the package and you will have all the data exported.

How to export SQL Server results to an HTML in SSIS

The following example will show how to export the results of an SQL query into an HTML file.

1. In order to start, we will use the Execute SQL Task. Specify the option Full result set and create an OLEDB connection specifying the Server Name, database name and the select statement to a table:
   

   SSIS SQL query results

2. Secondly, go to the Result Set page and in the Result Name, set 0 and in Variable name, create a new variable:
   

   ssis sql result to variable

3. Also, create a variable of type object:
   

   SSIS object data type

4. Finally, use the ZS Logging Task to save the results in a File. Specify the Log file path and insert the variable:
   

   SSIS save to HTML file

Conclusion about read and write data to HTML in SSIS

To conclude, in this article, we learned how to export a list of files system properties and names to SQL Server. How to read an HTML table to SQL Server and finally how to export SQL Server results to an HTML in SSIS. There are nice tools to read and write data to HTML in SSIS

If you liked the tools, you can download them HERE.

References

• SSIS PowerPack ZappySys
• Integration Services (SSIS) Variables
• Execute SQL Task

The post How to read and write data to HTML in SSIS appeared first on ZappySys Blog.