How to Pivot CSV Data in SSIS

Introduction

In our previous blog we saw How to write data into CSV file in SSIS (GZip / Split). Now in this blog, we will see How to Pivot CSV Data in SSIS using CSV Source. It also supports Pivot mode so you can convert single CSV string value into Rows. In this article we also see how make some CSV columns as parent and not consider it in pivot and load it as it is using JSON Generator and JSON Parser Transform Pivot Option.

In nutshell, this post will focus on how to Pivot CSV Data into multiple columns and rows.

 

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:
  1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from the Microsoft site).
  2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
  3. Make sure ZappySys SSIS PowerPack is installed (download it, if you haven't already).
  4. (Optional step). Read this article, if you are planning to deploy packages to a server and schedule their execution later.

How to Pivot CSV Data in SSIS using CSV File Source

Let´s start with an example. In this SSIS CSV Parser Transform example, we will Parse database column string into multiple columns and rows.

  1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
    Drag and Drop SSIS Data Flow Task from SSIS Toolbox

    Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

  2. Furthermore, drag and drop the CSV Source and Double click on it to configure it.
    Drag and Drop : CSV Source from SSIS Toolbox

    Drag and Drop : CSV Source from SSIS Toolbox

  3. Enter the CSV file path and select desire column delimiter most common in CSV is comma and check the First Row as headers and click on preview.
    CSV Source Preview

    CSV Source Preview

  4. Now, go to Pivot Columns to Rows Tab and checked Enable Pivoting and click on Preview Data to view the data.
    CSV Source Pivoted Data Preview

    CSV Source Pivoted Data Preview

  5. That’s it we see how to read data from CSV File and also Pivot the CSV Data using CSV Source.
  6. Now In the next section we will see how to how make some CSV columns as parent and not consider it in pivot and load it as it is.

How make some CSV columns as parent.

Sometimes you just need to make child columns in to rows and want to leave parent level columns as it is.

  1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
    Drag and Drop SSIS Data Flow Task from SSIS Toolbox

    Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

  2. Secondly, drag and drop the CSV Source and Double click on it to configure it.
    Drag and Drop : CSV Source from SSIS Toolbox

    Drag and Drop : CSV Source from SSIS Toolbox

  3. Furthermore, enter the file path and click on preview.
    CSV Source Preview

    CSV Source Preview

  4. For example if we want to pivot Email and Phone only in pivoting and wants to exclude parent level columns FirstName and LastName as it is.
  5. For that drag and drop JSON Generator Transform and Select Output Mode as Output One JSON Per Parent Input Row  and Add Unbound Nested Element and named it as Persons and click on OK.
    JSON Generator Add Unbound Nested Element

    JSON Generator Add Unbound Nested Element

  6. Now add that two parent level fields FirstName and LastName in the Persons.
    JSON Generator Add Multiple Elements

    JSON Generator Add Multiple Elements

  7. Now select Person Element and right click and Add Unbound Nested Element and named it as Contacts and click on OK.
    JSON Generator Add Unbound Nested Element

    JSON Generator Add Unbound Nested Element

  8. Now add that two child level fields Email and Phone in the Contacts.
    JSON Generator Add Multiple Elements

    JSON Generator Add Multiple Elements

  9. Therefore JSON Generator Transform looks like below screen. Now Copy that Sample Preview JSON String to use it in the JSON Parser Transom as sample string.
    JSON Generator Transform - Sample Data

    JSON Generator Transform – Sample Data

  10. Similarly, drag and drop JSON Parser Transform and select the Input JSON Column and Paste that copied Sample JSON String and select the Contacts filter to read the Child data and click on Include Parent.
    JSON Parser Transform - Select Filter

    JSON Parser Transform – Select Filter

  11. Now go to Pivot Columns to Rows Tab and checked the Enable Pivoting and click on the Preview.
    JSON Parser Transform - Enable Pivoting

    JSON Parser Transform – Enable Pivoting

  12. Finally, we are ready to load this data.
    CSV Pivoted Data

    CSV Pivoted Data

  13. Let’s load this data in the SQL Server.

Load Pivoted CSV File data into SQL Server

Now let's look at how to load data into target like SQL Server, Oracle or Flat File. In below example we will see loading data into SQL Server database but steps may remain same for other targets which can be accessed using OLEDB Drivers (e.g. Oracle).
  1. Inside Data Flow, Drag and drop Upsert Destination Component from SSIS Toolbox
  2. Connect our Source component to Upsert Destination
  3. Double click Upsert Destination to configure it
  4. Select Target Connection or click NEW to create new connectionConfigure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS
  5. Select Target Table or click NEW to create new table based on source columns
  6. Click on Mappings Tab to Auto map columns by name. You can change mappings as you need SSIS Upsert Destination - Columns Mappings SSIS Upsert Destination - Columns Mappings
  7. Click OK to Save Upsert Destination Settings
  8. That's it, You are now ready to run data flow. NOTE: If you wish to debug data flow and see records when you run, add data viewer by right click on blue arrow > Click Enable Data Viewer
  9. To execute data flow, Right click anywhere inside Data Flow Surface and click Execute Task
 

Conclusion

After all, we saw you how to read CSV file data, pivot the data and Similarly, using SSIS JSON Generator and Parser Transform Pivot the CSV Data excluding some columns from pivoting.  To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following URL for more information.

Posted in SSIS CSV Source, SSIS JSON Generator Transform, SSIS JSON Parser Transform, Uncategorized and tagged , , .