ssis xml parser transform Archives | ZappySys Blog

How to import XML into SQL Server using SSIS

ZappySys — Tue, 30 Aug 2016 22:42:05 +0000

Introduction

In this post you will learn how to import XML file into target such as SQL Server, CSV, Excel, Oracle. Converting or Importing XML requires flattening XML structure (i.e. hierarchy) sometimes we refer this process as de-normalization. Parsing XML can be tricky depending on how many levels you have and from which level you need data. If you want to export XML rather than Import then check this article.

This article assumes you have installed SSIS PowerPack and watched introduction videos of SSIS XML Source Connector and SSIS XML Parser Transform

Import XML File using SSIS XML Source

ZappySys SSIS XML Source is a very powerful component. It supports not only reading from file but also reading from Direct XML String or REST API / SOAP Web Service.

How ZappySys XML Source parse/output data

XML Source created by ZappySys is very different than what Microsoft provides out the box because ZappySys component outputs one flat view of nested XML document (de-normalized) compared to many outputs what Microsoft XML Source produces which is hard to manage and join based on relation with parent node.

With ZappySys XML component you don’t have to worry about joins because by default parent columns included along with hierarchy you extracted (e.g. If you extract $.Customer.Orders[*].OrderDetails[*] then you get all columns listed under and nodes)

ZappySys XML Source (Read XML Files, Consume REST API, SOPA Web Service)

Importing single or multiple XML Files

To import single files specify full XML file path

Example: c:\ssis\File_001.xml

To import multiple files using XML source use wildcard pattern

Example: c:\ssis\File_*.xml

XML File Source – Specify File Path with wildcard pattern (Loop multiple files)

Importing data from multiple XML files

Using variable to make path dynamic

You can also use variable placeholders in your path to make it dynamic

Example: c:\ssis\{{User::FileName}}.xml

Importing data from XML Web Service or REST API

XML Source supports consuming REST API data or calling XML SOAP Web services. This article doesn’t include this feature in depth. But refer to learn more about this feature.

SSIS XML Source – Direct Mode – Call REST API web service or read from XML File

Parse XML string and split into columns using XML Parser Transform

Parsing XML Data – XML String into Columns using SSIS XML Parser Transform

Other Settings (XML Source, XML Parser Transform)

Preview Data

All access modes support preview data like below. Change your settings such as Filter expression, Source etc and click preview.

Preview XML Data

Filtering data using expression

You can filter XML data by specifying simple expressions (e.g. $.store.book[*]) to navigate various parts of XML where you can extract data (See above screenshot).

Things to remember for filter expressions

Expression must start with $ (means root from where parsing will start)
Use dot (.) to separate Hierarchy and use brackets to indicate array []
Use * to indicate all records

Sample XML for examples



  
    Bob
    2015-01-01
    
      Atlanta
      GA
    
  
  
    
      2005-02-01T10:00:09
      
        Tea
      
      
        Coffee
      
    
    
      2005-04-01T21:59:20
      
        Tea
      
      
        201
        Soda
      
    
  
  
    
      2005-02-01T21:59:20
      
        Apple
      
      
        Orange
      
    
    
      2005-01-01T21:59:20
      
        Soda
      
      
        Milk

Example of filter expression

Filter	Output
$	Get all records
$.Root.Customer[*]	Get all customers
$.Root.Customer[].Order[]	Get all orders for all customers
$.Root.Customer[].Order[](Include Parent Columns=True)	Get all orders for all customers (Include Parent Columns option checked)
$.Root.Customer[*].@CustId	Get only Customer Id for each customer record
$.Root.Customer[*].Order[:1]	Get first order for each customer
$.Root.Customer[*].Order[:3]	Get top 3 orders for each customer
$.Root.Customer[*].Order[:1]	Get last order of each customer
$.Root.Customer[*].Order[0,4]	Get orders starting from first row to fifth row (Zero based index). If you want 5th row to 10th row then use [4,9]
$.Root.Customer[*].Order[2]	Get 3rd order for each customer

Defining array elements

Any hierarchy you extracting with array indicator can be added to array element name list (On Array handling Tab). This helps when you only have one row for node specified as array.

For example if your filter expression is $.Customer.Order[*].OrderItem[*] in this case you indicated that you can have one or more Order nodes under Customer and you may have one or more OrderItem nodes under Order node. If this is the case then set array handling as below

Handling Rows/Array Elements (Repeated XML Elements)

Editing Column Metadata (Length and DataType)

XML Source and XML Parser both detects Datatype/Length by scanning sample data (Default 300 rows) but sometimes this is not accurate and it may produce invalid metadata. If that’s the case you can easily change it in Columns tab. Once you edit Length or DataType make sure you check Lock column (Last column in grid). Once Lock is applied metadata refresh wont affect your changes. You can also add new column manually if its not detected (Lets say your XML is very large and many columns not found in first 300 rows scanned). To add new column type in the last grid row or click new icon in the grid toolbar.

Edit XML File Source Column Metadata

Conclusion

In this post you have seen how to use SSIS XML Source Connector and SSIS XML Parser Transform to extract data from nested XML documents without any coding.

Related to: SSIS Import XML Files | SSIS Parse XML data | SSIS Extract from XML files | Read from XML Files

The post How to import XML into SQL Server using SSIS appeared first on ZappySys Blog.

Reading JSON Arrays from file / REST API

ZappySys — Mon, 11 Jan 2016 16:50:48 +0000

Introduction

REST API is becoming more and more common and with that you will see explosion in use of JSON data format. One of the questions we get a lot is “How to extract or read array from JSON data file” or “How to read multiple arrays from JSON data“. Example : Store JSON document has Customers[…] and Employees[…] arrays and lets say you want to extract both and save to target SQL server database. In this post you will learn how to make this possible and how to make it fast with some performance tips. We will use SSIS JSON Source (File, REST API Connector)

If you want to learn how to parse Multi-Dimensional JSON arrays then check this article.

Video Tutorial – Read from multiple arrays from JSON document

Read array from JSON document

To extract subdocument or array content from JSON document you need to use JSON Path expression. Lets check following example JSON document.

{
  "store": {
    "employees": [
      {
        "name": "bob",
        "hiredate": "2015-01-01"
      },
      {
        "name": "sam",
        "hiredate": "2015-01-02"
      },
      {
        "name": "ken",
        "hiredate": "2015-01-03"
      }
    ],
    "books": [
      {
        "category": "reference",
        "author": "bob",
        "title": "hellooo1",
        "price": 1.95,
        "sections": [
          "s1",
          "s2",
          "s3"
        ]
      },
      {
        "category": "fiction",
        "author": "sam",
        "title": "hellooo2",
        "price": 1.96,
        "sections": [
          "s4",
          "s1",
          "s3"
        ]
      },
      {
        "category": "science",
        "author": "steve",
        "title": "hellooo3",
        "tag": "1bcd",
        "price": 11,
        "sections": [
          "s9",
          "s2",
          "s3"
        ]
      }
    ],
    "location": {
      "street": "123 Main St.",
      "city": "Newyork",
      "state": "GA"
    }
  }
}

Below is example filter expression to extract common elements

$.store.books[*]       //get all books for store              
$.store.employees[*]       //get all employees for store                 
$.store.books[*].sections[*]       //get all sections from all books   
$.store.books[*].author //get all authors of all books for store           
$.store.books[*]    //get all books for store  
$.store.books[2]    //get 3rd book record
$.store.books[:2]   //get first 2 books from the top
$.store.books[-2:]  //get last 2 books
$.store.books[?(@.price < 10)] //get books where price is less than 10
$.store.books[?(@.tag)] //filter all books with tag

Read single array from JSON data (JSON Path expression)

SSIS PowerPack comes with powerful REST API Connectors for JSON and XML . In below example screenshot you will see how to extract single array section from your JSON File. But if you have multiple array sections (e.g. books and employees from above example) then you have to use Multiple JSON Source Connectors for same JSON File.

Read multiple arrays from JSON data

If you have scenario to extract multiple arrays from same JSON (e.g. extract Employees and Books from store ) then you have two options. You can pick any approach which suites your need (Method-1 is recommended).

Method-1: Use single JSON Source with [Output As Document] option and feed that to JSON Parser Transform
Method-2: Use multiple JSON Sources with different filter expressions
Method-3: Enable Array Flattening Option (Useful if you have smaller arrays – i.e. less than 10 items) – Version v2.7.6+

Now lets discuss each approach in detail.

Method-1: Use single JSON Source along with JSON Parser Transform

SSIS PowerPack v2.2 released new JSON Parser Transform and XML Parser Transform which can be used to parse any input JSON/XML data coming from upstream in pipeline. It parse input string column into multiple columns how JSON/XML Source does but the only difference is JSON Parser takes String as input from existing upstream column.

Here are the steps to parse multiple JSON arrays using JSON Parser Transform. For example purpose we will use sample store json listed above.

Drag new data flow in SSIS Control Flow designer. Double click on Data flow
Drag ZS JSON Source from SSIS Toolbox. Configure Source, Filter and other property.
Check [Output As Document] option (Found under Access Mode dropdown). Click preview. From grid copy one sample document from __DOCUMENT__ column. (Right click on Cell and Copy).This is raw JSON which we will use later to feed as sample in JSON Parser Transform (If you cant copy JSON then obtain some other way)
Enable Output As Raw Document Option – SSIS JSON Source
Click OK to save JSON Source.
Drag Multicast Transform (Native SSIS Transform) from Toolbox
Drag & Drop SSIS MultiCast Transform from Toolbox on designer surface
Connect JSON Source to Multicast Transform
Drag first JSON Parser Transform from Toolbox. Rename it to [Parse Employees]. Connect blue arrow coming from Multicast to JSON Parser Transform
Drag second JSON Parser Transform from Toolbox. Rename it to [Parse Books]. Connect blue arrow coming from Multicast to JSON Parser Transform
Double click [Parse Employees]. Select __DOCUMENT__ as upstream column. Enter sample JSON you obtained in step#3. Click on select Filter and highlight Employees node (i.e. $.store.employees[*] ). Click preview to see data.
Use JSON Parser Transform to parse Raw JSON document
Very Important:Click Columns tab and verify metadata (e.g. datatype, length). If you think you need extra columns or bigger column length then change in the grid and check [Lock] option found in the last grid column. If column length is smaller than output data then you will get error message when you run package. You can review error message carefully to determine which column needs to be adjusted (Repeat this until you clear all errors).
Click OK to save [Parse Employees].
Double click [Parse Books]. Select __DOCUMENT__ as upstream column. Enter sample JSON you obtained in step#3. Click on select Filter and highlight Books node (i.e. $.store.books[*] ). Click ok. Click preview to see data.
Perform same steps as #11 for [Parse Books] to set correct metadata.
Click OK to save [Parse Books].
Drag ZS Trash destination from Toolbox. Connect Trash destination to [Parse Employees]
Drag ZS Trash destination from Toolbox. Connect Trash destination to [Parse Books]
Run the package and you will see both 2 outputs from different JSON Parser with different row count.

Here is how it will look like

Extract multiple array output from JSON file / REST API response in (SSIS JSON Source, JSON Parser Transform)

Method-2: Use multiple JSON Sources with different filter expressions

You can consider using multiple JSON Source (each source with unique filter expression). Since one JSON source can only extract one array or Document.. You have to use below technique to read multiple arrays from same JSON document.

Read array from JSON data – Reading multiple array or single array

Method-3: Use Array Flattening Option (Preferred for smaller arrays)

If you have array inside extracted record and you like to flatten it so each items of array becomes column then use newly introduced feature [Enable Array Flattening]

Consider JSON like below. What if you don’t like to store every detail from below JSON in just one table (i.e. rather than splitting in 3 tables Customers, CustomerHobbies , CustomerAddresses ?) . To achieve full denormalization ZappySys offers powerful yet simple one click option.

{
	"Customer" : [{
			"Name" : "John",
			"Hobby" : ["Painting","Reading","Swimming"],
			"Addresses" : [
				{"Street":"100 Main St","ZipCode":"10062"}, 
				{"Street":"101 Fire St","ZipCode":"11121"}
			]
		}, {
			"Name" : "Peter",
			"Hobby" : ["Hiking","Golf"],
			"Addresses" : [
				{"Street":"200 Main St","ZipCode":"20062"}, 
				{"Street":"201 Fire St","ZipCode":"33321"}
			]
		}
	]
}

To enable flattening check below option on Filter

JSON Array Flattening Option in SSIS JSON Source

Column after Array Flattening Option

Using JSON REST API Connector to parse JSON data

If you want to access JSON data from REST API then you can use same JSON Source Connector. All you have to do is rather than local file path c:\data\xyz.json change it to URL as below and you will be able to access and filter JSON data using same technique as above.

Get JSON data from URL (REST API JSON response)

JSON Path for recursive data extract (Double dot expression)

There will be a time when you dont know how many levels deep you have to go for data extract. See below example. We will to get all employees id and name from hierarchy. For that you can use double dot like below. As you can see there are many nested levels and we want to extract id and name from each level so data can be

Expected Output

id      name
-------------------------
100	Jeff (CEO)
200	Bob (VP Sales)
300	Kim (VP HR)
400	Kumar (VP Dev)
210	Ron (Sales Mgr)	
220	Akash (Sales Mgr)	
410	Den (Sr. Dev)	
420	Den (Sr. Dev)
411	Den (Jr. Dev)
412	Sita (Jr. Dev)

Sample Data

{
  "employees": [
    {
      "id": 100,
      "name": "Jeff (CEO)",
      "employees": [
        {
          "id": 200,
          "name": "Bob (VP Sales)",
          "employees": [
            {
              "id": 210,
              "name": "Ron (Sales Mgr)"
            },
            {
              "id": 220,
              "name": "Akash (Sales Mgr)"
            }
          ]
        },
        {
          "id": 300,
          "name": "Kim (VP HR)"
        },
        {
          "id": 400,
          "name": "Kumar (VP Dev)",
          "employees": [
            {
              "id": 410,
              "name": "Den (Sr. Dev)",
              "employees": [
                {
                  "id": 411,
                  "name": "Den (Jr. Dev)"
                },
                {
                  "id": 412,
                  "name": "Sita (Jr. Dev)"
                }
              ]
            },
            {
              "id": 420,
              "name": "Den (Sr. Dev)"
            }
          ]
        }
      ]
    }
  ]
}

Configure JSON Source for Recursive Children Scan using Double Dot JSONPath

Open JSON Source and configure like below.

Set Filter as $..employees[*] (Notice how we used double dots in expression. This does recursive scan for N number of levels)

JSON Path for recursive scan of nested array (many levels deep)

Parse Multi-Dimensional Array

If you want to learn more how to parse Multi-Dimensional JSON arrays then check this article full length article.

Sample JSON with Multi Dimensional array

{
  "columns" : ["Id", "FirstName", "IsActive"],
  "rows" : [ [1,"bob",true], [2,"sam",false], [3,"joe",true] ]
}

Array Transformation Options for 2D array

Sample JSON with Multi Dimensional array (No columns in document)

{
  "rows" : [ [1,"bob",true], [2,"sam",false], [3,"joe",true] ]
}

Array Transformation Options for 2D array

Performance Tips

In this section we will describe how to make your things faster.

Reduce number of requests to server by changing page size

If you are doing REST API paging to get full resultset then make sure you adjust page size to appropriate number so total requests sent to server is reduced. Each request to server adds overhead. Most of API supports page size parameter. Refer to your API documentation how to adjust page size.

Avoid multiple REST API Round trips by caching response

If you extracting multiple arrays from REST API response using above technique then we recommend you to use REST API Task so save response to File or variable first and then use cached JSON data multiple times. This approach is not as friendly as others but it can certainly reduce your total API requests.

Use GZip compression

If your REST API supports GZip compression then check Enable Gzip on Advanced tab of SSIS JSON Source (File, REST API Connector). This will compress response in Gzip format from server and when it comes to your server it will be automatically decompressed by JSON Source connector. This seeds up thing significantly if you have lots of data to download.

Uncheck include parent columns if not needed

By default JSON Source connect includes parent column (Level above last property in filter) from the Filter path you specify. For Example if you specify $.store.employees[*] and “Include parent columns” checked then It will extract all employees and then also include parent level columns for store (Tip: Parent column name start with P_ by default) . If you don’t need parent columns then uncheck from output, this reduces many steps during data extract process.

Download sample package

Click here to download SSIS 2012 Package (Will work in SSIS 2014, 2016)

Conclusion

In this post you learned how to extract data from JSON array and how to make data extract fast. Download SSIS JSON Source (File, REST API Connector) to try everything listed above.

The post Reading JSON Arrays from file / REST API appeared first on ZappySys Blog.