In this blog, we will learn how to call SOAP / REST API using Dynamic Token in SSIS (i.e. Two steps authentication approach – Fist Call Login API to get token and then call API).

In our previous blog post, we saw how to call Web API using some industry standards approaches, such as OAuth 2.0 Authorization or Basic Authorization. However, not all APIs follow the industry standard to authenticate API requests. After talking with many customers we noticed that very common non-standard technique to authenticate API is Dynamic Token Approach. In this approach first, you have to call Login request to get a temporary token (i.e. sessionid ). Once you get that temp token you can supply it along with your API requests. These temp token/session id usually expires in short time (i.e. expire in one hour or so).

Sounds complicated? Well not really if you use ZappySys SSIS PowerPack or ODBC Drivers for API. In this post, we will go through the steps, to authenticate using Dynamic Token approach. We will show you features of  ZappySys SSIS HTTP Connection Manager. For example purpose, we will call XML SOAP API using SSIS XML Source Connector but same concepts can be applied to read from JSON API using  SSIS JSON Source Connector or CSV API using CSV Source Connector.

So let’s get started.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS XML Source Connector and  SSIS JSON Source Connector that will help you to call SOAP / REST API, read various data from SOAP / REST API with SSIS.

To get token from login / authentication URL you have to gather a few pieces from your API documentation.

• How to call Login API for your API – (This is not the same URL as Vendor Site Login Page, but its usually API Login endpoint e.g. https://somecompany/api/v2/login)
• API Response Format (e.g. JSON, XML ,CSV or Random Text )

In our below example steps, we will use XML API which requires POST method first to pass Userid / password to authenticate. If Authentication is successful it may return a response in XML format which contains token.

In the next step, we will call other API (e.g. get-data) and pass token as Authorization Header. All these steps will be done automatically using  ZappySys SSIS HTTP Connection Manager.

Create HTTP Connection for Dynamic Token

To get API response data using SOAP / REST API call, you need to get an API Token from your login request URL. I am using ZappySys sample SOAP API to show you for the Demo purpose. Follow the steps mentioned below to get API Token:

Create in SSIS

1. Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
   

   Connection Manager Window: Create New Connection
2. Select ZS HTTP Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
   

   List of SSIS Connection Managers – Select HTTP Connection Manager

Create in ODBC

To use HTTP Connection in ODBC Drivers you can change connection Type to HTTP like below and it will show similar UI discussed in this article.

Create an HTTP connection inside an ODBC driver

Configure HTTP Connection Credential Type

Once you add new SSIS ZS HTTP Connection Manager, Go to General Tab and Configure HTTP Connection Manager for Two Step Authorization Flow. Enter API URL, Login Credentials, and configure some basic settings for the same. We used Login URL as just example but you can enter any valid API URL endpoint here. Actual Login URL goes on Dynamic Token Tab (see next step)

SSIS ZS HTTP Connection Manager: General Tab Configurations Settings

Configure Request Settings

Once you enter Username / password details on General Tab now we need to setup some information like Login Request URL, Body, Headers. Lets look at few options.

Passing User ID /Password in Body

1. Go to Dynamic Token tab, Enter the following pieces.
   1. Token/API Login URL
      For our example enter API Login URL as:
      

      https://zappysys.com/downloads/files/test/soap-login.aspx

   2. Request Method as POST
   3. Request Body as below. Notice we used 

      [$userid$]

        and 

      [$password$]

        placeholders. These are replaced automatically at runtime so you do not have to hard code in Body and keep it secure. (See General Tab to enter UserID/Password)
      Pass User ID / Password in Body
      

      <soapenv:Envelope xmlns:soapenv=""http://schemas.xmlsoap.org/soap/envelope/"" xmlns:urn=""urn:partner.soap.sforce.com"">
         <soapenv:Body>
            <urn:login>
               <urn:username>[$userid$]</urn:username>
               <urn:password>[$password$]</urn:password>
            </urn:login>
         </soapenv:Body>
      </soapenv:Envelope>

   4. Select Request Body Content Type. For JSON data it may be application/json. For Key/Value Pair  key1=value1&key2=value2 leave it Default.Passing User ID /Password in Request BODY
      

      SSIS ZS HTTP Connection Manager: Dynamic Token Tab Configurations Settings of Request Tab

Passing User ID / Password in URL (GET Request)

Our Example API pass credentials in the POST body but some API may pass via URL as GET request.

In such case, you can enter like this

https://mysite.com/api/v2/login?userid=[$userid$]&password=[$password$]

[$userid$]

[$password$]

Passing User ID /Password in Header as Basic Authentication

Another scenario is API requires HTTP Basic authentication, (Pass User ID / Password in Header as Base64 encoded value) then you can pass credentials like this:

Authorization: Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>


Here we have used FUN_BASE64ENC function but there are many other functions can be used anywhere in Body or Headers

Using Basic Authentication in Dynamic Token approach

Configure Response Settings

Now lets configure how to extract token (single value) from response. Which can be XML, JSON or sometimes Raw Text. We support 4 modes (default = None means full response text is treated as token). If you want to extract just single value and not full text then use below settings.

XML Expression Type (Use XPath)

1. Now, Go to Response Settings tab under a Dynamic Token tab, Enter Response Settings like Token Extract Mode, Expression Type, Content Filter Expression as below XPath Expression. The expression basically extracts content from sessionid XML node using below XPath Expression.

   //*[local-name()='sessionid']

   

   ODBC ZS HTTP Connection Manager: Dynamic Token Tab Configurations Settings of Response Tab

2. That’s it click on OK button to save your Connection Manager Configurations.

JSON Expression Type (Use JSONPath)

If you API returns JSON response (like Walmart API Example) then you can do something like below. Choose Expression Type=JSON and in Response Content Filter You can enter expression like $.some_property_name  (e.g. $.access_token). Check more about How to write JSON Path expression here.

SSIS ZS HTTP Connection Manager – Get a JSON response for the token

Regular Expression Type (Use Regex)

Another advanced mode to extract token from response is Regex Mode. You can write any valid regular expression like this one to extract just one value.

For example if your response returns raw string with token in double quotes but you want just token and no double quotes around it then try something like below.

Extract Token from response – Remove double quotes around value using Regular Expression

Call API in SSIS with Dynamic API Token

Now once you have setup HTTP connection manager to authenticate and get API token, we are ready for next step. Now we can use that token in SOAP / REST API Service call. So, let’s move to the SSIS Part.

1. Once SSIS Package is open, Goto Data Flow Section, For that Drag and Drop DataFlow Task from the SSIS Toolbox.
   

   Drag and drop Data flow task

2. Now, Drag and Drop SSIS ZS XML Source Component from the SSIS Toolbox (if your API is JSON based then drag ZS JSON Source).
   

   Drag and Drop ZS SSIS XML Source Component

3. Let’s double-click on XML Source Component to configure it’s Properties. Enter Request URL.
   

   https://zappysys.com/downloads/files/test/soap-getdata.aspx

   

   SSIS XML Source: Enter your SOAP Request URL

4. Check Use Credentials (Username/Password OR Token), Select HTTP Connection Manager from the drop-down list, we have created just before.
   

   ZS SSIS XML Source Component: Check Use Credentials and Select Connection Manager

5. Select the Request Method and Enter SOAP Request Body something like this. Notice that we used [$token$] as a placeholder. This is automatically replaced at runtime by ZS-HTTP Connection Manager.Passing Token in Body
   

   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:partner.soap.sforce.com">
      <soapenv:Body>
         <urn:sessionid>[$token$]</urn:sessionid>
      </soapenv:Body>
   </soapenv:Envelope>

   Passing Token in Header
   
   If your API expects token via HTTP Header (e.g. Authorization: xxxx-my-token-xxxxxxxx ) then make sure you configure Auth Scheme and Header name on HTTP Connection. By default it sends Token in Authorization header if you dont do anything.
   

   ZS SSIS XML Source Component: Enter SOAP Request Body

6. Select Content-Type as per your Request, here we are selecting

   text/XML;charset=UTF-8

     and adding 1 HTTP Header:

   SOAPAction: "https://zappysys.com/soap-getdata"

   

   ZS SSIS XML Source Component: Select Content-Type Add HTTP Request Header

7. select Filter Path: set as per your requirements. here for this example, we are setting:
   

   $.soapenv:Envelope.soapenv:Body.urn:Row[*]

   

   ZS SSIS XML Source Component: Select Filter

    

8. Check Include Parent Columns checkbox, for including parent Columns, and then click on Preview button to see the Preview data.
   

   SSIS XML Source Component: See the Preview data

9. That’s it. Click on OK button to Save your Configuration.
10. Finally drag any destination (i.e. OLEDB Destination) from your SSIS toolbox. If you want to load this data into SQL, then Drag OLE DB Destination Component, and configure it.

Conclusion

In this blog, we learned how to call any SOAP/REST API with Dynamic Token, retrieve Service response data with SSIS using XML Source Component OR JSON Source Component   in a very simple way. Furthermore, you can achieve many more functionalities with this tool. Check our blogs/articles on XML Source Component to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the call any SOAP / REST API with Dynamic Token feature with our tools:

• Landing Page for XML Source, you can also find Tutorial Video here.
• Help File: Documentation of XML Source.

The post How to call SOAP / REST API using Dynamic Token in SSIS / ODBC appeared first on ZappySys Blog.

REST API is becoming more and more common and with that you will see explosion in use of JSON data format. One of the questions we get a lot is “How to extract or read array from JSON data file” or “How to read multiple arrays from JSON data“. Example : Store JSON document has Customers[…] and Employees[…] arrays and lets say you want to extract both and save to target SQL server database. In this post you will learn how to make this possible and how to make it fast with some performance tips. We will use SSIS JSON Source (File, REST API Connector)

If you want to learn how to parse Multi-Dimensional JSON arrays then check this article.

Video Tutorial – Read from multiple arrays from JSON document

Read array from JSON document

To extract subdocument or array content from JSON document you need to use JSON Path expression. Lets check following example JSON document.

{
  "store": {
    "employees": [
      {
        "name": "bob",
        "hiredate": "2015-01-01"
      },
      {
        "name": "sam",
        "hiredate": "2015-01-02"
      },
      {
        "name": "ken",
        "hiredate": "2015-01-03"
      }
    ],
    "books": [
      {
        "category": "reference",
        "author": "bob",
        "title": "hellooo1",
        "price": 1.95,
        "sections": [
          "s1",
          "s2",
          "s3"
        ]
      },
      {
        "category": "fiction",
        "author": "sam",
        "title": "hellooo2",
        "price": 1.96,
        "sections": [
          "s4",
          "s1",
          "s3"
        ]
      },
      {
        "category": "science",
        "author": "steve",
        "title": "hellooo3",
        "tag": "1bcd",
        "price": 11,
        "sections": [
          "s9",
          "s2",
          "s3"
        ]
      }
    ],
    "location": {
      "street": "123 Main St.",
      "city": "Newyork",
      "state": "GA"
    }
  }
}

Below is example filter expression to extract common elements

$.store.books[*]       //get all books for store              
$.store.employees[*]       //get all employees for store                 
$.store.books[*].sections[*]       //get all sections from all books   
$.store.books[*].author //get all authors of all books for store           
$.store.books[*]    //get all books for store  
$.store.books[2]    //get 3rd book record
$.store.books[:2]   //get first 2 books from the top
$.store.books[-2:]  //get last 2 books
$.store.books[?(@.price < 10)] //get books where price is less than 10
$.store.books[?(@.tag)] //filter all books with tag

Read single array from JSON data (JSON Path expression)

SSIS PowerPack comes with powerful REST API Connectors for JSON and XML . In below example screenshot you will see how to extract single array section from your JSON File. But if you have multiple array sections (e.g. books and employees from above example) then you have to use Multiple JSON Source Connectors for same JSON File.

Read multiple arrays from JSON data

If you have scenario to extract multiple arrays from same JSON (e.g. extract Employees and Books from store ) then you have two options. You can pick any approach which suites your need (Method-1 is recommended).

1. Method-1: Use single JSON Source with [Output As Document] option and feed that to JSON Parser Transform
2. Method-2: Use multiple JSON Sources with different filter expressions
3. Method-3: Enable Array Flattening Option (Useful if you have smaller arrays – i.e. less than 10 items) – Version v2.7.6+

Now lets discuss each approach in detail.

Method-1: Use single JSON Source along with JSON Parser Transform

SSIS PowerPack v2.2 released new JSON Parser Transform and XML Parser Transform which can be used to parse any input JSON/XML data coming from upstream in pipeline. It parse input string  column into multiple columns how JSON/XML Source does but the only difference is JSON Parser takes String as input from existing upstream column.

Here are the steps to parse multiple JSON arrays using JSON Parser Transform. For example purpose we will use sample store json listed above.

1. Drag new data flow in SSIS Control Flow designer. Double click on Data flow
   
2. Drag ZS JSON Source from SSIS Toolbox. Configure Source, Filter and other property.
   
3. Check [Output As Document] option (Found under Access Mode dropdown). Click preview. From grid copy one sample document from __DOCUMENT__ column. (Right click on Cell and Copy).This is raw JSON which we will use later to feed as sample in JSON Parser Transform (If you cant copy JSON then obtain some other way)
   

   Enable Output As Raw Document Option – SSIS JSON Source

4. Click OK to save JSON Source.
5. Drag Multicast Transform (Native SSIS Transform) from Toolbox
   

   Drag & Drop SSIS MultiCast Transform from Toolbox on designer surface

6. Connect JSON Source to Multicast Transform
7. Drag first JSON Parser Transform from Toolbox. Rename it to [Parse Employees]. Connect blue arrow coming from Multicast to JSON Parser Transform
8. Drag second JSON Parser Transform from Toolbox. Rename it to [Parse Books]. Connect blue arrow coming from Multicast to JSON Parser Transform
9. Double click [Parse Employees]. Select __DOCUMENT__ as upstream column. Enter sample JSON you obtained in step#3. Click on select Filter and highlight Employees node (i.e. $.store.employees[*] ). Click preview to see data.
   

   Use JSON Parser Transform to parse Raw JSON document

10. Very Important:Click Columns tab and verify metadata (e.g. datatype, length). If you think you need extra columns or bigger column length then change in the grid and check [Lock] option found in the last grid column. If column length is smaller than output data then you will get error message when you run package. You can review error message carefully to determine which column needs to be adjusted  (Repeat this until you clear all errors).
11. Click OK to save [Parse Employees].
12. Double click [Parse Books]. Select __DOCUMENT__ as upstream column. Enter sample JSON you obtained in step#3. Click on select Filter and highlight Books node (i.e. $.store.books[*] ). Click ok. Click preview to see data.
13. Perform same steps as #11 for  [Parse Books] to set correct metadata.
14. Click OK to save [Parse Books].
15. Drag ZS Trash destination from Toolbox. Connect Trash destination to [Parse Employees]
16. Drag ZS Trash destination from Toolbox. Connect Trash destination to [Parse Books]
17. Run the package and you will see both 2 outputs from different JSON Parser with different row count.

Here is how it will look like

Extract multiple array output from JSON file / REST API response in (SSIS JSON Source, JSON Parser Transform)

Method-2: Use multiple JSON Sources with different filter expressions

You can consider using multiple JSON Source (each source with unique filter expression). Since one JSON source can only extract one array or Document.. You have to use below technique to read multiple arrays from same JSON document.

Read array from JSON data – Reading multiple array or single array

Method-3: Use Array Flattening Option (Preferred for smaller arrays)

If you have array inside extracted record and you like to flatten it so each items of array becomes column then use newly introduced feature [Enable Array Flattening]

Consider JSON like below. What if you don’t like to store every detail from below JSON in just one table (i.e. rather than splitting in 3 tables Customers, CustomerHobbies , CustomerAddresses ?) . To achieve full denormalization ZappySys offers powerful yet simple one click option.

{
	"Customer" : [{
			"Name" : "John",
			"Hobby" : ["Painting","Reading","Swimming"],
			"Addresses" : [
				{"Street":"100 Main St","ZipCode":"10062"}, 
				{"Street":"101 Fire St","ZipCode":"11121"}
			]
		}, {
			"Name" : "Peter",
			"Hobby" : ["Hiking","Golf"],
			"Addresses" : [
				{"Street":"200 Main St","ZipCode":"20062"}, 
				{"Street":"201 Fire St","ZipCode":"33321"}
			]
		}
	]
}

To enable flattening check below option on Filter

JSON Array Flattening Option in SSIS JSON Source

Column after Array Flattening Option

Using JSON REST API Connector to parse JSON data

If you want to access JSON data from REST API then you can use same JSON Source Connector.  All you have to do is rather than local file path c:\data\xyz.json change it to URL as below and you will be able to access and filter JSON data using same technique as above.

Get JSON data from URL (REST API JSON response)

JSON Path for recursive data extract (Double dot expression)

There will be a time when you dont know how many levels deep you have to go for data extract. See below example. We will to get all employees id and name from hierarchy. For that you can use double dot like below. As you can see there are many nested levels and we want to extract id and name from each level so data can be

Expected Output

id      name
-------------------------
100	Jeff (CEO)
200	Bob (VP Sales)
300	Kim (VP HR)
400	Kumar (VP Dev)
210	Ron (Sales Mgr)	
220	Akash (Sales Mgr)	
410	Den (Sr. Dev)	
420	Den (Sr. Dev)
411	Den (Jr. Dev)
412	Sita (Jr. Dev)

Sample Data

{
  "employees": [
    {
      "id": 100,
      "name": "Jeff (CEO)",
      "employees": [
        {
          "id": 200,
          "name": "Bob (VP Sales)",
          "employees": [
            {
              "id": 210,
              "name": "Ron (Sales Mgr)"
            },
            {
              "id": 220,
              "name": "Akash (Sales Mgr)"
            }
          ]
        },
        {
          "id": 300,
          "name": "Kim (VP HR)"
        },
        {
          "id": 400,
          "name": "Kumar (VP Dev)",
          "employees": [
            {
              "id": 410,
              "name": "Den (Sr. Dev)",
              "employees": [
                {
                  "id": 411,
                  "name": "Den (Jr. Dev)"
                },
                {
                  "id": 412,
                  "name": "Sita (Jr. Dev)"
                }
              ]
            },
            {
              "id": 420,
              "name": "Den (Sr. Dev)"
            }
          ]
        }
      ]
    }
  ]
}

Configure JSON Source for Recursive Children Scan using Double Dot JSONPath

Open JSON Source and configure like below.

Set Filter as $..employees[*] (Notice how we used double dots in expression. This does recursive scan for N number of levels)

JSON Path for recursive scan of nested array (many levels deep)

Parse Multi-Dimensional Array

If you want to learn more how to parse Multi-Dimensional JSON arrays then check this article full length article.

Sample JSON with Multi Dimensional array

{
  "columns" : ["Id", "FirstName", "IsActive"],
  "rows" : [ [1,"bob",true], [2,"sam",false], [3,"joe",true] ]
}

Array Transformation Options for 2D array

Sample JSON with Multi Dimensional array (No columns in document)

{
  "rows" : [ [1,"bob",true], [2,"sam",false], [3,"joe",true] ]
}

Array Transformation Options for 2D array

Performance Tips

In this section we will describe how to make your things faster.

Reduce number of requests to server by changing page size

If you are doing REST API paging to get full resultset then make sure you adjust page size to appropriate number so total requests sent to server is reduced. Each request to server adds overhead. Most of API supports page size parameter. Refer to your API documentation how to adjust page size.

Avoid multiple REST API Round trips by caching response

If you extracting multiple arrays from REST API response using above technique then we recommend you to use REST API Task so save response to File or variable first and then use cached JSON data multiple times. This approach is not as friendly as others but it can certainly reduce your total API requests.

Use GZip compression

If your REST API supports GZip compression then check Enable Gzip on Advanced tab of SSIS JSON Source (File, REST API Connector). This will compress response in Gzip format from server and when it comes to your server it will be automatically decompressed by JSON Source connector. This seeds up thing significantly if you have lots of data to download.

Uncheck include parent columns if not needed

By default JSON Source connect includes parent column (Level above last property in filter) from the Filter path you specify. For Example  if you specify $.store.employees[*] and “Include parent columns” checked then It will extract all employees and then also include parent level columns for store (Tip: Parent column name start with P_ by default) . If you don’t need parent columns then uncheck from output, this reduces many steps during data extract process.

Download sample package

Click here to download SSIS 2012 Package (Will work in SSIS 2014, 2016)

Conclusion

In this post you learned how to extract data from JSON array and how to make data extract fast. Download SSIS JSON Source (File, REST API Connector) to try everything listed above.

The post Reading JSON Arrays from file / REST API appeared first on ZappySys Blog.

JSONPath expression is an expression language to filter JSON Data. Its a similar concept to XPath expression in XML but has limited features compared to XPath. If you are extracting data from REST API Services using SSIS JSON / XML Source or ODBC JSON / XML Driver then you will quickly realize that it’s very important to extract nested data by navigating to a certain level.

Sample data for JSONPath expression

All examples of JSONPath expression in this post will use the following sample JSON for reference.

{
  "store": {
    "books": [
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95,
        "sections": ["s1", "s2", "s3"],
        "publicationDay": "2023-05-10"
      },
      {
        "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99,
        "sections": ["s4", "s1", "s3"],
        "publicationDay": "2022-08-10"
      },
      {
        "category": "fiction",
        "author": "Herman Melville",
        "title": "Moby Dick",
        "isbn": "0-553-21311-3",
        "price": 8.99,
        "sections": ["s1"],
        "publicationDay": "1851-10-11"
      },
      {
        "category": "fiction",
        "author": "J. R. R. Tolkien",
        "title": "The Lord of the Rings",
        "isbn": "0-395-19395-8",
        "price": 22.99,
        "sections": ["s8", "s9"],
        "publicationDay": "1954-07-12"
      }
    ],
    "time": {
      "starttime": "8:00",
      "endtime": "18:00"
    }
  }
}

JSONPath Examples

//get all books for store
$.store.books[*]
                     
//get all sections from all books
$.store.books[*].sections[*]     

//get all authors of all books for store
$.store.books[*].author  
             
//get 3rd book record 
$.store.books[2]    

//get first 2 books from the top
$.store.books[:2]   

//get last 2 books
$.store.books[-2:]
  
//get all books where author attribute equals to 'sam'
$.store.books[?(@author=='Nigel Rees')] 

//get all documents / sub documents (any level) where author attribute equals to 'sam'
$..[?(@author=='Nigel Rees')] 

//get books where price is less than 10
$.store.books[?(@.price<10)] 

//filter all books with tag
$.store.books[?(@.tag)]

//find all books which contains section s1 or s2 (Use of Logical operator OR ( || )
$.store.books[?(@.sections[*]=='s1' || @.sections[*]=='s2' )]

//find all books where first section is s1 and second section is s2 (Logical operator OR ( && )
$.store.books[?(@.sections[0]=='s1' && @.sections[1]=='s2' )]

// Find all books where the publication day is before 2000-01-01
$.store.books[?(@.publicationDay < '2000-01-01')]

JSON Path – Using Regular Expressions (Regex)

The new version of SSIS PowerPack and ODBC PowerPack Supports Regular expressions in the path. Learn more about Regular expressions check this article

//Using Regular Expression: get all books where author attribute contains 'Nigel' or 'Waugh'
//Learn more about regular expressions here : https://zappysys.com/blog/using-regular-expressions-in-ssis/
$.store.books[?(@author=~ /Nigel|Waugh/  )] 

//Using Regular Expression: get all books where category is 'reference' OR author attribute contains 'Nigel' or 'Waugh'
$.store.books[?(@category=='reference' || @author=~ /Nigel|Waugh/ )] 

//Using Regular Expression: get all books where category is 'reference' AND author attribute contains 'Nigel' or 'Waugh'
$.store.books[?(@category=='reference' && @author=~ /Nigel|Waugh/ )] 

//Using Regular Expression: get all books where author name either start with 'Nigel' (see ^ prefix) or ends with 'Waugh' (see $ suffix)
//Learn more about regular expressions here : https://zappysys.com/blog/using-regular-expressions-in-ssis/
$.store.books[?(@author=~ /^Nigel|Waugh$/  )] 

//Using Regular Expression: get all books where author name starts with 'Nigel'
//Learn more about regular expressions here : https://zappysys.com/blog/using-regular-expressions-in-ssis/
$.store.books[?(@author=~ /^Nigel/  )] 

//Using Regular Expression: get all books where author name ends with 'Waugh'
//Learn more about regular expressions here : https://zappysys.com/blog/using-regular-expressions-in-ssis/
$.store.books[?(@author=~ /Waugh$/  )]

JSON Path – Nested Scan for any level (double dots)

Now lets look at example of below JSON. Assume that you like to extract information from all nested “children”  nodes (basically extract id and name of every child from any level).

You can use double dots without specifying parent heirarchy like below to scan recusrsivly.

$..children[*]

{
  "children": [
    {
      "id": 1,
      "name": "c1",
      "children": [
        {
          "id": 2,
          "name": "c2",
          "children": [
            {
              "id": 3,
              "name": "c3",
              "children": [
                { "id": 4, "name": "c4" },
                { "id": 5, "name": "c5" }
              ]
            }
          ]
        }
      ]
    }
  ]
}

For this you can write following JSON Path

$..children[*]

OR (below to find all children where name is c1 or c2 or c3)

$..children[?(@name=~ /c1|c2|c3/ )]

Here is an example of a JSON Driver in ODBC. The same expression can be used in SSIS too.

ODBC JSON Driver – Using Filter with Regular Expressions / JSONPath

JSONPath for Root Objects in Array

Now lets look at another pattern of JSON where your documents are on the first level rather than under some array property. Root filter only works if you have a version released after Dec-29-2021. Previous version didn’t support filtering on Root Array.

Example JSON

[
 { id:1 , color:'red'},
 { id:2 , color:'red'},
 { id:3 , color:'blue'},
 { id:4 , color:'green'}
]

Example JSON Path Filters

$.[0]  //Extract first document
$..[?(@color=='red')]   //Recursive scan (any level) - Extract all documents where color=red use 2 dots after $ for filter

NOTE: using two dots after $ (e.g. $..[zzzz]) invokes nested scan so any level array will return documents, this may not be the acceptable case sometimes but if you want to filter the root array then this is the only way for now in ZappySys.

If you must not allow recursive scan then you can use the following two properties

EnableRawOutputModeSingleRow = true;
RawOutputDataRowTemplate = “{data: [$1] }”  –where [$1] is replaced at runtime with Document Text you like to parse (can be static or URL response)

OR

Use functions in RawOutputDataRowTemplate if needed

RawOutputDataRowTemplate = “{data: <<[$1],FUN_BASE64DEC>>}”

The above will wrap around the actual JSON / XML you need to parse so now after this setting you can use JSONPath = $.data[?(@id==1)] to avoid two dots to prevent a recursive scan

The post JSONPath Expression Examples – JSON Cheatsheet (SSIS / ODBC) appeared first on ZappySys Blog.