In our previous post, we saw how to read Salesforce data using SSIS Salesforce Source (Which uses SOAP API under the hood). This may be slow if you have millions of rows because Salesforce Source paginates every 2000 rows, so it’s not a good option to read millions of rows to iterate on a large Salesforce dataset (Table or Query). To increase the read speed, you can use BULK API. This article explains how to read salesforce data using BULK API and increase read speed.

NEW Version (v5.5+)

In the new version (v5.5), we have added an option to query in Bulk Mode. This means if you have millions of Salesforce object rows to read, no worries; it may be very fast in the new version. Previously, it used to take a long time because it read in batches of 2000 rows in each request. With the Bulk API, the batch size is really large.

Here is how to set Salesforce Bulk API Mode in SELECT query (Read operation).

1. Install new version of SSIS PowerPack. Make sure it is v5.5 or higher.
2. Open existing Salesforce Source UI
3. Make sure you check [Use Bulk API] option like below. if you do not see this option then most likely you are using older version.
   -OR-
   Just add #bulkmode prefix in the SQL query like below, it will use Bulk API
   

   Bulk API mode for SSIS Salesforce Source (Read Large amount of data using SOQL Query)

OLD Version

If you are using an older version of SSIS Power Pack (v5.4.0 or older), you can follow the steps mentioned in the following section.

We will see how to query using Salesforce Bulk API in SSIS. It involves three simple steps like the one below.

1. Call BULK API – Submit SOQL Query and receive a JobId back (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

2. Keep checking Job Status Until Status is JobComplete (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

3. Read Query Result  (Use SSIS CSV Source)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Now, let’s look at everything in detail.

Obtain Salesforce Credentials

The first step to call Salesforce API is to obtain credentials (security token or OAuth ClientId /Secret)

Method#1 – Check this article to use UserId/ Password / SecretToken
Method#2 – Check this article to use the OAuth Connection Option

Create Salesforce Connection

Once we have Salesforce Credentials, we can create a new Salesforce Connection (You can do this from Task UI or the below way)

1. Open Visual Studio ( Assuming you already have SSIS installed along with ZappySys PowerPack )
2. Create a New SSIS Package Project.
3. Right-click on the Connection Manager panel and click on New Connection.
   
4. Select ZS-SALESFORCE Connection Manager from the Connection Managers list and Click on the Add Button.
   
5. Now in Connection Manager UI, Enter your SalesForce Credentials (OAuth Client Id/. Secret –or— User Name, Password and Security Token), and leave all other properties as it is.
   

Call Create Job – Retrieve / Save JobId

Once the connection is created, we can create our first step. Basically, we will call this API to create Job (Bulk Read)

1. Here, in Visual Studio, drag and drop the ZS Salesforce API Task on the design surface.
2. Double click the Task and configure Request Settings Tab like below
   – Select Salesforce Connection (Created in the previous section)
   – Check Use direct URL
   – Enter URL like below (Replace {your-instance} with your own instance (i.e. na9) . For more information on how to find out my
   salesforce instance id check this article
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

   – Request Method POST
   – Enter Body like below (You can use your own SOQL query). Keep in mind SQOL must be encoded for JSON format. This means the entire SQL must be in one line (replace new lines with \r\n , quotes with \” and tabs with \t)
   For example, to execute SELECT Id, Name FROM Accounts, you can write something like the one below in BODY.
   

   {
     "operation": "query",
     "query": "SELECT Id, Name\r\nFROM Account"
   }

   The above is for below (with a new line)
   

   SELECT Id, Name
   FROM Account

   

   Create Job – Call Bulk API to Read / Query using SOQL

3. Now Click Test Request (Copy the JobId which we will use in the Variable for Designtime Testing)
4. On the Response Tab, configure it like the below.
   – Select Response content Type = Json
   – Enter $.id as the expression
   – Check Save response option
   – Select Save to Variable and click New to create a variable. Name it JobId and enter a sample JobId value (copied from the previous step)
   

   Extract Salesforce JobId save to SSIS Variable

Do Status Check – Look for JobComplete

Once we create a job and store JobId in the variable we have to keep checking Job status every X seconds. Don’t worry you don’t have to use Loop or timer control for that. ZappySys made it so simple to check such thing without Loop /Timer.

1. Drag and drop the ZS Salesforce API Task in the design surface.
2. Double click and configure like below. URL will be different this time. Use variable as below and replace your instance ID
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

   

   Check Salesforce Job Status Periodically (Without Loop + Timer Controls)

3. Now go to the Status Check Tab and configure it, as shown below. It will keep looking for the words below in the response to decide whether it is a Success or Failure. If those words are not found, they will be checked every 5 seconds (Maximum 1200 secs).
   – Check Enable Status Check option
   – For Success Value – Enter JobComplete
   -Check Fail task option
   – Enter Cancel|Failed|Abort in the text box
   

   Configure Job Status Check Loop using Salesforce API Task

Read data (Compressed CSV Format)

Last step is read the result from completed Salesforce Job. Its in CSV format so we will use Advanced SSIS CSV Source which supports API connection to Salesforce. For that, we will use the same Salesforce API Task used before.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
2. Double click on the DataFlow task to see DataFlow designer surface.
3. Here, In Visual Studio, drag and drop the ZS CSV Source (Web API or File) in the design panel
   
4. Double click on ZS CSV Source (Web API or File) Configure it like below.
   – Enter URL to read result like this (You can supply OPTIONAL maxRecords parameters to control how much data comes in a single response)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results
   --OR--
   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results?maxRecords=50000

   – Check Use Credentials and select Salesforce Connection
   – Click Preview (assuming JobId variable has a valid completed JobId)
   

   Read Salesforce Bulk API Job Result – Using SSIS CSV / API Source

5. Now, set up Pagination as below to read all records using an automated loop. You will need VERSION Released after Jun 20, 2022 to use Regular Expression feature in Header Value extractionBy default, Salesforce doesn’t send all records in response so we need to setup pagination. In below example we are reading Sforce-Locator attribute from Response and sending it to the next URL as locator=<continue-token> value
   – Choose Response Header Contains Continue Token 
   – Enter below expression in Page Num Indicator
   

   locator=Sforce-Locator(^((?!null\b).)*$)

   Basically, the above expression will keep looping until we find Sforce-Locator : null in the response header.
   

   Pagination for Salesforce Bulk Query API (Get Header Value from Sforce-Locator)

6. Click OK to save and now run entire package like below.

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Conclusion

In this article we saw how to use various ZappySys Components / Task to achieve complex Salesforce API calls to read Query data using Bulk API. Download SSIS PowerPack and try yourself and also explore 70+ other tasks /components not discussed in this article.

The post How to query salesforce data using Bulk API in SSIS appeared first on ZappySys Blog.

In our last post (Regex Cheat Sheet) we explained use cases of SSIS Regular Expression Parser Task . Now lets look at some real world usecase. Basically in this article we are going to Extract data from audit log using SSIS. For demo purpose we will use log generated by SSIS Export CSV File Task output. When you export many tables dynamically using this task you want to know how many Row count exported for each Table. This information is logged in SSIS Output Log but no easy way to capture in SSIS variable. So what we will do we will use few tricks to capture that data using Regular Expression usecase.

So let’s get started.

Setup Export CSV Task (Output Multiple Tables to CSV Files Dynamically)

For example purpose we will use Export CSV File Task but you can use this technique for any other Tasks.

1. Drag and Drop ZS Export CSV Task from SSIS Toolbox and configure as below.
   
2.   Configure Export CSV Task to output multiple tables as explained here
3. Now create 2 SSIS User Variables
   1. FilePath (String Type)
   2. RowCount (String Type)

That’s it… now in the next section we will setup event handler to capture output from Log.

Setup EventHandle / Extract Audit Data using Regex

1. Now select task and click Event Handler tab
2. Select Executable Name from the dropdown, Select Event Name OnInformation from Handler Dropdown
3. Click Create Hyper Link to Create Event Handler like below
   

   Create Task Level Event Handler in SSIS

4. Now Drag below 3 tasks and connect like this
   1. Script Task (We will use as dummy start (No configuration needed)
   2. ZS Regular Expression Parser Task
   3. ZS Logging Task (Optional) but we will use to Output Extracted Values
      
5. Connect Dummy Script Task to ZS Regular Expression Parser Task and Right click on Green connected Arrow and Use Expression as below. Click OK to save
6. Configure ZS Regular Expression Parser Task as below
   1. Enter

      {{System::ErrorDescription}}

        as Direct Value
   2. Enter Below two mappings
      1. For RowCount set
         

         Total (\d+) records written to : (.*){{1,1}}

      2. For FilePath set
         

         Total (\d+) records written to : (.*){{1,2}}

7. Here is how it will look like
   

   Extract data from audit log using SSIS Regular Expression Parser Task

8. Thats it… Now connect 2nd step to last Logging Task…. in that Task you can set Text like below just to log extracted data. You can also use Execute SQL Task to log into Database Table
   

   {{User::FilePath}} ===> {{User::RowCount}}

9. Now run Package and you will see audit data we extracted for each file. Once File is exported we capture OnInformation event… and if Message contains substring like “records written to” then we extract data into variable for Row Count and Path then you can save to desired place for auditing purpose

The post Extract audit data from SSIS Execution Log (EventHandler and Regular Expression) appeared first on ZappySys Blog.

In this short article, you will learn how to write Regular expressions in SSIS (i.e. Regex) and what tool to use to test them. You will also find helpful resources on how to write more sophisticated expressions and learn more about them. For demo purposes, we will use FREE SSIS Regex Parser Task to parse and extract the text using Regex.

You can use Regular expressions in several SSIS PowerPack connectors:

• SSIS Regex Parser Task (FREE),
• Azure Blob Storage Task,
• Advanced File System Task,
• Amazon S3 Storage Task and others.

Writing Regular Expressions in SSIS

Based on the SSIS component you use it will include the filename in filtering options or match the text: See below Syntax to write Regex in ZappySys tools. We support additional construct {{X, Y}} at the end of Regex to control two parameters. This additional construct is useful for data extracted from matching regex. If you want to Test patterns and not worry about data extraction then no need to use last {{

Occurance_Index

Group_Index_Or_Name

Syntax:

<your Regular Expression>[{{Occurance_Index|*[,Group_Index_Or_Name]}}]

Where:

Occurance_Index

Group_Index_Or_Name

How to name a group?

E.g. (\w+)@(?<domain>\w+.com)

How to use the group name in the match extract?

E.g. (\w+)@(?<domain>\w+.com){{0,domain}}

Example Input:

Let’s assume we have the following input text. We will test various Expressions.

Customer => AAA
Email => aaa@google.com
Phone => 101-222-3333
========
Customer => BBB
Email => bbb@yahoo.com
Phone => 102-222-3333
========
Customer => CCC
Email => ccc@hotmail.com
Phone => 103-222-3333
========
Customer => DDD
Email => ddd@outlook.com
Phone => 104-222-3333

Sample Regex Expressions

(.*)

More Regular Expression Examples

Regex Examples (Using SSIS Regular Expression Parser Task)

Here is an example how Regex

(.+)_(.+)_(.+){{1,2}}

Using Regular Expressions in SSIS Regex Parser Task (Extract Groups)

Using Groups / Occurrence Index

Some tasks like SSIS Regex Parser Task (FREE) supports extracting value from specific occurrence and specific part of matched pattern using special syntax at the end of your pattern (see below).

Your Regex Pattern Here{{OccuranceIndex,GroupIndex}}

Where :
OccuranceIndex is 0 based (0=extract first occurrence)
GroupIndex is 0 based (0=extract first matching group from pattern. First group is always entire text. )

See above screenshot in previous section for example.

Tools

The best tool we’ve found to write and test Regex is Regex Hero (will require IE with Silverlight if you want to use it online, in the browser):

Another great site for Regex testing is https://regex101.com (Works in any browser unlike previous one)

and few more sites as below

http://www.regexr.com/
http://www.regexlib.com/
http://www.regular-expressions.info/

Resources

Regular Expressions cheat-sheet to hang on the wall

Regular Expressions quick reference

The post Using Regular Expressions in SSIS appeared first on ZappySys Blog.

Many times you have requirement to search and replace in files (single or multiple file) content using SSIS. If you are not C# or VB.net programmer then you may find yourself at dead end struggling how to achieve this in SSIS. If you have this need then you can use Advanced File System Task

Search and replace in files using Simple text search

In this option you can specify text you want to search and text you want to replace with. You can check ignore text if you dont want case-sensitive search/replace.

Extended text search (special characters allowed)

For extended search mode you may use any of the following in your search/replace string

• \n – for line feed
• \r – for return character
• \0 – for null character
• \t – for tab character
• \v – for vertical tab character

Example:
Search: “Hello\r\nWorld”
Replace: “Hello World”

Regular expression search/replace

Search and Replace File Content in SSIS (Find sub string, Search for special character)

Advanced File System Task

If you want to perform some other operations then here is the list of few features supported by Advanced File System Task.

• File operations for single or multiple files using wildcard(e.g. copy, move, rename, delete, write to file, get properties)
• Folder operations such as delete folder and create folder
• Support for bulk rename (e.g. by specifying {%name%}.{%ext%}.old as new name will rename all selected files to *.old)
• Search and Replace in files (Regular expression search supported)
• Change file encoding
• Get file properties as data table which you can loop easily (Using ForEach Loop – ADO Enumerator)
• Option to continue on error (e.g. Copy file only when source file found)
• Auto create missing target directory
• Use placeholders anywhere in the path, makes it easy to timestamp folders or files (e.g. c:\Archive\{{System::StartTime,yyyMMdd}})
• Regular Expression based Include Pattern (e.g. ^(file\d\d).(\w+)$ will include file01.exe,file22.csv …)
• Regular Expression based Exclude Pattern (e.g. (.exe|.msi) will exclude files with extension exe or msi)
• Sort based on specific property (e.g. Order Size DESC – extract top 10 files by size)
• Advanced SQL like filter support (e.g. Size> 1024 AND AgeInDays > 10 AND Extension IN(‘.txt’,’.csv’) will include file greater than 1024 bytes and extension is txt or csv)
• Support for Regular Expression based Include Pattern (e.g. ^(Customer\d\d).(\w+)$ )
• Support for SQL Server 2005, 2008, 2012, 2014, 2016 (32 bit and 64 bit)

Download FREE SSIS Components

Conclusion

Search and replace in many files can be tricky but using Advanced File System Task it can be few click. Click here  to download.

The post Search and replace in files using SSIS – No Coding appeared first on ZappySys Blog.

JSONPath expression is an expression language to filter JSON Data. Its a similar concept to XPath expression in XML but has limited features compared to XPath. If you are extracting data from REST API Services using SSIS JSON / XML Source or ODBC JSON / XML Driver then you will quickly realize that it’s very important to extract nested data by navigating to a certain level.

Sample data for JSONPath expression

All examples of JSONPath expression in this post will use the following sample JSON for reference.

{
  "store": {
    "books": [
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95,
        "sections": ["s1", "s2", "s3"],
        "publicationDay": "2023-05-10"
      },
      {
        "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99,
        "sections": ["s4", "s1", "s3"],
        "publicationDay": "2022-08-10"
      },
      {
        "category": "fiction",
        "author": "Herman Melville",
        "title": "Moby Dick",
        "isbn": "0-553-21311-3",
        "price": 8.99,
        "sections": ["s1"],
        "publicationDay": "1851-10-11"
      },
      {
        "category": "fiction",
        "author": "J. R. R. Tolkien",
        "title": "The Lord of the Rings",
        "isbn": "0-395-19395-8",
        "price": 22.99,
        "sections": ["s8", "s9"],
        "publicationDay": "1954-07-12"
      }
    ],
    "time": {
      "starttime": "8:00",
      "endtime": "18:00"
    }
  }
}

JSONPath Examples

//get all books for store
$.store.books[*]
                     
//get all sections from all books
$.store.books[*].sections[*]     

//get all authors of all books for store
$.store.books[*].author  
             
//get 3rd book record 
$.store.books[2]    

//get first 2 books from the top
$.store.books[:2]   

//get last 2 books
$.store.books[-2:]
  
//get all books where author attribute equals to 'sam'
$.store.books[?(@author=='Nigel Rees')] 

//get all documents / sub documents (any level) where author attribute equals to 'sam'
$..[?(@author=='Nigel Rees')] 

//get books where price is less than 10
$.store.books[?(@.price<10)] 

//filter all books with tag
$.store.books[?(@.tag)]

//find all books which contains section s1 or s2 (Use of Logical operator OR ( || )
$.store.books[?(@.sections[*]=='s1' || @.sections[*]=='s2' )]

//find all books where first section is s1 and second section is s2 (Logical operator OR ( && )
$.store.books[?(@.sections[0]=='s1' && @.sections[1]=='s2' )]

// Find all books where the publication day is before 2000-01-01
$.store.books[?(@.publicationDay < '2000-01-01')]

JSON Path – Using Regular Expressions (Regex)

The new version of SSIS PowerPack and ODBC PowerPack Supports Regular expressions in the path. Learn more about Regular expressions check this article

//Using Regular Expression: get all books where author attribute contains 'Nigel' or 'Waugh'
//Learn more about regular expressions here : https://zappysys.com/blog/using-regular-expressions-in-ssis/
$.store.books[?(@author=~ /Nigel|Waugh/  )] 

//Using Regular Expression: get all books where category is 'reference' OR author attribute contains 'Nigel' or 'Waugh'
$.store.books[?(@category=='reference' || @author=~ /Nigel|Waugh/ )] 

//Using Regular Expression: get all books where category is 'reference' AND author attribute contains 'Nigel' or 'Waugh'
$.store.books[?(@category=='reference' && @author=~ /Nigel|Waugh/ )] 

//Using Regular Expression: get all books where author name either start with 'Nigel' (see ^ prefix) or ends with 'Waugh' (see $ suffix)
//Learn more about regular expressions here : https://zappysys.com/blog/using-regular-expressions-in-ssis/
$.store.books[?(@author=~ /^Nigel|Waugh$/  )] 

//Using Regular Expression: get all books where author name starts with 'Nigel'
//Learn more about regular expressions here : https://zappysys.com/blog/using-regular-expressions-in-ssis/
$.store.books[?(@author=~ /^Nigel/  )] 

//Using Regular Expression: get all books where author name ends with 'Waugh'
//Learn more about regular expressions here : https://zappysys.com/blog/using-regular-expressions-in-ssis/
$.store.books[?(@author=~ /Waugh$/  )]

JSON Path – Nested Scan for any level (double dots)

Now lets look at example of below JSON. Assume that you like to extract information from all nested “children”  nodes (basically extract id and name of every child from any level).

You can use double dots without specifying parent heirarchy like below to scan recusrsivly.

$..children[*]

{
  "children": [
    {
      "id": 1,
      "name": "c1",
      "children": [
        {
          "id": 2,
          "name": "c2",
          "children": [
            {
              "id": 3,
              "name": "c3",
              "children": [
                { "id": 4, "name": "c4" },
                { "id": 5, "name": "c5" }
              ]
            }
          ]
        }
      ]
    }
  ]
}

For this you can write following JSON Path

$..children[*]

OR (below to find all children where name is c1 or c2 or c3)

$..children[?(@name=~ /c1|c2|c3/ )]

Here is an example of a JSON Driver in ODBC. The same expression can be used in SSIS too.

ODBC JSON Driver – Using Filter with Regular Expressions / JSONPath

JSONPath for Root Objects in Array

Now lets look at another pattern of JSON where your documents are on the first level rather than under some array property. Root filter only works if you have a version released after Dec-29-2021. Previous version didn’t support filtering on Root Array.

Example JSON

[
 { id:1 , color:'red'},
 { id:2 , color:'red'},
 { id:3 , color:'blue'},
 { id:4 , color:'green'}
]

Example JSON Path Filters

$.[0]  //Extract first document
$..[?(@color=='red')]   //Recursive scan (any level) - Extract all documents where color=red use 2 dots after $ for filter

NOTE: using two dots after $ (e.g. $..[zzzz]) invokes nested scan so any level array will return documents, this may not be the acceptable case sometimes but if you want to filter the root array then this is the only way for now in ZappySys.

If you must not allow recursive scan then you can use the following two properties

EnableRawOutputModeSingleRow = true;
RawOutputDataRowTemplate = “{data: [$1] }”  –where [$1] is replaced at runtime with Document Text you like to parse (can be static or URL response)

OR

Use functions in RawOutputDataRowTemplate if needed

RawOutputDataRowTemplate = “{data: <<[$1],FUN_BASE64DEC>>}”

The above will wrap around the actual JSON / XML you need to parse so now after this setting you can use JSONPath = $.data[?(@id==1)] to avoid two dots to prevent a recursive scan

The post JSONPath Expression Examples – JSON Cheatsheet (SSIS / ODBC) appeared first on ZappySys Blog.