MongoDB comes with SSL support, which can secure your end-to-end communication. However, setting up a MongoDB SSL Connection may require some configuration on both sides (i.e., Server and Client side). In this post, we aim to secure your MongoDB Integration in SSIS.

Now, let’s look at the steps to configure SSL in MongoDB.

Configure SSL Support in MongoDB Server

To configure MongoDB for SSL connectivity, the very first step is to generate a self-signed certificate in *.pem format (This must be stored on the server)

Download and Install OpenSSL

First step is to download and install OpenSSL. Assuming you are running a 64 Bit Windows OS, you can download the 64 Bit Lite version of OpenSSL (about 3 MB). If you install it using default settings, then all binaries will be installed under C:\OpenSSL-Win64\bin\

Create a self-signed certificate and *.pem file using OpenSSL.

If you don’t have a trust issue and want to use a self-signed certificate rather than buying from a 3rd-party SSL Certificate Authority, you can create a self-signed certificate using the OpenSSL Command line as shown below. Now let’s look at how to generate a private key, a certificate file, and finally the *.pem file, which is the only required file by MongoDB.

A PEM file is the most common file format (container file format), which includes a Certificate and a Private Key in a single file.

Let’s look at how to create a PEM file using OpenSSL.

1. First, run the command below. When prompted, enter “passphrase” (in our case, enter “test”).
   

   openssl genrsa -des3 -out server.key 2048

2. Now run the command below to generate a private key (*.key file)
   

   openssl rsa -in server.key -out server.key

3. After the private key is created, run the command below to create a CSR file (i.e., “Certificate Signing Request”). This file is needed to generate an actual certificate (Self-Signed or Signed by a 3rd Party, such as VeriSign or RapidSSL). If you omit the sub argument from below, then it may ask you a few questions (e.g, company name, department, etc), but for testing purposes, let’s make it easy 🙂
   To enter full details or just provide CN=xxxx
   

   openssl req -sha256 -new -key server.key -out server.csr -subj "/C=US/ST=Georgia/L=Atlanta/O=ZappySys LLC/OU=Org/E=testuser@zappysys.com/CN=localhost"
   -- or --
   openssl req -sha256 -new -key server.key -out server.csr -subj "/CN=localhost"

4. Now, let’s run the command below to generate a certificate file (*.crt). We are setting the expiration to 365 days; you can adjust as per your needs.
   

   openssl x509 -req -sha256 -days 365 -in server.csr -signkey server.key -out server.crt

5. Now, the last step is to combine the certificate (*.crt) and private key (*.key) files into a single PEM file using the command below. It will generate the desired *.pem file format, which can be used by the MongoDB server (In Unix OS, you can use the cat command rather than typing)
   

   type server.crt server.key > cert.pem

Once done, your cert.pem file will look like the following (Open and view in Notepad)

-----BEGIN CERTIFICATE-----
MIIDjjCCAnYCCQCDKO4/pCBfqTANBgkqhkiG9w0BAQsFADCBiDELMAkGA1UEBhMC
VVMxCzAJBgNVBAgMAkdBMRAwDgYDVQQHDAdBdGxhbnRhMREwDwYDVQQKDAhaYXBw
eVN5czELMAkGA1UECwwCSVQxFTATBgNVBAMMDHphcHB5c3lzLmNvbTEjMCEGCSqG
RF5DmNx6gKlRWlOFmokMCz7wAglErDL4h/I3w6iZaVVkBmeDj30cms/fQ+upAI8U
KrBpNEwOglU9fjqjpnW0u8qGpXIk3NfdCm+6Lr91GL/u/9+gmSYw5YCCF6kYaRA/
2mqLXZ3Xp6WfTfVyQskKMB9D1+7QljqBZmFnTjLCaJ9MdJzzMpFyWPRB4Ix1kAzV
yZk=
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAyo++fr54Wh8anfEDZrr5O1NXwrPZW6KI3dPQ8FVHVDcBPETi
VX3qGcq8vywXH/LdwMEC4EQGED+zo50GmQdNiIzUfsnDbkq6rkqVNKjh7agkDlip
5vK3yuHtYAVp+laJ8X8kKISRttY7kcnl9gSkfTilwpN8//49G45Nx0JFu3o3Ortb
H4FPWjidQBAA6hPxoQi2h7YfEfpICL1RwYDCaQpdwMEJJqzcvaHCpdYVYd5PHLks
xRi8iwKBgA/S5w4/pwrzq0Jab/Afzf8AleZm/xLmXFpglgZF25AtVhpnayoHiTIk
9BdCXydi0ZOg4LtMqkSwt427UWCEnoRRH1WbKH2F9ZRWaELw/ix9RrZgTNmF821F
1/ILtO+QrN8tFnrLX3NAWurxUJxvzhBnvq40ZIHCyWvhRXUTkzTH
-----END RSA PRIVATE KEY-----

Edit the MongoDB Server config file (v3.0 or higher)

Now that we have the *.pem file, we can use it to launch a MongoDB server instance. Before we do that, one more step is needed. We have to specify a few parameters in the MongoDB config file.

1. Navigate to the MongoDB installation directory (where mongod.exe is found)
2. Create an empty text file called mongo-config.txt and enter the following text (assuming cer.pem, mongo-config.txt, and all mongod.exe are in the same folder). The password was tested when we created the PEM file. If you used a different one, then change it below.
   mongo-config.txt
   

   net:  
       bindIp: localhost
       ssl:
           mode: requireSSL
           PEMKeyFile: cert.pem
           PEMKeyPassword: test
           allowInvalidCertificates: true
           allowInvalidHostnames: true

Start the MongoDB Server using the Config file

Once you create a config file, we can start the MongoDB instance using the following command line to use the new config file with SSL mode enabled. That’s it, your server is now accepting Secure connections using SSL.

mongod -f "mongod-config.txt"

Using the SSL option in the SSIS MongoDB Connection

Now, let’s look at the steps to secure the MongoDB connection in SSIS.

Prerequisites

Before we look at the Step-By-Step section on securing the MongoDB Connection in SSIS, let’s make sure you meet the following requirements.

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from the Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have basic familiarity with MongoDB.

Using SSL in the SSIS MongoDB Connection Manager

To enable SSL in SSIS, open the MongoDB connection manager and go to the SSL Tab. Check the Enable SSL option as shown below. Once you do this, and if your MongoDB server is configured correctly to use SSL, then your connection Test will be green, and any traffic between you and your MongoDB server will be encrypted using SSL.

SSIS MongoDB Connection – SSL Settings

Using X509 Client Authentication in SSIS MongoDB Connection Manager

Now, let’s look at how to authenticate using the X509 Client Certificate (X509) method. For this method, you must use the PFX certificate format (do not use PEM). To create a PFX file from the PEM we created earlier, you can run the following command. When prompted, use the same password you used before to generate a private key.

openssl pkcs12 -export -out cert.pfx -inkey server.key -in server.crt

To learn more about the whole process of how to configure the MongoDB Server to use Client X509 Certificate, check this Article.

Let’s look at the steps to configure the SSIS MongoDB Connection Manager to use X509 Client Certificate Mode.

1. Open ZappySys MongoDB Connection Manager UI
2. Enter the User ID you created in the $external database (This UserID usually matches the Subject of your Certificate). For example, in our case, UserID may appear as follows. Leave the password blank because it’s not used in the X. 509 method.
   

   emailAddress=user@myemail.com,CN=127.0.0.1,OU=UNIT,O=DEMO,L=city,ST=test,C=AU

   

   Configure SSIS MongoDB Connection for Client X509 Certificate (Set UserID of $external database)
3.  Now go to the SSL Tab. Check SSL Option, You can check Ignore SSL Validation error (If it’s a signed certificate)
4. Select the PFX file path and enter the certificate password if needed. If you type Path, then Subject Line appears automatically (You can use it as UserID field if you were unsure in Step#1 however, we still recommend making sure UserID is an  exact match based on the original name your System Admin created in $external DB)
   

   Configure SSIS MongoDB Connection for Client X509 Certificate (Select PFX File – Specify Password)
5. Now go to the Advanced Tab and enter the Below Option. By default, MongoDB Connection Manager uses the userid/password mode. To use X509 Auth mode, you must set an option in the Advanced Options table as shown below.
   

   authMechanism=MONGODB-X509

   

   Configure SSIS MongoDB Connection for Client X509 Certificate (Specify authMechanism=MONGODB-X509)
6. Click Test connection to make sure it’s successful.

The post How to configure MongoDB SSL Connection in SSIS appeared first on ZappySys Blog.

In our previous post we explained how to read/load MongoDB data in SSIS. This post covers specifically how to parse MongoDB date time stored inside your MongoDB documents. By default any well known date formats (e.g. ISO date) will be parsed as valid datetime (e.g. DT_DBTIMESTAMP) when you use SSIS MongoDB Source. But if you have date stored as other format (e.g. MM-dd-yyyy) then system will not parse as string unless you specify custom date format on JSON Datetime tab. If you want to learn How to query Date in MongoDB then read this article

Parse MongoDB Date time using SSIS MongoDB Source

Perform the following steps to enable custom date format parsing for date not stored as ISODate in MongoDB. Below steps assume you have Downloaded and Installed SSIS PowerPack from Here

NOTE: Below steps not necessary if your date is stored as ISODate in MongoDB…. e.g.  { “OrderDate” : ISODate(“2015-12-31T00:00:00Z”)  }

1. Open MongoDB Source. Change Mode to Query Mode
2. Check Enable JSON option. You don’t have to type filter (its optional)
3. Goto JSON Options and then Datetime options Tab
4. Enter custom date format (The way your date stored in MongoDB)

Now go to columns and click refresh columns. It will detect column as DT_DBTIMESTAMP (e.g. date/time)

SSIS MongoDB Source- Date Time Options ( Parse MongoDB Date)

SSIS MongoDB Source – Query and Parse MongoDB Date with Custom format

The post How to parse MongoDB Date time in SSIS appeared first on ZappySys Blog.

Steps – Extract multiple arrays from MongoDB

1. Extract data using MongoDB source (Select output mode as Raw Document)
2. Connect MongoDB Source to Multicast Transform
3. Connect multi cast output to JSON Parser Transform which will take upstream raw JSON doc and convert to Flat table output just like MongoDB source. On Json Parser specify filter e.g. $.doc.Array1[*]
4. Connect Json Parser Transform output to desired target e.g. sql server/flat file.
5. Repeat step #3,4,5 for  as many outputs you need with different filter settings (e.g. $.doc.Array2[*]).

Here is more info about JSON Parser Transform (This is new Transform added in v2.2)

//zappysys.com/products/ssis-powerpack/ssis-json-parser-transform/

Video Tutorial – Step By Step – How to read multiple MongoDB arrays

Below video tutorial demonstrates following things

* How to execute MongoDB Shell commands in SSIS (e.g. create Collection)
* How to read data from multiple arrays from MongoDB document (i.e. Orders, Products)
* Save result into two SQL Server tables

The post How to extract multiple arrays from MongoDB using SSIS appeared first on ZappySys Blog.

In this post you will learn how to write flexible MongoDB Aggregation Queries in SSIS (i.e. Group By Query) using SSIS MongoDB Source Component. MongoDB provides powerful Aggregation Pipeline Engine which is conceptually same as writing Group By queries in traditional SQL world but its not exactly same. SSIS MongoDB Source returns nested MongoDB JSON documents into flat looking table with rows and column (denormalized data).

If you want to learn how to write data to MongoDB (Insert, Update, Delete, Upsert) then click here. If you want learn how to read data from MongoDB then click here.

So let’s explore concepts of MongoDB Aggregation Pipeline and see some MongoDB query examples for SSIS use case.

SSIS MongoDB Source Introduction

SSIS MongoDB Source supports three modes to read data from MongoDB collection. Each mode also supports JSONPath Expression to extract nested array.

1. Table Mode
2. SQL Query Mode
3. JSON Query Mode (Native)

To read more about supported SQL Syntax read help file here.

Using SSIS MongoDB Source to Query Data (Aggregation)

Now lets look at how to use SSIS MongoDB Source to query MongoDB data using Aggregation Engine.

MongoDB Aggregation Pipeline Syntax in SSIS

SSIS MongoDB Source support all pipeline commands to write aggregation queries but its slighly different syntax than shell command so your SHell Query won’t work directly but you can copy some part of your Shell query. Check below Syntax.

MongoDB Native Query Syntax (Shell Command) for Aggregation:

db.YourCollectionName.aggregate(
[
[ <pipeline-stage1>]
[,<pipeline-stage2>]
...
...
[,<pipeline-stageN>]
]
)

SSIS MongoDB Source Query Syntax for Aggregation:


{Table:YourTableName ,Method:aggregate [,MaxRows: N] [,AllowDiskUse:true] [,AllowCursor:true] [,Timeout:#####] }
{
pipeline: [
[ <pipeline-stage1>]
[,<pipeline-stage2>]
...
...
[,<pipeline-stageN>]
]
}

Example Aggregation Queries (SQL to SSIS Mapping)

Assume that we want to write following SQL Query using SSIS MongoDB Source or using MongoDB Shell

SELECT State as StateAbbr,SUM(population) as TotalPopulation 
FROM ZipCodes 
WHERE status='Active' 
HAVING SUM(population) > 25
Order By TotalPopulation DESC

Above SQL Query can be translated to below JSON Query which can be used in SSIS MongoDB Source. As we noted before Below syntax is not same as Shell command so it will only run in SSIS MongoDB Source.

{Table: ZipCodes,Method: aggregate}
{
  pipeline: 
  [
    {$match : { status: "A" } } ,
    {$group : {_id :  "$state", TotalPopulation: { $sum: "$pop" }}},
    {$match: { TotalPopulation: { $gt: 25 } } },
    {$project : { StateAbbr: "$_id" , _id: 0, TotalPopulation:1 } },
    {$sort : {TotalPopulation : -1}}
  ]
}

So as you saw in above query various clause of SELECT queries are defined as pipeline stages. You can add any possible stages in different order. Based on their order it will be applied so order of stage is very important.  For more information about various pipeline stages click here. Below is high level translation of SQL Query to MongoDB Aggregation Query.

SSIS MongoDB Source – Example of MongoDB Group By Query (Aggregation Framework – Pipeline)

SQL to MongoDB Aggregation Mapping

Now let’s look at few examples how to translate traditional SQL  Queries to JSON Query for SSIS MongoDB Source. We will use sample data we inserted in previous section.

Creating Sample Data for MongoDB Query Examples

For this exercise to write sample queries using below data. So let’s first create a collection called Order and insert following records using your favourite MongoDB client tool.

db.Orders.insert(
  {
    "OrderId" : 1000,
    "CustomerId" : "CUS001",
    "OrderTotal" : 2000.50,
    "Quantity" : 20,
    "OrderDate" : ISODate("2010-01-01"),
    "Country" : "USA",
    "Items" : [ {ItemId:10, UnitPrice: 100.00,Quantity: 10}, {ItemId:11, UnitPrice: 200.10,Quantity: 5}],
    "PaymentInfo" : { 
      "Method" : "Card",
      "Currency" : "USD" 
    }
  }
)

db.Orders.insert(
  {
    "OrderId" : 1001,
    "CustomerId" : "CUS001",
    "OrderTotal" : 1000.00,
    "Quantity" : 10,
    "OrderDate" : ISODate("2011-01-02"),
    "Country" : "USA",
    "Items" : [ {ItemId:10, UnitPrice: 100.00,Quantity: 10}],
    "PaymentInfo" : { 
      "Method" : "Card",
      "Currency" : "USD" 
    }
  }
)

db.Orders.insert(
  {
    "OrderId" : 1002,
    "CustomerId" : "CUS002",
    "OrderTotal" : 1000.00,
    "Quantity" : 5,
    "OrderDate" : ISODate("2012-01-01"),
    "Country" : "USA",
    "Items" : [ {ItemId:13, UnitPrice: 200.00,Quantity: 5}],
    "PaymentInfo" : { 
      "Method" : "Bank",
      "Currency" : "USD" 
    }
  }
)

db.Orders.insert(
  {
    "OrderId" : 1003,
    "CustomerId" : "CUS003",
    "OrderTotal" : 59000.00,
    "Quantity" : 10,
    "OrderDate" : ISODate("2005-01-02"),
    "Country" : "India",
    "Items" : [ {ItemId:14, UnitPrice: 5900.00,Quantity: 10}],
    "PaymentInfo" : { 
      "Method" : "Check",
      "Currency" : "INR" 
    }
  }
)

Here is how it would look like (We used RoboMongo i.e. Robo 3T) but you can use mongo.exe command line to issue above commands.

Inserting Example data for MongoDB Aggregation Queries (RoboMongo / Robo 3T)

Example 1 – MongoDB Aggregation Query with COUNT(*) – $group, $project

Here is how to write COUNT(*) query in MongoDB. Below query will find order count by country. Each query must output _id field and you can refer Document column using $ prefix (i.e. “$Country” or for nested object use dot separator “$PaymentInfo.Method” ) . To output _id column with different name we have used Country: “$_id”. Because we including copy of _id we can remove _id column by doing  _id: 0.  To include column in output use YourColumanName: 1 in $project stage.

SQL Query:

SELECT Country,Count(*) as OrderCount 
FROM Orders
GROUP BY Country

Mongo Shell Query:

db.Orders.aggregate(
 [
   {$group : {_id : "$Country",OrderCount: { $sum: 1 }}} 
  ,{$project : {Country: "$_id", _id: 0, OrderCount: 1 }}
 ]
)

SSIS MongoDB Source Query:

{Table: Orders,Method: aggregate} 
{ pipeline: 
   [
    {$group : {_id : "$Country",OrderCount: { $sum: 1 }}} 
   ,{$project : {Country: "$_id", _id: 0, OrderCount: 1 }}
  ]
}

Example 2 – MongoDB Aggregation Query with WHERE and ORDER BY- $match, $group, $project, $sort

Here is how to write COUNT(*) query in MongoDB. Below query will find order count by country.

SQL Query:

SELECT Country,Count(*) as OrderCount 
WHERE Quantity > 1
FROM Orders
GROUP BY Country
ORDER BY Country

Mongo Shell Query:

db.Orders.aggregate(
 [
   {$match : {Quantity : { $gt: 1 }}} 
  ,{$group : {_id : "$Country",OrderCount: { $sum: 1 }}} 
  ,{$project : {Country: "$_id", _id: 0, OrderCount: 1 }}
  ,{$sort : {Country: 1}}
 ]
)

SSIS MongoDB Source Query:

{Table: Orders,Method: aggregate} 
{ pipeline: 
   [
     {$match : {Quantity : { $gt: 1 }}} 
    ,{$group : {_id : "$Country",OrderCount: { $sum: 1 }}} 
    ,{$project : {Country: "$_id", _id: 0, OrderCount: 1 }} ,{$sort : {Country: 1}}  
   ]
}

Example 3 – MongoDB Aggregation Query with WHERE and HAVING Clause – $match, $group

Here is an example of WHERE and HAVING clause. They both use $match stage but their orders are different. WHERE condition is applied before GROUP BY (i.e. $group) and HAVING condition is applied after GROUP BY.

SQL Query:

SELECT Country,Count(*) as OrderCount
FROM Orders 
WHERE Quantity > 1
GROUP BY Country
HAVING Count(*) > 0

Mongo Shell Query:

db.Orders.aggregate(
 [ 
   {$match : {Quantity: { $gt: 1 }}}
  ,{$group : {_id : "$Country",OrderCount: { $sum: 1 }}} 
  ,{$match : {OrderCount: { $gte: 1 }}} 
  ,{$project : {Country: "$_id", _id: 0, OrderCount: 1 }}
 ]
)

SSIS MongoDB Source Query:

{Table: Orders,Method: aggregate} 
{ pipeline: 
   [
      {
        $group : {_id : "$Country",OrderCount: { $sum: 1 }}
      }      
   ]
}

Example 4 – MongoDB Aggregation Query with $unwind (Flatten nested Array)

Now lets look at an example of $unwind operator. $unwind can be used to perform operations on nested array. Common things would be flatten the hierarchy and then apply group by operation. Lets find out Average Quantity of Items By Order Number

SQL Query:

SELECT Orders.OrderId as OrderNumber,Avg(Items.Quantity) as AvgItemQty
FROM Orders 
JOIN Items ON Orders.OrderId = Items.OrderId
GROUP BY Orders.OrderId

Mongo Shell Query:

db.Orders.aggregate(
 [ 
   {$unwind : "$Items"}
  ,{$group : {_id : "$OrderId",AvgItemQty: { $avg: "$Items.Quantity" }}} 
  ,{$project : {OrderNumber: "$_id", _id: 0, AvgItemQty: 1 }}
 ]
)

SSIS MongoDB Source Query:

{Table: Orders,Method: aggregate} 
{ pipeline: 
  [ 
    {$unwind : "$Items"}
   ,{$group : {_id : "$OrderId",AvgItemQty: { $avg: "$Items.Quantity" }}} 
   ,{$project : {OrderNumber: "$_id", _id: 0, AvgItemQty: 1 }}
  ]
)

MongoDB Query with Dynamic Value / ISO DateTime

There will be a time when you have to supply Dynamic Value from Variable or supply Dynamic Date time inside your query. Dates are little different than other data types. Check this article to learn more about passing dates in MongoDB Query.

For Dynamic Date use SSIS Variables Format Specifiers

Example of DateTime query (Static Value)

{Table: Orders,Method: aggregate} 
{ pipeline: 
   [
     {$match : {OrderDate : { $gt: new ISODate("2010-01-01") }}} 
    ,{$group : {_id : "$Country",OrderCount: { $sum: 1 }}} 
    ,{$project : {Country: "$_id", _id: 0, OrderCount: 1 }} ,{$sort : {Country: 1}}  
   ]
}

Example of using Dynamic Value from Variable

To pass date from SSIS Variable you can use Format Specifiers. (e.g. yyyy-MM-dd  or for full ISO date yyyy-MM-ddTHH:mm:ssZ )

{Table: Orders,Method: aggregate} 
{ pipeline: 
   [
     {$match : {OrderDate : { $gt: new ISODate("{{User::varOrderDate,yyyy-MM-dd}}") }}} 
    ,{$group : {_id : "$Country",OrderCount: { $sum: 1 }}} 
    ,{$project : {Country: "$_id", _id: 0, OrderCount: 1 }} ,{$sort : {Country: 1}}  
   ]
}

Conclusion

MongoDB is one of the most popular NoSQL / Document Storage Engine. SSIS PowerPack make super easy and efficient to read/write data in MongoDB without any coding effort. Download SSIS PowerPack to try out MongoDB features along with 70+ other components .

Search terms:
mongodb group by count
mongodb group by multiple fields
mongodb group by date
mongodb group example
mongodb group sort
mongodb group by month
mongodb group by having
mongodb group distinct
mongodb group by sum
mongodb aggregate
mongodb aggregate sort
mongodb aggregate unwind

The post How to write MongoDB Aggregation Queries in SSIS (Group By) appeared first on ZappySys Blog.

In this post you will learn how to read MongoDB data from Array. We recently introduced new JSON Path expression feature in SSIS MongoDB Source Connector to extract nested information from MongoDB documents.

Read documents from Array in MongoDB

By default SSIS MongoDB Source Connector doesn’t support extracting data using JSON Path expression. So if you want to read nested data from sub-document or array then you have enable that option.

Consider following sample Document stored in MongoDB. Lets say you want to extract all orders for each customer.

For that you have enter $.Orders[*] in the JSON Path expression as below screenshots. Notice it also brings parent columns of each Orders (By default parent columns prefixed with P_ ).

To exclude parent columns from your resultset got to Advanced JSON Filter Options tab -> Uncheck “Include Parent Columns” option.

Sample MongoDB Document

{
    "_id" : ObjectId("56a2ae6369e0f137a449f58b"),
    "CustomerID" : "ALFKI",
    "CompanyName" : "Alfreds Futterkiste",
    "ContactName" : "Maria Anders",
    "ContactTitle" : "Sales Representative",
    "LocationInfo" : {
        "City" : "Berlin",
        "Address" : "Obere Str. 57",
        "Region" : null,
        "PostalCode" : "12209",
        "Country" : "Germany"
    },
    "Phone" : "030-0074321",
    "Fax" : "030-0076545",
    "OrdersIDList" : [ 
        10643, 
        10692, 
        10702
    ],
    "Orders" : [ 
        {
            "OrderID" : 10643,
            "EmployeeID" : null,
            "OrderDate" : "1997-08-25 00:00:00"
        }, 
        {
            "OrderID" : 10692,
            "EmployeeID" : 4,
            "OrderDate" : "1997-10-03 00:00:00"
        }, 
        {
            "OrderID" : 10702,
            "EmployeeID" : 4,
            "OrderDate" : "1997-10-13 00:00:00"
        }
    ]
}

Read MongoDB Data from array (extract nested sub documents)

Server side Filtering using MongoDB Query

JSON Path filtering happens on client side so first your selected set of documents fetched from MongoDB server and then transformation of happens on client side based on JSON Path you specified (de-normalization).

Below screenshots shows how to use MongoDB SQL Query Language (Read more here) to filter data on server side and then apply JSON Path expression to extract sub-documents or array items from MongoDB documents.

MongoDB Server Side Filter – Extract MongoDB Data from Array or nested sub-documents using SQL Query and JSON Path

The post Read MongoDB data from array (extract nested sub documents) appeared first on ZappySys Blog.

ZappySys provides high-performance drag-and-drop connectors for MongoDB Integration. In our previous post, we discussed how to query/load MongoDB data (Insert, Update, Delete, Upsert). In this post, you will see how to query MongoDB by date (or ISODate) using the SSIS MongoDB Source. To see full details about possible query syntax, see online help.

MongoDB doesn’t have SQL Like query language so sometime you may feel odd…But has native JSON-like Query language (Sorry, SQL Guys.. )

No worry if you stumble upon how to query various things in MongoDB. It’s different but not too hard to learn.

MongoDB Query Example

Assume that you have the following Document stored in a MongoDB Collection. You want to query orders where the Order date is greater than a specific date. Here are some examples of MongoDB queries used by ZappySys SSIS MongoDB Source Connector. For more information about MongoDB Query, check the MongoDB Query Syntax documentation. An important thing is that your attribute must be stored as ISODate (not string). See the JSON document below; notice how OrderDate is saved as ISODate(…). This will allow Query to use any standard comparison operator (e.g. =, >, <,  != ) for date search.  If your data is not stored as ISODate, refer to online blogs/help on how to write update scripts to convert string dates to ISODate.

{
	"OrderID" : 100,
	"OrderDate" : ISODate("2015-02-28T00:00:00Z"),
	"Country" : "USA",
	"Total" : 5000
}

How to Query MongoDB ISODate Attribute

If you want to query the ISODate attribute, you may scratch your head about how to do it. There are two ways to do

Using SQL Query syntax (Only available in ZappySys SSIS MongoDB Source)

select * from Orders where OrderDate > 'DATE(2015-01-01)'

Using SSIS Variable in Query
You can also use an SSIS Variable to make your query dynamic. Variable placeholder supports datetime format specifiers too. E.g., for an ISO date with 3-digit millisecond precision, you may use yyyy-MM-ddTHH:mm:ss.fffZ format as below.

select * from Orders where OrderDate < 'DATE({{User::MyDateVar,yyyy-MM-ddTHH:mm:ss.fffZ}})'

Using ISODate syntax (SSIS MongoDB Source – Native Query)

{Table: Orders }
{
	"OrderDate" : { "$lt" : new ISODate("2015-01-02T00:16:15.184Z")}
}

Using $date syntax (SSIS MongoDB Source – Native Query)

{Table: Orders }
{
	"OrderDate" : { "$gt" : { "$date" : "2015-01-01"}}
}

Remember to suffix “Z” in your date format to indicate its UTC date

MongoDB stores datetime values in UTC (to confirm this, edit a Mongo document and you will see a Z at the end in the ISODate(….) value). When you use ZappySys MongoDB SQL Query Syntax like below, it may or may not perform Automatic Date Conversion from Local to UTC. If you append “Z” to a date/time value, it’s treated as UTC, and no DateTime conversion occurs. Dates without a Z at the end are treated as Local Date/Time, so they’re converted to UTC before being passed to MongoDB.

If you use JSON Style Native Query Syntax, ZappySys does not perform any date conversion.

Query-1: Date supplied as UTC (Suffixed with “Z”) – Date is passed as is to the Mongo engine

select * from Orders where OrderDate > 'DATE({{User::MyDateVar,yyyy-MM-ddTHH:mm:ss.fffZ}})'
select * from Orders where OrderDate > 'DATE({{User::MyDateVar,yyyy-MM-ddZ}})'

Query-2: Date supplied as Local Time (No “Z” at the end) – Date is converted to UTC and then passed to the Mongo Engine

select * from Orders where OrderDate > 'DATE({{User::MyDateVar,yyyy-MM-ddTHH:mm:ss.fff}})'
select * from Orders where OrderDate > 'DATE({{User::MyDateVar,yyyy-MM-dd}})'

MongoDB Query using SSIS MongoDB Source

ZappySys SSIS MongoDB Source supports querying nested Documents using JSON Path expressions (e.g., Extract order Items from Order Document using $.Items expression)

Example MongoDB Document

{
    "_id" : ObjectId("564bb3bea1e57e043363f1ea"),
    "OrderID" : 111,
    "OrderDate" : ISODate("2022-01-01T00:00:00.000Z"),
    "AuditInfo" : {
        "User" : "bob",
        "CDate" : ISODate("2015-01-01T00:00:00.000Z")
    },
    "Items" : [ 
        {
            "ProductID" : 100,
            "Qty" : 3
        }, 
        {
            "ProductID" : 101,
            "Qty" : 2
        }
    ]
}

MongoDB Source – SQL style MongoDB Query (filter by date)

Use the DATE function to convert a string value to an ISODate

SSIS MongoDB Source – MongoDB SQL Query – Filter by date (Query ISODate)

MongoDB Source – Native MongoDB Query  (filter by date)

Query MongoDB by Date Attribute (Query ISODate) using SSIS MongoDB Source

Dynamic MongoDB Query using Placeholders

MongoDB Source in SSIS PowerPack supports placeholders to make queries dynamic. You can use Query Placeholders like below (See Insert Placeholder menu)

Dynamic MongoDB Query – Use SSIS variable placeholder

SQL Like Query Language using SSIS PowerPack

SSIS PowerPack includes an SSIS MongoDB Source that supports a SQL-like query language for MongoDB. It automatically converts SQL queries to correct MongoDB JSON Query syntax. See the screenshot below. Here you can write an SQL Query. If you want to use a nested array element as a column name, you must use arrayname.elementname, e.g., LocationDetails.Country

MongoDB SQL Query Language – SSIS MongoDB Source

Download FREE SSIS Components

The post How to query MongoDB by date or ISODate appeared first on ZappySys Blog.