In our previous post, we discovered how to read and write MongoDB data using ZappySys MongoDB Source and Destination SSIS connectors. However, it was all about connecting to the on-premises hosted version of MongoDB. In this post, we will focus on connecting specifically to the Replica Set Cluster in MongoDB Atlas (a cloud database service by MongoDB). After you complete the steps listed below, you will be able to get and push data using the same MongoDB SSIS connectors, but this time — in MongoDB Atlas. Let’s not waste our precious time and begin!

MongoDB Atlas connection setup in SSIS

In this article, we are going to demonstrate how to create the MongoDB Connection Manager, which is used by MongoDB ExecuteSQL Task, MongoDB Source, and MongoDB Destination SSIS connectors. Let’s start by following these simple steps:

1. First, download and install SSIS PowerPack.
2. Once you installed SSIS PowerPack, open Visual Studio, and create a new SSIS project with a new SSIS package.
3. Then open the created package and right-click on the Connection Managers panel to create a new MongoDB connection. When a context menu appears, select the New Connection… menu item:
   
4. Next, select the ZS-MONGODB option from the connection manager list and click Add… button:
   
5. This will open the MongoDB Connection Manager configuration window. For now, let’s leave it open, and first go to your MongoDB Atlas account to get all MongoDB Atlas Cluster instance information for MongoDB Connection Manager configuration.
6. Once you are on the MongoDB Atlas landing page, continue by selecting the cluster you want to connect to (make sure you white-list your IP first). After the window appears, select the Compass option:
   

   Connecting to MongoDB Atlas Replica Set Cluster

7. Then proceed by simply copying and pasting the configuration information into the MongoDB Connection Manager we left to rest for a while. Move on by filling in your database credentials and the database you are connecting to. Also, don’t forget to enable SSL. Just patiently follow the numbers in orange circles below and you will be done faster than you think!
   

   Configuring MongoDB Atlas connection in SSIS by setting the host, user name, password, and database options

   

   Specifying MongoDB connection string options (i.e. replicaSet and authSource options)

   

   Enabling SSL Connection in MongoDB Connection Manager in SSIS (for encrypted data transfer)

8. Finally, press the Test Connection button to test the connection. If the test is successful, you can move on and start performing data operations on the MongoDB collections.

That’s it — you can now use the MongoDB ExecuteSQL Task, MongoDB Source, and MongoDB Destination as described in the below articles:

• How to call MongoDB JavaScript using SSIS
  
• How to read MongoDB data from an array (extract nested subdocuments)
• How to write MongoDB Aggregation Queries in SSIS (Group By)
• How to load SQL data into MongoDB (Insert, Upsert, Delete, Update)
• How to update MongoDB Array Items using SSIS

The post How to connect to MongoDB Atlas Replica Set Cluster in SSIS appeared first on ZappySys Blog.

In our previous blog post we saw how to update / delete mongodb array item. Now let’s look at how to Upsert MongoDB Array Items (i.e. Insert Item if not found in Array else Update existing record). Upsert into nested MongoDB Array requires two step process, unfortunately there is no easy way to do in a single step. You may see in below screenshot that first we will do Update Step for Existing MongoDB Array Items and then we will insert missing Records in second step. So let’s get started.

MongoDB Array Upsert – Update / Insert using Custom JOIN condition (such as $ne )

By default SSIS Mongodb Destination performs Lookup using $eq condition (Match rows using Equal operator). However there will be a time when you like to lookup and update target rows using custom join criteria (e.g. Use Not Equal condition – $ne ). V2.7.6  and later introduced new property called 

EnableCustomLookupQuery

Here is the description how to use custom Join using this new property.

Enables use of custom lookup query for Update / Upsert or Delete Operation. 
By default JOIN condition for target record match is generated automatically based on columns you supply in ColumnsForLookup property. 
However in some cases you need to supply custom condition for lookup to perform complex operations, in such case enable this option. 
When you enable this option you must supply map __DOCUMENT__ input column. String you supply to this column should be in this format 

<command>
  <query>YOUR_LOOKUP_QUERY</query>
  <document>YOUR_JSON_DOC</document>
</command>. 

Lookup query in <query> tag can be either Mongo JSON format (e.g. { \"CustomerID\" : \"AAA\", \"Orders.OrderID\" : { \"$ne\" : \"1000\" } }) 
OR
you can use ZappySys SQL query (e.g. select * from mytable where CustomerID='AAA' and [Orders.$.OrderID] != '1000' )

Now lets look at step by step. In below example we have a table called CustomerTest. We will load it with 2 records (With No orders). Then Update array using custom lookup condition, and later step we will insert record in array using custom lookup condition.

SSIS MongoDB Array Upsert Example (Update / Insert Array Items based on custom lookup condition)

Update Array using Custom Lookup Condition

1. Lets create few sample rows in MongoDB Collection. You can use following command in SSIS MongoDB Execute SQL Task
   

   {
    scope: 'database',
    db: 'Northwind',
    command: 'eval',
    args: 
    {
    code: 'db.CustomerTest.insert( [ {CustomerID:"AAA", Orders:[]}, {CustomerID:"BBB", Orders:[]}, {CustomerID:"CCC", Orders:[{OrderID:"1004","OrderDate" : "2008-02-05","Qty" : "5"} ]} ] )' 
    
    } 
   }

2. Now drag Data flow.
3. Drag ZS CSV Source or any other source. For example purpose we will configure CSV Source using Direct Value option with following Sample data. In the first run of package it will update only orders for CCC customer because only that customer will have records after table creation. one row. But in 2nd run all rows updated.
   

   CustomerID,Company,OrderID,OrderDate,Qty,Address
   AAA,Anthony Inc,1000,2008-01-01,90,Po Box 111\4612
   AAA,Anthony Inc,1001,2010-02-01,91,Po Box 111 / 4612
   BBB,Bob Inc,1002,2008-02-01,92,Po Box 222 / 4612
   BBB,Bob Inc,1003,2010-01-01,93,Po Box 222 / 4612
   CCC,Cindy Inc,1004,2010-01-07,2,Po Box 555 / 2345

    
4. Click OK to save Source UI
5. Now drag ZS Template Transform to build XML document (Data + Query). Connect Source to Template transform. You can also use ZS XML Generator Transform but for simplicity we will use Template Transform.
6. Double click to open Template Transform and enter text as below. Notice how we have entered MongoDB Query and Data in two separate XML nodes. First node is custom query for lookup. Second node is Operation we like to perform (e.g. $set in our case to update existing data). You can use Insert Variable and then select Columns option to insert placeholders. Also in some fields we used JSONENCODE function to make sure we escape double quote and slash correctly.
   

   <command>
   	<query><![CDATA[
   {
   	"CustomerID": "<%CustomerID%>", 
   	"Orders.OrderID": { "$eq" : "<%OrderID%>"}
   }
   ]]></query>
   	<document><![CDATA[
   {
     $set :
      { "CompanyName":"<%Company,JSONENCODE%>", 
        "Address":"<%Address,JSONENCODE%>", 
        "Orders.$.OrderID": "<%OrderID%>", 
        "Orders.$.OrderDate": "<%OrderDate%>", 
        "Orders.$.Qty": "<%Qty%>" 
       } 
     }
   }
   ]]></document>
   </command>

   

   Using ZS Template Transform to specify custom JOIN condition for MongoDB Update (Array Items)

7. Click OK to save Template Transform
8. Drag ZS MongoDB Destination. Connect Template Transform to Destination
9. Double click MongoDB Destination to edit. Set Connection, Table name, Operation=Update, EnableCustomLookupQuery=True
   

   Configure MongoDB Destination for Custom Join Condition (EnableCustomLookupQuery Setting)

10. Goto Mappings Tab and attach TemplateOutput to __DOCUMENT__
    

    MongoDB Mappings – Loading Document for Array Update (Custom Join Criteria)

11. Click OK to Save

Insert into Array using Custom Lookup Condition

Now let’s look at slightly modified steps to Insert into Array for new records. In this example we will insert Source records if CustomerID found but OrderID is not found. (See Template Transform Step). We will use Not Equal Condition this time.

1. Now drag Data flow rename it to something like [Add records to array]
2. Drag ZS CSV Source or any other source. For example purpose we will configure CSV Source using Direct Value option with following Sample data
   

   CustomerID,Company,OrderID,OrderDate,Qty
   AAA,Anthony Inc,1000,2008-01-01,30
   AAA,Anthony Inc,1001,2010-02-01,4
   BBB,Bob Inc,1002,2008-02-01,30
   BBB,Bob Inc,1003,2010-01-01,4

    
3. Click OK to save Source UI
4. Now drag ZS Template Transform to build XML document (Data + Query). Connect Source to Template transform. You can also use ZS XML Generator Transform but for simplicity we will use Template Transform.
5. Double click to open Template Transform and enter text as below. Notice how we have entered MongoDB Query and Data in two separate XML nodes. First node is custom query for lookup. Second node is Operation we like to perform (e.g. $addToSet in our case to insert into array )
   

   <command>
   <query>
   {"CustomerID": "<%CustomerID%>", "Orders.OrderID":{ "$ne" : "<%OrderID%>"}}
   </query>
   <document>{
     $addToSet :
      { Orders : {"OrderID": "<%OrderID%>", "OrderDate": "<%OrderDate%>", "Qty": "<%Qty%>" } }    
    }
   </document>
   </command>

6. Click OK to save Template Transform
7. Drag ZS MongoDB Destination. Connect Template Transform to Destination
8. Double click MongoDB Destination to edit. Set Connection, Table name, Operation=Update, EnableCustomLookupQuery=True
9. Goto Mappings Tab and attach TemplateOutput to __DOCUMENT__
10. Click OK to Save

Download Sample SSIS Package (2012 format) – MongoDB_Upsert_ArrayItem

Conclusion

MongoDB integration can be challenging if you are new to NoSQL world. If you are using SSIS as your primary ETL tool then not to worry because SSIS PowerPack can give super power needed to complete your project on time with drag and drop high performance connectors.

Keywords: ssis mongodb upsert array item | ssis mongodb update array item | ssis mongodb update array elements | mongodb update array documents | MongoDB $set operator | MongoDB $addToSet operator | MongoDB update sub document items | MongoDB CRUD operations | MongoDB Bulk Updates | MongoDB bulk updates to array items

The post Update or Insert – Upsert MongoDB Array Items using SSIS appeared first on ZappySys Blog.

MongoDB comes with SSL support, which can secure your end-to-end communication. However, setting up a MongoDB SSL Connection may require some configuration on both sides (i.e., Server and Client side). In this post, we aim to secure your MongoDB Integration in SSIS.

Now, let’s look at the steps to configure SSL in MongoDB.

Configure SSL Support in MongoDB Server

To configure MongoDB for SSL connectivity, the very first step is to generate a self-signed certificate in *.pem format (This must be stored on the server)

Download and Install OpenSSL

First step is to download and install OpenSSL. Assuming you are running a 64 Bit Windows OS, you can download the 64 Bit Lite version of OpenSSL (about 3 MB). If you install it using default settings, then all binaries will be installed under C:\OpenSSL-Win64\bin\

Create a self-signed certificate and *.pem file using OpenSSL.

If you don’t have a trust issue and want to use a self-signed certificate rather than buying from a 3rd-party SSL Certificate Authority, you can create a self-signed certificate using the OpenSSL Command line as shown below. Now let’s look at how to generate a private key, a certificate file, and finally the *.pem file, which is the only required file by MongoDB.

A PEM file is the most common file format (container file format), which includes a Certificate and a Private Key in a single file.

Let’s look at how to create a PEM file using OpenSSL.

1. First, run the command below. When prompted, enter “passphrase” (in our case, enter “test”).
   

   openssl genrsa -des3 -out server.key 2048

2. Now run the command below to generate a private key (*.key file)
   

   openssl rsa -in server.key -out server.key

3. After the private key is created, run the command below to create a CSR file (i.e., “Certificate Signing Request”). This file is needed to generate an actual certificate (Self-Signed or Signed by a 3rd Party, such as VeriSign or RapidSSL). If you omit the sub argument from below, then it may ask you a few questions (e.g, company name, department, etc), but for testing purposes, let’s make it easy 🙂
   To enter full details or just provide CN=xxxx
   

   openssl req -sha256 -new -key server.key -out server.csr -subj "/C=US/ST=Georgia/L=Atlanta/O=ZappySys LLC/OU=Org/E=testuser@zappysys.com/CN=localhost"
   -- or --
   openssl req -sha256 -new -key server.key -out server.csr -subj "/CN=localhost"

4. Now, let’s run the command below to generate a certificate file (*.crt). We are setting the expiration to 365 days; you can adjust as per your needs.
   

   openssl x509 -req -sha256 -days 365 -in server.csr -signkey server.key -out server.crt

5. Now, the last step is to combine the certificate (*.crt) and private key (*.key) files into a single PEM file using the command below. It will generate the desired *.pem file format, which can be used by the MongoDB server (In Unix OS, you can use the cat command rather than typing)
   

   type server.crt server.key > cert.pem

Once done, your cert.pem file will look like the following (Open and view in Notepad)

-----BEGIN CERTIFICATE-----
MIIDjjCCAnYCCQCDKO4/pCBfqTANBgkqhkiG9w0BAQsFADCBiDELMAkGA1UEBhMC
VVMxCzAJBgNVBAgMAkdBMRAwDgYDVQQHDAdBdGxhbnRhMREwDwYDVQQKDAhaYXBw
eVN5czELMAkGA1UECwwCSVQxFTATBgNVBAMMDHphcHB5c3lzLmNvbTEjMCEGCSqG
RF5DmNx6gKlRWlOFmokMCz7wAglErDL4h/I3w6iZaVVkBmeDj30cms/fQ+upAI8U
KrBpNEwOglU9fjqjpnW0u8qGpXIk3NfdCm+6Lr91GL/u/9+gmSYw5YCCF6kYaRA/
2mqLXZ3Xp6WfTfVyQskKMB9D1+7QljqBZmFnTjLCaJ9MdJzzMpFyWPRB4Ix1kAzV
yZk=
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAyo++fr54Wh8anfEDZrr5O1NXwrPZW6KI3dPQ8FVHVDcBPETi
VX3qGcq8vywXH/LdwMEC4EQGED+zo50GmQdNiIzUfsnDbkq6rkqVNKjh7agkDlip
5vK3yuHtYAVp+laJ8X8kKISRttY7kcnl9gSkfTilwpN8//49G45Nx0JFu3o3Ortb
H4FPWjidQBAA6hPxoQi2h7YfEfpICL1RwYDCaQpdwMEJJqzcvaHCpdYVYd5PHLks
xRi8iwKBgA/S5w4/pwrzq0Jab/Afzf8AleZm/xLmXFpglgZF25AtVhpnayoHiTIk
9BdCXydi0ZOg4LtMqkSwt427UWCEnoRRH1WbKH2F9ZRWaELw/ix9RrZgTNmF821F
1/ILtO+QrN8tFnrLX3NAWurxUJxvzhBnvq40ZIHCyWvhRXUTkzTH
-----END RSA PRIVATE KEY-----

Edit the MongoDB Server config file (v3.0 or higher)

Now that we have the *.pem file, we can use it to launch a MongoDB server instance. Before we do that, one more step is needed. We have to specify a few parameters in the MongoDB config file.

1. Navigate to the MongoDB installation directory (where mongod.exe is found)
2. Create an empty text file called mongo-config.txt and enter the following text (assuming cer.pem, mongo-config.txt, and all mongod.exe are in the same folder). The password was tested when we created the PEM file. If you used a different one, then change it below.
   mongo-config.txt
   

   net:  
       bindIp: localhost
       ssl:
           mode: requireSSL
           PEMKeyFile: cert.pem
           PEMKeyPassword: test
           allowInvalidCertificates: true
           allowInvalidHostnames: true

Start the MongoDB Server using the Config file

Once you create a config file, we can start the MongoDB instance using the following command line to use the new config file with SSL mode enabled. That’s it, your server is now accepting Secure connections using SSL.

mongod -f "mongod-config.txt"

Using the SSL option in the SSIS MongoDB Connection

Now, let’s look at the steps to secure the MongoDB connection in SSIS.

Prerequisites

Before we look at the Step-By-Step section on securing the MongoDB Connection in SSIS, let’s make sure you meet the following requirements.

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from the Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have basic familiarity with MongoDB.

Using SSL in the SSIS MongoDB Connection Manager

To enable SSL in SSIS, open the MongoDB connection manager and go to the SSL Tab. Check the Enable SSL option as shown below. Once you do this, and if your MongoDB server is configured correctly to use SSL, then your connection Test will be green, and any traffic between you and your MongoDB server will be encrypted using SSL.

SSIS MongoDB Connection – SSL Settings

Using X509 Client Authentication in SSIS MongoDB Connection Manager

Now, let’s look at how to authenticate using the X509 Client Certificate (X509) method. For this method, you must use the PFX certificate format (do not use PEM). To create a PFX file from the PEM we created earlier, you can run the following command. When prompted, use the same password you used before to generate a private key.

openssl pkcs12 -export -out cert.pfx -inkey server.key -in server.crt

To learn more about the whole process of how to configure the MongoDB Server to use Client X509 Certificate, check this Article.

Let’s look at the steps to configure the SSIS MongoDB Connection Manager to use X509 Client Certificate Mode.

1. Open ZappySys MongoDB Connection Manager UI
2. Enter the User ID you created in the $external database (This UserID usually matches the Subject of your Certificate). For example, in our case, UserID may appear as follows. Leave the password blank because it’s not used in the X. 509 method.
   

   emailAddress=user@myemail.com,CN=127.0.0.1,OU=UNIT,O=DEMO,L=city,ST=test,C=AU

   

   Configure SSIS MongoDB Connection for Client X509 Certificate (Set UserID of $external database)
3.  Now go to the SSL Tab. Check SSL Option, You can check Ignore SSL Validation error (If it’s a signed certificate)
4. Select the PFX file path and enter the certificate password if needed. If you type Path, then Subject Line appears automatically (You can use it as UserID field if you were unsure in Step#1 however, we still recommend making sure UserID is an  exact match based on the original name your System Admin created in $external DB)
   

   Configure SSIS MongoDB Connection for Client X509 Certificate (Select PFX File – Specify Password)
5. Now go to the Advanced Tab and enter the Below Option. By default, MongoDB Connection Manager uses the userid/password mode. To use X509 Auth mode, you must set an option in the Advanced Options table as shown below.
   

   authMechanism=MONGODB-X509

   

   Configure SSIS MongoDB Connection for Client X509 Certificate (Specify authMechanism=MONGODB-X509)
6. Click Test connection to make sure it’s successful.

The post How to configure MongoDB SSL Connection in SSIS appeared first on ZappySys Blog.

In our previous blog post we saw how to perform Read and Write operations in MongoDB using SSIS (i.e. Bulk Update, Delete, Upsert, Insert). In this post we specifically focus on how to update MongoDB Array items / elements using SSIS.

To make things simple to follow we have used JSON Source to produce sample JSON documents for MongoDB update (NOTE: Output as Document option is checked for JSON Source). However in real world you may have to use SSIS JSON Generator Transform to produce input documents for MongoDB. To learn more about generating JSON documents for MongoDB Load process check this blog post or watch video found on this page.

Video Tutorial – Example of insert/update documents inside MongoDB Array (Use $set, $push operator)

Basic Concepts: Performing MongoDB CRUD operations using SSIS

If you never heard term called CRUD stands for Create, Read, Update, Delete. These are the most common operations you have to do with any data source. SSIS PowerPack comes with many MongoDB Components to help you with CRUD operations. You can use one or more following components to achieve drag and drop MongoDB CRUD operations in SSIS. When you use SSIS PowerPack for MongoDB all operations will be Bulk Operations by default which means it will provide very high throughput.

• SSIS MongoDB Source
• SSIS MongoDB Destination
• SSIS MongoDB ExecuteSQL Task

Download Example SSIS Package

Here is the link to download Sample SSIS Package

SSIS Example : Update MongoDB Array Items / Elements (Update, Delete, Insert – Using $set, $pull, $push)

Update MongoDB Array Items ($set operator)

In next couple of sections we will see how to perform Update, Insert and Delete operations with MongoDB Array Items using various update operators and array operators. Consider the following Sample JSON documents for MongoDB. First we will see how to use MongoDB Shell commands to perform CRUD operations on MongoDB Array items.

{
    "_id" : ObjectId("5883ebe669e0f22ba890b49b"),
    "CustomerID" : "BOLID",
    "Name" : "Some name",
    "Orders" : [ 
        {
            "OrderID" : 200,
            "ShipCountry" : "USA",
            "City" : "Atlanta"
        }, 
        {
            "OrderID" : 201,
            "ShipCountry" : "USA",
            "City" : "New York"
        }
    ]
}

Update Array element using Shell command

Assume you want to update ShipCountry attribute of one of the Orders (OrderID=200) and for CustomerID=”BOLID” in above sample document.

Shell Command for Update Array Item

Use below shell command in your favorite MongoDB Client Tool (e.g. RoboMongo). Notice two things in below command because we will apply same concepts when we take this approach to SSIS. In below command Actual JSON document is wrapped inside. This is MongoDB Update operator. Click here to learn more about $set. Also notice that rather than nested Array syntax we used Orders.$.ShipCountry to update ShipCountry which is part of Array element. MongoDB provides special way to navigate array elements using   $ Operator (MongoDB Positional Update)

db.MyCollection.update(
	{ "CustomerID" : "BOLID", "Orders.OrderID" : 200 }, 
	{ $set : { "Orders.$.ShipCountry" : "Test1"} } , 
	{ upsert: false, multi: false}
)

After Update (MongoDB Document)

After you run above shell command notice all other attributes remain there in Array document and only ShipCountry is changed. This is because we used $ Operator (MongoDB Positional Update) rather than supplying Raw JSON for Array document (i.e. Orders : [ { … } ] ).

{
    "_id" : ObjectId("5883ebe669e0f22ba890b49b"),
    "CustomerID" : "BOLID",
    "Name" : "Some name",
    "Orders" : [ 
        {
            "OrderID" : 200,
            "ShipCountry" : "Test1",
            "City" : "Atlanta"
        }, 
        {
            "OrderID" : 201,
            "ShipCountry" : "USA",
            "City" : "New York"
        }
    ]
}

The issue with this approach its not Bulk operation. Assume that you have 100,000 records to update in various documents and its nested array items then row by row operation is not effective. To solve this issue we will use SSIS MongoDB Destination which performs Bulk operations (e.g. Bulk Update, Delete, Insert and Upsert).

Update MongoDB Array Items using SSIS

Now lets implement above scenario using SSIS. Assume that you have sample MongoDB document (as per above example – Before Update). We want to update ShipCountry for OrderID=200 for Customer BOLID. To do that perform following steps in SSIS. If you are updating Second level array (i.e. Root >> Orders >> OrderItems)  then you have follow slight different process. Check next section in this article about updating second level array (Insert items)

1. Download and Install SSIS PowerPack
2. Create new SSIS Project with one data flow task
   

   Dragging and dropping Data Flow Task into Control Flow

3. Drag ZS JSON Source and check Output as Document option (as below sceenshot). You can use JSON Source is your JSON is stored inside file or Variable or coming from Some API call which eventually go into MongoDB. You can also use JSON Source for quick testing where you Hardcode JSON to feed to MongoDB.
4. Enter the following JSON in the text area. Click OK to save.
   

   {"CustomerID":"BOLID", "Orders.$.OrderID":201, "Orders.$.ShipCountry": "Test1"}
   {"CustomerID":"BOLID", "Orders.$.OrderID":202, "Orders.$.ShipCountry": "Test2"}
   /**Below is extra record not found in destination - will be ignored in Update**/
   {"CustomerID":"BOLID", "Orders.$.OrderID":203, "Orders.$.ShipCountry": "Test3"}

5. Right click in Connections area and Create new ZS-MONGODB connection. Specify credentials and click Test before you hit OK to save.
6. Now drag ZS MongoDB Destination from SSIS Toolbox.
7. Select Runtime connection.
8. On the properties tab edit following properties
   1. Set ColumnsForLookup property as below (Notice how we use 2 columns for JOIN. Also used AS keywords for 2nd Column so we have <column-from-input-doc> AS <column-for-server-side-find>
      CustomerID,Orders.$.OrderID AS Orders.OrderID
   2. Set LoadOptions as below (Op is Update Operator you like to use. In our case its $set, Multi=True means if multiple match found then all matching documents will be updated). If you want to use multiple update operators then you have to supply that from Input document and use op=none option rather op=$set.
      op=$set;multi=true
   3. Set Operation property to Update
9. On the Mappings tab Map __DOCUMENT__ from upstream (In our case you will see same name column from JSON Source if you checked Output as Document option)
10. Click OK to save MongoDB Destination UI
11. Run package. You will notice after execution two orders of CustomerID=BOLID will be updated.

SSIS JSON Source – Read JSON (Raw Documents)

SSIS MongoDB Destination – Update MongoDB Array Item using $set update operator

Insert MongoDB Array Item ($push / $addToSet Operator)

Now lets look at how to insert new item into existing Array. Lets say we already have two orders for a customer and now we want to add new order. For this you can either use  $push operator or $addToSet operator. $push operator doesn’t check for duplicate item so its little faster. If you wish to skip append if item already exists (by matching all attributes) then use $addToSet operator.

Using Shell Command : Insert MongoDB Array Item (Append to array)

Here is the shell command which will push new item to the array

db.MyCollection.update(
	{CustomerID:"BOLID"},
	{$push:{Orders: {"OrderID":202,"ShipCountry":"USA","ShipCity":"Atlanta" } }},
	{upsert: false,multi: false}
)

Using Shell Command : Insert MongoDB Array Item at second level

Here is the shell command which will push new item to the array which is stored at second level e.g. { Orders :  [ { Items : [ insert-here ] } ] }

db.MyCollection.update(
	{CustomerID:"BOLID", "Orders.OrderID" : 202 },
	{$push:{ "Orders.$.Items" : {"ProductID":1001,"Quantity":3} } },
	{upsert: false,multi: false}
)

Using SSIS : Insert MongoDB Array Item at Second Level

Now lets look at some example how to Insert Items inside Orders array. Since it requires two JOIN columns to perform second level array insert you may need to add metadata for join.

First step is create documents for update. There are atleast 4 ways you can create input documents for MongoDB Destination.

1. You can use JSON generator Transform to create documents for load process
2. You can use Export JSON Task to generate input documents file and then use JSON Source to Read Documents (Check Output as Document option on JSON Source to read as RAW JSON)
3. You can use JSON Source to feed direct JSON (This approach is demonstrated in the below example sceenshot)
4. You can also use SQL query to output JSON Strings along with columns used for JOIN condition
   

   select CustomerID, OrderID as 'Orders.$.OrderID', DOC = '{Orders.$.Items : {ItemID: 1, Qty: 5} }'

To make it simple lets use #3 approach from above list. We will create few documents and insert New orders Items

Sample Document for MongoDB Collection

For testing purpose you can create MongoDB collection called test with following one document. As you see Items array is empty for both orders but once you run sample each will have 2 items.

{
    "_id" : ObjectId("58c01bee0610fea8ddda04b9"),
    "CustomerID" : "ALFKI",
    "Orders" : [ 
        {
            "OrderID" : 100,
            "Items" : [ ]
        }, 
        {
            "OrderID" : 101,
            "Items" : [ ]
        }
    ]
}

Now lets look at how to configure SSIS package

Step-1 : Create JSON Documents for Array Insert

Lets use JSON Source like below. Use following Sample JSON

/*Insert new items into order#100*/
{CustomerID:"ALFKI","Orders.$.OrderID" : 100, Doc: "{ \"Orders.$.Items\" : {ItemID: 1, Qty: 10} }" }
{CustomerID:"ALFKI","Orders.$.OrderID" : 100, Doc: "{ \"Orders.$.Items\" : {ItemID: 2, Qty: 11} }" }

/*Insert new items into order#101*/
{CustomerID:"ALFKI","Orders.$.OrderID" : 101, Doc: "{ \"Orders.$.Items\" : {ItemID: 1, Qty: 20} }" }
{CustomerID:"ALFKI","Orders.$.OrderID" : 101, Doc: "{ \"Orders.$.Items\" : {ItemID: 2, Qty: 21} }" }

Create Sample JSON Documents for MongoDB Array Update (2nd level array)

Step-2 : Configure MongoDB Destination for $push operation on Second level Array

Now lets connect source to Destination and configure MongoDB Destination. Select connection on first tab. Then On second tab change properties like below.

Configure MongoDB Destination for second level array update (Insert item using $push update operator)

Step-3 : Add missing JOIN column for Mapping

When you click on Mappings tab you will notice only root level columns are listed there for mapping but if you are JOINing based on Nested column found inside array then you have to add missing mapping column by hand like below screenshot.  In our case we Join by CustomerID and OrderID where OrderID is found in Orders array which is missing on mappings tab. So lets add it. Just make sure name new column using $ positional operator… e.g. Orders.$.OrderID  this will tell MongoDB that our OrderID column must be lookup under Array called Orders: [  ]

MongoDB Destination – Add missing metadata column for JOIN

Step-4 : Map JOIN columns and Input DOCUMENT

Now finally map all columns

SSIS MongoDB Destination – Column Mappings

Using SSIS : Insert MongoDB Array Item (Append to array)

Now lets look at how to perform same operation to insert array item in SSIS using $push operator. In this case we will do little different than Update Item example ($set operator). First we have not included CustomerID as part of Input columns and we mapped it on Mapping Screen rather than supplying columns names in ColumnsForLookup property (On Component Properties Tab of Destination UI). If you are using JSON Generator Transform to generate your JSON then you have option to Hide/Include certain columns part of your flow. Check Include column to downstream option if you want to use for JOIN column.

Set following settings

• Set LoadOptions as below
  op=$push;multi=true
• Set Operation=Update
• Leave ColumnsForLookup as empty
• On the mappings tab map __DOCUMENT__ and CustomerID in the Target. If you don’t see CustomerID column in the mapping create atleast one sample doc in MongoDB Collection so metadata is detected.

Download Sample for more detail.

SSIS JSON Source – generate JSON for MongoDB Array Insert

SSIS MongoDB Destination – Insert MongoDB Array Item (Use $push, or $addToSet operator)

Delete MongoDB Array Item ($pull operator)

If you wish to remove array item then you can use $pull operator. Only difference is rather than specifying two conditions in first argument… you just specify JOIN columns without array column (i.e. OrderID). To specify Array item condition you have to set inside $pull : { … }

Using Shell Command for MongoDB Delete Array Item (Remove by condition)

Below example shows how to remove OrderID=201 from Orders array for CustomerID=BOLID.

db.MyCollection.update(
	{CustomerID:"BOLID"},
	{$pull:{Orders: {"OrderID":201} }},
	{upsert: false,multi: false}
)

Using SSIS for MongoDB Delete Array Item

Now lets look at how to implement Delete logic for MongoDB Array Item in SSIS. As you see in below screenshots You have to modify following things

• Set ColumnsForLookup property to CustomerID (no need to add OrderID here because its part of input document)
• Set LoadOptions property to op=none;multi=false. op=none means we are supplying Update operator part of input document.

Assume you have doc like below in MongoDB  and you like to remove Order#101 from array

{
    "_id" : ObjectId("58c01bee0610fea8ddda04b9"),
    "ParentID" : 1,
    "Orders" : [ 
        100, 
        101, 
        102
    ]
}

For this supply 2 columns from upstream
col1  = ParentID
col2  = DOCUMENT

Document column should look like below
{Orders: 101}

1. On MongoDB destination change LoadOptions property to  op=$pull;multi=true
2. Change Operation to Update
3. Map ParentID and DOCUMENT columns on Mapping screen
4. Run the package … It will remove Order# 101 from array

Array Upsert – Update / Insert using Custom JOIN condition (such as $ne )

If you like to perform Upsert Operation on MongoDB Array Items then refer to below blog post.

Update or Insert – Upsert MongoDB Array Items using SSIS

Conclusion

MongoDB integration can be challenging if you are new to NoSQL world. If you are using SSIS as your primary ETL tool then not to worry because SSIS PowerPack can give super power needed to complete your project on time with drag and drop high performance connectors.

Keywords: mongodb update array item | mongodb update array elements | mongodb update array documents | MongoDB $ operator | MongoDB update sub document items | MongoDB CRUD operations | MongoDB Bulk Updates | MongoDB bulk updates to array items

The post Update MongoDB Array Items using SSIS appeared first on ZappySys Blog.

In our previous post we explained how to read/load MongoDB data in SSIS. This post covers specifically how to parse MongoDB date time stored inside your MongoDB documents. By default any well known date formats (e.g. ISO date) will be parsed as valid datetime (e.g. DT_DBTIMESTAMP) when you use SSIS MongoDB Source. But if you have date stored as other format (e.g. MM-dd-yyyy) then system will not parse as string unless you specify custom date format on JSON Datetime tab. If you want to learn How to query Date in MongoDB then read this article

Parse MongoDB Date time using SSIS MongoDB Source

Perform the following steps to enable custom date format parsing for date not stored as ISODate in MongoDB. Below steps assume you have Downloaded and Installed SSIS PowerPack from Here

NOTE: Below steps not necessary if your date is stored as ISODate in MongoDB…. e.g.  { “OrderDate” : ISODate(“2015-12-31T00:00:00Z”)  }

1. Open MongoDB Source. Change Mode to Query Mode
2. Check Enable JSON option. You don’t have to type filter (its optional)
3. Goto JSON Options and then Datetime options Tab
4. Enter custom date format (The way your date stored in MongoDB)

Now go to columns and click refresh columns. It will detect column as DT_DBTIMESTAMP (e.g. date/time)

SSIS MongoDB Source- Date Time Options ( Parse MongoDB Date)

SSIS MongoDB Source – Query and Parse MongoDB Date with Custom format

The post How to parse MongoDB Date time in SSIS appeared first on ZappySys Blog.

Steps – Extract multiple arrays from MongoDB

1. Extract data using MongoDB source (Select output mode as Raw Document)
2. Connect MongoDB Source to Multicast Transform
3. Connect multi cast output to JSON Parser Transform which will take upstream raw JSON doc and convert to Flat table output just like MongoDB source. On Json Parser specify filter e.g. $.doc.Array1[*]
4. Connect Json Parser Transform output to desired target e.g. sql server/flat file.
5. Repeat step #3,4,5 for  as many outputs you need with different filter settings (e.g. $.doc.Array2[*]).

Here is more info about JSON Parser Transform (This is new Transform added in v2.2)

//zappysys.com/products/ssis-powerpack/ssis-json-parser-transform/

Video Tutorial – Step By Step – How to read multiple MongoDB arrays

Below video tutorial demonstrates following things

* How to execute MongoDB Shell commands in SSIS (e.g. create Collection)
* How to read data from multiple arrays from MongoDB document (i.e. Orders, Products)
* Save result into two SQL Server tables

The post How to extract multiple arrays from MongoDB using SSIS appeared first on ZappySys Blog.

SSIS PowerPack comes with great features for MongoDB Integration Scenarios but in this post we will focus on how to call MongoDB Javascript (Server side) using SSIS. For this purpose we will use SSIS MongoDB Execute SQL Task

This powerful task not only easy to use but it supports calling any valid MongoDB commands including Server side Java Script.

How to call MongoDB JavaScript – Example

To call various MongoDB shell commands or execute JavaScript in MongoDB using SSIS you have Download and Install SSIS PowerPack . Once its installed perform following steps.

1. From SSIS Toolbox drag ZS MongoDB ExecuteSQL Task
2. Click New to create new MongoDB connection manager
3. Once connection is configured click Test and OK to save
4. Now you can use {{Examples}} dropdown to see possible scenarios with MongoDB ExecuteSQL task
5. Below is sample script to call Aggregate command

 NOTE: See how we returned command output using toArray() Javascript function at the end? This will give us all records in clean JSON Array format which can be consumed easily using SSIS JSON Source. If you don’t use toArray then you may get Cursor with only first batch.

{
scope: 'database',
db: '{{User::varDB}}',
command: 'eval',
args: 
 { 
  code : 'function()
  {
   var varState="GA"
   return db.zipcodes.aggregate( 
   [
   {$match : { state: varState } },
   {$group : {_id :  "$state", TotalPopulation: { $sum: "$pop" }}},
   {$match: { TotalPopulation: { $gt: 25 } } },
   {$project : { StateAbbr: "$_id" , _id: 0, TotalPopulation:1 } },
   {$sort : {TotalPopulation : -1}}
   ]).toArray(); 
  }'
 }
}

Call MongoDB single line commands

If you wish to call commands which are single line only then you may use below code which is simpler than previous example. Notice it doesn’t require function() keyword like previous one. Make sure your code has to be in single line or try to use \r\n for new line

{
 scope: 'database',
 command: 'eval',
 args: { code: 'db.hostInfo()' } 
}

Permissions needed to call JavaScript

Calling eval command requires special permission so make sure you setup permission so MongoDB user can call db.runCommand ( {eval command} ). Here is more information about calling eval command in MongoDB.

Save Result JSON of MongoDB command into SSIS variable

If you want to save MongoDB command output as JSON into SSIS Variable then set Result mode = RawResult

Call MongoDB JavaScript in SSIS Using Execute SQL Task – Aggregate Command Example – Save raw JSON output into SSIS variable

How to use MongoDB Eval command to call runCommand function

Now lets look at how to call some admin functions using runCommand function. In below example we setting Global Timeout setting for Cursor (This helps to avoid timeout for long running cursor query)

{
 scope: 'database',
 command: 'eval',
 args: { code: "db.getSiblingDB('admin').runCommand( { setParameter: 1, cursorTimeoutMillis: 400000 } )" } 
}

Setting MongoDB Cursor Timeout ( set cursorTimeoutMillis parameter )

Save Result of MongoDB command into SSIS variable as ADO.net Table

If you want to use ForEach Loop to iterate through certain records from MongoDB result then set Result mode = FullResult this will save result as ADO.net table rather than JSON. Example : Get all collections and loop through them in SSIS using ForEachLoop Task?

Call MongoDB JavaScript in SSIS – Aggregate Command – Save JSON output as ADO.net Table – Loop Through ForEachLoop

Conclusion

MongoDB is the most popular NoSQL Database engine but integrating that into SSIS can be challenging without right tools. ZappySys SSIS PowerPack comes with many MongoDB Tasks/Components for SSIS.

The post How to call MongoDB JavaScript using SSIS appeared first on ZappySys Blog.

In our previous article, we discussed how to load data into MongoDB (Insert, Update, or Upsert Mode) with various options. In this post, we will focus on creating a MongoDB Document for the load process. We will see how to produce BSON (MongoDB-specific JSON that may include functions such as ObjectID and ISODate).

This article assumes you have installed SSIS PowerPack and watched the introduction videos of SSIS JSON Generator Transform and MongoDB Destination Connector

Create MongoDB document Example (Produce JSON, BSON for Load)

Here is the screenshot, which shows how to fetch SQL Server data > create a MongoDB document in JSON / BSON format, and load it into a MongoDB collection

The high-level steps we performed are

1. Used OLEDB Source to fetch data from SQL Server. Used a query like below
   

   select CustomerID, 'ISODate("' + convert(varchar(30),getdate(),126) + '")' as DOB from Customers

   In the above query, you will notice that we are building some BSON fragments, which will be treated as raw values in the JSON generator. Also, we are using the convert function from SQL Server to produce an ISO date format (Style=126 will produce an ISO date)
2. Used the SSIS JSON Generator Transform to produce a JSON document for each customer. Define a custom layout using the drag-and-drop interface. You can control how many documents you want to output using settings on Transform (Single document for all input rows or one document per row in the Main dataset)
3. Load JSON documents coming from upstream into MongoDB using the SSIS MongoDB Destination Connector. We changed Action=Upsert and set ColumnsForLookup to CustomerID (Consider this as Primary Key or your Join Column to find existing document)
4. On Target, we mapped __DOCUMENT__ rather than individual columns, so that we can load the full document.
   

   Sample SSIS Package – Create MongoDB Document in BSON format (JSON with functions). Update or Insert Mode

Configure JSON Generator Transform – Create JSON Document

If you want to learn how to configure the SSIS JSON Generator Transform, then watch the video here. Important thing to remember in the configuration is to check the Treat value of this column as Raw JSON

Create MongoDB Document in JSON or BSON format – Load into Collection (Upsert – Update or Insert)

Configure MongoDB Destination – For Upsert (Update or Insert Mode)

By default, the SSIS MongoDB Destination Connector performs an insert, but if you want to perform an Upsert (i.e., insert if not found, else update), you have to change a few settings.

1. Change Operation to UpdateOrInsert
2. Set ColumnsForLookup (e.g., in our case, it was just one CustomerID). If you have multiple columns for join, then use a comma-separated list (e.g., City, State, Zip)
3. If you want to affect various documents if more than one match is found for the join criteria, then use multi=true as shown in the screenshot below (see LoadOptions )
   

   Configure the SSIS MongoDB destination for the Upsert operation

Conclusion

In this post, you have seen how easy it is to create MongoDB Documents in JSON / BSON format and load them into a MongoDB collection using ZappySys SSIS PowerPack. This innovative drag-and-drop approach can simplify the entire ETL process if you are in a time crunch and don’t know scripting languages.

The post How to create MongoDB documents (JSON, BSON) for loading in SSIS appeared first on ZappySys Blog.

In this post you will learn how to write flexible MongoDB Aggregation Queries in SSIS (i.e. Group By Query) using SSIS MongoDB Source Component. MongoDB provides powerful Aggregation Pipeline Engine which is conceptually same as writing Group By queries in traditional SQL world but its not exactly same. SSIS MongoDB Source returns nested MongoDB JSON documents into flat looking table with rows and column (denormalized data).

If you want to learn how to write data to MongoDB (Insert, Update, Delete, Upsert) then click here. If you want learn how to read data from MongoDB then click here.

So let’s explore concepts of MongoDB Aggregation Pipeline and see some MongoDB query examples for SSIS use case.

SSIS MongoDB Source Introduction

SSIS MongoDB Source supports three modes to read data from MongoDB collection. Each mode also supports JSONPath Expression to extract nested array.

1. Table Mode
2. SQL Query Mode
3. JSON Query Mode (Native)

To read more about supported SQL Syntax read help file here.

Using SSIS MongoDB Source to Query Data (Aggregation)

Now lets look at how to use SSIS MongoDB Source to query MongoDB data using Aggregation Engine.

MongoDB Aggregation Pipeline Syntax in SSIS

SSIS MongoDB Source support all pipeline commands to write aggregation queries but its slighly different syntax than shell command so your SHell Query won’t work directly but you can copy some part of your Shell query. Check below Syntax.

MongoDB Native Query Syntax (Shell Command) for Aggregation:

db.YourCollectionName.aggregate(
[
[ <pipeline-stage1>]
[,<pipeline-stage2>]
...
...
[,<pipeline-stageN>]
]
)

SSIS MongoDB Source Query Syntax for Aggregation:


{Table:YourTableName ,Method:aggregate [,MaxRows: N] [,AllowDiskUse:true] [,AllowCursor:true] [,Timeout:#####] }
{
pipeline: [
[ <pipeline-stage1>]
[,<pipeline-stage2>]
...
...
[,<pipeline-stageN>]
]
}

Example Aggregation Queries (SQL to SSIS Mapping)

Assume that we want to write following SQL Query using SSIS MongoDB Source or using MongoDB Shell

SELECT State as StateAbbr,SUM(population) as TotalPopulation 
FROM ZipCodes 
WHERE status='Active' 
HAVING SUM(population) > 25
Order By TotalPopulation DESC

Above SQL Query can be translated to below JSON Query which can be used in SSIS MongoDB Source. As we noted before Below syntax is not same as Shell command so it will only run in SSIS MongoDB Source.

{Table: ZipCodes,Method: aggregate}
{
  pipeline: 
  [
    {$match : { status: "A" } } ,
    {$group : {_id :  "$state", TotalPopulation: { $sum: "$pop" }}},
    {$match: { TotalPopulation: { $gt: 25 } } },
    {$project : { StateAbbr: "$_id" , _id: 0, TotalPopulation:1 } },
    {$sort : {TotalPopulation : -1}}
  ]
}

So as you saw in above query various clause of SELECT queries are defined as pipeline stages. You can add any possible stages in different order. Based on their order it will be applied so order of stage is very important.  For more information about various pipeline stages click here. Below is high level translation of SQL Query to MongoDB Aggregation Query.

SSIS MongoDB Source – Example of MongoDB Group By Query (Aggregation Framework – Pipeline)

SQL to MongoDB Aggregation Mapping

Now let’s look at few examples how to translate traditional SQL  Queries to JSON Query for SSIS MongoDB Source. We will use sample data we inserted in previous section.

Creating Sample Data for MongoDB Query Examples

For this exercise to write sample queries using below data. So let’s first create a collection called Order and insert following records using your favourite MongoDB client tool.

db.Orders.insert(
  {
    "OrderId" : 1000,
    "CustomerId" : "CUS001",
    "OrderTotal" : 2000.50,
    "Quantity" : 20,
    "OrderDate" : ISODate("2010-01-01"),
    "Country" : "USA",
    "Items" : [ {ItemId:10, UnitPrice: 100.00,Quantity: 10}, {ItemId:11, UnitPrice: 200.10,Quantity: 5}],
    "PaymentInfo" : { 
      "Method" : "Card",
      "Currency" : "USD" 
    }
  }
)

db.Orders.insert(
  {
    "OrderId" : 1001,
    "CustomerId" : "CUS001",
    "OrderTotal" : 1000.00,
    "Quantity" : 10,
    "OrderDate" : ISODate("2011-01-02"),
    "Country" : "USA",
    "Items" : [ {ItemId:10, UnitPrice: 100.00,Quantity: 10}],
    "PaymentInfo" : { 
      "Method" : "Card",
      "Currency" : "USD" 
    }
  }
)

db.Orders.insert(
  {
    "OrderId" : 1002,
    "CustomerId" : "CUS002",
    "OrderTotal" : 1000.00,
    "Quantity" : 5,
    "OrderDate" : ISODate("2012-01-01"),
    "Country" : "USA",
    "Items" : [ {ItemId:13, UnitPrice: 200.00,Quantity: 5}],
    "PaymentInfo" : { 
      "Method" : "Bank",
      "Currency" : "USD" 
    }
  }
)

db.Orders.insert(
  {
    "OrderId" : 1003,
    "CustomerId" : "CUS003",
    "OrderTotal" : 59000.00,
    "Quantity" : 10,
    "OrderDate" : ISODate("2005-01-02"),
    "Country" : "India",
    "Items" : [ {ItemId:14, UnitPrice: 5900.00,Quantity: 10}],
    "PaymentInfo" : { 
      "Method" : "Check",
      "Currency" : "INR" 
    }
  }
)

Here is how it would look like (We used RoboMongo i.e. Robo 3T) but you can use mongo.exe command line to issue above commands.

Inserting Example data for MongoDB Aggregation Queries (RoboMongo / Robo 3T)

Example 1 – MongoDB Aggregation Query with COUNT(*) – $group, $project

Here is how to write COUNT(*) query in MongoDB. Below query will find order count by country. Each query must output _id field and you can refer Document column using $ prefix (i.e. “$Country” or for nested object use dot separator “$PaymentInfo.Method” ) . To output _id column with different name we have used Country: “$_id”. Because we including copy of _id we can remove _id column by doing  _id: 0.  To include column in output use YourColumanName: 1 in $project stage.

SQL Query:

SELECT Country,Count(*) as OrderCount 
FROM Orders
GROUP BY Country

Mongo Shell Query:

db.Orders.aggregate(
 [
   {$group : {_id : "$Country",OrderCount: { $sum: 1 }}} 
  ,{$project : {Country: "$_id", _id: 0, OrderCount: 1 }}
 ]
)

SSIS MongoDB Source Query:

{Table: Orders,Method: aggregate} 
{ pipeline: 
   [
    {$group : {_id : "$Country",OrderCount: { $sum: 1 }}} 
   ,{$project : {Country: "$_id", _id: 0, OrderCount: 1 }}
  ]
}

Example 2 – MongoDB Aggregation Query with WHERE and ORDER BY- $match, $group, $project, $sort

Here is how to write COUNT(*) query in MongoDB. Below query will find order count by country.

SQL Query:

SELECT Country,Count(*) as OrderCount 
WHERE Quantity > 1
FROM Orders
GROUP BY Country
ORDER BY Country

Mongo Shell Query:

db.Orders.aggregate(
 [
   {$match : {Quantity : { $gt: 1 }}} 
  ,{$group : {_id : "$Country",OrderCount: { $sum: 1 }}} 
  ,{$project : {Country: "$_id", _id: 0, OrderCount: 1 }}
  ,{$sort : {Country: 1}}
 ]
)

SSIS MongoDB Source Query:

{Table: Orders,Method: aggregate} 
{ pipeline: 
   [
     {$match : {Quantity : { $gt: 1 }}} 
    ,{$group : {_id : "$Country",OrderCount: { $sum: 1 }}} 
    ,{$project : {Country: "$_id", _id: 0, OrderCount: 1 }} ,{$sort : {Country: 1}}  
   ]
}

Example 3 – MongoDB Aggregation Query with WHERE and HAVING Clause – $match, $group

Here is an example of WHERE and HAVING clause. They both use $match stage but their orders are different. WHERE condition is applied before GROUP BY (i.e. $group) and HAVING condition is applied after GROUP BY.

SQL Query:

SELECT Country,Count(*) as OrderCount
FROM Orders 
WHERE Quantity > 1
GROUP BY Country
HAVING Count(*) > 0

Mongo Shell Query:

db.Orders.aggregate(
 [ 
   {$match : {Quantity: { $gt: 1 }}}
  ,{$group : {_id : "$Country",OrderCount: { $sum: 1 }}} 
  ,{$match : {OrderCount: { $gte: 1 }}} 
  ,{$project : {Country: "$_id", _id: 0, OrderCount: 1 }}
 ]
)

SSIS MongoDB Source Query:

{Table: Orders,Method: aggregate} 
{ pipeline: 
   [
      {
        $group : {_id : "$Country",OrderCount: { $sum: 1 }}
      }      
   ]
}

Example 4 – MongoDB Aggregation Query with $unwind (Flatten nested Array)

Now lets look at an example of $unwind operator. $unwind can be used to perform operations on nested array. Common things would be flatten the hierarchy and then apply group by operation. Lets find out Average Quantity of Items By Order Number

SQL Query:

SELECT Orders.OrderId as OrderNumber,Avg(Items.Quantity) as AvgItemQty
FROM Orders 
JOIN Items ON Orders.OrderId = Items.OrderId
GROUP BY Orders.OrderId

Mongo Shell Query:

db.Orders.aggregate(
 [ 
   {$unwind : "$Items"}
  ,{$group : {_id : "$OrderId",AvgItemQty: { $avg: "$Items.Quantity" }}} 
  ,{$project : {OrderNumber: "$_id", _id: 0, AvgItemQty: 1 }}
 ]
)

SSIS MongoDB Source Query:

{Table: Orders,Method: aggregate} 
{ pipeline: 
  [ 
    {$unwind : "$Items"}
   ,{$group : {_id : "$OrderId",AvgItemQty: { $avg: "$Items.Quantity" }}} 
   ,{$project : {OrderNumber: "$_id", _id: 0, AvgItemQty: 1 }}
  ]
)

MongoDB Query with Dynamic Value / ISO DateTime

There will be a time when you have to supply Dynamic Value from Variable or supply Dynamic Date time inside your query. Dates are little different than other data types. Check this article to learn more about passing dates in MongoDB Query.

For Dynamic Date use SSIS Variables Format Specifiers

Example of DateTime query (Static Value)

{Table: Orders,Method: aggregate} 
{ pipeline: 
   [
     {$match : {OrderDate : { $gt: new ISODate("2010-01-01") }}} 
    ,{$group : {_id : "$Country",OrderCount: { $sum: 1 }}} 
    ,{$project : {Country: "$_id", _id: 0, OrderCount: 1 }} ,{$sort : {Country: 1}}  
   ]
}

Example of using Dynamic Value from Variable

To pass date from SSIS Variable you can use Format Specifiers. (e.g. yyyy-MM-dd  or for full ISO date yyyy-MM-ddTHH:mm:ssZ )

{Table: Orders,Method: aggregate} 
{ pipeline: 
   [
     {$match : {OrderDate : { $gt: new ISODate("{{User::varOrderDate,yyyy-MM-dd}}") }}} 
    ,{$group : {_id : "$Country",OrderCount: { $sum: 1 }}} 
    ,{$project : {Country: "$_id", _id: 0, OrderCount: 1 }} ,{$sort : {Country: 1}}  
   ]
}

Conclusion

MongoDB is one of the most popular NoSQL / Document Storage Engine. SSIS PowerPack make super easy and efficient to read/write data in MongoDB without any coding effort. Download SSIS PowerPack to try out MongoDB features along with 70+ other components .

Search terms:
mongodb group by count
mongodb group by multiple fields
mongodb group by date
mongodb group example
mongodb group sort
mongodb group by month
mongodb group by having
mongodb group distinct
mongodb group by sum
mongodb aggregate
mongodb aggregate sort
mongodb aggregate unwind

The post How to write MongoDB Aggregation Queries in SSIS (Group By) appeared first on ZappySys Blog.

In this post you will learn how to use MongoDB ExecuteSQL Task for SSIS to insert multiple documents into MongoDB collection using SSIS. This task can be used to perform any DDL or DML operations for MongoDB natively inside SSIS without using any command line utilities. It gives you flexibility and security of SSIS.

Step-By-Step – Insert Multiple Documents into MongoDB Collection using SSIS

Lets look at how to insert multiple MongoDB documents into

1. Download SSIS PowerPack from here
2. Create new SSIS Package
3. Drag ZS MongoDB ExecuteSQL Task from SSIS Toolbox
4. Double click task to configure
5. Create new MongoDB connection by clicking on [New] button next to connection dropdown
6. Click OK to close connection UI
7. Now you can enter following Command in MongoDB ExecuteSQL Task to insert multiple documents into MongoDB. If you want to pass json stored inside SSIS variable then use variable placeholders (e.g. {{User::varMyJson}}  ) anywhere inside below command.
   

   {
    scope: 'database',
    db: 'DemoDB',
    command: 'eval',
    args: { code: 'db.MyCollection.insert( [ {ID:1,Name:"AAA"}, {ID:2,Name:"BBB"} ] )' } 
   }

Insert multiple documents in MongoDB Collection (JSON or BSON format)

Conclusion

Using MongoDB ExecuteSQL Task for SSIS you can execute ad-hoc MongoDB Shell commands without installing any command line tools. This is totally native SSIS approaching. You can Try SSIS PowerPack for FREE to find out many more MongoDB features not discussed in this article.

The post How to Insert Multiple Documents into MongoDB using SSIS appeared first on ZappySys Blog.