In this version we focused on many bug fixes. Among them most important bug was limitation of only 2000 characters for SQL Query using OPENQUERY statement. Now you can use up to 8000 characters for OPENQUERY and if that’s not enough use EXEC AT statement to send query to Data Gateway. We also made many improvements with our Salesforce Driver so now you can use it along with Data Gateway or any other Reporting / ETL tools.

Version 1.1.4.10904 [Sep 11, 2019]

New Features/Improvements

• NEW: All Drivers – New DSN dialogbox should suggest unique DSN name if already exists (Prevent accidental overwrite)
• NEW: All Drivers – Add support for named parameters for stored procs (e.g. exec webrequest @uri=’https://abc.com’ )
• NEW: Data Gateway – Give Export / Import Settings option
• NEW: Data Gateway – Provide list of database to caller App
• NEW: General – Use default log file name if only folder name is supplied
• NEW: Salesforce Driver – Add Placeholder Engine Support

Bug fixes

• FIX: All API Drivers – Query builders shows Password and Default Attributes on first time launch
• FIX: All API Drivers – RetryInfo not included in Connectionstring or generated SQL
• FIX: All API Drivers – URL JOIN may cause error for decimal values – Query execution error: Input string was not in a correct format.
• FIX: All API Drivers – URL on general tab is wiped out when you use save HTTP Connection
• FIX: All API Drivers – Field names are wrong StatucFieldxxxxxxx and StatucCheckxxxxxx rather than StatusCheckxxxxxx and StatusFieldxxxxxxx
• FIX: CSV Driver – JOIN query fails in CSV driver with invalid format detected
• FIX: Data Gateway – MS SQL Server – Support more than 2000 characters for SQL in OPENROWSET and EXEC – In old version you may get error – Data is invalid if text is longer than 2000 characters
• FIX: Data Gateway – Query with Caching Option hangs
• FIX: Data Gateway – Use of META in WITH clause prevents error in some cases
• FIX: Data Gateway – You may get “The data is invalid” error when the query is too long
• FIX: Data Gateway – You may get Data is invalid error for OAuth Authorization failure
• FIX: Salesforce Driver – When you execute INSERT via Linked Server Query it Inserts row but fails with error at the end
• FIX: Salesforce Driver – Load ConnectionString action throws Object reference not set to an instance of an object.
• FIX: Salesforce Driver – Click on Generate SQL throws Object reference not set to an instance of an object
• FIX: Salesforce Driver – Data is not returned in Excel when you use [From Microsoft Query] option
• FIX: Salesforce Driver – DELETE statement not working
• FIX: Salesforce Driver – Escape sequence not supported (e.g. WHERE name =’Sam\’s’ )
• FIX: Salesforce Driver – OPENQUERY / EXEC AT doesn’t work with DELETE / UPDATE / INSERT
• FIX: Salesforce Driver – SSIS Failed to Generate Metadata due to character encoding issue
• FIX: Salesforce Driver – Using * in SELECT with WHERE is not working

The post ODBC PowerPack v1.1.4 Released appeared first on ZappySys Blog.

In our previous blog we saw how update / insert data into SQL Server using SSIS Upsert Destination. In this post we will look at specific example on Data migration from Access to Amazon Redshift using SSIS Upsert Destination (Insert, Update, Delete), along with few other topics such as how to create table using Upsert Destination. how to read all Customers data from Ms Access Table and Merge it in the Amazon Redshift.

We will go through the steps to read data from Access and Load into Amazon Redshift.

In nutshell, this post will focus on how to read access table data in SSIS.

So let’s get started.

Requirements

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, do not forget to install ZappySys SSIS PowerPack
4. Finally, Make sure that Microsoft Access installed.

How to Read MS Access table data and migrate that data in Amazon Redshift table.

Let´s start with an example. In this article we will see Data migration from Access to Amazon Redshift.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. Furthermore, drag and drop the OLE DB Source.
   

   OLE DB Source – Drag and Drop

3. Double click on OLE DB Source for configure it and click on New Connection and configure connection as below to connect access database and click on OK.
   

   OLE DB : Access Connection

4. Now in OLE DB Source Select the mode as Table or View and select Preview to view the access table data.
   

   OLE DB Source Preview

5. Now drag and drop Upsert Destination (Insert, Update, Delete) and create connection with Amazon Redshift Database.
   

   Upsert Destination : Amazon Redshift Connection

6. Now select Action as Sync and check all the checkboxes Insert, Update and Delete from target if not found in source. Select the table and Map all the columns and select the Key field(s) and click on OK.
   

   Upsert Destination Configuration

7. That’s it we are ready to migrate MS access table data into Amazon Redshift Table. Execute the package and it will migrate the data.
   

   Upsert Destination (Insert, Update, Delete)

Bulk Update data in Amazon Redshift

So in previous example we saw bulk update or insert (Upsert) in Redshift Table. Now let’s look at how to update data in target table if record exists.

Here is how you can perform bulk update in Amazon Redshift using Upsert Destination.

1. Double click on Upsert Destination for configure it.
2. Set Action Bulk Update => based on matching records on target. Select Connection and Target Table. Click on Map All to Mappings all columns and check on Only Primary Key columns.
   
3. Thats all, Click on OK to save Upsert Destination settings UI.

Bulk Delete data in Amazon Redshift

Here is how you can bulk delete data in Amazon Redshift.

1. Double click on Upsert Destination for configure it.
2. Set Action Bulk Delete => based on matching records on target. Select Connection and Target Table. Click on Map All to Mappings all columns and check on Only Primary Key columns.
   
3. Thats all, Click on OK to save Upsert Destination settings UI.

Conclusion

In this article, we show how to read MS Access table data and migrate the data in Amazon Redshift table using SSIS. We show how to do connect access MS Access using OLE DB Source. Also, we show how to write Sync Insert, Update and Delete in target if not found in Source Using ZS Upsert Destination. If you liked this article and you want to try, you can download the SSIS PowerPack from here (includes 70+ Components).

References

• ZappySys SSIS installer.
• Help File: Upsert Destination

The post Merge / Upsert data in Amazon Redshift using SSIS appeared first on ZappySys Blog.

In our previous blog we saw How to perform Upsert (Update or Insert) for SQL Server Table. In this post we will look at specific example on How to Load data into PostgreSQL – Upsert using SSIS Upsert Destination (Insert, Update, Delete), along with few other topics such as how to create target table using Upsert Destination, how to read data from Ms Access Table and Merge into PostgreSQL using SSIS.

We will go through the steps to read data from Access and Load into PostgreSQL – Upsert in SSIS.

In nutshell, this post will focus on how to read access table data in SSIS.

So let’s get started.

Requirements

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, do not forget to install ZappySys SSIS PowerPack
4. Finally, Make sure that Microsoft Access installed.

Read MS Access data and Upsert into PostgreSQL table

Let´s start with an example. In this article we will see Data migration from Access to PostgreSQL.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. Furthermore, drag and drop the OLE DB Source.
   

   OLE DB Source – Drag and Drop

3. Double click on OLE DB Source for configure it and click on New Connection and configure connection as below to connect access database and click on OK.
   

   OLE DB : Access Connection

4. Now in OLE DB Source Select the mode as Table or View and select Preview to view the access table data.
   

   OLE DB Source Preview

5. Now drag and drop Upsert Destination (Insert, Update, Delete) and create connection with PostgreSQL Database.
   

   Setup SSIS Upsert Destination for PostgreSQL data load (Bulk Update, Insert, Delete)

6. Now select Action as Sync and check all the checkboxes Insert, Update and Delete from target if not found in source. Select the table and Map all the columns and select the Key field(s) and click on OK.
7. That’s it we are ready to migrate MS access table data into PostgreSQL Table. Execute the package and it will migrate the data.

Conclusion

In this article, we show how to read MS Access table data and migrate the data in PostgreSQL table using SSIS Upsert Destination. We show how to do connect access MS Access using OLE DB Source. Also, we show how to write Sync Insert, Update and Delete in target if not found in Source Using ZS Upsert Destination. If you liked this article and you want to try, you can download the SSIS PowerPack from here (includes 70+ Components).

References

• ZappySys SSIS installer.
• Help File: Upsert Destination

The post Load data into PostgreSQL – Upsert using SSIS (Bulk Update, Insert, Delete) appeared first on ZappySys Blog.

In our previous blog, we saw how to export a REST API to MS Access using a VBA Command Button. In this post, we will look at a specific example of data migration from Access to SQL Server using the SSIS Upsert Destination (Insert, Update, Delete), along with a few other topics, such as how to create a table using the Upsert Destination. How to read all the customer data from the MS Access Table and merge it into the SQL Server.

We will go through the steps to read data from Access and load it into SQL Server.

In a nutshell, this post explains how to read data from an Access table in SSIS.

So let’s get started.

Requirements

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Thirdly, do not forget to install ZappySys SSIS PowerPack
4. Finally, make sure that Microsoft Access is installed.

How to read MS Access table data and migrate that data to a SQL Server table.

Let´s start with an example. In this article, we will see Data migration from Access to SQL Server.

1. First of all, drag and drop the Data Flow Task from the SSIS Toolbox and double-click it to edit.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox
2. Furthermore, drag and drop the OLE DB Source.
   

   OLE DB Source – Drag and Drop
3. Double-click the OLE DB Source to configure it. Click New Connection, configure the connection as shown below to connect to the Access database, and click OK.
   

   OLE DB Access Connection
4. Now, in the OLE DB Source, set the mode to Table or View, and select Preview to view the table data.
   

   OLE DB Source Preview
5. Now drag and drop Upsert Destination (Insert, Update, Delete), and create a connection to a SQL Server Database.
   

   Upsert Destination SQL Server Connection
6. Click the New Table button in the Target Table, change the table name, and click OK to create the table in SQL Server.
   

   Upsert Destination Create Table
7. Now select Action as Sync and check all the checkboxes: Insert, Update, and Delete from the target if not found in the source. Map all the columns and select the Key field(s) and click on OK.
   

   Upsert Destination Configuration
8. That’s it, we are ready to migrate the MS Access table data into the SQL Server Table. Execute the package, and it will migrate the data.
   

   Upsert Destination (Insert, Update, Delete)

Conclusion

In this article, we show how to read data from an MS Access table and migrate it to a SQL Server table using SSIS. We show how to connect to MS Access using an OLE DB Source. Also, we show how to write Sync Insert, Update, and Delete in the target if not found in the Source Using ZS Upsert Destination. If you liked this article and want to try it, you can download the SSIS PowerPack here (includes 70+ Components).

References

• ZappySys SSIS installer.
• Help File: Upsert Destination

The post Data migration from Access to SQL Server using SSIS Upsert Destination appeared first on ZappySys Blog.

In this release we focused on mainly focused on fixing high priority bugs related to Upsert Destination. We also added High DPI Monitor Support with Auto Scale UI option (4K Resolution now supported)

Full Notes: https://zappysys.com/onlinehelp/ssis-powerpack/scr/release-notes.htm

Version 2.9.2.10321 [Mar 21, 2019]

New Features/Improvements

• NEW: Excel Source – Add Option to read first sheet and last sheet rather than fixed name ( $last_sheet$ and $first_sheet$ )
• NEW: General – Support for UI Auto Scaling when High DPI / Font Scaling Enabled on your OS (e.g. Win10) – 4K Monitors or High Resolution Display now supported
• NEW: Generator Tarnsforms / Export Tasks for JSON, XML, CSV – Add Option to set Date Format at column level rather than global option
• NEW: Upsert Destination – Add Create Table Option based on Upstream Metadata
• NEW: Upsert Destination – Add option to preview exising data from selected table
• NEW: Upsert Destination – Add option to show source and target column datatypes

Bug fixes

• FIX: All Source Components – After Sorting Available Columns you may lose Output column renames in the columns grid, also output columns sorted in reverse order
• FIX: All Source Components – After Sorting Columns you cannot remove or add columns from the grid
• FIX: Upsert Destination – Mapped Column status icon is not refreshing after column is mapped
• FIX: Upsert Destination – varchar, char, text datatypes detected as DT_WSTR / DT_NTEXT rather than DT_STR / DT_TEXT
• FIX: Upsert Destination – When you have zero rows in source you may get error – Invalid object name #zappysys_xxxxxx_yyyyyyyy
• FIX: Upsert Destination – When you map to column with scale=zero (e.g. numeric(N, 0)) you may get error on save
• FIX: Upsert Destination – You may get error when you change source column after save – An item with the same key has already been added
• FIX: Upsert Destination – You may get object reference not set error in advanced edit

The post SSIS PowerPack v2.9.2 released (Maintenance Release) appeared first on ZappySys Blog.

In our previous blog post we saw how to update / delete mongodb array item. Now let’s look at how to Upsert MongoDB Array Items (i.e. Insert Item if not found in Array else Update existing record). Upsert into nested MongoDB Array requires two step process, unfortunately there is no easy way to do in a single step. You may see in below screenshot that first we will do Update Step for Existing MongoDB Array Items and then we will insert missing Records in second step. So let’s get started.

MongoDB Array Upsert – Update / Insert using Custom JOIN condition (such as $ne )

By default SSIS Mongodb Destination performs Lookup using $eq condition (Match rows using Equal operator). However there will be a time when you like to lookup and update target rows using custom join criteria (e.g. Use Not Equal condition – $ne ). V2.7.6  and later introduced new property called 

EnableCustomLookupQuery

Here is the description how to use custom Join using this new property.

Enables use of custom lookup query for Update / Upsert or Delete Operation. 
By default JOIN condition for target record match is generated automatically based on columns you supply in ColumnsForLookup property. 
However in some cases you need to supply custom condition for lookup to perform complex operations, in such case enable this option. 
When you enable this option you must supply map __DOCUMENT__ input column. String you supply to this column should be in this format 

<command>
  <query>YOUR_LOOKUP_QUERY</query>
  <document>YOUR_JSON_DOC</document>
</command>. 

Lookup query in <query> tag can be either Mongo JSON format (e.g. { \"CustomerID\" : \"AAA\", \"Orders.OrderID\" : { \"$ne\" : \"1000\" } }) 
OR
you can use ZappySys SQL query (e.g. select * from mytable where CustomerID='AAA' and [Orders.$.OrderID] != '1000' )

Now lets look at step by step. In below example we have a table called CustomerTest. We will load it with 2 records (With No orders). Then Update array using custom lookup condition, and later step we will insert record in array using custom lookup condition.

SSIS MongoDB Array Upsert Example (Update / Insert Array Items based on custom lookup condition)

Update Array using Custom Lookup Condition

1. Lets create few sample rows in MongoDB Collection. You can use following command in SSIS MongoDB Execute SQL Task
   

   {
    scope: 'database',
    db: 'Northwind',
    command: 'eval',
    args: 
    {
    code: 'db.CustomerTest.insert( [ {CustomerID:"AAA", Orders:[]}, {CustomerID:"BBB", Orders:[]}, {CustomerID:"CCC", Orders:[{OrderID:"1004","OrderDate" : "2008-02-05","Qty" : "5"} ]} ] )' 
    
    } 
   }

2. Now drag Data flow.
3. Drag ZS CSV Source or any other source. For example purpose we will configure CSV Source using Direct Value option with following Sample data. In the first run of package it will update only orders for CCC customer because only that customer will have records after table creation. one row. But in 2nd run all rows updated.
   

   CustomerID,Company,OrderID,OrderDate,Qty,Address
   AAA,Anthony Inc,1000,2008-01-01,90,Po Box 111\4612
   AAA,Anthony Inc,1001,2010-02-01,91,Po Box 111 / 4612
   BBB,Bob Inc,1002,2008-02-01,92,Po Box 222 / 4612
   BBB,Bob Inc,1003,2010-01-01,93,Po Box 222 / 4612
   CCC,Cindy Inc,1004,2010-01-07,2,Po Box 555 / 2345

    
4. Click OK to save Source UI
5. Now drag ZS Template Transform to build XML document (Data + Query). Connect Source to Template transform. You can also use ZS XML Generator Transform but for simplicity we will use Template Transform.
6. Double click to open Template Transform and enter text as below. Notice how we have entered MongoDB Query and Data in two separate XML nodes. First node is custom query for lookup. Second node is Operation we like to perform (e.g. $set in our case to update existing data). You can use Insert Variable and then select Columns option to insert placeholders. Also in some fields we used JSONENCODE function to make sure we escape double quote and slash correctly.
   

   <command>
   	<query><![CDATA[
   {
   	"CustomerID": "<%CustomerID%>", 
   	"Orders.OrderID": { "$eq" : "<%OrderID%>"}
   }
   ]]></query>
   	<document><![CDATA[
   {
     $set :
      { "CompanyName":"<%Company,JSONENCODE%>", 
        "Address":"<%Address,JSONENCODE%>", 
        "Orders.$.OrderID": "<%OrderID%>", 
        "Orders.$.OrderDate": "<%OrderDate%>", 
        "Orders.$.Qty": "<%Qty%>" 
       } 
     }
   }
   ]]></document>
   </command>

   

   Using ZS Template Transform to specify custom JOIN condition for MongoDB Update (Array Items)

7. Click OK to save Template Transform
8. Drag ZS MongoDB Destination. Connect Template Transform to Destination
9. Double click MongoDB Destination to edit. Set Connection, Table name, Operation=Update, EnableCustomLookupQuery=True
   

   Configure MongoDB Destination for Custom Join Condition (EnableCustomLookupQuery Setting)

10. Goto Mappings Tab and attach TemplateOutput to __DOCUMENT__
    

    MongoDB Mappings – Loading Document for Array Update (Custom Join Criteria)

11. Click OK to Save

Insert into Array using Custom Lookup Condition

Now let’s look at slightly modified steps to Insert into Array for new records. In this example we will insert Source records if CustomerID found but OrderID is not found. (See Template Transform Step). We will use Not Equal Condition this time.

1. Now drag Data flow rename it to something like [Add records to array]
2. Drag ZS CSV Source or any other source. For example purpose we will configure CSV Source using Direct Value option with following Sample data
   

   CustomerID,Company,OrderID,OrderDate,Qty
   AAA,Anthony Inc,1000,2008-01-01,30
   AAA,Anthony Inc,1001,2010-02-01,4
   BBB,Bob Inc,1002,2008-02-01,30
   BBB,Bob Inc,1003,2010-01-01,4

    
3. Click OK to save Source UI
4. Now drag ZS Template Transform to build XML document (Data + Query). Connect Source to Template transform. You can also use ZS XML Generator Transform but for simplicity we will use Template Transform.
5. Double click to open Template Transform and enter text as below. Notice how we have entered MongoDB Query and Data in two separate XML nodes. First node is custom query for lookup. Second node is Operation we like to perform (e.g. $addToSet in our case to insert into array )
   

   <command>
   <query>
   {"CustomerID": "<%CustomerID%>", "Orders.OrderID":{ "$ne" : "<%OrderID%>"}}
   </query>
   <document>{
     $addToSet :
      { Orders : {"OrderID": "<%OrderID%>", "OrderDate": "<%OrderDate%>", "Qty": "<%Qty%>" } }    
    }
   </document>
   </command>

6. Click OK to save Template Transform
7. Drag ZS MongoDB Destination. Connect Template Transform to Destination
8. Double click MongoDB Destination to edit. Set Connection, Table name, Operation=Update, EnableCustomLookupQuery=True
9. Goto Mappings Tab and attach TemplateOutput to __DOCUMENT__
10. Click OK to Save

Download Sample SSIS Package (2012 format) – MongoDB_Upsert_ArrayItem

Conclusion

MongoDB integration can be challenging if you are new to NoSQL world. If you are using SSIS as your primary ETL tool then not to worry because SSIS PowerPack can give super power needed to complete your project on time with drag and drop high performance connectors.

Keywords: ssis mongodb upsert array item | ssis mongodb update array item | ssis mongodb update array elements | mongodb update array documents | MongoDB $set operator | MongoDB $addToSet operator | MongoDB update sub document items | MongoDB CRUD operations | MongoDB Bulk Updates | MongoDB bulk updates to array items

The post Update or Insert – Upsert MongoDB Array Items using SSIS appeared first on ZappySys Blog.

In this article we’re going to focus on load data in Salesforce using SSIS (Bulk Insert, Update, Delete and Upsert). If you like to know how to read from Salesforce then refer to previous post here. For Demo purpose we will use AdventuresWorks sample database as our source but you can use your own Database for testing.

Salesforce developer account

To develop ETLs in Salesforce using SSIS you’ll need a Salesforce developer account. If you don’t have one, don’t worry, you can create a limited version FREE developer account. Click here to Sign up for FREE developer account.

Video Tutorial – Salesforce Insert, Upsert, Delete, Update

Getting Started

In this section, we will see how to perform bulk insert, update delete and upsert operations using ZS Salesforce Destination

Configure Salesforce Connection

First, we’ll need setup a connection to read or write Salesforce data is to make sure we have a working connection in SSIS.

1. Create new or open existing SSIS Project in Visual Studio.
2. Open SSIS Package and Go to SSIS designer.
3. Right click in the connection manager panel and click New connection
   

   Create a new SSIS Salesforce Connection

4. In the connection type selection, Select the ZS-SALESFORCE Connection.
5. We’ll use this information: Username, Password, Security Token (click the link learn how to obtain Salesforce Security Token )
6. Click OK to add the connection.
   

   Salesforce using SSIS, Salesforce connection

7. Save all changes

Bulk Insert – Import data into Salesforce using SSIS

Let’s start with the a bulk insert in Salesforce using SSIS. On this section, we’re going to generate retrieve data from and OLEDB Source and proceed to Insert the records into Salesforce using ZS Salesforce Destination.

Configure SQL Server Source (OLEDB)

1. Open SSIS Package and drag Data Flow task from Control Flow SSIS Toolbox
   

   Dragging and dropping Data Flow Task into Control Flow

2. Go to data flow designer. Drag and drop an OLE DB Source
3. Open the OLE DB Source and configure as follow:
   

   Configure SQL Server Source

Configure SSIS Salesforce Destination for Bulk Insert

Once SQL Server source is configure. Now let’s configure Salesforce Destination for Bulk Insert like below.

1. Drag and drop ZS Salesforce Destination
2. Open the destination component
3. Connection Manager
   

   Salesforce using SSIS, upsert destination connection manager

4. Component Properties
   

   salesforce-using-ssis-insert-properties

5. Column Mappings
   

   salesforce-using-ssis-insert-mapping

6. Click OK
7. Drag and drop two Trash Destination components
8. Connect ZS Salesforce Destination for: New records Output and Error Output
9. Right click on the joining arrows and enable data viewers on all previous connections
10. Save all changes
11. Right click in the data flow designer, click Execute Task
    

    salesforce-using-ssis-insert-results

Bulk Update – Modify data in Salesforce using SSIS

In this section we’re going to perform a bulk update on Salesforce using SSIS. We’re going to list some information from our source, list all our Salesforce Account data, merge it and only update the data we match. In below example what we will do is Find out matching records in Salesforce and our Source System based on some External Key field (e.g. AccountNumber in our case). And then obtain Salesforce Record Id for matching AccountNumber and use it to Update target record.

Configure SQL Server Source (OLEDB Source)

1. Open SSIS Package and drag Data Flow task from Control Flow SSIS Toolbox
2. Go to data flow designer. Drag and drop OLE DB Source
3. Open the OLE DB Source and configure
   

   Salesforce using SSIS, Update OLEDB source

Configure SSIS Salesforce Source

1. Now lets Drag and drop ZS Salesforce Source
2. Open the ZS Salesforce and configure. You can select Query Mode and enter query like this because we only care about two fields for now. Use query mode whenever possible because it transfers less data.
   

   select Id,AccountNumber from account

   

   Salesforce using SSIS, Update Salesforce source

Configure Sort Transforms

Now lets sort both previous datasets using Sort Transforms. This is needed to JOIN dataset using Merge Transform (See next section)

1. Drag and drop: 2 Sort and 1 Merge Join components
2. Connect the OLE DB Source to the first Sort component, and configure
   

   Salesforce using SSIS, Update OLEBD sort

3. Connect the Salesforce Source to the second Sort component, and configure
   

   Salesforce using SSIS,update Salesforce sort

Configure Merge Transforms

No next step would be to join previous datasets based on AccountNumber and then we will output Id from Salesforce which is needed for later step.

1. Drag and Drop Merge Transform from SSIS Toolbox like below.
   

   Drag SSIS Merge Join Transform from Toolbox

2. Connect Sorts to the Merge component (sort from OLEDB Source most be on the left)
3. Configure the Merge component
   

   Salesforce using SSIS, update merge

Configure SSIS Salesforce Destination for Bulk Update

1. Drag and drop ZS Salesforce Destination
2. Open the ZS Salesforce Destination and configure
3. Connection Manager tab
   

   Salesforce using SSIS, update destination connection manager

4. Component Properties tab
   

   Salesforce using SSIS, update destination component properties

5. For Update purposes, Salesforce Destinations needs to map the ID from the Account object. For more information, please use the following link SSIS Salesforce Destination – Update / Insert with Lookup Fields (Text to Id)
6. Column Mapping
   

   salesforce using SSIS, update destination column mappings

7. Click OK
8. Drag and drop three Trash Destination components
9. Create two connections from ZS Salesforce Destination as follows: Update records and Error Output
10. Right click on the joining arrows and enable data viewers on all previous connections
11. Save all changes
12. Right click in the data flow designer, click Execute Task
    

    Salesforce using SSIS, update results

Bulk Upsert – Import/Modify data in Salesforce using SSIS

In this section we’re going to perform a bulk upsert of 25 record’s on Salesforce using SSIS. Upsert is an operation that combines both insert an update.

ZS Salesforce Destination allows Upserts operations. You’ll need to select the Upsert as an action and indicate the Key value for the comparison.

Creating External ID field for Salesforce Upsert

For the Upsert action we have to use an Upsert Keyfield (i.e. External ID field). For more information read here External ID. Follow these steps to add a External ID field on Salesforce Accounts object:

1. Login to your Salesforce account.
2. Click on the configuration icon and select “Service Setup”
   

   Salesforce using SSIS, Salesforce edit object

3. On the new Tab, look for USER INTERFACE -> Objects And Fields -> Object Manager
4. Select Account object
5. Select Fields & Relationships
6. Click New, to create a field
   

   Salesforce using SSIS, Salesforce Fields & Configuration

7. For the field type, choose Number and click next
8. For Field Label and Field Name, type TableID. Check Unique and External ID boxes and click next
9. Leave field-level security as default, click next
10. Leave page layouts as default, click next.
11. After finishing the configuration, the field should appear in the Fields & Relationships page

Configure SQL Server Source

1. Open SSIS Package and drag Data Flow task from Control Flow SSIS Toolbox
   

   Dragging and dropping Data Flow Task into Control Flow

2. Go to data flow designer. Drag and drop OLE DB Source
3. Open the OLE DEB Source and configure as follow
   

   Salesforce using SSIS, Upsert OLE DB source

Configure SSIS Salesforce Destination for Upsert

1. Drag and drop ZS Salesforce Destination
2. Open the ZS Salesforce and configure
3. Connection Manager
   

   Salesforce using SSIS, upsert destination connection manager

4. Component Properties
   

   Salesforce using SSIS, upsert destination component properties

5. Column Mapping
   

   salesforce using SSIS, upsert destination column mapping

6. Click OK
7. Drag and drop three Trash Destination components

Configure Error Handling, Outputs for New, Updated, Failed Records

If you like to capture which rows are inserted and which rows are updated then you can drag Blue arrows coming from Salesforce destination component. You can also connect Red arrow to detect failed records and reason for error.

1. Connect ZS Salesforce Destination for: New records Output, Updated Records Output and Error Output
2. Right click on the joining arrows and enable data viewers on all previous connections
3. Right click in the data flow designer, click Execute Task
   

   Salesforce using SSIS, upsert results

Bulk Delete – Delete data in Salesforce using SSIS

In this section, we’re going to perform a bulk delete on Salesforce using SSIS ZS Salesforce Destination. In this example we’re going to list all our source customers (from previous sections), merge it wit the Salesforce Account object, retrieve the Salesforce Account Id and delete the rows. Delete Operation also requires to map Id field so we will use similar technique we used for Update.

Configure SQL Server Source (OLEDB Source)

1. Go to data flow designer. Drag and drop OLE DB Source
2. Open the OLE DEB Source and configure as follow
   

   salesforce-using-ssis-delete-oledb-source

Configure SSIS Salesforce Source

1. Drag and drop ZS Salesforce Source
2. Open the ZS Salesforce and configure.
3. You can select Query Mode and enter query like this because we only care about two fields for now. Use query mode whenever possible because it transfers less data.
   

   select Id,AccountNumber from account

Configure Sort Transforms

Now lets sort both previous datasets using Sort Transforms. This is needed to JOIN dataset using Merge Transform (See next section)

1. Drag and drop: 2 Sort and 1 Merge Join components
2. Connect the OLE DB Source to the first Sort component, and configure
   

   Salesforce using SSIS, Update OLEBD sort

3. Connect the Salesforce Source to the second Sort component, and configure
   

   Salesforce using SSIS,update Salesforce sort

Configure Merge Transforms

No next step would be to join previous datasets based on AccountNumber and then we will output Id from Salesforce which is needed for later step.

1. Now drag Merge Transform from SSIS toolbox
   

   Drag SSIS Merge Join Transform from Toolbox

2. Connect both Sort to the Merge component (sort from OLEDB Source most be on the left)
3. Configure the Merge component
   

   salesforce-using-ssis-delete-merge

Configure SSIS Salesforce Destination for Bulk Delete

1. Drag and drop ZS Salesforce Destination
2. Connect the Merge and ZS Salesforce Destination components. Right click the arrow and enable data viewer
3. Connection Manager tab
   

   Salesforce using SSIS, delete destination connection manager

4. Component Properties tab
   

   Salesforce using SSIS, delete destination component properties

5. For Delete purposes, Salesforce Destinations needs to map the ID from the Account object.For more information, please use the following link SSIS Salesforce Destination – Update / Insert with Lookup Fields (Text to Id) 
6. Column Mapping
   

   salesforce-using-ssis-delete-destination-column-mapping

7. Click OK
8. Save all changes
9. Right click in the data flow designer, click Execute Task
   

   salesforce-using-ssis-delete-results

Error Handling

When working on Salesforce with SSIS, we recommend Trash Destination component to help debugging outputs without performing DML on destinations.

Main benefits of usage:

1. Enable data viewers and get live feedback of the data
2. Save the results on different formats (XML, JSON, plain text files).

Conclusion

In this article, We have seen how easy it is to integrate Salesforce using SSIS. Using drag and drop connector you can perform Bulk insert, update, delete and Upsert operations.

Download ZappySys SSIS PowerPack to learn more about other possible scenarios which are not discussed in this article.

The post Load data in Salesforce using SSIS – Insert, Upsert, Delete, Update appeared first on ZappySys Blog.

In our previous article, we talked about how to read from Dynamics CRM / Dataverse (Dynamics 365 / On-Premises CRM / PowerApp). Now let’s look at how to Load data into Dynamics CRM / Dataverse (Dynamics 365 / On-Premises CRM / PowerApp) using SSIS. We will see a few important aspects of loading data into Dynamics CRM / Dataverse Online (i.e. Dynamics 365) and On-premises / IFD (Internet facing hosted instance).

We will show you how to use SSIS Dynamics CRM Destination connector for Bulk Insert, Update, Delete and Upsert Operations. This article uses Dynamics 365 Online for CRM Integration Examples but concepts are same for other types of installation (e.g. IFD or On-Premises)

Requirements

1. Make sure you have SSIS Designer installed (i.e. SSDT-BI)
2. Download and Installed ZappySys SSIS PowerPack which includes an SSIS Dynamics CRM Connectors.
3. Finally, make sure you have Microsoft Dynamics CRM / Dataverse credentials handy.

That’s it. So let’s get started.

Video Tutorial – CRM Insert, Upsert, Delete, Update

Getting Started

In this section, we will see how to perform bulk insert, update delete and upsert operations using  SSIS Dynamics CRM Destination

Configure SSIS Dynamics CRM / Dataverse Connection

The first step to read or write Dynamics CRM / Dataverse data is to make sure we have working Dynamics CRM connection in SSIS.

1. Create new or open existing SSIS Project in Visual Studio.
2. Open SSIS Package and Go to SSIS designer.
3. Right click in the connection manager panel and click New connection:
   

   Create a new SSIS Dynamics CRM / Dataverse Connection

4. In the connection type selection, Select the ZS-DYNAMICS-CRM Connection.
5. We will use a CRM Online (Office 365) connection, region, a username and a password and select organization from the dropdown as below and then click Test connection:
   

   Configure Dynamics CRM / Dataverse Connection in SSIS

Bulk Insert – SQL Server data Import into Dynamics CRM / Dataverse using SSIS

Now, let’s see steps to do SQL Server data Import into Dynamics CRM / Dataverse using SSIS. Below steps assume you already have sample data in some SQL Server Table which will be used to create new CRM accounts. You can enter few account names you like to create in the CRM.

1. Open SSIS Package and drag Data Flow task from Control Flow SSIS Toolbox
2. Go to data flow designer. Drag and drop OLEDB Source
3. Configure OLEDB Source to read from some sample data table (e.g. dbo.Accounts). If you want to be lazy and just test quickly you can use ZS CSV Source (Direct Value Mode) instead of OLEDB Source.Sample Data May be like below
   

   name,address1_city,address1_country
   Flipkart,Bengaluru,India
   Microsoft,seattle,USA
   Nokia,Espoo,Finland

   Once you configure Source (Connection, Table Name etc.) click OK to save it.
   

   CSV Source – Sample Accounts for CRM Insert Demo

4. Drag and drop  ZS Dynamics CRM Destination from the toolbox.
5. Connect Source to your Destination component.
6. Double click CRM destination to configure it.
7. On the Connection Managers tab, Select Dynamics CRM / Dataverse connection we created in the previous section.
   

   Select Connection for Dynamics CRM / Dataverse Destination

8. On the Component Properties Tab Select Table Name.
   

   Select Target Table for Dynamics CRM / Dataverse Insert

9. On the Mapping Tab map few columns (e.g.  name, address1_city, and address1_country)
   

   Configure Target Column Mapping for CRM Data Load

10. Click OK to save CRM destination.
11. Right click in the data flow designer, click Execute Task
    

    Execute SSIS Package – Loading Data into Dynamics CRM / Dataverse from CSV / SQL Server

Bulk Upsert – Update or Insert Dynamics CRM / Dataverse Records in SSIS

In the previous section we saw how to Insert New records. However sometimes you want to Upsert (Insert or Update). So if record already exists Update else Insert. In order to achieve this you have to set two additional attributes. For CRM Upsert you have to perform few changes to previous steps

Create Alternate Key for CRM Upsert

To perform Upsert operation you can map Table’s Primary Key or Alternate Key. Primary Key usually starts with table name and ends with “id” (e.g. for account table its accountid ).

Many times Source system don’t have CRM Table Primary Key. In such case you may use Alternate Key in the mapping (JOIN Key). To use Alternate key for Upsert you have first define Alternate Key in CRM Portal. Check this link for exact steps.

Create Alternate Key for Dynamics CRM / Dataverse Online Table (Useful for Update or Upsert Operation)

Configure SSIS CRM Destination for Upsert

1. Double click CRM destination. On the Component Properties Tab, Change Operation to UpdateOrInsert
2. Specify UpsertKeyField to either Table’s Primary Key Column Name or Alternate Key Name.
   

   Dynamics CRM / Dataverse Upsert Records using SSIS CRM Destination (Specify Alternate Key / Primary Key)

3. Now go to Mappings tab and confirm Key field is mapped.
   

   Configure Dynamics CRM / Dataverse Mapping for Upsert in SSIS

4. Click OK and Run Package to test CRM Upsert Operation.

If you want to use primary key for Upsert or Update Operation and not sure correct column name then you may use ZappySys Dynamics CRM / Dataverse Source FetchXml Query Mode and check Object Browser (Table Info Tab >> PrimaryIdAttribute in the grid).

Primary Key name usually starts with table name and ends with “id” (e.g. for account table its accountid ).

Find Primary Key for Dynamics CRM / Dataverse Table

Handling Inserted / Updated Records Output

Now lets look at how to detect which records got inserted and which records got Update? In Upsert Mode each record can be directed to one of the following three Outputs.

• Inserted Records [Blue Arrow]
• Updated Records [Blue Arrow]
• Records with Error [Red Arrow]

You can select Red or Blue arrow coming out from CRM Destination component. Drag and drop output arrow on any down stream component.

Bulk Update – Modify Dynamics CRM / Dataverse Records in SSIS

If you want to do bulk Update its very same steps as Upsert Operation except you have to set Operation = Update in the properties grid. For Key field you have to set primary key or Alternate Key.

Dynamics CRM / Dataverse Update in SSIS

Bulk Delete – Delete Dynamics CRM / Dataverse Records in SSIS

Bulk delete doesn’t require many settings. All you have to change is Operation = Delete on Component Properties Screen. On the Mapping Screen Map only one column (Primary Key). If you are not sure which one is primary key please refer to the previous section (Upsert Operation).

Delete records from Dynamics CRM / Dataverse Online using SSIS (Primary Key mapping for CRM Bulk Delete)

Some times you may get below errors. If you get this error make sure you delete all child records first before deleting parent. For example if you are deleting account then make sure you have to delete leads / contacts for that account or update leads / contacts so they don’t refer to account being deleted.

Error: 0xC0047062 at Dynamics CRM / Dataverse Bulk Delete, Delete Accounts [2]: System.Exception: Exception: The object you tried to delete is associated with another object and cannot be deleted.
Error Details: 
ApiExceptionSourceKey=Plugin/Microsoft.Crm.Common.ObjectModel.AccountService

Error Handling

Many times you want to ignore bad records and continue processing good records. In such case you can use Dynamics CRM / Dataverse Error Output. Just grad Red arrow from CRM Destination and Drop on target component. Select Redirect Option for Error , Truncation both and Click OK.

Once you set Error Handling your component will continue to run without failing whole data flow. Later you can review errors (if you saving to destination).

Configure Error Handling in SSIS Dynamics CRM / Dataverse Destination

Loading records into Dynamics CRM / Dataverse Table using SSIS . Redirect Bad rows to Error Output.

Handling Special Data Types (PartyList, Lookup, PickList)

Certain datatypes in CRM requires special handling. Check this article to learn more about how to insert data into columns with PartyList, Lookup or PickList field.

Conclusion

In this article, We have seen how easy it is to integrate SQL Server data with Dynamics CRM / Dataverse. Using drag and drop connector you can perform Bulk insert, update, delete and Upsert operations. Download ZappySys SSIS PowerPack to learn more about other possible scenarios which are not discussed in this article.

The post Load data into Dynamics CRM using SSIS – Insert, Upsert, Delete, Update appeared first on ZappySys Blog.

In our previous blog post we saw how to perform Read and Write operations in MongoDB using SSIS (i.e. Bulk Update, Delete, Upsert, Insert). In this post we specifically focus on how to update MongoDB Array items / elements using SSIS.

To make things simple to follow we have used JSON Source to produce sample JSON documents for MongoDB update (NOTE: Output as Document option is checked for JSON Source). However in real world you may have to use SSIS JSON Generator Transform to produce input documents for MongoDB. To learn more about generating JSON documents for MongoDB Load process check this blog post or watch video found on this page.

Video Tutorial – Example of insert/update documents inside MongoDB Array (Use $set, $push operator)

Basic Concepts: Performing MongoDB CRUD operations using SSIS

If you never heard term called CRUD stands for Create, Read, Update, Delete. These are the most common operations you have to do with any data source. SSIS PowerPack comes with many MongoDB Components to help you with CRUD operations. You can use one or more following components to achieve drag and drop MongoDB CRUD operations in SSIS. When you use SSIS PowerPack for MongoDB all operations will be Bulk Operations by default which means it will provide very high throughput.

• SSIS MongoDB Source
• SSIS MongoDB Destination
• SSIS MongoDB ExecuteSQL Task

Download Example SSIS Package

Here is the link to download Sample SSIS Package

SSIS Example : Update MongoDB Array Items / Elements (Update, Delete, Insert – Using $set, $pull, $push)

Update MongoDB Array Items ($set operator)

In next couple of sections we will see how to perform Update, Insert and Delete operations with MongoDB Array Items using various update operators and array operators. Consider the following Sample JSON documents for MongoDB. First we will see how to use MongoDB Shell commands to perform CRUD operations on MongoDB Array items.

{
    "_id" : ObjectId("5883ebe669e0f22ba890b49b"),
    "CustomerID" : "BOLID",
    "Name" : "Some name",
    "Orders" : [ 
        {
            "OrderID" : 200,
            "ShipCountry" : "USA",
            "City" : "Atlanta"
        }, 
        {
            "OrderID" : 201,
            "ShipCountry" : "USA",
            "City" : "New York"
        }
    ]
}

Update Array element using Shell command

Assume you want to update ShipCountry attribute of one of the Orders (OrderID=200) and for CustomerID=”BOLID” in above sample document.

Shell Command for Update Array Item

Use below shell command in your favorite MongoDB Client Tool (e.g. RoboMongo). Notice two things in below command because we will apply same concepts when we take this approach to SSIS. In below command Actual JSON document is wrapped inside. This is MongoDB Update operator. Click here to learn more about $set. Also notice that rather than nested Array syntax we used Orders.$.ShipCountry to update ShipCountry which is part of Array element. MongoDB provides special way to navigate array elements using   $ Operator (MongoDB Positional Update)

db.MyCollection.update(
	{ "CustomerID" : "BOLID", "Orders.OrderID" : 200 }, 
	{ $set : { "Orders.$.ShipCountry" : "Test1"} } , 
	{ upsert: false, multi: false}
)

After Update (MongoDB Document)

After you run above shell command notice all other attributes remain there in Array document and only ShipCountry is changed. This is because we used $ Operator (MongoDB Positional Update) rather than supplying Raw JSON for Array document (i.e. Orders : [ { … } ] ).

{
    "_id" : ObjectId("5883ebe669e0f22ba890b49b"),
    "CustomerID" : "BOLID",
    "Name" : "Some name",
    "Orders" : [ 
        {
            "OrderID" : 200,
            "ShipCountry" : "Test1",
            "City" : "Atlanta"
        }, 
        {
            "OrderID" : 201,
            "ShipCountry" : "USA",
            "City" : "New York"
        }
    ]
}

The issue with this approach its not Bulk operation. Assume that you have 100,000 records to update in various documents and its nested array items then row by row operation is not effective. To solve this issue we will use SSIS MongoDB Destination which performs Bulk operations (e.g. Bulk Update, Delete, Insert and Upsert).

Update MongoDB Array Items using SSIS

Now lets implement above scenario using SSIS. Assume that you have sample MongoDB document (as per above example – Before Update). We want to update ShipCountry for OrderID=200 for Customer BOLID. To do that perform following steps in SSIS. If you are updating Second level array (i.e. Root >> Orders >> OrderItems)  then you have follow slight different process. Check next section in this article about updating second level array (Insert items)

1. Download and Install SSIS PowerPack
2. Create new SSIS Project with one data flow task
   

   Dragging and dropping Data Flow Task into Control Flow

3. Drag ZS JSON Source and check Output as Document option (as below sceenshot). You can use JSON Source is your JSON is stored inside file or Variable or coming from Some API call which eventually go into MongoDB. You can also use JSON Source for quick testing where you Hardcode JSON to feed to MongoDB.
4. Enter the following JSON in the text area. Click OK to save.
   

   {"CustomerID":"BOLID", "Orders.$.OrderID":201, "Orders.$.ShipCountry": "Test1"}
   {"CustomerID":"BOLID", "Orders.$.OrderID":202, "Orders.$.ShipCountry": "Test2"}
   /**Below is extra record not found in destination - will be ignored in Update**/
   {"CustomerID":"BOLID", "Orders.$.OrderID":203, "Orders.$.ShipCountry": "Test3"}

5. Right click in Connections area and Create new ZS-MONGODB connection. Specify credentials and click Test before you hit OK to save.
6. Now drag ZS MongoDB Destination from SSIS Toolbox.
7. Select Runtime connection.
8. On the properties tab edit following properties
   1. Set ColumnsForLookup property as below (Notice how we use 2 columns for JOIN. Also used AS keywords for 2nd Column so we have <column-from-input-doc> AS <column-for-server-side-find>
      CustomerID,Orders.$.OrderID AS Orders.OrderID
   2. Set LoadOptions as below (Op is Update Operator you like to use. In our case its $set, Multi=True means if multiple match found then all matching documents will be updated). If you want to use multiple update operators then you have to supply that from Input document and use op=none option rather op=$set.
      op=$set;multi=true
   3. Set Operation property to Update
9. On the Mappings tab Map __DOCUMENT__ from upstream (In our case you will see same name column from JSON Source if you checked Output as Document option)
10. Click OK to save MongoDB Destination UI
11. Run package. You will notice after execution two orders of CustomerID=BOLID will be updated.

SSIS JSON Source – Read JSON (Raw Documents)

SSIS MongoDB Destination – Update MongoDB Array Item using $set update operator

Insert MongoDB Array Item ($push / $addToSet Operator)

Now lets look at how to insert new item into existing Array. Lets say we already have two orders for a customer and now we want to add new order. For this you can either use  $push operator or $addToSet operator. $push operator doesn’t check for duplicate item so its little faster. If you wish to skip append if item already exists (by matching all attributes) then use $addToSet operator.

Using Shell Command : Insert MongoDB Array Item (Append to array)

Here is the shell command which will push new item to the array

db.MyCollection.update(
	{CustomerID:"BOLID"},
	{$push:{Orders: {"OrderID":202,"ShipCountry":"USA","ShipCity":"Atlanta" } }},
	{upsert: false,multi: false}
)

Using Shell Command : Insert MongoDB Array Item at second level

Here is the shell command which will push new item to the array which is stored at second level e.g. { Orders :  [ { Items : [ insert-here ] } ] }

db.MyCollection.update(
	{CustomerID:"BOLID", "Orders.OrderID" : 202 },
	{$push:{ "Orders.$.Items" : {"ProductID":1001,"Quantity":3} } },
	{upsert: false,multi: false}
)

Using SSIS : Insert MongoDB Array Item at Second Level

Now lets look at some example how to Insert Items inside Orders array. Since it requires two JOIN columns to perform second level array insert you may need to add metadata for join.

First step is create documents for update. There are atleast 4 ways you can create input documents for MongoDB Destination.

1. You can use JSON generator Transform to create documents for load process
2. You can use Export JSON Task to generate input documents file and then use JSON Source to Read Documents (Check Output as Document option on JSON Source to read as RAW JSON)
3. You can use JSON Source to feed direct JSON (This approach is demonstrated in the below example sceenshot)
4. You can also use SQL query to output JSON Strings along with columns used for JOIN condition
   

   select CustomerID, OrderID as 'Orders.$.OrderID', DOC = '{Orders.$.Items : {ItemID: 1, Qty: 5} }'

To make it simple lets use #3 approach from above list. We will create few documents and insert New orders Items

Sample Document for MongoDB Collection

For testing purpose you can create MongoDB collection called test with following one document. As you see Items array is empty for both orders but once you run sample each will have 2 items.

{
    "_id" : ObjectId("58c01bee0610fea8ddda04b9"),
    "CustomerID" : "ALFKI",
    "Orders" : [ 
        {
            "OrderID" : 100,
            "Items" : [ ]
        }, 
        {
            "OrderID" : 101,
            "Items" : [ ]
        }
    ]
}

Now lets look at how to configure SSIS package

Step-1 : Create JSON Documents for Array Insert

Lets use JSON Source like below. Use following Sample JSON

/*Insert new items into order#100*/
{CustomerID:"ALFKI","Orders.$.OrderID" : 100, Doc: "{ \"Orders.$.Items\" : {ItemID: 1, Qty: 10} }" }
{CustomerID:"ALFKI","Orders.$.OrderID" : 100, Doc: "{ \"Orders.$.Items\" : {ItemID: 2, Qty: 11} }" }

/*Insert new items into order#101*/
{CustomerID:"ALFKI","Orders.$.OrderID" : 101, Doc: "{ \"Orders.$.Items\" : {ItemID: 1, Qty: 20} }" }
{CustomerID:"ALFKI","Orders.$.OrderID" : 101, Doc: "{ \"Orders.$.Items\" : {ItemID: 2, Qty: 21} }" }

Create Sample JSON Documents for MongoDB Array Update (2nd level array)

Step-2 : Configure MongoDB Destination for $push operation on Second level Array

Now lets connect source to Destination and configure MongoDB Destination. Select connection on first tab. Then On second tab change properties like below.

Configure MongoDB Destination for second level array update (Insert item using $push update operator)

Step-3 : Add missing JOIN column for Mapping

When you click on Mappings tab you will notice only root level columns are listed there for mapping but if you are JOINing based on Nested column found inside array then you have to add missing mapping column by hand like below screenshot.  In our case we Join by CustomerID and OrderID where OrderID is found in Orders array which is missing on mappings tab. So lets add it. Just make sure name new column using $ positional operator… e.g. Orders.$.OrderID  this will tell MongoDB that our OrderID column must be lookup under Array called Orders: [  ]

MongoDB Destination – Add missing metadata column for JOIN

Step-4 : Map JOIN columns and Input DOCUMENT

Now finally map all columns

SSIS MongoDB Destination – Column Mappings

Using SSIS : Insert MongoDB Array Item (Append to array)

Now lets look at how to perform same operation to insert array item in SSIS using $push operator. In this case we will do little different than Update Item example ($set operator). First we have not included CustomerID as part of Input columns and we mapped it on Mapping Screen rather than supplying columns names in ColumnsForLookup property (On Component Properties Tab of Destination UI). If you are using JSON Generator Transform to generate your JSON then you have option to Hide/Include certain columns part of your flow. Check Include column to downstream option if you want to use for JOIN column.

Set following settings

• Set LoadOptions as below
  op=$push;multi=true
• Set Operation=Update
• Leave ColumnsForLookup as empty
• On the mappings tab map __DOCUMENT__ and CustomerID in the Target. If you don’t see CustomerID column in the mapping create atleast one sample doc in MongoDB Collection so metadata is detected.

Download Sample for more detail.

SSIS JSON Source – generate JSON for MongoDB Array Insert

SSIS MongoDB Destination – Insert MongoDB Array Item (Use $push, or $addToSet operator)

Delete MongoDB Array Item ($pull operator)

If you wish to remove array item then you can use $pull operator. Only difference is rather than specifying two conditions in first argument… you just specify JOIN columns without array column (i.e. OrderID). To specify Array item condition you have to set inside $pull : { … }

Using Shell Command for MongoDB Delete Array Item (Remove by condition)

Below example shows how to remove OrderID=201 from Orders array for CustomerID=BOLID.

db.MyCollection.update(
	{CustomerID:"BOLID"},
	{$pull:{Orders: {"OrderID":201} }},
	{upsert: false,multi: false}
)

Using SSIS for MongoDB Delete Array Item

Now lets look at how to implement Delete logic for MongoDB Array Item in SSIS. As you see in below screenshots You have to modify following things

• Set ColumnsForLookup property to CustomerID (no need to add OrderID here because its part of input document)
• Set LoadOptions property to op=none;multi=false. op=none means we are supplying Update operator part of input document.

Assume you have doc like below in MongoDB  and you like to remove Order#101 from array

{
    "_id" : ObjectId("58c01bee0610fea8ddda04b9"),
    "ParentID" : 1,
    "Orders" : [ 
        100, 
        101, 
        102
    ]
}

For this supply 2 columns from upstream
col1  = ParentID
col2  = DOCUMENT

Document column should look like below
{Orders: 101}

1. On MongoDB destination change LoadOptions property to  op=$pull;multi=true
2. Change Operation to Update
3. Map ParentID and DOCUMENT columns on Mapping screen
4. Run the package … It will remove Order# 101 from array

Array Upsert – Update / Insert using Custom JOIN condition (such as $ne )

If you like to perform Upsert Operation on MongoDB Array Items then refer to below blog post.

Update or Insert – Upsert MongoDB Array Items using SSIS

Conclusion

MongoDB integration can be challenging if you are new to NoSQL world. If you are using SSIS as your primary ETL tool then not to worry because SSIS PowerPack can give super power needed to complete your project on time with drag and drop high performance connectors.

Keywords: mongodb update array item | mongodb update array elements | mongodb update array documents | MongoDB $ operator | MongoDB update sub document items | MongoDB CRUD operations | MongoDB Bulk Updates | MongoDB bulk updates to array items

The post Update MongoDB Array Items using SSIS appeared first on ZappySys Blog.

MongoDB is one of the most poplar NoSQL database out there. In this article you will learn how to use ZappySys MongoDB Destination Connector to perform various bulk operations on MongoDB such as Insert, Update, Delete and Upsert. If you wish to learn how to Update MongoDB Array items then check this blog post instread.

MongoDB data read/write Example SSIS Package

Before we talk more about loading data into MongoDB lets look at screenshot and steps involved in our sample package for this tutorial. Below sample SSIS package uses 4 Custom SSIS Tasks

1. SSIS MongoDB Source Connector (Used to read data from MongoDB Collection)
2. SSIS MongoDB Destination Connector (Used to write data into MongoDB Collection also you can do other bulk operations such as MongoDB update, MongoDB upsert and MongoDB delete)
3. SSIS MongoDB ExecuteSQL Task (Used to call any MongoDB Shell commands including MongoDB DDL or DML type operations such as CREATE/DROP, Insert, Update, Delete, Read, you can also call server side JavaScript)
4. Dummy Data Source (Used to generate sample JSON Documents which can be loaded in MongoDB)

MongoDB Examples

Here is the list of various MongoDB examples included in attached sample package (see at the end for download links)

• MongoDB example – How to read data from MongoDB collection using SQL query language
• MongoDB example – How to load raw JSON document into MongoDB Collection (Bulk Insert with Reload)
• MongoDB example – How to do MongoDB Upsert (Bulk Update or Insert) for specific fields or entire document
• MongoDB example – How to perform MongoDB Delete ( Bulk Delete based on Matching Rows)
• MongoDB example – How to create new MongoDB collection (Only if not exist – Use safe option)
• MongoDB example – How to fetch MongoDB collection record count and save to SSIS variable
• MongoDB example – How to fetch MongoDB collection size and save to SSIS variable
• MongoDB example – How to get collection names and loop through them using ForEach Loop Task
• MongoDB example – How to save query output into Recordset variable and loop though records using ForEach Loop Task

Video Tutorial – Update/Delete/Write/Bulk Load data into MongoDB from SQL Server/ORACLE

In below video tutorial you will see how easy it is to load data into MongoDB from any Datasource such as SQL Server, Oracle, MySQL, Flatfile etc. You will also learn how to product nested JSON from multiple tables using JSON Generator Transform. You can also use JSON Source with Output as Document Mode to load raw JSON data.

Video Tutorial – Read data from MongoDB

In the following video tutorial you will learn how to consume data from MongoDB Collection using SSIS MongoDB Source Connector. You will notice that it uses SQL query syntax rather than MongoDB specific native query syntax (JSON query). This makes it super easy to query data without any learning curve. Data stored in MongoDB is in JSON document format but Data coming out from MongoDB Source Connector is Flat table structure (It de-normalize nested nodes). You can also Query inner Hierarchy using JSON Path expression (e.g. query Orders from Customer Document using $.Customer.Orders[*] ) .

SSIS Example – Loading data into MongoDB, Read from MongoDB, Upsert, Upsert, Delete Insert JSON Documents, Execute Shell Commands

MongoDB Insert – Bulk Loading data into MongoDB

Using MongoDB Destination Connector you can easily Insert JSON documents in Bulk. There are two input modes for inserting records into MongoDB.

Simple Mode – Loading data in simple mode (array not allowed)

In simple loading mode you can map source columns to target column in MongoDB destination. If you Insert data using this mode then you cant load complex documents with array.

Document Mode – Loading JSON documents into MongoDB

In this mode you can insert/update/delete documents by mapping inbuilt __DOCUMENT__ input column which appears on mapping tab (Target columns). When you perform Insert you have two options (Insert (i.e. Append) and Reload (Truncate + Insert). When you select Operation=Reload in that case Collection is first Truncated and then new records are loaded.

In JSON document load mode if you don’t supply _id column part of your input JSON then new _id is automatically generated by MongoDB for newly inserted document.

Loading JSON Documents into MongoDB Collection (Map Raw Document Column)

Loading JSON files into MongoDB

Another scenario could be loading JSON files into MongoDB. You can load JSON Files into MongoDB two ways.

• Use Flat File Source with just one column with DT_NTEXT datatype (use this method when you each JSON documents in one line and they are separated by new line as below screenshot)
• Use JSON Source Connector with Output as Document option checked. You can map __DOCUMENT__ source column to Target MongoDB collection as below. Advantage of this method is you extract JSON from array too and specify many other advanced options.
  

  SSIS Extract JSON Documents or Sub Documents from JSON file or JSON Array

Specifying LoadOptions

MongoDB Destination Connector supports following LoadOptions property (see Component Properties Tab on UI) which controls how target data gets modified. LoadOptions property is ignored for Reload and Insert operations.

• op : this option specify update operators (Valid for Update, Replace and Upsert operation (i.e. UpdateOrInsert) ).  See this link for more information on various Update operators available in MongoDB. Most common operators listed below. Refer official MongoDB help for more information.
  • op=none : Replace entire document
  • op=$set  : Update only those fields specified in mapping. If Specified mapping field not available in target then its added to target document.
  • op=$push : Adds new item to destination array. See following examples (How to insert single or multiple documents into MongoDB Array by supplying JSON into  __DOCUMENT__ column
    

    /* Insert single item into array - Construct input JSON as below */
    {YourArrayField : 111 }
    
    /* Insert multiple items into array - Construct document as below */
    {YourArrayField : { $each: [ 111, 222, 333 ] } }
    
    /* Insert multiple documents into array */
    {YourArrayField : { $each: [ {OrderID:1, Total:20.00}, {OrderID:2, Total:12.00} ] } }

• multi : This option controls if you want to Update or Delete multiple records for matching condition for each upstream input record.  For example if you joining by Country and City columns for update and target system finds 5 records then by default they all will be updated if multi=true is set in LoadOptions property. If you don’t want to update multiple records

MongoDB Update

MongoDB Destination Connector supports Batch Update and Batch Upsert Operations. There are few things to remember how Update operation works with MongoDB Destination Connector.

• To perform Update operation you have to specify Join Criteria for record lookup on target (see ColumnsForLookup property).
• Specify JOIN columns in ColumnsForLookup property. Use comma to specify multiple columns (e.g. CustomerID,OrderID). If you dont specify columns in this property then any columns mapped other than __DOCUMENT__ will be considered as JOIN column.
• With MongoDB destination you can only specify simple Update condition by mapping JOIN input columns. Behind the scene it will construct search query for update. For example if you Map 3 input columns (__DOCUMENT__ , Country, State) and you specify LoadOptions : op=$set;multi=true then its similar as below SQL Statement.
  

  UPDATE MyCollection
  SET [properties specified in __DOCUMENT__]
  WHERE Country=[input value] AND State=[input value]

• Join columns can be individually mapped or can be supplied as part of JSON document mapped to __DOCUMENT__ column.
• If __DOCUMENT__ column is not mapped then you must specify at least one column name in ColumnsForLookup property to identify search column.
• If __DOCUMENT__ column is mapped and column name is not specified in ColumnsForLookup property then you must map at least one more input column which can treated as JOIN column

Update specific fields  ($set operator)

To update specific fields in target MongoDB collection you can map JOIN column(s) and then map fields you want to update or map __DOCUMENT__ if you suppling JSON document with fields you want to update.

See below screenshot where its updating Customers record with new coordinates  based on City and Country information.

SSIS MongoDB Update Bulk – $set update operator – Update specific fields – Update single or multiple documents

Update single or multiple target records for single input row

If you want to update all matching records in target then set LoadOptions property to op=$set,multi=true
If you want to update only one record per matching criteria in target then set LoadOptions property to op=$set,multi=false

Update entire document (i.e. Overwrite/Replace documents)

If you like to replace entire target document then you can do two ways. Few things you have to change for MongoDB Destination.

Approach#1 

1. Select Operation=Replace (Rather than Update or Upsert)
2. Specify Join column in ColumnsForLookup
3. On the mapping tab Map __DOCUMENT__ (Assuming you have Raw JSON coming from Upstream)

Approach#2

Another approach is use $set operator in LoadOptions property then it will only update specified fields in mappings. But if you want to overwrite entire document (Replace document) then set LoadOptions property to op=none,multi=true or op=none,multi=false (If you want to update one record for matching condition)

Insert single item into MongoDB Array using $push operator

If you want to insert new items into array then set LoadOptions property to op=$push,multi=true or op=$push,multi=false (If you want to update one record for matching condition). You have to map __DOCUMENT__ column on target mappings tab to supply item(s) or document(s) you want to insert into target array.

/* Insert single number or string into MongoDB array - Construct input JSON as below */
{YourArrayField : 111 }

/* Insert single document into MongoDB array - Construct input JSON as below */
{YourArrayField : {OrderID:1, Total:20.00} }

Insert multiple items into MongoDB Array using $push operator along with $each

Inserting multiple items into array requires use of $each modifier as below. Use above instructions except input document (e.g. data passed to __DOCUMENT__ column) will look like below (see how $each used).

/* Insert multiple items into MongoDB array - Construct document as below */
{YourArrayField : { $each: [ 111, 222, 333 ] } }

/* Insert multiple documents into MongoDB array */
{YourArrayField : { $each: [ {OrderID:1, Total:20.00}, {OrderID:2, Total:12.00} ] } }

Other MongoDB Update Operators

So far we have seen $set and $push update operators. To learn more about other operators see this link

MongoDB Update Array Items ($pull, $push, $addToSet)

If you wish to update items found inside nested array of document then Check this full length article

MongoDB Upsert – Bulk Update or Insert JSON documents

If you wish to perform Bulk Upsert using MongoDB Destination connector then select Upsert Action from dropdown (Action=UpdateOrInsert). In Upset if document is not found for matching criteria then new document is inserted in MongoDB Collection. If document is found then Update operation occurs. To set specific fields use op=$set in LoadOptions   and if you wish to overwrite entire document then use $op=none in LoadOptions

SSIS MongoDB Upsert – Bulk Update or Insert JSON documents

MongoDB Delete – Bulk delete MongoDB JSON documents

To delete documents from collection you can set operation=Delete on Component properties page. For delete you can simply map _id column on target or just supply columns based on you want to search records you want to delete.

To delete maximum one record for each input row set LoadOptions property to op=none,multi=false or set op=none,multi=false if you want to delete more than one record for matching condition.

MongoDB Collection Reload – Remove All (Truncate collection)

To reload collection set operation to Reload on Component properties page. Reload operation first clears target collection and perform bulk Insert operation

Generate JSON for MongoDB data load

Now lets talk about different ways to produce simple or complex JSON using ZappySys components.  There are 3 ways you can produce JSON which can be loaded into MongoDB Collection. Following section will describe each approach and you can select which one best suited for you.

Generate JSON – Using Export JSON Task

Export JSON Task can export JSON to File or variable which you can consume via JSON Source Connector (Check Output as Document option) and supply to MongoDB Destination Connector (map to __DOCUMEN__).

Check this link for more info :  Export Json from sql server using ssis

Generate JSON – Using JSON Generator Transform

JSON generator transform is useful when you want to Produce JSON inside pipeline from single or multiple input.

Check this link for more info :  JSON generator Transform (Watch Video on this page)

JSON Generator Transform – Generate JSON documents and load into MongoDB

Generate JSON – Using Dummy Data Source (Fake JSON for Testing)

If you need to produce JSON for testing purpose then use Dummy Data Source (Select JsonDocuments Template from dropdown and click ok)

Query/Read data from MongoDB

Visit following links to learn more about reading data from MongoDB.

SSIS MongoDB Source Homepage
MongoDB SQL Query Reference
MongoDB data extract – from nested sub documents and array

Execute MongoDB Shell commands in SSIS (e.g. Create/Drop collection)

MongoDB SOurce and Destination Connectors are great for read/write data. But what if you want to issue ad-hoc MongoDB shell command such as create collection/drop collection ? No worry.. SSIS PowerPack v2.0 released new Task called SSIS MongoDB ExecuteSQL Task This task comes with many examples listed on the UI itself (Look for Examples dropdown menu on UI). You can use SSIS MongoDB ExecuteSQL Task to perform many operations such as

• Create/Drop MongoDB collections
• Get collection list as ADO.net recordset so you can loop through using ForEachLoop task
• Call serverside JavaScript
• Fetch data as ADO.net recordset and store in SSIS variable so you can loop using ForEachLoop task
• Drop database
• Call many shell commands

Download MongoDB Example Files used in this tutorial

To test this package you first need to download SSIS PowerPack
Once SSIS PowerPack installed you can download following SSIS sample project file(s).

Download sample files – SSIS 2008
Download sample files – SSIS 2012
Download sample files – SSIS 2014
Download sample files – SSIS 2016

Conclusion

MongoDB makes it super easy to store JSON documents and gives you whole new way of manipulating documents with ease of use. ZappySys SSIS Connectors and Tasks for MongoDB makes it possible to implement various data integration scenarios with MongoDB. Click here to Download SSIS PowerPack and try it your self.

Keywords:
Loading data into MongoDB | mongodb upsert array | mongodb upsert example | mongodb insert or update | mongo insert or update | mongo upsert | mongo insert JSON | mongo upsert JSON | update data into MongoDB

The post SSIS – Loading data into MongoDB (Upsert, Delete, Update) appeared first on ZappySys Blog.