Sometimes Data Flow source does not contain an ID column, or a number indicating a row number, which you need in further steps. To solve this, you may use a free ZappySys PowerPack component Template Transform. So let’s take a look at how to accomplish that.

Tip

Add Template Transform

Go to the Data Flow, drag and drop Template Transform from SSIS toolbox, and connect it right after the source, which does not provide ID/row number:

Configure Template Transform

Template Transform will provide the row number we want. So configure it like this: in the text area enter <<FUN_SEQUENCE>> and optionally you can give this column a name, e.g. RowNumber.

Configure Template Transform to get row numbers for each row in SSIS

By default Template Transform’s output type is DT_NTEXT. So if you wish to change the type to, for example, integer, use the Derived Column connector and CAST function to change it, e.g.:

(DT_I4)(DT_WSTR,100)RowNumber

Derived column added to convert DT_NTEXT column to an integer type.

Result

In the data viewer you would see that you have a column with a different row number in each row:

Results: How to add row numbers to rows in SSIS data flow

The post How to add row numbers in SSIS data flow appeared first on ZappySys Blog.

In this blog,  we are going to learn to how to compare records in SSIS using hash method. We will use FREE Template Transform component along with Sort Transform, Merge Join Transform and Conditional Split Transform. We will see how to produce hash value from multiple columns and later compare source\target hash values to detect changes\new rows.

Data Hashing

Here is sample design how to compare records using hash method

1. In Step 1, we need two data tables for record comparison from source and destination.Source Data:
   Target Data:
   
2. In Step 2, both the source and target data table records would be hash using the ZS Template Transform component as below. You can use FUN_MD5 to produce hash from multiple columns and. Please make sure that you change the Output Column name as below e.g. HashCrm or HashSQL
3. In Step 3, both the source and target data table need to be sorted by primary key using SSIS “Sort Transformation Editor” component as shown in above sample design screenshot.
4. In Step 4, both the source and target data table need to be merge using SSIS “Merge Join” component providing appropriate join type and key mapping
   as below
5. In Step 4, we adding the conditional split with appropriate condition based on your need e.g. we want to check similar records as below
6. From conditional split, we can feed the modified or same records into destination component based on our needs.
7. This is how we can compare the data for source and target tables for updated records and also we can set up different Merge Join and conditional split for desired set of output records set.

Detect Update/Insert

1. In previous section, we have chosen the updated records using hash. Here are going to see the Updated/New records using Left Outer Join as belowLeft Outer Join
   
2. Here is conditional split criteria for Update/New records as below

Conclusion

In this tutorial we learned how to compare the records in SSIS using hash method. If you want take a look it then download the ZappySys SSIS PowerPack here.

The post How to compare records in SSIS using hash appeared first on ZappySys Blog.

In our previous article, we saw how to get Office 365 data in Power BI. Now, let’s look at how to get Office 365 Mail Attachments using SSIS without any coding. Microsoft Graph API is a unified way to access many Microsoft services, including the Office 365 API. In this post, we will use the SSIS JSON/REST API Source Connector to retrieve the Outlook Messages list and its attachments.

Use Case of Microsoft Graph API

Here are some use cases for why you want to use the Microsoft Graph API

• Read/Write events from Outlook Calendar
• Send email  / Read emails
• Get the list of files from OneDrive
• Upload/ Download files to OneDrive
• Read/Write Excel Sheet (Use range or a specific range of a specific sheet)
• Read the SharePoint Document library
• Search content from the SharePoint document library
• Get AD users for your Organization (AD Accounts)
• Update OneNote
• Track changes to Users, Events, Calendar Items using the delta API

Step-By-Step to call the Microsoft Graph Rest API Using SSIS

Now, let’s look at examples to access your Outlook.com emails using SSIS. You can use the same techniques to call pretty much any Office 365 API.

Configure SSIS OAuth Connection for Graph REST API

The first step to access any Graph API is to configure an OAuth connection

1. Open Visual Studio
2. Open an existing SSIS Project or create a new one using File > New > Project > Choose “Integration Services Project” Type under Business Intelligence template category.
3. Create Package
4. Right-click inside the Connection Managers area and click New Connection…
5. From the connection type list, select the ZS-OAUTH connection type.
   

   Create a new SSIS OAuth API Connection Manager

6. On the OAuth Connection Manager, configure the following options
   1. Select “Custom” from the Provider dropdown
   2. Select OAuth2 from the OAuth version
   3. Enter your Application Id and Secret Key (i.e., App Password) obtained in the previous section from here.
   4. Enter the following URL in the Authorization URL
      

      https://login.microsoftonline.com/common/oauth2/v2.0/authorize

   5. Use the following URL in the token URL field
      

      https://login.microsoftonline.com/common/oauth2/v2.0/token

   6. Enter the following Graph API Scopes (each scope must be entered on a new line).
      

      user.read
      mail.read
      offline_access

      NOTE: Always include the offline_access scope, which returns a refresh_token, which allows you to renew the token without going through the login process again. For more information about which scopes are needed, always refer to the API help page (e.g., if you want to read mail using this API, then you will see it says you can use Mail.Read or Mail.ReadWrite scope).
   7. Go to the Advanced tab and enter the following URL in the Callback/Return URL (assuming the same URL used when you registered the App in the previous section)
      

      https://zappysys.com/oauth

   8. Now go back to the first tab and click Generate Token. When prompted, log in using your Personal Microsoft Account or Work Account (Office 365 or AD login)
   9. After logging in, you will see the Accept option. Just click it. If things go right, then you will see the Access Token and Refresh Token fields populated.
7. Once everything is configured, you can click Test Connection to verify the connection works.
   

   SSIS OAuth Connection Manager for Office 365 REST API Microsoft Graph API

Download the Messages Attachment Files from Outlook

Let’s start with an example. We use the SSIS JSON Source and Web API Destination components to make a call to the Microsoft Graph REST API.

1. First of all, drag and drop the Data Flow Task from the SSIS Toolbox and double-click it to edit.
   

   Drag and drop Data flow

2. From the SSIS toolbox, drag and drop the JSON Source onto the Data Flow Designer surface.
   

   SSIS JSON Source – Drag and Drop

3. Double-click the JSON Source, then enter the following URL to invoke the Messages API call with search options to retrieve emails with attachments.
   

   https://graph.microsoft.com/v1.0/me/messages?$select=sentDateTime,from,subject,hasAttachments&$search="hasAttachments:True"

   Set the filter to $.value[*] and click the preview.
   

   SSIS JSON Source – Get data from Microsoft Graph API (Office 365 API – Read Mail Example)

   To customize the URL with additional parameters, check this help link. It’s a standard OData Protocol, so you can leverage many common OData features (e.g., use $top parameters to set pageSize. Default is 10 for List Mail, but you can include more records per response by changing it, such as …/messages?$top=50).

4.  Now, let’s use Template Transform to set the URL using the message ID to make the Get attachment API call.

   https://graph.microsoft.com/v1.0/me/messages/<%id%>/attachments

   

   SSIS Template Transform sending attachment id
5. Now we need to use the Web API Destination to make the Get Attachment API call by message ID. Select the Input column for URL as that TemplateOutput.
   

   SSIS Web API Destination – Get data from Microsoft Graph API (Office 365 API – Get Attachment Example)

6. Now, the next step is to parse the JSON Response String of the get attachment api call. For that, we need to use a JSON Parser. Select the input as Web PAI Destination ResponseText and use the below sample JSON string to configure the metadata(Columns).
   

   {"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#users('support%40zappysys.com')/messages('AQMkADAwATMwMAItMDkyMC0zZTg4AC0wMAItMDAKAEYAAAP_JUy')/attachments"
   ,"value":[{"@odata.type":"#microsoft.graph.fileAttachment","@odata.mediaContentType":"text/plain"
   ,"id":"AQMkADAwATMwMAItMDkyMC0zZTg4AC0wMAItMDAKAEYAAAP_JUy_27M"
   ,"lastModifiedDateTime":"2020-02-21T07:36:16Z"
   ,"name":"Test.txt","contentType":"text/plain","size":870,"isInline":false
   ,"contentId":null,"contentLocation":null,"contentBytes":"VGVzdCBNZXNzYWdl"}]}

   

   JSON Parser Transform – using JSON example
7.  Now, in the JSON Parser, go to the Go-to columns Tab and set the contentBytes DataType to DT_IMAGE, and increase the other columns’ DT_WSTR lengths by 1000 and 1500. We set contentBytes to the DT_IMAGE  datatype, which is like varbinary(MAX).
   

   JSON Parser Transform Columns Tab fix the datatype

8. Furthermore, in the next step, we need to use an SSIS Derived Column to set the FilePath where we want to save the attachments, and then concatenate the local file path with the attachment name column.
   

   SSIS Derived Column: Add New Column

9. Finally, we need to use the Export Column Transform (Native) to export attachment bytes to a local file. It needs to know 2 upstream info Bytes and the target file path.
   

   Export Column configuration

10. Make sure to attach the export column to some destination (e.g., our ZS Trash Destination); the engine might remove it. At the same time, it optimizes the runtime. It may remove all transformations that don’t have a destination, such as deploying the package to SQL Server, or when you set optimize in Visual Studio.
11. That’s it, execute the package, and it will download all the email attachments.
    

    Downloaded Mail Attachments

Conclusion

After all, in this article, we learned how to make a Microsoft Graph API REST API call. We used the SSIS JSON / REST API Connector to extract data from Outlook Mail using OAuth. Made the Get attachment API call using the Web API Destination and parsed the JSON response using the JSON Parser. And using the Native SSIS Export Column, we saved the export attachment’s bytes to a local file. To explore many other scenarios not discussed in this article, download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • Help File: Web API Destination
  • Help File: JSON Parser Transform
  • Overview of the Microsoft Graph API

The post How to Get Office 365 Mail Attachments using SSIS appeared first on ZappySys Blog.

In our previous blog post we saw how to update / delete mongodb array item. Now let’s look at how to Upsert MongoDB Array Items (i.e. Insert Item if not found in Array else Update existing record). Upsert into nested MongoDB Array requires two step process, unfortunately there is no easy way to do in a single step. You may see in below screenshot that first we will do Update Step for Existing MongoDB Array Items and then we will insert missing Records in second step. So let’s get started.

MongoDB Array Upsert – Update / Insert using Custom JOIN condition (such as $ne )

By default SSIS Mongodb Destination performs Lookup using $eq condition (Match rows using Equal operator). However there will be a time when you like to lookup and update target rows using custom join criteria (e.g. Use Not Equal condition – $ne ). V2.7.6  and later introduced new property called 

EnableCustomLookupQuery

Here is the description how to use custom Join using this new property.

Enables use of custom lookup query for Update / Upsert or Delete Operation. 
By default JOIN condition for target record match is generated automatically based on columns you supply in ColumnsForLookup property. 
However in some cases you need to supply custom condition for lookup to perform complex operations, in such case enable this option. 
When you enable this option you must supply map __DOCUMENT__ input column. String you supply to this column should be in this format 

<command>
  <query>YOUR_LOOKUP_QUERY</query>
  <document>YOUR_JSON_DOC</document>
</command>. 

Lookup query in <query> tag can be either Mongo JSON format (e.g. { \"CustomerID\" : \"AAA\", \"Orders.OrderID\" : { \"$ne\" : \"1000\" } }) 
OR
you can use ZappySys SQL query (e.g. select * from mytable where CustomerID='AAA' and [Orders.$.OrderID] != '1000' )

Now lets look at step by step. In below example we have a table called CustomerTest. We will load it with 2 records (With No orders). Then Update array using custom lookup condition, and later step we will insert record in array using custom lookup condition.

SSIS MongoDB Array Upsert Example (Update / Insert Array Items based on custom lookup condition)

Update Array using Custom Lookup Condition

1. Lets create few sample rows in MongoDB Collection. You can use following command in SSIS MongoDB Execute SQL Task
   

   {
    scope: 'database',
    db: 'Northwind',
    command: 'eval',
    args: 
    {
    code: 'db.CustomerTest.insert( [ {CustomerID:"AAA", Orders:[]}, {CustomerID:"BBB", Orders:[]}, {CustomerID:"CCC", Orders:[{OrderID:"1004","OrderDate" : "2008-02-05","Qty" : "5"} ]} ] )' 
    
    } 
   }

2. Now drag Data flow.
3. Drag ZS CSV Source or any other source. For example purpose we will configure CSV Source using Direct Value option with following Sample data. In the first run of package it will update only orders for CCC customer because only that customer will have records after table creation. one row. But in 2nd run all rows updated.
   

   CustomerID,Company,OrderID,OrderDate,Qty,Address
   AAA,Anthony Inc,1000,2008-01-01,90,Po Box 111\4612
   AAA,Anthony Inc,1001,2010-02-01,91,Po Box 111 / 4612
   BBB,Bob Inc,1002,2008-02-01,92,Po Box 222 / 4612
   BBB,Bob Inc,1003,2010-01-01,93,Po Box 222 / 4612
   CCC,Cindy Inc,1004,2010-01-07,2,Po Box 555 / 2345

    
4. Click OK to save Source UI
5. Now drag ZS Template Transform to build XML document (Data + Query). Connect Source to Template transform. You can also use ZS XML Generator Transform but for simplicity we will use Template Transform.
6. Double click to open Template Transform and enter text as below. Notice how we have entered MongoDB Query and Data in two separate XML nodes. First node is custom query for lookup. Second node is Operation we like to perform (e.g. $set in our case to update existing data). You can use Insert Variable and then select Columns option to insert placeholders. Also in some fields we used JSONENCODE function to make sure we escape double quote and slash correctly.
   

   <command>
   	<query><![CDATA[
   {
   	"CustomerID": "<%CustomerID%>", 
   	"Orders.OrderID": { "$eq" : "<%OrderID%>"}
   }
   ]]></query>
   	<document><![CDATA[
   {
     $set :
      { "CompanyName":"<%Company,JSONENCODE%>", 
        "Address":"<%Address,JSONENCODE%>", 
        "Orders.$.OrderID": "<%OrderID%>", 
        "Orders.$.OrderDate": "<%OrderDate%>", 
        "Orders.$.Qty": "<%Qty%>" 
       } 
     }
   }
   ]]></document>
   </command>

   

   Using ZS Template Transform to specify custom JOIN condition for MongoDB Update (Array Items)

7. Click OK to save Template Transform
8. Drag ZS MongoDB Destination. Connect Template Transform to Destination
9. Double click MongoDB Destination to edit. Set Connection, Table name, Operation=Update, EnableCustomLookupQuery=True
   

   Configure MongoDB Destination for Custom Join Condition (EnableCustomLookupQuery Setting)

10. Goto Mappings Tab and attach TemplateOutput to __DOCUMENT__
    

    MongoDB Mappings – Loading Document for Array Update (Custom Join Criteria)

11. Click OK to Save

Insert into Array using Custom Lookup Condition

Now let’s look at slightly modified steps to Insert into Array for new records. In this example we will insert Source records if CustomerID found but OrderID is not found. (See Template Transform Step). We will use Not Equal Condition this time.

1. Now drag Data flow rename it to something like [Add records to array]
2. Drag ZS CSV Source or any other source. For example purpose we will configure CSV Source using Direct Value option with following Sample data
   

   CustomerID,Company,OrderID,OrderDate,Qty
   AAA,Anthony Inc,1000,2008-01-01,30
   AAA,Anthony Inc,1001,2010-02-01,4
   BBB,Bob Inc,1002,2008-02-01,30
   BBB,Bob Inc,1003,2010-01-01,4

    
3. Click OK to save Source UI
4. Now drag ZS Template Transform to build XML document (Data + Query). Connect Source to Template transform. You can also use ZS XML Generator Transform but for simplicity we will use Template Transform.
5. Double click to open Template Transform and enter text as below. Notice how we have entered MongoDB Query and Data in two separate XML nodes. First node is custom query for lookup. Second node is Operation we like to perform (e.g. $addToSet in our case to insert into array )
   

   <command>
   <query>
   {"CustomerID": "<%CustomerID%>", "Orders.OrderID":{ "$ne" : "<%OrderID%>"}}
   </query>
   <document>{
     $addToSet :
      { Orders : {"OrderID": "<%OrderID%>", "OrderDate": "<%OrderDate%>", "Qty": "<%Qty%>" } }    
    }
   </document>
   </command>

6. Click OK to save Template Transform
7. Drag ZS MongoDB Destination. Connect Template Transform to Destination
8. Double click MongoDB Destination to edit. Set Connection, Table name, Operation=Update, EnableCustomLookupQuery=True
9. Goto Mappings Tab and attach TemplateOutput to __DOCUMENT__
10. Click OK to Save

Download Sample SSIS Package (2012 format) – MongoDB_Upsert_ArrayItem

Conclusion

MongoDB integration can be challenging if you are new to NoSQL world. If you are using SSIS as your primary ETL tool then not to worry because SSIS PowerPack can give super power needed to complete your project on time with drag and drop high performance connectors.

Keywords: ssis mongodb upsert array item | ssis mongodb update array item | ssis mongodb update array elements | mongodb update array documents | MongoDB $set operator | MongoDB $addToSet operator | MongoDB update sub document items | MongoDB CRUD operations | MongoDB Bulk Updates | MongoDB bulk updates to array items

The post Update or Insert – Upsert MongoDB Array Items using SSIS appeared first on ZappySys Blog.

In this post we will show you how to convert varbinary to Base64 in SSIS. We will use FREE Tasks provided by ZappySys.

Preparing Sample Data

First let’s create a sample table with some Varbinary datatype. Run following command in SSMS to create a sample table with one sample row.

create table binarydata(
 data varbinary(max)
)
go
insert into binarydata(data )
values(0x5468697320697320612074657374) /*binary of "This is a test"*/

Method 1 – Using Template Transform

Let’s use Template Transform to convert Varbinary data to Base64 format. When you use columns with binary data in Template transform it automatically converts to Base64 by default.

1. Go to Data flow
2. Drag your OLEDB Source and configure it to outputs SQL Data with Varbinary column  (e.g. select data from dbo.binarydata).
3. Now drag ZS Template Transform from SSIS Toolbox
4. Connect OLEDB Source to Template Transform
5. Double click Template Transform  to configure. Now you can click Insert Placeholders > Columns > Select your Upstream column name (which has Varbinary data)
6. You will see Template text as  <%MyVarBinaryColumn%>
7. Click OK to Save Template Transform
8. Now connect Template transform to any Destination e.g. OLEDB Destination.  You can Map TemplateOutput column to target column. This will contain Base64 representation of Varbinary data. For example purpose we used ZS Trash destination (Check save to file)

Convert Varbinary to BASE64 using SSIS Template Transform

Save Base64 string to Variable

In previous step we saw how Template Transform can convert binary data (i.e. Varbinary datatype) to BASE64 string. Now let’s look at how to save this value to Variable using Set Variable Transform.

Using SSIS Set Variable Transform in Data flow (Capture binary data as base64 example)

Method 2 – Using Trash Destination

Another way to dump Binary data to CSV file is use ZS Trans Destination. See previous screenshot. If you map OLEDB Source directly to Trash destination without Template Transform then you will see something like below.

data
VGhpcyBpcyBhIHRlc3Q=

The post How to convert Varbinary to Base64 in SSIS appeared first on ZappySys Blog.

In this post we will lean SSIS Magento data read / write operations. Magento is a very popular eCommerce platform and they offer JSON based REST API and XML based SOAP API. You can use either API based on your need to automate common integration needs.

We recommend using REST API (JSON API) if possible because they are simpler and faster. We will focus only on JSON API in this article but we do have blog post on how to call SOAP API.

About Calling Magento REST API

If you are new to Magento API we highly recommend to get familiar with Magento REST API here. Magento REST API  offers two ways to authenticate.

1. Token based Authentication
2. OAuth Based Authentication (OAuth 1.0a)

In this article we will look at only Token based approach. In token based approach we have to get new token using admin / customer account information.

We will cover later how to access admin or customer level API later in this article (See HTTP connection configuration).

Configure Connection / Call Magento REST API in SSIS

Now lets look at very simple example how to use Token based method to call Magento REST API in SSIS. We will first configure HTTP connection and then call REST API.

Let’s get started.

1. First, open SSIS Package after installing SSIS PowerPack
2. Drag ZS REST API Task from SSIS Toolbox on Control Flow Designer. Let’s rename task to Get Magento Categories
3. Enter REST API URL you like to call. For example we can use below URL. Replace **your-magento-host** with your own name.
   

   http://**your-magento-host**/index.php/rest/V1/categories

4. Select Url from Connection mode
5. From Connection Dropdown select new ZS-HTTP connection
   

   Create new connection for SSIS REST API Task

6. Configure General Tab of HTTP Connection like below
   URL can be anything for now because we will use override option in next step (On REST API Task).
7. Select Credentials Type as Dynamic Token.
8. Enter Userid / password
   

   Configure Magento REST API Connect for Token based Authentication

9. Now goto Dynamic Token Tab (Or click Configure Link) and configure like below.
10. Enter Token URL for API Login URL field.
    NOTE: If you are a customer and want to access your own account (e.g. view your orders) then replace /admin with /customer  in URL)
    Replace **my-host** with your host name.
    

    http://***my-host***/index.php/rest/V1/integration/admin/token

11. Change Method to POST.
12. Enter Body as below (Placeholders are automatically replaced when /token endpoint is called.
    

    {"username":"[$userid$]", "password":"[$password$]"}

13. Select Content Type as Application/Json
    

    Configure Magento Token Request

14. Now click Dynamic Token – Response Tab and configure like below.
    Select Regex and enter Expression as below. This will remove double quotes around token in response.
    

    "(.*)"{{0,1}}

    

    Extract Token from response – Remove double quotes around value using Regular Expression

15. Click OK to save connection
16. Now on REST API Task check Use Direct URL Option
17. Click Test Request to confirm its working
    

    Call Magento REST API using SSIS REST API Task

Read data from Magento and Load into SQL Server

Now lets look at how to read Magento data and load into SQL Server. Assume that you like to read all products.

Configure JSON Source (Read Magento REST API)

1. Drag Data flow and double click to edit:
   

   Dragging and dropping Data Flow Task into Control Flow

2. Drag ZS JSON Source on data flow and double click to edit
3. Configure JSON Source like below if you like to get product list for example
   1. Enter URL as below (change http to https if your host support secure channel).
      

      http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=1000
      --- to extract only specific fields do below way --
      http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=1000&fields=items[sku,name]

   2. Check Use Credentials. Use same HTTP connection we created in previous section
   3. In the Filter enter $.items[*] or click Select Filter to Browse and select items node.
   4. Click Preview to confirm
4. Here is how your setup will look like after configuration.
   

   SSIS JSON Source Configuration – Read data from Magento REST API

5. If you expecting more than 1000 rows then you have to set up pagination. Now go to pagination tab and configure as below (Below options are only available in v2.9.1 or higher). If you have older version then check next section for workaround.
   1. Pagination Mode =URL Parameter Mode
   2. Page Num Indicator = searchCriteria[currentPage]
   3. Max Rows Expression = $.total_count
   4. Page Data Expression = $.items[*]
6. Here is how your Magento REST API Pagination Setup will look like.
   

   REST API Pagination using Max Row count Mode

7. Click OK to save JSON Source. In next section we will see how to load data into Target like SQL Server.

Configure OLEDB Destination (Load into SQL Server)

1. Drag OLEDB Destination.
2. Select / Create Connection
3. Select or create NEW target Table (Click NEW button)
4. Click Mapping Tab to map source columns to target table
5. Click OK to save
6. Execute Package

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Configure Magento Pagination (For Old Version)

In previous section we configured pagination using newer version (v2.9.1 or higher). But what if you have older version and you dont see that feature. Use below steps for workaround.

Now let’s look at another common scenario when you real lots of data. Let’s say you have 5000 products but API call we saw earlier returns max 1000 rows. In that case we have to keep reading next page until all rows are returned. ZappySys provides many settings to configure various REST APIs out there, unfortunately there is no standard around pagination methods. Since Magento API has some bug in API pagination we have to do following hack. Magento keep sending same data of last page if you try to access page which doesn’t exists. Here is how to overcome that bug.

Configure REST API task to get total row count

First step to configure magento pagination is get total rows returned in our request which you like to paginate (in our case /products).

1. Go to Control Flow designer
2. Drag ZS REST API Task from SSIS Toolbox
3. Let’s rename task to Get Magento Product Count
4. Double click it and configure like below (Change HTTP to HTTPS if needed)
   

   http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=10

   
5. Go to Response Tab and configure like below. Enter expression as $.total_count
6. Click OK to save

Define Expression for JSON Source – MaxRows Property

Now next step is to define expression on MaxRows property of JSON Rows so we stop once all rows consumed from Paginated response. If you don’t do this it will keep paginating forever 🙁

1. On Control Flow designer select Data Flow which contains JSON Source for Magento
2. Right click and go to Properties of that data flow
3. Find Expression and click Button to add new expression
4. On Expression dialogbox select [Your JSON Source].[MaxRows] and for expression enter variable name which holds total count (e.g. @[User::MaxRows] )
5. Click OK to Save

Configure JSON Source for Magento Pagination

Now last thing we have to do is go to JSON Source and configure few things for pagination.

1. Go to data flow and double click JSON Source
2. Click on Pagination tab and select Pagination by URL Parameter Mode
3. Enter Page name as below
   

   searchCriteria[currentPage]

4. Click OK to save UI
5. Now run entire page to test (NOTE: Do not execute just data flow because if you do that way, it will not extract total row count)
   

Write data to Magento using SSIS Web API destination

Now let’s look at example to load data into Magento. In below example we will create few sample products.

1. Create new data flow and go to data flow designer
2. Drag and drop ZS CSV Source from SSIS toolbox. You can use any Data source but for simple demo we will use it.
3. Double click CSV source and select Direct Value mode from dropdown. Enter following sample data (Lets create 12 sample products.)
   

   SKU,ProductName
   PROD-1,Product 1
   PROD-2,Product 2
   PROD-3,Product 3
   PROD-4,Product 4
   PROD-5,Product 5
   PROD-6,Product 6
   PROD-7,Product 7
   PROD-8,Product 8
   PROD-9,Product 9
   PROD-10,Product 10
   PROD-11,Product 11
   PROD-12,Product 12

4. Now click OK to save UI
5. Now drag ZS Template Transform from toolbox.
6. Connect CSV Source to Template Transform
7. Enter following sample text to build request for new product. Refer to Magento API help file / tutorial to learn more.
   

   {
     "product": {
       "sku": "<%SKU%>",
       "name": "<%ProductName%>",
       "attribute_set_id": 4,
       "price": 25,
       "status": 1,
       "visibility": 1
      }
   }

8. Notice that when you click Insert Variable > Columns > select desired upstream column placeholder. You can also encode special characters in your data (e.g. new lines) using function like <%MyColumn,JSONENCODE%>
9. Click OK and save UI
10. Now drag ZS Web API Destination from SSIS Toolbox. Configure like below.Enter URL same as before (used to read product), change Method to POST, change content type to Application/JSON.
11. Now run entire flow (You can watch your API requests in Fiddler for debugging)

Debug Magento Web Requests using Fiddler

Click on below article to learn how to debug web requests

How to use Fiddler to analyze HTTP Web Requests

Magento Integration in Other Apps (e.g. Power BI / Informatica / SQL Server)

You can use techniques listed on this page with ODBC Drivers too for integration in other tools like Power BI, SQL Server code, Informatica etc.

Conclusion

In this post we saw how easy it is to perform Magento Integration in SSIS without doing any programming. In few steps you can read or write data in Magento using SSIS PowerPack REST API components. Download FREE Trial and explore many other scenarios.

The post SSIS Magento data Read / Write using REST API Call appeared first on ZappySys Blog.

We will look at step-by-step instructions to read Zoho CRM data (e.g. Leads, Accounts, Contacts) and load into SQL Server Database Table. In the next section, we will look at Write scenario too.

Now let’s look at step-by-step approach to call Zoho API  using SSIS.

Authentication Using Zoho V2 REST API (OAuth 2.0)

Zoho recently released v2 API. It uses OAuth 2.0 instead of static Token based (described in the next section). OAuth approach is more secure an recommended. If you must use static token-method then check next section. Zoho may discontinue Token based method in future.

Register Zoho OAuth App and get ClientID/ Secret (*** MUST READ ***)

Check this article to register your OAuth App and obtain Client ID  /Client Secret

Create Zoho OAuth Connection in SSIS (for API call)

Once you create Zoho OAuth App now its time to call some Zoho API. We will use ZappySys REST API Task to make simple API call. In the next section we will see how to load data in SQL Server.

1. Open SSIS Package and go to SSIS Designer (Control Flow)
2. Drag ZS REST API Task from Control Flow SSIS Toolbox as below
   

   Drag and drop SSIS REST API Task from SSIS Toolbox

3. Double click the task to configure.
4. Change URL Access mode to URL from Connection
5. Enter API URL you like to call. For example you can enter below
   

   https://www.zohoapis.com/crm/v2/settings/modules

6. Select New ZS-OAUTH connection from Connection drop down
   

   Call Zoho REST API using OAuth 2.0

7. Configure your OAuth connection General Tab as below
   1. OAuth Provider: Custom
   2. OAuth Version: OAuth2
   3. Client Id , Client Secret
   4. Enter One Scope Per Line (For more info on scope read here)
      

      ZohoCRM.modules.READ
      ZohoCRM.settings.READ

   5. Auth URL : https://accounts.zoho.com/oauth/v2/auth
   6. Token URL:  https://accounts.zoho.com/oauth/v2/token
8. Configure Advanced tab as below
   1. Enter Callback URL : https://zappysys.com/oauth
   2. Enter Extra Attribute for /Auth : access_type=offline&prompt=consent
      This is needed to obtain refresh_token (if you ommit this then only access_token returned which cannot be renewed once expired after 1hr)

      

9. Now go back to general tab and Click Generate Token.
   

   Generate OAuth Token for Zoho CRM REST API Access

10. Click OK to save the connection UI

Call Zoho API using OAuth Connection

1. On the REST API Task click Test Request / Response
   

   SSIS REST API Task – Call Zoho REST API in SSIS (OAuth 2.0)

Read data from Zoho Table in SSIS JSON Source

In this section we will learn how to read data from Zoho using SSIS JSON Source. We will use OAuth connection created in earlier section.

Now lets look at step by step example of reading Zoho CRM data using SSIS (e.g. read Leads, Accounts, Contacts, Invoice etc). Zoho provides JSON API for read action so we will Use SSIS JSON Source for this purpose. Full sample package is attached at the end of this article but see below steps for high level configuration.

Configure JSON Source

So below steps shows example of reading from Zoho Lead modules using v2 records API Call.

1. Open SSIS Package
2. Drag data flow task from Control flow SSIS Toolbox
   

   Drag data flow from SSIS Toolbox

3. Go to data flow and drag ZS JSON Source from SSIS Toolbox
   

   Drag and drop SSIS JSON Source (REST API or File)

4. Double click JSON Source to configure. Enter URL as below. You can supply few more parameters. Click Here to get full list of parameters and usage. In below example we are reading records from Leads Module but you can replace Leads to something else (e.g. Accounts). Refer to above help link for available Modules.
   

   https://www.zohoapis.com/crm/v2/Leads?per_page=200

5. Check Use Credentials. Select OAuth connection manager we created in the previous section.
6. Under filter options tab, enter Array Filter as  

   $.data[*]

7. Now click Preview to see your data
   

   Read data from Zoho API v2 using SSIS – OAuth (Use Last Modified Date for Incremental Extract)

8. Click OK to save UI. In the next section we will see how to setup pagination and incremental extract of Zoho records. After that we will see how to load data into SQL Server Table.

Zoho API Pagination (Read all rows)

Just like most API zoho also uses pagination. See below for how to configure pagination in JSON Source for Zoho API.

1. Double click JSON Source
2. Go to the Pagination Tab and select URL Parameter Mode
3. For Page Num Indicator Enter

   page

4. For Last Page Detection select

   Detect last page based on status code

5. In the Status code field
   enter 304 (if you added If-Modified-Since header for incremental extract – See next section)
   enter 204 (if you do not add If-Modified-Since)

Once you configure this way now your API will return all records until last page is detected. For more information on pagination check this article

Paginate Zoho API records

Incremental Extract based on Last Modified Date/Time field

Each module in Zoho has Last Modified By and LastModified Date/Time fields. You can filter extracted data for incremental extract by supplying

If-Modified-Since

1. On the JOSN Source, click Raw Edit above HTTP Headers Grid.  We can define Header to extract data after last modified date. Skip this step if you like to extract all records.
   For example to read Lead records which are modified after 12/31/2019 you can enter header like below. Date/Time is UTC. When record is created Modified date is set as creation date too so this will include created records too.
   

   If-Modified-Since: 2019-12-31T00:00:00

   If you have local time saved in a variable and you like to use SSIS variable then you can use below way. In this example we converted Local time to UTC.
   

   If-Modified-Since: <<{{User::varExtractDate,yyyy-MM-ddTHH:mm:ss}},FUN_TO_UTC_DATE>>

Loading Zoho data into SQL Server Table / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Write data to Zoho CRM using OAuth 2.0

If you like to load data from any source (i.e. SQL Server) to Zoho CRM then you can use Records API (POST). Use combination of the following Transforms  / Components. Later in this article we have some more examples (Older API) which may give you one more ways to load data into Zoho (Use of Template Transform).

Basically you have to build POST Body using JSON Generator Transform or Template Transform and then use that data in Web API Destination to call POST API request to push data to Zoho.

	Web API Destination
	JSON Generator Transform
	JSON Parser Transform

Create JSON for POST request (Single dataset pattern)

Insert data to Zoho Table using SSIS (Lead, Account Modules) – Bulk Load using POST API call

Authentication Using Zoho Tokens

If you don’t want to use a newer version of Zoho API (v2) then you can use Token based Authentication method.

In Zoho Portal perform following steps to obtain API token

1. Click on Setup (Toolbox icon at the top-right corner)
2. Search for “API” > Select CRM API  or find under Developer space > APIs
3. Click on Gear icon found above Usage Panel (small icon so find carefully)
4. Click on Authentication Token generation
5. Specify the APP Name and Click Generate
6. This will direct to the new page which may contain text like below. Just copy AUTHTOKEN part (i.e. 8a08xxxxxxxxxx13630d )
   

   #
   #Mon May 15 12:21:42 PDT 2017
   AUTHTOKEN=8a083f76xxxxxxxxxxxxxxxxxxxdxxxxxxxx13630d
   RESULT=TRUE

7. Now use the newly created auth token to your API call.

How to get Zoho CRM API Token (For SSIS or other app integration)

Read data from Zoho using SSIS (Export Zoho CRM data to SQL Server Table)

Now lets look at step by step example of reading Zoho CRM data using SSIS (e.g. read Leads, Accounts, Contacts, Invoice etc). Zoho provides JSON API for read action so we will Use SSIS JSON Source for this purpose. Full sample package is attached at the end of this article but see below steps for high level configuration.

So below steps shows example of reading Zoho Lead modules using getRecords API Call.

1. Goto SSIS > Variable and create a new string variable called token. Enter your Zoho API token as value (see previous section to get API token)
2. Drag data flow task from Control flow SSIS Toolbox
3. Go to data flow and drag ZS JSON Source from SSIS Toolbox
4. Double click JSON Source to configure as below
   1. Enter URL as below. Note that we have use Variable for Token to make URL dynamic.To read from different module change URL (e.g. rather than Leads in URL use Accounts).  Click Here to get full list of Modules and Fields
      

      https://crm.zoho.com/crm/private/json/Leads/getRecords?newFormat=1&authtoken={{User::token}}&scope=crmapi&fromIndex=0&toIndex=200

      You can also use lastModifiedTime parameter along leads modified after certain date/time (this time is Local time and not the UTC). To extract leads modified after certain date (Create a datetime variable called varExtractDate) and use it as below in the URL. You can use any valid format specifiers for date time
      

      https://crm.zoho.com/crm/private/json/Leads/getRecords?newFormat=1&authtoken={{User::token}}&scope=crmapi&fromIndex=0&toIndex=200&lastModifiedTime={{User::varExtractDate,yyyy-MM-dd HH:mm:ss}}

   2. Click on the Select Filter and select node with Array icon as below. Each Module may have different structure for example for Leads module Filter may look like $.response.result.Leads.row[*]
      

      Configure SSIS JSON Source – Read Zoho CRM Leads (getRecords API call with lastModifiedDate Example)

   3. Now go to 2D Array Tab and configure like below
      1. Select Transform Type to Key/Value Pivot Mode.
      2. Select or type Column Filter as $.FL[*].val
      3. Select or type Row Value Filter as $.FL[*].content
   4. Once you configure 2D Array Transform click Preview to see sample data
   5. .
   6. Click OK to save JSON Source and attach it to target such as SQL Server Destination like below.
   7. Execute Package to test.
      

      SSIS Example : Loading Zoho CRM data to SQL Server Table (JSON API Source)

Write data to Zoho using SSIS (Import SQL Server Table to Zoho CRM)

Now you know how to read data from Zoho CRM using JSON API Source next step is to load data to Zoho CRM. For writing data we will use following three components

	Web API Destination
	Template Transform
	XML Parser Transform

For this example we will use hard coded data in CSV Source (Sample Leads ) but in real world you will have data coming from some RDBMS such as SQL Server.

1. Drag Data flow Task and double click to go to Data flow designer.
2. Drag ZS CSV Source from the SSIS Toolbox. Change Access mode to Direct Value. Enter following sample data for demo.
   

   Company,First Name,Last Name,Email,Lead Status,Lead Source,Phone
   Microsoft,TestMaawia,TestBrito,Maawia@outlook.com,Lunk Lead,Web Download,111-111-3333
   Microsoft,Testraja,Testvedaiyan,raja@outlook.com,Lunk Lead,Web Download,111-222-3333
   Yahoo,TestJayder,Testelnageib,Jayder@outlook.com,Lunk Lead,Web Download,111-333-3333
   Yahoo,TestStephen,TestTjebane,Stephen@outlook.com,Lunk Lead,Web Download,111-444-3333
   Google,TestHarold,TestBatista,Harold@gmail.com,Lunk Lead,Web Download,111-555-3333
   Google,TestVennon,TestMills,Vennon@gmail.com,Lunk Lead,Web Download,111-666-3333
   Google,TestMichael,TestTremblay,Michael@gmail.com,Lunk Lead,Web Download,111-777-3333

3. Now drag ZS Template Transform. Connect it with previous step.
4. Double click Template Transform to configure it with following sample Text (This transform was introduced in v2.6.6. If you dont see it then update your SSIS PowerPack to latest version). Use of XMLENC is optional for Non string columns or Alphanumeric values but if you not sure about column type then just add it anyways.
   

   <Leads>
    <row no="1">
     <FL val="Lead Status"><%Lead Status,XMLENC%></FL>
     <FL val="Company"><%Company,XMLENC%></FL>
     <FL val="First Name"><%First Name,XMLENC%></FL>
     <FL val="Last Name"><%Last Name,XMLENC%></FL>
     <FL val="Email"><%Email,XMLENC%></FL>
     <FL val="Phone"><%Phone,XMLENC%></FL>
     <FL val="Lead Source">Web Download</FL>
    </row>
   </Leads>

   
5. Now drag another ZS Template Transform and connect it with previous step.
6. Double click Template Transform to configure. Enter below text. This is final text we will use as Request Body in the Web API Destination.
   

   xmlData=<%TemplateOutput,URLENC%>

   
7. Drag Microsoft Derived Column Transform. Connect with previous step and double click it to configure.
8. Select Replace ‘TemplateOutput’ action. Type new column name in the first column e.g. TemplateOutput_Encoded
9. Now drag ZS Web API Destination, Connect with previous step and double click to configure as below.
   1. Select New HTTP Connection. Enter some valid URL e.g. https://zoho.com  (This will be ignored anyways because we will use Dynamic URL on main UI)
   2. Enter URL as below (see how we again added API token as dynamic variable) . If you inserting to different Module (e.g. Accounts rather than Leads then change accordingly).   Click Here to get full list of Modules and Fields
      

      https://crm.zoho.com/crm/private/xml/Leads/insertRecords?newFormat=1&authtoken={{User::token}}&scope=crmapi&duplicateCheck

   3. Select Input Body column as TemplateOutput_Encoded
   4. Enter Sample Body as below if you like to use Test feature else ignore this step. Click Test to see its working. Capture Response XML because you may need in the next step to feed as sample to XML Parser Transform (again optional step).
      

      <Leads>
       <row no="1">
        <FL val="Lead Status">Junk Lead</FL>
        <FL val="Company">ABCD Inc</FL>
        <FL val="First Name">Bob</FL>
        <FL val="Last Name">Smith</FL>
        <FL val="Email">bob@abc.com</FL>
        <FL val="Phone">111-222-3333</FL>
        <FL val="Lead Source">Web Download</FL>
       </row>
      </Leads>

      
10. Now once API destination is set we can optionally configure below steps if you want to store status and ID of newly created records. If you don’t care for that then skip below steps.
11. Drag ZS XML Parser Transform, Connect it Web API Destination (Response Output – Blue Arrow). Double click XML Parser to configure as below
    1. Select Input XMl Column as ResponseText
    2. Enter Sample Text as below
       

       <?xml version="1.0" encoding="UTF-8" ?>
       <response uri="/crm/private/xml/Leads/insertRecords">
       	<result>
       		<message>Record(s) added successfully</message>
       		<recorddetail>
       			<FL val="Id">1558554000015597002</FL>
       			<FL val="Created Time">2018-02-13 16:14:10</FL>
       			<FL val="Modified Time">2018-02-13 16:14:10</FL>
       			<FL val="Created By">
       				<![CDATA[ZappySys Support]]>
       			</FL>
       			<FL val="Modified By">
       				<![CDATA[ZappySys Support]]>
       			</FL>
       		</recorddetail>
       	</result>
       </response>

        
    3. Enter Filter as below
       

       $.response.result.recorddetail[*]

    4. 
    5. Now Go to 2D Array Transform Tab. Configure as below
    6. Select Transform Mode to Key/Value
    7. Column name filter as  

       $.FL[:3].@val

    8. Row value filter as  

       $.FL[:3].#text

       
    9. Now click OK to close.
12. Drag ZS Trash Destination or some other destination (e.g. SQL Server) and connect Red arrow coming from Web API destination to this component to capture any errors.
13. Thats it now you ready to run your SSIS sample package which Inserts Leads to Zoho CRM. See below for full package.

SSIS Example: Import data to Zoho CRM using SSIS Web API Destination (Loading Leads, Accounts, Contacts)

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Sample SSIS Package Download

Click here to download sample package (SSIS 2012 or Higher).

Conclusion

Zoho CRM API provides great way to automate data read/write operations. However to call Zoho API  you have to use SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides great way to integrate any Zoho API call via simple drag and drop approach without coding. Try  SSIS PowerPack for free and call virtually any REST API in few clicks.

The post Read / Write Zoho CRM data using SSIS REST API Call appeared first on ZappySys Blog.

In this post you will learn how to Read / write Smartsheet data using SSIS (Drag and drop approach without any coding).  We will use SSIS JSON/ REST API Source to extract data from Smartsheet API and use SSIS Web API Destination to write data to Smartsheet. This approach is similar to our previous blog where we described how to read and write Google sheet using SSIS

Now let’s look at step by step approach to call Smartsheet REST API.

Create Token to access Smartsheet REST API

First step to access smartsheet API in SSIS is create API Token. We will use this API token later on to read data from Smartsheet Sheet.

1. Login to your Smartsheet account
2. Click on Account > Apps & Integrations > Click API Access > Click Generate Token
   

   Create Smartsheet API Token for REST API Access (Step 1)

   

   Generate new Smartsheet token (Step 2)

3. Once Token is generated copy that for later use (Make sure you treat this token as password and do not share with unauthorized user.

Getting started with Smartsheet API

In this article we will use few out of many API provided by Smartsheet. To get started with Smartsheet API concepts click here. To read full details about each Smartsheet API click here.

Step-By-Step – Import/Export Smartsheet data in SSIS

Now lets see how to read and write smartsheet data using SSIS.

Read data from Smartsheet using SSIS JSON / REST API Source (Load Smartsheet to SQL Server)

To read data from smartsheet you need to perform following steps.

Find Sheet ID for Smartsheet API access

1. First you need to know how to find Sheet ID. Perform below steps to find Sheet ID.
2. Goto your portal and open sheet you like to read. Right click on the Sheet Tab > Click Properties and Copy Sheet ID
   

   Find Sheet ID for Smartsheet API call

Once you know Sheet ID. Now lets create SSIS package to extract data from Smartsheet and load into SQL Server.

1. Create new SSIS Package or Open existing one.
2. From control flow SSIS toolbox drag Data Flow task
3. Double click Data flow
4. From SSIS Toolbox drag and drop ZS JSON Source (for REST API / File)
5. Double click JSON Source and configure following properties
   1. Enter Shartsheet API Access URL as below (Change last part with your own sheet id)
      

      https://api.smartsheet.com/2.0/sheets/YOUR-SHEET-ID-GOES-HERE

   2. Check use credentials and then create a new HTTP connection. Select Static Token / API Key and enter enter your token we got in previous section
      
   3. Click on Select Filter and pick Rows node or just type below text in the Filter textbox
      

      $.rows[*]

      
   4. Now goto 2D Array Transformation Tab and set following settings
      1. Set Transformation Type to Complex 2-dimensional array
      2. Column Name Filter to $.columns[*].title
      3. Row Value Filter to $.cells[*].value
         
         

         SSIS Smartsheet API Configuration – Complex Array transformation (Read nested array)

   5. Once everything is configured Click Preview. Click OK to save.
   6. Drag OLEDB Destination from SSIS Toolbox.
   7. Connect JSON SOurce to OLEDB Destination
   8. Double click OLEDB Destination and configure connection (e.g. Select SQL Server connection) and select Target Table or click New to Create new Table based on Source columns.
   9. Click on Mappings to map source columns to target
   10. Click OK to save
   11. Execute Package
       

       SSIS Example : Loading Smartsheet data into SQL Server (REST API Call)

Read Smartsheet data with Pagination option

Smartsheet API support options to read large amount of data in small chunks by using pagination options. ZappySys offers many ways to paginate for various API.

Disable Smartsheet Pagination

Many API endpoints in Smartsheet by default send you paginated response. Pagination requires some extra configuration so try to disable it if possible by using includeAll=true option in your URL as below. If you cannot includeAll for some reason then perform steps listed in next section.

https://api.smartsheet.com/2.0/sheets?includeAll=true

Smartsheet Pagination in SSIS

To paginate Smartsheet response in SSIS JSON Source perform the following steps. Below steps assume that you trying to get all sheets by calling

https://api.smartsheet.com/2.0/sheets

1. Go to Control Flow designer right click > Select Variables
2. Create new string datatype variable called name it TotalPages. Set default value to 0
3. Now drag ZS REST API Task from SSIS Toolbox
4. Enter Following URL (we don’t need many rows to so reduced default page size to just 10)
   

   https://api.smartsheet.com/2.0/sheets?pageSize=10

5. Now click on Raw Edit (Above Headers grid)
6. Enter following header (Replace Token with your own token we obtained in the previous section)
   

   Authorization: Bearer YOUR-TOKEN-GOES-HERE

7. Go to Response Settings Tab
8. Select Content Type Json and enter filter as $.totalPages
9. Check Save response to Variable. Select Variable we created in Previous step (Only string datatype variables are visible)
   

   Save Total Page Count to SSIS variable

10. Click Test Request/Response to confirm it works
11. Click OK to save
12. Now let’s use this variable to configure MaxPageNumber setting in JSON Source. Using following way you can make any Data flow component property dynamic.
    

    Define expression on SSIS Data flow Component Property (Dynamic Value)

13. Now connect your REST API Task to Data flow
14. On the JSON Source go to Pagination Tab and enter following settings.
    

    Smartsheet API Pagination settings – SSIS JSON / REST Source

15. That’s it. Now if you run it you will see all records will be pulled.

Write data to Smartsheet using SSIS Web API Destination (SQL Server to Smartsheet Import)

Now lets see how to write some data to Smartsheet Sheet. We will read records from SQL Server and write to SmartSheet using ZS Web API Destination

For inserting multiple rows we can call this Smartsheet API Endpoint

Basically High level steps are.

1. Find out Smartsheet Column ID to use along with API call
2. Read Records from SQL Server using OLEDB Source
3. Construct desired JSON for Add New Row API Call
4. Send JSON to Smartsheet Add Row API  (Configure Bulk Options )

Find out Smartsheet Column ID

Very first step you have to perform is get Column ID which correspond to certain title. You can use following URL endpoint to get list of all columns (Change your Sheet ID). Make sure you append includeAll=true else it may only fetch 100 columns. You can save Column result to some database table to File to copy ID. Preview Grid also allows to copy cell.

https://api.smartsheet.com/2.0/sheets/YOUR-SHEET-ID/columns?includeAll=true

Get Smartsheet Column ID using API call in SSIS JSON Source

Configure SQL Server Source (OLEDB Source)

Once you have column ID you can drag OLEDB Source from SSIS Toolbox and configure to read from SQL Server. Below is sample source query to extract data from SQL Server.

Select OrderID,CustomerID,EmployeeID,OrderDate from Orders

Configure SSIS Template Transform

Once SQL Source is configure we are ready for next step. SSIS PowerPack v2.6.4 and later introduced new SSIS transform called Template Transform. This transform allows you to produce desired JSON /XML from single input record by using Column name as placeholder anywhere in your text. This is much simpler method than constructing JSON using JSON Generator Transform

Here is the sample JSON we want to produce to insert into Sheet which contains 4 columns OrderID, CustomerID, EmployeeID and OrderDate.

{
  "toTop": true,
  "cells": [
    {
      "columnId": 8146714579232644,
      "value": 10548
    },
    {
      "columnId": 828365184755588,
      "value": "TOMSP"
    },
    {
      "columnId": 5331964812126084,
      "value": 3
    },
    {
      "columnId": 3080164998440836,
      "value": "1997-05-26T00:00:00"
    }
  ]
}

Now lets configure SSIS Template Transform to create desired JSON Request for API call.

1. Drag ZS Template Transform from the SSIS toolbox
2. Connect OLEDB Source to ZS Template Transform
3. Double click Template Transform. Paste above sample request in the Template Text
4. Now highlight sample value you wish to replace from upstream by some input column.
5. Click on Insert Placeholders > Columns >  Your Column
6. Perform above steps for each value you wish to replace. You can also add use placeholder functions such as JSONENC after column name (useful if you expecting new line or special characters such as tab or double quotes in your input text ). Date formatting (e.g. yyyy-MM-dd ) also allowed. See below example after placeholders added.
   

   {
     "toTop": true,
     "cells": [
       {
         "columnId": 8146714579232644,
         "value": <%OrderID%>
       },
       {
         "columnId": 828365184755588,
         "value": "<%CustomerID,JSONENC%>"
       },
       {
         "columnId": 5331964812126084,
         "value": <%EmployeeID%>
       },
       {
         "columnId": 3080164998440836,
         "value": "<%OrderDate,yyyy-MM-ddTHH:mm:ss%>"
       }
     ]
   }

7. Here is final configuration of Template Transform
   

   Create JSON for API Request (POST) using SSIS Template Transform

Configure SSIS Web API Destination (POST New Rows to Smartsheet)

Now lets configure last step. This will POST API request to insert multiple rows to Smartsheet. Previous template transform crafts request for one new row but later in this section we will enable Batch mode so we can JOIN multiple JSON and created Bulk requests in one go so we avoid many API call. Always use Bulk mode if API endpoint supports it.

1. Drag and drop ZS Web API destination from SSIS Toolbox
2. Connect previous Template Transform to your Web API Destination.
3. Select new HTTP connection. When Prompted enter following URL on HTTP Connection UI (Replace your Sheet ID)
   

   https://api.smartsheet.com/2.0/sheets/YOUR-SHEET-ID/rows

4. In the input column for Body select TemplateOutput
5. For Body you can enter Sample Body if you wish to Test at design time else leave it blank
6. Select Body Content Type as application/json
7. In the Headers enter Authorization Header like we did in previous section (Read from Smartsheet)
   

   Authorization: Bearer YOUR-TOKEN-GOES-HERE

   

   Configure SSIS Web API Destination – Add / Insert Rows to Smartsheet (Bulk API call)

8. Go to Batch Setting mode. Change following settings
   1. Check on Enable submitting multiple records
   2. For Body Header enter [
   3. For Body Footer enter  ]
   4. For Body Row separator enter , (i.e comma)
   5. You can change Body Batch Size to desired value (100 is recommended)
      

      Configure SSIS Web API Destination for Bulk API call (Smartsheet REST API – Insert multiple rows)

9. That’s it run your package to test the entire flow.
   

   Loading data from SQL Server to Smartsheet using SSIS (Web API POST Example – Bulk Mode)

Conclusion

Smartsheet API provides great way to automate data read/write. However to call Smartsheet API  you have to use SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides great way to integrate any Smartsheet API call via simple drag and drop approach without coding. Try  SSIS PowerPack for free and call virtually any REST API in few clicks.

Keywords: Import smartsheet into sql server | export smartsheet to sql server | ssis smartsheet read | ssis smartsheet write | reading smartsheet data using API call | write to smartsheet

The post Read / Write Smartsheet data using SSIS REST API Call appeared first on ZappySys Blog.

In our previous article, we saw step-by-step approach to read data from workday using SSIS. In this article, we will focus on how to load SQL Server data to Workday (e.g. POST, Create, Update). We will use SSIS Web API Destination and the combination of other Transforms such as SSIS Template Transform and SSIS XML Generator Transform .

The main thing in this article how to create SOAP Request correctly inside your request. This article explains how to use Free tools like SoapUI to create Workday Request. Once you do that you can call virtually any Workday API using the same technique. If you need more help contact our Support and we will be happy to help you step by step.

Step-By-Step : Import SQL Server data to Workday using SSIS

Lets build our SSIS Package to load data from SQL Server or any Source (e.g. Oracle, DB2) to Workday using SOAP API calls. Using below approach you can Create new records or Update existing records in Workday. You can also delete records with same concept.

Basic steps outlined below.

1. Fetch records from SQL Source and build XML Request for each Row (e.g. Create Account)
2. Build XML Request (SOAP Body) using Template Transform or  SSIS XML Generator Transform. If you have Array nodes (e.g. One to Many) then you have to use SSIS XML Generator Transform else use Template Transform for ease of use.
3. Pass input record (e.g. SOAP Body) to Web API destination to call Workday API call (CREATE, UPDATE, DELETE requests)
4. Parse XML Response (i.e. output) using SSIS XML Parser Transform or save raw XML to SQL Server database.
5. Redirect Bad rows or failed requests to log file for review

Now let’s look at each step in detail

Obtain Workday API URL

Once you have WSDL file, next step is craft correct URL for API service you like to call. The service name can be obtained from here (Check service column)

Syntax: https://<workday host name>.workday.com/ccx/service/<tenant name>/<service-name>
Example: https://MY-INSTANCE.workday.com/ccx/service/MY-TenantID/Human_Resources

Craft SOAP Body (XML API Request) using SoapUI

Now its time to craft some SOAP Request. Check steps outlined here (Use SoapUI tool) . Once you have Request Body XML you can change parameters as per your need.

Here is sample  SOAP XML Body for Get Employee call from Human_Resopurces service.

Creating SSIS Connection for Workday SOAP API call using WSS Security

Loading SQL Server data to Workday using SSIS

Let’s look at the real-world scenario. You have Accounts table stored in SQL Server and you like to create same accounts in Workday by calling appropriate API calls.

1. Drag Data flow task from SSIS Toolbox. Double click to edit.
2. Drag OLEDB Source configure to read SQL Table (e.g. Accounts)
3. Drag ZS Template Transform from the toolbox. Connect OLEDB Source to Template Transform. If you need flexible XML Generation then use XML Generator Transform but it may require some learning curve so for simplicity we are skipping that from this article.
4. Enter your SOAP request in the template text (like below) you like to call (This is obtained from the previous section – using tool like SoapUI)
   For Example: To create a new account you can use enter like below. Replace xxxxxxxxxx with Columns placeholder.
   To insert Column name as Placeholder click <<Insert Placeholder>> and then Select [Columns] node. Template Transform outputs column name TemplateOutput. You can use this as Body to feed next step (i.e. Call Workday API using Web API Destination )When you insert placeholder to make sure you use XML Encoded Columns if you expecting a Long text or special characters part of your data.
   Syntax for encoded value is   

   <%CustomerName,FUN_XMLENC%>

     . You don’t need FUN_XMLENC for numeric fields. For normal placeholder without encoding use just name with column placeholder indicators e.g.  

   <%Amount%>

   <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:bsvc="urn:com.workday/bsvc">
      <soapenv:Header/>
      <soapenv:Body>
         <bsvc:Workday_Account_for_Worker_Add>
            <bsvc:Worker_Reference>
               <!--You have a CHOICE of the next 2 items at this level-->
               <bsvc:Employee_Reference>
                  <bsvc:Integration_ID_Reference>
                     <bsvc:ID>xxxxxxxxxx</bsvc:ID>
                  </bsvc:Integration_ID_Reference>
               </bsvc:Employee_Reference>
               <bsvc:Contingent_Worker_Reference>
                  <bsvc:Integration_ID_Reference>
                     <bsvc:ID>xxxxxxxxxxxx</bsvc:ID>
                  </bsvc:Integration_ID_Reference>
               </bsvc:Contingent_Worker_Reference>
            </bsvc:Worker_Reference>
            <bsvc:Workday_Account_for_Worker_Data>
               <!--type: string-->
               <bsvc:User_Name>xxxxxxxxxxxxxx</bsvc:User_Name>
            </bsvc:Workday_Account_for_Worker_Data>
         </bsvc:Workday_Account_for_Worker_Add>
      </soapenv:Body>
   </soapenv:Envelope>

   

   Template Transform –
   Create Workday SOAP Request – Create new records using SSIS

5. Drag ZS Web API Destination from SSIS Toolbox. Connect Template Transform to Web API Destination.
6. Configure Web API Destination as below (See we used same HTTP connection created in the previous section)
7. Select HTTP Connection
8. Select Input Column for Body (select TemplateOutput)
9. Select Body Content Type as XML (text/xml;charset=UTF-8)
10. Click Raw Edit above Headers grid. Enter following text (We just need one header)
    

    SOAPAction: ""

    

    SSIS Web API Destination –
    Loading SQL Server data to Workday (SOAP Call – Create, Insert, Update, Delete Records)

11. Now you can connect Web API Destination Output (Blue Arrow to Either XML Parser or some other Destination). Web API destination gives you a response in Raw XML Format. This article explains how to use JSON or XML Parser to parse API response coming from Web API destination.

Video Tutorial (See Part#4) – Create XML and POST data to SOAP Web Service or REST API

Now lets look at examples of creating XML from multiple data sources and POST XML request to SOAP Web Service URL or any other XML Based REST API URL. This video has 5 parts to cover full length tutorial but if you want to fast forward to see XML Generator Transform and Web API Destination then see part#4.

Sending complex SOAP Request (Nested Parent-Child Array)

If you have to call more complex SOAP request which requires Parent-child structure (Nested Array) then you cant use Template Transform. In that case, check this article for real-world example.

Create nested XML for SOAP Request – Multiple inputs (POST data to API using SSIS Web API Destination)

Extract single XML node from SOAP Response

There will be a time when you need to extract just one value out of your response and save into SSIS variable / use it later on. If that’s the case then refer to this article.

Conclusion

In this article we have learned how to load data from SQL Server to Workday using SSIS ( drag and drop approach without coding). Combination of few ZappySys Components (e.g. Web API Destination, XML Generator ) makes it super simple to call any Web API to load data from one system to API endpoint. Download SSIS PowerPack to try many other automation scenarios not discussed in this article.

The post Load SQL Server data to Workday using SSIS / SOAP API appeared first on ZappySys Blog.

In our previous blog post we saw how to perform Read and Write operations in MongoDB using SSIS (i.e. Bulk Update, Delete, Upsert, Insert). In this post we specifically focus on how to update MongoDB Array items / elements using SSIS.

To make things simple to follow we have used JSON Source to produce sample JSON documents for MongoDB update (NOTE: Output as Document option is checked for JSON Source). However in real world you may have to use SSIS JSON Generator Transform to produce input documents for MongoDB. To learn more about generating JSON documents for MongoDB Load process check this blog post or watch video found on this page.

Video Tutorial – Example of insert/update documents inside MongoDB Array (Use $set, $push operator)

Basic Concepts: Performing MongoDB CRUD operations using SSIS

If you never heard term called CRUD stands for Create, Read, Update, Delete. These are the most common operations you have to do with any data source. SSIS PowerPack comes with many MongoDB Components to help you with CRUD operations. You can use one or more following components to achieve drag and drop MongoDB CRUD operations in SSIS. When you use SSIS PowerPack for MongoDB all operations will be Bulk Operations by default which means it will provide very high throughput.

• SSIS MongoDB Source
• SSIS MongoDB Destination
• SSIS MongoDB ExecuteSQL Task

Download Example SSIS Package

Here is the link to download Sample SSIS Package

SSIS Example : Update MongoDB Array Items / Elements (Update, Delete, Insert – Using $set, $pull, $push)

Update MongoDB Array Items ($set operator)

In next couple of sections we will see how to perform Update, Insert and Delete operations with MongoDB Array Items using various update operators and array operators. Consider the following Sample JSON documents for MongoDB. First we will see how to use MongoDB Shell commands to perform CRUD operations on MongoDB Array items.

{
    "_id" : ObjectId("5883ebe669e0f22ba890b49b"),
    "CustomerID" : "BOLID",
    "Name" : "Some name",
    "Orders" : [ 
        {
            "OrderID" : 200,
            "ShipCountry" : "USA",
            "City" : "Atlanta"
        }, 
        {
            "OrderID" : 201,
            "ShipCountry" : "USA",
            "City" : "New York"
        }
    ]
}

Update Array element using Shell command

Assume you want to update ShipCountry attribute of one of the Orders (OrderID=200) and for CustomerID=”BOLID” in above sample document.

Shell Command for Update Array Item

Use below shell command in your favorite MongoDB Client Tool (e.g. RoboMongo). Notice two things in below command because we will apply same concepts when we take this approach to SSIS. In below command Actual JSON document is wrapped inside. This is MongoDB Update operator. Click here to learn more about $set. Also notice that rather than nested Array syntax we used Orders.$.ShipCountry to update ShipCountry which is part of Array element. MongoDB provides special way to navigate array elements using   $ Operator (MongoDB Positional Update)

db.MyCollection.update(
	{ "CustomerID" : "BOLID", "Orders.OrderID" : 200 }, 
	{ $set : { "Orders.$.ShipCountry" : "Test1"} } , 
	{ upsert: false, multi: false}
)

After Update (MongoDB Document)

After you run above shell command notice all other attributes remain there in Array document and only ShipCountry is changed. This is because we used $ Operator (MongoDB Positional Update) rather than supplying Raw JSON for Array document (i.e. Orders : [ { … } ] ).

{
    "_id" : ObjectId("5883ebe669e0f22ba890b49b"),
    "CustomerID" : "BOLID",
    "Name" : "Some name",
    "Orders" : [ 
        {
            "OrderID" : 200,
            "ShipCountry" : "Test1",
            "City" : "Atlanta"
        }, 
        {
            "OrderID" : 201,
            "ShipCountry" : "USA",
            "City" : "New York"
        }
    ]
}

The issue with this approach its not Bulk operation. Assume that you have 100,000 records to update in various documents and its nested array items then row by row operation is not effective. To solve this issue we will use SSIS MongoDB Destination which performs Bulk operations (e.g. Bulk Update, Delete, Insert and Upsert).

Update MongoDB Array Items using SSIS

Now lets implement above scenario using SSIS. Assume that you have sample MongoDB document (as per above example – Before Update). We want to update ShipCountry for OrderID=200 for Customer BOLID. To do that perform following steps in SSIS. If you are updating Second level array (i.e. Root >> Orders >> OrderItems)  then you have follow slight different process. Check next section in this article about updating second level array (Insert items)

1. Download and Install SSIS PowerPack
2. Create new SSIS Project with one data flow task
   

   Dragging and dropping Data Flow Task into Control Flow

3. Drag ZS JSON Source and check Output as Document option (as below sceenshot). You can use JSON Source is your JSON is stored inside file or Variable or coming from Some API call which eventually go into MongoDB. You can also use JSON Source for quick testing where you Hardcode JSON to feed to MongoDB.
4. Enter the following JSON in the text area. Click OK to save.
   

   {"CustomerID":"BOLID", "Orders.$.OrderID":201, "Orders.$.ShipCountry": "Test1"}
   {"CustomerID":"BOLID", "Orders.$.OrderID":202, "Orders.$.ShipCountry": "Test2"}
   /**Below is extra record not found in destination - will be ignored in Update**/
   {"CustomerID":"BOLID", "Orders.$.OrderID":203, "Orders.$.ShipCountry": "Test3"}

5. Right click in Connections area and Create new ZS-MONGODB connection. Specify credentials and click Test before you hit OK to save.
6. Now drag ZS MongoDB Destination from SSIS Toolbox.
7. Select Runtime connection.
8. On the properties tab edit following properties
   1. Set ColumnsForLookup property as below (Notice how we use 2 columns for JOIN. Also used AS keywords for 2nd Column so we have <column-from-input-doc> AS <column-for-server-side-find>
      CustomerID,Orders.$.OrderID AS Orders.OrderID
   2. Set LoadOptions as below (Op is Update Operator you like to use. In our case its $set, Multi=True means if multiple match found then all matching documents will be updated). If you want to use multiple update operators then you have to supply that from Input document and use op=none option rather op=$set.
      op=$set;multi=true
   3. Set Operation property to Update
9. On the Mappings tab Map __DOCUMENT__ from upstream (In our case you will see same name column from JSON Source if you checked Output as Document option)
10. Click OK to save MongoDB Destination UI
11. Run package. You will notice after execution two orders of CustomerID=BOLID will be updated.

SSIS JSON Source – Read JSON (Raw Documents)

SSIS MongoDB Destination – Update MongoDB Array Item using $set update operator

Insert MongoDB Array Item ($push / $addToSet Operator)

Now lets look at how to insert new item into existing Array. Lets say we already have two orders for a customer and now we want to add new order. For this you can either use  $push operator or $addToSet operator. $push operator doesn’t check for duplicate item so its little faster. If you wish to skip append if item already exists (by matching all attributes) then use $addToSet operator.

Using Shell Command : Insert MongoDB Array Item (Append to array)

Here is the shell command which will push new item to the array

db.MyCollection.update(
	{CustomerID:"BOLID"},
	{$push:{Orders: {"OrderID":202,"ShipCountry":"USA","ShipCity":"Atlanta" } }},
	{upsert: false,multi: false}
)

Using Shell Command : Insert MongoDB Array Item at second level

Here is the shell command which will push new item to the array which is stored at second level e.g. { Orders :  [ { Items : [ insert-here ] } ] }

db.MyCollection.update(
	{CustomerID:"BOLID", "Orders.OrderID" : 202 },
	{$push:{ "Orders.$.Items" : {"ProductID":1001,"Quantity":3} } },
	{upsert: false,multi: false}
)

Using SSIS : Insert MongoDB Array Item at Second Level

Now lets look at some example how to Insert Items inside Orders array. Since it requires two JOIN columns to perform second level array insert you may need to add metadata for join.

First step is create documents for update. There are atleast 4 ways you can create input documents for MongoDB Destination.

1. You can use JSON generator Transform to create documents for load process
2. You can use Export JSON Task to generate input documents file and then use JSON Source to Read Documents (Check Output as Document option on JSON Source to read as RAW JSON)
3. You can use JSON Source to feed direct JSON (This approach is demonstrated in the below example sceenshot)
4. You can also use SQL query to output JSON Strings along with columns used for JOIN condition
   

   select CustomerID, OrderID as 'Orders.$.OrderID', DOC = '{Orders.$.Items : {ItemID: 1, Qty: 5} }'

To make it simple lets use #3 approach from above list. We will create few documents and insert New orders Items

Sample Document for MongoDB Collection

For testing purpose you can create MongoDB collection called test with following one document. As you see Items array is empty for both orders but once you run sample each will have 2 items.

{
    "_id" : ObjectId("58c01bee0610fea8ddda04b9"),
    "CustomerID" : "ALFKI",
    "Orders" : [ 
        {
            "OrderID" : 100,
            "Items" : [ ]
        }, 
        {
            "OrderID" : 101,
            "Items" : [ ]
        }
    ]
}

Now lets look at how to configure SSIS package

Step-1 : Create JSON Documents for Array Insert

Lets use JSON Source like below. Use following Sample JSON

/*Insert new items into order#100*/
{CustomerID:"ALFKI","Orders.$.OrderID" : 100, Doc: "{ \"Orders.$.Items\" : {ItemID: 1, Qty: 10} }" }
{CustomerID:"ALFKI","Orders.$.OrderID" : 100, Doc: "{ \"Orders.$.Items\" : {ItemID: 2, Qty: 11} }" }

/*Insert new items into order#101*/
{CustomerID:"ALFKI","Orders.$.OrderID" : 101, Doc: "{ \"Orders.$.Items\" : {ItemID: 1, Qty: 20} }" }
{CustomerID:"ALFKI","Orders.$.OrderID" : 101, Doc: "{ \"Orders.$.Items\" : {ItemID: 2, Qty: 21} }" }

Create Sample JSON Documents for MongoDB Array Update (2nd level array)

Step-2 : Configure MongoDB Destination for $push operation on Second level Array

Now lets connect source to Destination and configure MongoDB Destination. Select connection on first tab. Then On second tab change properties like below.

Configure MongoDB Destination for second level array update (Insert item using $push update operator)

Step-3 : Add missing JOIN column for Mapping

When you click on Mappings tab you will notice only root level columns are listed there for mapping but if you are JOINing based on Nested column found inside array then you have to add missing mapping column by hand like below screenshot.  In our case we Join by CustomerID and OrderID where OrderID is found in Orders array which is missing on mappings tab. So lets add it. Just make sure name new column using $ positional operator… e.g. Orders.$.OrderID  this will tell MongoDB that our OrderID column must be lookup under Array called Orders: [  ]

MongoDB Destination – Add missing metadata column for JOIN

Step-4 : Map JOIN columns and Input DOCUMENT

Now finally map all columns

SSIS MongoDB Destination – Column Mappings

Using SSIS : Insert MongoDB Array Item (Append to array)

Now lets look at how to perform same operation to insert array item in SSIS using $push operator. In this case we will do little different than Update Item example ($set operator). First we have not included CustomerID as part of Input columns and we mapped it on Mapping Screen rather than supplying columns names in ColumnsForLookup property (On Component Properties Tab of Destination UI). If you are using JSON Generator Transform to generate your JSON then you have option to Hide/Include certain columns part of your flow. Check Include column to downstream option if you want to use for JOIN column.

Set following settings

• Set LoadOptions as below
  op=$push;multi=true
• Set Operation=Update
• Leave ColumnsForLookup as empty
• On the mappings tab map __DOCUMENT__ and CustomerID in the Target. If you don’t see CustomerID column in the mapping create atleast one sample doc in MongoDB Collection so metadata is detected.

Download Sample for more detail.

SSIS JSON Source – generate JSON for MongoDB Array Insert

SSIS MongoDB Destination – Insert MongoDB Array Item (Use $push, or $addToSet operator)

Delete MongoDB Array Item ($pull operator)

If you wish to remove array item then you can use $pull operator. Only difference is rather than specifying two conditions in first argument… you just specify JOIN columns without array column (i.e. OrderID). To specify Array item condition you have to set inside $pull : { … }

Using Shell Command for MongoDB Delete Array Item (Remove by condition)

Below example shows how to remove OrderID=201 from Orders array for CustomerID=BOLID.

db.MyCollection.update(
	{CustomerID:"BOLID"},
	{$pull:{Orders: {"OrderID":201} }},
	{upsert: false,multi: false}
)

Using SSIS for MongoDB Delete Array Item

Now lets look at how to implement Delete logic for MongoDB Array Item in SSIS. As you see in below screenshots You have to modify following things

• Set ColumnsForLookup property to CustomerID (no need to add OrderID here because its part of input document)
• Set LoadOptions property to op=none;multi=false. op=none means we are supplying Update operator part of input document.

Assume you have doc like below in MongoDB  and you like to remove Order#101 from array

{
    "_id" : ObjectId("58c01bee0610fea8ddda04b9"),
    "ParentID" : 1,
    "Orders" : [ 
        100, 
        101, 
        102
    ]
}

For this supply 2 columns from upstream
col1  = ParentID
col2  = DOCUMENT

Document column should look like below
{Orders: 101}

1. On MongoDB destination change LoadOptions property to  op=$pull;multi=true
2. Change Operation to Update
3. Map ParentID and DOCUMENT columns on Mapping screen
4. Run the package … It will remove Order# 101 from array

Array Upsert – Update / Insert using Custom JOIN condition (such as $ne )

If you like to perform Upsert Operation on MongoDB Array Items then refer to below blog post.

Update or Insert – Upsert MongoDB Array Items using SSIS

Conclusion

MongoDB integration can be challenging if you are new to NoSQL world. If you are using SSIS as your primary ETL tool then not to worry because SSIS PowerPack can give super power needed to complete your project on time with drag and drop high performance connectors.

Keywords: mongodb update array item | mongodb update array elements | mongodb update array documents | MongoDB $ operator | MongoDB update sub document items | MongoDB CRUD operations | MongoDB Bulk Updates | MongoDB bulk updates to array items

The post Update MongoDB Array Items using SSIS appeared first on ZappySys Blog.