In our previous post, we explored how to read and write PostgreSQL data using the ZappySys PostgreSQL Source and Destination connectors for SSIS. In this article, you will learn how to connect MariaDB to PostgreSQL using ZappySys SSIS PowerPack and transfer data between both systems.

We will demonstrate a practical ETL pattern using ZappySys ODBC/JDBC connectivity to extract data from MariaDB and load it into PostgreSQL within SSIS.

Prerequisites

Before you begin, make sure you have the following:

• First, you will need to have SSIS installed
• Secondly, make sure to have SSDT
• Thirdly, access credentials for both MariaDB and PostgreSQL
• Finally, do not forget to install ZappySys SSIS PowerPack and ZappySys ODBC PowerPack

Steps

Configure MariaDB Connection Using ODBC (JDBC Bridge)

Since MariaDB connectivity is handled via JDBC, we will use the ZappySys JDBC Bridge Driver to expose it as an ODBC data source.

1. Visit the official MariaDB website.
2. Download the latest MariaDB JDBC driver.
3. Save the .jar file locally, for example: D:\Zappysys\drivers\mariadb-java-client-3.5.7.jar
4. Download and install ZappySys ODBC PowerPack if not already installed.
5. Open ODBC Data Sources (64-bit).
   

   Open ODBC Data Sources

6. Create a User data source (User DSN) based on ZappySys JDBC Bridge Driver:
   

   Create a User data source

7. Configure the JDBC Connection. Fill in the connection details
   • Connection String  jdbc:mariadb://{host-name}:{port}/{database-name}

   • JDBC Driver File D:\Zappysys\drivers\mariadb-java-client-3.5.7.jar

   • Username root

   • Password ********

   • Connection Parameters[]
8. Click Test Connection. If configured correctly, you should see a success message.
   

   Zappysys JDBC – MariaDB

9. You can also use the Preview tab to select the table and execute SQL queries. For advanced SQL examples, refer to the JDBC Bridge documentation.
   

   Zappysys JDBC – MariaDB preview result

Connect MariaDB to SSIS

1. Open Visual Studio.
2. Create a New Project.
3. Select Integration Services Project.
4. Provide a project name and location, then click OK.
5. Drag and drop a Data Flow Task from the SSIS Toolbox onto the Control Flow surface.
   

   Drag and drop Data flow.

6. Double-click the task to open the Data Flow Designer.
7. Drag and drop an ODBC Source component into the Data Flow.
   

   Drag and drop the ODBC source.

8. Double-click the component to configure it.
9. Click New… to create a new ODBC Connection Manager.
   

   Creating an ODBC Connection Manager in SSIS

10. Select the MariaDB ODBC DSN you created earlier.
11. Click Test Connection and click OK to save the configuration.
    

    Creating an ODBC Connection Manager in SSIS from JDBC

12. Choose Data Access Mode: Table name or SQL command.
13. Select the desired table or enter a custom query.
14. Click Preview to validate the data, and click OK to save the configuration.
    
    

    Preview an ODBC Connection Manager in SSIS from JDBC

Configure PostgreSQL Destination

1. In the Connection Manager pane, click New Connection.
2. Select ZS-POSTGRESQL.
3. Configure the connection:
   • Host
   • Username
   • Password
   • Port
   • Database name
4. Use the other tabs for more configuration if needed
5. Click Test Connection and Click OK.
   

   PostgreSQL connection configuration

6. Drag and drop the PostgreSQL Destination component.
   

   Drag and drop the PostgreSQL destination.

7. Connect it to the ODBC Source and double-click to configure it.
8. Select the PostgreSQL connection.
   

   PostgreSQL Destination – Choose the connection

9. Go to the Properties tab and choose the target table.
   

   PostgreSQL Destination – Choose the target table

10. Go to the Column Mappings tab and map source columns to destination columns.
    

    PostgreSQL Destination – Map the columns

11. Click OK to save the configuration.
12. Run the SSIS package. If everything is configured correctly, the data will be transferred from MariaDB to PostgreSQL.
    

    PostgreSQL Destination – Final result

Alternative Option — Using Upsert Destination

An alternative destination component is the Upsert Destination; you can replace it with the PostgreSQL destination. You can preview data from the element, select the options to insert or/and update, and more.

1. Replace the PostgreSQL Destination with the Upsert Destination.
2. Select your PostgreSQL connection.
3. Choose the target table.
4. Choose Insert or/and Update action.n
5. Map the columns.
6. Select the primary key column(s).
7. Configure additional options under the Advanced tab if needed.
   

   Upsert Destination – PostgreSQL destination table

8. Run the package to perform the upsert operation.
   

   Upsert Destination – Final result

Conclusion

With ZappySys SSIS PowerPack and ODBC PowerPack, connecting MariaDB to PostgreSQL becomes straightforward and reliable.

By configuring the ODBC/JDBC bridge for MariaDB and using the native PostgreSQL destination in SSIS, you can build a robust ETL pipeline between the two databases.

Explore the full capabilities of the ZappySys SSIS PowerPack and ZappySys ODBC PowerPack to optimize your data integration tasks.

Need Help?

If you encounter any issues, our support team is here to help:

• Live Chat — Use the chat widget (bottom-right corner of this page)
• Email — support@zappysys.com
• Support Center — Support | ZappySys

The post SSIS tutorial: Migrating MariaDB data to PostgreSQL appeared first on ZappySys Blog.

In this article, you will learn how to download Outlook (Office 365) emails as .EML or .MSG files using ZappySys SSIS PowerPack without writing any code. We’ll leverage the power of Microsoft Graph API, which provides unified access to Microsoft 365 services like Outlook Mail, and integrate it seamlessly using the API Connector for SSIS Outlook Mail (Office 365), Web API Destination and other components. By the end, you’ll be able to automate the process of extracting emails and saving them to disk in standard email formats directly from your SSIS package.

Step 1: Register an Application in Azure Portal

To access Outlook emails via Microsoft Graph API, the first step is to register an application in the Azure Portal. This allows your SSIS package to authenticate securely and retrieve mail data from Office 365.
Step by step info is available here Outlook Mail (Office 365) Connector Authentication section.

User Credentials

Use delegated access (User Credentials) whenever you want to let a signed-in user work with their own resources or resources they can access. Whether it’s an admin setting up policies for their entire organization or a user deleting an email in their inbox, all scenarios involving user actions should use delegated access.
Refer to this links for the detailed steps: User Credentials [OAuth]

Application Credentials

Application-only access is broader and more powerful than delegated access (User Credentials), so you should only use app-only access where needed. Use it when:

1. The application needs to run in an automated way, without user input (for example, a daily script that checks emails from certain contacts and sends automated responses).
2. The application needs to access resources belonging to multiple different users (for example, a backup or data loss prevention app might need to retrieve messages from many different chat channels, each with different participants).
3. You find yourself tempted to store credentials locally and allow the app to sign in ‘as’ the user or admin.

Refer to this links for the detailed steps: Application Credentials [OAuth]

Step 2: Download Outlook (Office 365) Emails as .EML Files using SSIS

Now let’s look at some examples of how to access your Outlook emails using our easy-to-use Outlook Mail Connector in ZappySys SSIS PowerPack. This connector simplifies calling the Microsoft Graph API, so you can retrieve emails with just a few clicks.
In this section we will learn how to configure and use Outlook Mail (Office 365) Connector in API Source to extract data from Outlook Mail (Office 365).

1. Begin with opening Visual Studio and Create a New Project.
2. Select Integration Service Project and in new project window set the appropriate name and location for project.
   And click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on left side bar
   2. Solution Explorer and Property Window on right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window in the bottom

   

   Note: If you don’t see ZappySys SSIS PowerPack Task or Components in SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox. Double click on the Data Flow Task to see Data Flow designer.
5. From the SSIS toolbox drag and API Source (Predefined Templates) on the data flow designer surface, and double click on it to edit it:
   
6. Select New Connection to create a new connection:
   
7. Use a preinstalled Outlook Mail (Office 365) Connector from Popular Connector List or press Search Online radio button to download Outlook Mail (Office 365) Connector. Once downloaded simply use it in the configuration:
   

   Outlook Mail (Office 365) Connector Selection

8. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed. You may press a link Steps to Configure which will help set certain parameters. More info is available in Authentication section.
   1. User Credentials [OAuth]
      

      User Credentials [OAuth]

      OR
   2. Application Credentials [OAuth]
      

      Application Credentials [OAuth]

9. Select the Get Messsages endpoint, change/pass the properties values, and click on Preview Data button to make the API call.
   

   API Source – Get Messages Endpoint

10. That’s it — click OK and you’re done! In just a few clicks, you’ve successfully configured a connection to Outlook Mail (Office 365) using the Outlook Mail Connector.
11. Now let’s move to the next step — retrieving the full email content.
    You’ll use the Web API Destination component to make the Get MIME content API call. Within this task, you can dynamically pass the Message ID to fetch each email’s raw .EML content from Outlook (Office 365).
    For more details, refer to the official Microsoft documentation: Get Outlook Mail MIME content
12. Drag and drop the Web API Destination component from the SSIS Toolbox onto your Data Flow.
    Double-click to open it, then configure as follows and create a new OAuth Connection if needed:URL:

    https://graph.microsoft.com/v1.0/me/messages/<%Id%>/$value

    The <%Id%> placeholder is dynamically replaced by SSIS at runtime with the actual Message ID, allowing the Web API Destination to download each email’s .EML content individually.

    

    Get MIME content’ API call using the Web API Destination

13. You’ll also need to create the OAuth connection using the same credentials.
    On the Advanced tab, make sure to set the Redirect URL correctly.For detailed steps, refer to this guide:
    How to set up OAuth 2.0 in SSIS
    

    Outlook Mail OAuth 2.0 Connection

14. Next, add a Derived Column transformation to create a new column that defines the file path for saving the .EML file.
    In this example, we’ll use the Message ID to generate a unique file name for each email:Expression:

    (DT_WSTR,1000)”E:\\zsTemp\\” + [Id] + “.eml”

    

    Derived Column – Dynamic File Path

15. Now, add an Export Column transformation to save the email content to a file.
    • Extract Column: set to ResponseText (this holds the email content from Web API Destination)
    • File Path Column: set to the column you created in the Derived Column step (e.g., FilePath)
    • Enable Force Truncate and Write Byte-Order Mark options
    

    Export Column – write the email content to a file on disk.

16. Set File Encoding to UTF-8 (Optional)
    Next, go to the Control Flow and use the Advanced File System Task to convert the downloaded .eml file’s encoding to UTF-8.
    This step is optional — I had to apply it on my system because the files weren’t opening correctly due to a non-UTF-8 encoding.
    If you encounter a similar issue, follow this step to ensure proper compatibility with email clients.
    

    Advanced File System Task – Change Encoding

17. That’s it! The process will now automatically download your Outlook messages to your local system as .EML files.
    

    Outlook .eml file’s

Conclusion

Using ZappySys SSIS PowerPack and Microsoft Graph API, you can easily automate the download of Outlook (Office 365) emails in .EML format — all within SSIS and without writing any code.
This approach is ideal for archiving, compliance, or integrating email content into downstream systems.
With a few simple components and steps, your email extraction workflow can be up and running in minutes.

The post How to download Outlook (Office 365) Emails as .EML or .MSG Files appeared first on ZappySys Blog.

In our previous post we saw how to read API data in SSIS. This post we will cover how to call NetSuite RESTlet to read / write data in SSIS or ODBC Apps. For demo purpose we will use SSIS PowerPack REST API Task but concepts and many UI elements remain same for ODBC JSON Driver too. This article assumes you know how to write Netsuite RESTlet and you have obtained Credentials to call RESTlet from extranal app.

What is NetSuite RESTlets

NetSuite added a really nice feature to expose business data and workflows as API by writing Javascript functions which they call it RESTlets. You can secure and customize these restlets as per your need.

You can automate CRUD (Create /Read / Write / Delete) operations in NetSuite by writing special functions in JavaScript and expose those functions as API URL which can be called by any external Systems (i.e. SSIS Packages , ODBC Apps like Power BI, Informatica, SSRS).

RESTlet can be secured few ways such as OAuth 1.0 and OAuth 2.0 authentication. We will cover both approach briefly in this article. You can choose any one as per your need

Creating NetSuite RESTlet / Obtain OAuth Credentials (Client ID and Secret)

Below video shows how to obtain necessary credentials to call Netsuites API (RESTlet).

At mark 4:30 seconds it will show you How to use OAuth 2.0 for calling RESTlet

Create NetSuite RESTlet Connection

There are few ways to authenticate NetSuite RESTlet. If you are using ZappySys Products then it makes it super easy to use either approach. Lets look at both approaches.

Using OAuth 2.0 (Recommended)

As per above video if you obtained Client ID and Secret for OAuth 2.0 you can follow below steps to configure OAuth connection to call RESTlet.

What you need to use OAuth 2.0.

• Your Account ID (it goes in URL)
• Your Client ID and Secret (Obtained from Netsuite Portal) – See above video Mark 04:30
• Register Redirect URL in NetSuite portal (we recommend you use https://zappysys.com/oauth as call back Url for Automatic Code extraction)

Steps to configure OAuth Connection

1. Right click on Connection Panel and choose Choose “New Connection..”. Select ZS-OAUTH type
   
   
   

   Create new SSIS OAuth API Connection Manager

    

    

2.  Enter following things
   ————————-
   On General Tab
   ————————-
   ClientID and Secret: <get it from NetSuite>
   Authorization URL: https://<accountID>.app.netsuite.com/app/login/oauth2/authorize.nl
   Token URL: https://<accountID>.suitetalk.api.netsuite.com/services/rest/auth/oauth2/v1/token
   Scope: restlet
   

   Configure NetSuite OAuth 2.0 Connection for API call

    

   ————————-
   On Advanced Tab
   ————————-
   Callback / Redirect URL:  https://zappysys.com/oauth  (this must be entered same way in NetSuite Portal)

   

   OAuth2 Connection – Callback / Redirect URL

3. Now go back to General Tab and click Generate Token button
4. Follow login screen and try to finish the flow. At the end it will generate Tokens and populate on UI
5. Click OK to save

Using OAuth 1.0

Another approach is use OAuth 1.0 (Older version). If you have enabled OAuth 1.0 in NetSuite then follow below steps (Create connection is same as before).

1. Setup General Tab and Advanced Tab as below.
   On ==== General Tab====:
   Enter ClientID, ClientSecret, AccessToken, AccessToken Secret
   On ==== Advanced Tab ====:
   Check Do Not Include Empty Key/Value Pairs
   Select Signature Method as HMAC-SHA256
   For Extra Token Attribute enter realm=<your-account-number>_SB1
2. Click OK to Save

Call NetSuite RESTlet in SSIS

1. Create a new SSIS Package
2. Drag REST API Task from SSIS Toolbox
3. Double click the Task to configure
4. Select URL from Connection
5. From Connection dropdown select OAuth connection we created in the earlier section (either OAuth 1 or OAuth 2)
6. Enter the URL as below. Assuming you copied URL from NetSuite Replace script ID and account number
   

   https://<account-number>-sb1.restlets.api.netsuite.com/app/site/hosting/restlet.nl?script=<scriptid>&deploy=1

   Enter <account-number>
   Enter <scriptid>
7. Change Method to POST if your script supports Parameters (Else you can supply id=xxxx and recordtype=xxxxx in the URL for GET request)
8. If its POST request then chnage ContentType to ApplicationJSON (application/json)
9. Enter your Input in the body. In below example our RESTlet takes sql query as input so we enter just one parameter named query but you can have different parameter(s)
   

   Syntax:  { param1_name : <value> , param2_name : <value> ....  }

10. Now click Test see everything works
    

    Calling NetSuite RESTlet in SSIS (Using REST API Task)

Conclusion

We saw how easy it is to call NetSuite RESTlets or other NetSuite API in SSIS / other BI Apps using ODBC Drivers provided by ZappySys. Download SSIS PowerPack and try your self see how easy it is to consume virtually any API in SSIS.

The post How to call NetSuite RESTlet API in SSIS / ODBC appeared first on ZappySys Blog.

In this post, you will learn how to read / write ManageEngine ServiceDesk data in SSIS (Drag and drop approach without any coding).  We will use SSIS JSON/ REST API Source to extract data from Servicedesk API and use SSIS Web API Destination to write data to Zoho.

We will look at step-by-step instructions to read ServiceDesk  data (e.g. Request, Problems, Contacts) and load into SQL Server Database Table.

Authentication ServiceDesk REST API in SSIS (Use OAuth 2.0)

Very first step to call ServiceDesk API is you need to register OAuth App in Zoho Portal. Don’t get confused if you see Zoho Portal for ServiceDesk a Product from ManageEngine is in fact Zoho subsidiary (in other words Zoho is the parent company). ServiceDesk has On-Premises version too but this article is only about Cloud version.

Register Zoho OAuth App and get ClientID/ Secret (*** MUST READ ***)

Check this article to register your OAuth App and obtain Client ID  /Client Secret

Create ServiceDesk API Connection in SSIS

Once you create Zoho OAuth App now its time to call some Zoho API. Let’s create the OAuth Connection first. In the next section we will see how to load data in SQL Server.

1. Right click on Connection Panel and choose Choose “New Connection..”. Select ZS-OAUTH type
   

   Create new SSIS OAuth API Connection Manager

2. Configure your OAuth connection General Tab as below
3. OAuth Provider: Custom
4. OAuth Version: OAuth2
5. Client Id , Client Secret
6. Enter One Scope Per Line (For more info on scope read here) . For write you can change READ to WRITE or use ALL as per the help link.
   

   SDPOnDemand.requests.READ

7. Auth URL : https://accounts.zoho.com/oauth/v2/auth
8. Token URL:  https://accounts.zoho.com/oauth/v2/token
9. Configure Advanced tab as below
10. Enter Callback URL : https://zappysys.com/oauth
11. Enter Extra Attribute for /Auth : access_type=offline&prompt=consent
    This is needed to obtain refresh_token (if you ommit this then only access_token returned which cannot be renewed once expired after 1hr)

    

12. Now go back to general tab and Click Generate Token. Close the Popup and use Full browser instead if you see prompt.
13. Click OK to save the connection UI

Read data from ServiceDesk API in SSIS JSON Source

In this section we will learn how to read data from ServiceDesk using SSIS JSON Source. We will use OAuth connection created in earlier section.

Now lets look at step by step example of reading ManageEngine ServiceDesk data using SSIS (e.g. Requests).

ManageEngine provides JSON API for read action so we will Use SSIS JSON Source for this purpose.

Configure JSON Source

So below steps shows example of reading from ManageEngine Requests API

1. Open SSIS Package
2. Drag data flow task from Control flow SSIS Toolbox
   

   Drag data flow from SSIS Toolbox

3. Go to data flow and drag ZS JSON Source from SSIS Toolbox
   

   Drag and drop SSIS JSON Source (REST API or File)

4. Double click JSON Source to configure. Enter URL as below. You can supply few more parameters. Click Here to get full list of parameters and usage. In below example we are reading records from Leads Module but you can replace Leads to something else (e.g. Accounts). Refer to above help link for available Modules.
   

   https://sdpondemand.manageengine.com/api/v3/requests

5. Check Use Credentials. Select OAuth connection manager we created in the previous section.
6. Add the Accept header in HTTP Headers.
   

   Accept: application/vnd.manageengine.sdp.v3+json

7. Under filter options tab, enter Array Filter as  

   $.requests[*]

8. Now click Preview to see your data
9. Click OK to save UI. In the next section, we will see how to setup pagination and incremental extract of Servicedesk records. After that we will see how to load data into SQL Server Table.

Read ManageEngine ServiceDesk Plus data in SSIS (Cloud Version) – Zoho API

ServiceDesk API Pagination (Read all Requests)

Just like most API ServiceDesk also uses pagination. See below to learn how to configure pagination in JSON Source for ServiceDesk API.

1. Double click JSON Source
2. Change URL to include two things (Page size (how many rows per page and start row). Note <%page%> must match with Page Num Indicator you going enter in the next screen
   

   https://sdpondemand.manageengine.com/api/v3/requests?input_data={"list_info":{"row_count":100,"start_index":<%page%>}}

3. Go to Pagination Tab and select URL Path Mode
4. For Page Num Indicator : Enter

   <%page%>

5. For Increment By : Enter

   100

6. Page Start Number select New variable call it StartIndex and start default value 1

Once you configure this way now your API will return all records until last page is detected. For more information on pagination check this article

How to paginate ManageEngine ServiceDesk API

Loading ServiceDesk data into SQL Server Table / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Write data to ServiceDesk using OAuth 2.0

If you like to load data from any source (i.e. SQL Server) to Zoho CRM then you can use Records API (POST). Use a combination of the following Transforms  / Components. Later in this article we have some more examples (Older API) which may give you one more ways to load data into Zoho (Use of Template Transform).

Basically you have to build POST Body using JSON Generator Transform or Template Transform and then use that data in Web API Destination to call POST API request to push data to ServiceDesk. Basically screnshot below is just an example using some other Zoho API but you get the idea.

	Web API Destination
	JSON Generator Transform
	JSON Parser Transform

Create JSON for POST request (Single dataset pattern)

Insert data to Zoho Table using SSIS (Lead, Account Modules) – Bulk Load using POST API call

Write data to ServiceDesk using SSIS (Import SQL Server Table to ServiceDesk)

Now you know how to read data from Zoho CRM using JSON API Source next step is to load data to Zoho CRM. For writing data we will use following three components

	Web API Destination
	Template Transform
	XML Parser Transform

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Conclusion

Manageengine  Servicedesk API provides a great way to automate data read/write operations. However, to call Service API  you have to use SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Servicedesk API call via a  simple drag and drop approach without coding. Try  SSIS PowerPack for free and call virtually any REST API in a few clicks.

The post Read/Write ManageEngine ServiceDesk Data in SSIS (Zoho API) appeared first on ZappySys Blog.

In our previous post, we saw how to read Salesforce data using SSIS Salesforce Source (Which uses SOAP API under the hood). This may be slow if you have millions of rows because Salesforce Source paginates every 2000 rows, so it’s not a good option to read millions of rows to iterate on a large Salesforce dataset (Table or Query). To increase the read speed, you can use BULK API. This article explains how to read salesforce data using BULK API and increase read speed.

NEW Version (v5.5+)

In the new version (v5.5), we have added an option to query in Bulk Mode. This means if you have millions of Salesforce object rows to read, no worries; it may be very fast in the new version. Previously, it used to take a long time because it read in batches of 2000 rows in each request. With the Bulk API, the batch size is really large.

Here is how to set Salesforce Bulk API Mode in SELECT query (Read operation).

1. Install new version of SSIS PowerPack. Make sure it is v5.5 or higher.
2. Open existing Salesforce Source UI
3. Make sure you check [Use Bulk API] option like below. if you do not see this option then most likely you are using older version.
   -OR-
   Just add #bulkmode prefix in the SQL query like below, it will use Bulk API
   

   Bulk API mode for SSIS Salesforce Source (Read Large amount of data using SOQL Query)

OLD Version

If you are using an older version of SSIS Power Pack (v5.4.0 or older), you can follow the steps mentioned in the following section.

We will see how to query using Salesforce Bulk API in SSIS. It involves three simple steps like the one below.

1. Call BULK API – Submit SOQL Query and receive a JobId back (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

2. Keep checking Job Status Until Status is JobComplete (Use SSIS Salesforce API Task)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

3. Read Query Result  (Use SSIS CSV Source)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Now, let’s look at everything in detail.

Obtain Salesforce Credentials

The first step to call Salesforce API is to obtain credentials (security token or OAuth ClientId /Secret)

Method#1 – Check this article to use UserId/ Password / SecretToken
Method#2 – Check this article to use the OAuth Connection Option

Create Salesforce Connection

Once we have Salesforce Credentials, we can create a new Salesforce Connection (You can do this from Task UI or the below way)

1. Open Visual Studio ( Assuming you already have SSIS installed along with ZappySys PowerPack )
2. Create a New SSIS Package Project.
3. Right-click on the Connection Manager panel and click on New Connection.
   
4. Select ZS-SALESFORCE Connection Manager from the Connection Managers list and Click on the Add Button.
   
5. Now in Connection Manager UI, Enter your SalesForce Credentials (OAuth Client Id/. Secret –or— User Name, Password and Security Token), and leave all other properties as it is.
   

Call Create Job – Retrieve / Save JobId

Once the connection is created, we can create our first step. Basically, we will call this API to create Job (Bulk Read)

1. Here, in Visual Studio, drag and drop the ZS Salesforce API Task on the design surface.
2. Double click the Task and configure Request Settings Tab like below
   – Select Salesforce Connection (Created in the previous section)
   – Check Use direct URL
   – Enter URL like below (Replace {your-instance} with your own instance (i.e. na9) . For more information on how to find out my
   salesforce instance id check this article
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query

   – Request Method POST
   – Enter Body like below (You can use your own SOQL query). Keep in mind SQOL must be encoded for JSON format. This means the entire SQL must be in one line (replace new lines with \r\n , quotes with \” and tabs with \t)
   For example, to execute SELECT Id, Name FROM Accounts, you can write something like the one below in BODY.
   

   {
     "operation": "query",
     "query": "SELECT Id, Name\r\nFROM Account"
   }

   The above is for below (with a new line)
   

   SELECT Id, Name
   FROM Account

   

   Create Job – Call Bulk API to Read / Query using SOQL

3. Now Click Test Request (Copy the JobId which we will use in the Variable for Designtime Testing)
4. On the Response Tab, configure it like the below.
   – Select Response content Type = Json
   – Enter $.id as the expression
   – Check Save response option
   – Select Save to Variable and click New to create a variable. Name it JobId and enter a sample JobId value (copied from the previous step)
   

   Extract Salesforce JobId save to SSIS Variable

Do Status Check – Look for JobComplete

Once we create a job and store JobId in the variable we have to keep checking Job status every X seconds. Don’t worry you don’t have to use Loop or timer control for that. ZappySys made it so simple to check such thing without Loop /Timer.

1. Drag and drop the ZS Salesforce API Task in the design surface.
2. Double click and configure like below. URL will be different this time. Use variable as below and replace your instance ID
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}

   

   Check Salesforce Job Status Periodically (Without Loop + Timer Controls)

3. Now go to the Status Check Tab and configure it, as shown below. It will keep looking for the words below in the response to decide whether it is a Success or Failure. If those words are not found, they will be checked every 5 seconds (Maximum 1200 secs).
   – Check Enable Status Check option
   – For Success Value – Enter JobComplete
   -Check Fail task option
   – Enter Cancel|Failed|Abort in the text box
   

   Configure Job Status Check Loop using Salesforce API Task

Read data (Compressed CSV Format)

Last step is read the result from completed Salesforce Job. Its in CSV format so we will use Advanced SSIS CSV Source which supports API connection to Salesforce. For that, we will use the same Salesforce API Task used before.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
2. Double click on the DataFlow task to see DataFlow designer surface.
3. Here, In Visual Studio, drag and drop the ZS CSV Source (Web API or File) in the design panel
   
4. Double click on ZS CSV Source (Web API or File) Configure it like below.
   – Enter URL to read result like this (You can supply OPTIONAL maxRecords parameters to control how much data comes in a single response)
   

   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results
   --OR--
   https://{your-instance}.salesforce.com/services/data/v50.0/jobs/query/{{User::JobId}}/results?maxRecords=50000

   – Check Use Credentials and select Salesforce Connection
   – Click Preview (assuming JobId variable has a valid completed JobId)
   

   Read Salesforce Bulk API Job Result – Using SSIS CSV / API Source

5. Now, set up Pagination as below to read all records using an automated loop. You will need VERSION Released after Jun 20, 2022 to use Regular Expression feature in Header Value extractionBy default, Salesforce doesn’t send all records in response so we need to setup pagination. In below example we are reading Sforce-Locator attribute from Response and sending it to the next URL as locator=<continue-token> value
   – Choose Response Header Contains Continue Token 
   – Enter below expression in Page Num Indicator
   

   locator=Sforce-Locator(^((?!null\b).)*$)

   Basically, the above expression will keep looping until we find Sforce-Locator : null in the response header.
   

   Pagination for Salesforce Bulk Query API (Get Header Value from Sforce-Locator)

6. Click OK to save and now run entire package like below.

Query Salesforce Data using Bulk API in SSIS (read using SOQL query in Bulk mode / large dataset)

Conclusion

In this article we saw how to use various ZappySys Components / Task to achieve complex Salesforce API calls to read Query data using Bulk API. Download SSIS PowerPack and try yourself and also explore 70+ other tasks /components not discussed in this article.

The post How to query salesforce data using Bulk API in SSIS appeared first on ZappySys Blog.

In this section, we will discuss how to register OAuth App to call Salesforce API using Client ID / Secret rather than using your User Id / Password. Once you follow these steps below to register your Salesforce App (OAuth App), at the end you will get a Client ID (sometimes referred to as App Id) and Client Secret (or App Secret). These two pieces can be used later on to call Salesforce API using OAuth. So let's get started.

Register Salesforce App (OAuth)

1. Login into your account and go to Setup as shown below (click on gear icon at the top right corner):
2. Then in the search bar input "app" and open App Manager:
3. Then click New Connected App to create a new OAuth App: NOTE: In some version, it may say New External Client App

   Look for one of the following buttons:

   • New Connected App (Classic name)

   • New External Client App (Lightning name)

   
4. Give your App a name and: - Enable OAuth Settings - Set callback URL to following two URLs (one in each line) https://login.salesforce.com/services/oauth2/success https://zappysys.com/oauth - Add an OAuth scope you want or simply add "Full access". - Add "Perform requests on your behalf at any time (offline_token)" scope to get a refresh token so that you don't have to re-authenticate each time requesting an access token: IMPORTANT: You can use "Access and manage your data (api)" if you do not like to grant full permission to this app but make sure you request same permission from the caller app too  (Example on ZappySys Salesforce Connection UI -> Advanced Tab -> Set scope as "api offline_access" rather than "full offline_access"
5. Click "Save".

Get Client Id and Client Secret for Salesforce Connected App

Once the App is created, we need to find out the App Id / Secret so we can use it for API calls.

1. Go to App Manager
2. Find out the recently created App and click View (click on the small arrow to the right)
3. Once App page is loaded you can Copy Consumer Key and Consumer Secret (click on reveal to see). Also copy anyone Call back URL (We will need this for OAuth Connection > Advanced Tab > Call Back URL or Redirect URL property depending on which ZappySys Product / Driver you are using)
4. That's it, you can now use these 3 pieces to ZappySys OAuth connection UI or on Salesforce Connection UI (new version with OAuth support). The above screenshot shows where to enter these 3 pieces in the New Salesforce Connection. For the OAuth Connection, the redirect URL goes to the Advanced Tab > Callback URL textbox.

Get Client Id and Secret for Salesforce External Client App

If you created a new App by clicking on the New External Client App button, then your App may not be visible in the App Manager Screen. In that case, find it below way. [caption id="attachment_11637" align="alignnone" width="1222"] Get Client Id, Client Secret - For New External Client App[/caption]

Enable Client Credentials Grant (Non-Interactive Run-As)

If you like to enable Non-Interactive mode, just like UserId / Password Authentication, but without actually passing a real UserId or Password on the Connection UI, then you can enable Client Credentials Flow as below. NOTE: Only the New version of ZappySys Salesforce Connection supports Client Credentials Grant. Version released after Nov 1, 2025 supports this grant type. [caption id="attachment_11639" align="alignnone" width="419"] Enable Salesforce Client Credentials Grant (Run As)[/caption]

(OPTIONAL) Configure Salesforce OAuth refresh token validity

By default, Salesforce Refresh Token doesn't expire once you generate. If you want to change the default behavior, follow these steps.

1. Go to "Manage Connected Apps" menu item and press on your App name:
2. Click "Edit Policies" to configure refresh token validity:
3. Set "All users may self-authorize" and "Refresh token is valid until revoked" options:
4. Click "Save".

Using Salesforce Connection with OAuth App in SSIS / ODBC

Once you have Client ID / Secret, you can use it in the Salesforce Connection in SSIS Connector / ODBC Driver for Salesforce like below (Only the new version will have OAuth option)

Troubleshooting Known Issues

Let’s look at some common issues with Salesforce Connection and how to fix them.

OAuth Error: Missing Required Code Challenge

When you click Generate Token button, you may get an error like the one below.

Salesforce OAuth Generate Token Error – Missing Required Code Challenge

Solution

Here is how to fix it.

For Salesforce Connection UI: Go to Advanced Tab => Check Enable PKCE Code Verifier option
–OR–
For OAuth Connection UI: Go to OAuth2 Grant Options Tab => Check Enable Code Verifier Option

Enable PKCE code verifier option – OAuth Connection for Salesforce.

The post How to register Salesforce App and obtain Client ID / Secret (for API Call / OAuth) appeared first on ZappySys Blog.

In this article, we will show you step by step approach on how to register an Azure AD (Active Directory) app to access Dynamics CRM 365 API (also same for CDS / Dataverse API). You might have heard that Microsoft is deprecating an older way of authenticating (i.e. User ID / Password-based WS-Trust Authentication) on April 2022. You must update your jobs / code to use OAuth 2.0 by April 2022 for any CRM related API calls.

In other words rather than User Id / password now you can use more secure way by using OAuth rather than WS-Trust.

Going forward Microsoft wants you to use OAuth Token rather than user name/password to call Dynamics CRM 365 API programmatically.

Here is the official link which describes the dates for WS-Trust sunset

Register App for CRM / CDS / Dataverse in Azure / Active Directory

The first step is to register your app in Azure Portal under Active Directory using the below steps.

1. Ask your Active Directory Admin to login to Azure Portal and navigate to > Active Directory Blade.
   OR
   You can access it by searching “Azure Active directory” like below.
   

   Open Azure Active Directory

2. Now click on the App Registrations icon and click New App Registration
   
   

   New App Registrations under Azure Active Directory

3. Now next step is to give your App name to identify easily. Select App is used for your Org only option (Default) and last enter Redirect URL as below (This URL is used for 3-legged Authorization Code Grant only). We prefer you enter a URL which is OAuth friendly (If you don’t know one Use

   https://zappysys.com/oauth

     this is recommended URL if you working with ZappySys Products)
   

   Azure AD App Registration – Single / Multi-Tenant with Redirect URL (i.e. Call back)

    

4. Now once App is created you can find the Application (Client) Id from overview page
   

   Azure AD App – Find Application / Client Id on overview page

Add API Permissions for the App

Once App is created we need to Add Permission. Basically, we will set which style of APIs can be called by this App and what Actions can be performed.

1. To add API Permissions for this AD App click on API Permissions and then click on Add Permission
   

   Add API Permissions for Azure AD App

2. Find the Dynamics CRM icon and click on it
   

   Add API Permission for Dynamics CRM API access (Azure AD App)

3. Now you will see just one permission for Dynamics CRM (user_impersonation). As we said earlier for Dynamics CRM API security is controlled by User Roles in CRM Portal itself which will see in the next section. For now, just choose Delegated Permissions,  check user_impersonation and click Add Permissions
   

   Select API Permissions for Dynamics CRM App (Azure AD App)

4. Now the last step in Adding API permissions is Grant Admin Consent. If you are logged in as an Azure AD User with the correct admin role then no worries else you have to ask your Azure AD admin to consent for you so this App can be used to read/write data from CRM.

Create Client Secret

Once App is created next step is to generate a Client Secret. Perform the following steps to create a secret that you can use when you call API

1. Click on Certificates and Secrets and then New Secret
   

   Azure AD App Create Client Secret / Set Name, Expiration Date

2. Enter Name for Secret (You can give some friendly name if you create multiple secrets to control multiple processes). For example SecretForTesting, SecretForProduction etc.Select a longer expiration date (i.e. 24 months) if you don’t like to change secret often in production jobs else choose shorter expiration (i.e. 6 Months or Custom).
3. Once you Create a Secret make sure you copy it to a secure location. Once the page is closed it won’t appear again.
   

   Copy Client Secret for Azure AD App

Associate App User / Set Roles in CRM 365 / Dataverse Environment

Once we have App Id and Client Secret we can go to CRM Portal to add our App as a valid CRM User. And the last step will be to attach some role(s) to App User (See Next section).

Method-1 (New way)

Here are the steps to configure User / Security Roles in New account. Below steps will give you access to Dataverse Entities via API Call. To see all Dataverse Entity you can log in to https://make.powerapps.com/ and click Tables

1. Click on the below link to login to new portal to manage CRM /Dataverse environment
   https://admin.powerplatform.microsoft.com/environments
2. Create an Application User using these steps (When prompted choose the same App we created in the previous section)
3. Now Go to App Users Screen and Edit Security Roles using the below steps
   

   See available CRM 365 Online / Datavese Environments

   

   See available CRM 365 Online / Datavese Application Users

   

   Edit Security Roles for CRM 365 Online / Datavese Application Users

Method-2 (Legacy)

If you prefer to use an older interface (and it’s still available to you) then you can use the below steps to perform the same thing we described in the previous section.

NOTE: Microsoft may soon retire the older interface and force to use the previous way only.

1. To Add our Azure AD App as CRM Application user goto your CRM Portal. Click on Gear icon for Settings > Advanced Settings
   

   Dynamics CRM Advanced Settings

2. Click on Settings dropdown > Security (Under System Column)
   

   Dynamics CRM Security Settings (Under Advanced Settings – System)

3. On the security, screen click on the Users icon
   
   

   Manage Dynamics CRM Users (Add, Edit, Delete)

4. Once User Screen shows up click on the User Type Dropdown and choose Application Users and click New
   

   Add new Application User in Dynamics CRM

5. Once User screen shows up. Enter your Application Id (obtained from the Azure AD App > Overview screen – app created earlier). Click Save and you will notice all other fields auto populates.
   

   Adding Azure AD App as Dynamics CRM 365 – Application User

6. And the last step is to Attach some CRM Role to the new Application User.
   For demo purposes we will choose “System Admin” but for production use, you should grant correct permission carefully. There are many inbuilt roles you can choose from.
   

   Associate Security Role(s) for Dynamics CRM Application User

Find your Organization Id for CRM / Dataverse API calls

Now find your Org Id before we can call CRM / Dataverse APIs in the next section.

To find your Org Id you can perform the following steps.

Method-1 (New way – Dataverse Portal)

1. Go to https://admin.powerplatform.microsoft.com/
2. Click on the Environment you like to access
3. On the Details, page copy your Org Id from the Environment URL For example URL might look like below… You just need to copy the first part (Use this on CRM Connection > Organization Field)
   org8640f7e7.crm.dynamics.com
   

   How to find Organization Id for Microsoft Dataverse / CDS / CRM 365 API calls

Method-2 (Old way)

Another way to find your Organization Id is from CRM Portal

If you have multiple environments (i.e. Prod, Sandbox) then each environment might have its own unique URL)

1. To find your API Service URL, Click on Gear Icon
2. Choose Advanced Settings
3. Click down arrow on Settings
4. Click Customization
5. Click Developer Resource Link
6. On this page now you can see various URLs. For examples used in this article, we need to get is Service Root URL for Web API
   

   How to Find Organization Id / Web API URL / SOAP API Service URL (Unique Environment Instance URL)

Using Client ID (i.e. App Id) and Client Secret to Call Dynamics CRM 365 API / CDS / Dataverse

That’s it now we are ready to use this App Id and Client Secret to make CRM API calls.

To confirm if it’s working you can create a Dynamics CRM Connection (using SSIS PowerPack ). Choose OAuth option on Dynamics CRM Connection

Connect to Dynamics CRM / Dataverse / PowerApps API using OAuth (Client ID /Secret)

Conclusion

In this article, we saw how to Register Azure AD App to call various Microsoft APIs (for this article we focused mainly on Dynamics CRM 365 API only)

The post How to register App for Dynamics CRM 365 / CDS / Dataverse API (Azure AD / OAuth) appeared first on ZappySys Blog.

OneDrive allows to store files on the cloud and and we have seen growing requirements in downloading/Uploading files from there. This post will provide step by step details of how to Upload/Download files from OneDrive.

Register Microsoft App for OAuth Authentication

In this section, you will learn how to register a custom app in Microsoft Azure portal that will allow access to the OneDrive. So, let’s get started –

1. Log into Microsoft Azure portal to register a custom app.
2. Register a new application by clicking New Registration link.
   

   New App Registration in Azure portal

3. Provide the name of the custom app and who can access the app in the organization.
   

   Register an OneDrive App

4. Go to the App overview and add a Redirect URL.
   

   Add a Redirect URL

5. Click on “Add a Platform” under Platform Configuration section and then select “Web” under Web applications section to enter a Redirect URL.
   

   https://zappysys.com/oauth

   

   Redirect URL

6. Create a Client Secret key which will be used to Authenticate the custom Azure app.
   

   Add a Client Secret

   

   Secret Key Expiration Period

   

   Specify Secret Key

   Note – Take a note of Client Secret, it will be required while configuring OAuth connection in the SSIS later.
7. Add API Permissions for OneDrive API.
   

   OneDrive API Permissions

8. Select the following permissions from the Delegated Permissions section.
   

   User.ReadBasic.All
   Files.Read
   offline_access

   

   Select Delegated Permissions

9. Take note of Client ID, it will be required while configuring OAuth connection in the SSIS later.
   

   Client ID

Get OneDrive File ID

In this section, you will learn how to get the File ID of the CSV file through OneDrive API. So, let’s get started –

1. Go to the Microsoft Graph Explorer.
2. Sign in to Graph Explorer.
   

   SSIS GraphExplorer SignIn

3. Run below OneDrive API to get a list of the files from the OneDrive.
   

   https://graph.microsoft.com/v1.0/me/drive/root/children

   

   Microsoft Graph Explorer

   NOTE: Take a note of File ID of the CSV file. For this example, we will download invoices.csv file.

Create OAuth Connection in SSIS

In this section, you will learn how to create an OAuth connection in SSIS to an Azure custom app. So, let’s get started –

1. Create a new ZS-OAuth connection in SSIS.
   

   ZappySys OAuth Connection in SSIS

2. Configure OAuth connection with following settings.

   Client ID	Provide Client ID from App Registration steps
   Client Secret	Provide Client Secret from App Registration steps
   Authorization URL	https://login.microsoftonline.com/organizations/oauth2/v2.0/authorize
   Access Token URL	https://login.microsoftonline.com/organizations/oauth2/v2.0/token
   Scope/Permissions	User.ReadBasic.All Files.Read.All offline_access

   

   OAuth Connection Configuration

3. Go to the Advanced tab and provide Return URL.
   

   https://zappysys.com/oauth

   

   Add Redirect URL

4. Click on Generate Token will give a login screen which will allow to generate access and refresh token for the app.
5. Approve the requested permissions bu Accepting it.
   

   Approve Permissions Requested

6. Token has been generate successfully.
   

   Refresh Token Successful

7. Let’s test the connection to make sure it is working fine.
   

   Connection Successful

Download CSV File Through OneDrive REST API in SSIS

In this section, you will learn how to download a CSV file through OneDrive API in SSIS and save it into a CSV file.

1. Here is the OneDrive API which allows to read content of the file.
   

   https://graph.microsoft.com/v1.0/me/drive/Items/<File ID>/content

2. Replace <File ID> in above step#1 with the file ID that we have got in the Get OneDrive File ID section. Final URL will look something like below.
   

   https://graph.microsoft.com/v1.0/me/drive/Items/01W7L3VCUBS27ILUQKENAK4LQGGTJ2R3TT/content

3. Let’s create a SSIS package with Data Flow Task from SSIS Toolbox.
4. Replace <File ID> in above step#1 with the file ID that we have got in the Get OneDrive File ID section. Final URL will look something like below.
   

   https://graph.microsoft.com/v1.0/me/drive/Items/01W7L3VCWMCGPYZLM2IRAYWROSH4XMQT7A/content

5. Configure REST API task as follows.
   

   Binary File Configuration in ZappySys REST API Task

6. Configure the responses setting as below.
   

   Binary File Configuration in ZappySys REST API Task

   NOTE: Ensure that the Treat response as the binary checkbox is checked otherwise the file will be downloaded but corrupted as zip/mp3 etc. file format is binary.
7. Check if the request is made correctly by clicking Test Request/Response.
   

   Example of Testing the Request

8. Let’s have a look at the folder where the file has been downloaded. Here is an example –
   

   Downloaded File Example

Download a Text File Through OneDrive API in SSIS

Here is another technique that can be used to download a CSV/text file through OneDrive API without knowing it’s file ID but file name. Please check this link for more details.

Download Excel Sheets Through OneDrive API in SSIS

JSON Source also supports reading API response which returns 2D arrays. Such as Office Excel Spreadsheet API to read excel data. Check this blog post to learn a similar scenario to parse 2D arrays. There are few other options for parsing 2D arrays which are documented here.

https://graph.microsoft.com/v1.0/me/drive/items/01CYZLFJDYxxxxxxx/workbook/worksheets('test')/range(address='A1:B200')

Upload a File Through OneDrive API in SSIS

In this section, you will learn how to upload a file through OneDrive API in SSIS. Uploading a file has few additional steps and has been documented separately. Please check this link for more details.

Delete a File Through OneDrive API in SSIS

In this section, you will learn how to delete a file from OneDrive through OneDrive APIs in SSIS.

1. Add a REST API task and configure as follows. Make sure to use the File ID of the file which needs to be deleted from OneDrive. Example URL –
   

   https://graph.microsoft.com/v1.0/me/drive/Items/01W7L3VCSEADQ457W54RBI3I47YGFJOD2I

   

   Delete OneDrive File

2. Execute the task and you will have file deleted from OneDrive.

Conclusion

In this article, we have learned how to Upload/Download/Delete data/file from OneDrive API in SSIS. We used SSIS REST API Task to extract data through OneDrive API in SSIS. Download SSIS PowerPack to try many other automation scenarios not discussed in this article.

The post How to call OneDrive API in SSIS (Upload, Download) appeared first on ZappySys Blog.

In our previous posts we saw how to read and write Salesforce Data using SSIS. In this post we will see how to read Salesforce Metadata in SSIS using JSON Source.

Step-By-Step

In this section you will learn how to use ZappySys JSON Source Adapter to extract data from Salesforce REST API (JSON Format). You can call pretty much Any REST API to fetch data from salesforce but our example for this post would be read metadata (i.e. table list) from salesforce (we will call sobjects to get all table information) .

So lets get started.

1. In case you missed the previous section, You need to Download and Install SSIS ZappySys PowerPack.
2. Once you finished first step, Open Visual Studio and Create New SSIS Package Project.
3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
4. Double click on the Data Flow task to see Data Flow designer surface.
5. From the SSIS toolbox drag and drop JSON Source on the Data Flow designer surface.
   
6. Now double click JSON Source enter any valid REST API URL like below. Make sure you used correct instance name. Using wrong instance name might slow down connection or fail your request. If you are not sure then just login to salesforce portal using credentials you like to use and observe browser URL.. It will have instance number.
   

   https://na34.salesforce.com/services/data/v20.0/sobjects/Account/describe/

7. Check Use Connection and select New ZS-Salesforce Connection from the list.
   

   Create new REST API Connection in SSIS (For JSON, XML or CSV Source)

8. Configure your salesforce connection
   
9. Select or type filter. For example in this case you will type 

   $.sobjects[*]

10. Click OK to Save UI and attach your source to some target like OLEDB Destination and run your data flow to test.
11. Here is full UI after all configurations.
    

    Read Salesforce Metadata in SSIS (Get Table List Example – SOObject REST API Call)

How to read Salesforce Table Fields Metadata

If you have need to read fields and its information for tables then use below steps

1. Use almost identical steps as previous section. Except two things. Our URL and Filter would be different.
2. Configure your API URL as below
   

   https://YOUR-INSTANCE.salesforce.com/services/data/v20.0/sobjects/YOUR-TABLE/describe/

   Example:
   

   https://demo.salesforce.com/services/data/v20.0/sobjects/Account/describe/

3. Configure Filter as a 

   $.fields[*]

Get Salesforce Table Fields Metadata using REST API call in SSIS

The post How to read Salesforce Metadata in SSIS using REST API (JSON Source) appeared first on ZappySys Blog.

In our previous post, we discovered how to read and write MongoDB data using ZappySys MongoDB Source and Destination SSIS connectors. However, it was all about connecting to the on-premises hosted version of MongoDB. In this post, we will focus on connecting specifically to the Replica Set Cluster in MongoDB Atlas (a cloud database service by MongoDB). After you complete the steps listed below, you will be able to get and push data using the same MongoDB SSIS connectors, but this time — in MongoDB Atlas. Let’s not waste our precious time and begin!

MongoDB Atlas connection setup in SSIS

In this article, we are going to demonstrate how to create the MongoDB Connection Manager, which is used by MongoDB ExecuteSQL Task, MongoDB Source, and MongoDB Destination SSIS connectors. Let’s start by following these simple steps:

1. First, download and install SSIS PowerPack.
2. Once you installed SSIS PowerPack, open Visual Studio, and create a new SSIS project with a new SSIS package.
3. Then open the created package and right-click on the Connection Managers panel to create a new MongoDB connection. When a context menu appears, select the New Connection… menu item:
   
4. Next, select the ZS-MONGODB option from the connection manager list and click Add… button:
   
5. This will open the MongoDB Connection Manager configuration window. For now, let’s leave it open, and first go to your MongoDB Atlas account to get all MongoDB Atlas Cluster instance information for MongoDB Connection Manager configuration.
6. Once you are on the MongoDB Atlas landing page, continue by selecting the cluster you want to connect to (make sure you white-list your IP first). After the window appears, select the Compass option:
   

   Connecting to MongoDB Atlas Replica Set Cluster

7. Then proceed by simply copying and pasting the configuration information into the MongoDB Connection Manager we left to rest for a while. Move on by filling in your database credentials and the database you are connecting to. Also, don’t forget to enable SSL. Just patiently follow the numbers in orange circles below and you will be done faster than you think!
   

   Configuring MongoDB Atlas connection in SSIS by setting the host, user name, password, and database options

   

   Specifying MongoDB connection string options (i.e. replicaSet and authSource options)

   

   Enabling SSL Connection in MongoDB Connection Manager in SSIS (for encrypted data transfer)

8. Finally, press the Test Connection button to test the connection. If the test is successful, you can move on and start performing data operations on the MongoDB collections.

That’s it — you can now use the MongoDB ExecuteSQL Task, MongoDB Source, and MongoDB Destination as described in the below articles:

• How to call MongoDB JavaScript using SSIS
  
• How to read MongoDB data from an array (extract nested subdocuments)
• How to write MongoDB Aggregation Queries in SSIS (Group By)
• How to load SQL data into MongoDB (Insert, Upsert, Delete, Update)
• How to update MongoDB Array Items using SSIS

The post How to connect to MongoDB Atlas Replica Set Cluster in SSIS appeared first on ZappySys Blog.