dynamics crm Archives | ZappySys Blog

How to register App for Dynamics CRM 365 / CDS / Dataverse API (Azure AD / OAuth)

ZappySys — Mon, 29 Nov 2021 21:20:53 +0000

Introduction

In this article, we will show you step by step approach on how to register an Azure AD (Active Directory) app to access Dynamics CRM 365 API (also same for CDS / Dataverse API). You might have heard that Microsoft is deprecating an older way of authenticating (i.e. User ID / Password-based WS-Trust Authentication) on April 2022. You must update your jobs / code to use OAuth 2.0 by April 2022 for any CRM related API calls.

In other words rather than User Id / password now you can use more secure way by using OAuth rather than WS-Trust.

Going forward Microsoft wants you to use OAuth Token rather than user name/password to call Dynamics CRM 365 API programmatically.

Here is the official link which describes the dates for WS-Trust sunset

Register App for CRM / CDS / Dataverse in Azure / Active Directory

The first step is to register your app in Azure Portal under Active Directory using the below steps.

Ask your Active Directory Admin to login to Azure Portal and navigate to > Active Directory Blade.
OR
You can access it by searching “Azure Active directory” like below.

Open Azure Active Directory
Now click on the App Registrations icon and click New App Registration

New App Registrations under Azure Active Directory
Now next step is to give your App name to identify easily. Select App is used for your Org only option (Default) and last enter Redirect URL as below (This URL is used for 3-legged Authorization Code Grant only). We prefer you enter a URL which is OAuth friendly (If you don’t know one Use
```
https://zappysys.com/oauth
```
this is recommended URL if you working with ZappySys Products)
Azure AD App Registration – Single / Multi-Tenant with Redirect URL (i.e. Call back)
Now once App is created you can find the Application (Client) Id from overview page
Azure AD App – Find Application / Client Id on overview page

Add API Permissions for the App

Once App is created we need to Add Permission. Basically, we will set which style of APIs can be called by this App and what Actions can be performed.

For Apps specific to CRM / CDS / Dataverse API, actual app permissions are controlled by User Roles. Check later in this article how to associate one or more Roles to this AD App so it can perform certation actions via API call

To add API Permissions for this AD App click on API Permissions and then click on Add Permission
Add API Permissions for Azure AD App
Find the Dynamics CRM icon and click on it
Add API Permission for Dynamics CRM API access (Azure AD App)
Now you will see just one permission for Dynamics CRM (user_impersonation). As we said earlier for Dynamics CRM API security is controlled by User Roles in CRM Portal itself which will see in the next section. For now, just choose Delegated Permissions, check user_impersonation and click Add Permissions
Select API Permissions for Dynamics CRM App (Azure AD App)
Now the last step in Adding API permissions is Grant Admin Consent. If you are logged in as an Azure AD User with the correct admin role then no worries else you have to ask your Azure AD admin to consent for you so this App can be used to read/write data from CRM.

Create Client Secret

Once App is created next step is to generate a Client Secret. Perform the following steps to create a secret that you can use when you call API

Click on Certificates and Secrets and then New Secret
Azure AD App Create Client Secret / Set Name, Expiration Date
Enter Name for Secret (You can give some friendly name if you create multiple secrets to control multiple processes). For example SecretForTesting, SecretForProduction etc.Select a longer expiration date (i.e. 24 months) if you don’t like to change secret often in production jobs else choose shorter expiration (i.e. 6 Months or Custom).
Once you Create a Secret make sure you copy it to a secure location. Once the page is closed it won’t appear again.
Copy Client Secret for Azure AD App

Associate App User / Set Roles in CRM 365 / Dataverse Environment

Once we have App Id and Client Secret we can go to CRM Portal to add our App as a valid CRM User. And the last step will be to attach some role(s) to App User (See Next section).

Method-1 (New way)

Here are the steps to configure User / Security Roles in New account. Below steps will give you access to Dataverse Entities via API Call. To see all Dataverse Entity you can log in to https://make.powerapps.com/ and click Tables

Click on the below link to login to new portal to manage CRM /Dataverse environment
https://admin.powerplatform.microsoft.com/environments
Create an Application User using these steps (When prompted choose the same App we created in the previous section)
Now Go to App Users Screen and Edit Security Roles using the below steps
See available CRM 365 Online / Datavese Environments

See available CRM 365 Online / Datavese Application Users

Edit Security Roles for CRM 365 Online / Datavese Application Users

Method-2 (Legacy)

If you prefer to use an older interface (and it’s still available to you) then you can use the below steps to perform the same thing we described in the previous section.

NOTE: Microsoft may soon retire the older interface and force to use the previous way only.

To Add our Azure AD App as CRM Application user goto your CRM Portal. Click on Gear icon for Settings > Advanced Settings
Dynamics CRM Advanced Settings
Click on Settings dropdown > Security (Under System Column)
Dynamics CRM Security Settings (Under Advanced Settings – System)
On the security, screen click on the Users icon

Manage Dynamics CRM Users (Add, Edit, Delete)
Once User Screen shows up click on the User Type Dropdown and choose Application Users and click New
Add new Application User in Dynamics CRM
Once User screen shows up. Enter your Application Id (obtained from the Azure AD App > Overview screen – app created earlier). Click Save and you will notice all other fields auto populates.
Adding Azure AD App as Dynamics CRM 365 – Application User
And the last step is to Attach some CRM Role to the new Application User.
For demo purposes we will choose “System Admin” but for production use, you should grant correct permission carefully. There are many inbuilt roles you can choose from.

Associate Security Role(s) for Dynamics CRM Application User

Find your Organization Id for CRM / Dataverse API calls

Now find your Org Id before we can call CRM / Dataverse APIs in the next section.

To find your Org Id you can perform the following steps.

Method-1 (New way – Dataverse Portal)

Go to https://admin.powerplatform.microsoft.com/
Click on the Environment you like to access
On the Details, page copy your Org Id from the Environment URL For example URL might look like below… You just need to copy the first part (Use this on CRM Connection > Organization Field)
org8640f7e7.crm.dynamics.com

How to find Organization Id for Microsoft Dataverse / CDS / CRM 365 API calls

Method-2 (Old way)

Another way to find your Organization Id is from CRM Portal

If you have multiple environments (i.e. Prod, Sandbox) then each environment might have its own unique URL)

To find your API Service URL, Click on Gear Icon
Choose Advanced Settings
Click down arrow on Settings
Click Customization
Click Developer Resource Link
On this page now you can see various URLs. For examples used in this article, we need to get is Service Root URL for Web API
How to Find Organization Id / Web API URL / SOAP API Service URL (Unique Environment Instance URL)

Using Client ID (i.e. App Id) and Client Secret to Call Dynamics CRM 365 API / CDS / Dataverse

That’s it now we are ready to use this App Id and Client Secret to make CRM API calls.

To confirm if it’s working you can create a Dynamics CRM Connection (using SSIS PowerPack ). Choose OAuth option on Dynamics CRM Connection

Connect to Dynamics CRM / Dataverse / PowerApps API using OAuth (Client ID /Secret)

Conclusion

In this article, we saw how to Register Azure AD App to call various Microsoft APIs (for this article we focused mainly on Dynamics CRM 365 API only)

The post How to register App for Dynamics CRM 365 / CDS / Dataverse API (Azure AD / OAuth) appeared first on ZappySys Blog.

How to read Microsoft Dynamics 365 OData API data in SSIS (CRM Web API)

ZappySys — Sat, 25 May 2019 10:28:13 +0000

Introduction

In our previous article, we talked about how to read from Dynamics CRM using easy to use SSIS Dynamics CRM Source connector. In this post, you will learn how to use SSIS JSON Source (for REST API) instead of CRM Source. Using SSIS JSON Source you can read virtually any REST API or JSON file data including Dynamics CRM / Dataverse. For this article, our goal is to learn How to read Microsoft Dynamics 365 data using Web API (OData API) in SSIS. We will use drag and drop approach without any coding.

Why use Web API (OAuth) for Dynamics 365 CRM?

Microsoft has announced depreciation of older CRM API (SOAP API with WS-Trust Authentication) which means they are no longer going to support the older style of authentication so no option but to switch to newer authentication / API.

In the older approach (SOAP API with WS-Trust Authentication) you must pass your User Name and Password to authenticate, also if MFA / 2FA (Multi-Factored Auth) is turned on by organization policy you cannot use your user id / password in API call. In new authentication (i.e. OAuth) you have to use Client Id and Client Secret only.

Now let’s look at the step by step approach to call Dynamics 365 REST API.

Step-1 : Register Dynamics CRM / Dataverse App obtain App Id / Client Secret

The very first step to access Microsoft Dynamics CRM 365 API is to obtain Application Id (also referred as Client Id) and Client Secret. We will use this information later in this article (on OAuth connection UI).

Step-2 : Read Microsoft Dynamics 365 API data in SSIS using JSON Source

Let’s start with an example. We use SSIS JSON Source Task to make calls to Microsoft Dynamics 365 API, we will load Dynamics 365 Leads data to SQL Server.

Please refer to this link for more information: Microsoft Dynamics 365 Web API Reference

Find out API Url for your environment (organization url)

Once we are done with the creation of the Azure AD app to obtain App Id /Secret out next step is to find out the base API URL for your organization / environment.

If you have multiple environments (i.e. Prod, Sandbox) then each environment might have its own unique URL)

To find your API Service URL, Click on Gear Icon
Choose Advanced Settings
Click down arrow on Settings
Click Customization
Click Developer Resource Link
On this page now you can see various URLs. For examples used in this article, we need to get is Service Root URL for Web API

How to Find Organization Id / Web API URL / SOAP API Service URL (Unique Environment Instance URL)

Create SSIS Data Flow to read data from Dynamics CRM / Dataverse

First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.

Drag and Drop SSIS Data Flow Task from SSIS Toolbox
Furthermore, Drag and Drop JSON Source(REST API and File) File Task from the SSIS Toolbox.

Drag and Drop JSON Source Component

Create OAuth Connection for Dynamics CRM / Dataverse

Now lets look at how to create and configure OAuth connection to call Web API for Dynamics CRM / Dataverse

Check Use Credentials Option and select a new ZS-OAuth Connection from the drop-down.

Check Use Credentials
Once OAuth Connection UI opens up. Configure like below on General Tab.
Change OAuth Version to OAuth2
Change Grant Type to Client Credentials
Enter Client Id (Azure AD App Id) (Obtained in the Previous Section)
Enter Client Secret (Obtained in the Previous Section)
Enter Token URL (see below)

Configure OAuth Connection for Azure AD App – Client Credentials Grant (Tenant Specific Token EndPoints)

For Client Credentials Grant
Enter Token URL as https://login.microsoftonline.com//oauth2/token
(You can find Tenant ID on the Overview page of your Azure AD App)

For Default Grant (3-legged – requires User Login Prompt)
Enter Auth URL as https://login.microsoftonline.com/common/oauth2/authorize
Enter Token URL as https://login.microsoftonline.com/common/oauth2/token

Where to find Token Endpoint with Tenant ID in the URL (Azure AD App)
Now go to Advanced Tab and configure like belowEnter Extra Attribute for /auth as
```
resource=https://.api..dynamics.com
```
Client Credentials Grant for Dynamics CRM / Dataverse – API Access (Azure AD App / OAuth)
Click OK to Save connection.

Get all Accounts, Contacts, and Leads using Dynamics 365 API

Now let’s configure API URL from where you like to read data.

For example, if you just use the root URL then it will list all possible entries

Root URL:

https://{{organization-uri}}/api/data/v9.1/

Double click on the Component to Edit the Configurations.
Change Data Format to OData.
In the Web URL Field, you can enter any valid URL for Web API. Please refer to this link of Dynamics 365 Rest API for more information.Example Data URLs
```
https://{{organization-uri}}/api/data/v9.1/
https://{{organization-uri}}/api/data/v9.1/accounts
https://{{organization-uri}}/api/data/v9.1/contacts
https://{{organization-uri}}/api/data/v9.1/leads
```
Here, replace {{organization-uri}} in URL to your Microsoft Dynamics 365 site domain name (Example zappy6.api.crm.dynamics.com Dynamics 365 URL).

Set URL in JSON Source

Preview / Query Dynamics 365 Data

Now select Array filter as
```
$.value[*]
```
(Click Select Filter Button – Choose value node)
Click on the Preview button to get the data.

Preview Data
Finally, click OK to save JSON Source settings.
That’s it we are ready to load Dynamics 365 API data to SQL Server.

Load Microsoft Dynamics 365 API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

Drag and drop the Upsert Destination component from the SSIS Toolbox.
Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

Double-click the Upsert Destination component to open the configuration window.
Under Connection, select an existing target connection or click NEW to create a new connection.
- Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

In the Target Table dropdown, select the table where you want to load data.
Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

Go to the Mappings tab.
Click Auto Map to map source columns to target columns by name.
Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

In this post you have seen how to Read Microsoft Dynamics 365 API data in SSIS. We used JSON Source Component to call Dynamics 365 REST API and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information about the use of Dynamics 365 REST API with our tools:

JSON Source, you can also find Tutorial Video here.
Help File: Documentation of SSIS Json Source
Microsoft Dynamics CRM Web API Reference
Dynamics 365 Web API EntityType Reference

The post How to read Microsoft Dynamics 365 OData API data in SSIS (CRM Web API) appeared first on ZappySys Blog.

Load data into Dynamics CRM using SSIS – Insert, Upsert, Delete, Update

ZappySys — Thu, 05 Jul 2018 19:20:32 +0000

Introduction

In our previous article, we talked about how to read from Dynamics CRM / Dataverse (Dynamics 365 / On-Premises CRM / PowerApp). Now let’s look at how to Load data into Dynamics CRM / Dataverse (Dynamics 365 / On-Premises CRM / PowerApp) using SSIS. We will see a few important aspects of loading data into Dynamics CRM / Dataverse Online (i.e. Dynamics 365) and On-premises / IFD (Internet facing hosted instance).

We will show you how to use SSIS Dynamics CRM Destination connector for Bulk Insert, Update, Delete and Upsert Operations. This article uses Dynamics 365 Online for CRM Integration Examples but concepts are same for other types of installation (e.g. IFD or On-Premises)

Requirements

Make sure you have SSIS Designer installed (i.e. SSDT-BI)
Download and Installed ZappySys SSIS PowerPack which includes an SSIS Dynamics CRM Connectors.
Finally, make sure you have Microsoft Dynamics CRM / Dataverse credentials handy.

That’s it. So let’s get started.

Video Tutorial – CRM Insert, Upsert, Delete, Update

Getting Started

In this section, we will see how to perform bulk insert, update delete and upsert operations using SSIS Dynamics CRM Destination

Configure SSIS Dynamics CRM / Dataverse Connection

The first step to read or write Dynamics CRM / Dataverse data is to make sure we have working Dynamics CRM connection in SSIS.

Create new or open existing SSIS Project in Visual Studio.
Open SSIS Package and Go to SSIS designer.
Right click in the connection manager panel and click New connection:

Create a new SSIS Dynamics CRM / Dataverse Connection
In the connection type selection, Select the ZS-DYNAMICS-CRM Connection.
We will use a CRM Online (Office 365) connection, region, a username and a password and select organization from the dropdown as below and then click Test connection:

Configure Dynamics CRM / Dataverse Connection in SSIS

Bulk Insert – SQL Server data Import into Dynamics CRM / Dataverse using SSIS

Now, let’s see steps to do SQL Server data Import into Dynamics CRM / Dataverse using SSIS. Below steps assume you already have sample data in some SQL Server Table which will be used to create new CRM accounts. You can enter few account names you like to create in the CRM.

Open SSIS Package and drag Data Flow task from Control Flow SSIS Toolbox
Go to data flow designer. Drag and drop OLEDB Source
Configure OLEDB Source to read from some sample data table (e.g. dbo.Accounts). If you want to be lazy and just test quickly you can use ZS CSV Source (Direct Value Mode) instead of OLEDB Source.Sample Data May be like below
```
name,address1_city,address1_country
Flipkart,Bengaluru,India
Microsoft,seattle,USA
Nokia,Espoo,Finland
```
Once you configure Source (Connection, Table Name etc.) click OK to save it.

CSV Source – Sample Accounts for CRM Insert Demo
Drag and drop ZS Dynamics CRM Destination from the toolbox.
Connect Source to your Destination component.
Double click CRM destination to configure it.
On the Connection Managers tab, Select Dynamics CRM / Dataverse connection we created in the previous section.

Select Connection for Dynamics CRM / Dataverse Destination
On the Component Properties Tab Select Table Name.

Select Target Table for Dynamics CRM / Dataverse Insert
On the Mapping Tab map few columns (e.g. name, address1_city, and address1_country)

Configure Target Column Mapping for CRM Data Load
Click OK to save CRM destination.
Right click in the data flow designer, click Execute Task

Execute SSIS Package – Loading Data into Dynamics CRM / Dataverse from CSV / SQL Server

Bulk Upsert – Update or Insert Dynamics CRM / Dataverse Records in SSIS

In the previous section we saw how to Insert New records. However sometimes you want to Upsert (Insert or Update). So if record already exists Update else Insert. In order to achieve this you have to set two additional attributes. For CRM Upsert you have to perform few changes to previous steps

Create Alternate Key for CRM Upsert

To perform Upsert operation you can map Table’s Primary Key or Alternate Key. Primary Key usually starts with table name and ends with “id” (e.g. for account table its accountid ).

Many times Source system don’t have CRM Table Primary Key. In such case you may use Alternate Key in the mapping (JOIN Key). To use Alternate key for Upsert you have first define Alternate Key in CRM Portal. Check this link for exact steps.

Create Alternate Key for Dynamics CRM / Dataverse Online Table (Useful for Update or Upsert Operation)

Configure SSIS CRM Destination for Upsert

Double click CRM destination. On the Component Properties Tab, Change Operation to UpdateOrInsert
Specify UpsertKeyField to either Table’s Primary Key Column Name or Alternate Key Name.

Dynamics CRM / Dataverse Upsert Records using SSIS CRM Destination (Specify Alternate Key / Primary Key)
Now go to Mappings tab and confirm Key field is mapped.

Configure Dynamics CRM / Dataverse Mapping for Upsert in SSIS
Click OK and Run Package to test CRM Upsert Operation.

If you want to use primary key for Upsert or Update Operation and not sure correct column name then you may use ZappySys Dynamics CRM / Dataverse Source FetchXml Query Mode and check Object Browser (Table Info Tab >> PrimaryIdAttribute in the grid).

Primary Key name usually starts with table name and ends with “id” (e.g. for account table its accountid ).

Find Primary Key for Dynamics CRM / Dataverse Table

Handling Inserted / Updated Records Output

Now lets look at how to detect which records got inserted and which records got Update? In Upsert Mode each record can be directed to one of the following three Outputs.

Inserted Records [Blue Arrow]
Updated Records [Blue Arrow]
Records with Error [Red Arrow]

You can select Red or Blue arrow coming out from CRM Destination component. Drag and drop output arrow on any down stream component.

Bulk Update – Modify Dynamics CRM / Dataverse Records in SSIS

If you want to do bulk Update its very same steps as Upsert Operation except you have to set Operation = Update in the properties grid. For Key field you have to set primary key or Alternate Key.

Dynamics CRM / Dataverse Update in SSIS

Bulk Delete – Delete Dynamics CRM / Dataverse Records in SSIS

Bulk delete doesn’t require many settings. All you have to change is Operation = Delete on Component Properties Screen. On the Mapping Screen Map only one column (Primary Key). If you are not sure which one is primary key please refer to the previous section (Upsert Operation).

Delete records from Dynamics CRM / Dataverse Online using SSIS (Primary Key mapping for CRM Bulk Delete)

Some times you may get below errors. If you get this error make sure you delete all child records first before deleting parent. For example if you are deleting account then make sure you have to delete leads / contacts for that account or update leads / contacts so they don’t refer to account being deleted.

Error: 0xC0047062 at Dynamics CRM / Dataverse Bulk Delete, Delete Accounts [2]: System.Exception: Exception: The object you tried to delete is associated with another object and cannot be deleted.
Error Details: 
ApiExceptionSourceKey=Plugin/Microsoft.Crm.Common.ObjectModel.AccountService

Error Handling

Many times you want to ignore bad records and continue processing good records. In such case you can use Dynamics CRM / Dataverse Error Output. Just grad Red arrow from CRM Destination and Drop on target component. Select Redirect Option for Error , Truncation both and Click OK.

Once you set Error Handling your component will continue to run without failing whole data flow. Later you can review errors (if you saving to destination).

Configure Error Handling in SSIS Dynamics CRM / Dataverse Destination

Loading records into Dynamics CRM / Dataverse Table using SSIS . Redirect Bad rows to Error Output.

Handling Special Data Types (PartyList, Lookup, PickList)

Certain datatypes in CRM requires special handling. Check this article to learn more about how to insert data into columns with PartyList, Lookup or PickList field.

Conclusion

In this article, We have seen how easy it is to integrate SQL Server data with Dynamics CRM / Dataverse. Using drag and drop connector you can perform Bulk insert, update, delete and Upsert operations. Download ZappySys SSIS PowerPack to learn more about other possible scenarios which are not discussed in this article.

The post Load data into Dynamics CRM using SSIS – Insert, Upsert, Delete, Update appeared first on ZappySys Blog.

SSIS Dynamics CRM / Dataverse – Read / Import data into SQL Server

ZappySys Team — Tue, 15 May 2018 20:36:40 +0000

Introduction

Today Microsoft Dynamics CRM / Dataverse (Dynamics 365 / On-Premises CRM / PowerApp) is one of the most popular CRMs in the world. SSIS Dynamics CRM / Dataverse (Dynamics 365 / On-Premises CRM / PowerApp) read operation can be achieved using Dynamics CRM / Dataverse Source Connector. It allows connecting to Dynamics CRM / Dataverse 365 Online or On-Premises Installation / Hosted CRM instance (IFD – internet facing deployment). You can read data from Dynamics CRM / Dataverse using SSIS connector and then exports to other sources (e.g. SQL Server, Oracle or Flat File).

In this new article, we will show how to connect and download the Dynamics CRM / Dataverse 365 (Online) data using Dynamics CRM / Dataverse Source Connector, how to query CRM data using FetchXML in SSIS and how to use variables in CRM FetchXML Query.

Requirements for SSIS Dynamics CRM / Dataverse (Dynamics 365 / On-Premises CRM / PowerApp) read component

First, you will need SSDT installed
Second, You will need to install ZappySys SSIS PowerPack which includes an SSIS Dynamics CRM / Dataverse Source (Dynamics 365 / On-Premises CRM / PowerApp) task.
Finally, a Microsoft Dynamics account

SSIS Dynamics CRM / Dataverse Source – Video Tutorial

[youtube https://www.youtube.com/watch?v=zl0ecdjuLGk?rel=0&showinfo=0&w=560&h=315]

Getting started with SSIS Dynamics CRM / Dataverse read component

Register Dynamics CRM / Dataverse App obtain App Id / Client Secret

The very first step to access Microsoft Dynamics CRM / Dataverse 365 API is to obtain Application Id (also referred as Client Id) and Client Secret. We will use this information later in this article (on OAuth connection UI).
Register Dynamics CRM / Dataverse App With Azure and get the credential (App Id, Client Secret).

Configure Dynamics CRM / Dataverse Connection

In the connection manager, press the new button to create a new connection:

SSIS create a new connection CRM Dynamics
Select the ZS-DYNAMICS-CRM Connection. We will use a CRM Online connection, username, password, and organization:

NOTE: Please check the Discovery and ORG Service URL are the same as your organization URL. And if you are using Customize URL make sure that you are using it in both Discovery URL as well as in ORG Service URL also.

SSIS Dynamics CRM / Dataverse credentials

How to create a Dynamics CRM / Dataverse Source

In the control flow, drag and drop the data flow:
Drag and drop the ZS Dynamics CRM / Dataverse Source:

SSIS Dynamics CRM / Dataverse source
Second, select the connection created before and select the account table:

SSIS entity table account

Configure SQL Server Destination

In the following example, we will show how to export the data from the account entity to SQL Server.

First, drag and drop the OLEDB Destination and join with our ZS CRM Dynamics source:

SSIS source dynamics SQL destination
Make sure to select the account table from Dynamics:

SSIS Dynamics CRM / Dataverse read entity table account

In addition, select an SQL Server and a database. In the name of the table, we will create a table with the following columns:

CREATE TABLE [crm dynamics] (
[accountcategorycode] nvarchar(255),

[accountid] uniqueidentifier,
[accountnumber] nvarchar(20),
[accountratingcode] nvarchar(255),
[accountratingcodename] nvarchar(255),
[address1_addressid] uniqueidentifier,
[address1_city] nvarchar(80),

)

SSIS OLEBD SQL Server

Go to the mapping page to map all the source and destination columns:

SSIS Dynamics CRM / Dataverse read mapping

Load Dynamics CRM / Dataverse Data to SQL Server

Execute the package:

SSIS start package
Finally, you can see the data exported in SQL Server. Select the data from the CRM Dynamics table:

Data exported from SSIS Dynamic CRM read

How to query Dynamics CRM / Dataverse

You can query Dynamics CRM / Dataverse data in two modes (Table mode or Query Mode). To use query mode you have to use a special language called FetchXML (sorry no SQL support). FetchXML is an XML based query language which can be used to SELECT, FILTER, Aggregate, Transform data from CRM. Not all features from traditional SQL Language Found in FetchXML but basic query features can be implemented by writing FetchXML. Here is an example of typical FetchXML.

Above FetchXML query is equivalent to below SQL query in traditional RDBMS.

SELECT accountid,name,country 
FROM account
WHERE country IN ('USA','UK','INDIA') 
AND status LIKE 'ACT%'

Now let’s look at how to build FetchXML queries by UI or by Hand.

How to generate a FetchXML query using UI

The Dynamics CRM / Dataverse portal contains an option to create queries visually that can be exported to XML. To do that use the Advanced Find option in the portal:

Dynamics CRM / Dataverse fetch XML
For example, we will query the Accounts entity where the account name is A. Datum:

dynamics 365 CRM Create a query
Next, we can download the Fetch XML:

Dynamics CRM / Dataverse xml download

Let’s check some examples about fetch XML:

SELECT All Columns (*)

This example shows how to write a simple FetchXML query to select all columns
of the account entity.

This FetchXML is the equivalent of below SQL

SELECT * FROM account

The option all-attributes show all the attributes. In this case, of the account entity.

To build custom FetxhXML:
1. Login to your Dynamics CRM / Dataverse Portal
2. Click on Advanced Find icon (Found next to the search textbox)
3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

SELECT Specified Columns and Alias with SSIS Dynamics CRM / Dataverse

This example shows how to write a simple FetchXML query to select two columns and use an alias for column name (e.g. name is renamed to accountname).

This FetchXML is the equivalent of below SQL

SELECT accountid, name as accountname
FROM account

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

WHERE (AND conditions)

This example shows how to write FetchXML query to filter data using multiple conditions (AND).

This FetchXML is the equivalent of below SQL

SELECT * FROM account WHERE revenue > 0 AND name like 'A%'

To build custom FetxhXML
1. Login to your Dynamics CRM / Dataverse Portal
2. Click on Advanced Find icon (Found next to the search textbox)
3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

WHERE (OR conditions)

This example shows how to write FetchXML query to filter data using multiple conditions (OR).

This FetchXML is the equivalent of below SQL

SELECT * FROM account WHERE revenue > 0 OR name like 'A%'

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

WHERE (Mixing AND / OR conditions)

This example shows how to write FetchXML query to filter data using multiple conditions (Mixing AND / OR).

This FetchXML is the equivalent of below SQL

SELECT * FROM account 
WHERE revenue > 0 AND
    (
     name LIKE 'A%' OR 
     name LIKE 'B%'
    )

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

SELECT DISTINCT (Exclude duplicates)

This example shows how to write FetchXML query to get distinct records (exclude duplicates).

This FetchXML is the equivalent of below SQL

SELECT DISTINCT accountid,name FROM account

For more information about FetchXML query syntax check below links:
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

SELECT TOP using SSIS Dynamics CRM / Dataverse read

This example shows how to write FetchXML query to select TOP N rows. This setting may not work if you using preview mode.

This FetchXML is the equivalent of below SQL

SELECT TOP 10 * FROM account

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

GROUP BY / Aggregate functions

This example shows how to use aggregate functions such as SUM, MIN, MAX, AVG along with GROUP BY clause. When you use the Aggregate function in SQL Query you can also use an alias for the field. Refer FetchXML help to learn more about aggregate functions.

This FetchXML is the equivalent of below SQL

SELECT AVG(estimatedvalue) AS estimatedvalue_avg, COUNT(name) AS opportunity_count, ownerid AS ownerid
FROM opportunity 
GROUP BY ownerid

You can use COUNT, COUNT(attribute_name), SUM, MIN, MAX, AVG as an aggregate function

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

INNER JOIN (matching child records)

This example shows how to write FetchXML query to get account and associated contacts where email is @gmail.com or yahoo.com or hotmail.com and revenue for account is > 10000.

This FetchXML is the equivalent of below SQL

SELECT a.name, c.firstname, c.lastname, c.email
FROM account a
INNER JOIN contact c ON a.accountid=c.parentcustomerid
                     AND (c.email LIKE '%@yahoo.com'
                          OR c.email LIKE '%@gmail.com'
                          OR c.email LIKE '%@hotmail.com')
WHERE a.revenue > 10000

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

LEFT OUTER JOIN (‘not in’ type query)

This example shows how to write FetchXML query to find all leads that have no tasks, using an alias.

This FetchXML is the equivalent of below SQL

SELECT lead.FullName
FROM Leads as lead
LEFT OUTER JOIN Tasks as ab
    ON (lead.leadId  =  ab.RegardingObjectId)
WHERE ab.RegardingObjectId is null

For more information on left outer join syntax check below link
https://msdn.microsoft.com/en-us/library/dn531006.aspx

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

ORDER BY

This example shows how to write FetchXML query with order by specified for one or more attributes.

This FetchXML is the equivalent of below SQL

SELECT firstname, lastname, email FROM contact
ORDER BY lastname ASC, firstname DESC

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – Equal ( = )

This example shows how to write FetchXML query using equal operator.

This FetchXML is the equivalent of below SQL

SELECT name FROM account WHERE revenue=0

For a list of supported operators check below XSD and search for name=”operator”
https://msdn.microsoft.com/en-us/library/gg309405.aspx

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – Not Equal ( != )
This example shows how to write FetchXML query using not equal operator.
This FetchXML is the equivalent of below SQL
SELECT name FROM account WHERE revenue != 0

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – Greater Than ( > )
This example shows how to write FetchXML query using greater than the operator.
This FetchXML is the equivalent of below SQL
SELECT name FROM account WHERE revenue > 0

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – Less Than ( < )
This example shows how to write FetchXML query using greater than the operator.
This FetchXML is the equivalent of below SQL
SELECT name FROM account WHERE revenue < 0

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – IN (Check for multiple matching values)
This example shows how to write FetchXML query to find multiple matching values using IN operator.
This FetchXML is the equivalent of below SQL
SELECT name FROM account WHERE name IN ('Microsoft', 'Yahoo', 'Google')

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

Microsoft
Yahoo
Google

OPERATOR – NULL (Check for NULL values)
This example shows how to write FetchXML query to find NULL values (search for records where email not specified).
This FetchXML is the equivalent of below SQL
SELECT firstname, lastname, email FROM contact WHERE email is NULL

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – NOT NULL (Check for NOT NULL values)
This example shows how to write FetchXML query to find NOT NULL values (search for records where email not specified).
This FetchXML is the equivalent of below SQL
SELECT firstname, lastname, email FROM contact WHERE email is NOT NULL

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – LIKE (Pattern search)
This example shows how to write FetchXML query using LIKE operator to match string by pattern search.
This FetchXML is the equivalent of below SQL
SELECT name FROM account WHERE name LIKE 'zappy%'

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – NOT LIKE (Pattern search)
This example shows how to write FetchXML query using LIKE operator to match string by pattern search.
This FetchXML is the equivalent of below SQL
SELECT name FROM account WHERE name NOT LIKE 'zappy%'

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – UNDER (Find all children for given hierarchy) with the SSIS Dynamics CRM / Dataverse read task
This example shows how to write FetchXML query to get children count for a given hierarchy.
For more information about querying hierarchical data check below link

https://msdn.microsoft.com/en-us/library/dn817893.aspx
For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

Using DateTime Operators with the SSIS Dynamics CRM / Dataverse read task
This example shows how to write FetchXML query to get all tickets which are created 30 mins before.
Other similar operators are

olderthan-x-hours

olderthan-x-days

olderthan-x-weeks

olderthan-x-months

olderthan-x-years
For a list of supported DateTime operators

https://msdn.microsoft.com/en-us/library/gg334216.aspx
For a list of all supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

Query – All activities (sort, outer join, alias) with the SSIS Dynamics CRM / Dataverse read task
This example shows how to get all activities performed on lead or contact. It shows how to get attributes from lead and contact by using OUTER JOIN.
This FetchXML is the equivalent of below SQL
SELECT a.*
, l.firstname as lead_name,l.emailadddress1 as lead_email1
, c.firstname as contact_name
FROM activitypointer a
LEFT OUTER JONI lead l ON a.regardingobjectid=l.leadid
LEFT OUTER JONI lead c ON a.regardingobjectid=c.contactid
ORDER BY a.createdon DESC

To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

How to use SSIS variables with Fetch XML in SSIS for SSIS Dynamics
CRM read
You can include variables in an XML fetch.

First, create your variable in the SSDT. Go to SSIS and Variables and create your variable:

SSIS variable value

In addition, create your fetch, include the variable in the XM ({{User::value}}):

Conclusion about the SSIS Dynamics CRM / Dataverse read task
To conclude, we can say that the ZS Dynamic CRM source is a great tool to get data from Microsoft CRM Dynamics and export any other destination format. In this article, we learn how to create queries to CRM Dynamics using fetch data. We used several examples to query data. We also learned to use SSIS variables in our queries and how to export the data from Dynamics CRM / Dataverse to SQL Server. Download SSIS PowerPack to explore Dynamics CRM / Dataverse integration possibilities in SSIS.
References about SSIS Dynamics CRM / Dataverse read

What is CRM?
SSIS Dynamics CRM / Dataverse Source Connector (Dynamics 365 / On-Premises CRM)

The post SSIS Dynamics CRM / Dataverse – Read / Import data into SQL Server appeared first on ZappySys Blog.