office 365 Archives | ZappySys Blog

How to Get Office 365 Mail Attachments using SSIS

ZappySys — Sat, 22 Feb 2020 08:27:34 +0000

Introduction

UPDATE: ZappySys has released a brand new Outlook Mail (Office 365) connector, which makes it much simpler to download attachments, read/search emails, users, mail folders, send email, and more in SSIS compared to the steps listed in this article. You can still use the steps from this article, but if you are new to APIs or want to avoid the learning curve, use a newer approach.

In our previous article, we saw how to get Office 365 data in Power BI. Now, let’s look at how to get Office 365 Mail Attachments using SSIS without any coding. Microsoft Graph API is a unified way to access many Microsoft services, including the Office 365 API. In this post, we will use the SSIS JSON/REST API Source Connector to retrieve the Outlook Messages list and its attachments.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from the Microsoft site).
Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
Make sure ZappySys SSIS PowerPack is installed (download it, if you haven't already).
(Optional step). Read this article, if you are planning to deploy packages to a server and schedule their execution later.

Use Case of Microsoft Graph API

Here are some use cases for why you want to use the Microsoft Graph API

Read/Write events from Outlook Calendar
Send email / Read emails
Get the list of files from OneDrive
Upload/ Download files to OneDrive
Read/Write Excel Sheet (Use range or a specific range of a specific sheet)
Read the SharePoint Document library
Search content from the SharePoint document library
Get AD users for your Organization (AD Accounts)
Update OneNote
Track changes to Users, Events, Calendar Items using the delta API

Register Application (OAuth2 App for Graph API)

The first step to access any Office 365 API / Graph API is to register an OAuth App in the Azure Portal. After following these steps, you will get the following two items to use in the next section:

Application Id
Application Secret

In this section, you will learn how to register a custom app in Microsoft Azure portal that will allow access to the OneDrive. So, let's get started -

Log into Microsoft Azure portal to register a custom app.
Register a new application by clicking New Registration link.
Provide the name of the custom app and who can access the app in the organization.
Go to the App overview and add a Redirect URL.
Click on "Add a Platform" under Platform Configuration section and then select "Web" under Web applications section to enter a Redirect URL.
```
https://zappysys.com/oauth
```
Create a Client Secret key which will be used to Authenticate the custom Azure app.
Note - Take a note of Client Secret, it will be required while configuring OAuth connection in the SSIS later
Add API Permissions.
Select following permissions from the Delegated Permissions section.
```
User.ReadBasic.All
Files.Read
offline_access
```
Take a note of Client ID, it will be required while configuring OAuth connection in the SSIS later.

Step-By-Step to call the Microsoft Graph Rest API Using SSIS

Now, let’s look at examples to access your Outlook.com emails using SSIS. You can use the same techniques to call pretty much any Office 365 API.

Configure SSIS OAuth Connection for Graph REST API

The first step to access any Graph API is to configure an OAuth connection

Open Visual Studio
Open an existing SSIS Project or create a new one using File > New > Project > Choose “Integration Services Project” Type under Business Intelligence template category.
Create Package
Right-click inside the Connection Managers area and click New Connection…
From the connection type list, select the ZS-OAUTH connection type.
Create a new SSIS OAuth API Connection Manager
On the OAuth Connection Manager, configure the following options
1. Select “Custom” from the Provider dropdown
2. Select OAuth2 from the OAuth version
3. Enter your Application Id and Secret Key (i.e., App Password) obtained in the previous section from here.
4. Enter the following URL in the Authorization URL
```
https://login.microsoftonline.com/common/oauth2/v2.0/authorize
```
5. Use the following URL in the token URL field
```
https://login.microsoftonline.com/common/oauth2/v2.0/token
```
6. Enter the following Graph API Scopes (each scope must be entered on a new line).
```
user.read
mail.read
offline_access
```
  NOTE: Always include the offline_access scope, which returns a refresh_token, which allows you to renew the token without going through the login process again. For more information about which scopes are needed, always refer to the API help page (e.g., if you want to read mail using this API, then you will see it says you can use Mail.Read or Mail.ReadWrite scope).
7. Go to the Advanced tab and enter the following URL in the Callback/Return URL (assuming the same URL used when you registered the App in the previous section)
```
https://zappysys.com/oauth
```
8. Now go back to the first tab and click Generate Token. When prompted, log in using your Personal Microsoft Account or Work Account (Office 365 or AD login)
9. After logging in, you will see the Accept option. Just click it. If things go right, then you will see the Access Token and Refresh Token fields populated.
Once everything is configured, you can click Test Connection to verify the connection works.

SSIS OAuth Connection Manager for Office 365 REST API Microsoft Graph API

NOTE: Typically, RefreshToken is long-lived, but in some cases, they may expire soon too. If that’s the case, then you can configure the above OAuth connection to change the refresh token pattern. Simply enter the token file path and re-authenticate by clicking Generate Token. After that, each time you make an API call, it will save a new refresh token in the file.

Download the Messages Attachment Files from Outlook

Let’s start with an example. We use the SSIS JSON Source and Web API Destination components to make a call to the Microsoft Graph REST API.

First of all, drag and drop the Data Flow Task from the SSIS Toolbox and double-click it to edit.
Drag and drop Data flow
From the SSIS toolbox, drag and drop the JSON Source onto the Data Flow Designer surface.

SSIS JSON Source – Drag and Drop
Double-click the JSON Source, then enter the following URL to invoke the Messages API call with search options to retrieve emails with attachments.
```
https://graph.microsoft.com/v1.0/me/messages?$select=sentDateTime,from,subject,hasAttachments&$search="hasAttachments:True"
```
Set the filter to $.value[*] and click the preview.

SSIS JSON Source – Get data from Microsoft Graph API (Office 365 API – Read Mail Example)

To customize the URL with additional parameters, check this help link. It’s a standard OData Protocol, so you can leverage many common OData features (e.g., use $top parameters to set pageSize. Default is 10 for List Mail, but you can include more records per response by changing it, such as …/messages?$top=50).
Now, let’s use Template Transform to set the URL using the message ID to make the Get attachment API call.
```
https://graph.microsoft.com/v1.0/me/messages/<%id%>/attachments
```
SSIS Template Transform sending attachment id
Now we need to use the Web API Destination to make the Get Attachment API call by message ID. Select the Input column for URL as that TemplateOutput.
SSIS Web API Destination – Get data from Microsoft Graph API (Office 365 API – Get Attachment Example)

Now, the next step is to parse the JSON Response String of the get attachment api call. For that, we need to use a JSON Parser. Select the input as Web PAI Destination ResponseText and use the below sample JSON string to configure the metadata(Columns).

{"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#users('support%40zappysys.com')/messages('AQMkADAwATMwMAItMDkyMC0zZTg4AC0wMAItMDAKAEYAAAP_JUy')/attachments"
,"value":[{"@odata.type":"#microsoft.graph.fileAttachment","@odata.mediaContentType":"text/plain"
,"id":"AQMkADAwATMwMAItMDkyMC0zZTg4AC0wMAItMDAKAEYAAAP_JUy_27M"
,"lastModifiedDateTime":"2020-02-21T07:36:16Z"
,"name":"Test.txt","contentType":"text/plain","size":870,"isInline":false
,"contentId":null,"contentLocation":null,"contentBytes":"VGVzdCBNZXNzYWdl"}]}

JSON Parser Transform – using JSON example

Now, in the JSON Parser, go to the Go-to columns Tab and set the contentBytes DataType to DT_IMAGE, and increase the other columns’ DT_WSTR lengths by 1000 and 1500. We set contentBytes to the DT_IMAGE datatype, which is like varbinary(MAX).
JSON Parser Transform Columns Tab fix the datatype
Furthermore, in the next step, we need to use an SSIS Derived Column to set the FilePath where we want to save the attachments, and then concatenate the local file path with the attachment name column.
SSIS Derived Column: Add New Column
Finally, we need to use the Export Column Transform (Native) to export attachment bytes to a local file. It needs to know 2 upstream info Bytes and the target file path.
Export Column configuration
Make sure to attach the export column to some destination (e.g., our ZS Trash Destination); the engine might remove it. At the same time, it optimizes the runtime. It may remove all transformations that don’t have a destination, such as deploying the package to SQL Server, or when you set optimize in Visual Studio.
That’s it, execute the package, and it will download all the email attachments.
Downloaded Mail Attachments

Conclusion

After all, in this article, we learned how to make a Microsoft Graph API REST API call. We used the SSIS JSON / REST API Connector to extract data from Outlook Mail using OAuth. Made the Get attachment API call using the Web API Destination and parsed the JSON response using the JSON Parser. And using the Native SSIS Export Column, we saved the export attachment’s bytes to a local file. To explore many other scenarios not discussed in this article, download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

- Help File: JSON Source(REST API or File)
- Help File: Web API Destination
- Help File: JSON Parser Transform
- Overview of the Microsoft Graph API

The post How to Get Office 365 Mail Attachments using SSIS appeared first on ZappySys Blog.

Get Office 365 data in Power BI using Microsoft Graph API and ODBC

ZappySys — Wed, 19 Dec 2018 17:57:00 +0000

Introduction

In this article, we will get Office 365 data in Power BI using Microsoft Graph API and ODBC drivers. Specifically, we will get Excel file data in OneDrive, events from a Calendar and finally, a list in a note in OneNote and then load that data into a Power BI report. We will achieve that by creating an ODBC data source using ZappySys ODBC PowerPack and then querying corresponding Microsoft Graph API methods. ODBC PowerPack allows to query REST/SOAP APIs in an easy way and lets to view responses in a tabular format, which will be perfect to accomplish our goal.

Prerequisites

A Power BI account.
Wield basic Power BI skills.
Power BI Desktop installed.
ZappySys ODBC PowerPack installed.

Step By Step – Get Office 365 data in Power BI using Microsoft Graph API and ODBC

Register Application (OAuth2 App for Graph API)

The first step to access any Office 365 API / Graph API is to register an OAuth App in the Azure Portal. After following these steps, you will get the following two items to use in the next section:

Application Id
Application Secret

In this section, you will learn how to register a custom app in Microsoft Azure portal that will allow access to the OneDrive. So, let's get started -

Log into Microsoft Azure portal to register a custom app.
Register a new application by clicking New Registration link.
Provide the name of the custom app and who can access the app in the organization.
Go to the App overview and add a Redirect URL.
Click on "Add a Platform" under Platform Configuration section and then select "Web" under Web applications section to enter a Redirect URL.
```
https://zappysys.com/oauth
```
Create a Client Secret key which will be used to Authenticate the custom Azure app.
Note - Take a note of Client Secret, it will be required while configuring OAuth connection in the SSIS later
Add API Permissions.
Select following permissions from the Delegated Permissions section.
```
User.ReadBasic.All
Files.Read
offline_access
```
Take a note of Client ID, it will be required while configuring OAuth connection in the SSIS later.

Open ODBC Data Source Administrator

Hit Windows Start button and search for "odbc".
You will see two versions of ODBC Data Source administrator: 64-bit and 32-bit. Choose:
1. - 64-bit if your client application is a 64-bit program and you plan to create a System DSN.
  - 32-bit if your client application is a 32-bit program and you plan to create a System DSN.
  - 32-bit or 64-bit version if you plan to create a User DSN.
Windows ODBC Data Sources
Once you open, you should see a similar window:

ODBC Data Source Administrator

Set up an ODBC data source based on ZappySys JSON Driver

Add a data source

Go to System DSN and press Add to add a system DSN. If you use apps like Power BI which needs access only your User account then you can create User DSN.
Add a data source based on ZappySys JSON Driver:

Configure the data source

Once a window appears, give a name to the data source, e.g. System - JSON:
Then input https://graph.microsoft.com/v1.0/me in the URL field.
Select OAuth as the Connection Type.
Click Click to Configure button/link to open OAuth configuration window.
Select Custom as OAuth Provider.
Then input Client Id, Client Secret which we got when created the Graph API application.
Set https://login.microsoftonline.com/common/oauth2/v2.0/authorize as Authorization Url.
Set https://login.microsoftonline.com/common/oauth2/v2.0/token as Access Token Url.
In Scopes / Permissions section set these scopes, separated by a new line. E.g. offline_access, Files.Read.All, Calendars.Read, Notes.Read, User.Read
In Advanced tab configure Callback/Redirect Url and set it to the one we configured in Graph API app:
Finally, click Generate Token button to generate a token.
Click OK to close the window.

Set up an ODBC data source based on ZappySys XML Driver

Before adding another data source based on ZappySys XML Driver, copy the connection string of the data source we just created:
Close the window and then add a new data source, but this time based on ZappySys XML Driver.
Once a window is opened, give it a name, e.g. System - XML.
Then press Load connection string button to load the connection string.
Paste the copied connection string and replace JSON text to XML, so that you have { ZappySys XML Driver } in DRIVER property.
Close the window.

Create a Power BI report

Reading files from OneDrive (getting Excel data)

Let’s say you have Stock.xlsx stored in Sample Documents folder, OneDrive with this kind of data and you want to show it in a Power BI report:

To have that data in a Power BI report, perform these steps:

Open Power BI Desktop and create a new Power BI report.
Create a data source based on System – JSON DSN (since we get a JSON response):
Then in SQL statement area enter this query:
SELECT * FROM $ WITH(Src='https://graph.microsoft.com/v1.0/me/drive/root:/YOUR FOLDER/Stock.xlsx:/workbook/worksheets/SHEET NAME/usedRange' ,Filter='$.values[1:]' ,ArrayTransformType='TransformSimpleTwoDimensionalArray' ,ArrayTransColumnNameFilter='$.values[0][*]' )Using WITH clause lets us override the properties. You can generate the query above using the Query Builder:
In this step we are overriding Url, Filter and array transformation fields you configured in JSON DSN. The Src value overrides the URL. The Filter property enables us to select the data we want to see in the report. We use [1:] in $.values[1:] to skip the first row because the first row contains the headers and we want only the data. You can configure this option in the Filter Options tab (read more about JSON path).We also have to make the transformation on the JSON and turn JSON arrays into rows. To accomplish that, we override ArrayTransformType. With ArrayTransColumnNameFilter property and [0][*] text, we specify that the first array of parent array will be the headers of the columns. You can read more about transformations in Parse multi-dimensional JSON array article.
Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other OneDrive endpoints.
Click OK.
Drag a table into the report, select the columns and you should see a similar view:

Reading calendar entries

Now, let’s say you use a calendar in Office 365 and you want your appointments to be visible in a Power BI report:

To start, create a new Power BI report.
Then add a data source based on System – JSON DSN (again, we get the response in JSON):
In SQL statement section enter this query:
SELECT * FROM $ WITH(Src='https://graph.microsoft.com/v1.0/me/events?$select=subject,start,end,location' ,Filter='$.value[*]' ,Header='Prefer: outlook.timezone="FLE Standard Time"' )Here we also are overriding HTTP Header so that we get the event times not in UTC/GMT but for the specified timezone. It must be specified in Prefer: outlook.timezone=”{time-zone-string}” format (check an article on how to get a list of supported time zones).
Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other Calendar endpoints.
Click OK.
Drag a table control into the report, select several columns and you should see a similar view:

Reading notes from OneNote

Suppose, you have these notes in OneNote and you want “Groceries list” to be shown in a Power BI report:

First, we will need to find NOTE ID we are trying to get. To do that, open ODBC Administrator, find System – JSON data source and hit Configure.
Then go to the Preview tab, and input this SQL query and hit Preview Data:
SELECT * FROM $ WITH (Src='https://graph.microsoft.com/v1.0/me/onenote/pages' ,Filter='$.value[*]' )
Once you get a result set, find your note name and copy its ID:
Then create a new Power BI report.
Create a new data source based on System – XML DSN (we are using XML-based DSN because the response will be in HTML):
In SQL statement section enter this query (use the NOTE ID you retrieved in the previous step):
SELECT * FROM $ WITH(ElementsToTreatAsArray='div,p' ,Src='https://graph.microsoft.com/v1.0/me/onenote/pages/NOTE ID/content' ,Filter='$.html.body.div[0].p[1:]' )

Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other OneNote endpoints.
Click OK.
Drag and drop Multi-row card visual, select #text column and you should see a similar view:
We finished! Now you can publish your Power BI reports to Power BI service.

Working with Gateways in Power BI (Schedule Import)

If the data needs to be updated, it is necessary to create a gateway on-premises. In this new section, we will install a Power BI Gateway and in the next section schedule it to update the REST API information.

In the last section, we Published the report. Power BI may ask you to SIGN IN.

Sign in in Power BI
Select the Workspace and select Datasets
[caption id="attachment_10110" align="alignnone" width="702"] Go to workspace and dataset[/caption]
Go to workspace and dataset
Right-click the report and select Settings.

Specify settings for the report
The system will ask for a Gateway. Stay here.

Add Power BI Gateway

Use the following link to install a Data Gateway:

https://docs.microsoft.com/en-us/power-bi/service-gateway-onprem

Run the installer and press Next

Gateway installer
Select the option On-premises data gateway (recommended). This option allows access to multiple users and can be used by more applications than Power BI.

Choose Power BI option
The installer will show a warning message.

Warning message during installation
Select the path to install and check the I accept the terms.

Accept terms
Specify the email address to use the gateway.

Register email address
After entering the email, write the gateway name and a recovery key. Make sure to confirm the recovery key.

Enter recovery key

Manage gateways and configure the schedule

Once that the gateway is installed we will configure it and add the connection strings.

The next step is to go to manage gateway

Power BI - Manage Gateway Setting
In order to get the connection string, we will need the connection string of the ZappySys JSON Driver. In the first section of this post, we explained how to configure it. Press Copy Connection String

Connection properties
Once that the data is copied, add a New data Source. In Data Source Name, enter the Data Source Name of the ZappySys JSON driver in step 13 and in Data Source Type, select ODBC. In connection string copy and paste from the clipboard of the step 13 and press Add.
[caption id="attachment_10113" align="alignnone" width="720"] ZappySys connection properties in Power BI[/caption]
ZappySys connection properties in Power BI
Once added the gateway. You can see the schedule refresh to On and Add another time to add the time where you want to refresh the data.

Schedule gateway

Conclusion

Today we learned how to call Microsoft Graph API methods using ZappySys ODBC PowerPack and load response data into Power BI without any ETL procedures. Particularly, we used OneDrive, Calendar and OneNote Graph API endpoints to achieve that. We created two ODBC data sources based on JSON and XML ODBC Driver to query those endpoints, which let us read data directly in a Power BI report. Finally, we examined how to refresh data source on Power BI using on-premises Power BI gateway.

References

https://powerbi.microsoft.com/en-us/

https://apps.dev.microsoft.com/

https://developer.microsoft.com/en-us/graph/graph-explorer/

https://docs.microsoft.com/en-us/graph/overview

https://docs.microsoft.com/en-us/graph/onedrive-concept-overview

https://docs.microsoft.com/en-us/graph/outlook-calendar-concept-overview

https://docs.microsoft.com/en-us/graph/integrate-with-onenote

The post Get Office 365 data in Power BI using Microsoft Graph API and ODBC appeared first on ZappySys Blog.

Load data into Dynamics CRM using SSIS – Insert, Upsert, Delete, Update

ZappySys — Thu, 05 Jul 2018 19:20:32 +0000

Introduction

In our previous article, we talked about how to read from Dynamics CRM / Dataverse (Dynamics 365 / On-Premises CRM / PowerApp). Now let’s look at how to Load data into Dynamics CRM / Dataverse (Dynamics 365 / On-Premises CRM / PowerApp) using SSIS. We will see a few important aspects of loading data into Dynamics CRM / Dataverse Online (i.e. Dynamics 365) and On-premises / IFD (Internet facing hosted instance).

We will show you how to use SSIS Dynamics CRM Destination connector for Bulk Insert, Update, Delete and Upsert Operations. This article uses Dynamics 365 Online for CRM Integration Examples but concepts are same for other types of installation (e.g. IFD or On-Premises)

Requirements

Make sure you have SSIS Designer installed (i.e. SSDT-BI)
Download and Installed ZappySys SSIS PowerPack which includes an SSIS Dynamics CRM Connectors.
Finally, make sure you have Microsoft Dynamics CRM / Dataverse credentials handy.

That’s it. So let’s get started.

Video Tutorial – CRM Insert, Upsert, Delete, Update

Getting Started

In this section, we will see how to perform bulk insert, update delete and upsert operations using SSIS Dynamics CRM Destination

Configure SSIS Dynamics CRM / Dataverse Connection

The first step to read or write Dynamics CRM / Dataverse data is to make sure we have working Dynamics CRM connection in SSIS.

Create new or open existing SSIS Project in Visual Studio.
Open SSIS Package and Go to SSIS designer.
Right click in the connection manager panel and click New connection:

Create a new SSIS Dynamics CRM / Dataverse Connection
In the connection type selection, Select the ZS-DYNAMICS-CRM Connection.
We will use a CRM Online (Office 365) connection, region, a username and a password and select organization from the dropdown as below and then click Test connection:

Configure Dynamics CRM / Dataverse Connection in SSIS

Bulk Insert – SQL Server data Import into Dynamics CRM / Dataverse using SSIS

Now, let’s see steps to do SQL Server data Import into Dynamics CRM / Dataverse using SSIS. Below steps assume you already have sample data in some SQL Server Table which will be used to create new CRM accounts. You can enter few account names you like to create in the CRM.

Open SSIS Package and drag Data Flow task from Control Flow SSIS Toolbox
Go to data flow designer. Drag and drop OLEDB Source
Configure OLEDB Source to read from some sample data table (e.g. dbo.Accounts). If you want to be lazy and just test quickly you can use ZS CSV Source (Direct Value Mode) instead of OLEDB Source.Sample Data May be like below
```
name,address1_city,address1_country
Flipkart,Bengaluru,India
Microsoft,seattle,USA
Nokia,Espoo,Finland
```
Once you configure Source (Connection, Table Name etc.) click OK to save it.

CSV Source – Sample Accounts for CRM Insert Demo
Drag and drop ZS Dynamics CRM Destination from the toolbox.
Connect Source to your Destination component.
Double click CRM destination to configure it.
On the Connection Managers tab, Select Dynamics CRM / Dataverse connection we created in the previous section.

Select Connection for Dynamics CRM / Dataverse Destination
On the Component Properties Tab Select Table Name.

Select Target Table for Dynamics CRM / Dataverse Insert
On the Mapping Tab map few columns (e.g. name, address1_city, and address1_country)

Configure Target Column Mapping for CRM Data Load
Click OK to save CRM destination.
Right click in the data flow designer, click Execute Task

Execute SSIS Package – Loading Data into Dynamics CRM / Dataverse from CSV / SQL Server

Bulk Upsert – Update or Insert Dynamics CRM / Dataverse Records in SSIS

In the previous section we saw how to Insert New records. However sometimes you want to Upsert (Insert or Update). So if record already exists Update else Insert. In order to achieve this you have to set two additional attributes. For CRM Upsert you have to perform few changes to previous steps

Create Alternate Key for CRM Upsert

To perform Upsert operation you can map Table’s Primary Key or Alternate Key. Primary Key usually starts with table name and ends with “id” (e.g. for account table its accountid ).

Many times Source system don’t have CRM Table Primary Key. In such case you may use Alternate Key in the mapping (JOIN Key). To use Alternate key for Upsert you have first define Alternate Key in CRM Portal. Check this link for exact steps.

Create Alternate Key for Dynamics CRM / Dataverse Online Table (Useful for Update or Upsert Operation)

Configure SSIS CRM Destination for Upsert

Double click CRM destination. On the Component Properties Tab, Change Operation to UpdateOrInsert
Specify UpsertKeyField to either Table’s Primary Key Column Name or Alternate Key Name.

Dynamics CRM / Dataverse Upsert Records using SSIS CRM Destination (Specify Alternate Key / Primary Key)
Now go to Mappings tab and confirm Key field is mapped.

Configure Dynamics CRM / Dataverse Mapping for Upsert in SSIS
Click OK and Run Package to test CRM Upsert Operation.

If you want to use primary key for Upsert or Update Operation and not sure correct column name then you may use ZappySys Dynamics CRM / Dataverse Source FetchXml Query Mode and check Object Browser (Table Info Tab >> PrimaryIdAttribute in the grid).

Primary Key name usually starts with table name and ends with “id” (e.g. for account table its accountid ).

Find Primary Key for Dynamics CRM / Dataverse Table

Handling Inserted / Updated Records Output

Now lets look at how to detect which records got inserted and which records got Update? In Upsert Mode each record can be directed to one of the following three Outputs.

Inserted Records [Blue Arrow]
Updated Records [Blue Arrow]
Records with Error [Red Arrow]

You can select Red or Blue arrow coming out from CRM Destination component. Drag and drop output arrow on any down stream component.

Bulk Update – Modify Dynamics CRM / Dataverse Records in SSIS

If you want to do bulk Update its very same steps as Upsert Operation except you have to set Operation = Update in the properties grid. For Key field you have to set primary key or Alternate Key.

Dynamics CRM / Dataverse Update in SSIS

Bulk Delete – Delete Dynamics CRM / Dataverse Records in SSIS

Bulk delete doesn’t require many settings. All you have to change is Operation = Delete on Component Properties Screen. On the Mapping Screen Map only one column (Primary Key). If you are not sure which one is primary key please refer to the previous section (Upsert Operation).

Delete records from Dynamics CRM / Dataverse Online using SSIS (Primary Key mapping for CRM Bulk Delete)

Some times you may get below errors. If you get this error make sure you delete all child records first before deleting parent. For example if you are deleting account then make sure you have to delete leads / contacts for that account or update leads / contacts so they don’t refer to account being deleted.

Error: 0xC0047062 at Dynamics CRM / Dataverse Bulk Delete, Delete Accounts [2]: System.Exception: Exception: The object you tried to delete is associated with another object and cannot be deleted.
Error Details: 
ApiExceptionSourceKey=Plugin/Microsoft.Crm.Common.ObjectModel.AccountService

Error Handling

Many times you want to ignore bad records and continue processing good records. In such case you can use Dynamics CRM / Dataverse Error Output. Just grad Red arrow from CRM Destination and Drop on target component. Select Redirect Option for Error , Truncation both and Click OK.

Once you set Error Handling your component will continue to run without failing whole data flow. Later you can review errors (if you saving to destination).

Configure Error Handling in SSIS Dynamics CRM / Dataverse Destination

Loading records into Dynamics CRM / Dataverse Table using SSIS . Redirect Bad rows to Error Output.

Handling Special Data Types (PartyList, Lookup, PickList)

Certain datatypes in CRM requires special handling. Check this article to learn more about how to insert data into columns with PartyList, Lookup or PickList field.

Conclusion

In this article, We have seen how easy it is to integrate SQL Server data with Dynamics CRM / Dataverse. Using drag and drop connector you can perform Bulk insert, update, delete and Upsert operations. Download ZappySys SSIS PowerPack to learn more about other possible scenarios which are not discussed in this article.

The post Load data into Dynamics CRM using SSIS – Insert, Upsert, Delete, Update appeared first on ZappySys Blog.

SSIS Dynamics CRM / Dataverse – Read / Import data into SQL Server

ZappySys Team — Tue, 15 May 2018 20:36:40 +0000

Introduction

Today Microsoft Dynamics CRM / Dataverse (Dynamics 365 / On-Premises CRM / PowerApp) is one of the most popular CRMs in the world. SSIS Dynamics CRM / Dataverse (Dynamics 365 / On-Premises CRM / PowerApp) read operation can be achieved using Dynamics CRM / Dataverse Source Connector. It allows connecting to Dynamics CRM / Dataverse 365 Online or On-Premises Installation / Hosted CRM instance (IFD – internet facing deployment). You can read data from Dynamics CRM / Dataverse using SSIS connector and then exports to other sources (e.g. SQL Server, Oracle or Flat File).

In this new article, we will show how to connect and download the Dynamics CRM / Dataverse 365 (Online) data using Dynamics CRM / Dataverse Source Connector, how to query CRM data using FetchXML in SSIS and how to use variables in CRM FetchXML Query.

Requirements for SSIS Dynamics CRM / Dataverse (Dynamics 365 / On-Premises CRM / PowerApp) read component

First, you will need SSDT installed
Second, You will need to install ZappySys SSIS PowerPack which includes an SSIS Dynamics CRM / Dataverse Source (Dynamics 365 / On-Premises CRM / PowerApp) task.
Finally, a Microsoft Dynamics account

SSIS Dynamics CRM / Dataverse Source – Video Tutorial

[youtube https://www.youtube.com/watch?v=zl0ecdjuLGk?rel=0&showinfo=0&w=560&h=315]

Getting started with SSIS Dynamics CRM / Dataverse read component

Register Dynamics CRM / Dataverse App obtain App Id / Client Secret

The very first step to access Microsoft Dynamics CRM / Dataverse 365 API is to obtain Application Id (also referred as Client Id) and Client Secret. We will use this information later in this article (on OAuth connection UI).
Register Dynamics CRM / Dataverse App With Azure and get the credential (App Id, Client Secret).

Configure Dynamics CRM / Dataverse Connection

In the connection manager, press the new button to create a new connection:

SSIS create a new connection CRM Dynamics
Select the ZS-DYNAMICS-CRM Connection. We will use a CRM Online connection, username, password, and organization:

NOTE: Please check the Discovery and ORG Service URL are the same as your organization URL. And if you are using Customize URL make sure that you are using it in both Discovery URL as well as in ORG Service URL also.

SSIS Dynamics CRM / Dataverse credentials

How to create a Dynamics CRM / Dataverse Source

In the control flow, drag and drop the data flow:
Drag and drop the ZS Dynamics CRM / Dataverse Source:

SSIS Dynamics CRM / Dataverse source
Second, select the connection created before and select the account table:

SSIS entity table account

Configure SQL Server Destination

In the following example, we will show how to export the data from the account entity to SQL Server.

First, drag and drop the OLEDB Destination and join with our ZS CRM Dynamics source:

SSIS source dynamics SQL destination
Make sure to select the account table from Dynamics:

SSIS Dynamics CRM / Dataverse read entity table account

In addition, select an SQL Server and a database. In the name of the table, we will create a table with the following columns:

CREATE TABLE [crm dynamics] (
[accountcategorycode] nvarchar(255),

[accountid] uniqueidentifier,
[accountnumber] nvarchar(20),
[accountratingcode] nvarchar(255),
[accountratingcodename] nvarchar(255),
[address1_addressid] uniqueidentifier,
[address1_city] nvarchar(80),

)

SSIS OLEBD SQL Server

Go to the mapping page to map all the source and destination columns:

SSIS Dynamics CRM / Dataverse read mapping

Load Dynamics CRM / Dataverse Data to SQL Server

Execute the package:

SSIS start package
Finally, you can see the data exported in SQL Server. Select the data from the CRM Dynamics table:

Data exported from SSIS Dynamic CRM read

How to query Dynamics CRM / Dataverse

You can query Dynamics CRM / Dataverse data in two modes (Table mode or Query Mode). To use query mode you have to use a special language called FetchXML (sorry no SQL support). FetchXML is an XML based query language which can be used to SELECT, FILTER, Aggregate, Transform data from CRM. Not all features from traditional SQL Language Found in FetchXML but basic query features can be implemented by writing FetchXML. Here is an example of typical FetchXML.

Above FetchXML query is equivalent to below SQL query in traditional RDBMS.

SELECT accountid,name,country 
FROM account
WHERE country IN ('USA','UK','INDIA') 
AND status LIKE 'ACT%'

Now let’s look at how to build FetchXML queries by UI or by Hand.

How to generate a FetchXML query using UI

The Dynamics CRM / Dataverse portal contains an option to create queries visually that can be exported to XML. To do that use the Advanced Find option in the portal:

Dynamics CRM / Dataverse fetch XML
For example, we will query the Accounts entity where the account name is A. Datum:

dynamics 365 CRM Create a query
Next, we can download the Fetch XML:

Dynamics CRM / Dataverse xml download

Let’s check some examples about fetch XML:

SELECT All Columns (*)

This example shows how to write a simple FetchXML query to select all columns
of the account entity.

This FetchXML is the equivalent of below SQL

SELECT * FROM account

The option all-attributes show all the attributes. In this case, of the account entity.

To build custom FetxhXML:
1. Login to your Dynamics CRM / Dataverse Portal
2. Click on Advanced Find icon (Found next to the search textbox)
3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

SELECT Specified Columns and Alias with SSIS Dynamics CRM / Dataverse

This example shows how to write a simple FetchXML query to select two columns and use an alias for column name (e.g. name is renamed to accountname).

This FetchXML is the equivalent of below SQL

SELECT accountid, name as accountname
FROM account

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

WHERE (AND conditions)

This example shows how to write FetchXML query to filter data using multiple conditions (AND).

This FetchXML is the equivalent of below SQL

SELECT * FROM account WHERE revenue > 0 AND name like 'A%'

To build custom FetxhXML
1. Login to your Dynamics CRM / Dataverse Portal
2. Click on Advanced Find icon (Found next to the search textbox)
3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

WHERE (OR conditions)

This example shows how to write FetchXML query to filter data using multiple conditions (OR).

This FetchXML is the equivalent of below SQL

SELECT * FROM account WHERE revenue > 0 OR name like 'A%'

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

WHERE (Mixing AND / OR conditions)

This example shows how to write FetchXML query to filter data using multiple conditions (Mixing AND / OR).

This FetchXML is the equivalent of below SQL

SELECT * FROM account 
WHERE revenue > 0 AND
    (
     name LIKE 'A%' OR 
     name LIKE 'B%'
    )

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

SELECT DISTINCT (Exclude duplicates)

This example shows how to write FetchXML query to get distinct records (exclude duplicates).

This FetchXML is the equivalent of below SQL

SELECT DISTINCT accountid,name FROM account

For more information about FetchXML query syntax check below links:
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

SELECT TOP using SSIS Dynamics CRM / Dataverse read

This example shows how to write FetchXML query to select TOP N rows. This setting may not work if you using preview mode.

This FetchXML is the equivalent of below SQL

SELECT TOP 10 * FROM account

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

GROUP BY / Aggregate functions

This example shows how to use aggregate functions such as SUM, MIN, MAX, AVG along with GROUP BY clause. When you use the Aggregate function in SQL Query you can also use an alias for the field. Refer FetchXML help to learn more about aggregate functions.

This FetchXML is the equivalent of below SQL

SELECT AVG(estimatedvalue) AS estimatedvalue_avg, COUNT(name) AS opportunity_count, ownerid AS ownerid
FROM opportunity 
GROUP BY ownerid

You can use COUNT, COUNT(attribute_name), SUM, MIN, MAX, AVG as an aggregate function

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

INNER JOIN (matching child records)

This example shows how to write FetchXML query to get account and associated contacts where email is @gmail.com or yahoo.com or hotmail.com and revenue for account is > 10000.

This FetchXML is the equivalent of below SQL

SELECT a.name, c.firstname, c.lastname, c.email
FROM account a
INNER JOIN contact c ON a.accountid=c.parentcustomerid
                     AND (c.email LIKE '%@yahoo.com'
                          OR c.email LIKE '%@gmail.com'
                          OR c.email LIKE '%@hotmail.com')
WHERE a.revenue > 10000

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

LEFT OUTER JOIN (‘not in’ type query)

This example shows how to write FetchXML query to find all leads that have no tasks, using an alias.

This FetchXML is the equivalent of below SQL

SELECT lead.FullName
FROM Leads as lead
LEFT OUTER JOIN Tasks as ab
    ON (lead.leadId  =  ab.RegardingObjectId)
WHERE ab.RegardingObjectId is null

For more information on left outer join syntax check below link
https://msdn.microsoft.com/en-us/library/dn531006.aspx

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

ORDER BY

This example shows how to write FetchXML query with order by specified for one or more attributes.

This FetchXML is the equivalent of below SQL

SELECT firstname, lastname, email FROM contact
ORDER BY lastname ASC, firstname DESC

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – Equal ( = )

This example shows how to write FetchXML query using equal operator.

This FetchXML is the equivalent of below SQL

SELECT name FROM account WHERE revenue=0

For a list of supported operators check below XSD and search for name=”operator”
https://msdn.microsoft.com/en-us/library/gg309405.aspx

For more information about FetchXML query syntax check below links :
https://msdn.microsoft.com/en-us/library/gg328117.aspx
https://msdn.microsoft.com/en-us/library/gg328332.aspx

For complete syntax and validation rule for FetchXML check below XSD
https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – Not Equal ( != )
This example shows how to write FetchXML query using not equal operator.
This FetchXML is the equivalent of below SQL
SELECT name FROM account WHERE revenue != 0

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – Greater Than ( > )
This example shows how to write FetchXML query using greater than the operator.
This FetchXML is the equivalent of below SQL
SELECT name FROM account WHERE revenue > 0

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – Less Than ( < )
This example shows how to write FetchXML query using greater than the operator.
This FetchXML is the equivalent of below SQL
SELECT name FROM account WHERE revenue < 0

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – IN (Check for multiple matching values)
This example shows how to write FetchXML query to find multiple matching values using IN operator.
This FetchXML is the equivalent of below SQL
SELECT name FROM account WHERE name IN ('Microsoft', 'Yahoo', 'Google')

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

Microsoft
Yahoo
Google

OPERATOR – NULL (Check for NULL values)
This example shows how to write FetchXML query to find NULL values (search for records where email not specified).
This FetchXML is the equivalent of below SQL
SELECT firstname, lastname, email FROM contact WHERE email is NULL

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – NOT NULL (Check for NOT NULL values)
This example shows how to write FetchXML query to find NOT NULL values (search for records where email not specified).
This FetchXML is the equivalent of below SQL
SELECT firstname, lastname, email FROM contact WHERE email is NOT NULL

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – LIKE (Pattern search)
This example shows how to write FetchXML query using LIKE operator to match string by pattern search.
This FetchXML is the equivalent of below SQL
SELECT name FROM account WHERE name LIKE 'zappy%'

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – NOT LIKE (Pattern search)
This example shows how to write FetchXML query using LIKE operator to match string by pattern search.
This FetchXML is the equivalent of below SQL
SELECT name FROM account WHERE name NOT LIKE 'zappy%'

For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

OPERATOR – UNDER (Find all children for given hierarchy) with the SSIS Dynamics CRM / Dataverse read task
This example shows how to write FetchXML query to get children count for a given hierarchy.
For more information about querying hierarchical data check below link

https://msdn.microsoft.com/en-us/library/dn817893.aspx
For a list of supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

Using DateTime Operators with the SSIS Dynamics CRM / Dataverse read task
This example shows how to write FetchXML query to get all tickets which are created 30 mins before.
Other similar operators are

olderthan-x-hours

olderthan-x-days

olderthan-x-weeks

olderthan-x-months

olderthan-x-years
For a list of supported DateTime operators

https://msdn.microsoft.com/en-us/library/gg334216.aspx
For a list of all supported operators check below XSD and search for name=”operator”

https://msdn.microsoft.com/en-us/library/gg309405.aspx
To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

Query – All activities (sort, outer join, alias) with the SSIS Dynamics CRM / Dataverse read task
This example shows how to get all activities performed on lead or contact. It shows how to get attributes from lead and contact by using OUTER JOIN.
This FetchXML is the equivalent of below SQL
SELECT a.*
, l.firstname as lead_name,l.emailadddress1 as lead_email1
, c.firstname as contact_name
FROM activitypointer a
LEFT OUTER JONI lead l ON a.regardingobjectid=l.leadid
LEFT OUTER JONI lead c ON a.regardingobjectid=c.contactid
ORDER BY a.createdon DESC

To build custom FetxhXML

1. Login to your Dynamics CRM / Dataverse Portal

2. Click on Advanced Find icon (Found next to the search textbox)

3. Once done building query, Click on Download FetchXML icon. Copy that XML into Query editor of Dynamics CRM / Dataverse Source.
For more information about FetchXML query syntax check below links :

https://msdn.microsoft.com/en-us/library/gg328117.aspx

https://msdn.microsoft.com/en-us/library/gg328332.aspx
For complete syntax and validation rule for FetchXML check below XSD

https://msdn.microsoft.com/en-us/library/gg309405.aspx

How to use SSIS variables with Fetch XML in SSIS for SSIS Dynamics
CRM read
You can include variables in an XML fetch.

First, create your variable in the SSDT. Go to SSIS and Variables and create your variable:

SSIS variable value

In addition, create your fetch, include the variable in the XM ({{User::value}}):

Conclusion about the SSIS Dynamics CRM / Dataverse read task
To conclude, we can say that the ZS Dynamic CRM source is a great tool to get data from Microsoft CRM Dynamics and export any other destination format. In this article, we learn how to create queries to CRM Dynamics using fetch data. We used several examples to query data. We also learned to use SSIS variables in our queries and how to export the data from Dynamics CRM / Dataverse to SQL Server. Download SSIS PowerPack to explore Dynamics CRM / Dataverse integration possibilities in SSIS.
References about SSIS Dynamics CRM / Dataverse read

What is CRM?
SSIS Dynamics CRM / Dataverse Source Connector (Dynamics 365 / On-Premises CRM)

The post SSIS Dynamics CRM / Dataverse – Read / Import data into SQL Server appeared first on ZappySys Blog.

How to use Office 365 API with SSIS (Mail, Calendar, Contacts, OneDrive, Excel)

ZappySys — Sat, 12 Aug 2017 23:45:15 +0000

Introduction

NOTE: UPDATE: ZappySys has released a brand new API Connector for SSIS Outlook Mail (Office 365) which makes it much simpler to download attachment, read / search Emails, Users, MailFolders, Send email and more in SSIS compared to the steps listed in this article. You can still use steps from this article but if you are new to API or want to avoid learning curve with API then use newer approach.

Please visit this page to see all Pre-Configured ready to use API connectors which you can use in SSIS API Source / SSIS API Destination OR API ODBC Driver (for non-SSIS Apps such as Excel, Power BI, Informatica).

In this post you will learn how to access Microsoft Graph API (Office 365 REST API / Sharepoint API) inside SSIS without any coding. Microsoft Graph API is a unified way to access many Microsoft services API including Office 365 API. In this post we will use SSIS JSON Source / REST API Connector to load data from Office 365 to SQL Server.

Use Case of Microsoft Graph API

Here are some use cases why you want to use Microsoft Graph API

Read/Write events from Outlook Calendar
Send email / Read emails
Get list of files from OneDrive
Upload/ Download files to OneDrive
Read/Write Excel Sheet (Use range or specific range of specific sheet)
Read Sharepoint Document library
Search content from Sharepoint document library
Get AD users for your Organization (AD Accounts)
Update OneNote
Track changes to Users, Events, Calendar Items using delta API

Register Application (OAuth2 App for Graph API)

The first step to access any Office 365 API / Graph API is to register an OAuth App in the Azure Portal. After following these steps, you will get the following two items to use in the next section:

Application Id
Application Secret

In this section, you will learn how to register a custom app in Microsoft Azure portal that will allow access to the OneDrive. So, let's get started -

Log into Microsoft Azure portal to register a custom app.
Register a new application by clicking New Registration link.
Provide the name of the custom app and who can access the app in the organization.
Go to the App overview and add a Redirect URL.
Click on "Add a Platform" under Platform Configuration section and then select "Web" under Web applications section to enter a Redirect URL.
```
https://zappysys.com/oauth
```
Create a Client Secret key which will be used to Authenticate the custom Azure app.
Note - Take a note of Client Secret, it will be required while configuring OAuth connection in the SSIS later
Add API Permissions.
Select following permissions from the Delegated Permissions section.
```
User.ReadBasic.All
Files.Read
offline_access
```
Take a note of Client ID, it will be required while configuring OAuth connection in the SSIS later.

Step-By-Step : Access Outlook.com Emails using SSIS

Now lets look at examples to access your outlook.com emails using SSIS. You can use same techniques to call pretty much any Office 365 API.

Configure SSIS OAuth Connection for Graph REST API

First step to access any Graph API is configure OAuth connection

Open Visual Studio
Open existing SSIS Project or create new using File > New > Project > Choose “Integration Services Project” Type under Business Intelligence template category
Open Package
Right click inside Connection Managers area and click “New Connection…”
From the connection type list select “ZS-OAUTH” connection type.
Create new SSIS OAuth API Connection Manager
On the OAuth Connection Manager configure following options
1. Select “Custom” from Provider dropdown
2. Select OAuth2 from OAuth version
3. Enter your Application Id and Secret Key (i.e. App Password) obtained in the previous section from here.
4. Enter following URL in the Authorization URL
```
https://login.microsoftonline.com/common/oauth2/v2.0/authorize
```
5. Enter the following URL in the token URL field
```
https://login.microsoftonline.com/common/oauth2/v2.0/token
```
6. Enter the following Graph API Scopes (each scope must be entered in a new line).
```
user.read
mail.read
offline_access
```
  NOTE: Always include offline_access scope returns refreh_token which allows you to renew token without going through login process again. For more information about which scopes needed always refer to API help page (e.g. if you want to read mail using this API then you will see it says you can use Mail.Read or Mail.ReadWrite scope).
7. Go to Advanced tab and enter the following URL in the Callback/Return URL (Assuming exact same URL used when you registered App in the previous section)
```
https://zappysys.com/oauth
```
8. Now go back to first tab and Click Generate Token. When prompted Login using your Personal Microsoft Account or Work Account (Office 365 or AD login)
9. After login you will see Accept option just click it. If things go right then you will see Access Token and Refresh Token fields will be populated.
Once all configured you can click Test to make sure connection works.
SSIS OAuth Connection Manager for Office 365 REST API / Microsoft Graph API

NOTE: Typically RefreshToken is longed lived but in some cases they may expire soon too. If that’s the case then you can configure above oauth connection for Changing RefreshToken pattern. Simply enter token file path and re-authenticate by clicking Generate Token. After that each time you make an API call it will save new refresh token in file.

Configure SSIS JSON/REST API Source

Once you configure OAuth connection you are ready to read data from Office 365 API. Lets look at step by step how to read emails using Mail API and save to SQL Server.

Drag and drop Data flow task from SSIS toolbox
Goto data flow designer and drag and drop ZS JSON Source/REST API Connector on the designer
Double click to edit JSON Source.
In the URL field enter below URL
```
https://graph.microsoft.com/v1.0/me/messages
```
To customize URL with additional parameters check this help link. It’s a standard OData Protocol so you can leverage many common OData features (e.g. Use $top parameters to set pageSize. Default is 10 for List Mail but you can include more records per response by changing it such as …/messages?$top=50).
Check Use credentials option
From the connections dropdown select OAuth Connection manager created in the previous section
Change Data Format option to OData
In the Filter enter following or click “Select Filter” and select Value Array node > Click OK.
```
$.value[*]
```
In the Max rows set to 200 so we don’t pull all data for now 🙂 … This is our hello world example.
Now click Preview to see data. If all is well then Save UI by clicking OK
SSIS JSON Source – Get data from Microsoft Graph API (Office 365 API – Read Mail Example)

Loading Graph API / Office 365 data to SQL Server

Now last step is loading Outlook data to SQL Server. For example purpose we will load data into SQL Server but you can load into any Target (e.g. Flat file, Oracle, Excel) using Microsoft or ZappySys Destination connectors

To load Office 365 data into SQL Server perform the following steps

Drag OLEDB destination on Data Flow surface
Connect ZS JSON Source to Destination
On OLEDB destination select / create new SQL Connection and then Click “New Table”
Click on Mappings tab and click OK to save
Execute Package

Configure SSIS OLEDB Destination – Loading Xero Data into SQL Server Table

Graph API / Office 365 to SQL Server Column Mappings for OLEDB Destination

Loading Graph REST API / Office 365 data to SQL Server in SSIS

POST data to Graph API (Insert or Update)

So far we have seen how to read data from Office 365 API. Now let’s look at how to write data to Office 365.

There are two ways you can achieve this.

Use Web API Destination in Data Flow
Use REST API Task in Control Flow

Read this article to learn more about API POST using SSIS

Using SSIS Web API Task (Write data from SQL Server to Office 365 / Graph API)

Using REST API Task to POST data to Graph API

If you have JSON/XML data already prepared and if you like to POST it to Graph API then REST API Task would be easy to use.

REST API Task Body can be direct string, variable (e.g. {{User::varSomeData}} or Body can come from File

SSIS REST API Task – POST data to Graph API

Download file from OneDrive

To download file from OneDrive perform these steps

https://zappysys.zendesk.com/hc/en-us/articles/115005010573-How-to-download-file-from-OneDrive-using-Graph-API

Upload file to OneDrive

https://zappysys.zendesk.com/hc/en-us/articles/115004893713-How-to-upload-file-to-OneDrive-using-Office-365-Graph-API

Reading Office Excel Sheets from OneDrive

JSON Source also support reading API response which returns 2D arrays. Such as Office Excel Spreadsheet API to read excel data. Check this blog post to learn similar scenario to parse 2D arrays. There are few other options for parsing 2D arrays which are documented here.

https://graph.microsoft.com/v1.0/me/drive/items/01CYZLFJDYxxxxxxx/workbook/worksheets('test')/range(address='A1:B200')

Testing Office 365 API using Graph API Explorer

If you like to test API request/response inside browser then Microsoft provides really great tool to called Graph API Explorer. Click here to try Graph API Explorer.

See below screenshot

Common Errors

Truncation related error

The most common error you may face when you run an SSIS package is truncation error. During the design time only 300 rows are scanned from a source (a file or a REST API call response) to detect datatypes but at runtime, it is likely you will retrieve far more records. So it is possible that you will get longer strings than initially expected. For detailed instructions on how to fix common metadata related errors read an article "How to handle SSIS errors (truncation, metadata issues)".

Authentication related error

Another frequent error you may get is an authentication error, which happens when you deploy/copy a package to another machine and run it there. Check the paragraph below to see why it happens and how to solve this problem.

Deployment to Production

In SSIS package sensitive data such as tokens and passwords are by default encrypted by SSIS with your Windows account which you use to create a package. So SSIS will fail to decrypt tokens/passwords when you run it from another machine using another Windows account. To circumvent this when you are creating an SSIS package which uses authentication components (e.g. an OAuth Connection Manager or an HTTP Connection Manager with credentials, etc.), consider using parameters/variables to pass tokens/passwords. In this way, you won’t face authentication related errors when a package is deployed to a production server.

Check our article on how to configure packages with sensitive data on your production or development server.

Conclusion. What’s next?

In this article we have learned how to load data from Microsoft Graph API to SQL Server using SSIS ( drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from Outlook Mail using OAuth. JSON Source makes it super simple to parsing complex / large JSON Files or any Web API Response into rows and column so you can load into database like SQL Server. Download SSIS PowerPack to try many other automation scenarios not discussed in this article.

Keywords:

The post How to use Office 365 API with SSIS (Mail, Calendar, Contacts, OneDrive, Excel) appeared first on ZappySys Blog.