OneDrive allows to store files on the cloud and and we have seen growing requirements in downloading/Uploading files from there. This post will provide step by step details of how to Upload/Download files from OneDrive.

Register Microsoft App for OAuth Authentication

In this section, you will learn how to register a custom app in Microsoft Azure portal that will allow access to the OneDrive. So, let’s get started –

1. Log into Microsoft Azure portal to register a custom app.
2. Register a new application by clicking New Registration link.
   

   New App Registration in Azure portal

3. Provide the name of the custom app and who can access the app in the organization.
   

   Register an OneDrive App

4. Go to the App overview and add a Redirect URL.
   

   Add a Redirect URL

5. Click on “Add a Platform” under Platform Configuration section and then select “Web” under Web applications section to enter a Redirect URL.
   

   https://zappysys.com/oauth

   

   Redirect URL

6. Create a Client Secret key which will be used to Authenticate the custom Azure app.
   

   Add a Client Secret

   

   Secret Key Expiration Period

   

   Specify Secret Key

   Note – Take a note of Client Secret, it will be required while configuring OAuth connection in the SSIS later.
7. Add API Permissions for OneDrive API.
   

   OneDrive API Permissions

8. Select the following permissions from the Delegated Permissions section.
   

   User.ReadBasic.All
   Files.Read
   offline_access

   

   Select Delegated Permissions

9. Take note of Client ID, it will be required while configuring OAuth connection in the SSIS later.
   

   Client ID

Get OneDrive File ID

In this section, you will learn how to get the File ID of the CSV file through OneDrive API. So, let’s get started –

1. Go to the Microsoft Graph Explorer.
2. Sign in to Graph Explorer.
   

   SSIS GraphExplorer SignIn

3. Run below OneDrive API to get a list of the files from the OneDrive.
   

   https://graph.microsoft.com/v1.0/me/drive/root/children

   

   Microsoft Graph Explorer

   NOTE: Take a note of File ID of the CSV file. For this example, we will download invoices.csv file.

Create OAuth Connection in SSIS

In this section, you will learn how to create an OAuth connection in SSIS to an Azure custom app. So, let’s get started –

1. Create a new ZS-OAuth connection in SSIS.
   

   ZappySys OAuth Connection in SSIS

2. Configure OAuth connection with following settings.

   Client ID	Provide Client ID from App Registration steps
   Client Secret	Provide Client Secret from App Registration steps
   Authorization URL	https://login.microsoftonline.com/organizations/oauth2/v2.0/authorize
   Access Token URL	https://login.microsoftonline.com/organizations/oauth2/v2.0/token
   Scope/Permissions	User.ReadBasic.All Files.Read.All offline_access

   

   OAuth Connection Configuration

3. Go to the Advanced tab and provide Return URL.
   

   https://zappysys.com/oauth

   

   Add Redirect URL

4. Click on Generate Token will give a login screen which will allow to generate access and refresh token for the app.
5. Approve the requested permissions bu Accepting it.
   

   Approve Permissions Requested

6. Token has been generate successfully.
   

   Refresh Token Successful

7. Let’s test the connection to make sure it is working fine.
   

   Connection Successful

Download CSV File Through OneDrive REST API in SSIS

In this section, you will learn how to download a CSV file through OneDrive API in SSIS and save it into a CSV file.

1. Here is the OneDrive API which allows to read content of the file.
   

   https://graph.microsoft.com/v1.0/me/drive/Items/<File ID>/content

2. Replace <File ID> in above step#1 with the file ID that we have got in the Get OneDrive File ID section. Final URL will look something like below.
   

   https://graph.microsoft.com/v1.0/me/drive/Items/01W7L3VCUBS27ILUQKENAK4LQGGTJ2R3TT/content

3. Let’s create a SSIS package with Data Flow Task from SSIS Toolbox.
4. Replace <File ID> in above step#1 with the file ID that we have got in the Get OneDrive File ID section. Final URL will look something like below.
   

   https://graph.microsoft.com/v1.0/me/drive/Items/01W7L3VCWMCGPYZLM2IRAYWROSH4XMQT7A/content

5. Configure REST API task as follows.
   

   Binary File Configuration in ZappySys REST API Task

6. Configure the responses setting as below.
   

   Binary File Configuration in ZappySys REST API Task

   NOTE: Ensure that the Treat response as the binary checkbox is checked otherwise the file will be downloaded but corrupted as zip/mp3 etc. file format is binary.
7. Check if the request is made correctly by clicking Test Request/Response.
   

   Example of Testing the Request

8. Let’s have a look at the folder where the file has been downloaded. Here is an example –
   

   Downloaded File Example

Download a Text File Through OneDrive API in SSIS

Here is another technique that can be used to download a CSV/text file through OneDrive API without knowing it’s file ID but file name. Please check this link for more details.

Download Excel Sheets Through OneDrive API in SSIS

JSON Source also supports reading API response which returns 2D arrays. Such as Office Excel Spreadsheet API to read excel data. Check this blog post to learn a similar scenario to parse 2D arrays. There are few other options for parsing 2D arrays which are documented here.

https://graph.microsoft.com/v1.0/me/drive/items/01CYZLFJDYxxxxxxx/workbook/worksheets('test')/range(address='A1:B200')

Upload a File Through OneDrive API in SSIS

In this section, you will learn how to upload a file through OneDrive API in SSIS. Uploading a file has few additional steps and has been documented separately. Please check this link for more details.

Delete a File Through OneDrive API in SSIS

In this section, you will learn how to delete a file from OneDrive through OneDrive APIs in SSIS.

1. Add a REST API task and configure as follows. Make sure to use the File ID of the file which needs to be deleted from OneDrive. Example URL –
   

   https://graph.microsoft.com/v1.0/me/drive/Items/01W7L3VCSEADQ457W54RBI3I47YGFJOD2I

   

   Delete OneDrive File

2. Execute the task and you will have file deleted from OneDrive.

Conclusion

In this article, we have learned how to Upload/Download/Delete data/file from OneDrive API in SSIS. We used SSIS REST API Task to extract data through OneDrive API in SSIS. Download SSIS PowerPack to try many other automation scenarios not discussed in this article.

The post How to call OneDrive API in SSIS (Upload, Download) appeared first on ZappySys Blog.

In this article, we will get Office 365  data in Power BI using Microsoft Graph API and ODBC drivers. Specifically, we will get Excel file data in OneDrive, events from a Calendar and finally, a list in a note in OneNote and then load that data into a Power BI report. We will achieve that by creating an ODBC data source using ZappySys ODBC PowerPack and then querying corresponding Microsoft Graph API methods. ODBC PowerPack allows to query REST/SOAP APIs in an easy way and lets to view responses in a tabular format, which will be perfect to accomplish our goal.

Prerequisites

1. A Power BI account.
2. Wield basic Power BI skills.
3. Power BI Desktop installed.
4. ZappySys ODBC PowerPack installed.

Step By Step – Get Office 365 data in Power BI using Microsoft Graph API and ODBC

Set up an ODBC data source based on ZappySys JSON Driver

Add a data source

Configure the data source

Set up an ODBC data source based on ZappySys XML Driver

1. Before adding another data source based on ZappySys XML Driver, copy the connection string of the data source we just created:
2. Close the window and then add a new data source, but this time based on ZappySys XML Driver.
3. Once a window is opened, give it a name, e.g. System - XML.
4. Then press Load connection string button to load the connection string.
5. Paste the copied connection string and replace JSON text to XML, so that you have { ZappySys XML Driver } in DRIVER property.
6. Close the window.

Create a Power BI report

Reading files from OneDrive (getting Excel data)

Let’s say you have Stock.xlsx stored in Sample Documents folder, OneDrive with this kind of data and you want to show it in a Power BI report:

To have that data in a Power BI report, perform these steps:

1. Open Power BI Desktop and create a new Power BI report.
2. Create a data source based on System – JSON DSN (since we get a JSON response):
   
3. Then in SQL statement area enter this query:
   SELECT * FROM $
WITH(Src='https://graph.microsoft.com/v1.0/me/drive/root:/YOUR FOLDER/Stock.xlsx:/workbook/worksheets/SHEET NAME/usedRange'
,Filter='$.values[1:]'
,ArrayTransformType='TransformSimpleTwoDimensionalArray'
,ArrayTransColumnNameFilter='$.values[0][*]'
)
Using WITH clause lets us override the properties. You can generate the query above using the Query Builder:
   In this step we are overriding Url, Filter and array transformation fields you configured in JSON DSN. The Src value overrides the URL. The Filter property enables us to select the data we want to see in the report. We use [1:] in $.values[1:] to skip the first row because the first row contains the headers and we want only the data. You can configure this option in the Filter Options tab (read more about JSON path).We also have to make the transformation on the JSON and turn JSON arrays into rows. To accomplish that, we override ArrayTransformType. With ArrayTransColumnNameFilter property and [0][*] text, we specify that the first array of parent array will be the headers of the columns. You can read more about transformations in Parse multi-dimensional JSON array article.
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other OneDrive endpoints.
4. Click OK.
5. Drag a table into the report, select the columns and you should see a similar view:
   

Reading calendar entries

Now, let’s say you use a calendar in Office 365 and you want your appointments to be visible in a Power BI report:

1. To start, create a new Power BI report.
2. Then add a data source based on System – JSON DSN (again, we get the response in JSON):
   
3. In SQL statement section enter this query:
   SELECT * FROM $
WITH(Src='https://graph.microsoft.com/v1.0/me/events?$select=subject,start,end,location'
,Filter='$.value[*]'
,Header='Prefer: outlook.timezone="FLE Standard Time"'
)Here we also are overriding HTTP Header so that we get the event times not in UTC/GMT but for the specified timezone. It must be specified in Prefer: outlook.timezone=”{time-zone-string}” format (check an article on how to get a list of supported time zones).
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other Calendar endpoints.
4. Click OK.
5. Drag a table control into the report, select several columns and you should see a similar view:
   

Reading notes from OneNote

Suppose, you have these notes in OneNote and you want “Groceries list” to be shown in a Power BI report:

1. First, we will need to find NOTE ID we are trying to get. To do that, open ODBC Administrator, find System – JSON data source and hit Configure.
2. Then go to the Preview tab, and input this SQL query and hit Preview Data:
   SELECT * FROM $
WITH (Src='https://graph.microsoft.com/v1.0/me/onenote/pages'
,Filter='$.value[*]'
)
3. Once you get a result set, find your note name and copy its ID:
   
4. Then create a new Power BI report.
5. Create a new data source based on System – XML DSN (we are using XML-based DSN because the response will be in HTML):
   
6. In SQL statement section enter this query (use the NOTE ID you retrieved in the previous step):
   SELECT * FROM $
WITH(ElementsToTreatAsArray='div,p'
,Src='https://graph.microsoft.com/v1.0/me/onenote/pages/NOTE ID/content'
,Filter='$.html.body.div[0].p[1:]'
)

   
   Note: Use Microsoft Graph API Explorer to make a call and see what response you get. You can try calling other OneNote endpoints.
7. Click OK.
8. Drag and drop Multi-row card visual, select #text column and you should see a similar view:
   
9. We finished! Now you can publish your Power BI reports to Power BI service.

Conclusion

Today we learned how to call Microsoft Graph API methods using ZappySys ODBC PowerPack and load response data into Power BI without any ETL procedures. Particularly, we used OneDrive, Calendar and OneNote Graph API endpoints to achieve that. We created two ODBC data sources based on JSON and XML ODBC Driver to query those endpoints, which let us read data directly in a Power BI report. Finally, we examined how to refresh data source on Power BI using on-premises Power BI gateway.

References

https://powerbi.microsoft.com/en-us/

https://apps.dev.microsoft.com/

https://developer.microsoft.com/en-us/graph/graph-explorer/

https://docs.microsoft.com/en-us/graph/overview

https://docs.microsoft.com/en-us/graph/onedrive-concept-overview

https://docs.microsoft.com/en-us/graph/outlook-calendar-concept-overview

https://docs.microsoft.com/en-us/graph/integrate-with-onenote

The post Get Office 365 data in Power BI using Microsoft Graph API and ODBC appeared first on ZappySys Blog.

In this post you will learn how to access Microsoft Graph API  (Office 365 REST API / Sharepoint API) inside SSIS without any coding. Microsoft Graph API is a unified way to access many Microsoft services API including Office 365 API. In this post we will use SSIS JSON Source / REST API Connector to load data from Office 365 to SQL Server.

Use Case of Microsoft Graph API

Here are some use cases why you want to use Microsoft Graph API

• Read/Write events from Outlook Calendar
• Send email  / Read emails
• Get list of files from OneDrive
• Upload/ Download files to OneDrive
• Read/Write Excel Sheet (Use range or specific range of specific sheet)
• Read Sharepoint Document library
• Search content from Sharepoint document library
• Get AD users for your Organization (AD Accounts)
• Update OneNote
• Track changes to Users, Events, Calendar Items using delta API

Step-By-Step : Access Outlook.com Emails using SSIS

Now lets look at examples to access your outlook.com emails using SSIS. You can use same techniques to call pretty much any Office 365 API.

Configure SSIS OAuth Connection for Graph REST API

First step to access any Graph API is configure OAuth connection

1. Open Visual Studio
2. Open existing SSIS Project or create new using File > New > Project > Choose “Integration Services Project” Type under Business Intelligence template category
3. Open Package
4. Right click inside Connection Managers area and click “New Connection…”
5. From the connection type list select “ZS-OAUTH” connection type.
   

   Create new SSIS OAuth API Connection Manager

6. On the OAuth Connection Manager configure following options
   1. Select “Custom” from Provider dropdown
   2. Select OAuth2 from OAuth version
   3. Enter your Application Id and Secret Key (i.e. App Password) obtained in the previous section from here.
   4. Enter following URL in the Authorization URL
      

      https://login.microsoftonline.com/common/oauth2/v2.0/authorize

   5. Enter the following URL in the token URL field
      

      https://login.microsoftonline.com/common/oauth2/v2.0/token

   6. Enter the following Graph API Scopes (each scope must be entered in a new line).
      

      user.read
      mail.read
      offline_access

      NOTE: Always include offline_access scope returns refreh_token which allows you to renew token without going through login process again. For more information about which scopes needed always refer to API help page (e.g. if you want to read mail using this API then you will see it says you can use Mail.Read or Mail.ReadWrite scope).
   7. Go to Advanced tab and enter the following URL in the Callback/Return URL (Assuming exact same URL used when you registered App in the previous section)
      

      https://zappysys.com/oauth

   8. Now go back to first tab and Click Generate Token. When prompted Login using your Personal Microsoft Account or Work Account (Office 365 or AD login)
   9. After login you will see Accept option just click it. If things go right then you will see Access Token and Refresh Token fields will be populated.
7. Once all configured you can click Test to make sure connection works.
   

   SSIS OAuth Connection Manager for Office 365 REST API / Microsoft Graph API

Configure SSIS JSON/REST API Source

Once you configure OAuth connection you are ready to read data from Office 365 API. Lets look at step by step how to read emails using Mail API and save to SQL Server.

1. Drag and drop Data flow task from SSIS toolbox
2. Goto data flow designer and drag and drop ZS JSON Source/REST API Connector on the designer
3. Double click to edit JSON Source.
4. In the URL field enter below URL
   

   https://graph.microsoft.com/v1.0/me/messages

   To customize URL with additional parameters check this help link. It’s a standard OData Protocol so you can leverage many common OData features (e.g. Use $top parameters to set pageSize. Default is 10 for List Mail but you can include more records per response by changing it such as …/messages?$top=50).
5. Check Use credentials option
6. From the connections dropdown select OAuth Connection manager created in the previous section
7. Change Data Format option to OData 
8. In the Filter enter following or click “Select Filter” and select Value Array node > Click OK.
   

   $.value[*]

9. In the Max rows set to 200 so we don’t pull all data for now 🙂 … This is our hello world example.
10. Now click Preview to see data. If all is well then Save UI by clicking OK
    

    SSIS JSON Source – Get data from Microsoft Graph API (Office 365 API – Read Mail Example)

Loading Graph API / Office 365 data to SQL Server

Now last step is loading Outlook data to SQL Server. For example purpose we will load data into SQL Server but you can load into any Target (e.g. Flat file, Oracle, Excel) using Microsoft or ZappySys Destination connectors

To load Office 365 data into SQL Server perform the following steps

1. Drag OLEDB destination on Data Flow surface
2. Connect ZS JSON Source to Destination
3. On OLEDB destination select / create new SQL Connection and then Click “New Table”
4. Click on Mappings tab and click OK to save
5. Execute Package

Configure SSIS OLEDB Destination – Loading Xero Data into SQL Server Table

Graph API / Office 365 to SQL Server Column Mappings for OLEDB Destination

Loading Graph REST API  / Office 365 data to SQL Server in SSIS

POST data to Graph API (Insert or Update)

So far we have seen how to read data from Office 365 API. Now let’s look at how to write data to Office 365.

There are two ways you can achieve this.

1. Use Web API Destination in Data Flow
2. Use REST API Task in Control Flow

Read this article to learn more about API POST using SSIS

Using SSIS Web API Task (Write data from SQL Server to Office 365 / Graph API)

Using REST API Task to POST data to Graph API

If you have JSON/XML data already prepared and if you like to POST it to Graph API then REST API Task would be easy to use.

REST API Task Body can be direct string, variable (e.g. {{User::varSomeData}} or Body can come from File 

SSIS REST API Task – POST data to Graph API

Download file from OneDrive

To download file from OneDrive perform these steps

https://zappysys.zendesk.com/hc/en-us/articles/115005010573-How-to-download-file-from-OneDrive-using-Graph-API

Upload file to OneDrive

https://zappysys.zendesk.com/hc/en-us/articles/115004893713-How-to-upload-file-to-OneDrive-using-Office-365-Graph-API

Reading Office Excel Sheets from OneDrive

JSON Source also support reading API response which returns 2D arrays. Such as Office Excel Spreadsheet API to read excel data. Check this blog post to learn similar scenario to parse 2D arrays. There are few other options for parsing 2D arrays which are documented here.

https://graph.microsoft.com/v1.0/me/drive/items/01CYZLFJDYxxxxxxx/workbook/worksheets('test')/range(address='A1:B200')

Testing Office 365 API using Graph API Explorer

If you like to test API request/response inside browser then Microsoft provides really great tool to called Graph API Explorer. Click here to try Graph API Explorer.

See below screenshot

Common Errors

Deployment to Production

Conclusion. What’s next?

In this article we have learned how to load data from Microsoft Graph API to SQL Server using SSIS ( drag and drop approach without coding). We used SSIS JSON / REST API Connector to extract data from Outlook Mail using OAuth. JSON Source makes it super simple to parsing complex / large JSON Files or any Web API Response into rows and column so you can load into database like SQL Server. Download SSIS PowerPack to try many other automation scenarios not discussed in this article.

Keywords:

Office 365 API Integration with SQL Server | How to extract Office 365 data in SSIS? | How to read outlook mail, calendar event and contacts? | Calling Microsoft Graph API using SSIS. | Office 365 to SQL Server | SQL Server to Office 365 | SSIS Office 365 API Integration | SSIS Graph API Integration

The post How to use Office 365 API with SSIS (Mail, Calendar, Contacts, OneDrive, Excel) appeared first on ZappySys Blog.