In our previous article, we saw how to get Office 365 data in Power BI. Now, let’s look at how to get Office 365 Mail Attachments using SSIS without any coding. Microsoft Graph API is a unified way to access many Microsoft services, including the Office 365 API. In this post, we will use the SSIS JSON/REST API Source Connector to retrieve the Outlook Messages list and its attachments.

Use Case of Microsoft Graph API

Here are some use cases for why you want to use the Microsoft Graph API

• Read/Write events from Outlook Calendar
• Send email  / Read emails
• Get the list of files from OneDrive
• Upload/ Download files to OneDrive
• Read/Write Excel Sheet (Use range or a specific range of a specific sheet)
• Read the SharePoint Document library
• Search content from the SharePoint document library
• Get AD users for your Organization (AD Accounts)
• Update OneNote
• Track changes to Users, Events, Calendar Items using the delta API

Step-By-Step to call the Microsoft Graph Rest API Using SSIS

Now, let’s look at examples to access your Outlook.com emails using SSIS. You can use the same techniques to call pretty much any Office 365 API.

Configure SSIS OAuth Connection for Graph REST API

The first step to access any Graph API is to configure an OAuth connection

1. Open Visual Studio
2. Open an existing SSIS Project or create a new one using File > New > Project > Choose “Integration Services Project” Type under Business Intelligence template category.
3. Create Package
4. Right-click inside the Connection Managers area and click New Connection…
5. From the connection type list, select the ZS-OAUTH connection type.
   

   Create a new SSIS OAuth API Connection Manager

6. On the OAuth Connection Manager, configure the following options
   1. Select “Custom” from the Provider dropdown
   2. Select OAuth2 from the OAuth version
   3. Enter your Application Id and Secret Key (i.e., App Password) obtained in the previous section from here.
   4. Enter the following URL in the Authorization URL
      

      https://login.microsoftonline.com/common/oauth2/v2.0/authorize

   5. Use the following URL in the token URL field
      

      https://login.microsoftonline.com/common/oauth2/v2.0/token

   6. Enter the following Graph API Scopes (each scope must be entered on a new line).
      

      user.read
      mail.read
      offline_access

      NOTE: Always include the offline_access scope, which returns a refresh_token, which allows you to renew the token without going through the login process again. For more information about which scopes are needed, always refer to the API help page (e.g., if you want to read mail using this API, then you will see it says you can use Mail.Read or Mail.ReadWrite scope).
   7. Go to the Advanced tab and enter the following URL in the Callback/Return URL (assuming the same URL used when you registered the App in the previous section)
      

      https://zappysys.com/oauth

   8. Now go back to the first tab and click Generate Token. When prompted, log in using your Personal Microsoft Account or Work Account (Office 365 or AD login)
   9. After logging in, you will see the Accept option. Just click it. If things go right, then you will see the Access Token and Refresh Token fields populated.
7. Once everything is configured, you can click Test Connection to verify the connection works.
   

   SSIS OAuth Connection Manager for Office 365 REST API Microsoft Graph API

Download the Messages Attachment Files from Outlook

Let’s start with an example. We use the SSIS JSON Source and Web API Destination components to make a call to the Microsoft Graph REST API.

1. First of all, drag and drop the Data Flow Task from the SSIS Toolbox and double-click it to edit.
   

   Drag and drop Data flow

2. From the SSIS toolbox, drag and drop the JSON Source onto the Data Flow Designer surface.
   

   SSIS JSON Source – Drag and Drop

3. Double-click the JSON Source, then enter the following URL to invoke the Messages API call with search options to retrieve emails with attachments.
   

   https://graph.microsoft.com/v1.0/me/messages?$select=sentDateTime,from,subject,hasAttachments&$search="hasAttachments:True"

   Set the filter to $.value[*] and click the preview.
   

   SSIS JSON Source – Get data from Microsoft Graph API (Office 365 API – Read Mail Example)

   To customize the URL with additional parameters, check this help link. It’s a standard OData Protocol, so you can leverage many common OData features (e.g., use $top parameters to set pageSize. Default is 10 for List Mail, but you can include more records per response by changing it, such as …/messages?$top=50).

4.  Now, let’s use Template Transform to set the URL using the message ID to make the Get attachment API call.

   https://graph.microsoft.com/v1.0/me/messages/<%id%>/attachments

   

   SSIS Template Transform sending attachment id
5. Now we need to use the Web API Destination to make the Get Attachment API call by message ID. Select the Input column for URL as that TemplateOutput.
   

   SSIS Web API Destination – Get data from Microsoft Graph API (Office 365 API – Get Attachment Example)

6. Now, the next step is to parse the JSON Response String of the get attachment api call. For that, we need to use a JSON Parser. Select the input as Web PAI Destination ResponseText and use the below sample JSON string to configure the metadata(Columns).
   

   {"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#users('support%40zappysys.com')/messages('AQMkADAwATMwMAItMDkyMC0zZTg4AC0wMAItMDAKAEYAAAP_JUy')/attachments"
   ,"value":[{"@odata.type":"#microsoft.graph.fileAttachment","@odata.mediaContentType":"text/plain"
   ,"id":"AQMkADAwATMwMAItMDkyMC0zZTg4AC0wMAItMDAKAEYAAAP_JUy_27M"
   ,"lastModifiedDateTime":"2020-02-21T07:36:16Z"
   ,"name":"Test.txt","contentType":"text/plain","size":870,"isInline":false
   ,"contentId":null,"contentLocation":null,"contentBytes":"VGVzdCBNZXNzYWdl"}]}

   

   JSON Parser Transform – using JSON example
7.  Now, in the JSON Parser, go to the Go-to columns Tab and set the contentBytes DataType to DT_IMAGE, and increase the other columns’ DT_WSTR lengths by 1000 and 1500. We set contentBytes to the DT_IMAGE  datatype, which is like varbinary(MAX).
   

   JSON Parser Transform Columns Tab fix the datatype

8. Furthermore, in the next step, we need to use an SSIS Derived Column to set the FilePath where we want to save the attachments, and then concatenate the local file path with the attachment name column.
   

   SSIS Derived Column: Add New Column

9. Finally, we need to use the Export Column Transform (Native) to export attachment bytes to a local file. It needs to know 2 upstream info Bytes and the target file path.
   

   Export Column configuration

10. Make sure to attach the export column to some destination (e.g., our ZS Trash Destination); the engine might remove it. At the same time, it optimizes the runtime. It may remove all transformations that don’t have a destination, such as deploying the package to SQL Server, or when you set optimize in Visual Studio.
11. That’s it, execute the package, and it will download all the email attachments.
    

    Downloaded Mail Attachments

Conclusion

After all, in this article, we learned how to make a Microsoft Graph API REST API call. We used the SSIS JSON / REST API Connector to extract data from Outlook Mail using OAuth. Made the Get attachment API call using the Web API Destination and parsed the JSON response using the JSON Parser. And using the Native SSIS Export Column, we saved the export attachment’s bytes to a local file. To explore many other scenarios not discussed in this article, download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • Help File: Web API Destination
  • Help File: JSON Parser Transform
  • Overview of the Microsoft Graph API

The post How to Get Office 365 Mail Attachments using SSIS appeared first on ZappySys Blog.

In this article, you will learn Using SSIS Trash Destination and Dummy Data Source (FREE). You can write data into the file using Trash Destination, You can use Trash Destination to write data into a raw file from any source. We will use FREE Dummy Data Source to generate random data for testing purpose. Trash Destination can be used to terminate your dataflow pipeline without doing any expensive processing on incoming data. This component is very useful during testing where you want to focus on the source and transforms but don’t really care about the destination.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).

Components mention in this article

	ZS Trash Destination
	ZS Dummy Data Source

Getting Started

SSIS ZS Trash Destination and Dummy Data Source are free. If you check save Data to file then by default all input columns are logged to file. If you wish to log the selected input column then use Advanced Editor. This component does not support an error output. Any bad random value ill cause error or Truncation (e.g. If your sample value is 5 characters long but output column length is set to 3 chars). If you have error output attached with RedirectRow option on error, Truncation then you can capture bad rows along with an exact reason of error.

Write data into the file using Trash Destination

Overview

In this tutorial, you will learn how to Using SSIS Trash Destination and Dummy Data Source.  Let’s use ZS DummyData Source to write data into ZS Trash Destination. Here you can select our, any of suitable Source Adapter.

Method 1 – Using ZS DummyData Source

1. Open Visual Studio and Create New SSIS Package Project.
2. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

3. Double click on the Data Flow task to see the Data Flow designer surface.
4. From the SSIS toolbox drag and drop ZS DummyData Source on the data flow designer surface.
   

   Drag and Drop ZS DummyData Source

5. Double click DummyData Source and select Customer profile from Template dropdown.
6. Once you change Template you will notice that a lot of columns automatically added. Notice how sample values are set for many columns along with their value delimiter. You may use many inbuilt placeholders in your sample values such as ({NULL} for a null value, {TAB} for the tab character, {CRLF} for new line).
7. Change [How many rows] setting to 5000.
   

   SSIS DummyData Source – Configure

8. Click OK to save settings.
9. We can use DummyData Source like this, click here.
10. Here, you can insert data into SQL Server Table, Check Here.
11. Now, Just Drag and Drop Our Free ZS Trash Destination on the data flow designer surface from SSIS Toolbox.
    

    SSIS Trash Destination – Drag and Drop

12. Now single click on ZS DummyData Source once you see the blue arrow from the source … connect it to ZS Trash Destination.
13. Double click on ZS Trash Destination to configure it.
    

    SSIS Trash Destination – Configure

14. Click OK to save configure setting UI.
15. Right-click on the path and Add Data Viewer.
16. Execute the package and verify source data in the data viewer.
    

    Execute Package and Verify Data Source

Method 2 – Using OLE DB Source

Let’s use ZS OLE DB Source to write data into ZS Trash Destination.

1. Open Visual Studio and Create New SSIS Package Project.
2. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

3. We are going through SQL Server Database, need OLE DB Connection for Data Source.

How to create OLE DB Connection.

1. Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
   

   Create OLE DB Connection

2. Now, click on New button for create Connection.
   

   Create New OLE DB Connection

3. In the Connection Manager, just follow the steps in the following image we have created.
   

   OLE DB Connection Configure

4. Click on OK button to save connection configure setting UI.

How to write into Trash Destination from OLE DB Source

1. Drag and Drop OLE DB Source from SSIS toolbox.
   

   OLE DB Source – Drag and Drop

2. Double click on OLE DB Source for configure it.
3. In the Connection Manager, just follow the steps in the following image we have created.
   

   OLE DB Source – Configure

4. Click on OK button to save OLE DB Source Editor setting UI.
5. Now, Just Drag and Drop Our Free ZS Trash Destination on the data flow designer surface from SSIS Toolbox.
   

   SSIS ZS Trash Destination – Drag and Drop

6. Now, single click on OLE DB Source once you see the blue arrow from the source … connect it to ZS Trash Destination.
7. Double click on ZS Trash Destination to configure it.
   

   SSIS ZS Trash Destination – Configure

8. Click OK to save configure setting UI.
9. Right-click on the path and Add data viewer.
10. Execute the package and verify source data in the data viewer.
    

    Execute Package and Verify Data

Conclusion

In this blog, we learned how to write data into a raw file and read data in SSIS. We used the Trash Destination for write data and Dummy Data Source to read data. You can download SSIS PowerPack from here to try many other scenarios not discussed in this blog. SSIS PowerPack is a collection of 70+ high performance, just drag and drop connector/task for SSIS(i.e. Microsoft SQL Server Integration Service). After you install SSIS PowerPack you will see many new Custom Tasks and Components in your Visual Studio SSIS Toolbox.

References

Finally, you can use the following links for more information:

• ZS Trash Destination
• ZS Dummy Data Source
• Help File:
  • ZS Trash Destination
  • ZS Dummy Data Source

The post Using SSIS Trash Destination and Dummy Data Source appeared first on ZappySys Blog.

In this post we will show you how to convert varbinary to Base64 in SSIS. We will use FREE Tasks provided by ZappySys.

Preparing Sample Data

First let’s create a sample table with some Varbinary datatype. Run following command in SSMS to create a sample table with one sample row.

create table binarydata(
 data varbinary(max)
)
go
insert into binarydata(data )
values(0x5468697320697320612074657374) /*binary of "This is a test"*/

Method 1 – Using Template Transform

Let’s use Template Transform to convert Varbinary data to Base64 format. When you use columns with binary data in Template transform it automatically converts to Base64 by default.

1. Go to Data flow
2. Drag your OLEDB Source and configure it to outputs SQL Data with Varbinary column  (e.g. select data from dbo.binarydata).
3. Now drag ZS Template Transform from SSIS Toolbox
4. Connect OLEDB Source to Template Transform
5. Double click Template Transform  to configure. Now you can click Insert Placeholders > Columns > Select your Upstream column name (which has Varbinary data)
6. You will see Template text as  <%MyVarBinaryColumn%>
7. Click OK to Save Template Transform
8. Now connect Template transform to any Destination e.g. OLEDB Destination.  You can Map TemplateOutput column to target column. This will contain Base64 representation of Varbinary data. For example purpose we used ZS Trash destination (Check save to file)

Convert Varbinary to BASE64 using SSIS Template Transform

Save Base64 string to Variable

In previous step we saw how Template Transform can convert binary data (i.e. Varbinary datatype) to BASE64 string. Now let’s look at how to save this value to Variable using Set Variable Transform.

Using SSIS Set Variable Transform in Data flow (Capture binary data as base64 example)

Method 2 – Using Trash Destination

Another way to dump Binary data to CSV file is use ZS Trans Destination. See previous screenshot. If you map OLEDB Source directly to Trash destination without Template Transform then you will see something like below.

data
VGhpcyBpcyBhIHRlc3Q=

The post How to convert Varbinary to Base64 in SSIS appeared first on ZappySys Blog.