In this article, you will learn how to download Outlook (Office 365) emails as .EML or .MSG files using ZappySys SSIS PowerPack without writing any code. We’ll leverage the power of Microsoft Graph API, which provides unified access to Microsoft 365 services like Outlook Mail, and integrate it seamlessly using the API Connector for SSIS Outlook Mail (Office 365), Web API Destination and other components. By the end, you’ll be able to automate the process of extracting emails and saving them to disk in standard email formats directly from your SSIS package.

Step 1: Register an Application in Azure Portal

To access Outlook emails via Microsoft Graph API, the first step is to register an application in the Azure Portal. This allows your SSIS package to authenticate securely and retrieve mail data from Office 365.
Step by step info is available here Outlook Mail (Office 365) Connector Authentication section.

User Credentials

Use delegated access (User Credentials) whenever you want to let a signed-in user work with their own resources or resources they can access. Whether it’s an admin setting up policies for their entire organization or a user deleting an email in their inbox, all scenarios involving user actions should use delegated access.
Refer to this links for the detailed steps: User Credentials [OAuth]

Application Credentials

Application-only access is broader and more powerful than delegated access (User Credentials), so you should only use app-only access where needed. Use it when:

1. The application needs to run in an automated way, without user input (for example, a daily script that checks emails from certain contacts and sends automated responses).
2. The application needs to access resources belonging to multiple different users (for example, a backup or data loss prevention app might need to retrieve messages from many different chat channels, each with different participants).
3. You find yourself tempted to store credentials locally and allow the app to sign in ‘as’ the user or admin.

Refer to this links for the detailed steps: Application Credentials [OAuth]

Step 2: Download Outlook (Office 365) Emails as .EML Files using SSIS

Now let’s look at some examples of how to access your Outlook emails using our easy-to-use Outlook Mail Connector in ZappySys SSIS PowerPack. This connector simplifies calling the Microsoft Graph API, so you can retrieve emails with just a few clicks.
In this section we will learn how to configure and use Outlook Mail (Office 365) Connector in API Source to extract data from Outlook Mail (Office 365).

1. Begin with opening Visual Studio and Create a New Project.
2. Select Integration Service Project and in new project window set the appropriate name and location for project.
   And click OK.
3. In the new SSIS project screen you will find the following:
   1. SSIS ToolBox on left side bar
   2. Solution Explorer and Property Window on right bar
   3. Control flow, data flow, event Handlers, Package Explorer in tab windows
   4. Connection Manager Window in the bottom

   

   Note: If you don’t see ZappySys SSIS PowerPack Task or Components in SSIS Toolbox, please refer to this help link.
4. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox. Double click on the Data Flow Task to see Data Flow designer.
5. From the SSIS toolbox drag and API Source (Predefined Templates) on the data flow designer surface, and double click on it to edit it:
   
6. Select New Connection to create a new connection:
   
7. Use a preinstalled Outlook Mail (Office 365) Connector from Popular Connector List or press Search Online radio button to download Outlook Mail (Office 365) Connector. Once downloaded simply use it in the configuration:
   

   Outlook Mail (Office 365) Connector Selection

8. Proceed with selecting the desired Authentication Type. Then select API Base URL (in most cases default one is the right one). Finally, fill in all the required parameters and set optional parameters if needed. You may press a link Steps to Configure which will help set certain parameters. More info is available in Authentication section.
   1. User Credentials [OAuth]
      

      User Credentials [OAuth]

      OR
   2. Application Credentials [OAuth]
      

      Application Credentials [OAuth]

9. Select the Get Messsages endpoint, change/pass the properties values, and click on Preview Data button to make the API call.
   

   API Source – Get Messages Endpoint

10. That’s it — click OK and you’re done! In just a few clicks, you’ve successfully configured a connection to Outlook Mail (Office 365) using the Outlook Mail Connector.
11. Now let’s move to the next step — retrieving the full email content.
    You’ll use the Web API Destination component to make the Get MIME content API call. Within this task, you can dynamically pass the Message ID to fetch each email’s raw .EML content from Outlook (Office 365).
    For more details, refer to the official Microsoft documentation: Get Outlook Mail MIME content
12. Drag and drop the Web API Destination component from the SSIS Toolbox onto your Data Flow.
    Double-click to open it, then configure as follows and create a new OAuth Connection if needed:URL:

    https://graph.microsoft.com/v1.0/me/messages/<%Id%>/$value

    The <%Id%> placeholder is dynamically replaced by SSIS at runtime with the actual Message ID, allowing the Web API Destination to download each email’s .EML content individually.

    

    Get MIME content’ API call using the Web API Destination

13. You’ll also need to create the OAuth connection using the same credentials.
    On the Advanced tab, make sure to set the Redirect URL correctly.For detailed steps, refer to this guide:
    How to set up OAuth 2.0 in SSIS
    

    Outlook Mail OAuth 2.0 Connection

14. Next, add a Derived Column transformation to create a new column that defines the file path for saving the .EML file.
    In this example, we’ll use the Message ID to generate a unique file name for each email:Expression:

    (DT_WSTR,1000)”E:\\zsTemp\\” + [Id] + “.eml”

    

    Derived Column – Dynamic File Path

15. Now, add an Export Column transformation to save the email content to a file.
    • Extract Column: set to ResponseText (this holds the email content from Web API Destination)
    • File Path Column: set to the column you created in the Derived Column step (e.g., FilePath)
    • Enable Force Truncate and Write Byte-Order Mark options
    

    Export Column – write the email content to a file on disk.

16. Set File Encoding to UTF-8 (Optional)
    Next, go to the Control Flow and use the Advanced File System Task to convert the downloaded .eml file’s encoding to UTF-8.
    This step is optional — I had to apply it on my system because the files weren’t opening correctly due to a non-UTF-8 encoding.
    If you encounter a similar issue, follow this step to ensure proper compatibility with email clients.
    

    Advanced File System Task – Change Encoding

17. That’s it! The process will now automatically download your Outlook messages to your local system as .EML files.
    

    Outlook .eml file’s

Conclusion

Using ZappySys SSIS PowerPack and Microsoft Graph API, you can easily automate the download of Outlook (Office 365) emails in .EML format — all within SSIS and without writing any code.
This approach is ideal for archiving, compliance, or integrating email content into downstream systems.
With a few simple components and steps, your email extraction workflow can be up and running in minutes.

The post How to download Outlook (Office 365) Emails as .EML or .MSG Files appeared first on ZappySys Blog.

In this article, we will show how to work with the Web API Destination and create a dynamic URL, Body or header. For this purpose, we will use the following components:

ZappySys SSIS PowerPack component to make things work:

	SSIS Web API Destination (POST data to URL)
	SSIS JSON Generator Transform

In these examples, we are going to show different examples of how to send a dynamic URL, dynamic body and dynamic headers.

How to make Web API Destination dynamic URL with the OLEDB Source

1. First of all, in this example, we are going to create a dynamic URL, we will use need the following URLs:
   https://jsonplaceholder.typicode.com/posts, https://jsonplaceholder.typicode.com/comments,
   https://jsonplaceholder.typicode.com/todos, https://jsonplaceholder.typicode.com/users, and other similar URLs.
2. So, we will set the last part of the URL as a variable.
3. Also, will have a table with the values of the variable (posts, users, comments, etc):
   

   CREATE TABLE [dbo].[webApiDest](
   	[id] [smallint] NOT NULL,
   	[type] [nchar](10) NULL,
    CONSTRAINT [PK_webApiDest] PRIMARY KEY CLUSTERED 
   (
   	[id] ASC
   )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
   ) ON [PRIMARY]
   GO
   INSERT [dbo].[webApiDest] ([id], [type]) VALUES (1, N'posts     ')
   INSERT [dbo].[webApiDest] ([id], [type]) VALUES (2, N'comments  ')
   INSERT [dbo].[webApiDest] ([id], [type]) VALUES (3, N'todos     ')
   INSERT [dbo].[webApiDest] ([id], [type]) VALUES (4, N'albums    ')
   INSERT [dbo].[webApiDest] ([id], [type]) VALUES (5, N'photos    ')
   INSERT [dbo].[webApiDest] ([id], [type]) VALUES (6, N'users     ')

    
4. In addition, we are going to create a Data Flow with the following components as follows:
5. Secondly, we will connect to the table created in the previous section
   using the OLEDB Source component:
   

   OLEDB-Source-Data

How to make Web API Destination dynamic URL in SSIS with the ZS JSON Generator Transform

1. Also, in the JSON Generator transform, right-click on Mapping and select add elements:
   

   add-elements-JSON

2. Check the add multiple bound options and check the two columns:
   

   Add JSON elements

3. Pay attention to this step because this and the next one are the most important steps. Right-click the type element and select edit.
   

   Edit JSON element

4. The hide from output will hide the type from the output but will be present as a variable. Also, when we want to hide in the JSON created that element.
   

   Column downstream

5. In addition, the include columns in the downstream option are the option that will allow sending the type as a variable from the JSON Generator Transform to the Web API destination.

How to make Web API Destination dynamic URL in SSIS with the ZS JSON Generator Transform

1. First of all, we will add the input column for the body. This will send the ZSON generator into the body.
   

   Body input

2. In connections, we will create a ZS-HTTP connection to the https://jsonplaceholder.typicode.com/ site:
   

   SSIS HTTP Connection

3. In the URL we will create a dynamic part with the type as a variable that gets the values from the SQL Server table and the type column.
4. You can write <%type%> manually, or edit the URL to use placeholders and get the type variable from the columns folder:
   

   SSIS web api dynamic link

5. The Web API configuration will be something like this:
   

   Final SSIS configuration

6. Finally, run the package.

Conclusion

In this article, we learned how to send a dynamic URL to REST API. If you want to test the tool, you can download it here:

• ZappySys PowerPack

The post How to make Web API Destination dynamic (URL, Body, Headers) appeared first on ZappySys Blog.

In our previous article, we saw how to get Office 365 data in Power BI. Now, let’s look at how to get Office 365 Mail Attachments using SSIS without any coding. Microsoft Graph API is a unified way to access many Microsoft services, including the Office 365 API. In this post, we will use the SSIS JSON/REST API Source Connector to retrieve the Outlook Messages list and its attachments.

Use Case of Microsoft Graph API

Here are some use cases for why you want to use the Microsoft Graph API

• Read/Write events from Outlook Calendar
• Send email  / Read emails
• Get the list of files from OneDrive
• Upload/ Download files to OneDrive
• Read/Write Excel Sheet (Use range or a specific range of a specific sheet)
• Read the SharePoint Document library
• Search content from the SharePoint document library
• Get AD users for your Organization (AD Accounts)
• Update OneNote
• Track changes to Users, Events, Calendar Items using the delta API

Step-By-Step to call the Microsoft Graph Rest API Using SSIS

Now, let’s look at examples to access your Outlook.com emails using SSIS. You can use the same techniques to call pretty much any Office 365 API.

Configure SSIS OAuth Connection for Graph REST API

The first step to access any Graph API is to configure an OAuth connection

1. Open Visual Studio
2. Open an existing SSIS Project or create a new one using File > New > Project > Choose “Integration Services Project” Type under Business Intelligence template category.
3. Create Package
4. Right-click inside the Connection Managers area and click New Connection…
5. From the connection type list, select the ZS-OAUTH connection type.
   

   Create a new SSIS OAuth API Connection Manager

6. On the OAuth Connection Manager, configure the following options
   1. Select “Custom” from the Provider dropdown
   2. Select OAuth2 from the OAuth version
   3. Enter your Application Id and Secret Key (i.e., App Password) obtained in the previous section from here.
   4. Enter the following URL in the Authorization URL
      

      https://login.microsoftonline.com/common/oauth2/v2.0/authorize

   5. Use the following URL in the token URL field
      

      https://login.microsoftonline.com/common/oauth2/v2.0/token

   6. Enter the following Graph API Scopes (each scope must be entered on a new line).
      

      user.read
      mail.read
      offline_access

      NOTE: Always include the offline_access scope, which returns a refresh_token, which allows you to renew the token without going through the login process again. For more information about which scopes are needed, always refer to the API help page (e.g., if you want to read mail using this API, then you will see it says you can use Mail.Read or Mail.ReadWrite scope).
   7. Go to the Advanced tab and enter the following URL in the Callback/Return URL (assuming the same URL used when you registered the App in the previous section)
      

      https://zappysys.com/oauth

   8. Now go back to the first tab and click Generate Token. When prompted, log in using your Personal Microsoft Account or Work Account (Office 365 or AD login)
   9. After logging in, you will see the Accept option. Just click it. If things go right, then you will see the Access Token and Refresh Token fields populated.
7. Once everything is configured, you can click Test Connection to verify the connection works.
   

   SSIS OAuth Connection Manager for Office 365 REST API Microsoft Graph API

Download the Messages Attachment Files from Outlook

Let’s start with an example. We use the SSIS JSON Source and Web API Destination components to make a call to the Microsoft Graph REST API.

1. First of all, drag and drop the Data Flow Task from the SSIS Toolbox and double-click it to edit.
   

   Drag and drop Data flow

2. From the SSIS toolbox, drag and drop the JSON Source onto the Data Flow Designer surface.
   

   SSIS JSON Source – Drag and Drop

3. Double-click the JSON Source, then enter the following URL to invoke the Messages API call with search options to retrieve emails with attachments.
   

   https://graph.microsoft.com/v1.0/me/messages?$select=sentDateTime,from,subject,hasAttachments&$search="hasAttachments:True"

   Set the filter to $.value[*] and click the preview.
   

   SSIS JSON Source – Get data from Microsoft Graph API (Office 365 API – Read Mail Example)

   To customize the URL with additional parameters, check this help link. It’s a standard OData Protocol, so you can leverage many common OData features (e.g., use $top parameters to set pageSize. Default is 10 for List Mail, but you can include more records per response by changing it, such as …/messages?$top=50).

4.  Now, let’s use Template Transform to set the URL using the message ID to make the Get attachment API call.

   https://graph.microsoft.com/v1.0/me/messages/<%id%>/attachments

   

   SSIS Template Transform sending attachment id
5. Now we need to use the Web API Destination to make the Get Attachment API call by message ID. Select the Input column for URL as that TemplateOutput.
   

   SSIS Web API Destination – Get data from Microsoft Graph API (Office 365 API – Get Attachment Example)

6. Now, the next step is to parse the JSON Response String of the get attachment api call. For that, we need to use a JSON Parser. Select the input as Web PAI Destination ResponseText and use the below sample JSON string to configure the metadata(Columns).
   

   {"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#users('support%40zappysys.com')/messages('AQMkADAwATMwMAItMDkyMC0zZTg4AC0wMAItMDAKAEYAAAP_JUy')/attachments"
   ,"value":[{"@odata.type":"#microsoft.graph.fileAttachment","@odata.mediaContentType":"text/plain"
   ,"id":"AQMkADAwATMwMAItMDkyMC0zZTg4AC0wMAItMDAKAEYAAAP_JUy_27M"
   ,"lastModifiedDateTime":"2020-02-21T07:36:16Z"
   ,"name":"Test.txt","contentType":"text/plain","size":870,"isInline":false
   ,"contentId":null,"contentLocation":null,"contentBytes":"VGVzdCBNZXNzYWdl"}]}

   

   JSON Parser Transform – using JSON example
7.  Now, in the JSON Parser, go to the Go-to columns Tab and set the contentBytes DataType to DT_IMAGE, and increase the other columns’ DT_WSTR lengths by 1000 and 1500. We set contentBytes to the DT_IMAGE  datatype, which is like varbinary(MAX).
   

   JSON Parser Transform Columns Tab fix the datatype

8. Furthermore, in the next step, we need to use an SSIS Derived Column to set the FilePath where we want to save the attachments, and then concatenate the local file path with the attachment name column.
   

   SSIS Derived Column: Add New Column

9. Finally, we need to use the Export Column Transform (Native) to export attachment bytes to a local file. It needs to know 2 upstream info Bytes and the target file path.
   

   Export Column configuration

10. Make sure to attach the export column to some destination (e.g., our ZS Trash Destination); the engine might remove it. At the same time, it optimizes the runtime. It may remove all transformations that don’t have a destination, such as deploying the package to SQL Server, or when you set optimize in Visual Studio.
11. That’s it, execute the package, and it will download all the email attachments.
    

    Downloaded Mail Attachments

Conclusion

After all, in this article, we learned how to make a Microsoft Graph API REST API call. We used the SSIS JSON / REST API Connector to extract data from Outlook Mail using OAuth. Made the Get attachment API call using the Web API Destination and parsed the JSON response using the JSON Parser. And using the Native SSIS Export Column, we saved the export attachment’s bytes to a local file. To explore many other scenarios not discussed in this article, download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • Help File: Web API Destination
  • Help File: JSON Parser Transform
  • Overview of the Microsoft Graph API

The post How to Get Office 365 Mail Attachments using SSIS appeared first on ZappySys Blog.

Connect to YouTube API in SSIS (OAuth Connection)

Very first step to call any Google API is to create an API project for Google Service and then enable APIs you like to call in Console. In our case we need to enable YouTube API.

Check this article to learn how to create API Project and obtain Client ID and Client Secret to connect to YouTube API. For ease of use use Default Application provided by ZappySys but we still recommend using your own Custom Application  on OAuth Connection.

Before we try our first YouTube API example, we recommend you to get familiar with YouTube API here. Once you done with above steps you can perform the following steps to call YouTube API in SSIS.

1. Open existing SSIS Project or Create new One in Visual Studio (i.e. SSDT)
2. Create a new SSIS Package. Right click in the Connection Manager Panel and click “New Connection…”.
   
3. When Dialog box opens select ZS-OAUTH connection type.
   

   SSIS Connection Managers List: Select O-AUTH Connection Manager

4. Once OAuth UI opens, Configure Google connection like below.
   1. Select Service Type as Google
   2. You can select Default App for ease of use or Select Custom Application (Enter Client Id and Secret Obtained using these steps )
   3. In the Scope Enter below – One scope per line for calling correct YouTube API. Refer to YouTube API documentation incase you need extra permissions. For read only operation we just need below scopes. Last scope is only needed if you need to use YouTube Analytics Reports
      

      https://www.googleapis.com/auth/youtube
      https://www.googleapis.com/auth/youtube.readonly
      https://www.googleapis.com/auth/yt-analytics.readonly

5. Click Generate Token to obtain Refresh Token (You may see Login Prompt, and then Grant Permission Confirmation Screen)
   

   Connect to Google API using OAuth 2.0 (YouTube API Example)

6. Click OK to save UI.

Call YouTube REST API Example

Now lets look at how to call YouTube API REST API Task this task is more suitable to call GET, POST, DELETE, PUT requests without parsing data in Rows and Columns. If you need to parse data and load into Database table then see next section (Use JSON Source)

1. From SSIS Toolbox look for items starting with “ZS”. Drag and Drop [ZS Rest API Task] to Designer Surface.
   
2. Double click it to configure like below.
   1. Select URL From Connection option
   2. Select OAuth connection created in previous section.
   3. Enter URL like below (For example get all playlists). Replace xxxxxxxxxxx with your own channel ID (Its usually found in URL when you visit Youtube Channel Homepage)
      

      https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&channelId=xxxxxxxxxxxxx

   4. Click Test Request Button
3. Here is how it looks like
   

   Call Youtube API in SSIS Example (Dynamic URL)

That’s it we have successfully configured Connection for YouTube API in SSIS. In the next section we will see how to use this connection and read various data from YouTube API.

Read YouTube Playlists in SSIS

Once we done creating OAuth Connection Manager we can move forward to read YouTube data inside Data Flow.

Configure SSIS JSON / REST API Source

1. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
2. Double click on the Data Flow task to see Data Flow designer surface.
3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   
4. Double click JSON Source and enter URL as below (Use variable or Hardcode Channel Id). maxResults controls how many rows you want to get in each response. Rep
   

   https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&maxResults=50&channelId={{User::varChannelId}}
   --OR-- 
   https://www.googleapis.com/youtube/v3/playlists?part=snippet,id&maxResults=50&channelId=xxxxxxxxxxxxxxxxxxxxxx

5. Check Use Credentials and select existing OAuth connection or Create New one
6. Select Array Filter (Items node) or type

   $.items[*]

     as below.
   

   Get Playlists for YouTube Channel

7. That’s it you can now Click Preview to see sample data. In the next section we will configure Pagination to read all records if you have more than 100 rows.

Configure YouTube REST API Pagination

Most REST APIs dont return data in one big response. So you have to make sure you implement looping / pagination to read more records. Luckily ZappySys supports many different Pagination Settings. Here is how to configure YouTube REST API Pagination.

1. Go to Pagination Tab found on JSON Source and select following 3 settings
2. Select Pagination Mode as

   Response Attribute Mode

3. Enter Next Link as

   $.nextPageToken

4. Enter Suffix for Next URL as

   &pageToken=<%nextlink%>

YouTube REST API Pagination Settings

Read YouTube Playlist Videos in SSIS

So in previous section we saw simple API call to read all playlists from YouTube. Now lets see how to get information about Playlist Videos.

For that everything should remain same as previous section except change URL as below. Where xxxxxxxxxxxxxxxxxxxxx is your Playlist ID obtained from Previous Call.

https://www.googleapis.com/youtube/v3/playlistItems?part=snippet,id&maxResults=50&playlistId=xxxxxxxxxxxxxxxxxxxxx

Get all Videos for YouTube Channel

If you want to get all video name and ID for specific channel then use below URL. Click here to read more. Replace xxxxxxxxxxxxxxxxxxx with your Channel ID.

https://www.googleapis.com/youtube/v3/search?channelId=xxxxxxxxxxxxxxxxxxx&part=snippet,id&order=date&maxResults=50

Read YouTube Video Views, Likes, Dislikes (Extended Videos)

Another common request from YouTube API is get Video Information such as Views, Likes, Dislikes, Comment Count etc. For that you can use this api . However the issue is this endpoint requires you to pass Comma separated Ids of Video. You can only pass 25 Ids in one requests so if you have 200 videos for which you need to get information then you have to make 8 requests (25 Ids in each Request). In below screenshot you can see we used Fiddler to debug web requests.

So here is how high level process to fetch extended information for all Videos in your channel. Full SSIS Package is attached here (SSIS 2012 Format).

1. Get all videos for your channel by calling (Assuming you will paginate in below request.
   

   https://www.googleapis.com/youtube/v3/search?channelId=xxxxxxxxxxxxxxxxxxx&part=snippet,id&order=date&maxResults=50

2. Build Video Id groups for 25 Ids from above result and submit requests. For example if above request returns 200 Videos then your request to fetch extended information may look like below.
   

   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID1,ID2,ID3........ID25
   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID26,ID27,ID28........ID50
   ........
   ........
   https://www.googleapis.com/youtube/v3/videos?part=snippet,contentDetails,statistics&id=ID176,ID177,ID178........ID200

3. Parse response coming from each Request Above…. Each Response will give you details about 25 videos you requesting.

Here is the flow of our complete data flow to read extended information for all videos. You can read extended information one by one too but it wont be good idea if you have many videos.

We have used Following Components to achieve Bulk Mode

• JSON Source (REST API Source)
• CSV Parser Transform
• Web API Destination
• JSON Parser Transform

Get YouTube Video View Clicks, Likes, Dislikes (Bulk Mode for all videos)

Read data from YouTube Analytics / Reporting API (Metrics, Dimensions Report)

So far we have seen basic YouTube APIs to extract your Channel information. However there is another very powerful API endpoint to query custom reports from YouTube. Check this YouTube Analytics API endpoint. In this section we will learn how to extract useful information from YouTube using Analytics API.

Step-By-Step

Let’s look at an example to read total views, estimated watch time, average view duration and some other metrics by date for channel you have access.

1. Go to Data Flow, Drag ZS JSON Source from SSIS Toolbox
2. Double click to configure.
3. Check Use Credentials. Select OAuth connection we created in previous section.
4. In the URL field, Enter Below Sample URL. Change Parameters as per your need or leave it default.
   For example:
   ids=channel==MINE   (where MINE can be replaced by other channel id or keep it as MINE to get data for your own channel)
   metrics (can be changed from any valid list here)
   dimensions (can be changed to any valid list from here)
   startDate (Must be YYYY-MM-DD format)
   endDate (Must be YYYY-MM-DD format)
   

   https://youtubeanalytics.googleapis.com/v2/reports?ids=channel==MINE&metrics=views,estimatedMinutesWatched,averageViewDuration,averageViewPercentage,subscribersGained&dimensions=day&startDate=2019-03-01&endDate=2019-04-01

5. Change Filter to

   $.rows[*]

6. Go to 2D Array Transform tab.
   Select Method as

   Simple 2-dimensional array

   EnterColumn Filter as 

   $.columnHeaders[*].name

   

   YouTube Analytics API Advanced Transformation

7. That’s it now Preview your data. See below over all steps.
   

   Read data from YouTube Analytics API (Custom Reports using Dimensions, Metrics Query)

Build YouTube Analytics / Reporting API Query (API Playground)

With Analytics API You can extract many interesting data. You can use Test Tool found on this page to build some interesting queries. Scroll to Common Use Cases section on that link. On the right side you can select Credentials Type as OAuth and click Execute.

https://developers.google.com/youtube/analytics/reference/reports/query

YouTube Analytics / Reporting API Explorer (Testing API Calls)

Loading YouTube API Data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Download Sample SSIS Package

SSIS Package is attached here (SSIS 2012 Format).

Conclusion

In this article we saw how to extract information from REST API such as YouTube REST API using OAuth 2.0. We also learned techniques like Pagination, JSON Parsing, Request Batching using multiple components to achieve full scale API integration. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

The post How to read YouTube API data in SSIS (Videos, Channels, Playlists) appeared first on ZappySys Blog.

In our previous article we saw how to POST data to REST API using few different ways. Now let’s go one step further and discuss another common scenario to batch REST API requests in SSIS. For efficient data transfer many API provides you to submit multiple records in a single request. In this article we will look at few ways to batch our requests using SSIS JSON Generator Transform and  SSIS Web API destination.

Method 1 – Batch records using SSIS JSON / XML Generator Transform

First let’s look at common way to group multiple records and generate single JSON or XML document for desired batch size. We will also see another technique where you can assign Unique BatchID to each document.

For example below screenshot shows how to product one JSON document for every 10 input rows. We have also use BatchID column on the root level. This will use use BatchID from first row of that batch. You can use XML Generator Transform if you like to produce XML Document rather than JSON.

To generate Unique BatchID you can use function like newid() in SQL Server.

SELECT newid() as BatchID,CustomerID,CompanyName FROM Customers

Using SSIS JSON Generator Transform with Batch Option (Multiple Records in a single document)

Here if full flow to batch your JSON Requests and Submit to REST API.

POST data to REST API in Bulk Mode (Batch multiple rows)

Method 2 – Using SSIS Web API destination Batch Mode with Header / Footer

Another method of batching is use Web API destination and turn on Batch mode.  You can use Fiddler to debug your web requests incase you are not sure how its generating and sending POST requests after this setting.

Configure SSIS Web API Destination for Bulk API call (Insert multiple rows)

Method 3 – Batch CSV records using CSV Generator Transform

So far we looked at how to batch JSON and XML format. Now let’s look at scenario where you have to POST CSV formatted data to REST API endpoint.

Assume that you have an API endpoint where you can submit data in multiple batches. Also you have few additional requirements

1. First row of First batch must have column names (any other batch must not contains CSV Header)
2. In each Batch we want to submit maximum 5 rows as per API call
3. Each batch must contain Unique Batch Number in URL

Here is high level flow.

Generate CSV using Batch Setting , Send data to REST API in multiple batches, Debug REST API in Fiddler

Include CSV Header in First Batch Only

Now let’s break down the problem and find out how to do. First problem is we need to include CSV Header only in First batch. To achieve this we use old school trick where we output column names as data row… and when we generate CSV in the next section we will turn off Column Headers option.

If you are consuming records from SQL Server or any other RDBMS you can write following style source query. If you don’t use relational source then use native UNION Transform in SSIS.

select 'CustomerID' as Column1, 'OrderID' as Column2
UNION ALL
select top 12 cast(CustomerID as varchar(1000)), cast(OrderID as varchar(1000)) from Invoices

Batch records and output CSV data

Now next step would be to generate CSV data but in batches… so in our example we have to output size of 5 rows per batch. For this perform the following steps

1. Drag ZS CSV Generator Transform (Included in v2.8 and Higher)
2. Connect source to CSV Generator Transform
3. Now before we edit in UI mode, right click and select Properties. Change following 3 properties  (You can also change QuoteAroundValue and Dimiliter if needed). In future version batch setting will be visible in UI mode too.
   1. ArrayBatchSize  = yourbatchsize (in our case it was just 5 rows per batch)
   2. OutputMode = SingleFileFormat
   3. FirstRowsHasColumnNames  =False
4. Now double click to configure ZS Generator Output
5. Right click on Mappings folder node and Add Elements > Select Multiple Columns  > Check columns you like to output
6. Click OK to Save UI

Generate CSV data in SSIS (Batch Output Setting)

Generate unique URL with Batch using Script Component

Now next task we have to achieve is generate Unique Batch Number. Put that Batch ID in URL like below…

http://httpbin.org/post?batch=0
http://httpbin.org/post?batch=1
http://httpbin.org/post?batch=2
.....

1. For this drag SSIS Script Component and select Transformation Mode
2. Now go to Inputs and Outputs tab > Select Output Columns > Click Add Column > Name it as URL > Configure DataType as DT_WSTR , Length = 500
3. Now got Script Tab and Click Edit. Remove old code and enter following code (Assuming You selected C#). This code will generate new URL with Unique batch ID each time. You can use this same technique to generate Unique row number too (in that case rather than DT_WSTR type you can use DT_I4 type)

   [Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
   public class ScriptMain : UserComponent
   {
       private static int _batch;
       public override void Input0_ProcessInputRow(Input0Buffer Row)
       {
           Row.Url = string.Format("http://httpbin.org/post?batch={0}",_batch);
           //If you have to send GUID rather than Integer for Batch ID then use below code
           //Row.Url = string.Format("http://httpbin.org/post?batch={0}",System.Guid.NewGuid());
   
           _batch = _batch+1;
       }
   }

4. Click OK to Save script.

Output Unique URL or Number using SSIS Script Transform

Configure Web API Destination to POST CSV Data

Now final step is to configure SSIS Web API Destination to POST CSV data to API.

1. Drag ZS Web API Destination and connect Source Transform to Web API destination.
2. Double click to configure
3. Select Connection Manager
4. Once connection setup, select Input Column for URL ( This will be same column from Script Transform)
5. Select column for Body (e.g. ZS_CSV_OUT)
6. Click OK to save

POST data to REST API in multiple batches using SSIS Web API destination

Debug Web API Requests using Fiddler

Now we ready to run entire flow. Once you execute data flow, observe Input rows before CSV Generator Transform and after. CSV Generator transform must produce less rows before we now grouping multiple rows in a single batch.

Another important aspect to play with API request is to debug it, We recommend you to to look at this article about using Fiddler to debug web requests.

Download Sample Package

Here is the Sample SSIS Package (2012 format) – Rest API_Batching

Conclusion

In this post, we saw how easy it is to handle complex REST API  / CSV /JSON / XML operations in SSIS. We configured REST API POST Operation using Batching Method. Hope this post helps you to achieve your API integration without coding.  Feel free to download SSIS PowerPack and try many other components not discussed in this article.

The post How to batch REST API requests in SSIS (Bulk Operation) appeared first on ZappySys Blog.

In our previous article, we saw how to call REST API in SSIS. Now in let’s use that knowledge and learn how to read/write Shopify data in SSIS. If you are not aware of Shopify then its one of the most popular eCommerce platforms out there for small shops who sell online. Shopify provides a total framework to sell/fulfill orders online including tools to build/customize your own web store.

In this article we will see few examples on how to call basic REST API calls to read shopify data, delete shopify data and write shopify data using various SSIS Components.

Create Shopify Custom App for Shopify Admin API

Before accessing Shopify API you must create Custom App (and then obtain Access Token)

New approach (Custom App) to generate App Credentials for Rest API Call

Before you call Shopify REST API, your first step is to obtain credentials. We will need Access Token or Storefront Access Token. We will use it in later section to call Shopify REST API.

1. First, in Shopify, go to sore Settings (Click Gear Icon) >  Apps and sales channels > click Develop apps for your Store
   OR just navigate to URL
   

   https://{your-store}.myshopify.com/admin/settings/apps/development

   

   Shopify App and Sales Channel

    

2. In App development, click on Create an app
   
   

   Shopify Create an App

    

3. Provide the appropriate name for your custom app.
   

   Shopify Custom App Name

    

4. Once we created the app name, we have 2 options to create Admin API or Storefront API in the Overview tab.                                                                               Shopify App Scopes and Configuration
5. Click on Configure Admin API scopes or Configure Storefront API scopes based on your need as shown in the above screenshot.
6. If click on Configure Admin API scopes, then select the appropriate scopes for your Admin API.
7. Also, change orders, transactions, and fulfilments permissions to read and write if you need to create, update and delete products.
8. In addition, change the products, variants, and collections to read and write if you need to create, update and delete products.
   

   Shopify Admin Api Scope

    

9. After saving it, it asks to install app and generate Shopify Access Token for your Admin API.
   

   Shopify Admin Install App

    

   

   Shopify Admin Access Token

10. If click at Configure Storefront API scopes, then select the appropriate scopes for your Storefront API.
11. Also, change orders, transactions, and fulfilments permissions to read and write if you need to create, update and delete products.
12. In addition, change the products, variants, and collections to read and write if you need to create, update and delete products.
    

    Shopify Storefront API Scope

     

13. After saving it, it asks to install app and generate Shopify Storefront Access Token for your Admin API shown above screenshot.
    

    Shopify Storefront Install App

     

     

    

    Shopify Storefront Access Token

     

14.  We have created Admin API Access Token and Storefront API Access Token to access the Shopify APIs.
    

    Shopify API Credentials

Old approach (Private App) to generate App Credentials for Rest API Call

Before you call Shopify REST API, your first step is to obtain credentials. We will need API Key and Password. We will use it in later section to call shopify REST API.

1. First, in Shopify, go to Apps and click Manage private apps.
   

   Shopify App – Manager private apps

2.  Secondly, In Private apps, click the Create a new private app:
   

   Shopify Private apps

3. Also, change orders, transactions, and fulfilments permissions to read and write if you need to create, update and delete products.
4. In addition, change the products, variants, and collections to read and write if you need to create, update and delete products.
   

   Permission for private apps.

5. Finally, you will have an API key, a password, an URL and a shared  secret that you will use in the next section:

Shopify API key

Read Shopify data in SSIS (Get Orders Example)

So in this section we will learn how to call Shopify REST API.  For example purpose we will call Shopify Orders Endpoint. For all other REST API endpoints check this link. Read carefully which parameters you can pass for each API call. So let’s read shopify orders using sample API call.

Configure SSIS JSON Source to read from Shopify

Now, that we created an App in Shopify, the rest is easy.

1. First, in an SSIS project, drag and drop the Data Flow and double click it.
   

   Use the Data Flow

2. Secondly, in the Data Flow, drag and drop the ZS JSON Source.
   

   SSIS JSON Source – Drag and Drop

3. Double-click on JSON Source to configure it.If you are not familiar with ZS JSON Source, check this link.
   New approach to establish the HTTP Connection Manager

   Using Admin API Access token

   1. Enter URL for the API you like to call. For Example, you can enter
      

      https://{your-store}.myshopify.com/admin/api/2022-10/products.json

   2. Check Use Credentials and click on NEW-HTTP connection as shown below,
   3. On HTTP Connection Manager you can enter some URL (it will be ignored and use JSON Source URL above)
   4. Choose Credentials Type as Static Token / API Key
   5. In the Token / Key text box enter your Admin API Key (obtained from the previous section)
   6. In the Auth Header Name enter

      X-Shopify-Access-Token

   7. In the Auth Scheme select {none} 
      

      SSIS JSON Source Admin API HTTP Connection Manager

       

4. Also, press the Select Filter button and press the orders node:
   

   SSIS Shopify filter

5. Now go to Pagination Tab and select RFC5988 Pagination mode as below. Check this link for more info on Shopify Paginated API calls.
   

   Link Header Pagination Method (RFC 5988)

   NOTE:  For Older API use Cursor-based pagination as below (version 2019-07 or older). 
   

   SSIS pagination for Shopify

6. Finally, make sure to the Use Credentials option and select the http connection of the step 2 and press the Preview button to see the data:

SSIS preview Shopify data

Load Shopify data into SQL Server / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Write Shopify data in SSIS REST API Task

In this new example, we will write Shopify data in SSIS. In addition, we will create a new order using REST API to write Shopify data in SSIS.

1. First, use the REST API task in SSDT.
2. Secondly, in the REST API use the http connection created on step 2 of the read Shopify data in SSIS.  Also, in HTTP Request Method, select POST, in Body (Request Data) write the following:
   

   {
     "order": {
       "line_items": [
         {
           "title": "Big Brown Bear Boots",
           "price": 74.99,
           "grams": "1300",
           "quantity": 3,
           "tax_lines": [
             {
               "price": 13.5,
               "rate": 0.06,
               "title": "State tax"
             }
           ]
         }
       ],
       "transactions": [
         {
           "kind": "sale",
           "status": "success",
           "amount": 238.47
         }
       ],
       "total_tax": 13.5,
       "currency": "EUR"
     }
   }

   

   Post JSON order Shopify

    

3. If you dont like to use URL supplied in HTTP Connection manager then just check override URL option and enter URL like below. For this example we used correct URL in HTTP connection manager so we will not override URL.
   

   https://{{your-store}}.myshopify.com/admin/orders.json

4. Now you can click on Test Request / Response and you will see one sample order created in your admin portal.
5. You can supply variables in Body by clicking on Insert Variable dropdown when you edit Body.

Calling Storefront API

In our previous section, we saw how to call ADMIN API which is the most common API you will use in most cases to read / write store data. However if you have a need to call Storefront API  instead then configure HTTP Connection as below

1. For Store Front API call you can follow the same steps as the previous section except for few things on HTTP Connection Manager UI
2. In API Key enter Storefront API Access Token (rather than Admin Token)
3. In the Auth Header Name enter

   X-Shopify-Storefront-Access-Token

   

   SSIS JSON Source Storefront HTTP Connection Manager

Legacy Approach – Using Basic Auth

If you created Shopify PrivateApp (before they introduced the new CustomApp approach) then your HTTP Connection needs to be like below. Use Basic Auth for Credentials Type

1. Using approaches to establish the HTTP Connection Manager 
   

   SSIS JSON Source HTTP connection

Create Multiple Shopify records using Web API Destination

With our ZS Web API destination, it is possible to create multiple records at the same time.  The following link shows the ZS Web API Destination information:

• SSIS Web API destination

Here are few useful articles to explain Web API destination use case.

How to POST data to URL

Few other articles

Delete Shopify records using REST API in SSIS

1. First, in SSDT, drag and drop the ZS REST API task.
2. Secondly, add the HTTP Url Connection.  Use the DELETE HTTP Request Method and the URL including the order id as below:
   

   https://{{your-store}}.myshopify.com/admin/orders/{{orderid}}

   

   SSIS delete data in Shopify using REST API call

    

3. Now click Test Request / Response to test delete API call.

Conclusion

In this article, we learned how to create a Shopify app. In addition, we learned how to connect to Shopify using SSIS with the ZappySys tools. Also, we learned to use ZappySys tools in SSIS to get, write and delete information.

References

Finally, for more information, refer to these links:

• ZappySys SSIS JSON Source
• ZappySys SSIS REST API task

The post Read / Write Shopify data in SSIS (REST API) appeared first on ZappySys Blog.

In this post we will lean SSIS Magento data read / write operations. Magento is a very popular eCommerce platform and they offer JSON based REST API and XML based SOAP API. You can use either API based on your need to automate common integration needs.

We recommend using REST API (JSON API) if possible because they are simpler and faster. We will focus only on JSON API in this article but we do have blog post on how to call SOAP API.

About Calling Magento REST API

If you are new to Magento API we highly recommend to get familiar with Magento REST API here. Magento REST API  offers two ways to authenticate.

1. Token based Authentication
2. OAuth Based Authentication (OAuth 1.0a)

In this article we will look at only Token based approach. In token based approach we have to get new token using admin / customer account information.

We will cover later how to access admin or customer level API later in this article (See HTTP connection configuration).

Configure Connection / Call Magento REST API in SSIS

Now lets look at very simple example how to use Token based method to call Magento REST API in SSIS. We will first configure HTTP connection and then call REST API.

Let’s get started.

1. First, open SSIS Package after installing SSIS PowerPack
2. Drag ZS REST API Task from SSIS Toolbox on Control Flow Designer. Let’s rename task to Get Magento Categories
3. Enter REST API URL you like to call. For example we can use below URL. Replace **your-magento-host** with your own name.
   

   http://**your-magento-host**/index.php/rest/V1/categories

4. Select Url from Connection mode
5. From Connection Dropdown select new ZS-HTTP connection
   

   Create new connection for SSIS REST API Task

6. Configure General Tab of HTTP Connection like below
   URL can be anything for now because we will use override option in next step (On REST API Task).
7. Select Credentials Type as Dynamic Token.
8. Enter Userid / password
   

   Configure Magento REST API Connect for Token based Authentication

9. Now goto Dynamic Token Tab (Or click Configure Link) and configure like below.
10. Enter Token URL for API Login URL field.
    NOTE: If you are a customer and want to access your own account (e.g. view your orders) then replace /admin with /customer  in URL)
    Replace **my-host** with your host name.
    

    http://***my-host***/index.php/rest/V1/integration/admin/token

11. Change Method to POST.
12. Enter Body as below (Placeholders are automatically replaced when /token endpoint is called.
    

    {"username":"[$userid$]", "password":"[$password$]"}

13. Select Content Type as Application/Json
    

    Configure Magento Token Request

14. Now click Dynamic Token – Response Tab and configure like below.
    Select Regex and enter Expression as below. This will remove double quotes around token in response.
    

    "(.*)"{{0,1}}

    

    Extract Token from response – Remove double quotes around value using Regular Expression

15. Click OK to save connection
16. Now on REST API Task check Use Direct URL Option
17. Click Test Request to confirm its working
    

    Call Magento REST API using SSIS REST API Task

Read data from Magento and Load into SQL Server

Now lets look at how to read Magento data and load into SQL Server. Assume that you like to read all products.

Configure JSON Source (Read Magento REST API)

1. Drag Data flow and double click to edit:
   

   Dragging and dropping Data Flow Task into Control Flow

2. Drag ZS JSON Source on data flow and double click to edit
3. Configure JSON Source like below if you like to get product list for example
   1. Enter URL as below (change http to https if your host support secure channel).
      

      http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=1000
      --- to extract only specific fields do below way --
      http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=1000&fields=items[sku,name]

   2. Check Use Credentials. Use same HTTP connection we created in previous section
   3. In the Filter enter $.items[*] or click Select Filter to Browse and select items node.
   4. Click Preview to confirm
4. Here is how your setup will look like after configuration.
   

   SSIS JSON Source Configuration – Read data from Magento REST API

5. If you expecting more than 1000 rows then you have to set up pagination. Now go to pagination tab and configure as below (Below options are only available in v2.9.1 or higher). If you have older version then check next section for workaround.
   1. Pagination Mode =URL Parameter Mode
   2. Page Num Indicator = searchCriteria[currentPage]
   3. Max Rows Expression = $.total_count
   4. Page Data Expression = $.items[*]
6. Here is how your Magento REST API Pagination Setup will look like.
   

   REST API Pagination using Max Row count Mode

7. Click OK to save JSON Source. In next section we will see how to load data into Target like SQL Server.

Configure OLEDB Destination (Load into SQL Server)

1. Drag OLEDB Destination.
2. Select / Create Connection
3. Select or create NEW target Table (Click NEW button)
4. Click Mapping Tab to map source columns to target table
5. Click OK to save
6. Execute Package

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Configure Magento Pagination (For Old Version)

In previous section we configured pagination using newer version (v2.9.1 or higher). But what if you have older version and you dont see that feature. Use below steps for workaround.

Now let’s look at another common scenario when you real lots of data. Let’s say you have 5000 products but API call we saw earlier returns max 1000 rows. In that case we have to keep reading next page until all rows are returned. ZappySys provides many settings to configure various REST APIs out there, unfortunately there is no standard around pagination methods. Since Magento API has some bug in API pagination we have to do following hack. Magento keep sending same data of last page if you try to access page which doesn’t exists. Here is how to overcome that bug.

Configure REST API task to get total row count

First step to configure magento pagination is get total rows returned in our request which you like to paginate (in our case /products).

1. Go to Control Flow designer
2. Drag ZS REST API Task from SSIS Toolbox
3. Let’s rename task to Get Magento Product Count
4. Double click it and configure like below (Change HTTP to HTTPS if needed)
   

   http://***yourhost***/index.php/rest/V1/products?searchCriteria[pageSize]=10

   
5. Go to Response Tab and configure like below. Enter expression as $.total_count
6. Click OK to save

Define Expression for JSON Source – MaxRows Property

Now next step is to define expression on MaxRows property of JSON Rows so we stop once all rows consumed from Paginated response. If you don’t do this it will keep paginating forever 🙁

1. On Control Flow designer select Data Flow which contains JSON Source for Magento
2. Right click and go to Properties of that data flow
3. Find Expression and click Button to add new expression
4. On Expression dialogbox select [Your JSON Source].[MaxRows] and for expression enter variable name which holds total count (e.g. @[User::MaxRows] )
5. Click OK to Save

Configure JSON Source for Magento Pagination

Now last thing we have to do is go to JSON Source and configure few things for pagination.

1. Go to data flow and double click JSON Source
2. Click on Pagination tab and select Pagination by URL Parameter Mode
3. Enter Page name as below
   

   searchCriteria[currentPage]

4. Click OK to save UI
5. Now run entire page to test (NOTE: Do not execute just data flow because if you do that way, it will not extract total row count)
   

Write data to Magento using SSIS Web API destination

Now let’s look at example to load data into Magento. In below example we will create few sample products.

1. Create new data flow and go to data flow designer
2. Drag and drop ZS CSV Source from SSIS toolbox. You can use any Data source but for simple demo we will use it.
3. Double click CSV source and select Direct Value mode from dropdown. Enter following sample data (Lets create 12 sample products.)
   

   SKU,ProductName
   PROD-1,Product 1
   PROD-2,Product 2
   PROD-3,Product 3
   PROD-4,Product 4
   PROD-5,Product 5
   PROD-6,Product 6
   PROD-7,Product 7
   PROD-8,Product 8
   PROD-9,Product 9
   PROD-10,Product 10
   PROD-11,Product 11
   PROD-12,Product 12

4. Now click OK to save UI
5. Now drag ZS Template Transform from toolbox.
6. Connect CSV Source to Template Transform
7. Enter following sample text to build request for new product. Refer to Magento API help file / tutorial to learn more.
   

   {
     "product": {
       "sku": "<%SKU%>",
       "name": "<%ProductName%>",
       "attribute_set_id": 4,
       "price": 25,
       "status": 1,
       "visibility": 1
      }
   }

8. Notice that when you click Insert Variable > Columns > select desired upstream column placeholder. You can also encode special characters in your data (e.g. new lines) using function like <%MyColumn,JSONENCODE%>
9. Click OK and save UI
10. Now drag ZS Web API Destination from SSIS Toolbox. Configure like below.Enter URL same as before (used to read product), change Method to POST, change content type to Application/JSON.
11. Now run entire flow (You can watch your API requests in Fiddler for debugging)

Debug Magento Web Requests using Fiddler

Click on below article to learn how to debug web requests

How to use Fiddler to analyze HTTP Web Requests

Magento Integration in Other Apps (e.g. Power BI / Informatica / SQL Server)

You can use techniques listed on this page with ODBC Drivers too for integration in other tools like Power BI, SQL Server code, Informatica etc.

Conclusion

In this post we saw how easy it is to perform Magento Integration in SSIS without doing any programming. In few steps you can read or write data in Magento using SSIS PowerPack REST API components. Download FREE Trial and explore many other scenarios.

The post SSIS Magento data Read / Write using REST API Call appeared first on ZappySys Blog.

In this tutorial, we will cover the topics of how to perform geocoding on the addresses and reverse geocoding on the location coordinates using SSIS and Google Maps API. So what is geocoding, exactly? Geocoding is the process of translating an address (e.g. a street address) or a place to coordinates on the Earth’s surface. Simply put, geocoding takes a description of a location and gives back latitude and longitude values as a result. For example, 

1600 Amphitheatre Parkway, Mountain View, CA 94043, USA

(37.4224764, -122.0842499)

(37.4224764, -122.0842499)

1600 Amphitheatre Parkway, Mountain View, CA 94043, USA

To accomplish our goal we will use an add-on for SSIS – ZappySys SSIS PowerPack. It contains many additional SSIS Tasks and Data Flow Components that lets you accomplish what you cannot do with standard SSIS connectors. We will use these ZappySys SSIS PowerPack connectors:

Web API Destination

JSON Parser Transform

Let’s start!

Step-by-Step – SSIS Geocoding with Google Maps API

Before we start, let’s consider the use-case for our tutorial. Suppose, you have a CRM system filled with user-entered addresses of your leads. Since they are user-entered, it means that they can be incomplete, malformed, or just plain rubbish. Also, you have a marketing campaign and you want to send letters with offers to your leads so that they become your clients. Before sending them any letters, you have to make sure the addresses are valid and well-formed. You are also interested in the geographical location of your leads that you could do some business intelligence analysis on the coordinates of your leads and maybe future clientele. Here is where Geocoding API kicks in.

Thus in this example, we will take various street addresses – sometimes invalid, incomplete or unformatted – and submit them to Google Geocoding API for processing. As a result, we will get nicely formatted, cleansed, full addresses and their coordinates. We will take a look on how to determine if lookup worked successfully and to what extent you should trust Geocoding API’s response. Finally, we will save retrieved addresses and their individual components (e.g. street number, street name, city, country, etc.) to a SQL Server database table.

Load addresses from a SQL Server database table

We prepared the Google Geocoding API part, now we are ready to get to the place where the rubber meets the road – we will create an SSIS package and load addresses from a database table:

1. Create a new SSIS package.
2. Drag and drop Data Flow from SSIS Toolbox onto the Control Flow.
   

   Dragging and dropping Data Flow Task into Control Flow

3. Drag and drop OLE DB Source onto the Data Flow.
4. Configure OLE DB Source to retrieve addresses from your table. We will use a table from Microsoft’s Northwind’s modified database “Marketing” (download it):

Loading user-entered addresses to be looked up in Google Geocoding API using SSIS.

Construct lookup URL

1. Drag and drop Derived Column component onto the Data Flow.
2. Add a new column and construct Geocoding lookup URL in this format (just like it says in Geocoding API documentation):

https://maps.googleapis.com/maps/api/geocode/json?address=ADDRESS_TO_LOOKUP&key=YOUR_API_KEY

Adding Google Geocoding API lookup URL as Derived Column to be used for geocoding in SSIS.

Configure the Geocoding API request

1. Drag and drop Web API Destination onto the Data Flow and connect it with Derived Column. We will use Web API Destination to make HTTP requests to Geocoding API.
2. Then double-click on Web API Destination and create a new HTTP connection:
   

   Setting up HTTP Connection Manager for Web API Destination to make HTTP requests to Google Geocoding API using SSIS.

3. Finally, configure Input Column for URL, HTTP Request Method and input into URL field a sample lookup URL:
   

   https://maps.googleapis.com/maps/api/geocode/json?address=One Microsoft Way Redmond 98052-6399 USA&key=YOUR_API_KEY

   

   Configuring Web API Destination to make geocoding lookups in Google Geocoding API using SSIS.

   Once you are done, click Test Request/Response button. This will actually make a geocoding lookup for Microsoft headquarters:

4. In the Response (Raw) tab you see the actual response from Geocoding API. Within it –  cleansed, formatted and full Microsoft headquarters address together with geographical coordinates (even we looked it up using just the part of it). Now copy-paste the response body somewhere, so that we can use it in the next step while parsing:
   

   Checking Google Geocoding API HTTP response in Web API Destination using SSIS.

Parsing Geocoding API JSON response into columns

1. Drag and drop JSON Parser Transform onto the Data Flow and connect it to Web API Destination.
2. Then open it and configure Input JSON Column.
3. After that, set the Filter to $.results[*].
4. Continue by setting Sample JSON string to the one you saved in the previous step. Actually, before doing that let’s modify it and add partial_match node next to place_id so that exact and partial matches could be told apart (otherwise this property won’t be parsed):
   

   {
       "results": [{
           "address_components": [{
               "long_name": "One",
               "short_name": "One",
               "types": ["street_number"]
           }, {
               "long_name": "Microsoft Way",
               "short_name": "Microsoft Way",
               "types": ["route"]
           }, {
               "long_name": "Overlake",
               "short_name": "Overlake",
               "types": ["neighborhood", "political"]
           }, {
               "long_name": "Redmond",
               "short_name": "Redmond",
               "types": ["locality", "political"]
           }, {
               "long_name": "King County",
               "short_name": "King County",
               "types": ["administrative_area_level_2", "political"]
           }, {
               "long_name": "Washington",
               "short_name": "WA",
               "types": ["administrative_area_level_1", "political"]
           }, {
               "long_name": "United States",
               "short_name": "US",
               "types": ["country", "political"]
           }, {
               "long_name": "98052",
               "short_name": "98052",
               "types": ["postal_code"]
           }],
           "formatted_address": "One Microsoft Way, Redmond, WA 98052, USA",
           "geometry": {
               "location": {
                   "lat": 47.6423318,
                   "lng": -122.1369302
               },
               "location_type": "ROOFTOP",
               "viewport": {
                   "northeast": {
                       "lat": 47.6436807802915,
                       "lng": -122.1355812197085
                   },
                   "southwest": {
                       "lat": 47.6409828197085,
                       "lng": -122.1382791802915
                   }
               }
           },
           "partial_match": false,
           "place_id": "ChIJGxUEuHFtkFQRnokD7mkppQE",
           "plus_code": {
               "compound_code": "JVR7+W6 Redmond, Washington, United States",
               "global_code": "84VVJVR7+W6"
           },
           "types": ["establishment", "point_of_interest"]
       }],
       "status": "OK"
   }

   

   Configuring JSON Parser Transform to parse looked up addresses in Google Geocoding API HTTP JSON response using SSIS.

5. To proceed and parse individual address components from JSON response (e.g. number, street, city, country, etc.), you will need to specify columns for each address component manually, just like portrayed in Parse multi-dimensional JSON array in SSIS or Drivers article. Just open 2D Array Transform tab and select Multiple columns using expressions as Transform Type and configure them:
   

   Configuring JSON Parser Transform to parse individual address components from Google Geocoding API HTTP JSON response using SSIS.

   We specify these columns in our example:

   Number = $.address_components[?(@.types[0]=='street_number')].long_name
   Street = $.address_components[?(@.types[0]=='route')].long_name
   City = $.address_components[?(@.types[0]=='locality')].long_name
   State = $.address_components[?(@.types[0]=='administrative_area_level_1')].long_name
   Country = $.address_components[?(@.types[0]=='country')].long_name
   PostalCode = $.address_components[?(@.types[0]=='postal_code')].long_name

6. Finally, be sure to set Include all upstream columns to downstream (quite important), so that CustomerID is passed downstream and which we can save into a destination table. Later it can be a useful column to join the source table and the destination table on. In that way, you will be able to map user-entered addresses with looked up, fully formatted addresses (won’t be demonstrated in this tutorial).

Save geocoding results into a SQL Server database table

1. We are now ready to add OLE DB Destination, execute the package, make the lookups, and save the results into the database:
   

   Creating a database table for the Google Geocoding API looked up addresses in SSIS.

2. Execute the package!
   

   SSIS geocoding package execution and its results.

The results of SSIS geocoding

Viewing the results of SSIS geocoding in SQL Server destination table.

Once you query the table of looked up results, you will notice several things:

1. Perhaps there are more rows in the results table than in the source table
2. partial_match column is either 1 or NULL
3. geometry.location_type column has several values, e.g. ROOFTOP, RANGE_INTERPOLATED, APPROXIMATE, etc.

So to what extent should you trust the results? The answer is to take a look at the documentation and see what these values mean.

Basically, when partial_match is equal to 1, according to Google:

• Partial matches most often occur for street addresses that do not exist within the locality you pass in the request.
• Partial matches may also be returned when a request matches two or more locations in the same locality. For example, “21 Henr St, Bristol, UK” will return a partial match for both Henry Street and Henrietta Street. Note that if a request includes a misspelled address component, the geocoding service may suggest an alternative address. Suggestions triggered in this way will also be marked as a partial match.

While geometry.location_type column tells how accurate the lookup was. Again, according to Google:

• “ROOFTOP” indicates that the returned result is a precise geocode for which we have location information accurate down to street address precision.
• “RANGE_INTERPOLATED” indicates that the returned result reflects an approximation (usually on a road) interpolated between two precise points (such as intersections). Interpolated results are generally returned when rooftop geocodes are unavailable for a street address.
• “GEOMETRIC_CENTER” indicates that the returned result is the geometric center of a result such as a polyline (for example, a street) or polygon (region).
• “APPROXIMATE” indicates that the returned result is approximate.

Simply put, “ROOFTOP” is the most accurate lookup, while “APPROXIMATE” is the least accurate.

Another valuable piece of information is how many individual address components are filled. The more NULLs are there, the less accurate is the address. So if you have NULL in the Number column and in the Street column, most likely this is not a very accurate lookup:

Individual address components of looked up addresses in Google Geocoding API using SSIS.

Concluding, in the figure above, you can see how user-entered address can be validated, cleansed and formatted.

SSIS reverse geocoding with Google Maps API

The concepts presented above about “forward” geocoding will help you to make reverse geocoding requests too. The only difference is the request URL:

https://maps.googleapis.com/maps/api/geocode/json?latlng=40.714224,-73.961452&key=YOUR_API_KEY

For reference and help visit official Google Geocoding API documentation.

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Common errors

Parameterization and deployment to production 

Download a sample package

Be sure to download a sample SQL Server 2012 SSIS package, in case you want to try it right away (you can upgrade it to a higher version). If you want to use the database that was used in this tutorial, don’t forget to create the Marketing database from the script (a modified Microsoft’s Northwind database).

Conclusion

In this tutorial, we step-by-step covered how to make geocoding lookups using Geocoding API, part of Google Maps API. We learned how to cleanse, validate and format address just by submitting it to Geocoding API and parsing the results. We moved forward and looked at how to interpret the results and what geocoding results to trust. Then we touched the point that the same process could be applied to reverse geocoding. Concluding, we took a look at how to handle common errors and how to parameterize an SSIS package and deploy it to production.

The post SSIS Geocoding with Google Maps API appeared first on ZappySys Blog.

We will look at step-by-step instructions to read Zoho CRM data (e.g. Leads, Accounts, Contacts) and load into SQL Server Database Table. In the next section, we will look at Write scenario too.

Now let’s look at step-by-step approach to call Zoho API  using SSIS.

Authentication Using Zoho V2 REST API (OAuth 2.0)

Zoho recently released v2 API. It uses OAuth 2.0 instead of static Token based (described in the next section). OAuth approach is more secure an recommended. If you must use static token-method then check next section. Zoho may discontinue Token based method in future.

Register Zoho OAuth App and get ClientID/ Secret (*** MUST READ ***)

Check this article to register your OAuth App and obtain Client ID  /Client Secret

Create Zoho OAuth Connection in SSIS (for API call)

Once you create Zoho OAuth App now its time to call some Zoho API. We will use ZappySys REST API Task to make simple API call. In the next section we will see how to load data in SQL Server.

1. Open SSIS Package and go to SSIS Designer (Control Flow)
2. Drag ZS REST API Task from Control Flow SSIS Toolbox as below
   

   Drag and drop SSIS REST API Task from SSIS Toolbox

3. Double click the task to configure.
4. Change URL Access mode to URL from Connection
5. Enter API URL you like to call. For example you can enter below
   

   https://www.zohoapis.com/crm/v2/settings/modules

6. Select New ZS-OAUTH connection from Connection drop down
   

   Call Zoho REST API using OAuth 2.0

7. Configure your OAuth connection General Tab as below
   1. OAuth Provider: Custom
   2. OAuth Version: OAuth2
   3. Client Id , Client Secret
   4. Enter One Scope Per Line (For more info on scope read here)
      

      ZohoCRM.modules.READ
      ZohoCRM.settings.READ

   5. Auth URL : https://accounts.zoho.com/oauth/v2/auth
   6. Token URL:  https://accounts.zoho.com/oauth/v2/token
8. Configure Advanced tab as below
   1. Enter Callback URL : https://zappysys.com/oauth
   2. Enter Extra Attribute for /Auth : access_type=offline&prompt=consent
      This is needed to obtain refresh_token (if you ommit this then only access_token returned which cannot be renewed once expired after 1hr)

      

9. Now go back to general tab and Click Generate Token.
   

   Generate OAuth Token for Zoho CRM REST API Access

10. Click OK to save the connection UI

Call Zoho API using OAuth Connection

1. On the REST API Task click Test Request / Response
   

   SSIS REST API Task – Call Zoho REST API in SSIS (OAuth 2.0)

Read data from Zoho Table in SSIS JSON Source

In this section we will learn how to read data from Zoho using SSIS JSON Source. We will use OAuth connection created in earlier section.

Now lets look at step by step example of reading Zoho CRM data using SSIS (e.g. read Leads, Accounts, Contacts, Invoice etc). Zoho provides JSON API for read action so we will Use SSIS JSON Source for this purpose. Full sample package is attached at the end of this article but see below steps for high level configuration.

Configure JSON Source

So below steps shows example of reading from Zoho Lead modules using v2 records API Call.

1. Open SSIS Package
2. Drag data flow task from Control flow SSIS Toolbox
   

   Drag data flow from SSIS Toolbox

3. Go to data flow and drag ZS JSON Source from SSIS Toolbox
   

   Drag and drop SSIS JSON Source (REST API or File)

4. Double click JSON Source to configure. Enter URL as below. You can supply few more parameters. Click Here to get full list of parameters and usage. In below example we are reading records from Leads Module but you can replace Leads to something else (e.g. Accounts). Refer to above help link for available Modules.
   

   https://www.zohoapis.com/crm/v2/Leads?per_page=200

5. Check Use Credentials. Select OAuth connection manager we created in the previous section.
6. Under filter options tab, enter Array Filter as  

   $.data[*]

7. Now click Preview to see your data
   

   Read data from Zoho API v2 using SSIS – OAuth (Use Last Modified Date for Incremental Extract)

8. Click OK to save UI. In the next section we will see how to setup pagination and incremental extract of Zoho records. After that we will see how to load data into SQL Server Table.

Zoho API Pagination (Read all rows)

Just like most API zoho also uses pagination. See below for how to configure pagination in JSON Source for Zoho API.

1. Double click JSON Source
2. Go to the Pagination Tab and select URL Parameter Mode
3. For Page Num Indicator Enter

   page

4. For Last Page Detection select

   Detect last page based on status code

5. In the Status code field
   enter 304 (if you added If-Modified-Since header for incremental extract – See next section)
   enter 204 (if you do not add If-Modified-Since)

Once you configure this way now your API will return all records until last page is detected. For more information on pagination check this article

Paginate Zoho API records

Incremental Extract based on Last Modified Date/Time field

Each module in Zoho has Last Modified By and LastModified Date/Time fields. You can filter extracted data for incremental extract by supplying

If-Modified-Since

1. On the JOSN Source, click Raw Edit above HTTP Headers Grid.  We can define Header to extract data after last modified date. Skip this step if you like to extract all records.
   For example to read Lead records which are modified after 12/31/2019 you can enter header like below. Date/Time is UTC. When record is created Modified date is set as creation date too so this will include created records too.
   

   If-Modified-Since: 2019-12-31T00:00:00

   If you have local time saved in a variable and you like to use SSIS variable then you can use below way. In this example we converted Local time to UTC.
   

   If-Modified-Since: <<{{User::varExtractDate,yyyy-MM-ddTHH:mm:ss}},FUN_TO_UTC_DATE>>

Loading Zoho data into SQL Server Table / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Write data to Zoho CRM using OAuth 2.0

If you like to load data from any source (i.e. SQL Server) to Zoho CRM then you can use Records API (POST). Use combination of the following Transforms  / Components. Later in this article we have some more examples (Older API) which may give you one more ways to load data into Zoho (Use of Template Transform).

Basically you have to build POST Body using JSON Generator Transform or Template Transform and then use that data in Web API Destination to call POST API request to push data to Zoho.

	Web API Destination
	JSON Generator Transform
	JSON Parser Transform

Create JSON for POST request (Single dataset pattern)

Insert data to Zoho Table using SSIS (Lead, Account Modules) – Bulk Load using POST API call

Authentication Using Zoho Tokens

If you don’t want to use a newer version of Zoho API (v2) then you can use Token based Authentication method.

In Zoho Portal perform following steps to obtain API token

1. Click on Setup (Toolbox icon at the top-right corner)
2. Search for “API” > Select CRM API  or find under Developer space > APIs
3. Click on Gear icon found above Usage Panel (small icon so find carefully)
4. Click on Authentication Token generation
5. Specify the APP Name and Click Generate
6. This will direct to the new page which may contain text like below. Just copy AUTHTOKEN part (i.e. 8a08xxxxxxxxxx13630d )
   

   #
   #Mon May 15 12:21:42 PDT 2017
   AUTHTOKEN=8a083f76xxxxxxxxxxxxxxxxxxxdxxxxxxxx13630d
   RESULT=TRUE

7. Now use the newly created auth token to your API call.

How to get Zoho CRM API Token (For SSIS or other app integration)

Read data from Zoho using SSIS (Export Zoho CRM data to SQL Server Table)

Now lets look at step by step example of reading Zoho CRM data using SSIS (e.g. read Leads, Accounts, Contacts, Invoice etc). Zoho provides JSON API for read action so we will Use SSIS JSON Source for this purpose. Full sample package is attached at the end of this article but see below steps for high level configuration.

So below steps shows example of reading Zoho Lead modules using getRecords API Call.

1. Goto SSIS > Variable and create a new string variable called token. Enter your Zoho API token as value (see previous section to get API token)
2. Drag data flow task from Control flow SSIS Toolbox
3. Go to data flow and drag ZS JSON Source from SSIS Toolbox
4. Double click JSON Source to configure as below
   1. Enter URL as below. Note that we have use Variable for Token to make URL dynamic.To read from different module change URL (e.g. rather than Leads in URL use Accounts).  Click Here to get full list of Modules and Fields
      

      https://crm.zoho.com/crm/private/json/Leads/getRecords?newFormat=1&authtoken={{User::token}}&scope=crmapi&fromIndex=0&toIndex=200

      You can also use lastModifiedTime parameter along leads modified after certain date/time (this time is Local time and not the UTC). To extract leads modified after certain date (Create a datetime variable called varExtractDate) and use it as below in the URL. You can use any valid format specifiers for date time
      

      https://crm.zoho.com/crm/private/json/Leads/getRecords?newFormat=1&authtoken={{User::token}}&scope=crmapi&fromIndex=0&toIndex=200&lastModifiedTime={{User::varExtractDate,yyyy-MM-dd HH:mm:ss}}

   2. Click on the Select Filter and select node with Array icon as below. Each Module may have different structure for example for Leads module Filter may look like $.response.result.Leads.row[*]
      

      Configure SSIS JSON Source – Read Zoho CRM Leads (getRecords API call with lastModifiedDate Example)

   3. Now go to 2D Array Tab and configure like below
      1. Select Transform Type to Key/Value Pivot Mode.
      2. Select or type Column Filter as $.FL[*].val
      3. Select or type Row Value Filter as $.FL[*].content
   4. Once you configure 2D Array Transform click Preview to see sample data
   5. .
   6. Click OK to save JSON Source and attach it to target such as SQL Server Destination like below.
   7. Execute Package to test.
      

      SSIS Example : Loading Zoho CRM data to SQL Server Table (JSON API Source)

Write data to Zoho using SSIS (Import SQL Server Table to Zoho CRM)

Now you know how to read data from Zoho CRM using JSON API Source next step is to load data to Zoho CRM. For writing data we will use following three components

	Web API Destination
	Template Transform
	XML Parser Transform

For this example we will use hard coded data in CSV Source (Sample Leads ) but in real world you will have data coming from some RDBMS such as SQL Server.

1. Drag Data flow Task and double click to go to Data flow designer.
2. Drag ZS CSV Source from the SSIS Toolbox. Change Access mode to Direct Value. Enter following sample data for demo.
   

   Company,First Name,Last Name,Email,Lead Status,Lead Source,Phone
   Microsoft,TestMaawia,TestBrito,Maawia@outlook.com,Lunk Lead,Web Download,111-111-3333
   Microsoft,Testraja,Testvedaiyan,raja@outlook.com,Lunk Lead,Web Download,111-222-3333
   Yahoo,TestJayder,Testelnageib,Jayder@outlook.com,Lunk Lead,Web Download,111-333-3333
   Yahoo,TestStephen,TestTjebane,Stephen@outlook.com,Lunk Lead,Web Download,111-444-3333
   Google,TestHarold,TestBatista,Harold@gmail.com,Lunk Lead,Web Download,111-555-3333
   Google,TestVennon,TestMills,Vennon@gmail.com,Lunk Lead,Web Download,111-666-3333
   Google,TestMichael,TestTremblay,Michael@gmail.com,Lunk Lead,Web Download,111-777-3333

3. Now drag ZS Template Transform. Connect it with previous step.
4. Double click Template Transform to configure it with following sample Text (This transform was introduced in v2.6.6. If you dont see it then update your SSIS PowerPack to latest version). Use of XMLENC is optional for Non string columns or Alphanumeric values but if you not sure about column type then just add it anyways.
   

   <Leads>
    <row no="1">
     <FL val="Lead Status"><%Lead Status,XMLENC%></FL>
     <FL val="Company"><%Company,XMLENC%></FL>
     <FL val="First Name"><%First Name,XMLENC%></FL>
     <FL val="Last Name"><%Last Name,XMLENC%></FL>
     <FL val="Email"><%Email,XMLENC%></FL>
     <FL val="Phone"><%Phone,XMLENC%></FL>
     <FL val="Lead Source">Web Download</FL>
    </row>
   </Leads>

   
5. Now drag another ZS Template Transform and connect it with previous step.
6. Double click Template Transform to configure. Enter below text. This is final text we will use as Request Body in the Web API Destination.
   

   xmlData=<%TemplateOutput,URLENC%>

   
7. Drag Microsoft Derived Column Transform. Connect with previous step and double click it to configure.
8. Select Replace ‘TemplateOutput’ action. Type new column name in the first column e.g. TemplateOutput_Encoded
9. Now drag ZS Web API Destination, Connect with previous step and double click to configure as below.
   1. Select New HTTP Connection. Enter some valid URL e.g. https://zoho.com  (This will be ignored anyways because we will use Dynamic URL on main UI)
   2. Enter URL as below (see how we again added API token as dynamic variable) . If you inserting to different Module (e.g. Accounts rather than Leads then change accordingly).   Click Here to get full list of Modules and Fields
      

      https://crm.zoho.com/crm/private/xml/Leads/insertRecords?newFormat=1&authtoken={{User::token}}&scope=crmapi&duplicateCheck

   3. Select Input Body column as TemplateOutput_Encoded
   4. Enter Sample Body as below if you like to use Test feature else ignore this step. Click Test to see its working. Capture Response XML because you may need in the next step to feed as sample to XML Parser Transform (again optional step).
      

      <Leads>
       <row no="1">
        <FL val="Lead Status">Junk Lead</FL>
        <FL val="Company">ABCD Inc</FL>
        <FL val="First Name">Bob</FL>
        <FL val="Last Name">Smith</FL>
        <FL val="Email">bob@abc.com</FL>
        <FL val="Phone">111-222-3333</FL>
        <FL val="Lead Source">Web Download</FL>
       </row>
      </Leads>

      
10. Now once API destination is set we can optionally configure below steps if you want to store status and ID of newly created records. If you don’t care for that then skip below steps.
11. Drag ZS XML Parser Transform, Connect it Web API Destination (Response Output – Blue Arrow). Double click XML Parser to configure as below
    1. Select Input XMl Column as ResponseText
    2. Enter Sample Text as below
       

       <?xml version="1.0" encoding="UTF-8" ?>
       <response uri="/crm/private/xml/Leads/insertRecords">
       	<result>
       		<message>Record(s) added successfully</message>
       		<recorddetail>
       			<FL val="Id">1558554000015597002</FL>
       			<FL val="Created Time">2018-02-13 16:14:10</FL>
       			<FL val="Modified Time">2018-02-13 16:14:10</FL>
       			<FL val="Created By">
       				<![CDATA[ZappySys Support]]>
       			</FL>
       			<FL val="Modified By">
       				<![CDATA[ZappySys Support]]>
       			</FL>
       		</recorddetail>
       	</result>
       </response>

        
    3. Enter Filter as below
       

       $.response.result.recorddetail[*]

    4. 
    5. Now Go to 2D Array Transform Tab. Configure as below
    6. Select Transform Mode to Key/Value
    7. Column name filter as  

       $.FL[:3].@val

    8. Row value filter as  

       $.FL[:3].#text

       
    9. Now click OK to close.
12. Drag ZS Trash Destination or some other destination (e.g. SQL Server) and connect Red arrow coming from Web API destination to this component to capture any errors.
13. Thats it now you ready to run your SSIS sample package which Inserts Leads to Zoho CRM. See below for full package.

SSIS Example: Import data to Zoho CRM using SSIS Web API Destination (Loading Leads, Accounts, Contacts)

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Sample SSIS Package Download

Click here to download sample package (SSIS 2012 or Higher).

Conclusion

Zoho CRM API provides great way to automate data read/write operations. However to call Zoho API  you have to use SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides great way to integrate any Zoho API call via simple drag and drop approach without coding. Try  SSIS PowerPack for free and call virtually any REST API in few clicks.

The post Read / Write Zoho CRM data using SSIS REST API Call appeared first on ZappySys Blog.

In this post you will learn how to Read / write Smartsheet data using SSIS (Drag and drop approach without any coding).  We will use SSIS JSON/ REST API Source to extract data from Smartsheet API and use SSIS Web API Destination to write data to Smartsheet. This approach is similar to our previous blog where we described how to read and write Google sheet using SSIS

Now let’s look at step by step approach to call Smartsheet REST API.

Create Token to access Smartsheet REST API

First step to access smartsheet API in SSIS is create API Token. We will use this API token later on to read data from Smartsheet Sheet.

1. Login to your Smartsheet account
2. Click on Account > Apps & Integrations > Click API Access > Click Generate Token
   

   Create Smartsheet API Token for REST API Access (Step 1)

   

   Generate new Smartsheet token (Step 2)

3. Once Token is generated copy that for later use (Make sure you treat this token as password and do not share with unauthorized user.

Getting started with Smartsheet API

In this article we will use few out of many API provided by Smartsheet. To get started with Smartsheet API concepts click here. To read full details about each Smartsheet API click here.

Step-By-Step – Import/Export Smartsheet data in SSIS

Now lets see how to read and write smartsheet data using SSIS.

Read data from Smartsheet using SSIS JSON / REST API Source (Load Smartsheet to SQL Server)

To read data from smartsheet you need to perform following steps.

Find Sheet ID for Smartsheet API access

1. First you need to know how to find Sheet ID. Perform below steps to find Sheet ID.
2. Goto your portal and open sheet you like to read. Right click on the Sheet Tab > Click Properties and Copy Sheet ID
   

   Find Sheet ID for Smartsheet API call

Once you know Sheet ID. Now lets create SSIS package to extract data from Smartsheet and load into SQL Server.

1. Create new SSIS Package or Open existing one.
2. From control flow SSIS toolbox drag Data Flow task
3. Double click Data flow
4. From SSIS Toolbox drag and drop ZS JSON Source (for REST API / File)
5. Double click JSON Source and configure following properties
   1. Enter Shartsheet API Access URL as below (Change last part with your own sheet id)
      

      https://api.smartsheet.com/2.0/sheets/YOUR-SHEET-ID-GOES-HERE

   2. Check use credentials and then create a new HTTP connection. Select Static Token / API Key and enter enter your token we got in previous section
      
   3. Click on Select Filter and pick Rows node or just type below text in the Filter textbox
      

      $.rows[*]

      
   4. Now goto 2D Array Transformation Tab and set following settings
      1. Set Transformation Type to Complex 2-dimensional array
      2. Column Name Filter to $.columns[*].title
      3. Row Value Filter to $.cells[*].value
         
         

         SSIS Smartsheet API Configuration – Complex Array transformation (Read nested array)

   5. Once everything is configured Click Preview. Click OK to save.
   6. Drag OLEDB Destination from SSIS Toolbox.
   7. Connect JSON SOurce to OLEDB Destination
   8. Double click OLEDB Destination and configure connection (e.g. Select SQL Server connection) and select Target Table or click New to Create new Table based on Source columns.
   9. Click on Mappings to map source columns to target
   10. Click OK to save
   11. Execute Package
       

       SSIS Example : Loading Smartsheet data into SQL Server (REST API Call)

Read Smartsheet data with Pagination option

Smartsheet API support options to read large amount of data in small chunks by using pagination options. ZappySys offers many ways to paginate for various API.

Disable Smartsheet Pagination

Many API endpoints in Smartsheet by default send you paginated response. Pagination requires some extra configuration so try to disable it if possible by using includeAll=true option in your URL as below. If you cannot includeAll for some reason then perform steps listed in next section.

https://api.smartsheet.com/2.0/sheets?includeAll=true

Smartsheet Pagination in SSIS

To paginate Smartsheet response in SSIS JSON Source perform the following steps. Below steps assume that you trying to get all sheets by calling

https://api.smartsheet.com/2.0/sheets

1. Go to Control Flow designer right click > Select Variables
2. Create new string datatype variable called name it TotalPages. Set default value to 0
3. Now drag ZS REST API Task from SSIS Toolbox
4. Enter Following URL (we don’t need many rows to so reduced default page size to just 10)
   

   https://api.smartsheet.com/2.0/sheets?pageSize=10

5. Now click on Raw Edit (Above Headers grid)
6. Enter following header (Replace Token with your own token we obtained in the previous section)
   

   Authorization: Bearer YOUR-TOKEN-GOES-HERE

7. Go to Response Settings Tab
8. Select Content Type Json and enter filter as $.totalPages
9. Check Save response to Variable. Select Variable we created in Previous step (Only string datatype variables are visible)
   

   Save Total Page Count to SSIS variable

10. Click Test Request/Response to confirm it works
11. Click OK to save
12. Now let’s use this variable to configure MaxPageNumber setting in JSON Source. Using following way you can make any Data flow component property dynamic.
    

    Define expression on SSIS Data flow Component Property (Dynamic Value)

13. Now connect your REST API Task to Data flow
14. On the JSON Source go to Pagination Tab and enter following settings.
    

    Smartsheet API Pagination settings – SSIS JSON / REST Source

15. That’s it. Now if you run it you will see all records will be pulled.

Write data to Smartsheet using SSIS Web API Destination (SQL Server to Smartsheet Import)

Now lets see how to write some data to Smartsheet Sheet. We will read records from SQL Server and write to SmartSheet using ZS Web API Destination

For inserting multiple rows we can call this Smartsheet API Endpoint

Basically High level steps are.

1. Find out Smartsheet Column ID to use along with API call
2. Read Records from SQL Server using OLEDB Source
3. Construct desired JSON for Add New Row API Call
4. Send JSON to Smartsheet Add Row API  (Configure Bulk Options )

Find out Smartsheet Column ID

Very first step you have to perform is get Column ID which correspond to certain title. You can use following URL endpoint to get list of all columns (Change your Sheet ID). Make sure you append includeAll=true else it may only fetch 100 columns. You can save Column result to some database table to File to copy ID. Preview Grid also allows to copy cell.

https://api.smartsheet.com/2.0/sheets/YOUR-SHEET-ID/columns?includeAll=true

Get Smartsheet Column ID using API call in SSIS JSON Source

Configure SQL Server Source (OLEDB Source)

Once you have column ID you can drag OLEDB Source from SSIS Toolbox and configure to read from SQL Server. Below is sample source query to extract data from SQL Server.

Select OrderID,CustomerID,EmployeeID,OrderDate from Orders

Configure SSIS Template Transform

Once SQL Source is configure we are ready for next step. SSIS PowerPack v2.6.4 and later introduced new SSIS transform called Template Transform. This transform allows you to produce desired JSON /XML from single input record by using Column name as placeholder anywhere in your text. This is much simpler method than constructing JSON using JSON Generator Transform

Here is the sample JSON we want to produce to insert into Sheet which contains 4 columns OrderID, CustomerID, EmployeeID and OrderDate.

{
  "toTop": true,
  "cells": [
    {
      "columnId": 8146714579232644,
      "value": 10548
    },
    {
      "columnId": 828365184755588,
      "value": "TOMSP"
    },
    {
      "columnId": 5331964812126084,
      "value": 3
    },
    {
      "columnId": 3080164998440836,
      "value": "1997-05-26T00:00:00"
    }
  ]
}

Now lets configure SSIS Template Transform to create desired JSON Request for API call.

1. Drag ZS Template Transform from the SSIS toolbox
2. Connect OLEDB Source to ZS Template Transform
3. Double click Template Transform. Paste above sample request in the Template Text
4. Now highlight sample value you wish to replace from upstream by some input column.
5. Click on Insert Placeholders > Columns >  Your Column
6. Perform above steps for each value you wish to replace. You can also add use placeholder functions such as JSONENC after column name (useful if you expecting new line or special characters such as tab or double quotes in your input text ). Date formatting (e.g. yyyy-MM-dd ) also allowed. See below example after placeholders added.
   

   {
     "toTop": true,
     "cells": [
       {
         "columnId": 8146714579232644,
         "value": <%OrderID%>
       },
       {
         "columnId": 828365184755588,
         "value": "<%CustomerID,JSONENC%>"
       },
       {
         "columnId": 5331964812126084,
         "value": <%EmployeeID%>
       },
       {
         "columnId": 3080164998440836,
         "value": "<%OrderDate,yyyy-MM-ddTHH:mm:ss%>"
       }
     ]
   }

7. Here is final configuration of Template Transform
   

   Create JSON for API Request (POST) using SSIS Template Transform

Configure SSIS Web API Destination (POST New Rows to Smartsheet)

Now lets configure last step. This will POST API request to insert multiple rows to Smartsheet. Previous template transform crafts request for one new row but later in this section we will enable Batch mode so we can JOIN multiple JSON and created Bulk requests in one go so we avoid many API call. Always use Bulk mode if API endpoint supports it.

1. Drag and drop ZS Web API destination from SSIS Toolbox
2. Connect previous Template Transform to your Web API Destination.
3. Select new HTTP connection. When Prompted enter following URL on HTTP Connection UI (Replace your Sheet ID)
   

   https://api.smartsheet.com/2.0/sheets/YOUR-SHEET-ID/rows

4. In the input column for Body select TemplateOutput
5. For Body you can enter Sample Body if you wish to Test at design time else leave it blank
6. Select Body Content Type as application/json
7. In the Headers enter Authorization Header like we did in previous section (Read from Smartsheet)
   

   Authorization: Bearer YOUR-TOKEN-GOES-HERE

   

   Configure SSIS Web API Destination – Add / Insert Rows to Smartsheet (Bulk API call)

8. Go to Batch Setting mode. Change following settings
   1. Check on Enable submitting multiple records
   2. For Body Header enter [
   3. For Body Footer enter  ]
   4. For Body Row separator enter , (i.e comma)
   5. You can change Body Batch Size to desired value (100 is recommended)
      

      Configure SSIS Web API Destination for Bulk API call (Smartsheet REST API – Insert multiple rows)

9. That’s it run your package to test the entire flow.
   

   Loading data from SQL Server to Smartsheet using SSIS (Web API POST Example – Bulk Mode)

Conclusion

Smartsheet API provides great way to automate data read/write. However to call Smartsheet API  you have to use SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides great way to integrate any Smartsheet API call via simple drag and drop approach without coding. Try  SSIS PowerPack for free and call virtually any REST API in few clicks.

Keywords: Import smartsheet into sql server | export smartsheet to sql server | ssis smartsheet read | ssis smartsheet write | reading smartsheet data using API call | write to smartsheet

The post Read / Write Smartsheet data using SSIS REST API Call appeared first on ZappySys Blog.