In our previous post we saw how to read API data in SSIS. This post we will cover how to call NetSuite RESTlet to read / write data in SSIS or ODBC Apps. For demo purpose we will use SSIS PowerPack REST API Task but concepts and many UI elements remain same for ODBC JSON Driver too. This article assumes you know how to write Netsuite RESTlet and you have obtained Credentials to call RESTlet from extranal app.

What is NetSuite RESTlets

NetSuite added a really nice feature to expose business data and workflows as API by writing Javascript functions which they call it RESTlets. You can secure and customize these restlets as per your need.

You can automate CRUD (Create /Read / Write / Delete) operations in NetSuite by writing special functions in JavaScript and expose those functions as API URL which can be called by any external Systems (i.e. SSIS Packages , ODBC Apps like Power BI, Informatica, SSRS).

RESTlet can be secured few ways such as OAuth 1.0 and OAuth 2.0 authentication. We will cover both approach briefly in this article. You can choose any one as per your need

Creating NetSuite RESTlet / Obtain OAuth Credentials (Client ID and Secret)

Below video shows how to obtain necessary credentials to call Netsuites API (RESTlet).

At mark 4:30 seconds it will show you How to use OAuth 2.0 for calling RESTlet

Create NetSuite RESTlet Connection

There are few ways to authenticate NetSuite RESTlet. If you are using ZappySys Products then it makes it super easy to use either approach. Lets look at both approaches.

Using OAuth 2.0 (Recommended)

As per above video if you obtained Client ID and Secret for OAuth 2.0 you can follow below steps to configure OAuth connection to call RESTlet.

What you need to use OAuth 2.0.

• Your Account ID (it goes in URL)
• Your Client ID and Secret (Obtained from Netsuite Portal) – See above video Mark 04:30
• Register Redirect URL in NetSuite portal (we recommend you use https://zappysys.com/oauth as call back Url for Automatic Code extraction)

Steps to configure OAuth Connection

1. Right click on Connection Panel and choose Choose “New Connection..”. Select ZS-OAUTH type
   
   
   

   Create new SSIS OAuth API Connection Manager

    

    

2.  Enter following things
   ————————-
   On General Tab
   ————————-
   ClientID and Secret: <get it from NetSuite>
   Authorization URL: https://<accountID>.app.netsuite.com/app/login/oauth2/authorize.nl
   Token URL: https://<accountID>.suitetalk.api.netsuite.com/services/rest/auth/oauth2/v1/token
   Scope: restlet
   

   Configure NetSuite OAuth 2.0 Connection for API call

    

   ————————-
   On Advanced Tab
   ————————-
   Callback / Redirect URL:  https://zappysys.com/oauth  (this must be entered same way in NetSuite Portal)

   

   OAuth2 Connection – Callback / Redirect URL

3. Now go back to General Tab and click Generate Token button
4. Follow login screen and try to finish the flow. At the end it will generate Tokens and populate on UI
5. Click OK to save

Using OAuth 1.0

Another approach is use OAuth 1.0 (Older version). If you have enabled OAuth 1.0 in NetSuite then follow below steps (Create connection is same as before).

1. Setup General Tab and Advanced Tab as below.
   On ==== General Tab====:
   Enter ClientID, ClientSecret, AccessToken, AccessToken Secret
   On ==== Advanced Tab ====:
   Check Do Not Include Empty Key/Value Pairs
   Select Signature Method as HMAC-SHA256
   For Extra Token Attribute enter realm=<your-account-number>_SB1
2. Click OK to Save

Call NetSuite RESTlet in SSIS

1. Create a new SSIS Package
2. Drag REST API Task from SSIS Toolbox
3. Double click the Task to configure
4. Select URL from Connection
5. From Connection dropdown select OAuth connection we created in the earlier section (either OAuth 1 or OAuth 2)
6. Enter the URL as below. Assuming you copied URL from NetSuite Replace script ID and account number
   

   https://<account-number>-sb1.restlets.api.netsuite.com/app/site/hosting/restlet.nl?script=<scriptid>&deploy=1

   Enter <account-number>
   Enter <scriptid>
7. Change Method to POST if your script supports Parameters (Else you can supply id=xxxx and recordtype=xxxxx in the URL for GET request)
8. If its POST request then chnage ContentType to ApplicationJSON (application/json)
9. Enter your Input in the body. In below example our RESTlet takes sql query as input so we enter just one parameter named query but you can have different parameter(s)
   

   Syntax:  { param1_name : <value> , param2_name : <value> ....  }

10. Now click Test see everything works
    

    Calling NetSuite RESTlet in SSIS (Using REST API Task)

Conclusion

We saw how easy it is to call NetSuite RESTlets or other NetSuite API in SSIS / other BI Apps using ODBC Drivers provided by ZappySys. Download SSIS PowerPack and try your self see how easy it is to consume virtually any API in SSIS.

The post How to call NetSuite RESTlet API in SSIS / ODBC appeared first on ZappySys Blog.

In this post, you will learn how to read / write ManageEngine ServiceDesk data in SSIS (Drag and drop approach without any coding).  We will use SSIS JSON/ REST API Source to extract data from Servicedesk API and use SSIS Web API Destination to write data to Zoho.

We will look at step-by-step instructions to read ServiceDesk  data (e.g. Request, Problems, Contacts) and load into SQL Server Database Table.

Authentication ServiceDesk REST API in SSIS (Use OAuth 2.0)

Very first step to call ServiceDesk API is you need to register OAuth App in Zoho Portal. Don’t get confused if you see Zoho Portal for ServiceDesk a Product from ManageEngine is in fact Zoho subsidiary (in other words Zoho is the parent company). ServiceDesk has On-Premises version too but this article is only about Cloud version.

Register Zoho OAuth App and get ClientID/ Secret (*** MUST READ ***)

Check this article to register your OAuth App and obtain Client ID  /Client Secret

Create ServiceDesk API Connection in SSIS

Once you create Zoho OAuth App now its time to call some Zoho API. Let’s create the OAuth Connection first. In the next section we will see how to load data in SQL Server.

1. Right click on Connection Panel and choose Choose “New Connection..”. Select ZS-OAUTH type
   

   Create new SSIS OAuth API Connection Manager

2. Configure your OAuth connection General Tab as below
3. OAuth Provider: Custom
4. OAuth Version: OAuth2
5. Client Id , Client Secret
6. Enter One Scope Per Line (For more info on scope read here) . For write you can change READ to WRITE or use ALL as per the help link.
   

   SDPOnDemand.requests.READ

7. Auth URL : https://accounts.zoho.com/oauth/v2/auth
8. Token URL:  https://accounts.zoho.com/oauth/v2/token
9. Configure Advanced tab as below
10. Enter Callback URL : https://zappysys.com/oauth
11. Enter Extra Attribute for /Auth : access_type=offline&prompt=consent
    This is needed to obtain refresh_token (if you ommit this then only access_token returned which cannot be renewed once expired after 1hr)

    

12. Now go back to general tab and Click Generate Token. Close the Popup and use Full browser instead if you see prompt.
13. Click OK to save the connection UI

Read data from ServiceDesk API in SSIS JSON Source

In this section we will learn how to read data from ServiceDesk using SSIS JSON Source. We will use OAuth connection created in earlier section.

Now lets look at step by step example of reading ManageEngine ServiceDesk data using SSIS (e.g. Requests).

ManageEngine provides JSON API for read action so we will Use SSIS JSON Source for this purpose.

Configure JSON Source

So below steps shows example of reading from ManageEngine Requests API

1. Open SSIS Package
2. Drag data flow task from Control flow SSIS Toolbox
   

   Drag data flow from SSIS Toolbox

3. Go to data flow and drag ZS JSON Source from SSIS Toolbox
   

   Drag and drop SSIS JSON Source (REST API or File)

4. Double click JSON Source to configure. Enter URL as below. You can supply few more parameters. Click Here to get full list of parameters and usage. In below example we are reading records from Leads Module but you can replace Leads to something else (e.g. Accounts). Refer to above help link for available Modules.
   

   https://sdpondemand.manageengine.com/api/v3/requests

5. Check Use Credentials. Select OAuth connection manager we created in the previous section.
6. Add the Accept header in HTTP Headers.
   

   Accept: application/vnd.manageengine.sdp.v3+json

7. Under filter options tab, enter Array Filter as  

   $.requests[*]

8. Now click Preview to see your data
9. Click OK to save UI. In the next section, we will see how to setup pagination and incremental extract of Servicedesk records. After that we will see how to load data into SQL Server Table.

Read ManageEngine ServiceDesk Plus data in SSIS (Cloud Version) – Zoho API

ServiceDesk API Pagination (Read all Requests)

Just like most API ServiceDesk also uses pagination. See below to learn how to configure pagination in JSON Source for ServiceDesk API.

1. Double click JSON Source
2. Change URL to include two things (Page size (how many rows per page and start row). Note <%page%> must match with Page Num Indicator you going enter in the next screen
   

   https://sdpondemand.manageengine.com/api/v3/requests?input_data={"list_info":{"row_count":100,"start_index":<%page%>}}

3. Go to Pagination Tab and select URL Path Mode
4. For Page Num Indicator : Enter

   <%page%>

5. For Increment By : Enter

   100

6. Page Start Number select New variable call it StartIndex and start default value 1

Once you configure this way now your API will return all records until last page is detected. For more information on pagination check this article

How to paginate ManageEngine ServiceDesk API

Loading ServiceDesk data into SQL Server Table / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Write data to ServiceDesk using OAuth 2.0

If you like to load data from any source (i.e. SQL Server) to Zoho CRM then you can use Records API (POST). Use a combination of the following Transforms  / Components. Later in this article we have some more examples (Older API) which may give you one more ways to load data into Zoho (Use of Template Transform).

Basically you have to build POST Body using JSON Generator Transform or Template Transform and then use that data in Web API Destination to call POST API request to push data to ServiceDesk. Basically screnshot below is just an example using some other Zoho API but you get the idea.

	Web API Destination
	JSON Generator Transform
	JSON Parser Transform

Create JSON for POST request (Single dataset pattern)

Insert data to Zoho Table using SSIS (Lead, Account Modules) – Bulk Load using POST API call

Write data to ServiceDesk using SSIS (Import SQL Server Table to ServiceDesk)

Now you know how to read data from Zoho CRM using JSON API Source next step is to load data to Zoho CRM. For writing data we will use following three components

	Web API Destination
	Template Transform
	XML Parser Transform

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Conclusion

Manageengine  Servicedesk API provides a great way to automate data read/write operations. However, to call Service API  you have to use SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Servicedesk API call via a  simple drag and drop approach without coding. Try  SSIS PowerPack for free and call virtually any REST API in a few clicks.

The post Read/Write ManageEngine ServiceDesk Data in SSIS (Zoho API) appeared first on ZappySys Blog.

In this article, we will show you steps to register OAuth App for Zoho API and use it ZappySys Product. The steps listed here can be used for Zoho Products like CRM or ManagedEngine Products like ServiceDesk (ManageEngine is Zoho’s subsidary). For detailed steps check Zoho’s link here.

Register OAuth App in Zoho

1. Go to Zoho API Console
2. Click on Add Client  link
   
3. Click on Server-Based App option
   
4. When prompted enter following
   1. Your APP NAME
   2. Your company Homepage URL (any)
   3. Redirect URL (We suggest you use https://zappysys.com/oauth )
   4. Click CREATE
5. After you create app you will see ClientID and ClientSecret (Copy them somewhere safe location). You will need this to enter on OAuth UI
   

Using Zoho OAuth in ZappySys Connection

Now last step is using 3 pieces on ZappySys OAuth Connection UI (same UI in ODBC PowerPack or SSIS PowerPack Products)

1. Depending which Product you are using find out OAuth Connection UI Edit / New Connection option and enter below
2. On General Page enter
   1. ClientID,
   2. ClientSecret
   3. Scopes (Check API help page to find these.. each API requires different scope)
   4. Authorization URL and Token URL as below (if its US Region) . For Other regions refer this article to get URL

      https://accounts.zoho.com/oauth/v2/auth

      https://accounts.zoho.com/oauth/v2/token

      
3. Now go to Advanced Page and enter below
   1. Callback URL (exact same URL you configured in previous section)
      We suggest to use :

      https://zappysys.com/oauth

   2. Extra Auth Attributes like below
      

      access_type=offline&prompt=consent

      

4. Now go to General Tab again and click >>Generate Token<<
5. Close Popup Browser and use Full Browser when Prompted
6. Follow the steps … If successful you will see Access Token and Refresh Token extracted on the OAuth Connection UI

The post How to register OAuth App for Zoho / ManageEngine API appeared first on ZappySys Blog.

In this section, we will discuss how to register OAuth App to call Salesforce API using Client ID / Secret rather than using your User Id / Password. Once you follow these steps below to register your Salesforce App (OAuth App), at the end you will get a Client ID (sometimes referred to as App Id) and Client Secret (or App Secret). These two pieces can be used later on to call Salesforce API using OAuth. So let's get started.

Register Salesforce App (OAuth)

1. Login into your account and go to Setup as shown below (click on gear icon at the top right corner):
2. Then in the search bar input "app" and open App Manager:
3. Then click New Connected App to create a new OAuth App: NOTE: In some version, it may say New External Client App

   Look for one of the following buttons:

   • New Connected App (Classic name)

   • New External Client App (Lightning name)

   
4. Give your App a name and: - Enable OAuth Settings - Set callback URL to following two URLs (one in each line) https://login.salesforce.com/services/oauth2/success https://zappysys.com/oauth - Add an OAuth scope you want or simply add "Full access". - Add "Perform requests on your behalf at any time (offline_token)" scope to get a refresh token so that you don't have to re-authenticate each time requesting an access token: IMPORTANT: You can use "Access and manage your data (api)" if you do not like to grant full permission to this app but make sure you request same permission from the caller app too  (Example on ZappySys Salesforce Connection UI -> Advanced Tab -> Set scope as "api offline_access" rather than "full offline_access"
5. Click "Save".

Get Client Id and Client Secret for Salesforce Connected App

Once the App is created, we need to find out the App Id / Secret so we can use it for API calls.

1. Go to App Manager
2. Find out the recently created App and click View (click on the small arrow to the right)
3. Once App page is loaded you can Copy Consumer Key and Consumer Secret (click on reveal to see). Also copy anyone Call back URL (We will need this for OAuth Connection > Advanced Tab > Call Back URL or Redirect URL property depending on which ZappySys Product / Driver you are using)
4. That's it, you can now use these 3 pieces to ZappySys OAuth connection UI or on Salesforce Connection UI (new version with OAuth support). The above screenshot shows where to enter these 3 pieces in the New Salesforce Connection. For the OAuth Connection, the redirect URL goes to the Advanced Tab > Callback URL textbox.

Get Client Id and Secret for Salesforce External Client App

If you created a new App by clicking on the New External Client App button, then your App may not be visible in the App Manager Screen. In that case, find it below way. [caption id="attachment_11637" align="alignnone" width="1222"] Get Client Id, Client Secret - For New External Client App[/caption]

Enable Client Credentials Grant (Non-Interactive Run-As)

If you like to enable Non-Interactive mode, just like UserId / Password Authentication, but without actually passing a real UserId or Password on the Connection UI, then you can enable Client Credentials Flow as below. NOTE: Only the New version of ZappySys Salesforce Connection supports Client Credentials Grant. Version released after Nov 1, 2025 supports this grant type. [caption id="attachment_11639" align="alignnone" width="419"] Enable Salesforce Client Credentials Grant (Run As)[/caption]

(OPTIONAL) Configure Salesforce OAuth refresh token validity

By default, Salesforce Refresh Token doesn't expire once you generate. If you want to change the default behavior, follow these steps.

1. Go to "Manage Connected Apps" menu item and press on your App name:
2. Click "Edit Policies" to configure refresh token validity:
3. Set "All users may self-authorize" and "Refresh token is valid until revoked" options:
4. Click "Save".

Using Salesforce Connection with OAuth App in SSIS / ODBC

Once you have Client ID / Secret, you can use it in the Salesforce Connection in SSIS Connector / ODBC Driver for Salesforce like below (Only the new version will have OAuth option)

Troubleshooting Known Issues

Let’s look at some common issues with Salesforce Connection and how to fix them.

OAuth Error: Missing Required Code Challenge

When you click Generate Token button, you may get an error like the one below.

Salesforce OAuth Generate Token Error – Missing Required Code Challenge

Solution

Here is how to fix it.

For Salesforce Connection UI: Go to Advanced Tab => Check Enable PKCE Code Verifier option
–OR–
For OAuth Connection UI: Go to OAuth2 Grant Options Tab => Check Enable Code Verifier Option

Enable PKCE code verifier option – OAuth Connection for Salesforce.

The post How to register Salesforce App and obtain Client ID / Secret (for API Call / OAuth) appeared first on ZappySys Blog.

In this article, we will show you step by step approach on how to register an Azure AD (Active Directory) app to access Dynamics CRM 365 API (also same for CDS / Dataverse API). You might have heard that Microsoft is deprecating an older way of authenticating (i.e. User ID / Password-based WS-Trust Authentication) on April 2022. You must update your jobs / code to use OAuth 2.0 by April 2022 for any CRM related API calls.

In other words rather than User Id / password now you can use more secure way by using OAuth rather than WS-Trust.

Going forward Microsoft wants you to use OAuth Token rather than user name/password to call Dynamics CRM 365 API programmatically.

Here is the official link which describes the dates for WS-Trust sunset

Register App for CRM / CDS / Dataverse in Azure / Active Directory

The first step is to register your app in Azure Portal under Active Directory using the below steps.

1. Ask your Active Directory Admin to login to Azure Portal and navigate to > Active Directory Blade.
   OR
   You can access it by searching “Azure Active directory” like below.
   

   Open Azure Active Directory

2. Now click on the App Registrations icon and click New App Registration
   
   

   New App Registrations under Azure Active Directory

3. Now next step is to give your App name to identify easily. Select App is used for your Org only option (Default) and last enter Redirect URL as below (This URL is used for 3-legged Authorization Code Grant only). We prefer you enter a URL which is OAuth friendly (If you don’t know one Use

   https://zappysys.com/oauth

     this is recommended URL if you working with ZappySys Products)
   

   Azure AD App Registration – Single / Multi-Tenant with Redirect URL (i.e. Call back)

    

4. Now once App is created you can find the Application (Client) Id from overview page
   

   Azure AD App – Find Application / Client Id on overview page

Add API Permissions for the App

Once App is created we need to Add Permission. Basically, we will set which style of APIs can be called by this App and what Actions can be performed.

1. To add API Permissions for this AD App click on API Permissions and then click on Add Permission
   

   Add API Permissions for Azure AD App

2. Find the Dynamics CRM icon and click on it
   

   Add API Permission for Dynamics CRM API access (Azure AD App)

3. Now you will see just one permission for Dynamics CRM (user_impersonation). As we said earlier for Dynamics CRM API security is controlled by User Roles in CRM Portal itself which will see in the next section. For now, just choose Delegated Permissions,  check user_impersonation and click Add Permissions
   

   Select API Permissions for Dynamics CRM App (Azure AD App)

4. Now the last step in Adding API permissions is Grant Admin Consent. If you are logged in as an Azure AD User with the correct admin role then no worries else you have to ask your Azure AD admin to consent for you so this App can be used to read/write data from CRM.

Create Client Secret

Once App is created next step is to generate a Client Secret. Perform the following steps to create a secret that you can use when you call API

1. Click on Certificates and Secrets and then New Secret
   

   Azure AD App Create Client Secret / Set Name, Expiration Date

2. Enter Name for Secret (You can give some friendly name if you create multiple secrets to control multiple processes). For example SecretForTesting, SecretForProduction etc.Select a longer expiration date (i.e. 24 months) if you don’t like to change secret often in production jobs else choose shorter expiration (i.e. 6 Months or Custom).
3. Once you Create a Secret make sure you copy it to a secure location. Once the page is closed it won’t appear again.
   

   Copy Client Secret for Azure AD App

Associate App User / Set Roles in CRM 365 / Dataverse Environment

Once we have App Id and Client Secret we can go to CRM Portal to add our App as a valid CRM User. And the last step will be to attach some role(s) to App User (See Next section).

Method-1 (New way)

Here are the steps to configure User / Security Roles in New account. Below steps will give you access to Dataverse Entities via API Call. To see all Dataverse Entity you can log in to https://make.powerapps.com/ and click Tables

1. Click on the below link to login to new portal to manage CRM /Dataverse environment
   https://admin.powerplatform.microsoft.com/environments
2. Create an Application User using these steps (When prompted choose the same App we created in the previous section)
3. Now Go to App Users Screen and Edit Security Roles using the below steps
   

   See available CRM 365 Online / Datavese Environments

   

   See available CRM 365 Online / Datavese Application Users

   

   Edit Security Roles for CRM 365 Online / Datavese Application Users

Method-2 (Legacy)

If you prefer to use an older interface (and it’s still available to you) then you can use the below steps to perform the same thing we described in the previous section.

NOTE: Microsoft may soon retire the older interface and force to use the previous way only.

1. To Add our Azure AD App as CRM Application user goto your CRM Portal. Click on Gear icon for Settings > Advanced Settings
   

   Dynamics CRM Advanced Settings

2. Click on Settings dropdown > Security (Under System Column)
   

   Dynamics CRM Security Settings (Under Advanced Settings – System)

3. On the security, screen click on the Users icon
   
   

   Manage Dynamics CRM Users (Add, Edit, Delete)

4. Once User Screen shows up click on the User Type Dropdown and choose Application Users and click New
   

   Add new Application User in Dynamics CRM

5. Once User screen shows up. Enter your Application Id (obtained from the Azure AD App > Overview screen – app created earlier). Click Save and you will notice all other fields auto populates.
   

   Adding Azure AD App as Dynamics CRM 365 – Application User

6. And the last step is to Attach some CRM Role to the new Application User.
   For demo purposes we will choose “System Admin” but for production use, you should grant correct permission carefully. There are many inbuilt roles you can choose from.
   

   Associate Security Role(s) for Dynamics CRM Application User

Find your Organization Id for CRM / Dataverse API calls

Now find your Org Id before we can call CRM / Dataverse APIs in the next section.

To find your Org Id you can perform the following steps.

Method-1 (New way – Dataverse Portal)

1. Go to https://admin.powerplatform.microsoft.com/
2. Click on the Environment you like to access
3. On the Details, page copy your Org Id from the Environment URL For example URL might look like below… You just need to copy the first part (Use this on CRM Connection > Organization Field)
   org8640f7e7.crm.dynamics.com
   

   How to find Organization Id for Microsoft Dataverse / CDS / CRM 365 API calls

Method-2 (Old way)

Another way to find your Organization Id is from CRM Portal

If you have multiple environments (i.e. Prod, Sandbox) then each environment might have its own unique URL)

1. To find your API Service URL, Click on Gear Icon
2. Choose Advanced Settings
3. Click down arrow on Settings
4. Click Customization
5. Click Developer Resource Link
6. On this page now you can see various URLs. For examples used in this article, we need to get is Service Root URL for Web API
   

   How to Find Organization Id / Web API URL / SOAP API Service URL (Unique Environment Instance URL)

Using Client ID (i.e. App Id) and Client Secret to Call Dynamics CRM 365 API / CDS / Dataverse

That’s it now we are ready to use this App Id and Client Secret to make CRM API calls.

To confirm if it’s working you can create a Dynamics CRM Connection (using SSIS PowerPack ). Choose OAuth option on Dynamics CRM Connection

Connect to Dynamics CRM / Dataverse / PowerApps API using OAuth (Client ID /Secret)

Conclusion

In this article, we saw how to Register Azure AD App to call various Microsoft APIs (for this article we focused mainly on Dynamics CRM 365 API only)

The post How to register App for Dynamics CRM 365 / CDS / Dataverse API (Azure AD / OAuth) appeared first on ZappySys Blog.

If you have need to integrate REST API in SSIS (i.e. fetch JSON, XML, CSV via HTTP call) and wondering how to Read API data in SSIS and Load into SQL DB (i.e. SQL Server, MySQL, Oracle) then you are at the right place.

In this article we will cover simple step by step approach how to achieve API integration in few clicks. For detailed introduction please refer to our previous article.

In next few sections we will cover the following SSIS Components to read data from API like a Table (Parse in Rows / Column). Our instructions will be mostly for JSON Format but concepts are same for other formats (i.e. XML / CSV).

	JSON Source Connector (Read from REST API, JSON File or OData Service): Use this dataflow component when you have to fetch data from REST API webservice like a table. This component allows you to extract JSON data from webservice and de-normalize nested structure so you can save to Relational database such as SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local JSON files or direct JSON string (Wildcard pattern supported too e.g. c:\data\file*.json).
	XML Source Connector (SOAP, File, REST) : Use this dataflow component when you have to fetch data from XML or SOAP webservice and consume data like a table. This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local XML files or direct XML string.
	CSV Source Connector (File, REST) : Use this dataflow component when you have to fetch data from File or REST API which returns data in CSV format . This component allows you to extract data from webservice and save to SQL Server or any other target (Oracle, FlatFile, Excel, MySQL). This component also supports reading local CSV files or direct CSV string.

There are few more API components not discussed in this articles. They are REST API Task (no Parser – Just raw API calls) and Web API Destination (Write / Send Data To API / Delete rows). They both have different use case to check them see it fits in your need.

Video Tutorial

Step-By-Step – Read API data in SSIS using JSON / XML / CSV Source

In this section you will learn how to use JSON Source Adapter to Read data from API URL.

Load API data into SQL Server Table / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

API Authentication

In our step by step example we did not cover Authentication to keep things simple but in real world you have to pass some sort of authentication details to make API calls (e.g. UserID/Password or Secure Token). There are many ways to authenticate with API Services. ZappySys offer mainly 2 Connection Managers to cover most Authentication scenarios.

1. HTTP Connection Manager
2. OAuth Connection Manager

Using HTTP Connection Manager

If you have use case of using UserID / Password or any other method supported by HTTP Connection Manager then use below option (Below example is for Basic Authentication)

SSIS JSON Source – Passing Basic Credentials (Supply UserID / Password) using HTTP Connection

Using OAuth Connection Manager

If your API support OAuth Standard for Authorization then use OAuth Connection Type as below.

Generate Token for Google API using OAuth 2.0

API Pagination

Pagination is another important concept to understand when you call API using ZappySys Connector. If you ready many records from your API then by default API might not return all rows so you have to paginate until all records are read (Like a Loop in Programming). ZappySys makes it easy to paginate. Check this article to configure many different Pagination Modes

Things have gone bad: Error handling & debugging

Incidentally, bad things can happen. A remote server may go offline or your server may go out of memory. In any case, you may want to know when that happens and take actions accordingly. For that purpose, you have to redirect bad rows to some other destination. For this example, we will take and use Web API Destination, but basically, you can use any SSIS component:

Handling errors

1. Add a Derived Column above Web API Destination with expression "(DT_WSTR,4000)ZS_JSON_OUT" and name it "JsonAsString". This will let you see what JSON you are actually passing.
2. Then add a database or file destination or use another Trash Destination for debugging purposes and redirect the bad rows (red arrow) from Web API Destination into it. Don't forget to set Redirect row option for both, Error and Truncation columns:
   

   Redirected failed requests from Web API Destination to a desired destination when loading from SQL Server to REST API Service is failing. Derived Column JsonAsString added to be able to read JSON which was passed to Elasticsearch

3. Finally, add a Data Viewer for the red path, if you want to debug the flow. You will be able to see URL, JSON and the error message for each record. You may want to copy-paste ErrorMessage to Notepad if you want it to be more readable:
   

   Use Data Viewer to view HTTP requests that failed to be fulfilled.

NOTE: You can read more about redirecting rows in SSIS Error Handling (Redirect bad rows) article.

Debugging HTTP requests

A common thing you have to do when working with HTTP requests is to debug those requests; e.g. to check what headers, body or URL was passed. To test how things look behind the scenes we strongly suggest to use Fiddler - a popular web debugging tool.

Inside it, you can double-click the URL entry (Right side) to see Request and Response Panels. The top panel is Request (URL, Headers, Body) and Bottom Panel is Response. For https:// (secure URL) make sure you enable HTTPS option in Fiddler (Tools > Options > HTTPS > Check Decrypt https request):

Debugging Web API call using Fiddler in SSIS

Conclusion

REST API is becoming more and more popular each day. With that Data integration from RESTful API services going to be challenge. Luckily ZappySys SSIS PowerPack provides a great way to integrate any API in SSIS via simple drag and drop approach without coding. Try SSIS PowerPack for FREE  see how much time / money you can save and to integrate virtually any REST API.

The post How to Read API data in SSIS and Load into SQL Table appeared first on ZappySys Blog.

In this tutorial, we will use the REST API Task to call some API. We will use the OAuth connection along the REST API Task to write / update data to Google Spreadsheet.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Google API.

Component Mention in this article

	JSON Generator Transform
	Web API Destination
	REST API Task

Create a Google API Project

The first step to access any Google API is to create an API Project in Google Console. If you don’t want to go through this then Skip this Step-1 and in the next section select the Default OAuth App option on OAth Connection Manager (This is the easiest option for now unless you want to use your own OAuth App).

Check this article for step-by-step instructions. When you follow these instructions make sure you enable Google Sheet API.

Create OAuth Connection Manager in SSIS

Once you create a Google API project and obtained a Client ID and Client Secret your next step is to create OAuth Connection Manager in SSIS. ZappySys OAuth connection manager comes with many predefined OAuth Providers (e.g. Facebook, Twitter, Google, etc) but you can also define custom OAuth settings for any OAuth enabled API.

To create SSIS OAuth 2.0 Connection for Google API perform the following steps.

1. Download and Install SSIS PowerPack.
2. Create a new SSIS Package.
3. Right-click in Connection Manager Area and Click “New Connection”.
4. When prompted select ZS-OAUTH connection type, On the OAuth Connection Manager Select Provider=Google.
5. Click here to create credentials. Then select the “Use Custom OAuth App” option and enter your ClientID, ClientSecret.
6. In the Scopes enter below scopes (or click Select Scopes). This will allow read/write access to Drive Files or Sheet (E.g. Reading file content / Export file to CSV). You can use the Scope browser to see many other available permissions. For more information click here.
   

   https://www.googleapis.com/auth/drive
   https://www.googleapis.com/auth/drive.file
   https://www.googleapis.com/auth/drive.readonly
   https://www.googleapis.com/auth/spreadsheets
   https://www.googleapis.com/auth/spreadsheets.readonly

7. Click on the Generate Token button.
   

   SSIS OAuth Connection Manager – Access Google Drive API using OAuth 2.0

8. You will see UI as below… Click Allow (You may have to scroll to see that button sometimes).
9. Click on Test Connection to see the connection is working and click on the OK button to save configure setting UI.

Find Google Sheet ID for API calls

Before we can write or update Sheet data in SSIS we need to know Google Driver File ID. We will use this ID in the next few sections.

The easiest way to find File ID is to look at the URL like below. If you need to know ID at runtime dynamically then refer to the last section.

Get Google Drive File ID for API call (Sheet ID)

Create Variables to make dynamic URL

1. Here, we are going to make a Dynamic URL value.
2. In Visual Studio Project Right Click on Design Panel, Select Variables.
   

   Create Variables

3. Let’s Create Variables with correct DataType and Value. In variable “varRange” set only the starting point(Sheet1!A2) of data if you don’t know the ending point and leave the first-row for giving header row later or click here.
   

Write SQL Server data to Google Spreadsheet

Now let’s look at a real-world example. How to write data from SQL Server table or other source and send to Google Sheet using Web API Destination.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Dragging and dropping Data Flow Task into Control Flow

2. Double click on the Data Flow task to see the Data Flow designer surface.
3. Here, In Visual Studio, drag and drop the OLE DB Source, JSON Generator Transform and Web API Destination in the design surface and join the components with the blue arrow.
   

   Drag and Drop OLE DB Source, JSON Generator Transform, and Web API Destination

4. Here, you can use our Free ZS Dummy Data Source too. Click here for more information.
5. But, we are going through OLE DB Source so we need OLEDB Connection.

Create OLE DB Connection

1. Lets, create the OLE DB Connection.
2. Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
   

   Create OLE DB Connection

3. Now, click on New Button to create Connection.
   

   Create New Connection

4. Let’s Configure Connection Manager, just Follow steps one by one as we have created.
   

   Configure OLE DB Connection

5. Click on Test Connection to see correct configure it.
6. Click on the OK button to save connection configure setting UI.

Configure JSON Generator Transform

1. Double click on JSON Generator Transform to configure it.
2. Set Output Mode to Single Dataset Array. Right-click on Mappings, select Add Static Element then enter Name: “range” and Select variable we have created for value.
   

   Configure JSON Generator Transform – Add Static Value

3. Again, Right-click on Mappings, select Add Static Element then enter Name: “majorDimension” and Value: “ROWS“.
   

   Configure JSON Generator Transform – Add Static Value

4. Now, Right-click on Mappings, select Add Document Array, Check to Treat as 2D array option and enter Element Name as “values“.
   

   Configure JSON Generator Transform – Add Document Array

5. Then, right-click on Document Array, select Add elements > Select Add Multiple Columns and check on columns you like to add.
   

   Configure JSON Generator Transform – Add Element

6. Click on the OK button to save JSON Generator Transform configure setting UI.

Configure Web API Destination

1. Double click on Web API Destination to configure it.
2. Select the OAuth connection manager created in an earlier section (Assuming you had all scopes with write permission).
3. In the Setting tab,
   1. In URL, Generate Dynamic URL like below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4:append?valueInputOption=USER_ENTERED

   2. Change Request Method to POST.
   3. Select JSON Generator variable into Body (Request Data) or you can select Input column for Body (JSON Generator Output).
   4. Set application/JSON to Body Content-type.
      

      Configure Web API Destination

4. Click on the OK button to save Web API Destination configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet. Now update header row values for click here.
   

   Run or Execute Package and verify data into Google Spreadsheet

Append data to Google SpreadSheet using REST API Task

Now let’s look at how to write to Google SpreadSheet. To write multiple rows in a single request. Also, check this link to learn more.

For more information, you can see the example here. For parameter, detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve the Spreadsheet write scenario. For a simple example, let’s use the REST API Task.

1. Drag and Drop ZS REST API Task on the control flow designer.
   

   Drag and Drop REST API Task

2. Double click on ZS REST API Task to configure it.
3. In the Request Settings tab,
   1. set Request URL Access Mode to Url from a connection. Select OAuth Connection from the dropdown (Same connection we created in the previous section).
   2. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A1:append?valueInputOption=USER_ENTERED

   3. Change Request Method to POST.
   4. Enter Request Body as below
      
      Note: For example purpose, we included header line in data cells but in the real-world header may already exist in the sheet.

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["Item", "Cost", "Stocked", "Ship Date"],
          ["Wheel", "$20.50", "4", "3/1/2016"],
          ["Door", "$15", "2", "3/15/2016"],
          ["Engine", "$100", "1", "3/20/2016"],
          ["Totals", "=SUM(B2:B4)", "=SUM(C2:C4)", "=MAX(D2:D4)"]
        ]
      }

   5. Change Request Content type to application/JSON.
      

      Write data to Google SpreadSheet using REST API Task (OAuth 2.0 Connection)

4. Click on the OK button to save REST API Task configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.
   

   Execute REST API Task and verify Google Spreadsheet data

Update Google Sheet Cells

Now let’s look at the below example of updating multiple cells in google Spreadsheet. For more information, you can see the example here. For parameter, detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve the Spreadsheet Update scenario. For a simple example, let’s use the REST API Task.

Update Google Sheet header cells manually

1. Drag and drop ZS REST API Tasks on the control flow designer.
2. Double click on the First REST API Task to configure it.
3. In the Request Settings tab,
   1. Enter the URL below. Change Range designator(including Sheet name: Sheet1!A1) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A1?valueInputOption=USER_ENTERED

   2. Change Request Method to PUT.
   3. Enter Request Body as below
      

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["CustomerID", "CustomerName"]
        ]
      }

4. Click on the OK button to save REST API Task configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.
   

Single Range (Set Values for multiple cells)

1. Follow the previous section(write data to Google Spreadsheet) to update sheet data.
2. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4?valueInputOption=USER_ENTERED

   2. Change Request Method to PUT.
   3. Enter Request Body as below
      

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["Laptop", "$2000.50"],
          ["Mobile", "$1150"]
        ]
      }

3. That’s all, click on the Test Request/Response button and verify data into Google spreadsheet.
   

   Calling Google Sheet API to Update Multiple Cell Values

Clear data to Google SpreadSheet using REST API Task

Clear a sheet of all values while preserving formats

Lets clear a sheet of all values by REST API Task, Before processed we need sheet Id. So let’s get the SheetId by API Call. For more information about click here.

1. Drag and Drop two ZS REST API Tasks on the control flow designer and rename(Get Sheet ID and Clear Google Sheet of all values) it.
2. Double click on the First REST API(Get Sheet ID) Task to configure it.
3. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc

   2. Change Request Method to GET.
      REST API Task – Get Google Sheet ID
4. Now go to Response Settings Tab, Select Response content Type Json and Write Response Content Filter Expression. Check on Save Response Content. Select Save Mode to Variable and Create a New variable or select it.
   Example of Response Content Filter Expression for the first Sheet : 
   

   $.sheets.[0].properties.sheetId

   

   REST API Task – Save Google Sheet ID into Variable

5. Now, click on the Test Request/Response to check configures are ok.
6. Click on the OK button to save REST API Task configure setting UI.
7. Double click on the second REST API(Clear Google Sheet of all values) Task to configure it.
8. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}:batchUpdate

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc:batchUpdate

   2. Change Request Method to POST. Enter Request Body as below and change a variable parameter as per your need.
      

      {
        "requests": [
          {
            "updateCells": {
              "range": {
                "sheetId": {{User::varGoogleSheetID}}
              },
              "fields": "userEnteredValue"
            }
          }
        ]
      }

   3. Change Request Content type to application/JSON.
      
      

      REST API Task Configure – Clear sheet of all values

9. Click on the OK button to save REST API Task configure setting UI.
10. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.

Clears range cell values from a spreadsheet

1. Double click on REST API Task to configure it. Then, follow the way just like the previous section configures REST API Task.
2. Here you need to set the value of Variable “varSpreadsheetValues” to “clear” or you can change it directly in URL append to clear and value of variable “varRange” (Change cell range and sheet name) as per your need. For more information about it click here.
   1. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4:clear?valueInputOption=USER_ENTERED

3. Change Request Method to POST.
4. Enter Request Body as below
   

   {}

5. Change Request Content type to application/JSON.
6. That’s all, click on the Test Request/Response button and verify data into Google spreadsheet.

Conclusion

Google Drive API and Google Sheets API provide a great way to automate file-related functionality. However, to call Google API  you have to use the SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Google API call via simple drag and drop approach without coding. Try out yourself see how long it takes to call virtually any REST API.

References

Finally, you can use the following links for more information:

• JSON Generator Transform
• Web API Destination
• REST API Task
• Help File:
  • JSON Generator Transform
  • Web API Destination
  • REST API Task

The post How to write/update Google Sheet in SSIS appeared first on ZappySys Blog.

In our previous few posts we saw how to call various google apis in SSIS.  In this post lets learn how to call Google Search Console API in SSIS or other ODBC Compatible Apps such as Power BI, Informatica, SSRS using API Drivers for ODBC

About Google Search Console API (Google Webmaster API)

If you are new to Google Search Console API (i.e. Webmaster API) then start from this link. You can also check API Explorer here to test API requests.

Search API can be called by supplying either API key in URL  or use OAuth

Search Console V3 Covers following APIs (May add more in future)

Search Console API Call Example

Here is simple example of Search Console API call. Below example returns all sites you added under your search console account.

Request

GET https://www.googleapis.com/webmasters/v3/sites?key={YOUR_API_KEY}

Response

{
 "siteEntry": [
  {
   "siteUrl": "https://my-google-search-api-1.com/",
   "permissionLevel": "siteOwner"
  },
  {
   "siteUrl": "https://www.my-google-search-api-1.com/",
   "permissionLevel": "siteOwner"
  },
  {
   "siteUrl": "https://my-google-search-api-2.com/",
   "permissionLevel": "siteOwner"
  },
  {
   "siteUrl": "https://www.my-google-search-api-2.com/",
   "permissionLevel": "siteOwner"
  }
 ]
}

Step-By-Step – Call Search Console API in SSIS

Now lets look at very simple API call to list search console sites

1. Open SSIS Package
2. Drag and drop ZS REST API Task from SSIS tool box
   

   Drag and Drop ZS REST API Task from SSIS Toolbox

3. Now double click REST API Task and configure (Skip Step 5 if you use API Key instead of OAuth).
4. Enter API URL you like to call. For This example we will use simple URL as below
   OAuth based Authentication
   

   https://www.googleapis.com/webmasters/v3/sites/

   API Key based Authentication
   

   https://www.googleapis.com/webmasters/v3/sites/?key={YOUR_API_KEY}

5. If you want to use OAuth based credentials then perform the following steps
   1. Select URL from Connection 
   2. Click New ZS-OAUTH connection from Dropdown
   3. On the OAuth UI select Google as Provider
   4. Enter Scopes as below
      

      https://www.googleapis.com/auth/webmasters
      https://www.googleapis.com/auth/webmasters.readonly

   5. Click Generate Token. When you get Login Prompt enter your Google Account information and click Accept
   6. Click OK to Save OAuth UI
6. This is how it will look like if you use OAuth Connection
   

   Call Google Search Console API using SSIS REST API Task (OAuth Authentication)

7. Now click Test Request / Response to check if its working. You should get response like below

Read from Search Console Analytics API (query)

Now let’s look at how to call Search Console Analytics API (i.e. query) for Specified Site and extract data in tabular format using ZS JSON Source. In below example we will obtain search Impression, clicks, position etc for specified site. We will aggregate this over date and country (dimensions).

In this section you will learn how to use JSON Source Adapter to extract data from JSON file (In this case its Web URL).

1. Firstly, You need to Download and Install SSIS ZappySys PowerPack.
2. Once you finished first step, Open Visual Studio and Create New SSIS Package Project.
3. Now, Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   
4. Double click on the Data Flow task to see Data Flow designer surface.
5. From the SSIS toolbox drag and drop JSON Source on the Data Flow designer surface.
   
6. Double click JSON Source and configure like below
   1. Enter URL as below. Notice two things, first we used <<somedata,FUN_URLENC>> placeholder function to encode : and // in the URL so it become %2F %3A. We use

      --dont-escape--

        suffix at the end.
      For OAuth based Credentials
      

      https://www.googleapis.com/webmasters/v3/sites/<<https://mysite.com,FUN_URLENC>>/searchAnalytics/query--dont-escape--

      For API Key based Credentials
      

      https://www.googleapis.com/webmasters/v3/sites/<<https://mysite.com,FUN_URLENC>>/searchAnalytics/query/?key={YOUR_API_KEY}--dont-escape--

   2. Check Use Credentials and select OAuth connection (created earlier) (if you are using API key in URL then skip this step).
   3. Select POST as HTTP Method
   4. Enter Body for Query. Here are some examples for various queries you can use. For our case we want to group by date and country so use below
      

      {
      "startDate": "2019-01-01",
      "endDate": "2019-01-10",
      "dimensions" : ["date","country"]
      }

   5. Select Content Type as

      application/json

        from Content Type drop down.
   6. Select Filter or enter as 

      $.rows[*]

   7. Now go to Extract Multiple Arrays Tab and Check Array Flattening Option (on older version it was on the different Tab)
7. Click Preview and you will see your Google Search Console API data as below.
   

   Read from Google Search API Console (Analytics Query Endpoint) – JSON Source

Loading Google Search Console API data into SQL Server / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Import Google Search Console API data in Reporting / other ETL tools (ODBC Usecase)

There will be a time when you dont want to use SSIS connector like mentioned in previous sections but extract Google Search API data in Reporting Tools / Other ETL Platform.  Good news is you can use ZappySys ODBC Driver for JSON / REST API in ODBC Apps. See few popular ODBC Apps below.

• Power BI connection for Google Search Console API
• Excel connection for Google Search Console API
• Informatica connection for Google Search Console API
• MS Access connection for Google Search Console API
• C# , PowerShell, VB.net connection for Google Search Console API

Preview / Generate Query in JSON Driver / XML Driver

Load Google Search Console API in SQL Server without any ETL

Now let’s look at even more interesting integration scenario. If you have SQL Server and like to load Google Search Console API without any ETL then you can use Data Gateway

Read this article to learn how to load REST API in SQL Server without any coding (Use just T-SQL) .

You can write query like below to load data inside SQL Table.

SELECT * FROM OPENQUERY([MY_LINKED_SERVER]
, 'SELECT * FROM $
WITH(
	 Src=''https://www.googleapis.com/webmasters/v3/sites/<<https://mysite.com,FUN_URLENC>>/searchAnalytics/query/?key={YOUR_API_KEY}--dont-escape--''
	,Filter=''$.rows[*]''
	,RequestData=''{
"startDate": "2019-01-01",
"endDate": "2019-01-10",
"dimensions" : ["date","country"]
}''
	,RequestContentTypeCode=''ApplicationJson''
	,Header=''cache-control: no-cache || Accept: */*''
	,RequestMethod=''POST''
	,EnableArrayFlattening=''True''
)')

SSMS Output – Query REST API via Linked Server OPENQUERY statement (Connect to ZappySys Data Gateway)

Conclusion

So in this post we saw how easy it is to achieve total REST API integration in SSIS using JSON Connector. We also saw how to use ODBC Drivers to query REST API data inside Apps like Excel, MS Access, Informatica and other ODBC Apps using ODBC JSON / REST API driver. If you are SSIS User download SSIS PowerPack and try for FREE and if you don’t have SSIS in house and want to try more generic approach then Download ODBC PowerPack Drivers for FREE Trial.

The post Get data from Google Search Console API in SSIS and ODBC Apps appeared first on ZappySys Blog.

In our previous article we see How to read call REST API data in SSIS. Now in this article, we will see How to Make MailPlus OAuth 1.0a REST API Call in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call MailPlus Oauth 1.0a REST API Call can be useful for any developer regardless of which programming language or tool set you use. We will also see how to configure OAuth1.0 connection.

We will go through the steps to make temporary lists API call which Returns a list of all available temporary lists and we load it into MS SQL Server.

In nutshell, this post will focus on how to Make MailPlus OAuth 1.0a REST API Call in SSIS.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of MailPlus API.

What is MailPlus?

MailPlus, founded in 2004, is the number one ESP (Email Service Provider) in the Netherlands. MailPlus is an award winning solution that enables companies to send out their online communication, such as newsletters, easily and professionally.

Where to get the MailPlus API Key and Secret?

After activation of the MailPlus REST API in a MailPlus account you can get unique Key and Secret from there. The MailPlus REST API uses Oauth 1.0a (one leg, see Oauth 1.0a (One Leg) ). A consumer key and secret are required to access your MailPlus account. These can be created in your MailPlus account. The Key and Secret are always available via the startpage of a MailPlus account. Click on the tab “MailPlus Settings” and then “MailPlus Rest API”. In MailPlus eCom an authentication is already available.

Call MailPlus Oauth 1.0a REST API using SSIS

Let’s start with an example. We use SSIS JSON Source component to make the call to MailPlus API, we will read list of all available temporary lists and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to get list of all available temporary lists and create the OAuth 1.0 Connection.
   

   https://restapi.mailplus.nl/integrationservice-1.1.0/templist

   

   MailPlus Oauth 1.0a Connection

4. Finally click on Preview button to load the data.
   

   MailPlus Preview Data

5. You can find more MailPlus API Lists here on MailPlus Interactive API Documentation link.
6. That’s it we are ready to load all available temporary lists in SQL Server.

Load MailPlus REST API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to extract information from REST API such as MailPlus REST API using OAuth1.0 Connection and load into SQL Server. We also learned techniques like How to create OAuth1.0 Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • MailPlus API

The post How to Make MailPlus OAuth 1.0a REST API Call in SSIS appeared first on ZappySys Blog.

In this article, we will see how to read PayPal API data in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call PayPal APIs can be useful for any developer regardless of which programming language or tool set you use. We will also see How to send basic Authorization header along with body in OAuth2.

We will go through the steps to read data from PayPal Transactions and Load into MS SQL Server.

In nutshell, this post will focus on how to make PayPal API Call using SSIS.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of PayPal API. Click here to create API.

What is PayPal?

PayPal is an American company operating a worldwide online payments system that supports online money transfers and serves as an electronic alternative to traditional paper methods like checks and money orders. The company operates as a payment processor for online vendors, auction sites, and many other commercial users, for which it charges a fee in exchange for benefits such as one-click transactions and password memory. PayPal’s payment system, also called PayPal, is considered a type of payment rail.

Get Started with PayPal API

If you have need to automate PayPal operations (e.g. read / write / update / delete ) then you can use PayPal REST API. Before you can integrate a PayPal product or solution, you must set up your development environment to get OAuth 2.0 client ID and secret credentials for the sandbox and live environments. You exchange these credentials for an access token that authorizes your REST API calls. To test your web and mobile apps, you create sandbox accounts.

Get PayPal credentials

To generate OAuth 2.0 credentials for the sandbox and live environments:

1. Log into Dashboard and type your PayPal personal or business account email and password.
2. In the REST API apps section, click Create App. The purpose of this app is to generate your credentials.
3. Type a name for your app and click Create App. The page shows your sandbox app information, which includes your credentials.
   

   Note: To show your live app information, toggle to Live.

4. Copy and save the client ID and secret for your sandbox app.
   

   PayPal Client ID and Secret

5. Review your app details and save your app.

Read PayPal API Data using SSIS JSON Source

Let’s start with an example. We use SSIS JSON Source component to make the call to PayPal API, we will read PayPal data and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

In REST API calls, include the URL to the API service for the environment:

• Sandbox: https://api.sandbox.paypal.com
• Live: https://api.paypal.com

Create Basic OAuth Client Credentials Grant (Configure ZS-OAuth Connection)

To get API response data using SOAP / REST API call, you need to get an API Token from your token request URL. Follow the steps mentioned below to get API Token:

1. Now, Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
   

   Connection Manager Window: Create New Connection

2. Select ZS-OAuth Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
   

   SSIS Connection Mangers List: Select O-AUTH Connection Manager

3. In the SSIS ZS OAuth Connection Manager, Select Provider as Custom, select version as OAuth2 and select Grant Type as  Client Credentials Grant. Enter the credentials and access token url and test the connection.
   

   https://api.sandbox.paypal.com/v1/oauth2/token

    
   

   ZS OAuth Connection – Client Credentials Grant

4. Click on OK button to save OAuth Connection configure settings.

That’s it we have successfully configured Connection for PayPal API in SSIS. In the next section, we will see how to use this connection and read various data from PayPal API.

Read PayPal API Data in SSIS

Once we have done creating HTTP Connection Manager we can move forward to read PayPal API data inside Data Flow. So lets Configure SSIS JSON / REST API Source.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to get invoices and select that created OAuth Connection.
   

   https://api.sandbox.paypal.com/v1/invoicing/invoices?page=0&page_size=100&total_count_required=true

   

   SSIS JSON Source

4. Now Go to Pagination tab select mode as Url Parameter and add indicator as

   page

     and Add the increment value same as

   page_size

     value we set in url, there for in our case we set 100. For pagination call please click here for reference to this link.
   

   PayPal Pagination

5. Select the desire Array Filter and click on preview.
   

   PayPal Preview

6. That’s it, you are ready to load PayPal API Data into SQL Server.

Load PayPal API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to extract information from REST API such as PayPal REST API using HTTP Connection and load into SQL Server. We also learned techniques like How to get Dynamic Token using HTTP Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File), ZS Logging Task
• PayPal API
• Dynamic token generation

The post How to read PayPal API data in SSIS appeared first on ZappySys Blog.