In our previous post we saw how to read API data in SSIS. This post we will cover how to call NetSuite RESTlet to read / write data in SSIS or ODBC Apps. For demo purpose we will use SSIS PowerPack REST API Task but concepts and many UI elements remain same for ODBC JSON Driver too. This article assumes you know how to write Netsuite RESTlet and you have obtained Credentials to call RESTlet from extranal app.

What is NetSuite RESTlets

NetSuite added a really nice feature to expose business data and workflows as API by writing Javascript functions which they call it RESTlets. You can secure and customize these restlets as per your need.

You can automate CRUD (Create /Read / Write / Delete) operations in NetSuite by writing special functions in JavaScript and expose those functions as API URL which can be called by any external Systems (i.e. SSIS Packages , ODBC Apps like Power BI, Informatica, SSRS).

RESTlet can be secured few ways such as OAuth 1.0 and OAuth 2.0 authentication. We will cover both approach briefly in this article. You can choose any one as per your need

Creating NetSuite RESTlet / Obtain OAuth Credentials (Client ID and Secret)

Below video shows how to obtain necessary credentials to call Netsuites API (RESTlet).

At mark 4:30 seconds it will show you How to use OAuth 2.0 for calling RESTlet

Create NetSuite RESTlet Connection

There are few ways to authenticate NetSuite RESTlet. If you are using ZappySys Products then it makes it super easy to use either approach. Lets look at both approaches.

Using OAuth 2.0 (Recommended)

As per above video if you obtained Client ID and Secret for OAuth 2.0 you can follow below steps to configure OAuth connection to call RESTlet.

What you need to use OAuth 2.0.

• Your Account ID (it goes in URL)
• Your Client ID and Secret (Obtained from Netsuite Portal) – See above video Mark 04:30
• Register Redirect URL in NetSuite portal (we recommend you use https://zappysys.com/oauth as call back Url for Automatic Code extraction)

Steps to configure OAuth Connection

1. Right click on Connection Panel and choose Choose “New Connection..”. Select ZS-OAUTH type
   
   
   

   Create new SSIS OAuth API Connection Manager

    

    

2.  Enter following things
   ————————-
   On General Tab
   ————————-
   ClientID and Secret: <get it from NetSuite>
   Authorization URL: https://<accountID>.app.netsuite.com/app/login/oauth2/authorize.nl
   Token URL: https://<accountID>.suitetalk.api.netsuite.com/services/rest/auth/oauth2/v1/token
   Scope: restlet
   

   Configure NetSuite OAuth 2.0 Connection for API call

    

   ————————-
   On Advanced Tab
   ————————-
   Callback / Redirect URL:  https://zappysys.com/oauth  (this must be entered same way in NetSuite Portal)

   

   OAuth2 Connection – Callback / Redirect URL

3. Now go back to General Tab and click Generate Token button
4. Follow login screen and try to finish the flow. At the end it will generate Tokens and populate on UI
5. Click OK to save

Using OAuth 1.0

Another approach is use OAuth 1.0 (Older version). If you have enabled OAuth 1.0 in NetSuite then follow below steps (Create connection is same as before).

1. Setup General Tab and Advanced Tab as below.
   On ==== General Tab====:
   Enter ClientID, ClientSecret, AccessToken, AccessToken Secret
   On ==== Advanced Tab ====:
   Check Do Not Include Empty Key/Value Pairs
   Select Signature Method as HMAC-SHA256
   For Extra Token Attribute enter realm=<your-account-number>_SB1
2. Click OK to Save

Call NetSuite RESTlet in SSIS

1. Create a new SSIS Package
2. Drag REST API Task from SSIS Toolbox
3. Double click the Task to configure
4. Select URL from Connection
5. From Connection dropdown select OAuth connection we created in the earlier section (either OAuth 1 or OAuth 2)
6. Enter the URL as below. Assuming you copied URL from NetSuite Replace script ID and account number
   

   https://<account-number>-sb1.restlets.api.netsuite.com/app/site/hosting/restlet.nl?script=<scriptid>&deploy=1

   Enter <account-number>
   Enter <scriptid>
7. Change Method to POST if your script supports Parameters (Else you can supply id=xxxx and recordtype=xxxxx in the URL for GET request)
8. If its POST request then chnage ContentType to ApplicationJSON (application/json)
9. Enter your Input in the body. In below example our RESTlet takes sql query as input so we enter just one parameter named query but you can have different parameter(s)
   

   Syntax:  { param1_name : <value> , param2_name : <value> ....  }

10. Now click Test see everything works
    

    Calling NetSuite RESTlet in SSIS (Using REST API Task)

Conclusion

We saw how easy it is to call NetSuite RESTlets or other NetSuite API in SSIS / other BI Apps using ODBC Drivers provided by ZappySys. Download SSIS PowerPack and try your self see how easy it is to consume virtually any API in SSIS.

The post How to call NetSuite RESTlet API in SSIS / ODBC appeared first on ZappySys Blog.

In our previous article we saw how to call PayPal API in SSIS. Now in this article, we will see How to Make PayPal API Call in ODBC from the SQL Server. This blog mainly focuses on ODBC approach but steps mentioned to call PayPal APIs can be useful for any developer regardless of which programming language or tool set you use. We will also see How to send basic Authorization header along with body in OAuth2.

We will go through the steps to Create PayPal Order from MS SQL Server.

In nutshell, this post will focus on how to make PayPal API Call using ODBC Driver.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. A first requirement, make sure that Microsoft Access installed
2. Make sure to have ZappySys ODBC PowerPack installed.
3. Credentials of PayPal API. Click here to create API.

What is PayPal?

PayPal is an American company operating a worldwide online payments system that supports online money transfers and serves as an electronic alternative to traditional paper methods like checks and money orders. The company operates as a payment processor for online vendors, auction sites, and many other commercial users, for which it charges a fee in exchange for benefits such as one-click transactions and password memory. PayPal’s payment system, also called PayPal, is considered a type of payment rail.

Get Started with PayPal API

If you have need to automate PayPal operations (e.g. read / write / update / delete ) then you can use PayPal REST API. Before you can integrate a PayPal product or solution, you must set up your development environment to get OAuth 2.0 client ID and secret credentials for the sandbox and live environments. You exchange these credentials for an access token that authorizes your REST API calls. To test your web and mobile apps, you create sandbox accounts.

Get PayPal credentials

To generate OAuth 2.0 credentials for the sandbox and live environments:

1. Log into Dashboard and type your PayPal personal or business account email and password.
2. In the REST API apps section, click Create App. The purpose of this app is to generate your credentials.
3. Type a name for your app and click Create App. The page shows your sandbox app information, which includes your credentials.
   

   Note: To show your live app information, toggle to Live.

4. Copy and save the client ID and secret for your sandbox app.
   

   PayPal Client ID and Secret

5. Review your app details and save your app.

Call PayPal Create Order API in SQL Server (T-SQL) using Data Gateway

Let’s start with an example. We use JSON Driver to make the call to PayPal API, we will Create the PayPal Order from SQL Server.

In REST API calls, include the URL to the API service for the environment:

• Sandbox: https://api.sandbox.paypal.com
• Live: https://api.paypal.com

1. To do this, first of all, we will open the ZappySys Data Gateway Configuration:
   

   Open ZappySys Data Gateway

2. Add the Native – ZappySys JSON Driver Data source.
   

   Add Gateway Data Source (Native JSON Driver)

3. Now it’s time to connect with PayPal. Let’s use JSON Driver Data source with the following URL:
   

   https://api.sandbox.paypal.com/v2/checkout/orders

   Body:
   

   {
     "intent": "CAPTURE",
     "purchase_units": [
       {
         "amount": {
           "currency_code": "USD",
           "value": "100.00"
         }
       }
     ]
   }

   

   PayPal : Create Order

4. Create Basic OAuth Client Credentials Grant (Configure ZS-OAuth Connection).
   In the SSIS ZS OAuth Connection Manager, Select Provider as Custom, select version as OAuth2 and select Grant Type as  Client Credentials Grant. Enter the credentials and access token url and click OK.
   

   https://api.sandbox.paypal.com/v1/oauth2/token

   

   ZS OAuth Connection – Client Credentials Grant

5. Furthermore, Now click on Test Connection button to test PayPal Connection.
   

   PayPal : Test Connection

6. Write your Query and Preview Data.
7. (Optional Step) If your Query use POST method (e.g. Create Invoice) to submit data then you have Save Metadata (Read More). We will use this meta file in later step when we call POST request.
   

   Generate Metadata File for REST API POST Operation using ZappySys API Driver (JSON / XML)

8. Now using Code Generator we will generate the query.
   

   Generate Query

Setup Linked Server in MS SQL Server for PayPal

Once you configured the data source in Gateway, we can now setup Linked Server in SQL Server to query API Calls.

1. Assuming you have installed SQL Server and SSMS. If not then get both for FREE from here: Get SQL Server Express and  Get SSMS
2. Open SSMS and connect to SQL Server.
3. Go to Root > Server Objects > Linked Servers node. Right click and click New Linked Server…

   

   Add Linked Server in SQL Server

4.  Now enter the linked server name, select Provider as SQL Native Client
5. Enter data source as 

   GatewayServerName, PORT_NUMBER

    where server name is where ZappySys Gateway is running (Can be same as SQL Server machine or remote machine). Default PORT_NUMBER is 5000 but confirm on Data gateway > General tab in case its different.
6. Enter Catalog Name. This must match name from Data gateway Data sources grid > Name column

   

   Configure Linked Server Provider, Catalog, Server, Port for ZappySys Data Gateway Connection

7. Click on Security Tab and select last option “Be made using this security context“. Enter your gateway user account here.
8. Click OK to save Linked Server

Create Order in PayPal from SQL Server (Call POST API Method)

1. To Create Order in PayPal we need to execute the query like this. Please refer to this link for more information: Create Order example.
   
   Important Note*: Below query uses META property (fully explained here). This is needed if you creating anew record using POST, PUT. If you don’t pass META='{xxxx}’ in query driver calls REST API two times (first time to parse metadata and second time to get actual data). So that you find some odd thing in PayPal if you execute the query without Meta then you will find two same Orders created in your PayPal as the driver makes two calls as mentioned above.

   -- In below query rather than WITH(META=''[{"Name": "Success","Type": "Int32"}]'' ....
   -- You can also use WITH(META=''c:\my-meta-file.txt'' .... We created this file in previous section
   
   SELECT * FROM OPENQUERY([MY_LINKED_SERVER_NAME]
   , 'SELECT * FROM $
   WITH(META=''[{"Name": "Success","Type": "Int32"}]''
   	 ,Src=''https://api.sandbox.paypal.com/v2/checkout/orders''
   	,RequestData=''{
     "intent": "CAPTURE",
     "purchase_units": [
       {
         "amount": {
           "currency_code": "USD",
           "value": "100.00"
         }
       }
     ]
   }''
   	,RequestContentTypeCode=''ApplicationJson''
   	,RequestMethod=''POST''
   )')

2. Here is the preview after you run REST API query in SQL Server. It will return created Order Id, Links and status. Notice that you can override default configuration by supplying many parameters in WITH clause.
   

   Create Order In PayPal Using SQL Query

3. You can also find this created Order by opening that URL in the browser.
   

   Created Order In PayPal

PayPal API integration in ODBC apps (e.g. Power BI, Excel, Informatica, Access, C#…)

So far we have looked at SQL Server integration only, but what if you like to use same powerful ZappySys API drivers part of ODBC PowerPack in other Apps like Power BI, Excel, MS Access, Informatica and many more… ?? Well not to worry if you know that App supports ODBC you can use ZappySys ODBC Driver same way described earlier… only difference is you dont need Linked Server and you dont have to use OPENQUERY statement when you use ODBC Drivers.

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

So in this blog, we learned how to generate a PayPal API Token, Load PayPal data in MS SQL Server table using ODBC JSON / REST API Driver in a very simple way. You can achieve many more functionalities with this tool. Check our blogs/articles on JSON / REST API Driver to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the use of PayPal API with our tools:

• Landing Page for ODBC JSON / REST API Driver, you can also find Tutorial Video here.
• Help File: Documentation of JSON Driver.
• PayPal API

The post How to Make PayPal API Call in ODBC / SQL Server (T-SQL) appeared first on ZappySys Blog.

In this article, we will see how to read PayPal API data in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call PayPal APIs can be useful for any developer regardless of which programming language or tool set you use. We will also see How to send basic Authorization header along with body in OAuth2.

We will go through the steps to read data from PayPal Transactions and Load into MS SQL Server.

In nutshell, this post will focus on how to make PayPal API Call using SSIS.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of PayPal API. Click here to create API.

What is PayPal?

PayPal is an American company operating a worldwide online payments system that supports online money transfers and serves as an electronic alternative to traditional paper methods like checks and money orders. The company operates as a payment processor for online vendors, auction sites, and many other commercial users, for which it charges a fee in exchange for benefits such as one-click transactions and password memory. PayPal’s payment system, also called PayPal, is considered a type of payment rail.

Get Started with PayPal API

If you have need to automate PayPal operations (e.g. read / write / update / delete ) then you can use PayPal REST API. Before you can integrate a PayPal product or solution, you must set up your development environment to get OAuth 2.0 client ID and secret credentials for the sandbox and live environments. You exchange these credentials for an access token that authorizes your REST API calls. To test your web and mobile apps, you create sandbox accounts.

Get PayPal credentials

To generate OAuth 2.0 credentials for the sandbox and live environments:

1. Log into Dashboard and type your PayPal personal or business account email and password.
2. In the REST API apps section, click Create App. The purpose of this app is to generate your credentials.
3. Type a name for your app and click Create App. The page shows your sandbox app information, which includes your credentials.
   

   Note: To show your live app information, toggle to Live.

4. Copy and save the client ID and secret for your sandbox app.
   

   PayPal Client ID and Secret

5. Review your app details and save your app.

Read PayPal API Data using SSIS JSON Source

Let’s start with an example. We use SSIS JSON Source component to make the call to PayPal API, we will read PayPal data and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

In REST API calls, include the URL to the API service for the environment:

• Sandbox: https://api.sandbox.paypal.com
• Live: https://api.paypal.com

Create Basic OAuth Client Credentials Grant (Configure ZS-OAuth Connection)

To get API response data using SOAP / REST API call, you need to get an API Token from your token request URL. Follow the steps mentioned below to get API Token:

1. Now, Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
   

   Connection Manager Window: Create New Connection

2. Select ZS-OAuth Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
   

   SSIS Connection Mangers List: Select O-AUTH Connection Manager

3. In the SSIS ZS OAuth Connection Manager, Select Provider as Custom, select version as OAuth2 and select Grant Type as  Client Credentials Grant. Enter the credentials and access token url and test the connection.
   

   https://api.sandbox.paypal.com/v1/oauth2/token

    
   

   ZS OAuth Connection – Client Credentials Grant

4. Click on OK button to save OAuth Connection configure settings.

That’s it we have successfully configured Connection for PayPal API in SSIS. In the next section, we will see how to use this connection and read various data from PayPal API.

Read PayPal API Data in SSIS

Once we have done creating HTTP Connection Manager we can move forward to read PayPal API data inside Data Flow. So lets Configure SSIS JSON / REST API Source.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to get invoices and select that created OAuth Connection.
   

   https://api.sandbox.paypal.com/v1/invoicing/invoices?page=0&page_size=100&total_count_required=true

   

   SSIS JSON Source

4. Now Go to Pagination tab select mode as Url Parameter and add indicator as

   page

     and Add the increment value same as

   page_size

     value we set in url, there for in our case we set 100. For pagination call please click here for reference to this link.
   

   PayPal Pagination

5. Select the desire Array Filter and click on preview.
   

   PayPal Preview

6. That’s it, you are ready to load PayPal API Data into SQL Server.

Load PayPal API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to extract information from REST API such as PayPal REST API using HTTP Connection and load into SQL Server. We also learned techniques like How to get Dynamic Token using HTTP Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File), ZS Logging Task
• PayPal API
• Dynamic token generation

The post How to read PayPal API data in SSIS appeared first on ZappySys Blog.

In our previous article, we talked about how to read from Dynamics CRM using easy to use SSIS Dynamics CRM Source connector. In this post, you will learn how to use SSIS JSON Source (for REST API) instead of CRM Source. Using SSIS JSON Source you can read virtually any REST API or JSON file data including Dynamics CRM / Dataverse. For this article, our goal is to learn How to read Microsoft Dynamics 365 data using Web API (OData API) in SSIS. We will use drag and drop approach without any coding.

Why use Web API (OAuth) for Dynamics 365 CRM?

Microsoft has announced depreciation of older CRM API (SOAP API with WS-Trust Authentication) which means they are no longer going to support the older style of authentication so no option but to switch to newer authentication / API.

In the older approach (SOAP API with WS-Trust Authentication) you must pass your User Name and Password to authenticate, also if MFA / 2FA (Multi-Factored Auth) is turned on by organization policy you cannot use your user id / password in API call. In new authentication (i.e. OAuth) you have to use Client Id and Client Secret only.

Now let’s look at the step by step approach to call Dynamics 365 REST API.

Step-1 : Register Dynamics CRM / Dataverse App obtain App Id / Client Secret

The very first step to access Microsoft Dynamics CRM 365 API is to obtain Application Id (also referred as Client Id)  and Client Secret. We will use this information later in this article (on OAuth connection UI).

1. Register Dynamics CRM / Dataverse App With Azure and get the credential (App Id, Client Secret).

Step-2 : Read Microsoft Dynamics 365 API data in SSIS using JSON Source

Let’s start with an example. We use SSIS JSON Source Task to make calls to Microsoft Dynamics 365 API, we will load Dynamics 365 Leads data to SQL Server.

Please refer to this link for more information: Microsoft Dynamics 365 Web API Reference

Find out API Url for your environment (organization url)

Once we are done with the creation of the Azure AD app to obtain App Id /Secret out next step is to find out the base API URL for your organization / environment.

If you have multiple environments (i.e. Prod, Sandbox) then each environment might have its own unique URL)

1. To find your API Service URL, Click on Gear Icon
2. Choose Advanced Settings
3. Click down arrow on Settings
4. Click Customization
5. Click Developer Resource Link
6. On this page now you can see various URLs. For examples used in this article, we need to get is Service Root URL for Web API

How to Find Organization Id / Web API URL / SOAP API Service URL (Unique Environment Instance URL)

Create SSIS Data Flow to read data from Dynamics CRM / Dataverse

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Furthermore, Drag and Drop JSON Source(REST API and File) File Task from the SSIS Toolbox.
   

   Drag and Drop JSON Source Component

Create OAuth Connection for Dynamics CRM / Dataverse

Now lets look at how to create and configure OAuth connection to call Web API for Dynamics CRM / Dataverse

1. Check Use Credentials Option and select a new ZS-OAuth Connection from the drop-down.
   

   Check Use Credentials

2. Once OAuth Connection UI opens up. Configure like below on General Tab.
   Change OAuth Version to OAuth2
   Change Grant Type to Client Credentials
   Enter Client Id (Azure AD App Id) (Obtained in the Previous Section)
   Enter Client Secret (Obtained in the Previous Section)
   Enter Token URL (see below)
   

   Configure OAuth Connection for Azure AD App – Client Credentials Grant (Tenant Specific Token EndPoints)

   For Client Credentials Grant
   Enter Token URL as https://login.microsoftonline.com/<your-tenant-id>/oauth2/token
   (You can find Tenant ID on the Overview page of your Azure AD App)

   For Default Grant (3-legged – requires User Login Prompt)
   Enter Auth URL as https://login.microsoftonline.com/common/oauth2/authorize
   Enter Token  URL as https://login.microsoftonline.com/common/oauth2/token

    

   

   Where to find Token Endpoint with Tenant ID in the URL (Azure AD App)

    

3. Now go to Advanced Tab and configure like belowEnter Extra Attribute for /auth as  

   resource=https://<your-instance-id>.api.<your-crm-region>.dynamics.com

   

   Client Credentials Grant for Dynamics CRM / Dataverse – API Access (Azure AD App / OAuth)

4. Click OK to Save connection.

Get all Accounts, Contacts, and Leads using Dynamics 365 API

Now let’s configure API URL from where you like to read data.

For example, if you just use the root URL then it will list all possible entries

Root URL:

https://{{organization-uri}}/api/data/v9.1/

1. Double click on the Component to Edit the Configurations.
2. Change Data Format to OData.
3. In the Web URL Field, you can enter any valid URL for Web API. Please refer to this link of Dynamics 365 Rest API for more information.Example Data URLs

   https://{{organization-uri}}/api/data/v9.1/
   https://{{organization-uri}}/api/data/v9.1/accounts
   https://{{organization-uri}}/api/data/v9.1/contacts
   https://{{organization-uri}}/api/data/v9.1/leads

   Here, replace {{organization-uri}} in URL to your Microsoft Dynamics 365 site domain name (Example zappy6.api.crm.dynamics.com Dynamics 365 URL).

   

   Set URL in JSON Source

Preview / Query Dynamics 365 Data

1. Now select Array filter as

   $.value[*]

     (Click Select Filter Button – Choose value node)
2. Click on the Preview button to get the data.
   

   Preview Data

3. Finally, click OK to save JSON Source settings.
4. That’s it we are ready to load Dynamics 365 API data to SQL Server.

Load Microsoft Dynamics 365 API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

In this post you have seen how to Read Microsoft Dynamics 365 API data in SSIS. We used JSON Source Component to call Dynamics 365 REST API and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information about the use of Dynamics 365 REST API with our tools:

• JSON Source, you can also find Tutorial Video here.
• Help File: Documentation of SSIS Json Source
• Microsoft Dynamics CRM Web API Reference
• Dynamics 365 Web API EntityType Reference

The post How to read Microsoft Dynamics 365 OData API data in SSIS (CRM Web API) appeared first on ZappySys Blog.

QuickBooks Online is a well-known Cloud-based Accounting Software. In this post, you will learn how to implement QuickBooks Online API Integration with SQL Server or any other RDBMS (e.g. Oracle, MySQL, Postgresql) using ODBC in few clicks. We will use ODBC JSON / REST API Driver to read data from QuickBooks Online and Load into SQL Server / other targets (Using OAuth Connection). We will discuss on How to Create an Intuit Developer Account, How to Create QuickBooks Online App for OAuth.

In nutshell, this post will focus on how to call QuickBooks Online API using ODBC.
So let’s get started.

Prerequisites

Before we look into Step-By-Step section to extract and read data from QuickBooks Online to SQL Server let’s make sure you met the following requirements.

1. A first requirement will be to SQL Server Database Engine Installed
2. The second requirement will be SSMS installed
3. Finally, make sure to have ZappySys ODBC PowerPack installed.

Create ODBC – Native Zappysys JSON Driver Data Source

1. To do this, first of all, we will open the ZappySys Data Gateway Configuration:
   

   Open ZappySys Data Gateway

2. Add the Native – ZappySys JSON Driver Data source.
   

   Add Gateway Data Source (Native JSON Driver)

3. Now it’s time to connect with QuickBooks. Let’s use JSON Driver Data source with the following URL:
   

   https://{RequestURL}/v3/company/{realmID}/query?query={selectStatement}

   here, We have used 3 placeholders.
   * {RequestURL}is either a SandBox base URL or Production base URL.
   

   Sandbox Base URL: https://sandbox-quickbooks.api.intuit.com
   Production Base URL: https://quickbooks.api.intuit.com

   * {realmID} is your QuickBooks Online sandbox/production company ID.
   * {selectStatement} can be any query something like Select * From Invoice (*Note: select statement must be URL encoded first)
   For more information on Query, an Invoice Request API see this
   I am using my sandbox URL like this:
   

   https://sandbox-quickbooks.api.intuit.com/v3/company/193514799372394/query?query=SELECT * FROM Invoice STARTPOSITION <%startposition%> MAXRESULTS 1000

   

   Setup QuickBooks ODBC-based data source configuration

4. Configure the new OAuth Connection for QuickBooks Online as below.
   Basically, you need to configure the configurations of App on the General tab (i.e. OAuth Provider, Client ID, Client Secret, Scopes, etc.)
   

   OAuth Connection Manager: General Tab Configurations

5. Most importantly, you also need to Enter your Callback URL in Advanced Tab.
   

   OAuth Connection Manager: Advanced Tab Configurations

6. Similarly, you need to check “Do not include credentials in header”
   

   OAuth Connection Manager: OAuth2 Grant Options Tab Configurations

7. Click on Generate Token button to Generate token and then click on Test Connection button and click on OK to save the connection manager configurations settings.
   

   Generate Token and Test Connection

8. Let’s continue on, and enable pagination by configuring it like in the image below in the Pagination tab. In the Page Num Indicator field input this value: <%startposition%> Then set Increment By value to a maximum number of results possible: 1000
   
   

   Configuring pagination to get QuickBooks all rows in ODBC JSON-based driver data source

9. Finally, now using Code Generator we will generate the query.
   

   Generate Query

Load QuickBooks Invoices data in MS SQL Server

Conclusion

So in this blog, we learned how to generate a QuickBooks API Token, Load QuickBooks Invoices data in MS SQL Server table using ODBC JSON / REST API Driver in a very simple way. You can achieve many more functionalities with this tool. Check our blogs/articles on JSON / REST API Driver https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/json-rest-api-driver/ to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the use of QuickBooks API with our tools:

• QuickBooks Documentation for API
• Landing Page for ODBC JSON / REST API Driver, you can also find Tutorial Video here.
• Help File: Documentation of JSON Driver.
• Blog/articles link: https://zappysys.com/blog/category/odbc-powerpack/odbc-drivers/json-rest-api-driver/

The post Read data from QuickBooks Online into SQL Server via ODBC Driver appeared first on ZappySys Blog.

Splunk is commonly used for searching, monitoring, and analyzing machine-generated big data, via a Web-style interface. In this post, you will learn how to implement Splunk API Integration with SQL Server or any other RDBMS (e.g. Oracle, MySQL, Postgresql) using SSIS in a few clicks. We will use SSIS XML Source to Read data from Splunk and Load into SQL Server / other targets (Using HTTP Connection).

We will discuss on How to Create an Intuit Developer Account, How to Create QuickBooks Online App for OAuth, We will also discuss reverse scenario to Write data to Splunk (API POST for Insert or Update in Splunk) using SSIS REST API Task

In nutshell, this post will focus on how to call Splunk API using SSIS.
So let’s get started.

What is Splunk

Splunk is a software platform to search, analyze and visualize the machine-generated data gathered from the websites, applications, sensors, devices etc. which make up your IT infrastructure and business.

The other benefits of implementing Splunk are:

• Input data can be in any format for e.g. .csv, or JSON or other formats
• Give Alerts / Events notification at the onset of a machine state
• Accurately predict the resources needed for scaling up the infrastructure
• Create knowledge objects for Operational Intelligence

Why Use Splunk REST API

So there will be a time when you want to automate certain things without using Splunk Web Portal. Here are some example scenarios which can be solved using the Splunk REST API.

• Reading/Update Splunk configuration files
• Creating Splunk searches using the REST API
• Create a new Splunk object for a specific context
• Edit a Splunk object

Step-By-Step – Import Splunk data into SQL Server

In order to start, we will show several examples. ZappySys includes an SSIS XML Source that will help you to call Splunk API, read search job results data from Splunk with SSIS, create a new Saved Search on Splunk with SSIS and do many more actions using REST API. To learn more about Splunk REST API check this help file.

To get Splunk data using the REST API call, you need to have a Free or Licensed Splunk Account. You can Download Splunk Enterprise or Register with Splunk see this link. You can find the videos for How to Install and Getting Data in on this link: Splunk Enterprise Videos.

Create / Configure Splunk Dataset

So first step in our demo would be make sure we have at least one Splunk Dataset which we can query using REST API. If you have already configured Dataset then skip this section.

1. First of all, let’s start the Splunk on your machine from the program menu.
   

   Start Splunk

2. Hence, Splunk opens into the browser. So let’s logged in and let’s Add Data for WinEvents to be searched in our next phase.
   

   Add Data

3. To start adding data first, choose a data source. For example, you can monitor WinEvents.
   

   Select Data Source

4. Select Search & Reporting in App Context and click next for review and click next to finish.
   

   Select Search & Reporting

5. That’s it local event logs input has been created successfully for searching.
   

   Start searching for local events

6. Therefore you will redirect to search page.
   

   Splunk search page

7. So, let’s import WinEvents search data into SQL Server in the next few sections.

Connect to Splunk in SSIS / Create Search JOB / Obtain JobID

Now once you have Splunk Dataset configured, next step is Call Splunk API. So most common thing you will do with splunk API is to call splunk search API. In splunk Search API is Job Style API means you perform following steps

1. First call /services/search/jobs/ endpoint with Search query (in POST) to create search Job. It returns you Job ID (also known as SID)
2. Wait until Job is done (This part may be tricky) … SO you may have to add timer task after first step and in the 3rd step enable Retry option on HTTP connection just incase it didnt finish within supplied delay.
3. Once Job finished you can read by calling below API endpoint.
   /services/search/jobs/{{your Job Id}}/results?output_mode=json

Thats it now let’s look at actual steps

1. Now let’s make the first call to search job using a POST method. If you are using a custom time range, pass it in with the POST request. Refer this link for more information: Export data using the Splunk REST API. If your URL is different than below HTTP connection then Check Use direct URL option on REST API Task and enter custom URL.
   

   URL:
   https://localhost:8089/services/search/jobs/
   
   Post Reuest Data:
   search=search source="WinEventLog:*" host="SCIFI08" earliest=-1d&output_mode=json

   

   Splunk POST Search request

2. So let’s get Response SID(Search job ID) into the variable to be used in the next call to get results.
   

   Store Splunk Response Search Job ID in a variable

3. Now add the Data Flow Task to retrieve Splunk search result data using JSON Source (REST API or File) in it. In below URL we are specifying count=5 but in real world change it to 100 or higher.
   

   TO Get Metadata deatils:
   https://localhost:8089/services/search/jobs/1549701697.1424/results?output_mode=json&offset=0&count=5

   

   Splunk Search JSON Results Data

4. Furthermore, Let’s set pagination to get all the results data.
   NOTE: In below screenshot we are specifying count=5 and Increment=5 just to show demo but in real world set both items to higher value (e.g. 100) so you request more records per page.
   

   Set Pagination

5. Let’s preview the results data.
   

   Preview Splunk Search Data

6. Edit JSON Source again and replace the below URL with it to get results By SID(Search Job ID) variable. And create the new HTTP connection as if the search job(SID) is newly created, it is possible that we will #get 204s(No Content) until the job is ready to respond.
   

   https://localhost:8089/services/search/jobs/{{User::v_sid}}/results?output_mode=json&offset=0&count=5

   

   204 No Content response error handling

7. Furthermore, click on OK button and make sure we need to click on NO.
   

   Splunk URL With variable

8. Finally, we are ready to Load Splunk data into the SQL Server.

Load Splunk data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

Above all, in this blog, we learned how to read Splunk data in SSIS. Furthermore we used XML Source Component and SSIS REST API Task to call Splunk REST API and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information about the use of Splunk Online REST API with our tools:

• REST API Task, you can also find Tutorial Video here.
• XML Source Component, you can also find Tutorial Video here.
• Help File: Documentation of REST API Task and XML Source Component.

The post How to read data from Splunk in SSIS appeared first on ZappySys Blog.

QuickBooks Online is a well-known Cloud-based Accounting Software. In this post, you will learn how to implement QuickBooks Online API Integration with SQL Server or any other RDBMS (e.g. Oracle, MySQL, Postgresql) using SSIS in few clicks. We will use SSIS XML Source to Read data from QuickBooks Online and Load into SQL Server / other targets (Using OAuth Connection). We will discuss on How to Create an Intuit Developer Account, How to Create QuickBooks Online App for OAuth, We will also discuss reverse scenario to Write data to QuickBooks Online (API POST for Insert or Update in QuickBooks Online) using SSIS REST API Task
In nutshell, this post will focus on how to call QuickBooks Online API using SSIS.
So let’s get started.

Prerequisites

Before we look into Step-By-Step section to extract and load data from QuickBooks Online to SQL Server let’s make sure you met the following requirements.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have the basic familiarity with REST API concepts and QuickBooks Online API.

What is QuickBooks Online

QuickBooks is an accounting software package developed and marketed by Intuit. QuickBooks products are geared mainly toward small and medium-sized businesses and offer on-premises accounting applications as well as cloud-based versions that accept business payments, manage and pay bills, and payroll functions.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS XML Source that will help you to call QuickBooks Online API, read various data from QuickBooks Online with SSIS, create a new Contact on QuickBooks Online with SSIS and do many more actions using REST API. To learn more about QuickBooks Online API check this help file.

To access QuickBooks Online data using REST API call, you need to create an Intuit Developer Account and then create QuickBooks Online App.

Create an Intuit Developer Account

Follow the mentioned steps below to create an Intuit developer account:

1. you can visit this site developer.intuit.com, and click on Sign up from the menu bar.
2. A registration form will be opened in a Popup. you can fill up and then click on Create Account.
   

   Signup Form: Intuit Developer Account

3. Complete the validation check and click Continue.
4. On the confirmation dialog, provide a few details to fill out your profile and click All set.

The post How to read data from QuickBooks Online in SSIS appeared first on ZappySys Blog.

Tasks / Components used in this article.

	REST API Web Service Task
	JSON Source (REST API Source)

Prerequisites

Before we look into Step-By-Step section to extract and load data from Zoom to SQL Server let’s make sure you met the following requirements.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have the basic familiarity with REST API concepts and Zoom API.

Zoom API Authentication

Now let’s look at step by step how to read data from Zoom API or POST data to Zoom using SSIS.

Zoom Provides 3 ways to authenticate API calls. In this article we will cover only JWT and OAuth 2.0 connection approach.

1. Use JWT Token
2. Use OAuth 2.0 Token (3-legged OAuth)
3. Use API Key / Secret directly in API call (Mostly used by older API like this one)

Method-1 : Call Zoom API in SSIS using JWT Token (RFC 7519)

In this section we will learn how to use JWT Token method to call Zoom API (i.e. JWT as per RFC 7519 and here ).

JWT method is simple compared to OAuth 2.0 (next section) so many people prefer it compared to OAuth.

Now let’s look at how to use JWT token approach to authenticate your Zoom API calls.

Obtain Zoom API Key / Secret for JWT Auth

1. First Visit your Zoom Developer account here
2. Click on API Tab and copy your API Key and Secret (We will use it later)
   

   Obtain Zoom API Key and Secret (Used to generate JWT Token)

3. Whitelist the Redirect URL
   

   Whitelist the Redirect URL

Once we have API key / Secret we can move on to SSIS part.

Configure SSIS HTTP Connection with JWT

Now lets get started with SSIS part. We will first create HTTP Connection for Zoom API access using JWT method.

1. Open SSIS Package and drag ZS REST API Task from SSIS Control Flow Toolbox
2. Change URL Access Mode to Url From Connection
3. Enter URL as https://api.zoom.us/v2/users/me/settings
4. Select New ZS-HTTP connection type from the connection dropdown
5. Configure HTTP Connection as below. Select Credentials Type as JWT Token. Enter API Key and Secret.
   

   Configure HTTP Connection Manager (for API using JWT Token Authentication)

6. Once HTTP connection is configured you can click OK to save connection.
7. On the REST API Task click Test to confirm see its working.
   

   Calling Zoom REST API using SSIS REST API Task

Debug Web API call using Fiddler

To test how things looking behind the scene we strongly suggest to use tool like  fiddler. You can double click the URL entry (Right side) to see Request and Response Panels. Top panel is Request (URL, Headers, Body) and Bottom Panel is Response.

Debugging Web API call using Fiddler in SSIS

Loading data from REST API to SQL Server

Once you setup HTTP / OAUTH connection we can use JSON Source or XML Source to extract data from API and  load into SQL Server. If you are API is XML based API then use XML Source in below examples else use JSON Source. For example purpose we will load data into SQL Server but you can load into any Target (e.g. Flat file, Oracle, Excel) using Microsoft or ZappySys Destination connectors To load REST API data into SQL Server perform the following steps (Screenshots are generic and used for demonstration for purpose, your values can be different than below)

1. Go to SSIS Data Flow tab
2. Drag ZS JSON Source from the SSIS Toolbox (Use ZS XML Source if your API is XML based API)
3. Double click to edit Source
4. Set API URL you like to call, Check Use Credentials if you need to use HTTP or OAuth connection
5. Set Method / Body if its other than GET. For method other than GET make sure you select correct Content-Type from Dropdown (e.g. application/json). This indicate input data format in Body.
6. Set HTTP Headers if needed (in Most cases you won't need to set this)
7. Select Filter (many times this is not needed). If your API response has array node then select it here. For example if your response looks like this { rows: [ {...}, {...}, {...} ] } then filter would be $.rows[*]. Select Array node you like to extract in the Filter selection UI.
8. Configure Pagination if needed on the pagination tab. Check your API documentation see it supports pagination.
9. Click OK to Save component settings and generate desired metadata / columns for the output.
10. Now drag OLEDB destination from the SSIS toolbox and drop on the Data Flow surface (technically you can use any valid Destination but for demo we will use OLEDB Destination)
11. Connect Source to Destination Component
12. On OLEDB destination select / create new SQL Connection and then Click "New Table"
13. Click on Mappings tab to map columns by names and click OK to save
14. Execute the Package

Configure SSIS OLEDB Destination – Loading REST API Data into SQL Server Table

REST API to SQL Server Column Mappings for OLEDB Destination

Loading REST API to SQL Server in SSIS

POST / Write Data to API (Insert, Update or Delete)

There will be a time you like to automate POST actions (e.g. create new incident via API call). Check this article to learn more on how to POST , DELETE, PUT data using API calls

Common Errors

There are many error can occur during runtime. Here are most common errors you may face at runtime.

Truncation related error

The most common error you may face when you run a SSIS package is truncation error. During the design time only 300 rows are scanned from a source (a file or a REST API call response) to detect datatypes but at runtime it is likely you will retrieve far more records. So it is possible that you will get longer strings than initially expected. For detailed instructions how to fix common metadata related errors read an article "How to handle SSIS errors (truncation, metadata issues)".

Authentication related error

Another frequent error you may get is authentication error, which happens when you deploy/copy a package to another machine and run it there. Check Deployment to Production below to see why it happens and how to solve this problem.

Deployment to Production

In SSIS package sensitive data such as tokens and passwords are by default encrypted by SSIS with your Windows account which you use to create a package. So SSIS will fail to decrypt tokens/passwords, when you run it from another machine using another Windows account. To circumvent this when you are creating a SSIS package which uses authentication components (e.g. an OAuth Connection Manager or an HTTP Connection Manager with credentials, etc.), consider using parameters/variables to pass tokens/passwords. In this way you won’t face authentication related errors when package is deployed to a production server. Check our article on how to configure packages with sensitive data on your production or development server.

Reading Zoom API data using JSON Source (with Pagination)

Now let’s look at how to use ZappySys JSON Source to read API data and parse into rows/column

1. Drag and drop data flow task from SSIS Toolbox
2. Double click task to go to data flow designer
3. Drag SSIS ZS JSON Source from toolbox
4. Edit JSON Source like below
5. Enter URL like below (Use different URL if needed.)
   https://api.zoom.us/v2/users/me/meetings?type=upcoming&page_size=100&page_number=1
6. Select connection
7. Goto pagination tab and enter information as below screenshot.
8. Click on Select Filter
9. Click Preview to test

Read Zoom API data using JSON Source (Configure Pagination)

Method-2 : Call Zoom API in SSIS using OAuth 2.0

Now let’s look at OAuth 2.0 approach to authenticate with Zoom API.

Register OAuth App

Very first step to authenticate your Zoom API call using OAuth method is to register App. Go through steps  listed here to register your App. Below are high level steps.

1. Visit http://marketplace.zoom.us/
2. Click Build menu item at the top
3. Select App Type (i.e. Account Level or User Level) and Click Begin
4. Once App credentials displayed enter Redirect URL (use https://zappysys.com/oauth2 if you are not sure what to enter here). This URL never receives data from API but its only used to get authentication code during OAuth Process so don’t worry.
5. Click on Scopes tab and select one or more permissions you like to assign to your App.

Configure SSIS OAuth Connection

Now lets get started with SSIS part. We will first create OAuth Connection for Zoom API access.

1. Open SSIS Package and drag ZS REST API Task from SSIS Control Flow Toolbox
2. Change URL Access Mode to Url From Connection
3. Enter URL as https://api.zoom.us/v2/users/me/settings
4. Select New ZS-OAUTH connection type from the connection dropdown
5. Configure OAuth connection as below. Enter Token file path, enter Redirect URL, ClientID, Client Secret and then click Generate Token (You may have to login and then click Accept button to grant permission).
   

   Configure OAuth Connection for Zoom API (Enter Token File Path – This is needed to handle changing refresh token scenario)

   

   Configure OAuth Connection for Zoom API (Enter Redirect URL / Callback URL)

   

   Generate Token for Zoom API using OAuth 2.0

6. Once you get token is generated you can click OK to save connection.
7. On the REST API Task click Test to confirm its working.
   

   Calling Zoom REST API using SSIS REST API Task

Debug Web API call using Fiddler

To test how things looking behind the scene we strongly suggest to use tool like  fiddler. You can double click the URL entry (Right side) to see Request and Response Panels. Top panel is Request (URL, Headers, Body) and Bottom Panel is Response.

Debugging Web API call using Fiddler in SSIS

Loading data from REST API to SQL Server

Once you setup HTTP / OAUTH connection we can use JSON Source or XML Source to extract data from API and  load into SQL Server. If you are API is XML based API then use XML Source in below examples else use JSON Source. For example purpose we will load data into SQL Server but you can load into any Target (e.g. Flat file, Oracle, Excel) using Microsoft or ZappySys Destination connectors To load REST API data into SQL Server perform the following steps (Screenshots are generic and used for demonstration for purpose, your values can be different than below)

1. Go to SSIS Data Flow tab
2. Drag ZS JSON Source from the SSIS Toolbox (Use ZS XML Source if your API is XML based API)
3. Double click to edit Source
4. Set API URL you like to call, Check Use Credentials if you need to use HTTP or OAuth connection
5. Set Method / Body if its other than GET. For method other than GET make sure you select correct Content-Type from Dropdown (e.g. application/json). This indicate input data format in Body.
6. Set HTTP Headers if needed (in Most cases you won't need to set this)
7. Select Filter (many times this is not needed). If your API response has array node then select it here. For example if your response looks like this { rows: [ {...}, {...}, {...} ] } then filter would be $.rows[*]. Select Array node you like to extract in the Filter selection UI.
8. Configure Pagination if needed on the pagination tab. Check your API documentation see it supports pagination.
9. Click OK to Save component settings and generate desired metadata / columns for the output.
10. Now drag OLEDB destination from the SSIS toolbox and drop on the Data Flow surface (technically you can use any valid Destination but for demo we will use OLEDB Destination)
11. Connect Source to Destination Component
12. On OLEDB destination select / create new SQL Connection and then Click "New Table"
13. Click on Mappings tab to map columns by names and click OK to save
14. Execute the Package

Configure SSIS OLEDB Destination – Loading REST API Data into SQL Server Table

REST API to SQL Server Column Mappings for OLEDB Destination

Loading REST API to SQL Server in SSIS

POST / Write Data to API (Insert, Update or Delete)

There will be a time you like to automate POST actions (e.g. create new incident via API call). Check this article to learn more on how to POST , DELETE, PUT data using API calls

Common Errors

There are many error can occur during runtime. Here are most common errors you may face at runtime.

Truncation related error

The most common error you may face when you run a SSIS package is truncation error. During the design time only 300 rows are scanned from a source (a file or a REST API call response) to detect datatypes but at runtime it is likely you will retrieve far more records. So it is possible that you will get longer strings than initially expected. For detailed instructions how to fix common metadata related errors read an article "How to handle SSIS errors (truncation, metadata issues)".

Authentication related error

Another frequent error you may get is authentication error, which happens when you deploy/copy a package to another machine and run it there. Check Deployment to Production below to see why it happens and how to solve this problem.

Deployment to Production

In SSIS package sensitive data such as tokens and passwords are by default encrypted by SSIS with your Windows account which you use to create a package. So SSIS will fail to decrypt tokens/passwords, when you run it from another machine using another Windows account. To circumvent this when you are creating a SSIS package which uses authentication components (e.g. an OAuth Connection Manager or an HTTP Connection Manager with credentials, etc.), consider using parameters/variables to pass tokens/passwords. In this way you won’t face authentication related errors when package is deployed to a production server. Check our article on how to configure packages with sensitive data on your production or development server.

Conclusion

In this post you have seen how easy it is to call Zoom API in SSIS. Just register OAuth App and start calling your API. Download SSIS PowerPack and explore many other API integration uses.

The post How to call Zoom REST API using SSIS (OAuth / JWT) appeared first on ZappySys Blog.

In this blog, we will learn How to read data from Azure DevOps in SSIS and load into SQL Server Table. Azure DevOps was formally known as TFS Online / VSTS.

In this article we will learn, how to read Issues (Work Items) using Azure DevOps API (i.e. VSTS API). In last few years Microsoft has changed marketing names few times for TFS Service. For example everything started with  Visual Source Safe (VSS) > TFS > TFS Online > Visual Studio Online > VSTS (Visual Studio Team Services)) and finally now its called Azure DevOps. In this article we might use terms like VSTS or TFS  Online or Azure DevOps but not to get confused its all about Calling Azure Dev Ops API.

In this article we will try to cover few topics such as.

• Register an OAuth App to call REST API for Azure DevOps API (i.e. VSTS / TFS Online)
• Read all Issues (Work Items) data from VSTS with SSIS and finally
• Create new Issue/Bug in Visual Studio Team Service Project WorkItems.
• Authenticate and Call Azure DevOps using PAT (Personal Access Tokens) rather than OAuth 2.0

We will go through the steps to read Issues data from Visual Studio Team Service and Load into SQL Server using  SSIS REST API Task and SSIS JSON Source.

In nutshell, this post will focus on how to call VSTS API using SSIS.

So let’s get started.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Finally, do not forget to install ZappySys SSIS PowerPack

Call Azure DevOps API Using PAT (Personal Access Token)

There are two ways to call Azure DevOps API. You can use OAuth 2.0 or use PAT (i.e. Personal Access Token) In the next section we will see how to use OAuth 2.0 if you like more secure way to authenticate (OAuth 2.0 requires more steps).

Microsoft offers Static API Token (referred as PAT – Personal Access Token) which you can use like a password (it remains valid for 1 year before you have regenerate).

So let’s get started.

Create PAT (Personal Access Token)

1. Visit to

   https://dev.azure.com/YOUR_COMPANY_NAME/_usersSettings/tokens

    . Create Personal Access Token (i.e. PAT ) by clicking User Settings icon > Personal Access Token (see below)  (detailed steps listed here )
   

   Call Azure DevOps API (formally known as VSTS / TFS Online) – Create Personal Access Token (i.e. PAT)

2. On Token Configuration Screen change Expiration to maximum (i.e. one year from today)
3. Now select permission – We will call Analytics API below (but you can also check Work Items section if you need to call some other API for read / write purpose)
   

   Change Permission / Expiration Date for Azure DevOps PAT (Personal Access Token) – Read / Write Work Items, Call Analytics API Example

Call Azure DevOps OData API in SSIS – Read WorkItems  Example (Analytics API Service)

Now lets configure JSON Source to read WorkItems by calling  Azure DevOps Analytics API (in Previous few sections we call different set of APIs). Check this link for more information on AzureDevOps Analytics API (its OData format).

For demo purpose we are calling WorkItems entity but you can read from many other entities like (Read Areas, Iterations, Projects, Teams, Tags etc). Check this link to see which Entity (i.e. Table) you can query for specific OData API version.

So let’s get started

1. Go to SSIS designer and drag Data Flow Task on the Designer Surface
   
2. Then double click Data Flow and from toolbox drag ZS JSON Source
   
3. Double click JSON Source to configure like below
   1. First change Data Format to OData (Analytics API we will call in this section is OData format)
   2. Enter URL to Query data like below. Change YOUR_COMPANY_NAME with your own name. Below API URL will call Analytics will
      

      https://analytics.dev.azure.com/YOUR_COMPANY_NAME/Odbc/_odata/v3.0/WorkItems?$select=WorkItemId,Title,WorkItemType,State,CreatedDate&$orderby=CreatedDate desc&$top=11

   3. Check Use Credentials and Select new ZS-HTTP connection (instead of OAUTH)
4. On HTTP Connection UI Configure below
   1. Enter Some URL – This will be ignored anyways (e.g. https://dev.azure.com )
   2. Change Credentials Type to Basic (User ID / Password)
   3. Enter some random word in User ID -i.e. fakeusername (this is ignored anyways but needs some non-empty value for older version of SSIS PowerPack else validation fails)
   4. Enter PAT (i.e. Personal Access Token) obtained in the previous section
5. Click OK to Save Connection UI and go back to JSON Source
6. On JSON Source, Enter or Select Array Filter as 

   $.value[*]

7. Click Preview data see its working using HTTP connection method

Configure JSON Source – Call Azure DevOps Api using PAT (Personal Access Token)

Calling Azure DevOps API in SSIS using OAuth 2.0

Now in this section we will see how to use more secure but little harder approach to authenticate. We will use OAuth 2.0 this time rather than static token (PAT) which we used in previous section.

In order to start, we will show several examples. ZappySys includes an SSIS JSON Source that will help you to call VSTS API, read various data from VSTS with SSIS, create a new issue on VSTS with SSIS and do many more actions using REST API. To learn more about VSTS API check this help file.

Create VSTS App (for OAuth 2.0 API Call)

To access VSTS data using REST API call you need to create a VSTS app. Follow the mentioned steps below to register VSTS OAuth App:

1. you can directly login to the Visual Studio Team Service – TFS Online site with your credentials, and redirect to the Visual Studio Team Service App Registration for creating an app.
2. you can open MyApps Page.
   

   Visual Studio Team Service: My Apps

   Visual Studio Team Service: My Apps[/caption]

3. Click on Create new application link, and you will be redirected to the Visual Studio Team Service App Registration Page.
4. Make sure you enter Authorization call base URL. We suggest to below call back URL if you are unsure. We will use exact same URL on OAuth connection UI (Advanced Tab)

   https://zappysys.com/oauth

5. 

   Visual Studio Team Service: New OAuth App Registration

6. Select the Authorization scopes as per your requirements, and click on Create Application button.
   For example to read Work Items or Create new Issues programmatically you can check following scope (i.e. OAuth permission).Check – Work items (full)  (i.e. vso.work_full scope)
   Note: The scopes selected on your registered application (below screenshot) must match the scopes entered in the ZappySys OAuth connection scopes field (see below). If they do not match exactly, an InvalidScope error can be produced.
   
   
   

   Visual Studio Team Service: Select Authorization Scopes

7. Once you are done with the Creation of an App certainly, you can edit the Application If you want to change Redirect URL or something.
8. furthermore, Go to MyApps Page and you have your App’s Lists there with App’s App ID as Client ID and App Secret as Client Secret.
   

   Visual Studio Team Service: Your Apps with App’s Credentials

finally, we are ready with an App. Let´s start with an example. In this example, we will show how to retrieve all Issues data from Visual Studio Team Service – TFS Online and create a new Issue in Visual Studio Team Service – TFS Online Work Items List under your Project with SSIS.

Load VSTS data to SQL Server using SSIS JSON Source

Now once you registered OAuth app in the Visual Studio Team Service – TFS Online website we can move to SSIS Part.

Configure OAuth Connection Manager

1. Right click on Connection Managers Panel to Create New OAuth Connection, and Context Menu will appear, Select New Connection from the Context Menu.
   

   Connection Manager Panel: Select Connection

2. Select ZS-OAUTH Connection Manager from the Connection Managers list.
   

   SSIS Connection Mangers List: Select O-AUTH Connection Manager

3. Configure the new OAuth Connection for VSTS(Visual Studio Team Service – TFS Online) as below.
4. First lets enter Call back URL (Same URL you entered when you created App earlier)SSIS OAuth Connection Manager: Advanced Tab Configurations
5. For VSTS API you need to set the Refresh Token File Path as below because VSTS API Token’s Refresh Token keeps changing.
6. Then configure the App on the General tab like below (i.e. OAuth Provider, Client ID, Client Secret, Scopes, etc.)
   Scope must be valid one from this list (only those scope allowed which are checked during app creation). For example during app creation you check Work items (full) then you can enter vso.work_full in scope.
   Note: The scopes selected on your registered application must match the scopes entered in the scopes field (below screenshot). If they do not match exactly, an InvalidScope error can be produced.
   
7. finally, you are done with the OAuth Configurations. Click on Generate Token button to Generate token and then click on Test Connection button and click on OK to save the connection manager configurations settings.

How to get ProjectID?

If you don’t know the ProjectID, then you can also enter ProjectName in place of ProjectID in Request URL. you can also call GET Request for getting Lists of Projects. the Request URL for getting the list of Projects is:

https://{accountName}.visualstudio.com/_apis/projects?api-version=4.1

For this, we will use SSIS REST API Task

1. First of All, Goto Control Flow Section, Drag and Drop SSIS REST API Task from the SSIS Toolbox.
   

   Drag and Drop REST API Task

2. furthermore, Double click on that Component to Edit the Configurations and Configure the task like the below-attached screesnshot.
   Some basic configurations need to be done in SSIS REST API Task.
   # Request Tab: 

   * Select Request URL Access Mode to Url from Connection.
   * Select Url Connection to your configured ZS-OAuth Connection Manager.
   * Enter Request URL: https://zappysys.visualstudio.com/_apis/projects?api-version=4.1
   * Select HTTP Request Method: GET

   

   SSIS REST API Tsk: Request Tab Configurations

   Create New Variable from Variable Menu

   

   SSIS User Variables: Create a new Variable and set Value

   # Response Tab: 

   * Select Response Content Type: Json
   * Set Response Content Filter Expression: $.value[1].id
   * check the checkbox of Save Response Content
   * Set Save Mode to Save to Variable
   * Select your Variable: {{User::ProjectID}}

   

   SSIS REST API Task: Response Tab Configurations

3.  finally, you are done with the configurations and do Test Request/Response and your ProjectID will be saved in {{User:: ProjectID}}

Configure JSON Source Component

1. First of all, Create a new SSIS Project or Open Existing Project in Visual Studio  (File > New > Project > Business Intelligence > Integration Services)
2. Once SSIS Package is open, Goto Data Flow Section, Drag and Drop ZS JSON Source Component from the SSIS Toolbox.
   

   Drag and Drop JSON Source Component

3. furthermore, Double click on that Component to Edit the Configurations. Select File path or web URL from the Access Mode drop-down.
   Enter Web URL like this:
   

   https://{accountName}.almsearch.visualstudio.com/{project}/_apis/search/workitemsearchresults?api-version=4.1-preview.1

   here, We have used 2 Parameters.

   * {accountName} is the name of the Visual Studio Team Services – TFS Online account.
   * {project} is Project ID or project name (For more See this Visual Studio Team Service API Documentation for getting filtered Issues data Request API)
   We have used Search Work Items for Fetching Paginated Work Items(Issues, Bugs)

   You can use Variable too in your Request URL, to make URL Dynamic, See the below-attached Screenshot.
   To make URL dynamic first create an SSIS variable like below screenshot and then change URL to use Variable Placeholder like this

   https://zappysys.almsearch.visualstudio.com/{{User::ProjectID}}/_apis/search/workitemsearchresults?api-version=4.1-preview.1

    

SSIS JSON Source Component: Enter the Request URL

JSON Source Configuration for Pagination

1. now, we are going to set Pagination related configurations of Visual Studio Team Service – TFS Online for Getting all WorkItems with Search POST REST Request API. Here in our example,
   Request Method: POST
   Body Content-Type: JSON (application/json)
   Request Body:
   
   

   {
     "searchText": "a",
     "$skip": <%page%>,
     "$top": 100,
     "filters": {
       "System.TeamProject": [
         "ProductTesting"
       ]
     }
   }

   here, SearchText: The SearchText for filtering WorkItems based on the keywords match.
   $skip: Number of results to be skipped
   $top: Number of results to be returned.
   
   you can take any number of the page size. for example, you can set 100 in both $skip and $top properties (I have set 10 for the Demo Purpose only)
   filters: Filters to be applied. Set it to null if there are no filters to be applied.
   System.TeamProject: ProjectName in which you want to do filtering of WorkItems.
   

   SSIS JSON Source Component: Configurations of Settings Tab

2.  You can Configure the Pagination Tab something like this:
   

   SSIS JSON Source Component: Pagination Tab Settings

3. furthermore, Select the Filter to set the Filter.
   

   SSIS JSON Source Component Configurations: Select Filter

4. It seems like you are all OK with the configurations, Click on the Preview button to see the response data.
   

   SSIS JSON Source Component: See the Preview

5. finally, Click on OK button to save JSON Source settings.

Configure OLE DB Destination Component for dumping Contacts data into SQL

1. first of all, Drag any destination (i.e. OLEDB Destination) from your SSIS toolbox. For demo purpose, we will load Contacts data into SQL Server Table.
   

   Drag and Drop OLE DB Destination Component

2. Configure OLEDB Destination (Select Connection and Target Table). You can Create a new Table for the destination by clicking a NEW button next to Table drop-down.
   

   OLE DB Destination Component – Configure for dumping WorkItems of Project from VSTS data in SQL Table

3. almost you are done with the configurations, just need to configure mappings, So for that Click on the Columns Tab to configure Mappings. also, Click on OK button to Save the configurations of OLE DB Destination Component.
   

   OLE DB Destination – Column Configurations

4. furthermore, You can see the requests in Fiddler to see the Paginated requests flows.
   

   Fiddler – See All The Requests

5. finally, Execute your package to Load Contacts data from Visual Studio Team Service – TFS Online to SQL Server.

Create a new WorkItem on Visual Studio Team Service – TFS Online (Use SSIS REST API Task)

Now let’s look at one more example to call Visual Studio Team Service(VSTS) – TFS Online REST API. This time we will create a new WorkItem using REST API call in SSIS. For this, we will use SSIS REST API Task

1. First of All, Goto Control Flow Section, Drag and Drop SSIS REST API Task from the SSIS Toolbox.
   

   Drag and Drop REST API Task

2. furthermore, Double click on that Component to Edit the Configurations and Configure the task like the below-attached screesnshot.
   

   ::::: Request Settings :::::
   
   * Select Request URL Access Mode to select Url from Connection.
   * Select Url Connection: Select your created OAUTH Connection Manager.
   * Enter Reqeust URL
         Request URL: POST https://{accountName}.visualstudio.com/{project}/_apis/wit/workitems/${type}?api-version=4.1
         here, 
         {accountName} - The name of the Visual Studio Team Services account.
         {project} - Project ID or project name
         {type} - WorkItems Type (Task, Bug, Issue, Requirement etc)
   * Select HTTP Request Method as PATCH method
   * Enter Request Body.
         Request Body:
         [
            {
                "op": "add",
                "path": "/fields/System.Title",
                "value": "Sample task"
            }
         ]
   * Select Body Content Type: Text/Plain
   *Enter Request Header:
           Content-Type: application/json-patch+json
   (For more Information See this Visual Studio Team Service - TFS Online API Documentation for Creating a new WorkItem
   https://docs.microsoft.com/en-us/rest/api/vsts/wit/work%20items/create?view=vsts-rest-4.1)

   

   REST API Task: Configurations

3. So all good with the Configurations let’s do Test Request/Response by clicking on Test Request/Response button. See the below-attached request/response screen.
   

   Create New WorkItem in VSTS Account: Do Test Request/Response

4. finally, you have created a WorkItem in VSTS successfully using VSTS REST API and REST API Task, See in VSTS Site.
   

   VSTS Site: See the Newly Created WorkItem

5. And that’s it. similarly, you can Update/Delete particular WorkItem in WorkItems List in VSTS.

Conclusion

In this blog, we learned how to read Visual Studio Team Services – TFS(Team Foundation Server) Online data in SSIS, How to register an OAuth App for Visual Studio Team Service REST API, How to get all WorkItems data from Visual Studio Team Service  – TFS(Team Foundation Server) Online and create a new WorkItem on VSTS with SSIS using ZS JSON Source Component and SSIS REST API Task in a very simple way. You can achieve many more functionalities with the use of both of these tools. Check our blogs/articles on REST API Task https://zappysys.com/blog/tag/ssis-rest-api-task/ and JSON Source Component https://zappysys.com/blog/tag/ssis-json-source/ to find out what these tools are capable of more.

References

Finally, you can use the following links for more information about the use of VSTS API with our tools:

• Visual Studio Team Service Documentation for API
• Landing Page for REST API Task, you can also find Tutorial Video here.
• Landing Page for JSON Source Component, you can also find Tutorial Video here.
• Help File: Documentation of SSIS REST API Task and JSON Source Component.

The post Read Azure DevOps data in SSIS (VSTS / TFS Online) – Call REST API / Load to SQL Server appeared first on ZappySys Blog.

In this blog, we will learn How to read Constant Contact data in SSIS and load into SQL Server Table, along with few other topics such as how to generate an API Token using Mashery Developer Account for Constant Contact REST API Call, how to read all Contacts from Constant Contact with SSIS and finally How to add new Contact in Constant Contact Contact List Collection.

We will go through the steps to read Contact data from Constant Contact and Load into SQL Server using SSIS JSON Source.

In nutshell, this post will focus on how to call Constant Contact API using SSIS. If you are MailChimp user and curious how to perform data integration in SSIS then check this article about reading MailChimp data in SSIS.

So let’s get started.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Finally, do not forget to install ZappySys SSIS PowerPack

What is Constant Contact

Keeping in touch with customers on a consistent basis is one key to maintaining current clients and bringing in new ones. A number of companies exist that provide businesses with a way to contact clients regularly through snail mail, email or social media. Constant Contact offers online marketing services for businesses to stay in touch with clients.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS REST API Task and SSIS JSON Source that will help you to call Constant Contact API to read Constant Contact data, retrieve lists of Contacts data from Constant Contact, Add new Contact in Constant Contact with SSIS and do many more actions using REST API. To learn more about Constant Contact API check this help file.

Setup your Mashery Developer Account

To access Constant Contact data using Mashery Developer Account you need to register a MasheryID with Constant Contact. Follow the steps mentioned below to register Mashery Developer Account for Constant Contact Access:

1. first of all, you can directly login to the Constant Contact site with your credentials, and redirect to the Mashrey Developer Console to register a MasheryID.
2. thereafter Fillup the form and click on Register button
   

   Register Mashery ID to Constant Contact

3. If you have already registered on Mashery Developer Account, then you just have to log in using your Credentials.
   

   Login to Mashery Developer Account

4. So, when you are done with the Mashery, you can go to MyApplication Page to create a new application.

Create Constant Contact App (for OAuth)

To access Constant Contact data using REST API call you need to create a Mashery app. Follow the steps mentioned below to register Constant Contact OAuth App:

1. First of all, you can directly login to the Mashery Developer Account with your credentials, and redirect to the Constant Contact My Application’s Page for creating an app.
2. also, you can open the MyApps Page.
   

   Constant Contact: My Apps

3. Click on Create a New Application button while, the new Application form appears, Fill it up.
   

   Constant Contact: Create New Application Form

4. Once you are done with the Creation of an App certainly, you can edit the Application.
   Enter the Redirect URL there.
   

   Constant Contact: Edit Application

5. furthermore, Go to Keys tab and you have your App’s Lists there with App’s Key as Client ID and Secret as Client Secret.
   

   Constant Contact: Apps Keys

finally, we are ready with an App. Let´s start with an example. In this example, we will show how to retrieve all Contacts data from Constant Contact and add a new Contact in Contact Collection List on Constant contact with SSIS.

Load Constant Contact data to SQL Server using SSIS JSON Source

Now once you registered OAuth app in the Constant Contact website we can move to SSIS Part.

Configure JSON Source Component

1. First of all, Create a new SSIS Project or Open Existing Project in Visual Studio  (File > New > Project > Business Intelligence > Integration Services)
2. Once SSIS Package is open, Goto Data Flow Section, Drag and Drop ZS JSON Source Component from the SSIS Toolbox.
   

   Drag and Drop JSON Source Component

3. furthermore, Double click on that Component to Edit the Configurations. Select File path or web URL from the Access Mode drop-down.
   Enter Web URL like this:
   

   https://api.constantcontact.com/v2/contacts?status=ALL&limit=100&api_key=<<<Your_App_Key>>>

   here, We have used 2 Parameters.

   * status is the filter the response based on the contact status property: ALL, ACTIVE, UNCONFIRMED, OPTOUT, or REMOVED.
   * limit is the number of contacts to return per page, 1-500, default 50 (For more See this Constant Contact API Documentation for getting Contacts Request API)
   * <<<Your_App_Key>>> is your App Key, So you need to add your app key.

   You can use Variable too in your Request URL, to make URL Dynamic, See the below-attached Screenshot.
   To make URL dynamic first create a SSIS variable like below screenshot and then change URL to use Variable Placeholder like this

   https://api.constantcontact.com/v2/contacts?status=ALL&limit=100&api_key={{User::AppKey}}

   

   SSIS User Variables: Create a new Variable and set Value

   

   SSIS JSON Source Component: Enter the Request URL

Configure OAuth Connection Manager

1. Check Use Credentials Option and select a new ZS-OAuth Connection from the drop-down.
2. Configure the new OAuth Connection for Constant Contact as below.
   Basically, you need to configure the configurations of App on the General tab (i.e. Client ID, Client Secret, Authorization URL, Access Token URL)
   

   SSIS OAuth Connection Manager: General Tab Configurations

   also, most importantly, you need to Enter your Callback URL in Advanced Tab.

   

   SSIS OAuth Connection Manager: Advance Tab Configurations.

JSON Source Configuration for Pagination

1.  now, we are going to set Pagination related configurations for Constant Contact Getting all Contacts request. Here in our example, we are setting Page size as 5. while We need to read the next page information from the response.
   So, we can set the status=ALL, Limit=100 and also we can specify Next Link Expression something like this: $.meta.pagination.next_link and Suffix for Next URL: ?api_key={{User::AppKey}}
   
   

   SSIS JSON Source – Constant Contact request Pagination Configurations Settings

2. furthermore Select the Filter to set the Filter. seems like you are all OK with the configurations, Click on Preview button to see the response data.
   

   JSON Source: Preview Contacts data of Constant Contact

3. finally, Click on OK button to save JSON Source settings.

Configure OLE DB Destination Component for dumping Contacts data into SQL

1. first of all, Drag any destination (i.e. OLEDB Destination) from your SSIS toolbox. For demo purpose, we will load Contacts data into SQL Server Table.
   

   Drag and Drop OLE DB Destination Component

2. Configure OLEDB Destination (Select Connection and Target Table). You can Create a new Table for the destination by clicking a NEW button next to Table drop-down.
   

   OLE DB Destination Component – Configure for dumping Constant Contact data in SQL Table

3. almost you are done with the configurations, just need to configure mappings, So for that Click on the Columns Tab to configure Mappings. also, Click on OK button to Save the configurations of OLE DB Destination Component.
   

   OLE DB Destination – Column Configurations

4. furthermore, You can see the requests in Fiddler to see the Paginated requests flows.
   

   Fiddler – See All The Requests

5. finally, Execute your package to Load Contacts data from Constant Contact to SQL Server.

Add a new Contact on Constant Contact (Use SSIS REST API Task)

Now let’s look at one more example to call Constant Contact REST API. This time we will add a new contact using REST API call in SSIS. For this, we will use SSIS REST API Task

1. First of All, Goto Control Flow Section, Drag and Drop SSIS REST API Task from the SSIS Toolbox.
   

   Drag and Drop REST API Task

2. furthermore, Double click on that Component to Edit the Configurations and Configure the task like the below-attached screesnshot.
   

   Request URL: 
   https://api.constantcontact.com/v2/contacts?action_by=ACTION_BY_OWNER&api_key=<<<YOUR_APP_KEY>>>
   
   here, action_by Can be ACTION_BY_OWNER or ACTION_BY_VISITOR (from the API Document,Click here for more Informaion)
   Sample Request Body: 
   {
   	"addresses": [{
   		"line1": "4928 Small Street",
   		"city": "New York",
   		"address_type": "PERSONAL",
   		"state_code": "MA",
   		"country_code": "us",
   		"postal_code": "10022"
   	}],
   	"lists": [{
   		"id": "1652428712"
   	}],
   	"email_addresses": [{
   		"email_address": "hbshah@zappysys.com"
   	}],
   	"prefix_name": "Mr.",
   	"first_name": "Hardik",
   	"last_name": "Shah",
   	"job_title": "Tech Supporter",
   	"company_name": "ZappySys",
   	"home_phone": "555-555-5555",
   	"work_phone": "555-555-5555",
   	"cell_phone": "555-555-5555",
   	"fax": "555-555-5555",
   	"custom_fields": [{
   		"name": "CustomField1",
   		"value": "Has control of $25 million budget"
   	}]
   }

   

   REST API Task: Configurations

3. now you are done with the configurations. you can do Test Request/Response. See the below-attached request/response screen.
   

   Add New Contact in Constant Contact Account: Do Test Request/Response

4. finally, you have successfully added a contact. furthermore, Goto your Constant Contact Account on its site, you can see the latest Contact added which is added by us with the use of ZappySys and SSIS.
   

   Constant Contact Site: See the Newly Added Contact

5. though, You are done with adding a contact in Constant Contact. similarly, you can Update/Delete particular Contact in Contact Collection List.

Conclusion

In this blog, we learned how to Setup your Mashery Developer Account, Create Constant Contact App for OAuth, Get all Contacts data from Constant Contact and add a new contact on Constant Contact with SSIS using ZS JSON Source Component and SSIS REST API Task in a very simple way. You can achieve many more functionalities with the use of both of these tools. Check our blogs/articles on REST API Task https://zappysys.com/blog/tag/ssis-rest-api-task/ and JSON Source Component https://zappysys.com/blog/tag/ssis-json-source/ to find out what these tools are capable of more.

References

Finally, you can use the following links for more information about the use of Constant Contact API with our tools:

• Constant Contact Documentation for API
• Landing Page for REST API Task, you can also find Tutorial Video here.
• Landing Page for JSON Source Component, you can also find Tutorial Video here.
• Help File: Documentation of SSIS REST API Task and JSON Source Component.

The post How to read Constant Contact data in SSIS – Call REST API / Load to SQL Server appeared first on ZappySys Blog.