In this tutorial, we will use the REST API Task to call some API. We will use the OAuth connection along the REST API Task to write / update data to Google Spreadsheet.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Google API.

Component Mention in this article

	JSON Generator Transform
	Web API Destination
	REST API Task

Create a Google API Project

The first step to access any Google API is to create an API Project in Google Console. If you don’t want to go through this then Skip this Step-1 and in the next section select the Default OAuth App option on OAth Connection Manager (This is the easiest option for now unless you want to use your own OAuth App).

Check this article for step-by-step instructions. When you follow these instructions make sure you enable Google Sheet API.

Create OAuth Connection Manager in SSIS

Once you create a Google API project and obtained a Client ID and Client Secret your next step is to create OAuth Connection Manager in SSIS. ZappySys OAuth connection manager comes with many predefined OAuth Providers (e.g. Facebook, Twitter, Google, etc) but you can also define custom OAuth settings for any OAuth enabled API.

To create SSIS OAuth 2.0 Connection for Google API perform the following steps.

1. Download and Install SSIS PowerPack.
2. Create a new SSIS Package.
3. Right-click in Connection Manager Area and Click “New Connection”.
4. When prompted select ZS-OAUTH connection type, On the OAuth Connection Manager Select Provider=Google.
5. Click here to create credentials. Then select the “Use Custom OAuth App” option and enter your ClientID, ClientSecret.
6. In the Scopes enter below scopes (or click Select Scopes). This will allow read/write access to Drive Files or Sheet (E.g. Reading file content / Export file to CSV). You can use the Scope browser to see many other available permissions. For more information click here.
   

   https://www.googleapis.com/auth/drive
   https://www.googleapis.com/auth/drive.file
   https://www.googleapis.com/auth/drive.readonly
   https://www.googleapis.com/auth/spreadsheets
   https://www.googleapis.com/auth/spreadsheets.readonly

7. Click on the Generate Token button.
   

   SSIS OAuth Connection Manager – Access Google Drive API using OAuth 2.0

8. You will see UI as below… Click Allow (You may have to scroll to see that button sometimes).
9. Click on Test Connection to see the connection is working and click on the OK button to save configure setting UI.

Find Google Sheet ID for API calls

Before we can write or update Sheet data in SSIS we need to know Google Driver File ID. We will use this ID in the next few sections.

The easiest way to find File ID is to look at the URL like below. If you need to know ID at runtime dynamically then refer to the last section.

Get Google Drive File ID for API call (Sheet ID)

Create Variables to make dynamic URL

1. Here, we are going to make a Dynamic URL value.
2. In Visual Studio Project Right Click on Design Panel, Select Variables.
   

   Create Variables

3. Let’s Create Variables with correct DataType and Value. In variable “varRange” set only the starting point(Sheet1!A2) of data if you don’t know the ending point and leave the first-row for giving header row later or click here.
   

Write SQL Server data to Google Spreadsheet

Now let’s look at a real-world example. How to write data from SQL Server table or other source and send to Google Sheet using Web API Destination.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Dragging and dropping Data Flow Task into Control Flow

2. Double click on the Data Flow task to see the Data Flow designer surface.
3. Here, In Visual Studio, drag and drop the OLE DB Source, JSON Generator Transform and Web API Destination in the design surface and join the components with the blue arrow.
   

   Drag and Drop OLE DB Source, JSON Generator Transform, and Web API Destination

4. Here, you can use our Free ZS Dummy Data Source too. Click here for more information.
5. But, we are going through OLE DB Source so we need OLEDB Connection.

Create OLE DB Connection

1. Lets, create the OLE DB Connection.
2. Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
   

   Create OLE DB Connection

3. Now, click on New Button to create Connection.
   

   Create New Connection

4. Let’s Configure Connection Manager, just Follow steps one by one as we have created.
   

   Configure OLE DB Connection

5. Click on Test Connection to see correct configure it.
6. Click on the OK button to save connection configure setting UI.

Configure JSON Generator Transform

1. Double click on JSON Generator Transform to configure it.
2. Set Output Mode to Single Dataset Array. Right-click on Mappings, select Add Static Element then enter Name: “range” and Select variable we have created for value.
   

   Configure JSON Generator Transform – Add Static Value

3. Again, Right-click on Mappings, select Add Static Element then enter Name: “majorDimension” and Value: “ROWS“.
   

   Configure JSON Generator Transform – Add Static Value

4. Now, Right-click on Mappings, select Add Document Array, Check to Treat as 2D array option and enter Element Name as “values“.
   

   Configure JSON Generator Transform – Add Document Array

5. Then, right-click on Document Array, select Add elements > Select Add Multiple Columns and check on columns you like to add.
   

   Configure JSON Generator Transform – Add Element

6. Click on the OK button to save JSON Generator Transform configure setting UI.

Configure Web API Destination

1. Double click on Web API Destination to configure it.
2. Select the OAuth connection manager created in an earlier section (Assuming you had all scopes with write permission).
3. In the Setting tab,
   1. In URL, Generate Dynamic URL like below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4:append?valueInputOption=USER_ENTERED

   2. Change Request Method to POST.
   3. Select JSON Generator variable into Body (Request Data) or you can select Input column for Body (JSON Generator Output).
   4. Set application/JSON to Body Content-type.
      

      Configure Web API Destination

4. Click on the OK button to save Web API Destination configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet. Now update header row values for click here.
   

   Run or Execute Package and verify data into Google Spreadsheet

Append data to Google SpreadSheet using REST API Task

Now let’s look at how to write to Google SpreadSheet. To write multiple rows in a single request. Also, check this link to learn more.

For more information, you can see the example here. For parameter, detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve the Spreadsheet write scenario. For a simple example, let’s use the REST API Task.

1. Drag and Drop ZS REST API Task on the control flow designer.
   

   Drag and Drop REST API Task

2. Double click on ZS REST API Task to configure it.
3. In the Request Settings tab,
   1. set Request URL Access Mode to Url from a connection. Select OAuth Connection from the dropdown (Same connection we created in the previous section).
   2. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A1:append?valueInputOption=USER_ENTERED

   3. Change Request Method to POST.
   4. Enter Request Body as below
      
      Note: For example purpose, we included header line in data cells but in the real-world header may already exist in the sheet.

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["Item", "Cost", "Stocked", "Ship Date"],
          ["Wheel", "$20.50", "4", "3/1/2016"],
          ["Door", "$15", "2", "3/15/2016"],
          ["Engine", "$100", "1", "3/20/2016"],
          ["Totals", "=SUM(B2:B4)", "=SUM(C2:C4)", "=MAX(D2:D4)"]
        ]
      }

   5. Change Request Content type to application/JSON.
      

      Write data to Google SpreadSheet using REST API Task (OAuth 2.0 Connection)

4. Click on the OK button to save REST API Task configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.
   

   Execute REST API Task and verify Google Spreadsheet data

Update Google Sheet Cells

Now let’s look at the below example of updating multiple cells in google Spreadsheet. For more information, you can see the example here. For parameter, detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve the Spreadsheet Update scenario. For a simple example, let’s use the REST API Task.

Update Google Sheet header cells manually

1. Drag and drop ZS REST API Tasks on the control flow designer.
2. Double click on the First REST API Task to configure it.
3. In the Request Settings tab,
   1. Enter the URL below. Change Range designator(including Sheet name: Sheet1!A1) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A1?valueInputOption=USER_ENTERED

   2. Change Request Method to PUT.
   3. Enter Request Body as below
      

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["CustomerID", "CustomerName"]
        ]
      }

4. Click on the OK button to save REST API Task configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.
   

Single Range (Set Values for multiple cells)

1. Follow the previous section(write data to Google Spreadsheet) to update sheet data.
2. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4?valueInputOption=USER_ENTERED

   2. Change Request Method to PUT.
   3. Enter Request Body as below
      

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["Laptop", "$2000.50"],
          ["Mobile", "$1150"]
        ]
      }

3. That’s all, click on the Test Request/Response button and verify data into Google spreadsheet.
   

   Calling Google Sheet API to Update Multiple Cell Values

Clear data to Google SpreadSheet using REST API Task

Clear a sheet of all values while preserving formats

Lets clear a sheet of all values by REST API Task, Before processed we need sheet Id. So let’s get the SheetId by API Call. For more information about click here.

1. Drag and Drop two ZS REST API Tasks on the control flow designer and rename(Get Sheet ID and Clear Google Sheet of all values) it.
2. Double click on the First REST API(Get Sheet ID) Task to configure it.
3. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc

   2. Change Request Method to GET.
      REST API Task – Get Google Sheet ID
4. Now go to Response Settings Tab, Select Response content Type Json and Write Response Content Filter Expression. Check on Save Response Content. Select Save Mode to Variable and Create a New variable or select it.
   Example of Response Content Filter Expression for the first Sheet : 
   

   $.sheets.[0].properties.sheetId

   

   REST API Task – Save Google Sheet ID into Variable

5. Now, click on the Test Request/Response to check configures are ok.
6. Click on the OK button to save REST API Task configure setting UI.
7. Double click on the second REST API(Clear Google Sheet of all values) Task to configure it.
8. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}:batchUpdate

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc:batchUpdate

   2. Change Request Method to POST. Enter Request Body as below and change a variable parameter as per your need.
      

      {
        "requests": [
          {
            "updateCells": {
              "range": {
                "sheetId": {{User::varGoogleSheetID}}
              },
              "fields": "userEnteredValue"
            }
          }
        ]
      }

   3. Change Request Content type to application/JSON.
      
      

      REST API Task Configure – Clear sheet of all values

9. Click on the OK button to save REST API Task configure setting UI.
10. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.

Clears range cell values from a spreadsheet

1. Double click on REST API Task to configure it. Then, follow the way just like the previous section configures REST API Task.
2. Here you need to set the value of Variable “varSpreadsheetValues” to “clear” or you can change it directly in URL append to clear and value of variable “varRange” (Change cell range and sheet name) as per your need. For more information about it click here.
   1. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4:clear?valueInputOption=USER_ENTERED

3. Change Request Method to POST.
4. Enter Request Body as below
   

   {}

5. Change Request Content type to application/JSON.
6. That’s all, click on the Test Request/Response button and verify data into Google spreadsheet.

Conclusion

Google Drive API and Google Sheets API provide a great way to automate file-related functionality. However, to call Google API  you have to use the SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Google API call via simple drag and drop approach without coding. Try out yourself see how long it takes to call virtually any REST API.

References

Finally, you can use the following links for more information:

• JSON Generator Transform
• Web API Destination
• REST API Task
• Help File:
  • JSON Generator Transform
  • Web API Destination
  • REST API Task

The post How to write/update Google Sheet in SSIS appeared first on ZappySys Blog.

In this article, we will see how to read Walmart API data in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call Walmart APIs can be useful for any developer regardless of which programming language or toolset you use.

Are you a Walmart.com Supplier or Marketplace Seller? Then you may need to click here to visit the site that is focused on these profiles.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Walmart API. Click here to create API.

Components Mentioned in this article

	ZS Logging Task
	JSON Source(REST API or File)

What is Walmart?

Walmart is the world’s largest retailer, and the Walmart Open API provides access to our extensive product catalog, thus enabling digital distribution partners to earn substantial affiliate revenues from customer referrals.

Read Walmart API Data using SSIS JSON Source

Let’s start with an example. We use SSIS JSON Source component to make the call to Walmart API, we will read Walmart data and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

Get Dynamic Token (Configure ZS-HTTP Connection)

To get API response data using SOAP / REST API call, you need to get an API Token from your login request URL. Follow the steps mentioned below to get API Token:

1. Now, Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
   

   Create a New HTTP Connection.

2. Select ZS HTTP Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
   

   Select ZS-HTTP Connection from SSIS Connection Manager.

3. In the SSIS ZS HTTP Connection Manager, Go to General Tab and Configure HTTP Connection Manager. Enter API URL, Set Credentials Type to Dynamic Token – Two-Step Auth, and configure some basic settings for the same. We used Login URL as just example but you can enter any valid API URL endpoint here. Actual Login URL goes on Dynamic Token Tab (see next step).
   

   https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20

   WM_SEC.ACCESS_TOKEN

   

   ZS HTTP Connection – Configure General Tab

4. Now, In the Dynamic Token Tab do configure like below image. Please click here to refer about Walmart API Token.
   

   https://marketplace.walmartapis.com/v3/token

   grant_type=client_credentials

   Accept: application/json
   Cache-Control: no-cache
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>
   Authorization: Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>

   

   ZS HTTP Connection – Dynamic Token Tab

5. On the Response Settings tab configure as below
   

   Walmart API – Extract Token – Response Settings

   1. Extract Mode = Extract from Body
   2. Expression Type = Json
   3. Response Content Filter =

      $.access_token

6. Click on OK button to save HTTP Connection configure settings.

That’s it we have successfully configured Connection for Walmart API in SSIS. In the next section, we will see how to use this connection and read various data from Walmart API.

Read Walmart API Data in SSIS

Once we have done creating HTTP Connection Manager we can move forward to read Walmart API data inside Data Flow. So lets Configure SSIS JSON / REST API Source.

1. Now, Drag and drop Data Flow Task from SSIS Toolbox in the currently open project and double click it to edit.
   
   

   Drag and Drop Data Flow Task.

2. Double click on the Data Flow task to see the Data Flow designer surface.
3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   

   Drag and Drop JSON Source(REST API or File).

4. Double click JSON Source and enter the following URL as below.
   

   https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20

5. Check Use Credentials and select existing HTTP connection we have created already it.
6. Enter Following HTTP Headers. If you prefer Raw edit .. Just click on Raw Edit (#6 in screenshot) and paste below headers. Make sure to replace YourClientIdGoesHere and YourSecretGoesHere
   

   Content-Type: application/xml
   Accept: application/xml
   Authorization: Basic <<YourClientIdGoesHere:YourSecretGoesHere,FUN_BASE64ENC>>
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>

    
7. Select Array Filter (ItemResponse node) or type $.ItemResponse[*]  as below.
   

   ZS JSON Source – Configure

8. That’s it, you are ready to load Wallmart API Data into SQL Server and you can use more ZappySys Destination components.
9. For pagination call please click here for reference to this link.

Load Walmart API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Walmart API integration using ODBC Driver (Power BI, Excel, MS Access)

So far we have seen how to access Walmart API using SSIS. However there will be a time when you like to access API data inside other Apps using ODBC Driver. ZappySys Provides JSON/XML/CSV drivers for virtually any API including Walmart API. Here is how to access Walmart API using XML ODBC Driver.

1. Type “ODBC” in Start menu. Open ODBC Data sources (64 bit)
   
2. Click Add on User DSN tab (if you need to use DSN under different Account …i.e. Service Account then click on System DSN Tab)
3. Select ZappySys XML Driver
4. When New DSN UI shows up, Change DSN name to anything you like (e.g. WalmartAPI)
5. Click on Load ConnectionString button and enter below connection. Make sure to replace few things
   First ######YourClientId####### with your own clientid
   Second ######YourSecret####### with your own secret
   Third ######Base64OfClientIdAndSecret####### with base64 encoded string of

   clientId:Secret

       (both separated with colon). You can use any site like this one to generate base64 string of your clientid and secret (seperated by “:”)
   

   DRIVER={ZappySys XML Driver};
   ElementsToTreatAsArray='ns2:ItemResponse';
   DataPath='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20';
   DataConnectionType=HTTP;
   AuthScheme='{none}';
   Url='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20';
   TokenUrl='https://marketplace.walmartapis.com/v3/token';
   TokenRequestData='grant_type=client_credentials';
   TokenRequestMethod='POST';
   TokenResponseContentFilter='$.access_token';
   TokenRequestHeaders='Accept:application/json||Cache-Control:no-cache||WM_SVC.NAME:Walmart Marketplace||WM_QOS.CORRELATION_ID:TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>||Authorization:Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>';
   TokenAuthHeader='WM_SEC.ACCESS_TOKEN';
   TokenResponseContentType=Json;
   UserName='######YourClientId#######';
   CredentialType=TokenDynamic;
   Password='######YourSecret#######';
   Filter='$.ns2:ItemResponses.ns2:ItemResponse[*]';
   RequestMethod='GET';
   RequestHeaders='Content-Type: application/xml || Accept: application/xml || Authorization: Basic ######Base64OfClientIdAndSecret####### || WM_SVC.NAME: Walmart Marketplace || WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>'

   

   ZappySys Driver – Load Connection String

    

6. Now go to preview tab and enter below query and click Run below SQL.

   select * from $

   –OR–
   NOTE: Change YourClientId and YourSecret in below sql before running

   SELECT * FROM $
   WITH(
   Src='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20'
   ,Header='Content-Type: application/xml
   Accept: application/xml
   Authorization: Basic <<YourClientid:YourSecret,FUN_BASE64ENC>>
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>'
   )

    

Walmart API Integration with Other BI Tools (Power BI, Excel, SSRS, MS Access…)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

How to encrypt Authorization Token as Base64Encode

1. If you want to directly pass Authorization header Token as Base64Encode(clientId:clientSecret) encrypted in the get token call, you can encrypted it using ZS Logging task and used it. Please refer to this link for the same.
2. In visual studio just Drag and Drop ZS Logging Task in the design panel.
   

   Drag and Drop ZS Logging Task

3. Double click on ZS Logging Task to configure it.
4. In the ZS Logging Task configure like the following image.
   

   Logging Task : Base64Enocde

5. That’s all, Save Base64Encoded token for use in HTTP Connection.

Conclusion

After all, we saw you how to extract information from REST API such as Walmart REST API using HTTP Connection and load into SQL Server. We also learned techniques like How to get Dynamic Token using HTTP Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File), ZS Logging Task
• Walmart API: HTTP Connection

The post How to read Walmart API data in SSIS / ODBC appeared first on ZappySys Blog.

In this new article, we will show how to read WordPress data using REST API. REST API is a very popular software architectural style for Web Services.
SSIS is a very popular ETL software used to integrate almost any data (SQL Server, Oracle, Excel, DB2, CSV files, Big Data, Facebook, Gmail, etc).

Requirements

For this example, we will use a popular plug-in. The name is ZappySys PowerPack for SSIS which includes some nice generic connectors to REST API. You can download it here:

• ZappySys PowerPack for SSIS Installer
• SSDT for Business Intelligence

Getting started

We will be using this ZappySys SSIS PowerPack component to make things work:

	SSIS REST API Web Service Task
	SSIS JSON Source (File, REST API, OData Connector)
	SSIS JSON File Destination (Create JSON File)

Read WordPress data using SSIS with REST API

REST API is also the name of one of the ZappySys components. These tools can connect to thousands of REST API Servers. In this example, we will connect to WordPress.

1. First, drag and drop the REST API task to the design pane.
2. Secondly, write the following URL in the REST API task URL:
   
   

   http://demo.wp-api.org/wp-json/wp/v2/pages

3. After that, press Test Request/Response.
4. That URL shows the page’s information in WordPress. This is a demo API. For more information about the WordPress links, check here.
   

   Call WordPress in SSIS

5. Next, you can also check the Tags, Comments, Categories, Posts, Taxonomies, Media. Here you have some examples:
   • To get the tags in WordPress, use this URL:
     

     http://demo.wp-api.org/wp-json/wp/v2/tags

   • In order to read comments in WordPress using SSIS and REST API, use this URL:
     

     http://demo.wp-api.org/wp-json/wp/v2/comments

   • For the categories, use this URL to read WordPress data using SSIS:
     

     http://demo.wp-api.org/wp-json/wp/v2/categories

6. In addition, for your blog, you need to add /wp-json/wp/v2/ and pages, posts or tags in your WordPress URL.
7. Finally, if you want to save the results, in Response Settings, enable save response content and select the option save to file, and specify a file path:
   

   Save response WordPress API

How to read WordPress data using SSIS and store it into a SQL Server database

In the next example, we will use the JSON Source component. This component is similar to the REST API, but this is used in the Data Flow to save the REST API results in any destination of your preference (SQL Server, PostgreSQL, MongoDB, etc). Also, this example will read WordPress data using SSIS and store it into a SQL Server database.

1. First of all, in the Control flow, drag and drop the Data Flow:
   

   Data Flow Task

2. Drag and drop the JSON Source in the Data flow with this configuration:
   

   JSON Source to WordPress API

3. Finally, you can save to an OLEDB Destination to save into a SQL Database:
   

   Store WordPress into SQL Server

How to read WordPress data using SSIS and modify the JSON structure

Sometimes we need to modify the WordPress JSON output to a custom output in SSIS. We will now use the JSON Generator Transform this time, which is a very powerful tool for this purpose.

1. First of all, we will use the JSON Source connecting to the following
   

   URL: http://demo.wp-api.org/wp-json/wp/v2/categories

2. Secondly, we need to convert the JSON output to the following format:
   

   {
      "id" : 
        {
            "count" : 1474359596826126768,
            "description" : 
             {
               "link" : "xxx"
             }
       }
   }

3. Next, we will add an element first:
   

   SSIS WordPress API

4. Also, we will select the id attribute:
   

   id attributes SSIS

5. In addition, we will add a static element to the id:
   

   Add JSON element

6. Furthermore, we will add the count element.
   

   Add element in JSON with SSIS

7. Finally, repeat the same process for the description and link. The JSON structure will be the following:

How to read WordPress data using SSIS and use pagination

If it is a lot of data in REST API, pagination allows showing the results in pages. Let’s take a look at it.

1. First of all, we will use page 2. To do it, we will use the following URL in the JSON Source task in the data flow:
   

   http://demo.wp-api.org/wp-json/wp/v2/posts?per_page=2

   

   SSIS URL pagination REST API

2. Secondly, there is a pagination tab that can be used to get the data using pagination. For more information about pagination, refer to our ZappySys Pagination article related.
   

   SSIS pagination in SSIS

How to write WordPress data using SSIS

1. In order to write WordPress data using SSIS, we will use the REST API task with the following URL:
   

   POST http://demo.wp-api.org/wp-json/wp/v2/posts/

2. Secondly, we have the post with id equal to 2 and we will use the body request as follows:

Conclusion

To conclude, in this article, we learned how to write and read WordPress data using SSIS. In addition, we learned also how to get pagination and how to change the format of the JSON results. If you want to try it yourself, do not hesitate to download ZappySys PowerPack!

References

Finally, for more information about how to read WordPress Data using SSIS, refer to these links:

• REST API Handbook
• WordPress REST API Tutorial (Real Examples)

The post Write/Read WordPress data using SSIS and REST API appeared first on ZappySys Blog.

In this article, you will learn how to load data from Exact Online into SQL Server using SSIS. Exact Online is a popular CRM, ERP and HRM tool which also lets you manage your finances. As an example, we will use Exact Online contacts as data to load it into a SQL Server database. We will use ZappySys SSIS PowerPack connector to get contacts from Exact Online and then use standard SSIS connector OLE DB Destination to load data into a table. This SSIS PowerPack connector will be used to get the data:

JSON Source

Step-by-step – Load data from Exact Online into SQL Server using SSIS

Getting started

We will get these contacts and load them into a SQL Server database:

Loading Exact Online contacts into SQL Server

The first thing you will need to do is to create an OAuth application in Exact App Center and then we will proceed in getting the data and loading it into a SQL Server database.

Create OAuth application in Exact App Center

1. Visit Exact App Center and login:
   

   Exact App Center login window

2. Once logged in, click on Manage my apps link:
   

   Exact App Center login area

3. Create a new OAuth application:
   

   Exact Online API applications management page

4. Give it a name.
5. In Redirect URI field enter: https://www.zappysyscom/oauth2
   
   

   Registering Exact Online API application to load data into SQL Server

6. Hit Register.
7. Don’t close the browser, we will use App information later.

Create OAuth Connection Manager to connect to Exact Online

1. Create a new SSIS package.
2. Create a new OAuth Connection Manager, by right-clicking on the Connection Managers pane:
   

   Creating OAuth Connection Manager to connect to Exact Online API application

3. Select ZS-OAUTH to create one:
   

   Selecting ZS-OAUTH option to create OAuth Connection Manager

4. Then open your Exact App Center API application configuration we left open in the previous step. We will use that data from that window.
5. Select Custom as OAuth Provider.
6. Select Authorization Code Grant as OAuth Grant Type.
7. Copy Client ID and paste it into Client Id field.
8. Copy Client secret and paste it into Client Secret field.
9. Set Authorization Url to https://start.exactonline.nl/api/oauth2/auth.
10. Set Access Token Url to https://start.exactonline.nl/api/oauth2/token.
    
    

    Configuring OAuth Connection Manager to access Exact Online data via API

11. Then open Advanced tab and copy paste Redirect URI to Callback/Return Url field:
    

    Configuring OAuth application Callback/Return URL

12. Then click on OAuth2 Grant Options tab and check option below.
13. Finally, since Exact Online refresh token keeps changing, enter a filename where to store the refresh token:
    

    Configuring OAuth2 Grant specifics

    NOTE: Can be a non-existent file or must be an empty, existing file.

Getting data from Exact Online

1. Drag and drop Data Flow and then JSON Source into it:
   

   Dragging and dropping Data Flow Task into Control Flow

2. Double click on JSON Source to modify it.
3. Check Use Credentials checkbox and select OAuth 1 as your Connection Manager.
4. Add HTTP Header named Accept and set the value to application/json.
5. In Array Filter field enter $.d.results[*].
6. In Enter Path or Web URL field enter this URL:
   https://start.exactonline.nl/api/v1/current/Me?$select=CurrentDivision
7. Hit Preview button.
8. Copy your division number, you will need it when constructing a URL for getting contacts.
   

   Getting division number

9. Now enter this URL in the field:
   https://start.exactonline.nl/api/v1/{division}/crm/Contacts?$select=FirstName,LastName
   Replace {division} with your division number.

Configuring JSON Source to get Exact Online data and load it into SQL Server

Pagination

To paginate through the results, perform these steps:

1. Open Pagination tab.
2. In Next Link/Cursor Expression field enter $.d.__next

Configuring pagination in JSON Source to paginate through Exact Online contacts

Loading Exact Online data into SQL Server

1. Drag and drop OLE DB Destination or similar connector.
2. Connect JSON Source with OLE DB Destination.
3. Create a new connection.
4. Click New to create a new table.
5. Hit OK.
6. Select Mappings to map the columns.
7. Hit OK to close the window.
8. Execute the package.

Configuring OLE DB Destination to load Exact Online data into SQL Server

The results

Once you execute the package, you should see similar results in Visual Studio and SQL Server Management Studio:

Executing SSIS package and loading Exact Online data  into SQL Server

Successfully loading Exact Online data into SQL Server

As you see, these are the same Exact Online contacts showed in the Getting Started section.

Conclusion

You learned how to load data from Exact Online into SQL Server using SSIS and ZappySys SSIS PowerPack. At first, we created an Exact Online OAuth application, then created an OAuth Connection Manager based on its information and finally loaded contacts using JSON Source.

References

Exact App Center

API methods

Get contacts API method

Pagination

The post Load data from Exact Online into SQL Server using SSIS appeared first on ZappySys Blog.

In our previous article, we saw how to call REST API in SSIS. Now in let’s use that knowledge and learn how to read/write Shopify data in SSIS. If you are not aware of Shopify then its one of the most popular eCommerce platforms out there for small shops who sell online. Shopify provides a total framework to sell/fulfill orders online including tools to build/customize your own web store.

In this article we will see few examples on how to call basic REST API calls to read shopify data, delete shopify data and write shopify data using various SSIS Components.

Create Shopify Custom App for Shopify Admin API

Before accessing Shopify API you must create Custom App (and then obtain Access Token)

New approach (Custom App) to generate App Credentials for Rest API Call

Before you call Shopify REST API, your first step is to obtain credentials. We will need Access Token or Storefront Access Token. We will use it in later section to call Shopify REST API.

1. First, in Shopify, go to sore Settings (Click Gear Icon) >  Apps and sales channels > click Develop apps for your Store
   OR just navigate to URL
   

   https://{your-store}.myshopify.com/admin/settings/apps/development

   

   Shopify App and Sales Channel

    

2. In App development, click on Create an app
   
   

   Shopify Create an App

    

3. Provide the appropriate name for your custom app.
   

   Shopify Custom App Name

    

4. Once we created the app name, we have 2 options to create Admin API or Storefront API in the Overview tab.                                                                               Shopify App Scopes and Configuration
5. Click on Configure Admin API scopes or Configure Storefront API scopes based on your need as shown in the above screenshot.
6. If click on Configure Admin API scopes, then select the appropriate scopes for your Admin API.
7. Also, change orders, transactions, and fulfilments permissions to read and write if you need to create, update and delete products.
8. In addition, change the products, variants, and collections to read and write if you need to create, update and delete products.
   

   Shopify Admin Api Scope

    

9. After saving it, it asks to install app and generate Shopify Access Token for your Admin API.
   

   Shopify Admin Install App

    

   

   Shopify Admin Access Token

10. If click at Configure Storefront API scopes, then select the appropriate scopes for your Storefront API.
11. Also, change orders, transactions, and fulfilments permissions to read and write if you need to create, update and delete products.
12. In addition, change the products, variants, and collections to read and write if you need to create, update and delete products.
    

    Shopify Storefront API Scope

     

13. After saving it, it asks to install app and generate Shopify Storefront Access Token for your Admin API shown above screenshot.
    

    Shopify Storefront Install App

     

     

    

    Shopify Storefront Access Token

     

14.  We have created Admin API Access Token and Storefront API Access Token to access the Shopify APIs.
    

    Shopify API Credentials

Old approach (Private App) to generate App Credentials for Rest API Call

Before you call Shopify REST API, your first step is to obtain credentials. We will need API Key and Password. We will use it in later section to call shopify REST API.

1. First, in Shopify, go to Apps and click Manage private apps.
   

   Shopify App – Manager private apps

2.  Secondly, In Private apps, click the Create a new private app:
   

   Shopify Private apps

3. Also, change orders, transactions, and fulfilments permissions to read and write if you need to create, update and delete products.
4. In addition, change the products, variants, and collections to read and write if you need to create, update and delete products.
   

   Permission for private apps.

5. Finally, you will have an API key, a password, an URL and a shared  secret that you will use in the next section:

Shopify API key

Read Shopify data in SSIS (Get Orders Example)

So in this section we will learn how to call Shopify REST API.  For example purpose we will call Shopify Orders Endpoint. For all other REST API endpoints check this link. Read carefully which parameters you can pass for each API call. So let’s read shopify orders using sample API call.

Configure SSIS JSON Source to read from Shopify

Now, that we created an App in Shopify, the rest is easy.

1. First, in an SSIS project, drag and drop the Data Flow and double click it.
   

   Use the Data Flow

2. Secondly, in the Data Flow, drag and drop the ZS JSON Source.
   

   SSIS JSON Source – Drag and Drop

3. Double-click on JSON Source to configure it.If you are not familiar with ZS JSON Source, check this link.
   New approach to establish the HTTP Connection Manager

   Using Admin API Access token

   1. Enter URL for the API you like to call. For Example, you can enter
      

      https://{your-store}.myshopify.com/admin/api/2022-10/products.json

   2. Check Use Credentials and click on NEW-HTTP connection as shown below,
   3. On HTTP Connection Manager you can enter some URL (it will be ignored and use JSON Source URL above)
   4. Choose Credentials Type as Static Token / API Key
   5. In the Token / Key text box enter your Admin API Key (obtained from the previous section)
   6. In the Auth Header Name enter

      X-Shopify-Access-Token

   7. In the Auth Scheme select {none} 
      

      SSIS JSON Source Admin API HTTP Connection Manager

       

4. Also, press the Select Filter button and press the orders node:
   

   SSIS Shopify filter

5. Now go to Pagination Tab and select RFC5988 Pagination mode as below. Check this link for more info on Shopify Paginated API calls.
   

   Link Header Pagination Method (RFC 5988)

   NOTE:  For Older API use Cursor-based pagination as below (version 2019-07 or older). 
   

   SSIS pagination for Shopify

6. Finally, make sure to the Use Credentials option and select the http connection of the step 2 and press the Preview button to see the data:

SSIS preview Shopify data

Load Shopify data into SQL Server / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Write Shopify data in SSIS REST API Task

In this new example, we will write Shopify data in SSIS. In addition, we will create a new order using REST API to write Shopify data in SSIS.

1. First, use the REST API task in SSDT.
2. Secondly, in the REST API use the http connection created on step 2 of the read Shopify data in SSIS.  Also, in HTTP Request Method, select POST, in Body (Request Data) write the following:
   

   {
     "order": {
       "line_items": [
         {
           "title": "Big Brown Bear Boots",
           "price": 74.99,
           "grams": "1300",
           "quantity": 3,
           "tax_lines": [
             {
               "price": 13.5,
               "rate": 0.06,
               "title": "State tax"
             }
           ]
         }
       ],
       "transactions": [
         {
           "kind": "sale",
           "status": "success",
           "amount": 238.47
         }
       ],
       "total_tax": 13.5,
       "currency": "EUR"
     }
   }

   

   Post JSON order Shopify

    

3. If you dont like to use URL supplied in HTTP Connection manager then just check override URL option and enter URL like below. For this example we used correct URL in HTTP connection manager so we will not override URL.
   

   https://{{your-store}}.myshopify.com/admin/orders.json

4. Now you can click on Test Request / Response and you will see one sample order created in your admin portal.
5. You can supply variables in Body by clicking on Insert Variable dropdown when you edit Body.

Calling Storefront API

In our previous section, we saw how to call ADMIN API which is the most common API you will use in most cases to read / write store data. However if you have a need to call Storefront API  instead then configure HTTP Connection as below

1. For Store Front API call you can follow the same steps as the previous section except for few things on HTTP Connection Manager UI
2. In API Key enter Storefront API Access Token (rather than Admin Token)
3. In the Auth Header Name enter

   X-Shopify-Storefront-Access-Token

   

   SSIS JSON Source Storefront HTTP Connection Manager

Legacy Approach – Using Basic Auth

If you created Shopify PrivateApp (before they introduced the new CustomApp approach) then your HTTP Connection needs to be like below. Use Basic Auth for Credentials Type

1. Using approaches to establish the HTTP Connection Manager 
   

   SSIS JSON Source HTTP connection

Create Multiple Shopify records using Web API Destination

With our ZS Web API destination, it is possible to create multiple records at the same time.  The following link shows the ZS Web API Destination information:

• SSIS Web API destination

Here are few useful articles to explain Web API destination use case.

How to POST data to URL

Few other articles

Delete Shopify records using REST API in SSIS

1. First, in SSDT, drag and drop the ZS REST API task.
2. Secondly, add the HTTP Url Connection.  Use the DELETE HTTP Request Method and the URL including the order id as below:
   

   https://{{your-store}}.myshopify.com/admin/orders/{{orderid}}

   

   SSIS delete data in Shopify using REST API call

    

3. Now click Test Request / Response to test delete API call.

Conclusion

In this article, we learned how to create a Shopify app. In addition, we learned how to connect to Shopify using SSIS with the ZappySys tools. Also, we learned to use ZappySys tools in SSIS to get, write and delete information.

References

Finally, for more information, refer to these links:

• ZappySys SSIS JSON Source
• ZappySys SSIS REST API task

The post Read / Write Shopify data in SSIS (REST API) appeared first on ZappySys Blog.

In this new article, we will show how to call REST API in MicroStrategy. MicroStrategy is a Business Intelligence company that provides powerful software to create nice charts, reports to take decisions. It started in 1993 and now it is one of the most popular Business Intelligence tools in the world because it is simple, intuitive and friendly. Also, It can connect to almost any database. We will use ZappySys ODBC PowerPack. Zappysys ODBC is a pretty nice software used to connect to REST API.  ZappySys ODBC is an excellent plugin for MicroStrategy, because, with this plugin, you can connect to almost any application that supports REST API.

Requirements

1. First of all, we will need MicroStrategy Desktop installed.
2. Secondly, you will require ZappySys PowerPack for ODBC installed.

Getting started

First of all, we will need to configure the ODBC ZappySys to connect to REST API. Once the ODBC connection is created, we will invoke the ODBC connection in Microstrategy.

Configuring the ODBC Zappysys

1. Search “odbc” in start menu and select ODBC Data sources (64-bit)
2. First of all, we will open the ODBC Data Source Administrator in the System DSN and press Add:
   

   ODBC Connect to REST API

   Use 64 bits if your machine supports it.
3. Secondly, add a new ZappySys JSON Driver:
   

   List of drivers supported

4. Also, we will add the following URL to the Simple View:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json

   

   ODBC configuration to connect to REST API

Configuring MicroStrategy

1. In MicroStrategy Desktop, press the New Dossier button:
   

   Create a new dossier in MicroStrategy

2. Secondly, in the Dossier, press the New Data button:
   

   Add new data in Microstrategy

3. Also, in Data Sources select Databases:
   

   Select the option to connect to Databases in Microstrategy

4. In addition, select the option to Type a Query:
   

   Create a query to get Database data

5. Next, in Data Sources press the + icon:
   

   Add Data Source in MicroStrategy

6. Finally, in Data Source, select DSN Data Sources and select the data source created in the previous section using the ZappySys ODBC Driver. In version, select Generic DBMS and write any username and password:
   

   Select DSN, version

Call REST API in MicroStrategy

1. You can write different queries using the ZappySys ODBC Driver. The driver supports SQL queries to REST API data including WHERE clauses, GROUP BY, HAVING, ORDER BY, LIMIT WITH:
   

   SELECT ShipName FROM value  
   WHERE ShipName="Around the Horn"

   

   Query to REST API using MicroStrategy

2. Secondly, you can connect online (Live) or use an in-memory Dataset:
   

   MicroStrategy connect Live

3. Finally, you can use the data for your MicroStrategy reports:
   

   MicroStrategy report created

Invoking a JSON file in MicroStrategy using ODBC

With ZappySys PowerPack, you can not only CALL REST API in Microstrategy, but you can also create queries to plain JSON files stored locally or in a web address. We will use the following JSON file to test:

1. First of all, in order to connect, you will need to specify on the ZappySys JSON Driver the path of the local JSON file:
   

   Configure a JSON file connection with ODBC

2. Secondly, you can create queries in MicroStrategy in the same way that we did in the Call REST API in MicroStrategy section:
   

   Query to JSON file in MicroStrategy

Call REST API in MicroStrategy to get Gmail information

It is also possible to invoke and query Gmail data using REST API in MicroStrategy using ZappySys ODBC PowerPack. With the ODBC PowerPack, you can write SQL queries and call REST API in MicroStrategy using OAuth or HTTP
connections.

1. First of all, in the ZappySys ODBC JSON driver, write the following URL:
   

   https://www.googleapis.com/gmail/v1/users/me/messages/

2. Secondly, in connection type select OAuth Provider.
3. Also, in the OAuth Parameter in OAuth Provider, select Google and in scope https://mail.google.com and press generate token button and specify your credentials:
   

   Configuration to connect MicroStrategy with Gmail

4. Additionally, in Zappysys you can write queries in MicroStrategy as we did with other REST API URLs:
   

   SELECT * FROM messages

   

   Query to Gmail information in MicroStrategy

Problems with the boolean data type in MicroStrategy

In addition, there is known issue related to boolean data in Microstrategy:

Error in accessing information from the external data source. Database Instance: (DataImport) Error Fetch Failed Error type: Invalid operation. Data of a string cell was truncated.

https://community.microstrategy.com/s/article/KB20514-How-to-map-a-boolean-datatype-as-integer-with-PostgreSQL

We have a workaround for this situation.

1. First of all, you will need to change the metadata. The following article shows how to change the metadata:https://zappysys.com/blog/metadata-odbc-api-drivers-query/Also, in order to fix, save your current metadata:
   

   Driver ODBC metadata

2. Additionally, modify your data from Boolean to String:
   

   MicroStrategy boolean to string metadata

REST API / SOAP API Pagination Settings for MicroStrategy

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

REST API / SOAP Web Service Connection Settings for MicroStrategy

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Other settings for REST API / SOAP XML Call in MicroStrategy

REST API / XML SOAP Performance Tips for MicroStrategy

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Calling SOAP Web Service in MicroStrategy

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

Conclusion

To conclude, we can say that the process to call REST API in MicroStrategy using ZappySys ODBC PowerPack is straightforward. With the REST API and Web service support, Microstrategy can connect to almost any source. If you liked this article and you want to try our product, you can download it here.

References

Finally, if you need more information about the process to call REST API in MicroStrategy, refer to the following links:

• ODBC PowerPack – Drivers for REST API, JSON, XML, SOAP
• Get started with MicroStrategy Desktop
• SQL Query Examples with ODBC ZappySys PowerPack

The post How to call REST API in MicroStrategy (JSON / XML SOAP) appeared first on ZappySys Blog.

Splunk is commonly used for searching, monitoring, and analyzing machine-generated big data, via a Web-style interface. In this post, you will learn how to implement Splunk API Integration with SQL Server or any other RDBMS (e.g. Oracle, MySQL, Postgresql) using SSIS in a few clicks. We will use SSIS XML Source to Read data from Splunk and Load into SQL Server / other targets (Using HTTP Connection).

We will discuss on How to Create an Intuit Developer Account, How to Create QuickBooks Online App for OAuth, We will also discuss reverse scenario to Write data to Splunk (API POST for Insert or Update in Splunk) using SSIS REST API Task

In nutshell, this post will focus on how to call Splunk API using SSIS.
So let’s get started.

What is Splunk

Splunk is a software platform to search, analyze and visualize the machine-generated data gathered from the websites, applications, sensors, devices etc. which make up your IT infrastructure and business.

The other benefits of implementing Splunk are:

• Input data can be in any format for e.g. .csv, or JSON or other formats
• Give Alerts / Events notification at the onset of a machine state
• Accurately predict the resources needed for scaling up the infrastructure
• Create knowledge objects for Operational Intelligence

Why Use Splunk REST API

So there will be a time when you want to automate certain things without using Splunk Web Portal. Here are some example scenarios which can be solved using the Splunk REST API.

• Reading/Update Splunk configuration files
• Creating Splunk searches using the REST API
• Create a new Splunk object for a specific context
• Edit a Splunk object

Step-By-Step – Import Splunk data into SQL Server

In order to start, we will show several examples. ZappySys includes an SSIS XML Source that will help you to call Splunk API, read search job results data from Splunk with SSIS, create a new Saved Search on Splunk with SSIS and do many more actions using REST API. To learn more about Splunk REST API check this help file.

To get Splunk data using the REST API call, you need to have a Free or Licensed Splunk Account. You can Download Splunk Enterprise or Register with Splunk see this link. You can find the videos for How to Install and Getting Data in on this link: Splunk Enterprise Videos.

Create / Configure Splunk Dataset

So first step in our demo would be make sure we have at least one Splunk Dataset which we can query using REST API. If you have already configured Dataset then skip this section.

1. First of all, let’s start the Splunk on your machine from the program menu.
   

   Start Splunk

2. Hence, Splunk opens into the browser. So let’s logged in and let’s Add Data for WinEvents to be searched in our next phase.
   

   Add Data

3. To start adding data first, choose a data source. For example, you can monitor WinEvents.
   

   Select Data Source

4. Select Search & Reporting in App Context and click next for review and click next to finish.
   

   Select Search & Reporting

5. That’s it local event logs input has been created successfully for searching.
   

   Start searching for local events

6. Therefore you will redirect to search page.
   

   Splunk search page

7. So, let’s import WinEvents search data into SQL Server in the next few sections.

Connect to Splunk in SSIS / Create Search JOB / Obtain JobID

Now once you have Splunk Dataset configured, next step is Call Splunk API. So most common thing you will do with splunk API is to call splunk search API. In splunk Search API is Job Style API means you perform following steps

1. First call /services/search/jobs/ endpoint with Search query (in POST) to create search Job. It returns you Job ID (also known as SID)
2. Wait until Job is done (This part may be tricky) … SO you may have to add timer task after first step and in the 3rd step enable Retry option on HTTP connection just incase it didnt finish within supplied delay.
3. Once Job finished you can read by calling below API endpoint.
   /services/search/jobs/{{your Job Id}}/results?output_mode=json

Thats it now let’s look at actual steps

1. Now let’s make the first call to search job using a POST method. If you are using a custom time range, pass it in with the POST request. Refer this link for more information: Export data using the Splunk REST API. If your URL is different than below HTTP connection then Check Use direct URL option on REST API Task and enter custom URL.
   

   URL:
   https://localhost:8089/services/search/jobs/
   
   Post Reuest Data:
   search=search source="WinEventLog:*" host="SCIFI08" earliest=-1d&output_mode=json

   

   Splunk POST Search request

2. So let’s get Response SID(Search job ID) into the variable to be used in the next call to get results.
   

   Store Splunk Response Search Job ID in a variable

3. Now add the Data Flow Task to retrieve Splunk search result data using JSON Source (REST API or File) in it. In below URL we are specifying count=5 but in real world change it to 100 or higher.
   

   TO Get Metadata deatils:
   https://localhost:8089/services/search/jobs/1549701697.1424/results?output_mode=json&offset=0&count=5

   

   Splunk Search JSON Results Data

4. Furthermore, Let’s set pagination to get all the results data.
   NOTE: In below screenshot we are specifying count=5 and Increment=5 just to show demo but in real world set both items to higher value (e.g. 100) so you request more records per page.
   

   Set Pagination

5. Let’s preview the results data.
   

   Preview Splunk Search Data

6. Edit JSON Source again and replace the below URL with it to get results By SID(Search Job ID) variable. And create the new HTTP connection as if the search job(SID) is newly created, it is possible that we will #get 204s(No Content) until the job is ready to respond.
   

   https://localhost:8089/services/search/jobs/{{User::v_sid}}/results?output_mode=json&offset=0&count=5

   

   204 No Content response error handling

7. Furthermore, click on OK button and make sure we need to click on NO.
   

   Splunk URL With variable

8. Finally, we are ready to Load Splunk data into the SQL Server.

Load Splunk data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

Above all, in this blog, we learned how to read Splunk data in SSIS. Furthermore we used XML Source Component and SSIS REST API Task to call Splunk REST API and load data into SQL server. You can download SSIS PowerPack here to try many other scenarios not discussed in this blog along with 70+ other components.

References

Finally, you can use the following links for more information about the use of Splunk Online REST API with our tools:

• REST API Task, you can also find Tutorial Video here.
• XML Source Component, you can also find Tutorial Video here.
• Help File: Documentation of REST API Task and XML Source Component.

The post How to read data from Splunk in SSIS appeared first on ZappySys Blog.

QuickBooks Online is a well-known Cloud-based Accounting Software. In this post, you will learn how to implement QuickBooks Online API Integration with SQL Server or any other RDBMS (e.g. Oracle, MySQL, Postgresql) using SSIS in few clicks. We will use SSIS XML Source to Read data from QuickBooks Online and Load into SQL Server / other targets (Using OAuth Connection). We will discuss on How to Create an Intuit Developer Account, How to Create QuickBooks Online App for OAuth, We will also discuss reverse scenario to Write data to QuickBooks Online (API POST for Insert or Update in QuickBooks Online) using SSIS REST API Task
In nutshell, this post will focus on how to call QuickBooks Online API using SSIS.
So let’s get started.

Prerequisites

Before we look into Step-By-Step section to extract and load data from QuickBooks Online to SQL Server let’s make sure you met the following requirements.

1. SSIS designer installed. Sometimes it is referred as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. ZappySys SSIS PowerPack installed. Click on the link to download a FREE trial.
4. You have the basic familiarity with REST API concepts and QuickBooks Online API.

What is QuickBooks Online

QuickBooks is an accounting software package developed and marketed by Intuit. QuickBooks products are geared mainly toward small and medium-sized businesses and offer on-premises accounting applications as well as cloud-based versions that accept business payments, manage and pay bills, and payroll functions.

Getting Started

In order to start, we will show several examples. ZappySys includes an SSIS XML Source that will help you to call QuickBooks Online API, read various data from QuickBooks Online with SSIS, create a new Contact on QuickBooks Online with SSIS and do many more actions using REST API. To learn more about QuickBooks Online API check this help file.

To access QuickBooks Online data using REST API call, you need to create an Intuit Developer Account and then create QuickBooks Online App.

Create an Intuit Developer Account

Follow the mentioned steps below to create an Intuit developer account:

1. you can visit this site developer.intuit.com, and click on Sign up from the menu bar.
2. A registration form will be opened in a Popup. you can fill up and then click on Create Account.
   

   Signup Form: Intuit Developer Account

3. Complete the validation check and click Continue.
4. On the confirmation dialog, provide a few details to fill out your profile and click All set.

The post How to read data from QuickBooks Online in SSIS appeared first on ZappySys Blog.

In this blog, we will learn How to read data from Azure DevOps in SSIS and load into SQL Server Table. Azure DevOps was formally known as TFS Online / VSTS.

In this article we will learn, how to read Issues (Work Items) using Azure DevOps API (i.e. VSTS API). In last few years Microsoft has changed marketing names few times for TFS Service. For example everything started with  Visual Source Safe (VSS) > TFS > TFS Online > Visual Studio Online > VSTS (Visual Studio Team Services)) and finally now its called Azure DevOps. In this article we might use terms like VSTS or TFS  Online or Azure DevOps but not to get confused its all about Calling Azure Dev Ops API.

In this article we will try to cover few topics such as.

• Register an OAuth App to call REST API for Azure DevOps API (i.e. VSTS / TFS Online)
• Read all Issues (Work Items) data from VSTS with SSIS and finally
• Create new Issue/Bug in Visual Studio Team Service Project WorkItems.
• Authenticate and Call Azure DevOps using PAT (Personal Access Tokens) rather than OAuth 2.0

We will go through the steps to read Issues data from Visual Studio Team Service and Load into SQL Server using  SSIS REST API Task and SSIS JSON Source.

In nutshell, this post will focus on how to call VSTS API using SSIS.

So let’s get started.

Prerequisite

1. First, you will need to have SSIS installed
2. Secondly, make sure to have SSDT
3. Finally, do not forget to install ZappySys SSIS PowerPack

Call Azure DevOps API Using PAT (Personal Access Token)

There are two ways to call Azure DevOps API. You can use OAuth 2.0 or use PAT (i.e. Personal Access Token) In the next section we will see how to use OAuth 2.0 if you like more secure way to authenticate (OAuth 2.0 requires more steps).

Microsoft offers Static API Token (referred as PAT – Personal Access Token) which you can use like a password (it remains valid for 1 year before you have regenerate).

So let’s get started.

Create PAT (Personal Access Token)

1. Visit to

   https://dev.azure.com/YOUR_COMPANY_NAME/_usersSettings/tokens

    . Create Personal Access Token (i.e. PAT ) by clicking User Settings icon > Personal Access Token (see below)  (detailed steps listed here )
   

   Call Azure DevOps API (formally known as VSTS / TFS Online) – Create Personal Access Token (i.e. PAT)

2. On Token Configuration Screen change Expiration to maximum (i.e. one year from today)
3. Now select permission – We will call Analytics API below (but you can also check Work Items section if you need to call some other API for read / write purpose)
   

   Change Permission / Expiration Date for Azure DevOps PAT (Personal Access Token) – Read / Write Work Items, Call Analytics API Example

Call Azure DevOps OData API in SSIS – Read WorkItems  Example (Analytics API Service)

Now lets configure JSON Source to read WorkItems by calling  Azure DevOps Analytics API (in Previous few sections we call different set of APIs). Check this link for more information on AzureDevOps Analytics API (its OData format).

For demo purpose we are calling WorkItems entity but you can read from many other entities like (Read Areas, Iterations, Projects, Teams, Tags etc). Check this link to see which Entity (i.e. Table) you can query for specific OData API version.

So let’s get started

1. Go to SSIS designer and drag Data Flow Task on the Designer Surface
   
2. Then double click Data Flow and from toolbox drag ZS JSON Source
   
3. Double click JSON Source to configure like below
   1. First change Data Format to OData (Analytics API we will call in this section is OData format)
   2. Enter URL to Query data like below. Change YOUR_COMPANY_NAME with your own name. Below API URL will call Analytics will
      

      https://analytics.dev.azure.com/YOUR_COMPANY_NAME/Odbc/_odata/v3.0/WorkItems?$select=WorkItemId,Title,WorkItemType,State,CreatedDate&$orderby=CreatedDate desc&$top=11

   3. Check Use Credentials and Select new ZS-HTTP connection (instead of OAUTH)
4. On HTTP Connection UI Configure below
   1. Enter Some URL – This will be ignored anyways (e.g. https://dev.azure.com )
   2. Change Credentials Type to Basic (User ID / Password)
   3. Enter some random word in User ID -i.e. fakeusername (this is ignored anyways but needs some non-empty value for older version of SSIS PowerPack else validation fails)
   4. Enter PAT (i.e. Personal Access Token) obtained in the previous section
5. Click OK to Save Connection UI and go back to JSON Source
6. On JSON Source, Enter or Select Array Filter as 

   $.value[*]

7. Click Preview data see its working using HTTP connection method

Configure JSON Source – Call Azure DevOps Api using PAT (Personal Access Token)

Calling Azure DevOps API in SSIS using OAuth 2.0

Now in this section we will see how to use more secure but little harder approach to authenticate. We will use OAuth 2.0 this time rather than static token (PAT) which we used in previous section.

In order to start, we will show several examples. ZappySys includes an SSIS JSON Source that will help you to call VSTS API, read various data from VSTS with SSIS, create a new issue on VSTS with SSIS and do many more actions using REST API. To learn more about VSTS API check this help file.

Create VSTS App (for OAuth 2.0 API Call)

To access VSTS data using REST API call you need to create a VSTS app. Follow the mentioned steps below to register VSTS OAuth App:

1. you can directly login to the Visual Studio Team Service – TFS Online site with your credentials, and redirect to the Visual Studio Team Service App Registration for creating an app.
2. you can open MyApps Page.
   

   Visual Studio Team Service: My Apps

   Visual Studio Team Service: My Apps[/caption]

3. Click on Create new application link, and you will be redirected to the Visual Studio Team Service App Registration Page.
4. Make sure you enter Authorization call base URL. We suggest to below call back URL if you are unsure. We will use exact same URL on OAuth connection UI (Advanced Tab)

   https://zappysys.com/oauth

5. 

   Visual Studio Team Service: New OAuth App Registration

6. Select the Authorization scopes as per your requirements, and click on Create Application button.
   For example to read Work Items or Create new Issues programmatically you can check following scope (i.e. OAuth permission).Check – Work items (full)  (i.e. vso.work_full scope)
   Note: The scopes selected on your registered application (below screenshot) must match the scopes entered in the ZappySys OAuth connection scopes field (see below). If they do not match exactly, an InvalidScope error can be produced.
   
   
   

   Visual Studio Team Service: Select Authorization Scopes

7. Once you are done with the Creation of an App certainly, you can edit the Application If you want to change Redirect URL or something.
8. furthermore, Go to MyApps Page and you have your App’s Lists there with App’s App ID as Client ID and App Secret as Client Secret.
   

   Visual Studio Team Service: Your Apps with App’s Credentials

finally, we are ready with an App. Let´s start with an example. In this example, we will show how to retrieve all Issues data from Visual Studio Team Service – TFS Online and create a new Issue in Visual Studio Team Service – TFS Online Work Items List under your Project with SSIS.

Load VSTS data to SQL Server using SSIS JSON Source

Now once you registered OAuth app in the Visual Studio Team Service – TFS Online website we can move to SSIS Part.

Configure OAuth Connection Manager

1. Right click on Connection Managers Panel to Create New OAuth Connection, and Context Menu will appear, Select New Connection from the Context Menu.
   

   Connection Manager Panel: Select Connection

2. Select ZS-OAUTH Connection Manager from the Connection Managers list.
   

   SSIS Connection Mangers List: Select O-AUTH Connection Manager

3. Configure the new OAuth Connection for VSTS(Visual Studio Team Service – TFS Online) as below.
4. First lets enter Call back URL (Same URL you entered when you created App earlier)SSIS OAuth Connection Manager: Advanced Tab Configurations
5. For VSTS API you need to set the Refresh Token File Path as below because VSTS API Token’s Refresh Token keeps changing.
6. Then configure the App on the General tab like below (i.e. OAuth Provider, Client ID, Client Secret, Scopes, etc.)
   Scope must be valid one from this list (only those scope allowed which are checked during app creation). For example during app creation you check Work items (full) then you can enter vso.work_full in scope.
   Note: The scopes selected on your registered application must match the scopes entered in the scopes field (below screenshot). If they do not match exactly, an InvalidScope error can be produced.
   
7. finally, you are done with the OAuth Configurations. Click on Generate Token button to Generate token and then click on Test Connection button and click on OK to save the connection manager configurations settings.

How to get ProjectID?

If you don’t know the ProjectID, then you can also enter ProjectName in place of ProjectID in Request URL. you can also call GET Request for getting Lists of Projects. the Request URL for getting the list of Projects is:

https://{accountName}.visualstudio.com/_apis/projects?api-version=4.1

For this, we will use SSIS REST API Task

1. First of All, Goto Control Flow Section, Drag and Drop SSIS REST API Task from the SSIS Toolbox.
   

   Drag and Drop REST API Task

2. furthermore, Double click on that Component to Edit the Configurations and Configure the task like the below-attached screesnshot.
   Some basic configurations need to be done in SSIS REST API Task.
   # Request Tab: 

   * Select Request URL Access Mode to Url from Connection.
   * Select Url Connection to your configured ZS-OAuth Connection Manager.
   * Enter Request URL: https://zappysys.visualstudio.com/_apis/projects?api-version=4.1
   * Select HTTP Request Method: GET

   

   SSIS REST API Tsk: Request Tab Configurations

   Create New Variable from Variable Menu

   

   SSIS User Variables: Create a new Variable and set Value

   # Response Tab: 

   * Select Response Content Type: Json
   * Set Response Content Filter Expression: $.value[1].id
   * check the checkbox of Save Response Content
   * Set Save Mode to Save to Variable
   * Select your Variable: {{User::ProjectID}}

   

   SSIS REST API Task: Response Tab Configurations

3.  finally, you are done with the configurations and do Test Request/Response and your ProjectID will be saved in {{User:: ProjectID}}

Configure JSON Source Component

1. First of all, Create a new SSIS Project or Open Existing Project in Visual Studio  (File > New > Project > Business Intelligence > Integration Services)
2. Once SSIS Package is open, Goto Data Flow Section, Drag and Drop ZS JSON Source Component from the SSIS Toolbox.
   

   Drag and Drop JSON Source Component

3. furthermore, Double click on that Component to Edit the Configurations. Select File path or web URL from the Access Mode drop-down.
   Enter Web URL like this:
   

   https://{accountName}.almsearch.visualstudio.com/{project}/_apis/search/workitemsearchresults?api-version=4.1-preview.1

   here, We have used 2 Parameters.

   * {accountName} is the name of the Visual Studio Team Services – TFS Online account.
   * {project} is Project ID or project name (For more See this Visual Studio Team Service API Documentation for getting filtered Issues data Request API)
   We have used Search Work Items for Fetching Paginated Work Items(Issues, Bugs)

   You can use Variable too in your Request URL, to make URL Dynamic, See the below-attached Screenshot.
   To make URL dynamic first create an SSIS variable like below screenshot and then change URL to use Variable Placeholder like this

   https://zappysys.almsearch.visualstudio.com/{{User::ProjectID}}/_apis/search/workitemsearchresults?api-version=4.1-preview.1

    

SSIS JSON Source Component: Enter the Request URL

JSON Source Configuration for Pagination

1. now, we are going to set Pagination related configurations of Visual Studio Team Service – TFS Online for Getting all WorkItems with Search POST REST Request API. Here in our example,
   Request Method: POST
   Body Content-Type: JSON (application/json)
   Request Body:
   
   

   {
     "searchText": "a",
     "$skip": <%page%>,
     "$top": 100,
     "filters": {
       "System.TeamProject": [
         "ProductTesting"
       ]
     }
   }

   here, SearchText: The SearchText for filtering WorkItems based on the keywords match.
   $skip: Number of results to be skipped
   $top: Number of results to be returned.
   
   you can take any number of the page size. for example, you can set 100 in both $skip and $top properties (I have set 10 for the Demo Purpose only)
   filters: Filters to be applied. Set it to null if there are no filters to be applied.
   System.TeamProject: ProjectName in which you want to do filtering of WorkItems.
   

   SSIS JSON Source Component: Configurations of Settings Tab

2.  You can Configure the Pagination Tab something like this:
   

   SSIS JSON Source Component: Pagination Tab Settings

3. furthermore, Select the Filter to set the Filter.
   

   SSIS JSON Source Component Configurations: Select Filter

4. It seems like you are all OK with the configurations, Click on the Preview button to see the response data.
   

   SSIS JSON Source Component: See the Preview

5. finally, Click on OK button to save JSON Source settings.

Configure OLE DB Destination Component for dumping Contacts data into SQL

1. first of all, Drag any destination (i.e. OLEDB Destination) from your SSIS toolbox. For demo purpose, we will load Contacts data into SQL Server Table.
   

   Drag and Drop OLE DB Destination Component

2. Configure OLEDB Destination (Select Connection and Target Table). You can Create a new Table for the destination by clicking a NEW button next to Table drop-down.
   

   OLE DB Destination Component – Configure for dumping WorkItems of Project from VSTS data in SQL Table

3. almost you are done with the configurations, just need to configure mappings, So for that Click on the Columns Tab to configure Mappings. also, Click on OK button to Save the configurations of OLE DB Destination Component.
   

   OLE DB Destination – Column Configurations

4. furthermore, You can see the requests in Fiddler to see the Paginated requests flows.
   

   Fiddler – See All The Requests

5. finally, Execute your package to Load Contacts data from Visual Studio Team Service – TFS Online to SQL Server.

Create a new WorkItem on Visual Studio Team Service – TFS Online (Use SSIS REST API Task)

Now let’s look at one more example to call Visual Studio Team Service(VSTS) – TFS Online REST API. This time we will create a new WorkItem using REST API call in SSIS. For this, we will use SSIS REST API Task

1. First of All, Goto Control Flow Section, Drag and Drop SSIS REST API Task from the SSIS Toolbox.
   

   Drag and Drop REST API Task

2. furthermore, Double click on that Component to Edit the Configurations and Configure the task like the below-attached screesnshot.
   

   ::::: Request Settings :::::
   
   * Select Request URL Access Mode to select Url from Connection.
   * Select Url Connection: Select your created OAUTH Connection Manager.
   * Enter Reqeust URL
         Request URL: POST https://{accountName}.visualstudio.com/{project}/_apis/wit/workitems/${type}?api-version=4.1
         here, 
         {accountName} - The name of the Visual Studio Team Services account.
         {project} - Project ID or project name
         {type} - WorkItems Type (Task, Bug, Issue, Requirement etc)
   * Select HTTP Request Method as PATCH method
   * Enter Request Body.
         Request Body:
         [
            {
                "op": "add",
                "path": "/fields/System.Title",
                "value": "Sample task"
            }
         ]
   * Select Body Content Type: Text/Plain
   *Enter Request Header:
           Content-Type: application/json-patch+json
   (For more Information See this Visual Studio Team Service - TFS Online API Documentation for Creating a new WorkItem
   https://docs.microsoft.com/en-us/rest/api/vsts/wit/work%20items/create?view=vsts-rest-4.1)

   

   REST API Task: Configurations

3. So all good with the Configurations let’s do Test Request/Response by clicking on Test Request/Response button. See the below-attached request/response screen.
   

   Create New WorkItem in VSTS Account: Do Test Request/Response

4. finally, you have created a WorkItem in VSTS successfully using VSTS REST API and REST API Task, See in VSTS Site.
   

   VSTS Site: See the Newly Created WorkItem

5. And that’s it. similarly, you can Update/Delete particular WorkItem in WorkItems List in VSTS.

Conclusion

In this blog, we learned how to read Visual Studio Team Services – TFS(Team Foundation Server) Online data in SSIS, How to register an OAuth App for Visual Studio Team Service REST API, How to get all WorkItems data from Visual Studio Team Service  – TFS(Team Foundation Server) Online and create a new WorkItem on VSTS with SSIS using ZS JSON Source Component and SSIS REST API Task in a very simple way. You can achieve many more functionalities with the use of both of these tools. Check our blogs/articles on REST API Task https://zappysys.com/blog/tag/ssis-rest-api-task/ and JSON Source Component https://zappysys.com/blog/tag/ssis-json-source/ to find out what these tools are capable of more.

References

Finally, you can use the following links for more information about the use of VSTS API with our tools:

• Visual Studio Team Service Documentation for API
• Landing Page for REST API Task, you can also find Tutorial Video here.
• Landing Page for JSON Source Component, you can also find Tutorial Video here.
• Help File: Documentation of SSIS REST API Task and JSON Source Component.

The post Read Azure DevOps data in SSIS (VSTS / TFS Online) – Call REST API / Load to SQL Server appeared first on ZappySys Blog.

In this article, we will learn how to read REST API in Pentaho Kettle. Pentaho is a very popular tool in Business Intelligence to integrate data for OLAP services and to create reports and dashboards. In this article, we will use our ZappySys ODBC PowerPack to connect to REST API in Pentaho Kettle.

Our software requires the JDBC connector for SQL Server and then Uses our ZappySys Gateway to connect.

Requirements

1. First of all, we will require the ZappySys ODBC PowerPack.
2. Secondly, we will use Pentaho Kettle (Our example in this post uses v8.1)
3. Finally, we will require the JDBC driver for SQL Server.

Getting started

First of all, we will need to create an ODBC connection to REST API using the Windows ODBC Data Source Administrator. Next, we will create a gateway and finally, we will invoke the data in Pentaho.

Create an ODBC connection to read REST API in Pentaho Kettle

In this example, we will first create an ODBC connection to an URL.

1. First of all, open the ODBC Data Source administrator in Windows in the System DSN press Add:
   

   Add new ODBC driver

2. Secondly, select the ZappySys JSON Driver:
   

   Select JSON driver in ODBC DS Administrator

3. Finally, add a data source name and specify a link to a URL. In this example, we will connect to the following url:
   

   URL: https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json

   

   REST API ODBC URL

Create a ZappySys gateway to JDBC to read REST API in Pentaho

In order to connect to REST API, ZappySys ODBC PowerPack uses the JDBC for SQL Server and a gateway. Let’s take a look at it:

1. First of all, open the ZappySys Gateway that is installed with the ODBC PowerPack and add a new data source:
   

   Add new Data Source in ZappySys Gateway

2. Secondly, add the data source name:
   

   Pentaho REST Client

3. In addition, in the user’s tab, add the user test and specify a password:
   

   Add user and login

4. Also, in settings, press edit and add the ODBC Data Source created in the previous section and add the user created before:
   

   Link Data Source in ODBC with the default gateway

5. Finally, in the General tab, you can specify the port. By default, it is 5000.
   Make sure to save the changes and check the gateway is started with the start button:
   

   Default gateway with port 5000

How to read REST API in Pentaho

1. First of all, in Pentaho, we will create a project like the following:
   

   Pentaho Project REST API csv

2. Secondly, we will create a query like the following:
   

   A query in Pentaho to get data

3. In addition, we will add the following connection. We will connect to the data source named customers created in the ZappySys Gateway with port 5000 and the user created in the Gateway:
   

   JDBC connection to SQL Server

4. If everything is OK, you will be able to see the CSV file generated:
   

   Data in CSV format

5. Also, you can use SQL statements. The following example shows how to do a where filter:
   

   SELECT Shipname 
   FROM value
   where Shipname='Consolidated Holdings'

6. Finally, we will show the first 10 rows using a group by:
   

   SELECT Shipname 
   FROM value
   group by shipname
   limit 10

How to read REST API in Pentaho (Gmail)

The following example will show how to get Gmail information from Pentaho.

1. First of all, we will need to create a connection to Gmail in ODBC using the ZappySys ODBC Driver (Check the Create an ODBC connection for a Pentaho REST Client section). In the ODBC Administrator, we will add a ZappySys JSON Driver connection.
2. Secondly, in the ZappySys ODBC driver, specify the URL and select the OAuth connection:
   

   ODBC configuration

3. Also, in the OAuth properties specify the Google OAuth provider and the https://mail.google.com scope. Press the generate token button and specify your credentials and accept the access permission request:
   

   Gmail OAuth connection

4. In addition, using the ZappysSys Data Gateway, add the new Data Source and associate with the ODBC connection just created:
   

   Connect Pentaho with Gmail

5. Additionally, add the test user to the data source:
   

   Include user and permission to the gateway

6. In Pentaho, change the connection in the table input to connect to the Gateway:
   

   Pentaho Rest client to Gmail

7. Finally, in the table input add the following query:
   

   Connect Pentaho to Gmail

Conclusions

To conclude, we can say that it is simple to connect to the REST API using Pentaho with the help of ZappySys ODBC PowerPack. This tool allows to easily query REST API data like a simple SQL Query. If you like this tool you can
download the ODBC PowerPack here.

References

For more information, refer to the following links:

• ZappySys ODBC PowerPack
• Introduction To Pentaho

The post How to read REST API in Pentaho Kettle appeared first on ZappySys Blog.