Using SOAP API with SharePoint in SSIS is a common requirement to Administer SharePoint. SharePoint on-premises is a nice Microsoft application used to share documents and collaborate with the company. You can have schedules, projects, documents, and more shared using a Web platform.

In addition, it is a very useful tool that your company may need. It is also possible to automate, administer tasks using SOAP API.
In this article, we will learn how to create SOAP API queries to SharePoint on-premises.

Also, we will show our ZappySys SSIS PowerPack that includes very powerful tools to export SharePoint data from REST API to any other source.

	SSIS REST API Web Service Task
	SSIS XML Source (File, REST, SOAP Web Service)

Requirements

1. Firstly, you will need SSDT installed.
2. Secondly, SSIS PowerPack installed.
3. SoapUI installed.

Getting started

We will start with SoapUI application. It will help us to get all the possible methods SharePoint SOAP API allows. Also, it will show the inner workings of how the requests should look, which we will later replicate on XML Source. SoapUI will be our polestar.

How to call SharePoint SOAP API (On-Premises)

To call SOAP API in SSIS you have to perform the following steps at a high level. Detailed steps are described in the next section

1. Import WSDL file or URL for your Service in SoapUI for testing.
   1. File > New SOAP Project.
   2. In the Initial WSDL field enter
      http://your.domain.com/_vti_bin/Lists.asmx?wsdl
      or
      http://your.domain.com/my-site/_vti_bin/Lists.asmx?wsdl
2. Add credentials for SOAP Web service (Basic Auth or WSS)
3. Create SOAP Request XML (Edit default parameters) and Click Play Button to test your request. Associative photo:
   

   Create SOAP Request Body from WSDL (Using SoapUI tool)

4. If your test works in SoapUI then you are ready to move to SSIS part.
5. Copy Service URL, Request Body and Headers to Notepad — we will use it in SSIS.
   For Header see Raw Tab on SoapUI > Copy two lines Content-Type and SOAPAction.
   

   Get ContentType and SOAPAction Headers from SoapUI Raw tab

In the next section, we will see how to call SharePoint SOAP Web Service in SSIS.

How to read data from a SharePoint List via SOAP Web Service

Now let’s move onto SSIS and get some data from SharePoint using XML Source.

1. Open SSIS, create a new Integration project and create a new SSIS package.
2. Drag and drop Data Flow onto the Control Flow from SSIS Toolbox.
3. Then drag and drop XML Source onto Data Flow (you may also use other similar connectors such as REST API Task or Web API Destination).
4. Then configure XML Source in this way:
   
   1. Enter URL:
      https://my.sharepoint.site.com/sites/mysite/_vti_bin/Lists.asmx
      or
      https://my.sharepoint.site.com/_vti_bin/Lists.asmx (depends if you have many sites or just one).
   2. Check Use Credentials checkbox and create ZS-HTTP connection. Use Basic or NTLM authorization scheme.
   3. Then copy configuration from SoapUI to XML Source. E.g. HTTP Request Method, Body, Body Content-Type, HTTP Headers (e.g. SOAPAction header).
   4. Use $.soap:Envelope.soap:Body.GetListItemsResponse.GetListItemsResult.listitems.rs:data.z:row[*] as filter. If it doesn’t work, use Select Filter button to select the path manually and select row node.
5. What’s left is to connect XML Source with OLEDB Destination or a similar destination component:
   

Conclusion

To conclude, we can say that using SSIS PowerPack component XML Source and SoapUI we could achieve our goal of getting data from SharePoint On-Premises. The next step could be writing data to SharePoint via the same SOAP API interface, but this time using Web API Destination component.

The post Read data from SharePoint List in SSIS (On-Premises SOAP API) appeared first on ZappySys Blog.

In this article, we will show how to work with the Web API Destination and create a dynamic URL, Body or header. For this purpose, we will use the following components:

ZappySys SSIS PowerPack component to make things work:

	SSIS Web API Destination (POST data to URL)
	SSIS JSON Generator Transform

In these examples, we are going to show different examples of how to send a dynamic URL, dynamic body and dynamic headers.

How to make Web API Destination dynamic URL with the OLEDB Source

1. First of all, in this example, we are going to create a dynamic URL, we will use need the following URLs:
   https://jsonplaceholder.typicode.com/posts, https://jsonplaceholder.typicode.com/comments,
   https://jsonplaceholder.typicode.com/todos, https://jsonplaceholder.typicode.com/users, and other similar URLs.
2. So, we will set the last part of the URL as a variable.
3. Also, will have a table with the values of the variable (posts, users, comments, etc):
   

   CREATE TABLE [dbo].[webApiDest](
   	[id] [smallint] NOT NULL,
   	[type] [nchar](10) NULL,
    CONSTRAINT [PK_webApiDest] PRIMARY KEY CLUSTERED 
   (
   	[id] ASC
   )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
   ) ON [PRIMARY]
   GO
   INSERT [dbo].[webApiDest] ([id], [type]) VALUES (1, N'posts     ')
   INSERT [dbo].[webApiDest] ([id], [type]) VALUES (2, N'comments  ')
   INSERT [dbo].[webApiDest] ([id], [type]) VALUES (3, N'todos     ')
   INSERT [dbo].[webApiDest] ([id], [type]) VALUES (4, N'albums    ')
   INSERT [dbo].[webApiDest] ([id], [type]) VALUES (5, N'photos    ')
   INSERT [dbo].[webApiDest] ([id], [type]) VALUES (6, N'users     ')

    
4. In addition, we are going to create a Data Flow with the following components as follows:
5. Secondly, we will connect to the table created in the previous section
   using the OLEDB Source component:
   

   OLEDB-Source-Data

How to make Web API Destination dynamic URL in SSIS with the ZS JSON Generator Transform

1. Also, in the JSON Generator transform, right-click on Mapping and select add elements:
   

   add-elements-JSON

2. Check the add multiple bound options and check the two columns:
   

   Add JSON elements

3. Pay attention to this step because this and the next one are the most important steps. Right-click the type element and select edit.
   

   Edit JSON element

4. The hide from output will hide the type from the output but will be present as a variable. Also, when we want to hide in the JSON created that element.
   

   Column downstream

5. In addition, the include columns in the downstream option are the option that will allow sending the type as a variable from the JSON Generator Transform to the Web API destination.

How to make Web API Destination dynamic URL in SSIS with the ZS JSON Generator Transform

1. First of all, we will add the input column for the body. This will send the ZSON generator into the body.
   

   Body input

2. In connections, we will create a ZS-HTTP connection to the https://jsonplaceholder.typicode.com/ site:
   

   SSIS HTTP Connection

3. In the URL we will create a dynamic part with the type as a variable that gets the values from the SQL Server table and the type column.
4. You can write <%type%> manually, or edit the URL to use placeholders and get the type variable from the columns folder:
   

   SSIS web api dynamic link

5. The Web API configuration will be something like this:
   

   Final SSIS configuration

6. Finally, run the package.

Conclusion

In this article, we learned how to send a dynamic URL to REST API. If you want to test the tool, you can download it here:

• ZappySys PowerPack

The post How to make Web API Destination dynamic (URL, Body, Headers) appeared first on ZappySys Blog.

In this tutorial, we will use the REST API Task to call some API. We will use the OAuth connection along the REST API Task to write / update data to Google Spreadsheet.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Google API.

Component Mention in this article

	JSON Generator Transform
	Web API Destination
	REST API Task

Create a Google API Project

The first step to access any Google API is to create an API Project in Google Console. If you don’t want to go through this then Skip this Step-1 and in the next section select the Default OAuth App option on OAth Connection Manager (This is the easiest option for now unless you want to use your own OAuth App).

Check this article for step-by-step instructions. When you follow these instructions make sure you enable Google Sheet API.

Create OAuth Connection Manager in SSIS

Once you create a Google API project and obtained a Client ID and Client Secret your next step is to create OAuth Connection Manager in SSIS. ZappySys OAuth connection manager comes with many predefined OAuth Providers (e.g. Facebook, Twitter, Google, etc) but you can also define custom OAuth settings for any OAuth enabled API.

To create SSIS OAuth 2.0 Connection for Google API perform the following steps.

1. Download and Install SSIS PowerPack.
2. Create a new SSIS Package.
3. Right-click in Connection Manager Area and Click “New Connection”.
4. When prompted select ZS-OAUTH connection type, On the OAuth Connection Manager Select Provider=Google.
5. Click here to create credentials. Then select the “Use Custom OAuth App” option and enter your ClientID, ClientSecret.
6. In the Scopes enter below scopes (or click Select Scopes). This will allow read/write access to Drive Files or Sheet (E.g. Reading file content / Export file to CSV). You can use the Scope browser to see many other available permissions. For more information click here.
   

   https://www.googleapis.com/auth/drive
   https://www.googleapis.com/auth/drive.file
   https://www.googleapis.com/auth/drive.readonly
   https://www.googleapis.com/auth/spreadsheets
   https://www.googleapis.com/auth/spreadsheets.readonly

7. Click on the Generate Token button.
   

   SSIS OAuth Connection Manager – Access Google Drive API using OAuth 2.0

8. You will see UI as below… Click Allow (You may have to scroll to see that button sometimes).
9. Click on Test Connection to see the connection is working and click on the OK button to save configure setting UI.

Find Google Sheet ID for API calls

Before we can write or update Sheet data in SSIS we need to know Google Driver File ID. We will use this ID in the next few sections.

The easiest way to find File ID is to look at the URL like below. If you need to know ID at runtime dynamically then refer to the last section.

Get Google Drive File ID for API call (Sheet ID)

Create Variables to make dynamic URL

1. Here, we are going to make a Dynamic URL value.
2. In Visual Studio Project Right Click on Design Panel, Select Variables.
   

   Create Variables

3. Let’s Create Variables with correct DataType and Value. In variable “varRange” set only the starting point(Sheet1!A2) of data if you don’t know the ending point and leave the first-row for giving header row later or click here.
   

Write SQL Server data to Google Spreadsheet

Now let’s look at a real-world example. How to write data from SQL Server table or other source and send to Google Sheet using Web API Destination.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Dragging and dropping Data Flow Task into Control Flow

2. Double click on the Data Flow task to see the Data Flow designer surface.
3. Here, In Visual Studio, drag and drop the OLE DB Source, JSON Generator Transform and Web API Destination in the design surface and join the components with the blue arrow.
   

   Drag and Drop OLE DB Source, JSON Generator Transform, and Web API Destination

4. Here, you can use our Free ZS Dummy Data Source too. Click here for more information.
5. But, we are going through OLE DB Source so we need OLEDB Connection.

Create OLE DB Connection

1. Lets, create the OLE DB Connection.
2. Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
   

   Create OLE DB Connection

3. Now, click on New Button to create Connection.
   

   Create New Connection

4. Let’s Configure Connection Manager, just Follow steps one by one as we have created.
   

   Configure OLE DB Connection

5. Click on Test Connection to see correct configure it.
6. Click on the OK button to save connection configure setting UI.

Configure JSON Generator Transform

1. Double click on JSON Generator Transform to configure it.
2. Set Output Mode to Single Dataset Array. Right-click on Mappings, select Add Static Element then enter Name: “range” and Select variable we have created for value.
   

   Configure JSON Generator Transform – Add Static Value

3. Again, Right-click on Mappings, select Add Static Element then enter Name: “majorDimension” and Value: “ROWS“.
   

   Configure JSON Generator Transform – Add Static Value

4. Now, Right-click on Mappings, select Add Document Array, Check to Treat as 2D array option and enter Element Name as “values“.
   

   Configure JSON Generator Transform – Add Document Array

5. Then, right-click on Document Array, select Add elements > Select Add Multiple Columns and check on columns you like to add.
   

   Configure JSON Generator Transform – Add Element

6. Click on the OK button to save JSON Generator Transform configure setting UI.

Configure Web API Destination

1. Double click on Web API Destination to configure it.
2. Select the OAuth connection manager created in an earlier section (Assuming you had all scopes with write permission).
3. In the Setting tab,
   1. In URL, Generate Dynamic URL like below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4:append?valueInputOption=USER_ENTERED

   2. Change Request Method to POST.
   3. Select JSON Generator variable into Body (Request Data) or you can select Input column for Body (JSON Generator Output).
   4. Set application/JSON to Body Content-type.
      

      Configure Web API Destination

4. Click on the OK button to save Web API Destination configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet. Now update header row values for click here.
   

   Run or Execute Package and verify data into Google Spreadsheet

Append data to Google SpreadSheet using REST API Task

Now let’s look at how to write to Google SpreadSheet. To write multiple rows in a single request. Also, check this link to learn more.

For more information, you can see the example here. For parameter, detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve the Spreadsheet write scenario. For a simple example, let’s use the REST API Task.

1. Drag and Drop ZS REST API Task on the control flow designer.
   

   Drag and Drop REST API Task

2. Double click on ZS REST API Task to configure it.
3. In the Request Settings tab,
   1. set Request URL Access Mode to Url from a connection. Select OAuth Connection from the dropdown (Same connection we created in the previous section).
   2. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A1:append?valueInputOption=USER_ENTERED

   3. Change Request Method to POST.
   4. Enter Request Body as below
      
      Note: For example purpose, we included header line in data cells but in the real-world header may already exist in the sheet.

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["Item", "Cost", "Stocked", "Ship Date"],
          ["Wheel", "$20.50", "4", "3/1/2016"],
          ["Door", "$15", "2", "3/15/2016"],
          ["Engine", "$100", "1", "3/20/2016"],
          ["Totals", "=SUM(B2:B4)", "=SUM(C2:C4)", "=MAX(D2:D4)"]
        ]
      }

   5. Change Request Content type to application/JSON.
      

      Write data to Google SpreadSheet using REST API Task (OAuth 2.0 Connection)

4. Click on the OK button to save REST API Task configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.
   

   Execute REST API Task and verify Google Spreadsheet data

Update Google Sheet Cells

Now let’s look at the below example of updating multiple cells in google Spreadsheet. For more information, you can see the example here. For parameter, detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve the Spreadsheet Update scenario. For a simple example, let’s use the REST API Task.

Update Google Sheet header cells manually

1. Drag and drop ZS REST API Tasks on the control flow designer.
2. Double click on the First REST API Task to configure it.
3. In the Request Settings tab,
   1. Enter the URL below. Change Range designator(including Sheet name: Sheet1!A1) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A1?valueInputOption=USER_ENTERED

   2. Change Request Method to PUT.
   3. Enter Request Body as below
      

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["CustomerID", "CustomerName"]
        ]
      }

4. Click on the OK button to save REST API Task configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.
   

Single Range (Set Values for multiple cells)

1. Follow the previous section(write data to Google Spreadsheet) to update sheet data.
2. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4?valueInputOption=USER_ENTERED

   2. Change Request Method to PUT.
   3. Enter Request Body as below
      

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["Laptop", "$2000.50"],
          ["Mobile", "$1150"]
        ]
      }

3. That’s all, click on the Test Request/Response button and verify data into Google spreadsheet.
   

   Calling Google Sheet API to Update Multiple Cell Values

Clear data to Google SpreadSheet using REST API Task

Clear a sheet of all values while preserving formats

Lets clear a sheet of all values by REST API Task, Before processed we need sheet Id. So let’s get the SheetId by API Call. For more information about click here.

1. Drag and Drop two ZS REST API Tasks on the control flow designer and rename(Get Sheet ID and Clear Google Sheet of all values) it.
2. Double click on the First REST API(Get Sheet ID) Task to configure it.
3. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc

   2. Change Request Method to GET.
      REST API Task – Get Google Sheet ID
4. Now go to Response Settings Tab, Select Response content Type Json and Write Response Content Filter Expression. Check on Save Response Content. Select Save Mode to Variable and Create a New variable or select it.
   Example of Response Content Filter Expression for the first Sheet : 
   

   $.sheets.[0].properties.sheetId

   

   REST API Task – Save Google Sheet ID into Variable

5. Now, click on the Test Request/Response to check configures are ok.
6. Click on the OK button to save REST API Task configure setting UI.
7. Double click on the second REST API(Clear Google Sheet of all values) Task to configure it.
8. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}:batchUpdate

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc:batchUpdate

   2. Change Request Method to POST. Enter Request Body as below and change a variable parameter as per your need.
      

      {
        "requests": [
          {
            "updateCells": {
              "range": {
                "sheetId": {{User::varGoogleSheetID}}
              },
              "fields": "userEnteredValue"
            }
          }
        ]
      }

   3. Change Request Content type to application/JSON.
      
      

      REST API Task Configure – Clear sheet of all values

9. Click on the OK button to save REST API Task configure setting UI.
10. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.

Clears range cell values from a spreadsheet

1. Double click on REST API Task to configure it. Then, follow the way just like the previous section configures REST API Task.
2. Here you need to set the value of Variable “varSpreadsheetValues” to “clear” or you can change it directly in URL append to clear and value of variable “varRange” (Change cell range and sheet name) as per your need. For more information about it click here.
   1. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4:clear?valueInputOption=USER_ENTERED

3. Change Request Method to POST.
4. Enter Request Body as below
   

   {}

5. Change Request Content type to application/JSON.
6. That’s all, click on the Test Request/Response button and verify data into Google spreadsheet.

Conclusion

Google Drive API and Google Sheets API provide a great way to automate file-related functionality. However, to call Google API  you have to use the SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Google API call via simple drag and drop approach without coding. Try out yourself see how long it takes to call virtually any REST API.

References

Finally, you can use the following links for more information:

• JSON Generator Transform
• Web API Destination
• REST API Task
• Help File:
  • JSON Generator Transform
  • Web API Destination
  • REST API Task

The post How to write/update Google Sheet in SSIS appeared first on ZappySys Blog.

In this article, we will see how to read Walmart API data in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call Walmart APIs can be useful for any developer regardless of which programming language or toolset you use.

Are you a Walmart.com Supplier or Marketplace Seller? Then you may need to click here to visit the site that is focused on these profiles.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Walmart API. Click here to create API.

Components Mentioned in this article

	ZS Logging Task
	JSON Source(REST API or File)

What is Walmart?

Walmart is the world’s largest retailer, and the Walmart Open API provides access to our extensive product catalog, thus enabling digital distribution partners to earn substantial affiliate revenues from customer referrals.

Read Walmart API Data using SSIS JSON Source

Let’s start with an example. We use SSIS JSON Source component to make the call to Walmart API, we will read Walmart data and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

Get Dynamic Token (Configure ZS-HTTP Connection)

To get API response data using SOAP / REST API call, you need to get an API Token from your login request URL. Follow the steps mentioned below to get API Token:

1. Now, Right Click on Connection Managers Window, and one Context menu appears for creating a new connection manager.
   

   Create a New HTTP Connection.

2. Select ZS HTTP Connection Manager from the list of Connection Managers and, double-click on it or click on Add button to add a new Connection Manager. By Default, The created connection manager is of Package-Level. You can set Project Level Connection manager too If you want.
   

   Select ZS-HTTP Connection from SSIS Connection Manager.

3. In the SSIS ZS HTTP Connection Manager, Go to General Tab and Configure HTTP Connection Manager. Enter API URL, Set Credentials Type to Dynamic Token – Two-Step Auth, and configure some basic settings for the same. We used Login URL as just example but you can enter any valid API URL endpoint here. Actual Login URL goes on Dynamic Token Tab (see next step).
   

   https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20

   WM_SEC.ACCESS_TOKEN

   

   ZS HTTP Connection – Configure General Tab

4. Now, In the Dynamic Token Tab do configure like below image. Please click here to refer about Walmart API Token.
   

   https://marketplace.walmartapis.com/v3/token

   grant_type=client_credentials

   Accept: application/json
   Cache-Control: no-cache
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>
   Authorization: Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>

   

   ZS HTTP Connection – Dynamic Token Tab

5. On the Response Settings tab configure as below
   

   Walmart API – Extract Token – Response Settings

   1. Extract Mode = Extract from Body
   2. Expression Type = Json
   3. Response Content Filter =

      $.access_token

6. Click on OK button to save HTTP Connection configure settings.

That’s it we have successfully configured Connection for Walmart API in SSIS. In the next section, we will see how to use this connection and read various data from Walmart API.

Read Walmart API Data in SSIS

Once we have done creating HTTP Connection Manager we can move forward to read Walmart API data inside Data Flow. So lets Configure SSIS JSON / REST API Source.

1. Now, Drag and drop Data Flow Task from SSIS Toolbox in the currently open project and double click it to edit.
   
   

   Drag and Drop Data Flow Task.

2. Double click on the Data Flow task to see the Data Flow designer surface.
3. From the SSIS toolbox drag and drop JSON Source on the dataflow designer surface.
   

   Drag and Drop JSON Source(REST API or File).

4. Double click JSON Source and enter the following URL as below.
   

   https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20

5. Check Use Credentials and select existing HTTP connection we have created already it.
6. Enter Following HTTP Headers. If you prefer Raw edit .. Just click on Raw Edit (#6 in screenshot) and paste below headers. Make sure to replace YourClientIdGoesHere and YourSecretGoesHere
   

   Content-Type: application/xml
   Accept: application/xml
   Authorization: Basic <<YourClientIdGoesHere:YourSecretGoesHere,FUN_BASE64ENC>>
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>

    
7. Select Array Filter (ItemResponse node) or type $.ItemResponse[*]  as below.
   

   ZS JSON Source – Configure

8. That’s it, you are ready to load Wallmart API Data into SQL Server and you can use more ZappySys Destination components.
9. For pagination call please click here for reference to this link.

Load Walmart API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Walmart API integration using ODBC Driver (Power BI, Excel, MS Access)

So far we have seen how to access Walmart API using SSIS. However there will be a time when you like to access API data inside other Apps using ODBC Driver. ZappySys Provides JSON/XML/CSV drivers for virtually any API including Walmart API. Here is how to access Walmart API using XML ODBC Driver.

1. Type “ODBC” in Start menu. Open ODBC Data sources (64 bit)
   
2. Click Add on User DSN tab (if you need to use DSN under different Account …i.e. Service Account then click on System DSN Tab)
3. Select ZappySys XML Driver
4. When New DSN UI shows up, Change DSN name to anything you like (e.g. WalmartAPI)
5. Click on Load ConnectionString button and enter below connection. Make sure to replace few things
   First ######YourClientId####### with your own clientid
   Second ######YourSecret####### with your own secret
   Third ######Base64OfClientIdAndSecret####### with base64 encoded string of

   clientId:Secret

       (both separated with colon). You can use any site like this one to generate base64 string of your clientid and secret (seperated by “:”)
   

   DRIVER={ZappySys XML Driver};
   ElementsToTreatAsArray='ns2:ItemResponse';
   DataPath='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20';
   DataConnectionType=HTTP;
   AuthScheme='{none}';
   Url='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20';
   TokenUrl='https://marketplace.walmartapis.com/v3/token';
   TokenRequestData='grant_type=client_credentials';
   TokenRequestMethod='POST';
   TokenResponseContentFilter='$.access_token';
   TokenRequestHeaders='Accept:application/json||Cache-Control:no-cache||WM_SVC.NAME:Walmart Marketplace||WM_QOS.CORRELATION_ID:TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>||Authorization:Basic <<[$userid$]:[$password$],FUN_BASE64ENC>>';
   TokenAuthHeader='WM_SEC.ACCESS_TOKEN';
   TokenResponseContentType=Json;
   UserName='######YourClientId#######';
   CredentialType=TokenDynamic;
   Password='######YourSecret#######';
   Filter='$.ns2:ItemResponses.ns2:ItemResponse[*]';
   RequestMethod='GET';
   RequestHeaders='Content-Type: application/xml || Accept: application/xml || Authorization: Basic ######Base64OfClientIdAndSecret####### || WM_SVC.NAME: Walmart Marketplace || WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>'

   

   ZappySys Driver – Load Connection String

    

6. Now go to preview tab and enter below query and click Run below SQL.

   select * from $

   –OR–
   NOTE: Change YourClientId and YourSecret in below sql before running

   SELECT * FROM $
   WITH(
   Src='https://marketplace.walmartapis.com/v3/items?nextCursor=*&offset=2000&limit=20'
   ,Header='Content-Type: application/xml
   Accept: application/xml
   Authorization: Basic <<YourClientid:YourSecret,FUN_BASE64ENC>>
   WM_SVC.NAME: Walmart Marketplace
   WM_QOS.CORRELATION_ID: TokenId_<<yyyy-MM-ddTHH:mm:ss.fffZ,FUN_GETDATE_UTC>>'
   )

    

Walmart API Integration with Other BI Tools (Power BI, Excel, SSRS, MS Access…)

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

How to encrypt Authorization Token as Base64Encode

1. If you want to directly pass Authorization header Token as Base64Encode(clientId:clientSecret) encrypted in the get token call, you can encrypted it using ZS Logging task and used it. Please refer to this link for the same.
2. In visual studio just Drag and Drop ZS Logging Task in the design panel.
   

   Drag and Drop ZS Logging Task

3. Double click on ZS Logging Task to configure it.
4. In the ZS Logging Task configure like the following image.
   

   Logging Task : Base64Enocde

5. That’s all, Save Base64Encoded token for use in HTTP Connection.

Conclusion

After all, we saw you how to extract information from REST API such as Walmart REST API using HTTP Connection and load into SQL Server. We also learned techniques like How to get Dynamic Token using HTTP Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File), ZS Logging Task
• Walmart API: HTTP Connection

The post How to read Walmart API data in SSIS / ODBC appeared first on ZappySys Blog.

In this new article, we will show how to read WordPress data using REST API. REST API is a very popular software architectural style for Web Services.
SSIS is a very popular ETL software used to integrate almost any data (SQL Server, Oracle, Excel, DB2, CSV files, Big Data, Facebook, Gmail, etc).

Requirements

For this example, we will use a popular plug-in. The name is ZappySys PowerPack for SSIS which includes some nice generic connectors to REST API. You can download it here:

• ZappySys PowerPack for SSIS Installer
• SSDT for Business Intelligence

Getting started

We will be using this ZappySys SSIS PowerPack component to make things work:

	SSIS REST API Web Service Task
	SSIS JSON Source (File, REST API, OData Connector)
	SSIS JSON File Destination (Create JSON File)

Read WordPress data using SSIS with REST API

REST API is also the name of one of the ZappySys components. These tools can connect to thousands of REST API Servers. In this example, we will connect to WordPress.

1. First, drag and drop the REST API task to the design pane.
2. Secondly, write the following URL in the REST API task URL:
   
   

   http://demo.wp-api.org/wp-json/wp/v2/pages

3. After that, press Test Request/Response.
4. That URL shows the page’s information in WordPress. This is a demo API. For more information about the WordPress links, check here.
   

   Call WordPress in SSIS

5. Next, you can also check the Tags, Comments, Categories, Posts, Taxonomies, Media. Here you have some examples:
   • To get the tags in WordPress, use this URL:
     

     http://demo.wp-api.org/wp-json/wp/v2/tags

   • In order to read comments in WordPress using SSIS and REST API, use this URL:
     

     http://demo.wp-api.org/wp-json/wp/v2/comments

   • For the categories, use this URL to read WordPress data using SSIS:
     

     http://demo.wp-api.org/wp-json/wp/v2/categories

6. In addition, for your blog, you need to add /wp-json/wp/v2/ and pages, posts or tags in your WordPress URL.
7. Finally, if you want to save the results, in Response Settings, enable save response content and select the option save to file, and specify a file path:
   

   Save response WordPress API

How to read WordPress data using SSIS and store it into a SQL Server database

In the next example, we will use the JSON Source component. This component is similar to the REST API, but this is used in the Data Flow to save the REST API results in any destination of your preference (SQL Server, PostgreSQL, MongoDB, etc). Also, this example will read WordPress data using SSIS and store it into a SQL Server database.

1. First of all, in the Control flow, drag and drop the Data Flow:
   

   Data Flow Task

2. Drag and drop the JSON Source in the Data flow with this configuration:
   

   JSON Source to WordPress API

3. Finally, you can save to an OLEDB Destination to save into a SQL Database:
   

   Store WordPress into SQL Server

How to read WordPress data using SSIS and modify the JSON structure

Sometimes we need to modify the WordPress JSON output to a custom output in SSIS. We will now use the JSON Generator Transform this time, which is a very powerful tool for this purpose.

1. First of all, we will use the JSON Source connecting to the following
   

   URL: http://demo.wp-api.org/wp-json/wp/v2/categories

2. Secondly, we need to convert the JSON output to the following format:
   

   {
      "id" : 
        {
            "count" : 1474359596826126768,
            "description" : 
             {
               "link" : "xxx"
             }
       }
   }

3. Next, we will add an element first:
   

   SSIS WordPress API

4. Also, we will select the id attribute:
   

   id attributes SSIS

5. In addition, we will add a static element to the id:
   

   Add JSON element

6. Furthermore, we will add the count element.
   

   Add element in JSON with SSIS

7. Finally, repeat the same process for the description and link. The JSON structure will be the following:

How to read WordPress data using SSIS and use pagination

If it is a lot of data in REST API, pagination allows showing the results in pages. Let’s take a look at it.

1. First of all, we will use page 2. To do it, we will use the following URL in the JSON Source task in the data flow:
   

   http://demo.wp-api.org/wp-json/wp/v2/posts?per_page=2

   

   SSIS URL pagination REST API

2. Secondly, there is a pagination tab that can be used to get the data using pagination. For more information about pagination, refer to our ZappySys Pagination article related.
   

   SSIS pagination in SSIS

How to write WordPress data using SSIS

1. In order to write WordPress data using SSIS, we will use the REST API task with the following URL:
   

   POST http://demo.wp-api.org/wp-json/wp/v2/posts/

2. Secondly, we have the post with id equal to 2 and we will use the body request as follows:

Conclusion

To conclude, in this article, we learned how to write and read WordPress data using SSIS. In addition, we learned also how to get pagination and how to change the format of the JSON results. If you want to try it yourself, do not hesitate to download ZappySys PowerPack!

References

Finally, for more information about how to read WordPress Data using SSIS, refer to these links:

• REST API Handbook
• WordPress REST API Tutorial (Real Examples)

The post Write/Read WordPress data using SSIS and REST API appeared first on ZappySys Blog.

In our previous article, we saw how to call REST API in SSIS. Now in let’s use that knowledge and learn how to read/write Shopify data in SSIS. If you are not aware of Shopify then its one of the most popular eCommerce platforms out there for small shops who sell online. Shopify provides a total framework to sell/fulfill orders online including tools to build/customize your own web store.

In this article we will see few examples on how to call basic REST API calls to read shopify data, delete shopify data and write shopify data using various SSIS Components.

Create Shopify Custom App for Shopify Admin API

Before accessing Shopify API you must create Custom App (and then obtain Access Token)

New approach (Custom App) to generate App Credentials for Rest API Call

Before you call Shopify REST API, your first step is to obtain credentials. We will need Access Token or Storefront Access Token. We will use it in later section to call Shopify REST API.

1. First, in Shopify, go to sore Settings (Click Gear Icon) >  Apps and sales channels > click Develop apps for your Store
   OR just navigate to URL
   

   https://{your-store}.myshopify.com/admin/settings/apps/development

   

   Shopify App and Sales Channel

    

2. In App development, click on Create an app
   
   

   Shopify Create an App

    

3. Provide the appropriate name for your custom app.
   

   Shopify Custom App Name

    

4. Once we created the app name, we have 2 options to create Admin API or Storefront API in the Overview tab.                                                                               Shopify App Scopes and Configuration
5. Click on Configure Admin API scopes or Configure Storefront API scopes based on your need as shown in the above screenshot.
6. If click on Configure Admin API scopes, then select the appropriate scopes for your Admin API.
7. Also, change orders, transactions, and fulfilments permissions to read and write if you need to create, update and delete products.
8. In addition, change the products, variants, and collections to read and write if you need to create, update and delete products.
   

   Shopify Admin Api Scope

    

9. After saving it, it asks to install app and generate Shopify Access Token for your Admin API.
   

   Shopify Admin Install App

    

   

   Shopify Admin Access Token

10. If click at Configure Storefront API scopes, then select the appropriate scopes for your Storefront API.
11. Also, change orders, transactions, and fulfilments permissions to read and write if you need to create, update and delete products.
12. In addition, change the products, variants, and collections to read and write if you need to create, update and delete products.
    

    Shopify Storefront API Scope

     

13. After saving it, it asks to install app and generate Shopify Storefront Access Token for your Admin API shown above screenshot.
    

    Shopify Storefront Install App

     

     

    

    Shopify Storefront Access Token

     

14.  We have created Admin API Access Token and Storefront API Access Token to access the Shopify APIs.
    

    Shopify API Credentials

Old approach (Private App) to generate App Credentials for Rest API Call

Before you call Shopify REST API, your first step is to obtain credentials. We will need API Key and Password. We will use it in later section to call shopify REST API.

1. First, in Shopify, go to Apps and click Manage private apps.
   

   Shopify App – Manager private apps

2.  Secondly, In Private apps, click the Create a new private app:
   

   Shopify Private apps

3. Also, change orders, transactions, and fulfilments permissions to read and write if you need to create, update and delete products.
4. In addition, change the products, variants, and collections to read and write if you need to create, update and delete products.
   

   Permission for private apps.

5. Finally, you will have an API key, a password, an URL and a shared  secret that you will use in the next section:

Shopify API key

Read Shopify data in SSIS (Get Orders Example)

So in this section we will learn how to call Shopify REST API.  For example purpose we will call Shopify Orders Endpoint. For all other REST API endpoints check this link. Read carefully which parameters you can pass for each API call. So let’s read shopify orders using sample API call.

Configure SSIS JSON Source to read from Shopify

Now, that we created an App in Shopify, the rest is easy.

1. First, in an SSIS project, drag and drop the Data Flow and double click it.
   

   Use the Data Flow

2. Secondly, in the Data Flow, drag and drop the ZS JSON Source.
   

   SSIS JSON Source – Drag and Drop

3. Double-click on JSON Source to configure it.If you are not familiar with ZS JSON Source, check this link.
   New approach to establish the HTTP Connection Manager

   Using Admin API Access token

   1. Enter URL for the API you like to call. For Example, you can enter
      

      https://{your-store}.myshopify.com/admin/api/2022-10/products.json

   2. Check Use Credentials and click on NEW-HTTP connection as shown below,
   3. On HTTP Connection Manager you can enter some URL (it will be ignored and use JSON Source URL above)
   4. Choose Credentials Type as Static Token / API Key
   5. In the Token / Key text box enter your Admin API Key (obtained from the previous section)
   6. In the Auth Header Name enter

      X-Shopify-Access-Token

   7. In the Auth Scheme select {none} 
      

      SSIS JSON Source Admin API HTTP Connection Manager

       

4. Also, press the Select Filter button and press the orders node:
   

   SSIS Shopify filter

5. Now go to Pagination Tab and select RFC5988 Pagination mode as below. Check this link for more info on Shopify Paginated API calls.
   

   Link Header Pagination Method (RFC 5988)

   NOTE:  For Older API use Cursor-based pagination as below (version 2019-07 or older). 
   

   SSIS pagination for Shopify

6. Finally, make sure to the Use Credentials option and select the http connection of the step 2 and press the Preview button to see the data:

SSIS preview Shopify data

Load Shopify data into SQL Server / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Write Shopify data in SSIS REST API Task

In this new example, we will write Shopify data in SSIS. In addition, we will create a new order using REST API to write Shopify data in SSIS.

1. First, use the REST API task in SSDT.
2. Secondly, in the REST API use the http connection created on step 2 of the read Shopify data in SSIS.  Also, in HTTP Request Method, select POST, in Body (Request Data) write the following:
   

   {
     "order": {
       "line_items": [
         {
           "title": "Big Brown Bear Boots",
           "price": 74.99,
           "grams": "1300",
           "quantity": 3,
           "tax_lines": [
             {
               "price": 13.5,
               "rate": 0.06,
               "title": "State tax"
             }
           ]
         }
       ],
       "transactions": [
         {
           "kind": "sale",
           "status": "success",
           "amount": 238.47
         }
       ],
       "total_tax": 13.5,
       "currency": "EUR"
     }
   }

   

   Post JSON order Shopify

    

3. If you dont like to use URL supplied in HTTP Connection manager then just check override URL option and enter URL like below. For this example we used correct URL in HTTP connection manager so we will not override URL.
   

   https://{{your-store}}.myshopify.com/admin/orders.json

4. Now you can click on Test Request / Response and you will see one sample order created in your admin portal.
5. You can supply variables in Body by clicking on Insert Variable dropdown when you edit Body.

Calling Storefront API

In our previous section, we saw how to call ADMIN API which is the most common API you will use in most cases to read / write store data. However if you have a need to call Storefront API  instead then configure HTTP Connection as below

1. For Store Front API call you can follow the same steps as the previous section except for few things on HTTP Connection Manager UI
2. In API Key enter Storefront API Access Token (rather than Admin Token)
3. In the Auth Header Name enter

   X-Shopify-Storefront-Access-Token

   

   SSIS JSON Source Storefront HTTP Connection Manager

Legacy Approach – Using Basic Auth

If you created Shopify PrivateApp (before they introduced the new CustomApp approach) then your HTTP Connection needs to be like below. Use Basic Auth for Credentials Type

1. Using approaches to establish the HTTP Connection Manager 
   

   SSIS JSON Source HTTP connection

Create Multiple Shopify records using Web API Destination

With our ZS Web API destination, it is possible to create multiple records at the same time.  The following link shows the ZS Web API Destination information:

• SSIS Web API destination

Here are few useful articles to explain Web API destination use case.

How to POST data to URL

Few other articles

Delete Shopify records using REST API in SSIS

1. First, in SSDT, drag and drop the ZS REST API task.
2. Secondly, add the HTTP Url Connection.  Use the DELETE HTTP Request Method and the URL including the order id as below:
   

   https://{{your-store}}.myshopify.com/admin/orders/{{orderid}}

   

   SSIS delete data in Shopify using REST API call

    

3. Now click Test Request / Response to test delete API call.

Conclusion

In this article, we learned how to create a Shopify app. In addition, we learned how to connect to Shopify using SSIS with the ZappySys tools. Also, we learned to use ZappySys tools in SSIS to get, write and delete information.

References

Finally, for more information, refer to these links:

• ZappySys SSIS JSON Source
• ZappySys SSIS REST API task

The post Read / Write Shopify data in SSIS (REST API) appeared first on ZappySys Blog.

In this new article, we will show how to call REST API in MicroStrategy. MicroStrategy is a Business Intelligence company that provides powerful software to create nice charts, reports to take decisions. It started in 1993 and now it is one of the most popular Business Intelligence tools in the world because it is simple, intuitive and friendly. Also, It can connect to almost any database. We will use ZappySys ODBC PowerPack. Zappysys ODBC is a pretty nice software used to connect to REST API.  ZappySys ODBC is an excellent plugin for MicroStrategy, because, with this plugin, you can connect to almost any application that supports REST API.

Requirements

1. First of all, we will need MicroStrategy Desktop installed.
2. Secondly, you will require ZappySys PowerPack for ODBC installed.

Getting started

First of all, we will need to configure the ODBC ZappySys to connect to REST API. Once the ODBC connection is created, we will invoke the ODBC connection in Microstrategy.

Configuring the ODBC Zappysys

1. Search “odbc” in start menu and select ODBC Data sources (64-bit)
2. First of all, we will open the ODBC Data Source Administrator in the System DSN and press Add:
   

   ODBC Connect to REST API

   Use 64 bits if your machine supports it.
3. Secondly, add a new ZappySys JSON Driver:
   

   List of drivers supported

4. Also, we will add the following URL to the Simple View:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json

   

   ODBC configuration to connect to REST API

Configuring MicroStrategy

1. In MicroStrategy Desktop, press the New Dossier button:
   

   Create a new dossier in MicroStrategy

2. Secondly, in the Dossier, press the New Data button:
   

   Add new data in Microstrategy

3. Also, in Data Sources select Databases:
   

   Select the option to connect to Databases in Microstrategy

4. In addition, select the option to Type a Query:
   

   Create a query to get Database data

5. Next, in Data Sources press the + icon:
   

   Add Data Source in MicroStrategy

6. Finally, in Data Source, select DSN Data Sources and select the data source created in the previous section using the ZappySys ODBC Driver. In version, select Generic DBMS and write any username and password:
   

   Select DSN, version

Call REST API in MicroStrategy

1. You can write different queries using the ZappySys ODBC Driver. The driver supports SQL queries to REST API data including WHERE clauses, GROUP BY, HAVING, ORDER BY, LIMIT WITH:
   

   SELECT ShipName FROM value  
   WHERE ShipName="Around the Horn"

   

   Query to REST API using MicroStrategy

2. Secondly, you can connect online (Live) or use an in-memory Dataset:
   

   MicroStrategy connect Live

3. Finally, you can use the data for your MicroStrategy reports:
   

   MicroStrategy report created

Invoking a JSON file in MicroStrategy using ODBC

With ZappySys PowerPack, you can not only CALL REST API in Microstrategy, but you can also create queries to plain JSON files stored locally or in a web address. We will use the following JSON file to test:

1. First of all, in order to connect, you will need to specify on the ZappySys JSON Driver the path of the local JSON file:
   

   Configure a JSON file connection with ODBC

2. Secondly, you can create queries in MicroStrategy in the same way that we did in the Call REST API in MicroStrategy section:
   

   Query to JSON file in MicroStrategy

Call REST API in MicroStrategy to get Gmail information

It is also possible to invoke and query Gmail data using REST API in MicroStrategy using ZappySys ODBC PowerPack. With the ODBC PowerPack, you can write SQL queries and call REST API in MicroStrategy using OAuth or HTTP
connections.

1. First of all, in the ZappySys ODBC JSON driver, write the following URL:
   

   https://www.googleapis.com/gmail/v1/users/me/messages/

2. Secondly, in connection type select OAuth Provider.
3. Also, in the OAuth Parameter in OAuth Provider, select Google and in scope https://mail.google.com and press generate token button and specify your credentials:
   

   Configuration to connect MicroStrategy with Gmail

4. Additionally, in Zappysys you can write queries in MicroStrategy as we did with other REST API URLs:
   

   SELECT * FROM messages

   

   Query to Gmail information in MicroStrategy

Problems with the boolean data type in MicroStrategy

In addition, there is known issue related to boolean data in Microstrategy:

Error in accessing information from the external data source. Database Instance: (DataImport) Error Fetch Failed Error type: Invalid operation. Data of a string cell was truncated.

https://community.microstrategy.com/s/article/KB20514-How-to-map-a-boolean-datatype-as-integer-with-PostgreSQL

We have a workaround for this situation.

1. First of all, you will need to change the metadata. The following article shows how to change the metadata:https://zappysys.com/blog/metadata-odbc-api-drivers-query/Also, in order to fix, save your current metadata:
   

   Driver ODBC metadata

2. Additionally, modify your data from Boolean to String:
   

   MicroStrategy boolean to string metadata

REST API / SOAP API Pagination Settings for MicroStrategy

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

REST API / SOAP Web Service Connection Settings for MicroStrategy

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Other settings for REST API / SOAP XML Call in MicroStrategy

REST API / XML SOAP Performance Tips for MicroStrategy

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Calling SOAP Web Service in MicroStrategy

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

Conclusion

To conclude, we can say that the process to call REST API in MicroStrategy using ZappySys ODBC PowerPack is straightforward. With the REST API and Web service support, Microstrategy can connect to almost any source. If you liked this article and you want to try our product, you can download it here.

References

Finally, if you need more information about the process to call REST API in MicroStrategy, refer to the following links:

• ODBC PowerPack – Drivers for REST API, JSON, XML, SOAP
• Get started with MicroStrategy Desktop
• SQL Query Examples with ODBC ZappySys PowerPack

The post How to call REST API in MicroStrategy (JSON / XML SOAP) appeared first on ZappySys Blog.

In this article, we will learn how to read REST API in Pentaho Kettle. Pentaho is a very popular tool in Business Intelligence to integrate data for OLAP services and to create reports and dashboards. In this article, we will use our ZappySys ODBC PowerPack to connect to REST API in Pentaho Kettle.

Our software requires the JDBC connector for SQL Server and then Uses our ZappySys Gateway to connect.

Requirements

1. First of all, we will require the ZappySys ODBC PowerPack.
2. Secondly, we will use Pentaho Kettle (Our example in this post uses v8.1)
3. Finally, we will require the JDBC driver for SQL Server.

Getting started

First of all, we will need to create an ODBC connection to REST API using the Windows ODBC Data Source Administrator. Next, we will create a gateway and finally, we will invoke the data in Pentaho.

Create an ODBC connection to read REST API in Pentaho Kettle

In this example, we will first create an ODBC connection to an URL.

1. First of all, open the ODBC Data Source administrator in Windows in the System DSN press Add:
   

   Add new ODBC driver

2. Secondly, select the ZappySys JSON Driver:
   

   Select JSON driver in ODBC DS Administrator

3. Finally, add a data source name and specify a link to a URL. In this example, we will connect to the following url:
   

   URL: https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json

   

   REST API ODBC URL

Create a ZappySys gateway to JDBC to read REST API in Pentaho

In order to connect to REST API, ZappySys ODBC PowerPack uses the JDBC for SQL Server and a gateway. Let’s take a look at it:

1. First of all, open the ZappySys Gateway that is installed with the ODBC PowerPack and add a new data source:
   

   Add new Data Source in ZappySys Gateway

2. Secondly, add the data source name:
   

   Pentaho REST Client

3. In addition, in the user’s tab, add the user test and specify a password:
   

   Add user and login

4. Also, in settings, press edit and add the ODBC Data Source created in the previous section and add the user created before:
   

   Link Data Source in ODBC with the default gateway

5. Finally, in the General tab, you can specify the port. By default, it is 5000.
   Make sure to save the changes and check the gateway is started with the start button:
   

   Default gateway with port 5000

How to read REST API in Pentaho

1. First of all, in Pentaho, we will create a project like the following:
   

   Pentaho Project REST API csv

2. Secondly, we will create a query like the following:
   

   A query in Pentaho to get data

3. In addition, we will add the following connection. We will connect to the data source named customers created in the ZappySys Gateway with port 5000 and the user created in the Gateway:
   

   JDBC connection to SQL Server

4. If everything is OK, you will be able to see the CSV file generated:
   

   Data in CSV format

5. Also, you can use SQL statements. The following example shows how to do a where filter:
   

   SELECT Shipname 
   FROM value
   where Shipname='Consolidated Holdings'

6. Finally, we will show the first 10 rows using a group by:
   

   SELECT Shipname 
   FROM value
   group by shipname
   limit 10

How to read REST API in Pentaho (Gmail)

The following example will show how to get Gmail information from Pentaho.

1. First of all, we will need to create a connection to Gmail in ODBC using the ZappySys ODBC Driver (Check the Create an ODBC connection for a Pentaho REST Client section). In the ODBC Administrator, we will add a ZappySys JSON Driver connection.
2. Secondly, in the ZappySys ODBC driver, specify the URL and select the OAuth connection:
   

   ODBC configuration

3. Also, in the OAuth properties specify the Google OAuth provider and the https://mail.google.com scope. Press the generate token button and specify your credentials and accept the access permission request:
   

   Gmail OAuth connection

4. In addition, using the ZappysSys Data Gateway, add the new Data Source and associate with the ODBC connection just created:
   

   Connect Pentaho with Gmail

5. Additionally, add the test user to the data source:
   

   Include user and permission to the gateway

6. In Pentaho, change the connection in the table input to connect to the Gateway:
   

   Pentaho Rest client to Gmail

7. Finally, in the table input add the following query:
   

   Connect Pentaho to Gmail

Conclusions

To conclude, we can say that it is simple to connect to the REST API using Pentaho with the help of ZappySys ODBC PowerPack. This tool allows to easily query REST API data like a simple SQL Query. If you like this tool you can
download the ODBC PowerPack here.

References

For more information, refer to the following links:

• ZappySys ODBC PowerPack
• Introduction To Pentaho

The post How to read REST API in Pentaho Kettle appeared first on ZappySys Blog.

Using REST API with SharePoint in SSIS is a common requirement to Administer SharePoint. SharePoint 365 is a nice Microsoft application in the Cloud used to share documents and collaborate with the company. You can have schedules, projects, documents and more shared using a Web platform.

In addition, it is a very useful tool that your company may need. It is also possible to automate, administer task using Rest API.
This article, we will learn how to create REST API queries to SharePoint 365 using the Microsoft Graph API using SSIS. To learn about other Office 365 REST API in SSIS check this article (Outlook, Onedrive, mail, Excel API).

Also, we will show our ZappySys SSIS PowerPack that includes very powerful tools to export SharePoint data from REST API to any other source.

	SSIS REST API Web Service Task
	SSIS JSON Source (File, REST API, OData Connector)
	SSIS JSON File Destination (Create JSON File)

Requirements to REST API with SharePoint

1. First, you will need SSDT installed.
2. Second, SSIS PowerPack installed.
3. An Office 365 Account with SharePoint 365

Getting started

Microsoft included the Microsoft Graph to easily create API queries to different Microsoft components like OneDrive, Excel, Insights, Microsoft Team, Planner and more.

The Microsoft Graph is a gateway to access to 365 Data in an integrated platform. Let’s start with the Microsoft Graph Explorer and do some REST API queries to understand how to use REST API in SharePoint.

Microsoft Graph Explorer to SharePoint introduction

The best way to learn REST API with SharePoint 365 is to use the Graph Explorer. You can run API queries using the Graph Explorer and get your data. Here we will show how.

1. First, we will go to the Graph Explorer and sign-in with your Office 365 Account.
2. Secondly, in Run Query, you can specify your query, which will show SharePoint information.
   

   Graph Explorer

3. Let start with this link:
   

   https://graph.microsoft.com/v1.0/sites/root/lists

4. This shows properties about site resources. The response of the query is in JSON format:
   

   Graph API Microsoft JSON results

5. Also, it is possible to enable examples for SharePoint Sites, SharePoint Lists, and other Microsoft technologies:
   

   Graph API Applications

Create an OAuth connection to REST API with SharePoint in SSIS

1. 1. First, to get the Data from the Microsoft Graph, we will sign in to create an application in the Azure Portal app registration.
   2. Secondly, create a Microsoft Application. It will provide you an Application ID and a Password. For detailed steps about how to create a Microsoft Application, check our article related.
      

      Microsoft Graph API application id password

   3. Once that the application is created, go to SSDT or Visual Studio and create a new connection.
      

      SSIS OAuth connection

   4. Select the ZS-OAuth connection. Once selected, enter the Client ID (the Microsoft Graph Application ID), the client secret (application password).
   5. Add the Authorization URL is the following:
      

      https://login.microsoftonline.com/common/oauth2/v2.0/authorize

   6. Also, include the Access token URL is the following:
      

      https://login.microsoftonline.com/common/oauth2/v2.0/token

   7. The following permissions were applied in the example: User.Read
      

      offline_access
      Sites.ReadWrite.all
      Sites.Manage.All
      Sites.FullControl.All

      NOTE: It is advised to include offline_access scope in order to get a Refresh Token and to be free of any token restrictions.
   8. Once that the properties are entered, press the Generate token button:
      

      SSIS OAuth with REST API with SharePoint in SSIS

How to call SharePoint API in SSIS

1. Once that your OAuth Connection is created, we can use it in a REST API task. This task is included with the ZappySys SSIS PowerPack:
   

   SSIS REST API task

2. Secondly, in the REST API Task, select the option URL from connection and select the OAuth connection created before. We will first use the following URL to see the list of site properties:
   

   https://graph.microsoft.com/v1.0/sites/root/lists

3. Also, press Test request/Response button to get the results of the query:
   

   SSIS SharePoint rest API call

4. The response will be in JSON format:
   

   REST API SSIS Sharepoint 365 response

5. Also, it is possible to store the results in a file or a variable:
   

   SSIS save API SharePoint response to a file REST API with SharePoint

6. In Addition, you can run the following query to enumerate all the sites:
   

   https://graph.microsoft.com/beta/sites?select=siteCollection,webUrl&filter=siteCollection/root%20ne%20null

7. Also, to show the information on a specific site, the following query will be used:
   

   https://graph.microsoft.com/v1.0/sites/site-id

8. For example, if you site id is zappysys.sharepoint.com the URL would be the following:
   

   https://graph.microsoft.com/v1.0/sites/zappysys.sharepoint.com

9. To enumerate sites, the following example can be useful:
   

   https://graph.microsoft.com/beta/sites?select=siteCollection,webUrl&filter=siteCollection/root%20ne%20null

10. Also, to enumerate lists, you can use the following URL:
    

    https://graph.microsoft.com/beta/sites/zappysys.sharepoint.com/lists

How to read SharePoint lists in SSIS using REST API

We will learn how to show SharePoint lists using rest API.

In addition, you can get the list ids using this query:

https://graph.microsoft.com/beta/sites/zappysys.sharepoint.com/list

SharePoint list API

How to read data from a Sharepoint List

1. To get the list of ithems form a list use the following query:
   

   https://graph.microsoft.com/beta/sites/zappysys.sharepoint.com/lists/mylist/items/

2. Finally, to get metadata from a list, you can use the following query. To learn more how to import Sharepoint List into SQL Server use JSON / REST API Source.

https://graph.microsoft.com/beta/sites/zappysys.sharepoint.com/lists/fcf5fa2b-ceeb-4c9b-80e5-7bedd455fc9b

SSIS Sharepoint metadata list

How to create a SharePoint list item in SSIS using REST API (POST Method)

It is also possible to POST data using REST API. In the next example, we will create a new list in SharePoint 365 using REST API.

1. First, in SSDT, use the REST API Task used previously and specify the following URL:
   

   https://graph.microsoft.com/beta/sites/zappysys.sharepoint.com/lists/mynewlist/items/

   Where zappys.sharepoint.com is the site id and mynewlist is a new list created. Also, in the HTTP Request Method, use the POST method.
2. In addition, go to the Body Request, we will create a widget. We will use the following JSON data and make sure to select JSON (application/json) in the body:
   

   {
   "fields": {
   "Title": "Widget"
   }
   }

   

   SharePoint Post REST API

3. Finally, if everything is successful, you will be able to see the new item created:
   

   SSIS SharePoint list created with a POST

How to export SharePoint API response results to a file

We have a nice task named JSON Source used to get data from REST API or a JSON file and export to any source the results like SQL Server, Excel, a flat file, etc.
In this example, we will export data from the JSON Source to a JSON file using the JSON Destination Task.

1. First of all, we will use the Data Flow. Drag and drop the data flow to the design pane and double click to go to the Data Flow tab:
   

   SSIS Data Flow

2. Secondly, in the Data Flow tab, drag and drop the JSON Source, the JSON Destination and join them:
   

   SSIS JSON source to JSON destination

3. In addition, in the JSON Source, we will check the option Use Credentials and select the OAuth connection created and used before.
4. Enter Path or Web URL, we will use the following URL:
   

   https://graph.microsoft.com/beta/sites/zappysys.sharepoint.com/lists

5. Also, We will use the following filter:
   

   $.value[*].list

   

   SSIS JSON Source REST API with SharePoint in SSIS

6. In addition, go to JSON Destination task, select mappings and right click on add elements:
   

   SSIS JSON Destination

7. Also, select the attributes:
   

   SSIS add attributes

8. Finally, run the package to export the data.

Conclusion about REST API with SharePoint in SSIS

To conclude, we can say that SharePoint 365 is a very popular application to share documents and information. With REST API it is possible to get data properties and administer SharePoint. We used the REST API, JSON Source to get and send information. If you want to try yourself, you can download our tools here.

References

• Get access on behalf of a user
• Working with SharePoint sites in Microsoft Graph
• About SSIS PowerPack

The post Calling SharePoint 365 REST API in SSIS appeared first on ZappySys Blog.

In this example, we will show how to read emails by calling Gmail REST API using SSIS. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more.

Requirements

1. First, you will need ZappySys SSIS PowerPack installed.
2. Secondly, you will need SSDT installed.

Getting started

How to get information about Gmail with REST API

1. First, you will need to create a developer application for Gmail. The following tutorial shows how to create an application for Google Drive, but the steps are similar:
   • How to register Google OAuth Application (Get ClientID and ClientSecret)
2. Secondly, go to SSDT and create a new SQL Server Integration Project.
3. Subsequently, drag and drop a Data Flow task and double-click it:
   

   Drag and drop Data Flow

4. Also, in the connection manager create a new ZS-OAuth connection. Select Google as the OAuth provider. In scopes, add the following:
   

   https://mail.google.com/

   In addition, press generate token and add your credentials if necessary:
   

   OAuth credentials to connect to Gmail

    

5. To illustrate, the following example shows how to get the Gmail messages from an account. Check the option use credentials and select the OAuth connection just created. To get your current email messages, you can use the following URL:

   https://www.googleapis.com/gmail/v1/users/me/messages/

   The filter used is the following:

   $.messages[*].id

   The result displayed shows the ids of the messages. You can see the id of the message in Gmail in the URL when you click a message:

   

   JSON Source Rest API

    

6. If you want to check to information of a specific message you can use the following URL.

   https://www.googleapis.com/gmail/v1/users/me/messages/1632106341364d6f

   Where 1632106341364d6f is the message id. The filter used will be the following:

   $.id

   

   Gmail message information

   The query shows the snippet (body of the message), the history id, size estimated and more.

7. After this, we can use the following filter:

   $.payload.headers[*]

   This will show the sender, the receiver, subject, references, message ID and more information:

   

   Information about Gmail

How to get label information in Gmail with REST API

1. First, you can use the following URL to get the Gmail labels:
   

   https://www.googleapis.com/gmail/v1/users/me/labels/

   Filter:
   

   $.labels[*].id

    
2. Also, the information displayed is the following:
   

   Labels in Gmail

3. Finally, the labels are the same than in Gmail:
   

   Gmails labels, categories

How to get threads information in Gmail with REST API

1. Finally, you can also get the thread information using the following URL:
   

   https://www.googleapis.com/gmail/v1/users/me/threads

   The filter used will be the following:
   

   $.threads[*].id

    
2. The information displayed will be the following:
   

   Get Threads from Gmail

3. If you press the Preview button, you will see the following information:
   

   Thread information using Gmail

How to export the data from Gmail to a CSV file

1. First, to export the data, we will use the ZS Trash Destination:
   

   Export data from Gmail to a CSV file

2. Finally, in the Trash Destination specify the path to store the data from Gmail:
   

   CSV destination to store Gmail info

Conclusion

To conclude, we can say that ZappySys SSIS PowerPack is a powerful tool where you can get information from Gmail. This information can be exported to Excel, SQL Server, XML, JSON or any other format using SSIS.  Download FREE Trial of SSIS PowerPack to find out more use cases like this.

References

For more information about Gmail, you can refer to these links:

• API Reference
• How to read / write Google SpreadSheet using SSIS

The post Calling Gmail REST API using SSIS – Read Emails appeared first on ZappySys Blog.