In this article, you will learn how to create simple or complex JSON file in Informatica just in a few clicks. We will use the Powerful Export JSON Command line Builder from ZappyShell. Using this Portable / xCopy style command line you can build nested/complex JSON / XML or even CSV from single / multiple SQL Queries as Input or Select Database Tables as Source. Later you can copy generated command line from Tool and execute in any tool like Informatica / Talend / DataStage or Automation Script like Python / PowerShell.

Prerequisite

This article assumes you have access to Informatica Developer Tools and basic knowledge of command-line tools.

1. Download ZappyShell Command line utility (Click on Download Button). You can get msi and Zip (Poratble / xCopy version)
2. If you want to use the Portable version just extract ZappyShell-xCopy zip to some folder lets say (C:\ZappyShell). If you installed MSI then its usually in C:\Program Files (x86)\ZappySys
3. Double click on DbCommandBuilder.exe as listed here  to get started. At the end of this exercise, you will be able to generate something like the below JSON File in few clicks in Informatica or any other Tool / Programming Language you prefer which has option to invoke command line…
4. For all possible command line options Refer this
5. So lets get started

Sample JSON Created by ZappyShell from Informatica

Sample JSON Created by Informatica Workflow (Create From Database SQL Query or Tables)

Create Command Line to Export JSON File from Database SQL Query / Tables

In this section, we will see how to build Command Line which you can then execute in Informatica Command Time and later Schedule Export Job to Produce JSON File on a predefined schedule or manually.

1. Open DbCommandBuilder.exe and configure the Connection / Data source(s) as listed here .
2. Select JSON Export Option from the Dropdown. You can also choose XML, Excel or CSV
3. You can use the Custom Layout Option if you wish to Export using custom Output Layout else just choose tables you like to export or type SQL query.
   
4. Configure your Input Dataset(s) (can be SQL Queries or Tables) and define Desired Layout for your Output File.
5. Here is just an example of how to define an Array for a nested layout in a JSON or XML file.
6. t the end of configuration you may see command line like below (more examples listed here)Example: Simple mode Export (Microsoft SQL Server)
   

   db.exe export customers --json -o c:\data\cust.json --connstr "Data Source=localhost;Initial Catalog=Northwind;Integrated Security=SSPI;" -y

   Example: For Simple mode Export (MySQL SQL Server – ODBC Driver)
   

   export customers --out c:\data\customers.json --json --overwrite --connstr "Driver={MySQL ODBC 5.2 ANSI Driver};Server=your_server_name_or_ip;Port=3306;Uid=user1;pwd=SomeP@ssword;Database=your_db_name"

   Example: For custom layout mode Export
   
   

   export --map c:\data\MySampleMap.xml -o c:\data\customers.json --json --connstr "Data Source=localhost;Initial Catalog=Northwind;Integrated Security=SSPI;" -y

7. Once the Command line is created you can click Run to test and then copy it into tools like Informatica Command Task (see next section).

Export / Create JSON File In Informatica

1. Once the command line is prepared (see previous section) we can open Informatica Developer (D)
2. Connect to your Repository and create a new workflow
3. Drag new Command Task, from Toolbox and double click to edit
   

   Informatica Command Task

4. Enter the sample command line we obtained from the previous step
5. Save the workflow and run it… That’s it you will see the JSON file generated

Export / Create XML File In Informatica

Same Steps as above except choose File format as XML and you will see output like the below once you run the command line.

Export / Create XML File In Informatica

Export / Create CSV File In Informatica

Same Steps as above except choose File format as CSV and you will see output like the below once you run the command line.

Generate CSV Files in Informatica

Export / Create Excel File In Informatica

Same Steps as above except choose File format as Excel and you will see output like the below once you run the command line.

Conclusion

In this post, we just saw how easy it is to generate JSON / XML / Excel or CSV files in tools like Informatica, Talend, python, PowerShell in few clicks without coding. You can schedule job to generate JSON / XML files from Database tables or SQL Query. Download ZappyShell Today and try it to see what else you can do with it.

The post Export / Create JSON File in Informatica (from SQL Query / Database Tables) appeared first on ZappySys Blog.

In our previous article, we saw how to get Office 365 data in Power BI. Now, let’s look at how to get Office 365 Mail Attachments using SSIS without any coding. Microsoft Graph API is a unified way to access many Microsoft services, including the Office 365 API. In this post, we will use the SSIS JSON/REST API Source Connector to retrieve the Outlook Messages list and its attachments.

Use Case of Microsoft Graph API

Here are some use cases for why you want to use the Microsoft Graph API

• Read/Write events from Outlook Calendar
• Send email  / Read emails
• Get the list of files from OneDrive
• Upload/ Download files to OneDrive
• Read/Write Excel Sheet (Use range or a specific range of a specific sheet)
• Read the SharePoint Document library
• Search content from the SharePoint document library
• Get AD users for your Organization (AD Accounts)
• Update OneNote
• Track changes to Users, Events, Calendar Items using the delta API

Step-By-Step to call the Microsoft Graph Rest API Using SSIS

Now, let’s look at examples to access your Outlook.com emails using SSIS. You can use the same techniques to call pretty much any Office 365 API.

Configure SSIS OAuth Connection for Graph REST API

The first step to access any Graph API is to configure an OAuth connection

1. Open Visual Studio
2. Open an existing SSIS Project or create a new one using File > New > Project > Choose “Integration Services Project” Type under Business Intelligence template category.
3. Create Package
4. Right-click inside the Connection Managers area and click New Connection…
5. From the connection type list, select the ZS-OAUTH connection type.
   

   Create a new SSIS OAuth API Connection Manager

6. On the OAuth Connection Manager, configure the following options
   1. Select “Custom” from the Provider dropdown
   2. Select OAuth2 from the OAuth version
   3. Enter your Application Id and Secret Key (i.e., App Password) obtained in the previous section from here.
   4. Enter the following URL in the Authorization URL
      

      https://login.microsoftonline.com/common/oauth2/v2.0/authorize

   5. Use the following URL in the token URL field
      

      https://login.microsoftonline.com/common/oauth2/v2.0/token

   6. Enter the following Graph API Scopes (each scope must be entered on a new line).
      

      user.read
      mail.read
      offline_access

      NOTE: Always include the offline_access scope, which returns a refresh_token, which allows you to renew the token without going through the login process again. For more information about which scopes are needed, always refer to the API help page (e.g., if you want to read mail using this API, then you will see it says you can use Mail.Read or Mail.ReadWrite scope).
   7. Go to the Advanced tab and enter the following URL in the Callback/Return URL (assuming the same URL used when you registered the App in the previous section)
      

      https://zappysys.com/oauth

   8. Now go back to the first tab and click Generate Token. When prompted, log in using your Personal Microsoft Account or Work Account (Office 365 or AD login)
   9. After logging in, you will see the Accept option. Just click it. If things go right, then you will see the Access Token and Refresh Token fields populated.
7. Once everything is configured, you can click Test Connection to verify the connection works.
   

   SSIS OAuth Connection Manager for Office 365 REST API Microsoft Graph API

Download the Messages Attachment Files from Outlook

Let’s start with an example. We use the SSIS JSON Source and Web API Destination components to make a call to the Microsoft Graph REST API.

1. First of all, drag and drop the Data Flow Task from the SSIS Toolbox and double-click it to edit.
   

   Drag and drop Data flow

2. From the SSIS toolbox, drag and drop the JSON Source onto the Data Flow Designer surface.
   

   SSIS JSON Source – Drag and Drop

3. Double-click the JSON Source, then enter the following URL to invoke the Messages API call with search options to retrieve emails with attachments.
   

   https://graph.microsoft.com/v1.0/me/messages?$select=sentDateTime,from,subject,hasAttachments&$search="hasAttachments:True"

   Set the filter to $.value[*] and click the preview.
   

   SSIS JSON Source – Get data from Microsoft Graph API (Office 365 API – Read Mail Example)

   To customize the URL with additional parameters, check this help link. It’s a standard OData Protocol, so you can leverage many common OData features (e.g., use $top parameters to set pageSize. Default is 10 for List Mail, but you can include more records per response by changing it, such as …/messages?$top=50).

4.  Now, let’s use Template Transform to set the URL using the message ID to make the Get attachment API call.

   https://graph.microsoft.com/v1.0/me/messages/<%id%>/attachments

   

   SSIS Template Transform sending attachment id
5. Now we need to use the Web API Destination to make the Get Attachment API call by message ID. Select the Input column for URL as that TemplateOutput.
   

   SSIS Web API Destination – Get data from Microsoft Graph API (Office 365 API – Get Attachment Example)

6. Now, the next step is to parse the JSON Response String of the get attachment api call. For that, we need to use a JSON Parser. Select the input as Web PAI Destination ResponseText and use the below sample JSON string to configure the metadata(Columns).
   

   {"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#users('support%40zappysys.com')/messages('AQMkADAwATMwMAItMDkyMC0zZTg4AC0wMAItMDAKAEYAAAP_JUy')/attachments"
   ,"value":[{"@odata.type":"#microsoft.graph.fileAttachment","@odata.mediaContentType":"text/plain"
   ,"id":"AQMkADAwATMwMAItMDkyMC0zZTg4AC0wMAItMDAKAEYAAAP_JUy_27M"
   ,"lastModifiedDateTime":"2020-02-21T07:36:16Z"
   ,"name":"Test.txt","contentType":"text/plain","size":870,"isInline":false
   ,"contentId":null,"contentLocation":null,"contentBytes":"VGVzdCBNZXNzYWdl"}]}

   

   JSON Parser Transform – using JSON example
7.  Now, in the JSON Parser, go to the Go-to columns Tab and set the contentBytes DataType to DT_IMAGE, and increase the other columns’ DT_WSTR lengths by 1000 and 1500. We set contentBytes to the DT_IMAGE  datatype, which is like varbinary(MAX).
   

   JSON Parser Transform Columns Tab fix the datatype

8. Furthermore, in the next step, we need to use an SSIS Derived Column to set the FilePath where we want to save the attachments, and then concatenate the local file path with the attachment name column.
   

   SSIS Derived Column: Add New Column

9. Finally, we need to use the Export Column Transform (Native) to export attachment bytes to a local file. It needs to know 2 upstream info Bytes and the target file path.
   

   Export Column configuration

10. Make sure to attach the export column to some destination (e.g., our ZS Trash Destination); the engine might remove it. At the same time, it optimizes the runtime. It may remove all transformations that don’t have a destination, such as deploying the package to SQL Server, or when you set optimize in Visual Studio.
11. That’s it, execute the package, and it will download all the email attachments.
    

    Downloaded Mail Attachments

Conclusion

After all, in this article, we learned how to make a Microsoft Graph API REST API call. We used the SSIS JSON / REST API Connector to extract data from Outlook Mail using OAuth. Made the Get attachment API call using the Web API Destination and parsed the JSON response using the JSON Parser. And using the Native SSIS Export Column, we saved the export attachment’s bytes to a local file. To explore many other scenarios not discussed in this article, download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • Help File: Web API Destination
  • Help File: JSON Parser Transform
  • Overview of the Microsoft Graph API

The post How to Get Office 365 Mail Attachments using SSIS appeared first on ZappySys Blog.

In our previous article, we see how to load data into Elastic Search using SSIS. Now let’s look at how to read data from ElasticSearch using SSIS and load response into SQL Server.

Elasticsearch is a powerful engine that allows you to store, aggregate, and, most importantly, search data in a very analytical way. In this tutorial, you will learn how to load Elasticsearch data to SQL Server with SSIS (part of SQL Server) and ZappySys PowerPack. We also see how to set the maximum result window of the index using the Rest API Task.

In a nutshell, this post will focus on how to Make Elasticsearch Search API call using SSIS.

Step-By-Step to Call Elasticsearch API call using SSIS

Read from Elasticsearch Search API call

Let’s make the Elasticsearch Search API call using the JSON source to get all the records of the index with pagination.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Dragging and dropping Data Flow Task into Control Flow

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to make Search API Call on the index.
   

   http://{elasticsearch-site-url}/{index}/{index-type}/_search

4. Now check Use credentials and Select Connection section press

   <New ZS-HTTP Connection>

   .
   

   SSIS JSON Source – Passing Basic Credentials (Supply UserID / Password) using HTTP Connection

5. Once the HTTP Connection Manager window opens configure the connection to your Elasticsearch instance:
   • Set Web Url, which points to your Elasticsearch instance.
   • Set Credentials Type to

     Basic - UserID/Password

      (or other appropriate authentication methods).
   • Finally, set User Name and Password:
     
     

     Configure SSIS HTTP Connection to connect to Elasticsearch

6. Now, select method as POST and set for and size value in the body to get data using pagination. Set the content type as JSON (application/json).
   

   {"from" : <%page%>, "size" : 100}

7. Furthermore, go to the pagination tab and configure pagination like the below screenshot makes sure we need to enter increment size the same as given in body size value therefor 100.
   

   Pagination Mode: POST data in the body

8. That’s it now select the desired filter and click on the Preview button.
9. That’s it we are ready to load the data in the SQL Server.

For more than 10000 rows

When you try to get more than 100000 rows you will get the next error message

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [11000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
            }
        ]
    },
    "status": 400
}

Here is how to configure your JSON Source to pull more than 10K rows using Scroll Method.

1. In Settings Tab: In the Body and URL rather than Index name use

   [$tag$]

2. In the filter type:

   $.hits.hits[*]._source

3. Uncheck Include Parent
   

   Elasticsearch example for more than 10000 rows

4. In Pagination Tab, Set Next Link as below

   $._scroll_id

5. Stop Indicator Attribute as below

   $.hits.hits[0]._id

6. Stop Indicator Value as

   regex=^$

   

   Elasticsearch pagination tab configuration

7. In Advanced Pagination Tab, Check Has Different Page Info and Enable Page Token
8. Enter Page Place Holders as below (change YOUR_INDEX_NAME)
   

   url=YOUR_INDEX_NAME/_search?scroll=10m|_search/scroll

9. First Page Body as below – Change query as needed else keep it match all to fetch all rows from Index
   

   { "size":1000, "query": { "match_all": { }  } }

10. Next Page Body as below 

    {  "scroll": "5m", "scroll_id": "[$pagetoken$]" }

    

    Configuration for Elasticsearch advanced pagination tab

Loading Elasticsearch API data into SQL Server / Other Target

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Enable Deep Pagination Setting

By default index, the maximum result window size is 10,000 means you can paginate upto 10000 rows. If you ever want to get more than 10000 result then you have to perform the following steps to change default setting from 10000 to 500000 (that’s max).

We need to send PUT request to below URL:

http://{elasticsearch-site-url}/{your-index}/_settings

Need to pass below body as I am setting max result window size value as 500000.

{ "index" : { "max_result_window" : 500000 } }

And set content-type as JSON (application/json) and click on Test Request/Response and it will make the API call and set the max window size.

Rest API Task PUT Call

That’s it we have successfully made the call to set the maximum result window size of the index.

Common Errors

Deployment to Production

Conclusion

After all, we saw you How to Make Elasticsearch Search API call using SSIS JSON Source and load response into SQL Server. In this article, we have seen that how to make a setting call to set the maximum result window of the index. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• Help File: JSON Source(REST API or File)
• Elasticsearch Search APIs

The post How to read data from ElasticSearch using SSIS appeared first on ZappySys Blog.

In this tutorial, we will use the REST API Task to call some API. We will use the OAuth connection along the REST API Task to write / update data to Google Spreadsheet.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Google API.

Component Mention in this article

	JSON Generator Transform
	Web API Destination
	REST API Task

Create a Google API Project

The first step to access any Google API is to create an API Project in Google Console. If you don’t want to go through this then Skip this Step-1 and in the next section select the Default OAuth App option on OAth Connection Manager (This is the easiest option for now unless you want to use your own OAuth App).

Check this article for step-by-step instructions. When you follow these instructions make sure you enable Google Sheet API.

Create OAuth Connection Manager in SSIS

Once you create a Google API project and obtained a Client ID and Client Secret your next step is to create OAuth Connection Manager in SSIS. ZappySys OAuth connection manager comes with many predefined OAuth Providers (e.g. Facebook, Twitter, Google, etc) but you can also define custom OAuth settings for any OAuth enabled API.

To create SSIS OAuth 2.0 Connection for Google API perform the following steps.

1. Download and Install SSIS PowerPack.
2. Create a new SSIS Package.
3. Right-click in Connection Manager Area and Click “New Connection”.
4. When prompted select ZS-OAUTH connection type, On the OAuth Connection Manager Select Provider=Google.
5. Click here to create credentials. Then select the “Use Custom OAuth App” option and enter your ClientID, ClientSecret.
6. In the Scopes enter below scopes (or click Select Scopes). This will allow read/write access to Drive Files or Sheet (E.g. Reading file content / Export file to CSV). You can use the Scope browser to see many other available permissions. For more information click here.
   

   https://www.googleapis.com/auth/drive
   https://www.googleapis.com/auth/drive.file
   https://www.googleapis.com/auth/drive.readonly
   https://www.googleapis.com/auth/spreadsheets
   https://www.googleapis.com/auth/spreadsheets.readonly

7. Click on the Generate Token button.
   

   SSIS OAuth Connection Manager – Access Google Drive API using OAuth 2.0

8. You will see UI as below… Click Allow (You may have to scroll to see that button sometimes).
9. Click on Test Connection to see the connection is working and click on the OK button to save configure setting UI.

Find Google Sheet ID for API calls

Before we can write or update Sheet data in SSIS we need to know Google Driver File ID. We will use this ID in the next few sections.

The easiest way to find File ID is to look at the URL like below. If you need to know ID at runtime dynamically then refer to the last section.

Get Google Drive File ID for API call (Sheet ID)

Create Variables to make dynamic URL

1. Here, we are going to make a Dynamic URL value.
2. In Visual Studio Project Right Click on Design Panel, Select Variables.
   

   Create Variables

3. Let’s Create Variables with correct DataType and Value. In variable “varRange” set only the starting point(Sheet1!A2) of data if you don’t know the ending point and leave the first-row for giving header row later or click here.
   

Write SQL Server data to Google Spreadsheet

Now let’s look at a real-world example. How to write data from SQL Server table or other source and send to Google Sheet using Web API Destination.

1. Drag and Drop SSIS Data Flow Task from SSIS Toolbox.
   

   Dragging and dropping Data Flow Task into Control Flow

2. Double click on the Data Flow task to see the Data Flow designer surface.
3. Here, In Visual Studio, drag and drop the OLE DB Source, JSON Generator Transform and Web API Destination in the design surface and join the components with the blue arrow.
   

   Drag and Drop OLE DB Source, JSON Generator Transform, and Web API Destination

4. Here, you can use our Free ZS Dummy Data Source too. Click here for more information.
5. But, we are going through OLE DB Source so we need OLEDB Connection.

Create OLE DB Connection

1. Lets, create the OLE DB Connection.
2. Let’s, Right-click on Connection Managers Panel to Create OLEDB Connection, so you can use Source and Context Menu will appear, Select New OLEDB Connection from the Context Menu.
   

   Create OLE DB Connection

3. Now, click on New Button to create Connection.
   

   Create New Connection

4. Let’s Configure Connection Manager, just Follow steps one by one as we have created.
   

   Configure OLE DB Connection

5. Click on Test Connection to see correct configure it.
6. Click on the OK button to save connection configure setting UI.

Configure JSON Generator Transform

1. Double click on JSON Generator Transform to configure it.
2. Set Output Mode to Single Dataset Array. Right-click on Mappings, select Add Static Element then enter Name: “range” and Select variable we have created for value.
   

   Configure JSON Generator Transform – Add Static Value

3. Again, Right-click on Mappings, select Add Static Element then enter Name: “majorDimension” and Value: “ROWS“.
   

   Configure JSON Generator Transform – Add Static Value

4. Now, Right-click on Mappings, select Add Document Array, Check to Treat as 2D array option and enter Element Name as “values“.
   

   Configure JSON Generator Transform – Add Document Array

5. Then, right-click on Document Array, select Add elements > Select Add Multiple Columns and check on columns you like to add.
   

   Configure JSON Generator Transform – Add Element

6. Click on the OK button to save JSON Generator Transform configure setting UI.

Configure Web API Destination

1. Double click on Web API Destination to configure it.
2. Select the OAuth connection manager created in an earlier section (Assuming you had all scopes with write permission).
3. In the Setting tab,
   1. In URL, Generate Dynamic URL like below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4:append?valueInputOption=USER_ENTERED

   2. Change Request Method to POST.
   3. Select JSON Generator variable into Body (Request Data) or you can select Input column for Body (JSON Generator Output).
   4. Set application/JSON to Body Content-type.
      

      Configure Web API Destination

4. Click on the OK button to save Web API Destination configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet. Now update header row values for click here.
   

   Run or Execute Package and verify data into Google Spreadsheet

Append data to Google SpreadSheet using REST API Task

Now let’s look at how to write to Google SpreadSheet. To write multiple rows in a single request. Also, check this link to learn more.

For more information, you can see the example here. For parameter, detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve the Spreadsheet write scenario. For a simple example, let’s use the REST API Task.

1. Drag and Drop ZS REST API Task on the control flow designer.
   

   Drag and Drop REST API Task

2. Double click on ZS REST API Task to configure it.
3. In the Request Settings tab,
   1. set Request URL Access Mode to Url from a connection. Select OAuth Connection from the dropdown (Same connection we created in the previous section).
   2. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A1:append?valueInputOption=USER_ENTERED

   3. Change Request Method to POST.
   4. Enter Request Body as below
      
      Note: For example purpose, we included header line in data cells but in the real-world header may already exist in the sheet.

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["Item", "Cost", "Stocked", "Ship Date"],
          ["Wheel", "$20.50", "4", "3/1/2016"],
          ["Door", "$15", "2", "3/15/2016"],
          ["Engine", "$100", "1", "3/20/2016"],
          ["Totals", "=SUM(B2:B4)", "=SUM(C2:C4)", "=MAX(D2:D4)"]
        ]
      }

   5. Change Request Content type to application/JSON.
      

      Write data to Google SpreadSheet using REST API Task (OAuth 2.0 Connection)

4. Click on the OK button to save REST API Task configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.
   

   Execute REST API Task and verify Google Spreadsheet data

Update Google Sheet Cells

Now let’s look at the below example of updating multiple cells in google Spreadsheet. For more information, you can see the example here. For parameter, detail review this API documentation.

You can use SSIS REST API Task or Web API Destination to achieve the Spreadsheet Update scenario. For a simple example, let’s use the REST API Task.

Update Google Sheet header cells manually

1. Drag and drop ZS REST API Tasks on the control flow designer.
2. Double click on the First REST API Task to configure it.
3. In the Request Settings tab,
   1. Enter the URL below. Change Range designator(including Sheet name: Sheet1!A1) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A1?valueInputOption=USER_ENTERED

   2. Change Request Method to PUT.
   3. Enter Request Body as below
      

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["CustomerID", "CustomerName"]
        ]
      }

4. Click on the OK button to save REST API Task configure setting UI.
5. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.
   

Single Range (Set Values for multiple cells)

1. Follow the previous section(write data to Google Spreadsheet) to update sheet data.
2. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4?valueInputOption=USER_ENTERED

   2. Change Request Method to PUT.
   3. Enter Request Body as below
      

      {
        "range": "{{User::varRange}}",
        "majorDimension": "ROWS",
        "values": [
          ["Laptop", "$2000.50"],
          ["Mobile", "$1150"]
        ]
      }

3. That’s all, click on the Test Request/Response button and verify data into Google spreadsheet.
   

   Calling Google Sheet API to Update Multiple Cell Values

Clear data to Google SpreadSheet using REST API Task

Clear a sheet of all values while preserving formats

Lets clear a sheet of all values by REST API Task, Before processed we need sheet Id. So let’s get the SheetId by API Call. For more information about click here.

1. Drag and Drop two ZS REST API Tasks on the control flow designer and rename(Get Sheet ID and Clear Google Sheet of all values) it.
2. Double click on the First REST API(Get Sheet ID) Task to configure it.
3. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc

   2. Change Request Method to GET.
      REST API Task – Get Google Sheet ID
4. Now go to Response Settings Tab, Select Response content Type Json and Write Response Content Filter Expression. Check on Save Response Content. Select Save Mode to Variable and Create a New variable or select it.
   Example of Response Content Filter Expression for the first Sheet : 
   

   $.sheets.[0].properties.sheetId

   

   REST API Task – Save Google Sheet ID into Variable

5. Now, click on the Test Request/Response to check configures are ok.
6. Click on the OK button to save REST API Task configure setting UI.
7. Double click on the second REST API(Clear Google Sheet of all values) Task to configure it.
8. In the Request Settings tab,
   1. Enter the URL below. Change Sheet ID and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}:batchUpdate

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc:batchUpdate

   2. Change Request Method to POST. Enter Request Body as below and change a variable parameter as per your need.
      

      {
        "requests": [
          {
            "updateCells": {
              "range": {
                "sheetId": {{User::varGoogleSheetID}}
              },
              "fields": "userEnteredValue"
            }
          }
        ]
      }

   3. Change Request Content type to application/JSON.
      
      

      REST API Task Configure – Clear sheet of all values

9. Click on the OK button to save REST API Task configure setting UI.
10. That’s all, You are ready to Run or Execute Task and verify data into Google spreadsheet.

Clears range cell values from a spreadsheet

1. Double click on REST API Task to configure it. Then, follow the way just like the previous section configures REST API Task.
2. Here you need to set the value of Variable “varSpreadsheetValues” to “clear” or you can change it directly in URL append to clear and value of variable “varRange” (Change cell range and sheet name) as per your need. For more information about it click here.
   1. Enter the URL below. Change Sheet ID, Range designator(including Sheet name) and API URL Parameters as per your need.
      

      https://sheets.googleapis.com/v4/spreadsheets/{{User::varGoogleSpreadsheetID}}/values/{{User::varRange}}:{{User::varSpreadsheetValues}}?valueInputOption={{User::varValueInputOption}}

      Example URL without variables (some part masked with xxxxxxxx):
      

      https://sheets.googleapis.com/v4/spreadsheets/1tuGO3_-2JlSmyiHwX6xxxxxxxxCHrORJc/values/Sheet1!A2:B4:clear?valueInputOption=USER_ENTERED

3. Change Request Method to POST.
4. Enter Request Body as below
   

   {}

5. Change Request Content type to application/JSON.
6. That’s all, click on the Test Request/Response button and verify data into Google spreadsheet.

Conclusion

Google Drive API and Google Sheets API provide a great way to automate file-related functionality. However, to call Google API  you have to use the SDK / coding approach (e.g. C#, Java, Python, Ruby). Luckily ZappySys SSIS PowerPack provides a great way to integrate any Google API call via simple drag and drop approach without coding. Try out yourself see how long it takes to call virtually any REST API.

References

Finally, you can use the following links for more information:

• JSON Generator Transform
• Web API Destination
• REST API Task
• Help File:
  • JSON Generator Transform
  • Web API Destination
  • REST API Task

The post How to write/update Google Sheet in SSIS appeared first on ZappySys Blog.

In our previous article we see How to read call REST API data in SSIS. Now in this article, we will see How to Make UPS API call in SSIS and load response into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call UPS REST API Call can be useful for any developer regardless of which programming language or tool set you use. We will also see how to configure POST API Call.

We will go through the steps to make Pickup Creation Request API call which Returns a PRN and Rate Result and we load it into MS SQL Server.

In nutshell, this post will focus on how to Make UPS REST API Call in SSIS.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of UPS API.

What is UPS?

United Parcel Service Founded in 1907, UPS is the world’s largest package delivery company and a leading global provider of specialised transportation and logistics services. Using advanced technology, access to global resources, and an integrated network of physical, technological, and human assets, UPS provides a powerful competitive advantage that can help you earn repeat customers and grow your business.

Where to get the UPS Credentials?

You can get Username and password by signup in UPS website and also need to get Access Key from UPS Developer Kit.

Call UPS REST API using SSIS

Let’s start with an example. We use SSIS JSON Source component to make the call to UPS API, we will make Pickup Creation Request and load response into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to make Pickup Creation Request API Call and select Method as POST, I am using Testing URL but we need to use production one for real time API call.
   Testing:
   

   https://wwwcie.ups.com/rest/Pickup

   Production:
   

   https://onlinetools.ups.com/rest/Pickup

   

   JSON Source Set URL

4. Furthermore, set below JSON string in the body and also replace your UPS username, password and access key in the placeholder, i used my dummy data you need to pass yours valid data.
   

   {
   	"UPSSecurity": {
   		"UsernameToken": {
   			"Username": "[YOUR USERNAME]",
   			"Password": "[PASSWORD]"
   		},
   		"ServiceAccessToken": {
   			"AccessLicenseNumber": "[ACCESS KEY]"
   		}
   	},
   	"PickupCreationRequest": {
   		"Request": {
   			"TransactionReference": {
   				"CustomerContext": "CustomerContext."
   			}
   		},
   		"RatePickupIndicator": "Y",
   		"TaxInformationIndicator": "Y",
   		"PickupDateInfo": {
   			"CloseTime": "1700",
   			"ReadyTime": "1000",
   			"PickupDate": "20191123"
   		},
   		"PickupAddress": {
   			"CompanyName": "EIVR TEST NON-ACCOUNT PL 3",
   			"ContactName": "EIVR TES PL",
   			"AddressLine": "12380 Morris Rd",
   			"City": "Louisville",
   			"StateProvince": "KY",
   			"PostalCode": "40201",
   			"CountryCode": "US",
   			"ResidentialIndicator": "N",
   			"Phone": {
   				"Number": "+48223803070"
   			}
   		},
   		"AlternateAddressIndicator": "N",
   		"PickupPiece": {
   			"ServiceCode": "003",
   			"Quantity": "1",
   			"DestinationCountryCode": "US",
   			"ContainerCode": "01"
   		},
   		"OverweightIndicator": "N",
   		"PaymentMethod": "05"
   	}
   }

   

   JSON Source set Body

5. Now click on Select Filter button to get the desire filter.
   

   JSON Source Select Filter

6. Finally click on Preview button to load the data.
   

   JSON Source Preview

7. That’s it we are ready to load UPS Pickup Creation Request API call response in SQL Server.

Load UPS REST API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to make UPS API call using SSIS JSON Source and load response into SQL Server. In this article we made Pickup Creation Request POST API call same way you can make other UPS API calls like Tracking API and Address Verification API and so on by passing relevant URL and JSON string. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • UPS Developer Kit

The post How to Make UPS API call in SSIS appeared first on ZappySys Blog.

In our previous blog we saw How to write data into CSV file in SSIS (GZip / Split). Now in this blog, we will see How to Pivot CSV Data in SSIS using CSV Source. It also supports Pivot mode so you can convert single CSV string value into Rows. In this article we also see how make some CSV columns as parent and not consider it in pivot and load it as it is using JSON Generator and JSON Parser Transform Pivot Option.

In nutshell, this post will focus on how to Pivot CSV Data into multiple columns and rows.

How to Pivot CSV Data in SSIS using CSV File Source

Let´s start with an example. In this SSIS CSV Parser Transform example, we will Parse database column string into multiple columns and rows.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. Furthermore, drag and drop the CSV Source and Double click on it to configure it.
   

   Drag and Drop : CSV Source from SSIS Toolbox

3. Enter the CSV file path and select desire column delimiter most common in CSV is comma and check the First Row as headers and click on preview.
   

   CSV Source Preview

4. Now, go to Pivot Columns to Rows Tab and checked Enable Pivoting and click on Preview Data to view the data.
   

   CSV Source Pivoted Data Preview

5. That’s it we see how to read data from CSV File and also Pivot the CSV Data using CSV Source.
6. Now In the next section we will see how to how make some CSV columns as parent and not consider it in pivot and load it as it is.

How make some CSV columns as parent.

Sometimes you just need to make child columns in to rows and want to leave parent level columns as it is.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. Secondly, drag and drop the CSV Source and Double click on it to configure it.
   

   Drag and Drop : CSV Source from SSIS Toolbox

3. Furthermore, enter the file path and click on preview.
   

   CSV Source Preview

4. For example if we want to pivot Email and Phone only in pivoting and wants to exclude parent level columns FirstName and LastName as it is.
5. For that drag and drop JSON Generator Transform and Select Output Mode as Output One JSON Per Parent Input Row  and Add Unbound Nested Element and named it as Persons and click on OK.
   

   JSON Generator Add Unbound Nested Element

6. Now add that two parent level fields FirstName and LastName in the Persons.
   

   JSON Generator Add Multiple Elements

7. Now select Person Element and right click and Add Unbound Nested Element and named it as Contacts and click on OK.
   

   JSON Generator Add Unbound Nested Element

8. Now add that two child level fields Email and Phone in the Contacts.
   

   JSON Generator Add Multiple Elements

9. Therefore JSON Generator Transform looks like below screen. Now Copy that Sample Preview JSON String to use it in the JSON Parser Transom as sample string.
   

   JSON Generator Transform – Sample Data

10. Similarly, drag and drop JSON Parser Transform and select the Input JSON Column and Paste that copied Sample JSON String and select the Contacts filter to read the Child data and click on Include Parent.
    

    JSON Parser Transform – Select Filter

11. Now go to Pivot Columns to Rows Tab and checked the Enable Pivoting and click on the Preview.
    

    JSON Parser Transform – Enable Pivoting

12. Finally, we are ready to load this data.
    

    CSV Pivoted Data

13. Let’s load this data in the SQL Server.

Load Pivoted CSV File data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to read CSV file data, pivot the data and Similarly, using SSIS JSON Generator and Parser Transform Pivot the CSV Data excluding some columns from pivoting.  To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following URL for more information.

• Help File: CSV Source

The post How to Pivot CSV Data in SSIS appeared first on ZappySys Blog.

In our previous article we Read data from Twilio API in SSIS . Now in this article, we will see example of how to Call SurveyGizmo API in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call SurveyGizmo API Call can be useful for any developer regardless of which programming language or tool set you use. we will also see how to store access token and secret in the variable and pass it in the URL using variable.

We will go through the steps to make Survey GetList REST API call which Get a list of all of your surveys and we load it into MS SQL Server.

In nutshell, this post will focus on how to Make SurveyGizmo REST API Call in SSIS.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of SurveyGizmo API.

What is SurveyGizmo?

SurveyGizmo takes data out of dashboards and puts it into the hands of people who take action. Through the systems they use every day. SurveyGizmo is an online survey solution that makes it easy for businesses of all sorts to create and conduct surveys, polls, quizzes, and questionnaires.  Reaching out to potential and existing customers, managing organizational and employee relationships and conducting academic researches has become convenient through SurveyGizmo.

• Click here to read SurveyGizmo Rest API Documentation.

Call SurveyGizmo REST API using SSIS JSON Source

Let’s start with an example. We use SSIS JSON Source component to make the call to SurveyGizmo API, we will read list of all available temporary lists and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Now let’s add variable for SurveyGizmo Access Token and Secret and Set it in as a value.
   

   SSIS Add Variable

4. Now double click on JSON Source and enter the following URL with variable as placeholder to get list of your surveys.
   

   https://restapi.surveygizmo.com/v5/survey?page=1&resultsperpage=100&api_token={{User::v_api_token}}&api_token_secret={{User::v_api_token_secret}}

   

   JSON Source : URL with variable as placeholder

5. Let’s configure the pagination, go to Pagination tab and select mode as URL Parameter and set the indicator as

   Page

    and configure it like below screen.
   

   JSON Source Pagination

6. You can refer to this article for more details on pagination here Understanding REST API Pagination in SSIS / ODBC Drivers
7. Now click on Select Filter button and select the desire filter from the window and click on Preview to get the data.
   

   JSON Source Preview

8. That’s it we are ready to load all SurveyGizmo Surveys data in SQL Server.

Load SurveyGizmo REST API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to extract information from REST API such as SurveyGizmo and load into SQL Server. We also learned techniques like How to use variable as placeholder in URL and configure the pagination. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • SurveyGizmo Rest API

The post Call SurveyGizmo API in SSIS appeared first on ZappySys Blog.

In our previous article we see How to call REST API in SSIS . Now in this article, we will see example of how to Make Twilio API Call in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call Twilio API Call can be useful for any developer regardless of which programming language or tool set you use. we will also see how to configure HTTP basic connection.

We will go through the steps to make Accounts REST API call which Returns a list of all Accounts resources and we load it into MS SQL Server.

In nutshell, this post will focus on how to Make Twilio REST API Call in SSIS.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of Twilio API.

What is Twilio?

Twilio is a cloud communications platform as a service company based in San Francisco, California. It’s allows software developers programmatically to make and receive phone calls, send and receive text messages, and perform other communication functions using its web service APIs.

• Twilio API Documentation : Click here to read Twilio Rest API Documentation.

Make Call of Twilio REST API using SSIS

Let’s start with an example. We use SSIS JSON Source component to make the call to Twilio API, we will read list of all available temporary lists and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to get multiple Account resources and create the HTTP Connection use the same url in the http connection also.
   

   https://api.twilio.com/2010-04-01/Accounts.json

   

   Json Source HTTP connection

4. Now click on Select Filter button and select the desire filter from the window.
   

   Json Source Select Filter

5. Go to Pagination Tab and select mode as Response attribute Mode – Read next page information from response and set up the next page filter.
   

   JSON Source Preview

6. You can refer to this article for more details on pagination here Understanding REST API Pagination in SSIS / ODBC Drivers
7. Now select the desire options to get the data and click on preview, as here i want parent and missing child also i checked both options and clicked on Preview. if you don’t want to get parent details you can uncheck it.
   

   JSON Source Preview

8. That’s it we are ready to load all Twilio accounts resources in SQL Server.

Load Twilio REST API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to extract information from REST API such as Twilio using HTTP Connection and load into SQL Server. We also learned techniques like How to create HTTP Connection and configure the pagination. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • Twilio Rest API

The post Read data from Twilio API in SSIS appeared first on ZappySys Blog.

In our previous article we see How to read call REST API data in SSIS. Now in this article, we will see How to Make MailPlus OAuth 1.0a REST API Call in SSIS and load into SQL Server. This blog mainly focuses on SSIS approach but steps mentioned to call MailPlus Oauth 1.0a REST API Call can be useful for any developer regardless of which programming language or tool set you use. We will also see how to configure OAuth1.0 connection.

We will go through the steps to make temporary lists API call which Returns a list of all available temporary lists and we load it into MS SQL Server.

In nutshell, this post will focus on how to Make MailPlus OAuth 1.0a REST API Call in SSIS.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. SSIS designer installed. Sometimes it is referred to as BIDS or SSDT (download it from Microsoft site).
2. Basic knowledge of SSIS package development using Microsoft SQL Server Integration Services.
3. Make sure ZappySys SSIS PowerPack is installed (download it).
4. Credentials of MailPlus API.

What is MailPlus?

MailPlus, founded in 2004, is the number one ESP (Email Service Provider) in the Netherlands. MailPlus is an award winning solution that enables companies to send out their online communication, such as newsletters, easily and professionally.

Where to get the MailPlus API Key and Secret?

After activation of the MailPlus REST API in a MailPlus account you can get unique Key and Secret from there. The MailPlus REST API uses Oauth 1.0a (one leg, see Oauth 1.0a (One Leg) ). A consumer key and secret are required to access your MailPlus account. These can be created in your MailPlus account. The Key and Secret are always available via the startpage of a MailPlus account. Click on the tab “MailPlus Settings” and then “MailPlus Rest API”. In MailPlus eCom an authentication is already available.

Call MailPlus Oauth 1.0a REST API using SSIS

Let’s start with an example. We use SSIS JSON Source component to make the call to MailPlus API, we will read list of all available temporary lists and load into SQL Server. First of All, Open Visual Studio and Create New SSIS Package Project.

1. First of All, Drag and drop Data Flow Task from SSIS Toolbox and double click it to edit.
   

   Drag and Drop : SSIS Data Flow Task from SSIS Toolbox

2. From the SSIS toolbox drag and drop JSON Source on the data flow designer surface.
   

   Drag and Drop JSON Source Component

3. Double click JSON Source and enter the following URL as below to get list of all available temporary lists and create the OAuth 1.0 Connection.
   

   https://restapi.mailplus.nl/integrationservice-1.1.0/templist

   

   MailPlus Oauth 1.0a Connection

4. Finally click on Preview button to load the data.
   

   MailPlus Preview Data

5. You can find more MailPlus API Lists here on MailPlus Interactive API Documentation link.
6. That’s it we are ready to load all available temporary lists in SQL Server.

Load MailPlus REST API data into SQL Server

ZappySys SSIS PowerPack makes it easy to load data from various sources such as REST, SOAP, JSON, XML, CSV or from other source into SQL Server, or PostgreSQL, or Amazon Redshift, or other targets. The Upsert Destination component allows you to automatically insert new records and update existing ones based on key columns. Below are the detailed steps to configure it.

Step 1: Add Upsert Destination to Data Flow

1. Drag and drop the Upsert Destination component from the SSIS Toolbox.
2. Connect your source component (e.g., JSON / REST / Other Source) to the Upsert Destination.

SSIS - Data Flow - Drang and Drop Upsert Destination Component

Step 2: Configure Target Connection

1. Double-click the Upsert Destination component to open the configuration window.
2. Under Connection, select an existing target connection or click NEW to create a new connection.
   • Example: SQL Server, or PostgreSQL, or Amazon Redshift.

Step 3: Select or Create Target Table

1. In the Target Table dropdown, select the table where you want to load data.
2. Optionally, click NEW to create a new table based on the source columns.

Configure SSIS Upsert Destination Connection - Loading data (REST / SOAP / JSON / XML /CSV) into SQL Server or other target using SSIS

Step 4: Map Columns

1. Go to the Mappings tab.
2. Click Auto Map to map source columns to target columns by name.
3. Ensure you check the Primary key column(s) that will determine whether a record is inserted or updated.
4. You can manually adjust the mappings if necessary.

SSIS Upsert Destination - Columns Mappings

Step 5: Save Settings

• Click OK to save the Upsert Destination configuration.

Step 6: Optional: Add Logging or Analysis

• You may add extra destination components to log the number of inserted vs. updated records for monitoring or auditing purposes.

Step 7: Execute the Package

• Run your SSIS package and verify that the data is correctly inserted and updated in the target table.

SSIS Upsert Destination Execution

Conclusion

After all, we saw you how to extract information from REST API such as MailPlus REST API using OAuth1.0 Connection and load into SQL Server. We also learned techniques like How to create OAuth1.0 Connection. To explore many other scenarios not discussed in this article download SSIS PowerPack from here (includes 70+ Components).

References

Finally, you can use the following links for more information:

• • Help File: JSON Source(REST API or File)
  • MailPlus API

The post How to Make MailPlus OAuth 1.0a REST API Call in SSIS appeared first on ZappySys Blog.

In our previous article we saw how to call PayPal API in SSIS. Now in this article, we will see How to Make PayPal API Call in ODBC from the SQL Server. This blog mainly focuses on ODBC approach but steps mentioned to call PayPal APIs can be useful for any developer regardless of which programming language or tool set you use. We will also see How to send basic Authorization header along with body in OAuth2.

We will go through the steps to Create PayPal Order from MS SQL Server.

In nutshell, this post will focus on how to make PayPal API Call using ODBC Driver.

Prerequisites

Before we perform the steps listed in this article, you will need to make sure the following prerequisites are met:

1. A first requirement, make sure that Microsoft Access installed
2. Make sure to have ZappySys ODBC PowerPack installed.
3. Credentials of PayPal API. Click here to create API.

What is PayPal?

PayPal is an American company operating a worldwide online payments system that supports online money transfers and serves as an electronic alternative to traditional paper methods like checks and money orders. The company operates as a payment processor for online vendors, auction sites, and many other commercial users, for which it charges a fee in exchange for benefits such as one-click transactions and password memory. PayPal’s payment system, also called PayPal, is considered a type of payment rail.

Get Started with PayPal API

If you have need to automate PayPal operations (e.g. read / write / update / delete ) then you can use PayPal REST API. Before you can integrate a PayPal product or solution, you must set up your development environment to get OAuth 2.0 client ID and secret credentials for the sandbox and live environments. You exchange these credentials for an access token that authorizes your REST API calls. To test your web and mobile apps, you create sandbox accounts.

Get PayPal credentials

To generate OAuth 2.0 credentials for the sandbox and live environments:

1. Log into Dashboard and type your PayPal personal or business account email and password.
2. In the REST API apps section, click Create App. The purpose of this app is to generate your credentials.
3. Type a name for your app and click Create App. The page shows your sandbox app information, which includes your credentials.
   

   Note: To show your live app information, toggle to Live.

4. Copy and save the client ID and secret for your sandbox app.
   

   PayPal Client ID and Secret

5. Review your app details and save your app.

Call PayPal Create Order API in SQL Server (T-SQL) using Data Gateway

Let’s start with an example. We use JSON Driver to make the call to PayPal API, we will Create the PayPal Order from SQL Server.

In REST API calls, include the URL to the API service for the environment:

• Sandbox: https://api.sandbox.paypal.com
• Live: https://api.paypal.com

1. To do this, first of all, we will open the ZappySys Data Gateway Configuration:
   

   Open ZappySys Data Gateway

2. Add the Native – ZappySys JSON Driver Data source.
   

   Add Gateway Data Source (Native JSON Driver)

3. Now it’s time to connect with PayPal. Let’s use JSON Driver Data source with the following URL:
   

   https://api.sandbox.paypal.com/v2/checkout/orders

   Body:
   

   {
     "intent": "CAPTURE",
     "purchase_units": [
       {
         "amount": {
           "currency_code": "USD",
           "value": "100.00"
         }
       }
     ]
   }

   

   PayPal : Create Order

4. Create Basic OAuth Client Credentials Grant (Configure ZS-OAuth Connection).
   In the SSIS ZS OAuth Connection Manager, Select Provider as Custom, select version as OAuth2 and select Grant Type as  Client Credentials Grant. Enter the credentials and access token url and click OK.
   

   https://api.sandbox.paypal.com/v1/oauth2/token

   

   ZS OAuth Connection – Client Credentials Grant

5. Furthermore, Now click on Test Connection button to test PayPal Connection.
   

   PayPal : Test Connection

6. Write your Query and Preview Data.
7. (Optional Step) If your Query use POST method (e.g. Create Invoice) to submit data then you have Save Metadata (Read More). We will use this meta file in later step when we call POST request.
   

   Generate Metadata File for REST API POST Operation using ZappySys API Driver (JSON / XML)

8. Now using Code Generator we will generate the query.
   

   Generate Query

Setup Linked Server in MS SQL Server for PayPal

Once you configured the data source in Gateway, we can now setup Linked Server in SQL Server to query API Calls.

1. Assuming you have installed SQL Server and SSMS. If not then get both for FREE from here: Get SQL Server Express and  Get SSMS
2. Open SSMS and connect to SQL Server.
3. Go to Root > Server Objects > Linked Servers node. Right click and click New Linked Server…

   

   Add Linked Server in SQL Server

4.  Now enter the linked server name, select Provider as SQL Native Client
5. Enter data source as 

   GatewayServerName, PORT_NUMBER

    where server name is where ZappySys Gateway is running (Can be same as SQL Server machine or remote machine). Default PORT_NUMBER is 5000 but confirm on Data gateway > General tab in case its different.
6. Enter Catalog Name. This must match name from Data gateway Data sources grid > Name column

   

   Configure Linked Server Provider, Catalog, Server, Port for ZappySys Data Gateway Connection

7. Click on Security Tab and select last option “Be made using this security context“. Enter your gateway user account here.
8. Click OK to save Linked Server

Create Order in PayPal from SQL Server (Call POST API Method)

1. To Create Order in PayPal we need to execute the query like this. Please refer to this link for more information: Create Order example.
   
   Important Note*: Below query uses META property (fully explained here). This is needed if you creating anew record using POST, PUT. If you don’t pass META='{xxxx}’ in query driver calls REST API two times (first time to parse metadata and second time to get actual data). So that you find some odd thing in PayPal if you execute the query without Meta then you will find two same Orders created in your PayPal as the driver makes two calls as mentioned above.

   -- In below query rather than WITH(META=''[{"Name": "Success","Type": "Int32"}]'' ....
   -- You can also use WITH(META=''c:\my-meta-file.txt'' .... We created this file in previous section
   
   SELECT * FROM OPENQUERY([MY_LINKED_SERVER_NAME]
   , 'SELECT * FROM $
   WITH(META=''[{"Name": "Success","Type": "Int32"}]''
   	 ,Src=''https://api.sandbox.paypal.com/v2/checkout/orders''
   	,RequestData=''{
     "intent": "CAPTURE",
     "purchase_units": [
       {
         "amount": {
           "currency_code": "USD",
           "value": "100.00"
         }
       }
     ]
   }''
   	,RequestContentTypeCode=''ApplicationJson''
   	,RequestMethod=''POST''
   )')

2. Here is the preview after you run REST API query in SQL Server. It will return created Order Id, Links and status. Notice that you can override default configuration by supplying many parameters in WITH clause.
   

   Create Order In PayPal Using SQL Query

3. You can also find this created Order by opening that URL in the browser.
   

   Created Order In PayPal

PayPal API integration in ODBC apps (e.g. Power BI, Excel, Informatica, Access, C#…)

So far we have looked at SQL Server integration only, but what if you like to use same powerful ZappySys API drivers part of ODBC PowerPack in other Apps like Power BI, Excel, MS Access, Informatica and many more… ?? Well not to worry if you know that App supports ODBC you can use ZappySys ODBC Driver same way described earlier… only difference is you dont need Linked Server and you dont have to use OPENQUERY statement when you use ODBC Drivers.

ZappySys ODBC Drivers built using ODBC standard which is widely adopted by industry for a long time. Which mean the majority of BI Tools / Database Engines / ETL Tools already there will support native / 3rd party ODBC Drivers. Below is the small list of most popular tools / programming languages our Drivers support. If your tool / programming language doesn't appear in the below list, which means we have not documented use case but as long as your tool supports ODBC Standard, our drivers should work fine.  

BI / Reporting Tools Integration	ETL Tools Integration	Programming Languages Integration
Microsoft Power BI Tableau SSRS (SQL Reporting Services) QlikView /Qlik Sense MicroStrategy Google Sheet Microsoft Excel Microsoft Access Oracle OBIEE Many more (not in this list).....	Informatica PowerCenter (Windows) Informatica Cloud SSIS (SQL Integration Services) SQL Server Talend Data Studio Pentaho Kettle Oracle OBIEE Many more (not in this list).....	Visual Studio C# C++ JAVA Python PHP PowerShell T-SQL (Using Linked Server)

 

Conclusion

So in this blog, we learned how to generate a PayPal API Token, Load PayPal data in MS SQL Server table using ODBC JSON / REST API Driver in a very simple way. You can achieve many more functionalities with this tool. Check our blogs/articles on JSON / REST API Driver to find out what this tool is capable of more.

References

Finally, you can use the following links for more information about the use of PayPal API with our tools:

• Landing Page for ODBC JSON / REST API Driver, you can also find Tutorial Video here.
• Help File: Documentation of JSON Driver.
• PayPal API

The post How to Make PayPal API Call in ODBC / SQL Server (T-SQL) appeared first on ZappySys Blog.