Downloading Gmail Attachments in SSIS is always a little more difficult than working with simple text or numeric columns. If you are not an SSIS expert, doing it for the first time may be challenging. That is why we created this article for you. This is a common request we receive, so we are bringing this article to you.

Gmail is the most popular email service in the world, and it can be helpful to automate and get valuable information from it. We will connect to Gmail with the REST API. For that, if you are not an experienced user of the REST API, do not worry. This article is for you. We will provide simple steps for the connection.

Requirements

To do it, we will need the following components:

1. SSDT for SSIS must be installed on your computer
2. ZappySys SSIS Powerpack should be installed. This Powerpack Plugin includes connectors to Gmail via the REST API.

Getting Started

Creating the connection

1. First of all, in SSDT, create a new SSIS project
2. To create a connection in SSDT, open the Connection Manager and select New Connection.
   

   Create a new connection

3. Select the ZS OAuth connection. This is a special connection included with the ZS.
   SSIS PowerPack installer that we will use to connect to Gmail.
   

   Select ZS-OAuth connector
4. In the ZS OAuth connection, select the Google API OAuth Provider and specify the following credentials and scope permissions:
   

   SSIS Gmail connection

   https://mail.google.com/
   https://www.googleapis.com/auth/gmail.modify
   https://www.googleapis.com/auth/gmail.readonly
   https://www.googleapis.com/auth/gmail.addons.current.message.readonly
   https://www.googleapis.com/auth/gmail.addons.current.message.action

Creating the package to download Gmail attachments in SSIS

1. First of all, drag and drop the Data Flow Task from the SSIS Toolbox and double-click it
   to edit.
   

   Drag and Drop SSIS Data Flow Task from SSIS Toolbox

2. Secondly, in the Data Flow, we will use the JSON Source to connect to Gmail and get the attachments. JSON Source is a ZappySys component used now only to connect to JSON files, but also to connect to REST API Servers that support JSON. In this case, Gmail REST API can send information in JSON Format. Use the connection created before.
3. Basically, we need to specify the URL of the Gmail attachment. In Gmail, the email attachment is inside the message. Then you need to specify the message-id and the Gmail attachment-id like this:
   

   Zappysys JSON source Gmail attachments configuration

   https://www.googleapis.com/gmail/v1/users/me/messages/{{User::email_id}}/attachments/{{User::attachment_id}}

4. Now, in the go-to columns tab, set the data type to DT_IMAGE and increase the length of the other columns’ DT_WSTR fields by 1000 and 1500. We set

   contentBytes

   to the

   DT_IMAGE

     datatype, it’s like

   varbinary(MAX)

   

   JSON Source Columns Tab – attachment datatype

5. Next, we will use the derived column to set the path to the local folder where the attachment will be stored. The derived column will be used to set the local path for the attachment. Make sure to specify valid paths with double backslashes, since backslashes are special characters. The derived column is a special transformation component that creates new columns. SSIS expressions.
   

   Derived columns – Gmail attachment file path

6. Also, we will require the Export column to download into a file. This component is specially used to get images or binary files into the file system.
   

   Export columns transformation – Gmail attachment

7. Finally, we will use the Trash destination to get the attachment and finish the flow. This one is also a ZappySys component ubiquitous.
8. Your package should be something like this:
   

   Final Result – Gmail attachment

9. The next step is to run it and test it.
10. If everything is fine, we will be able to see the file in your system.

Conclusion

In conclusion, we learned how to download Gmail attachments in SSIS. Basically, you need to use the ZappySys JSON Source to connect to Gmail via the REST API, then use the derived column component to specify the path to store the attachment. The next step is to use the Export column component to export the attachment. Finally, the ZS Trash destination is the attachment’s destination. If you liked this article, do not hesitate to continue testing and using ZappySys PowerPack for SSIS.

API Connector

Here are a few examples with our API connector for Gmail

API connector – Gmail get message attachments

API connector – Gmail download message attachments

The post Download Gmail Attachment in SSIS appeared first on ZappySys Blog.

In this article, we will learn how to read REST API in Pentaho Kettle. Pentaho is a very popular tool in Business Intelligence to integrate data for OLAP services and to create reports and dashboards. In this article, we will use our ZappySys ODBC PowerPack to connect to REST API in Pentaho Kettle.

Our software requires the JDBC connector for SQL Server and then Uses our ZappySys Gateway to connect.

Requirements

1. First of all, we will require the ZappySys ODBC PowerPack.
2. Secondly, we will use Pentaho Kettle (Our example in this post uses v8.1)
3. Finally, we will require the JDBC driver for SQL Server.

Getting started

First of all, we will need to create an ODBC connection to REST API using the Windows ODBC Data Source Administrator. Next, we will create a gateway and finally, we will invoke the data in Pentaho.

Create an ODBC connection to read REST API in Pentaho Kettle

In this example, we will first create an ODBC connection to an URL.

1. First of all, open the ODBC Data Source administrator in Windows in the System DSN press Add:
   

   Add new ODBC driver

2. Secondly, select the ZappySys JSON Driver:
   

   Select JSON driver in ODBC DS Administrator

3. Finally, add a data source name and specify a link to a URL. In this example, we will connect to the following url:
   

   URL: https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json

   

   REST API ODBC URL

Create a ZappySys gateway to JDBC to read REST API in Pentaho

In order to connect to REST API, ZappySys ODBC PowerPack uses the JDBC for SQL Server and a gateway. Let’s take a look at it:

1. First of all, open the ZappySys Gateway that is installed with the ODBC PowerPack and add a new data source:
   

   Add new Data Source in ZappySys Gateway

2. Secondly, add the data source name:
   

   Pentaho REST Client

3. In addition, in the user’s tab, add the user test and specify a password:
   

   Add user and login

4. Also, in settings, press edit and add the ODBC Data Source created in the previous section and add the user created before:
   

   Link Data Source in ODBC with the default gateway

5. Finally, in the General tab, you can specify the port. By default, it is 5000.
   Make sure to save the changes and check the gateway is started with the start button:
   

   Default gateway with port 5000

How to read REST API in Pentaho

1. First of all, in Pentaho, we will create a project like the following:
   

   Pentaho Project REST API csv

2. Secondly, we will create a query like the following:
   

   A query in Pentaho to get data

3. In addition, we will add the following connection. We will connect to the data source named customers created in the ZappySys Gateway with port 5000 and the user created in the Gateway:
   

   JDBC connection to SQL Server

4. If everything is OK, you will be able to see the CSV file generated:
   

   Data in CSV format

5. Also, you can use SQL statements. The following example shows how to do a where filter:
   

   SELECT Shipname 
   FROM value
   where Shipname='Consolidated Holdings'

6. Finally, we will show the first 10 rows using a group by:
   

   SELECT Shipname 
   FROM value
   group by shipname
   limit 10

How to read REST API in Pentaho (Gmail)

The following example will show how to get Gmail information from Pentaho.

1. First of all, we will need to create a connection to Gmail in ODBC using the ZappySys ODBC Driver (Check the Create an ODBC connection for a Pentaho REST Client section). In the ODBC Administrator, we will add a ZappySys JSON Driver connection.
2. Secondly, in the ZappySys ODBC driver, specify the URL and select the OAuth connection:
   

   ODBC configuration

3. Also, in the OAuth properties specify the Google OAuth provider and the https://mail.google.com scope. Press the generate token button and specify your credentials and accept the access permission request:
   

   Gmail OAuth connection

4. In addition, using the ZappysSys Data Gateway, add the new Data Source and associate with the ODBC connection just created:
   

   Connect Pentaho with Gmail

5. Additionally, add the test user to the data source:
   

   Include user and permission to the gateway

6. In Pentaho, change the connection in the table input to connect to the Gateway:
   

   Pentaho Rest client to Gmail

7. Finally, in the table input add the following query:
   

   Connect Pentaho to Gmail

Conclusions

To conclude, we can say that it is simple to connect to the REST API using Pentaho with the help of ZappySys ODBC PowerPack. This tool allows to easily query REST API data like a simple SQL Query. If you like this tool you can
download the ODBC PowerPack here.

References

For more information, refer to the following links:

• ZappySys ODBC PowerPack
• Introduction To Pentaho

The post How to read REST API in Pentaho Kettle appeared first on ZappySys Blog.

In this new article, we will see how to connect Java to REST API using a simple connector. This ODBC driver allows connecting to REST API or a local file in Java using SQL queries. If you are familiar with SQL, you can easily access to REST API or JSON using simple SQL queries. This article mainly focus on JSON based REST API but techniques listed in this article is almost same for XML based API (e.g. SOAP Web Services).

Requirements

1. First of all, we will require the JDBC Driver for SQL Server.
2. Secondly, we will require ZappySys ODBC PowerPack.
3. Also, we will require a Java IDE. We will use Java NetBeans for this.

Getting started

First of all, we will create an ODBC connection with a Gateway to the SQL Server connector and then we will connect with Java code.

Add ZappySys ODBC Driver to connect to REST API

The ZappySys JSON Driver allows connecting to different resources like JSON files, REST API, OData, etc.

1. First of all, in the ODBC Data Source Administrator, press add to add a new System Data Source:
   

   Add new ODBC driver

2. Secondly, select the ZappySys JSON Driver:
   

   Select JSON driver in ODBC DS Administrator

3. Also, add this URL to the ZappySys JSON Driver:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json

   

   REST API ODBC URL

ZappySys Gateway to connect Java to REST API

In order to connect Java to REST API, we will use a Gateway that is installed with the ZappySys ODBC PowerPack. This gateway uses a SQL Server connection. Let’s take a look at the steps.

1. First of all, open the ZappysSys Data Gateway Configuration.
2. Secondly, press the add button and create a new data source named customer:
   

   Add new Data Source in ZappySys Gateway

3. Also, enable the custom data source and make sure to check the public option.
4. In addition, press the Edit and add the ODBC Data Source created in the Add ZappySys ODBC Driver to connect to REST API section:
   

   <strong>NOTE:</strong> Whenever possible use native driver option for better performance / security and ease of use.

   

   Zappysys ODBC driver

5. Additionally, we will add a new user:
   

   Add the user to the data source gateway

6. Also, add the user to the data source:
   

   Add user to the data source

7. Finally, check the default port in the general tab. By default this value is 5000:
   

   Default gateway with port 5000

Connect Java to REST API

1. First of all, we will first try to connect to the following URL:
   

   https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json

2. Secondly, we will get the countries and the sum or quantity orders using the following code:
   

   package padaone;
   
   import java.sql.*;
   
   public class zappy {
   
   	public static void main(String[] args) {
   		
   		// Create a variable for the connection string.
   		String connectionUrl = "jdbc:sqlserver://localhost:5000;databasename=customer;user=test;password=test";
   
   		// Declare the JDBC objects.
   		Connection con = null;
   		Statement stmt = null;
   		ResultSet rs = null;
   		
           	try {
           		// Establish the connection.
           		Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
               		con = DriverManager.getConnection(connectionUrl);
               
               		// Create and execute an SQL statement that returns some data.
               		String SQL = "SELECT Country , SUM(UnitPrice * Quantity) Total " 
   	+ "FROM value " 
   	+ "GROUP BY Country " 
   	+ "WITH (SRC='https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json')";
               		stmt = con.createStatement();
               		rs = stmt.executeQuery(SQL);
               
               		// Iterate through the data in the result set and display it.
               		while (rs.next()) {
               			System.out.println(rs.getString(1) + " " + rs.getString(2));
               		}
           	}
           
   		// Handle any errors that may have occurred.
   		catch (Exception e) {
   			e.printStackTrace();
   		}
   
   		finally {
   			if (rs != null) try { rs.close(); } catch(Exception e) {}
   	    		if (stmt != null) try { stmt.close(); } catch(Exception e) {}
   	    		if (con != null) try { con.close(); } catch(Exception e) {}
   		}
   	}
   }

3. In addition, in the project, go to libraries, right click and select the option to add JAR or folder and select the sqljdbc42.jar. This jar file is used to connect to SQL Server. The ZappySys ODBC PowerPack uses the SQL Server JDBC connector with a gateway to connect to REST API or JSON files:
   

   Add SQL Server jar file

4. Also, we will need to add the sqljdb4.2.jar and the sqljdbc_auth.dll in the C:/Windows/System32 and the path where the JDK is installed (in this example C:/Program Files/Java/jdk1.8.0_171/bin).
5. Finally, run the code and you will visualize the query results of the REST API:
   

   REST API results in Java

Explanation of the code to Connect Java to REST API

Previously, we run the code to query REST API data in Java. Now we will explain the code.

1. First of all, we will import the java.sql to connect to SQL Server using Java:
   

   import java.sql.*;

2. Secondly, we will create a connection to the ZappySys Gateway. Using the port 5000 and we are using the data source and user created in the Add ZappySys ODBC Driver to connect to REST API:
   

   String connectionUrl = "jdbc:sqlserver://localhost:5000;databasename=customer;
   user=test;password=test";

3. Also, we will set the variables to null to initialize them:
   

   Connection con = null;
   Statement stmt = null;
   ResultSet rs = null;

4. In addition, we will connect to the JDBC SQL driver:
   

   Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
   con = DriverManager.getConnection(connectionUrl);

5. Additionally, we will send a query to the URL and get the country and the sum of price multiplied by quantity, group by country:
   

   String SQL = "SELECT Country , SUM(UnitPrice * Quantity) Total " 
   	+ "FROM value " 
   	+ "GROUP BY Country " 
   	+ "WITH (SRC='https://services.odata.org/V3/Northwind/Northwind.svc/Invoices?$format=json')";
               		stmt = con.createStatement();
               		rs = stmt.executeQuery(SQL);

6. Next, we will print the results of the query:
   

   while (rs.next()) {
      System.out.println(rs.getString(1) + " " + rs.getString(2));
   }

7. Finally, we will close all the connections:
   

   finally {
   			if (rs != null) try { rs.close(); } catch(Exception e) {}
   	    		if (stmt != null) try { stmt.close(); } catch(Exception e) {}
   	    		if (con != null) try { con.close(); } catch(Exception e) {}
   		}

    

How to connect to multiple JSON files

The ZappySys ODBC PowerPack can be used to easily read local JSON files in Java and query using SQL with SELECT, WHERE, GROUP BY, HAVING and other SQL clauses. For more information, refer to this link.

1. The following code will list the files and the content of the JSON files:
   

   package padaone;
   
   import java.io.File;
   import java.sql.Connection;
   import java.sql.DriverManager;
   import java.sql.ResultSet;
   import java.sql.Statement;
   
   public class localfile {
       public static void main(String[] args) {
           File directory = new File ("C:\\temp");
           File[] list = directory.listFiles();
           
           printNames(list);	
       }
   
       private static void printNames(File[] list) {
           for(int i =0;i<list.length;i++){
               if(list[i].isFile()){
                   System.out.println(list[i].getName());
   
                   
                   showContent(list[i]);
                   
               }
           }
       }
   
       private static void showContent(File file) {
         
           // Create a variable for the connection string.
           String connectionUrl = "jdbc:sqlserver://localhost:5000;databasename=customer;user=test;password=test";
   
           // Declare the JDBC objects.
           Connection con = null;
           Statement stmt = null;
           ResultSet rs = null;
   
           try {
                   // Establish the connection.
                   Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
                   con = DriverManager.getConnection(connectionUrl);
   
                   // Create and execute an SQL statement that returns some data.
                   String SQL = "SELECT * FROM $ WITH (SRC='C:\\temp\\"+file.getName()+"')";
                   stmt = con.createStatement();
                   rs = stmt.executeQuery(SQL);
   
                   // Iterate through the data in the result set and display it.
                   while (rs.next()) {
                           System.out.println(rs.getString(1) + " " + rs.getString(2));//+ " " + rs.getString(3)+ " " + rs.getString(4)+ " " + rs.getString(5)+ " " + rs.getString(6));
                   }
           }
   
           // Handle any errors that may have occurred.
           catch (Exception e) {
                   e.printStackTrace();
           }
   
           finally {
                   if (rs != null) try { rs.close(); } catch(Exception e) {}
                   if (stmt != null) try { stmt.close(); } catch(Exception e) {}
                   if (con != null) try { con.close(); } catch(Exception e) {}
           }
   	
       }
   
   }

2. The JSON files are the following:
   

   example_1.json
   
   [{
     "id": 1,
     "first_name": "Jeanette",
     "last_name": "Penddreth",
     "email": "jpenddreth0@census.gov",
     "gender": "Female",
     "ip_address": "26.58.193.2"
   }, {
     "id": 2,
     "first_name": "Giavani",
     "last_name": "Frediani",
     "email": "gfrediani1@senate.gov",
     "gender": "Male",
     "ip_address": "229.179.4.212"
   }, {
     "id": 3,
     "first_name": "Noell",
     "last_name": "Bea",
     "email": "nbea2@imageshack.us",
     "gender": "Female",
     "ip_address": "180.66.162.255"
   }, {
     "id": 4,
     "first_name": "Willard",
     "last_name": "Valek",
     "email": "wvalek3@vk.com",
     "gender": "Male",
     "ip_address": "67.76.188.26"
   }]
   
   example_2.json
   
   {
       "quiz": {
           "sport": {
               "q1": {
                   "question": "Which one is correct team name in NBA?",
                   "options": [
                       "New York Bulls",
                       "Los Angeles Kings",
                       "Golden State Warriros",
                       "Huston Rocket"
                   ],
                   "answer": "Huston Rocket"
               }
           },
           "maths": {
               "q1": {
                   "question": "5 + 7 = ?",
                   "options": [
                       "10",
                       "11",
                       "12",
                       "13"
                   ],
                   "answer": "12"
               },
               "q2": {
                   "question": "12 - 8 = ?",
                   "options": [
                       "1",
                       "2",
                       "3",
                       "4"
                   ],
                   "answer": "4"
               }
           }
       }
   }

3. Finally, if everything is fine, you will be able to see the file names
   and the content:
   

   Read JSON files in Java

How to connect Java to REST API Gmail

Connect Java to Google, Facebook, OneDrive or other REST API tools is a straightforward process using the ZappySys ODBC Driver. You just need to create the Data Source in ODBC and then create a ZappySys Gateway, the rest is simple in Java. Let’s take a look at an example.

1. First of all, we will create a new Data source in the Windows ODBC Data
   Source Administrator. We will create a new ZappySys JSON Driver and add the following link:
   

   https://www.googleapis.com/gmail/v1/users/me/messages/

2. Secondly, select the OAuth connection type and $.messages[*] in the
   filter and press the Click to Configure link:
   

   Setup ODBC configuration to URL

3. Also, in the OAuth parameters link, select the Google OAuth Provider.
4. In addition in scopes write the scope https://mail.google.com:
   

   Rest API connect to Gmail

5. Additionally, in the preview tab, you can generate your queries and save
   your Data Source:
   

   Connect Java to REST API

6. Next, open your ZappySys Data Gateway. We will create a new Data Source
   named gm:
   

   Add Gmail Data Source

7. Also, we will associate the ODBC Data Source with the Gateway Data Source. Make sure to add the test user to the Gateway data source:
   

   Include the data source for the ODBC

8. Finally, you can run this code. The code in Java will connect to the Gateway data source using the port 5000 and the Gateway user and password. The query will show the Gmail message ids:
   

   package padaone;
   
   import java.sql.Connection;
   import java.sql.DriverManager;
   import java.sql.ResultSet;
   import java.sql.Statement;
   
   
   public class gmail {
       public static void main(String[] args) {
   		
   		// Create a variable for the connection string.
   		String connectionUrl = "jdbc:sqlserver://localhost:5000;databasename=gm;user=test;password=test";
   
   		// Declare the JDBC objects.
   		Connection con = null;
   		Statement stmt = null;
   		ResultSet rs = null;
   		
           	try {
           		// Establish the connection.
           		Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
               		con = DriverManager.getConnection(connectionUrl);
               
               		// Create and execute an SQL statement that returns some data.
               		String SQL = "SELECT id FROM [_root_]";
               		stmt = con.createStatement();
               		rs = stmt.executeQuery(SQL);
               
               		// Iterate through the data in the result set and display it.
               		while (rs.next()) {
               			System.out.println(rs.getString(1));
               		}
           	}
           
   		// Handle any errors that may have occurred.
   		catch (Exception e) {
   			e.printStackTrace();
   		}
   
   		finally {
   			if (rs != null) try { rs.close(); } catch(Exception e) {}
   	    		if (stmt != null) try { stmt.close(); } catch(Exception e) {}
   	    		if (con != null) try { con.close(); } catch(Exception e) {}
   		}
   	}
   }

9. You can find your message id in Google by clicking your messages:
   

   Display Gmail information in Java

ZappySys JSON /REST API Driver Query Examples

Reading from XML files or API can be done using the same way as previous sections except you have to use ZappySys XML Driver. Read help file here to see json query examples.

ZappySys XML / SOAP Driver Query Examples

Reading from XML files or API can be done using the same way as previous sections except you have to use ZappySys XML Driver. Read help file here to see xml query examples.

Calling XML SOAP Web Service in JAVA

So far we have looked at examples to consume data using JSON driver. Now lets look at an example, to call XML SOAP Web Service in JAVA.

To call SOAP API you need to know Request XML Body Structure. If you are not sure how to create SOAP Request body then no worries. Check this article to learn how to generate SOAP Request body using the Free tool SoapUI. Basically, you have to use SoapUI to generate Request XML and after that, you can replace parameters as needed in the generated body.

What is SOAP Web Service?

If you are new to SOAP Web Service sometimes referred as XML Web Service then please read some concept about SOAP Web service standard from this link There are two important aspects in SOAP Web service.

1. Getting WSDL file or URL
2. Knowing exact Web Service URL

What is WSDL

In very simple term WSDL (often pronounced as whiz-dull) is nothing but a document which describes Service metadata (e.g. Functions you can call, Request parameters, response structure etc). Some service simply give you WSDL as xml file you can download on local machine and then analyze or sometimes you may get direct URL (e.g. http://api.mycompany.com/hr-soap-service/?wsdl )

Example SQL Query for SOAP API call using ZappySys XML Driver

Here is an example SQL query you can write to call SOAP API. If you not sure about many details then check next few sections on how to use XML Driver User Interface to build desired SQL query to POST data to XML SOAP Web Service without any coding.

SELECT * FROM $
WITH(
	 Src='http://www.holidaywebservice.com/HolidayService_v2/HolidayService2.asmx'
	,DataConnectionType='HTTP'
	,CredentialType='Basic' --OR SoapWss
	,SoapWssPasswordType='PasswordText'
	,UserName='myuser'
	,Password='pass$$w123'
	,Filter='$.soap:Envelope.soap:Body.GetHolidaysAvailableResponse.GetHolidaysAvailableResult.HolidayCode[*]'
	,ElementsToTreatAsArray='HolidayCode'	
	,RequestMethod='POST'	
	,Header='Content-Type: text/xml;charset=UTF-8 || SOAPAction: "http://www.holidaywebservice.com/HolidayService_v2/GetHolidaysAvailable"'
	,RequestData='
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:hol="http://www.holidaywebservice.com/HolidayService_v2/">
   <soapenv:Header/>
   <soapenv:Body>
      <hol:GetHolidaysAvailable>
         <!--type: Country - enumeration: [Canada,GreatBritain,IrelandNorthern,IrelandRepublicOf,Scotland,UnitedStates]-->
         <hol:countryCode>UnitedStates</hol:countryCode>
      </hol:GetHolidaysAvailable>
   </soapenv:Body>
</soapenv:Envelope>'
)

Now let's look at steps to create SQL query to call SOAP API. Later we will see how to generate code for your desired programming language (e.g. C# or SQL Server)

Video Tutorial - Introduction to SOAP Web Service and SoapUI tool

Before we dive into details about calling SOAP API using ZappySys XML Driver, lets first understand what is SOAP API and how to create SOAP requests using SoapUI tool. You will learn more about this process in the later section. The video contains some fragment about using SOAP API in SSIS but just ignore that part because we will be calling Soap API using ZappySys ODBC Driver rather than SSIS Components.  

Using SoapUI to test SOAP API call / Create Request Body XML

Assuming you have downloaded and installed SoapUI from here, now we are ready to use WSDL for your SOAP Web Service Calls. If you do not have WSDL file or URL handy then contact your API provider (sometimes you just have to add ?wsdl at the end of your Service URL to get WSDL so try that. Example: http://mycompany/myservice?wsdl ). If you don't know what is WSDL then in short, WSDL is Web service Description Language (i.e. XML file which describes your SOAP Service). WSDL helps to craft SOAP API request Body for ZappySys XML Driver. So Let's get started.

1. Open SoapUI and click SOAP button to create new SOAP Project
2. Enter WSDL URL or File Path of WSDLFor example WSDL for our sample service can be accessed via this URL

   http://www.dneonline.com/calculator.asmx?wsdl

   
   Create new SOAP API Project in SoapUI tool for SOAP API Testing
3. Once WSDL is loaded you will see possible operations you can call for your SOAP Web Service.
4. If your web service requires credentials then you have to configure it. There are two common credential types for public services (SOAP WSS or BASIC )
   1. To use SOAP WSS Credentials select request node and enter UserId, Password, and WSS-PasswordType (PasswordText or PasswordHash)
      
      Configure SOAP WSS Credentials for SoapUI (SOAP API Testing Tool)
   2. To use BASIC Auth Credentials select request node and double-click it. At the bottom click on Auth (Basic) and From Authorization dropdown click Add New and Select Basic.
      Configure Basic Authorization for SoapUI (SOAP API Testing Tool)
5. Now you can test your request first Double-click on the request node to open request editor.
6. Change necessary parameters, remove optional or unwanted parameters. If you want to regenerate request you can click on Recreate default request toolbar icon.
   Create SOAP Request XML (With Optional Parameters)
7. Once your SOAP Request XML is ready, Click the Play button in the toolbar to execute SOAP API Request and Response will appear in Right side panel. Test SOAP API using SoapUI Tool (Change Default XML Body / Parameters, Execute and See Response)

Create DSN using ZappySys XML Driver to call SOAP API

Once you have tested your SOAP API in SoapUI tool, we are ready to use ZappySys XML driver to call SOAP API in your preferred BI tool or Programming language.

1. First open ODBC Data Sources (search ODBC in your start menu or go under ZappySys > ODBC PowerPack > ODBC 64 bit)
2. Goto System DSN Tab (or User DSN which is not used by Service account)
3. Click Add and Select ZappySys XML Driver ZappySys ODBC Driver for XML / SOAP API
4. Configure API URL, Request Method and Request Body as below ZappySys XML Driver - Calling SOAP API - Configure URL, Method, Body
5. (This step is Optional) If your SOAP API requires credentials then Select Connection Type to HTTP and configure as below.
   ZappySys XML Driver - Configure SOAP WSS Credentials or Basic Authorization (Userid, Password)
6. Configure-Request Headers as below (You can get it from Request > Raw tab from SoapUI after you test the request by clicking the Play button) Configure SOAP API Request Headers - ZappySys XML Driver
7. Once credentials entered you can select Filter to extract data from the desired node. Make sure to select array node (see special icon) or select the node which contains all necessary columns if you don't have array node. Select Filter - Extract data from nested XML / SOAP API Response (Denormalize Hierarchy)
8. If prompted select yes to treat selected node as Array (This is helpful when you expect one or more record for selected node) Treat selected node as XML Array Option for SOAP API Response XML

Preview SOAP API Response / Generate SQL Code for SOAP API Call

Once you configure settings for XML Driver now you can preview data or generate example code for desired language (e.g. C#, Python, Java, SQL Server). Go to Preview tab and you will see default query generated based on settings you entered in previous sections. Attributes listed in WITH clause are optional. If you omit attribute in WITH clause it will use it from Properties tab.

Preview Data

Preview SOAP API Response in ZappySys XML Driver

Generate Code Option

Generate Example Code for ZappySys Driver

POST data to REST API URL from file in JAVA

Above example was POST data to API URL but what if your Request Body is large and you have saved that to file? Well here is the way to get your request body from a file (Use @ symbol before path).

SELECT * FROM $
WITH 
(METHOD='POST' 
,HEADER='Content-Type:text/plain || x-hdr1:AAA'
,SRC='http://httpbin.org/post'
,BODY='@c:\files\dump.xml'
,IsMultiPart='True'
)

REST API Pagination in JAVA

SELECT * FROM $
WITH(
SRC=@'https://zappysys.com/downloads/files/test/pagination_nextlink_inarray_1.json'
,NextUrlAttributeOrExpr = '$.nextlink'  --keep reading until this attribute is missing. If attribute name contains dot then use brackets like this $.['my.attr.name']
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/page-xml.aspx?page=1&mode=DetectBasedOnResponseStatusCode'
,PagingMode='ByUrlParameter'
,PagingByUrlAttributeName='page'
,PagingByUrlEndStrategy='DetectBasedOnResponseStatusCode'
,PagingByUrlCheckResponseStatusCode=401
,IncrementBy=1
)

SELECT * FROM $
WITH (
SRC=@'https://zappysys.com/downloads/files/test/cust-<%page%>.xml'
,PagingMode='ByUrlPath'
,PagingByUrlAttributeName='<%page%>'
,PagingByUrlEndStrategy='DetectBasedOnRecordCount'
,IncrementBy=1
)

SELECT * FROM $
LIMIT 25
WITH(
	 Src='https://wordpress.org/news/wp-json/wp/v2/categories?per_page=10'
	,PagingMode='ByResponseHeaderRfc5988'
	,WaitTimeMs='200' --//wait 200 ms after each request
)

Error Handling in REST API / SOAP

Sometimes errors occur... they just do and there is nothing you can do! Or can you? Actually, in ODBC PowerPack you can handle them in two ways.

METHOD 1 - Using Error Handling Options

When to use?

You may want to use them when your source is a resource located on the Internet; e.g. a file on a website, a file on an FTP server or just a plain API HTTP response. By default, when a remote server returns an error, data retrieval is stopped, an error is raised and no data is given back to you. This might not be always desirable.

Scenario 1

Imagine a scenario, that there is a web server which each day at 12 AM releases a new JSON file with that day's date as filename, e.g. http://www.some-server.com/data/2018-06-20.json. And, of course, you want to download it and use it daily in your Power BI report. But you have a problem: Power BI report data sources are refreshed each hour and you may get HTTP 404 status code (no file was found) when a file is not released yet. Which consequentially means other data sources won't be updated as well and you will see old and cached data on the report. That's where you could use Continue on any error or Continue when Url is invalid or missing (404 Errors) to avoid an error being raised and let other data sources to be updated.

Scenario 2

Another scenario is when you expect a web server to raise some kind of HTTP error when accessing a URL. You don't want ODBC Data Source to raise an error but instead, you want to get response data. That's where you can use Continue on any error or alike together with  Get response data on error to continue on an error and get the data:

METHOD 2 - Using Connection [Retry Settings]

Another scenario you may run into is a buggy web server. You ask it to give you some file or data and it, like a snotty kid, just doesn't give it to you! You have to ask twice or thrice before it does its job. If that's the case, you have to retry HTTP requests using Connection:

REST API / SOAP Connection Types in JAVA (OAuth / HTTP)

If you need to authenticate or authorize your user to access a web resource, you will need to use one of the Connections:

• HTTP
• OAuth

HTTP Connection

Use HTTP Connection for simple Windows, Basic, NTLM or Kerberos authentication. Just fill in a username and a password and you are good to go!

You can also use HTTP Connection for more sophisticated authentication like:

• SOAP WSS (when accessing a SOAP WebService)
• Static Token / API Key (when need to pass an API key in HTTP header)
• Dynamic Token (same as Static Token method except that each time you need to log in and retrieve a fresh API key)
• JWT Token (As per RFC 7519)

OAuth

If you are trying to access REST API resource, it is a huge chance, you will need to use OAuth Connection. Read this article to understand how OAuth authentication and authorization works and how to use it (article originally was written for SSIS PowerPack, but the concepts and UI stay the same): 
https://zappysys.com/blog/rest-api-authentication-with-oauth-2-0-using-ssis/

Performance consideration for Web API Calls

SELECT * FROM value
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json&$filter=Country eq ''USA'''
	,DataFormat='Odata'
)

-- Slow query - Client-side filtering
SELECT * FROM value
WHERE Country ='USA'
WITH(
	 Src='https://services.odata.org/V3/Northwind/Northwind.svc/Customers?$format=json'
	,DataFormat='Odata'
)

SELECT * FROM $
WITH 
(  SRC='https://myhost.com/some-api'
  ,CachingMode='All'  --cache metadata and data rows both
  ,CacheStorage='File' --or Memory
  ,CacheFileLocation='c:\temp\myquery.cache'
  ,CacheEntryTtl=300 --cache for 300 seconds
)

SELECT * FROM $ 
LIMIT 10 --//add this just to test how fast you can get 10 rows
WITH(
  Filter='$.LargeArray[*]--FAST' --//Adding --FAST option turn on STREAM mode (large files)
 ,SRC='https://zappysys.com/downloads/files/test/large_file_100k_largearray_prop.json.gz'
 --,SRC='c:\data\large_file.json.gz'
 ,IncludeParentColumns='False'  --//This Must be OFF for STREAM mode (read very large files)
 ,FileCompressionType='GZip' --Zip or None (Zip format only available for Local files)
)

Other considerations for Web API calls in JAVA

Conclusion

To conclude, in this tutorial, we learned how to access the REST API using Java. We created a ZappySys ODBC connection and then used a ZappySys Gateway to connect to it.

References

• ZappySys ODBC PowerPack
• Java example ZappySys ODBC PowerPack
• Add connection in Netbeans 8 to the SQL Server 2012 database.

The post How to connect Java to REST API (JSON / SOAP XML) appeared first on ZappySys Blog.

In this example, we will show how to read emails by calling Gmail REST API using SSIS. To illustrate, we will use ZappySys SSIS PowerPack, which includes several tasks to import/export data from multiples sources to multiple destinations like flat files, Azure, AWS, databases, Office files and more.

Requirements

1. First, you will need ZappySys SSIS PowerPack installed.
2. Secondly, you will need SSDT installed.

Getting started

How to get information about Gmail with REST API

1. First, you will need to create a developer application for Gmail. The following tutorial shows how to create an application for Google Drive, but the steps are similar:
   • How to register Google OAuth Application (Get ClientID and ClientSecret)
2. Secondly, go to SSDT and create a new SQL Server Integration Project.
3. Subsequently, drag and drop a Data Flow task and double-click it:
   

   Drag and drop Data Flow

4. Also, in the connection manager create a new ZS-OAuth connection. Select Google as the OAuth provider. In scopes, add the following:
   

   https://mail.google.com/

   In addition, press generate token and add your credentials if necessary:
   

   OAuth credentials to connect to Gmail

    

5. To illustrate, the following example shows how to get the Gmail messages from an account. Check the option use credentials and select the OAuth connection just created. To get your current email messages, you can use the following URL:

   https://www.googleapis.com/gmail/v1/users/me/messages/

   The filter used is the following:

   $.messages[*].id

   The result displayed shows the ids of the messages. You can see the id of the message in Gmail in the URL when you click a message:

   

   JSON Source Rest API

    

6. If you want to check to information of a specific message you can use the following URL.

   https://www.googleapis.com/gmail/v1/users/me/messages/1632106341364d6f

   Where 1632106341364d6f is the message id. The filter used will be the following:

   $.id

   

   Gmail message information

   The query shows the snippet (body of the message), the history id, size estimated and more.

7. After this, we can use the following filter:

   $.payload.headers[*]

   This will show the sender, the receiver, subject, references, message ID and more information:

   

   Information about Gmail

How to get label information in Gmail with REST API

1. First, you can use the following URL to get the Gmail labels:
   

   https://www.googleapis.com/gmail/v1/users/me/labels/

   Filter:
   

   $.labels[*].id

    
2. Also, the information displayed is the following:
   

   Labels in Gmail

3. Finally, the labels are the same than in Gmail:
   

   Gmails labels, categories

How to get threads information in Gmail with REST API

1. Finally, you can also get the thread information using the following URL:
   

   https://www.googleapis.com/gmail/v1/users/me/threads

   The filter used will be the following:
   

   $.threads[*].id

    
2. The information displayed will be the following:
   

   Get Threads from Gmail

3. If you press the Preview button, you will see the following information:
   

   Thread information using Gmail

How to export the data from Gmail to a CSV file

1. First, to export the data, we will use the ZS Trash Destination:
   

   Export data from Gmail to a CSV file

2. Finally, in the Trash Destination specify the path to store the data from Gmail:
   

   CSV destination to store Gmail info

Conclusion

To conclude, we can say that ZappySys SSIS PowerPack is a powerful tool where you can get information from Gmail. This information can be exported to Excel, SQL Server, XML, JSON or any other format using SSIS.  Download FREE Trial of SSIS PowerPack to find out more use cases like this.

References

For more information about Gmail, you can refer to these links:

• API Reference
• How to read / write Google SpreadSheet using SSIS

The post Calling Gmail REST API using SSIS – Read Emails appeared first on ZappySys Blog.