Reading ElasticSearch data in SQL Server using ZappySys Data Gateway

This section covers steps for setting up ElasticSearch Connector in the ZappySys Data Gateway data source (based on API Driver) to read ElasticSearch data in SQL Server:

Create Data Source in ZappySys Data Gateway based on API Driver

1. Download and install ZappySys ODBC PowerPack.

2. Search for gateway in start menu and Open ZappySys Data Gateway:
   

3. Go to Users Tab to add our first Gateway user. Click Add; we will give it a name tdsuser and enter password you like to give. Check Admin option and click OK to save. We will use these details later when we create linked server:
   

4. Now we are ready to add a data source. Click Add, give data source a name (Copy this name somewhere, we will need it later) and then select Native - ZappySys API Driver. Finally, click OK. And it will create the Data Set for it and open the ZS driver UI.

   ElasticsearchDSN

   
   

5. When the Configuration window appears give your data source a name if you haven't done that already, then select "ElasticSearch" from the list of Popular Connectors. If "ElasticSearch" is not present in the list, then click "Search Online" and download it. Then set the path to the location where you downloaded it. Finally, click Continue >> to proceed with configuring the DSN:

   ElasticsearchDSN

   ElasticSearch

   

6. Now it's time to configure the Connection Manager. Select Authentication Type, e.g. Token Authentication. Then select API Base URL (in most cases, the default one is the right one). More info is available in the Authentication section.

    Basic Authentication (UserId/Password) [Http]

   Steps how to get and use ElasticSearch credentials
   

   For Local / Hosted Instance by you

   1. Get your userid / password and enter on the connection UI

   For Managed Instance (By Bonsai search)

   If your instance is hosted by bonsai then perform these steps to get your credentials for API call

   1. Go to https://app.bonsai.io/clusters/{your-instance-id}/tokens
   2. Copy Access Key and Access Secret and enter on the connection UI. Click Test connection.
   3. If your Cluster has no data you can generate sample data by visiting this URL and click Add Sample Data https://{your-cluster-id}.apps.bonsaisearch.net/app/home#/tutorial_directory

   Fill in all required parameters and set optional parameters if needed:

   ElasticsearchDSN

   ElasticSearch

   Basic Authentication (UserId/Password) [Http]

   http://localhost:9200

   Optional Parameters
   User Name (or Access Key)
   Password (or Access Secret)
   Ignore certificate related errors

   

    Windows Authentication (No Password) [Http]

   Fill in all required parameters and set optional parameters if needed:

   ElasticsearchDSN

   ElasticSearch

   Windows Authentication (No Password) [Http]

   http://localhost:9200

   Optional Parameters
   Ignore certificate related errors

   

7. Once the data source has been configured, you can preview data. Select the Preview tab and use settings similar to the following to preview data:
   

8. Click OK to finish creating the data source.

Read data in SQL Server

1. First, let's open SQL Server Management Studio, create a new Linked Server, and start configuring it:

   

2. Then click on Security option and configure username we created in ZappySys Data Gateway in one of the previous steps:
   

3. Optional step. Under the Server Options, Enable RPC and RPC Out and Disable Promotion of Distributed Transactions(MSDTC).

   

   You need to enable RPC Out if you plan to use EXEC(...) AT [LINKED_SERVER_TO_ELASTICSEARCH_IN_DATA_GATEWAY] rather than OPENQUERY.
   If don't enabled it, you will encounter the Server 'LINKED_SERVER_TO_ELASTICSEARCH_IN_DATA_GATEWAY' is not configured for RPC error.

   Query Example:

   EXEC('SELECT * from Products') AT [LINKED_SERVER_TO_ELASTICSEARCH_IN_DATA_GATEWAY]

   ---

   If you plan to use 'INSERT INTO...EXEC(....) AT [LINKED_SERVER_TO_ELASTICSEARCH_IN_DATA_GATEWAY]' in that case you need to Disable Promotion of Distributed Transactions(MSDTC).
   If don't disabled it, you will encounter the The operation could not be performed because OLE DB provider "SQLNCLI11" for linked server "MY_LINKED_SERVER_NAME" was unable to begin a distributed transaction. error.

   Query Example:

   INSERT INTO dbo.Products
   EXEC('SELECT * FROM Products') AT [LINKED_SERVER_TO_ELASTICSEARCH_IN_DATA_GATEWAY]

4. Finally, open a new query and execute a query we saved in one of the previous steps:
   

   SELECT * FROM OPENQUERY([LINKED_SERVER_TO_ELASTICSEARCH_IN_DATA_GATEWAY], 'SELECT * FROM Products')

   

   SELECT * FROM OPENQUERY([LINKED_SERVER_TO_ELASTICSEARCH_IN_DATA_GATEWAY], 'SELECT * FROM Products')

ElasticSearch Connector Examples

The ZappySys API Driver is a user-friendly interface designed to facilitate the seamless integration of various applications with the ElasticSearch API. With its intuitive design and robust functionality, the ZappySys API Driver simplifies the process of configuring specific API endpoints to efficiently read or write data from ElasticSearch.

Click here to find more ElasticSearch Connector examples designed for seamless integration with the ZappySys API ODBC Driver under ODBC Data Source (36/64) or ZappySys Data Gateway, enhancing your ability to connect and interact with Prebuilt Connectors effectively.

Consume Data inside your App / Programming Language

ODBC inside ETL / Reporting / BI Tools

• Power BI
• Tableau
• Excel
• Informatica PowerCenter
• QlikView
• SQL Server / Linked Server
• SSRS (SQL Server Reporting Services)

ODBC inside Programming Languages

• C# (CSharp)
• JAVA
• PHP
• PowerShell
• Python
• T-SQL / SQL Server Linked Server

Key features of the ZappySys API Driver include:

The API ODBC driver facilitates the reading and writing of data from numerous popular online services (refer to the complete list here) using familiar SQL language without learning complexity of REST API calls. The driver allows querying nested structure and output as a flat table. You can also create your own ODBC / Data Gateway API connector file and use it with this driver.

1. Intuitive Configuration: The interface is designed to be user-friendly, enabling users to easily set up the specific API endpoints within ElasticSearch without requiring extensive technical expertise or programming knowledge.

2. Customizable Endpoint Setup: Users can conveniently configure the API endpoint settings, including the HTTP request method, endpoint URL, and any necessary parameters, to precisely target the desired data within ElasticSearch.

3. Data Manipulation Capabilities: The ZappySys API Driver allows for seamless data retrieval and writing, enabling users to fetch data from ElasticSearch and perform various data manipulation operations as needed, all through an intuitive and straightforward interface.

4. Secure Authentication Integration: The driver provides secure authentication integration, allowing users to securely connect to the ElasticSearch API by inputting the necessary authentication credentials, such as API tokens or other authentication keys.

5. Error Handling Support: The interface is equipped with comprehensive error handling support, ensuring that any errors or exceptions encountered during the data retrieval or writing process are efficiently managed and appropriately communicated to users for prompt resolution.

6. Data Visualization and Reporting: The ZappySys API Driver facilitates the seamless processing and presentation of the retrieved data from ElasticSearch, enabling users to generate comprehensive reports and visualizations for further analysis and decision-making purposes.

Overall, the ZappySys API Driver serves as a powerful tool for streamlining the integration of applications with ElasticSearch, providing users with a convenient and efficient way to access and manage data, all through a user-friendly and intuitive interface.