ElasticSearch Connector for C#

Read / write ElasticSearch data inside your app; perform many ElasticSearch operations without coding, just use easy to use high performance API Connector for ElasticSearch

In this article you will learn how to quickly and efficiently integrate ElasticSearch data in C#. We will use high-performance ElasticSearch Connector to easily connect to ElasticSearch and then access the data inside C#.

Let's follow the steps below to see how we can accomplish that!

Download Documentation

ElasticSearch Connector for C# is based on ZappySys API Driver which is part of ODBC PowerPack. It is a collection of high-performance ODBC drivers that enable you to integrate data in SQL Server, SSIS, a programming language, or any other ODBC-compatible application. ODBC PowerPack supports various file formats, sources and destinations, including REST/SOAP API, SFTP/FTP, storage services, and plain files, to mention a few.

Connect to ElasticSearch in other apps

Connect to ElasticSearch in SSIS

Connect to ElasticSearch in SQL Server

Connect to ElasticSearch in Power BI

Connect to ElasticSearch in SSRS

Connect to ElasticSearch in Informatica

Connect to ElasticSearch in MS Access

Connect to ElasticSearch in MS Excel

Connect to ElasticSearch in SSAS

Connect to ElasticSearch in Python

Connect to ElasticSearch in JAVA

Connect to ElasticSearch in Tableau

Connect to ElasticSearch in SAP Crystal Reports

Connect to ElasticSearch in Azure Data Factory (Pipeline)

Connect to ElasticSearch in Azure Data Factory (SSIS)

Connect to ElasticSearch in Talend Studio

Connect to ElasticSearch in UiPath

Connect to ElasticSearch in PowerShell

Connect to ElasticSearch in ODBC

Create ODBC Data Source (DSN) based on ZappySys API Driver

Step-by-step instructions

To get data from ElasticSearch using C# we first need to create a DSN (Data Source) which will access data from ElasticSearch. We will later be able to read data using C#. Perform these steps:

Download and install ODBC PowerPack.
Open ODBC Data Sources (x64):
Create a User data source (User DSN) based on ZappySys API Driver

ZappySys API Driver
- Create and use User DSN if the client application is run under a User Account. This is an ideal option in design-time, when developing a solution, e.g. in Visual Studio 2019. Use it for both type of applications - 64-bit and 32-bit.
- Create and use System DSN if the client application is launched under a System Account, e.g. as a Windows Service. Usually, this is an ideal option to use in a production environment. Use ODBC Data Source Administrator (32-bit), instead of 64-bit version, if Windows Service is a 32-bit application.
When the Configuration window appears give your data source a name if you haven't done that already, then select "ElasticSearch" from the list of Popular Connectors. If "ElasticSearch" is not present in the list, then click "Search Online" and download it. Then set the path to the location where you downloaded it. Finally, click Continue >> to proceed with configuring the DSN:

ElasticsearchDSN

ElasticSearch

Now it's time to configure the Connection Manager. Select Authentication Type, e.g. Token Authentication. Then select API Base URL (in most cases, the default one is the right one). More info is available in the Authentication section.

Basic Authentication (UserId/Password) [Http]

Steps how to get and use ElasticSearch credentials

For Local / Hosted Instance by you

Get your userid / password and enter on the connection UI

For Managed Instance (By Bonsai search)

If your instance is hosted by bonsai then perform these steps to get your credentials for API call

Go to https://app.bonsai.io/clusters/{your-instance-id}/tokens
Copy Access Key and Access Secret and enter on the connection UI. Click Test connection.
If your Cluster has no data you can generate sample data by visiting this URL and click Add Sample Data https://{your-cluster-id}.apps.bonsaisearch.net/app/home#/tutorial_directory

Fill in all required parameters and set optional parameters if needed:

ElasticsearchDSN

ElasticSearch

Basic Authentication (UserId/Password) [Http]

http://localhost:9200

Optional Parameters
User Name (or Access Key)
Password (or Access Secret)
Ignore certificate related errors

Windows Authentication (No Password) [Http]

Fill in all required parameters and set optional parameters if needed:

ElasticsearchDSN

ElasticSearch

Windows Authentication (No Password) [Http]

http://localhost:9200

Optional Parameters
Ignore certificate related errors

Once the data source connection has been configured, it's time to configure the SQL query. Select the Preview tab and then click Query Builder button to configure the SQL query:

ZappySys API Driver - ElasticSearch

Read / write ElasticSearch data inside your app; perform many ElasticSearch operations without coding, just use easy to use high performance API Connector for ElasticSearch

ElasticsearchDSN
Start by selecting the Table or Endpoint you are interested in and then configure the parameters. This will generate a query that we will use in C# to retrieve data from ElasticSearch. Hit OK button to use this query in the next step.
```
SELECT * FROM Indexes
```
Some parameters configured in this window will be passed to the ElasticSearch API, e.g. filtering parameters. It means that filtering will be done on the server side (instead of the client side), enabling you to get only the meaningful data much faster.
Now hit Preview Data button to preview the data using the generated SQL query. If you are satisfied with the result, use this query in C#:
ZappySys API Driver - ElasticSearch

Read / write ElasticSearch data inside your app; perform many ElasticSearch operations without coding, just use easy to use high performance API Connector for ElasticSearch

ElasticsearchDSN
```
SELECT * FROM Indexes
```
You can also access data quickly from the tables dropdown by selecting <Select table>.

A WHERE clause, LIMIT keyword will be performed on the client side, meaning that the whole result set will be retrieved from the ElasticSearch API first, and only then the filtering will be applied to the data. If possible, it is recommended to use parameters in Query Builder to filter the data on the server side (in ElasticSearch servers).
Click OK to finish creating the data source.

Video Tutorial

Watch this video on YouTube

Read data in C#

Using ODBC DSN

Create a new Console project and use this code to read the data:

"ElasticsearchDSN";
Press F5 to run the code and read the data:

Here is the code in text format:

using System;
using System.Data.Odbc;

namespace ConsoleApp {
    class Program {
        static void Main() {
            var dsn = "ElasticsearchDSN";
            using (var conn = new OdbcConnection(String.Format("DSN={0}", dsn)))
            {
                conn.Open();
                var cmd = new OdbcCommand("SELECT * FROM Products", conn);
                
                //Increase the timeout duration from the default 30 seconds, which may be insufficient in certain scenarios
                cmd.CommandTimeout=600; // 600-Seconds
                
                var rdr = cmd.ExecuteReader();
                while (rdr.Read())
                {
                    for (int i = 0; i < rdr.FieldCount; i++)
                    {
                            Console.Write("{0}\t", rdr[i]);
                    }
                    Console.WriteLine();
                }
            }
            Console.ReadKey();
        }
    }
}

Using a full ODBC connection string

If you want to avoid being dependent on a DSN and creating multiple DSNs for each platform (x86, x64), then you can use a fully qualified connection string. Simply go to your DSN and copy the Connection String:

Open ODBC data source configuration and click Copy settings:

ZappySys API Driver - Configuration [Version: 2.0.1.10418]

ZappySys API Driver - ElasticSearch

Read / write ElasticSearch data inside your app; perform many ElasticSearch operations without coding, just use easy to use high performance API Connector for ElasticSearch

ElasticsearchDSN
The window opens, telling us the connection string was successfully copied to the clipboard:

Then in your C# code, pass the connection string as an argument when calling the OdbcConnection object's constructor, for example:

var connectionString = @"DRIVER={ZappySys API Driver};ServiceUrl=http://localhost:9200;UserName=[$UserName$];Password=[$Password$]";

using (var conn = new OdbcConnection(connectionString))
{
    // your code goes here
}

Limitations of using a full connection string

Despite using a full ODBC connection string may be very convenient it comes with a limitation: it's length is limited to 1024 symbols (or sometimes more). It usually happens when API provider generates a very long Refresh Token when OAuth is at play. If you are using such a long ODBC connection string, you may get this error:

"Connection string exceeds maximum allowed length of 1024"

But there is a solution to this by storing the full connection string in a file. Follow the steps below to achieve this:

Open your ODBC data source.
Click Copy settings button to copy a full connection string (see the previous section on how to accomplish that).
Then create a new file, let's say, in C:\temp\odbc-connection-string.txt.
Continue by pasting the copied connection string into a newly created file and save it.
Finally, the last step! Just construct a shorter ODBC connection string using this format:
```
DRIVER={ZappySys API Driver};SettingsFile=C:\temp\odbc-connection-string.txt
```
Our troubles are over! Now you should be able to use this connection string in C# with no problems.

This feature requires ODBC PowerPack v1.9.0 or later.