Reading data in SSIS

In this section we will learn how to configure and use ElasticSearch Connector in API Source to extract data from ElasticSearch.

Video tutorial

This video covers following and more so watch carefully. After watching this video follow the steps described in this article.

• How to download SSIS PowerPack for ElasticSearch integration in SSIS
• How to configure connection for ElasticSearch
• How to read data from ElasticSearch
• Features about SSIS API Source
• Using ElasticSearch Connector in SSIS

Step-by-step instructions

1. Open Visual Studio and click Create a new project.

2. Select Integration Services Project. Enter a name and location for your project, then click OK.

3. From the SSIS Toolbox, drag and drop a Data Flow Task onto the Control Flow surface, and double-click it:

   

4. Make sure you are in the Data Flow Task designer:

   

5. From the SSIS toolbox drag and API Source (Predefined Templates) on the data flow designer surface, and double click on it to edit it:
   

   

6. Select New Connection to create a new connection:
   

   

7. Use a preinstalled ElasticSearch Connector from Popular Connector List or press Search Online radio button to download ElasticSearch Connector. Once downloaded simply use it in the configuration:
   

   ElasticSearch

   

8. Select your authentication scenario below to expand connection configuration steps to:

   • Configure the authentication in ElasticSearch.
   • Enter those details into the API Connection Manager configuration.
   Basic Authentication (UserId/Password)

   ElasticSearch authentication

   Use HTTP Basic Authentication with any Elasticsearch-compatible cluster (self-hosted, Elastic Cloud, OpenSearch, or other managed hosts).

   1. Set Service URL to your cluster base URL (for example https://your-host:9200 or the HTTPS URL from your provider).
   2. Enter User Name and Password for a user that can call the Elasticsearch REST API. Some managed hosts label these as Access Key / Access Secret — use those values in the same fields.
   3. If you use a self-signed certificate and see a trust error, set Ignore certificate related errors to True (only if you trust the server).
   4. Click Test connection.

   Optional — Bonsai example: On Bonsai, open https://app.bonsai.io/clusters/{your-instance-id}/tokens, copy Access Key and Access Secret into User Name and Password, then test. Sample data (if needed): https://{your-cluster-id}.apps.bonsaisearch.net/app/home#/tutorial_directory.

   API Connection Manager configuration

   Just perform these simple steps to finish authentication configuration:

   1. Set Authentication Type to Basic Authentication (UserId/Password) [Http]
   2. Optional step. Modify API Base URL if needed (in most cases default will work).
   3. Fill in all the required parameters and set optional parameters if needed.
   4. Finally, hit OK button:

   ElasticSearch

   Basic Authentication (UserId/Password) [Http]

   http://localhost:9200

   Optional Parameters
   User Name (or Access Key)
   Password (or Access Secret)
   Ignore certificate related errors

   

   Find full details in the ElasticSearch Connector authentication reference.

   Windows Authentication (No Password)

   ElasticSearch authentication

   No instructions available.

   API Connection Manager configuration

   Just perform these simple steps to finish authentication configuration:

   1. Set Authentication Type to Windows Authentication (No Password) [Http]
   2. Optional step. Modify API Base URL if needed (in most cases default will work).
   3. Fill in all the required parameters and set optional parameters if needed.
   4. Finally, hit OK button:

   ElasticSearch

   Windows Authentication (No Password) [Http]

   http://localhost:9200

   Optional Parameters
   Ignore certificate related errors

   

   Find full details in the ElasticSearch Connector authentication reference.

9. Select the desired endpoint, change/pass the properties values, and click on Preview Data button to make the API call.

   API Source - ElasticSearch

   Read and write Elasticsearch data effortlessly. Integrate, manage, and automate indexes and documents — almost no coding required.

   

10. That's it! We are done! Just in a few clicks we configured the call to ElasticSearch using ElasticSearch Connector.

    You can load the source data into your desired destination using the Upsert Destination , which supports SQL Server, PostgreSQL, and Amazon Redshift. We also offer other destinations such as CSV , Excel , Azure Table , Salesforce , and more . You can check out our SSIS PowerPack Tasks and components for more options. (*loaded in Trash Destination)