Elasticsearch

Available when using an authoring Data Collector version 4.0.0 or later.

To create an Elasticsearch connection, one of the Elasticsearch stage libraries, streamsets-datacollector-elasticsearch_<version>-lib, must be installed on the selected authoring Data Collector.

Tip: To view the complete list of supported stage libraries, expand the list of libraries next to the Test Connection button when you create or edit a connection.

For a description of the Elasticsearch connection properties, see Elasticsearch Connection Properties.

After you create an Elasticsearch connection, you can use the connection in the following stages and locations:


Engine	Stages and Locations
Data Collector 4.0.0 or later	Elasticsearch origin Elasticsearch destination Write to Elasticsearch error record handling configured for a pipeline
Transformer 4.0.0 or later	Elasticsearch destination

Elasticsearch Security

When security is enabled for the Elasticsearch cluster, you must specify the authentication method:

Basic: Use Basic authentication for Elasticsearch clusters outside of Amazon OpenSearch Service. With Basic authentication, the connection passes the Elasticsearch user name and password.

AWS Signature V4

Use AWS Signature V4 authentication for Elasticsearch clusters within Amazon OpenSearch Service. The connection must sign HTTP requests with Amazon Web Services credentials. For details, see the Amazon OpenSearch Service documentation. Use one of the following methods to sign with AWS credentials:

Instance profile: When the execution engine - Data Collector or Transformer - runs on an Amazon EC2 instance that has an associated instance profile, the engine uses the instance profile credentials to automatically authenticate with AWS.; To use an instance profile, do not configure the Access Key ID and Secret Access Key properties.; For more information about associating an instance profile with an EC2 instance, see the Amazon EC2 documentation.
AWS access key pair: When the execution engine does not run on an Amazon EC2 instance or when the EC2 instance doesn’t have an instance profile, you must specify the Access Key ID and Secret Access Key properties.; Tip: To secure sensitive information, you can use credential stores or runtime resources.

Elasticsearch Connection Properties

When creating an Elasticsearch connection, configure the following properties on the Elasticsearch tab:


Elasticsearch Property	Description
HTTP URLs	Comma-separated list of HTTP or HTTPS URLs used to connect to each Elasticsearch server in the cluster. Use the following format: `http://<host1>,http://<host2>` You can specify a port number in the URLs to override the default port defined in the HTTP Port property, as follows: `http://<host1>:<port>,http://<host2>:<port>` When a port number is defined in both this property and in the HTTP Port property, the port in this property takes precedence. For example, if you define this property as follows: `http://server1,http://server2:1234` And you define the default HTTP Port property as 9200, then `server1` uses the default port of 9200 and `server2` uses the port 1234.
HTTP Port	Default port number to use for URLs that do not include a port. The default HTTP port is 9200. The default HTTPS port is 443.
Use Security	Specifies whether security is enabled on the Elasticsearch cluster.

Configure the following properties on the Security tab:


Security Property	Description
Mode	Authentication method to use: Basic - Authenticate with Elasticsearch user name and password. Select this option for Elasticsearch clusters outside of Amazon OpenSearch Service. AWS Signature V4 - Authenticate with AWS. Select this option for Elasticsearch clusters within Amazon OpenSearch Service.
User Name	Elasticsearch user name. Available when using Basic authentication.
Password	Password for the user account. Available when using Basic authentication.
Region	Amazon Web Services region that hosts the Elasticsearch domain. Available when using AWS Signature V4 authentication.
Access Key ID	AWS access key ID. Required when not using instance profile credentials. Available when using AWS Signature V4 authentication.
Secret Access Key	AWS secret access key. Required when not using instance profile credentials. Available when using AWS Signature V4 authentication.
Enable SSL	Enables the use of SSL.
SSL Truststore Path	Location of the truststore file. Configuring this property is the equivalent to configuring the shield.ssl.truststore.path Elasticsearch property. Not necessary for Elastic Cloud clusters.
SSL Truststore Password	Password for the truststore file. Configuring this property is the equivalent to configuring the shield.ssl.truststore.password Elasticsearch property. Not necessary for Elastic Cloud clusters.