Elasticsearch
Available when using an authoring Data Collector version 3.21.0 or later.
streamsets-datacollector-elasticsearch_<version>-lib
, must be
installed on the selected authoring Data Collector.For a description of the Elasticsearch connection properties, see Elasticsearch Connection Properties.
Engine | Stages and Locations |
---|---|
Data Collector 3.21.0 or later |
|
Transformer 3.18.0 or later | Elasticsearch destination |
Elasticsearch Security
- Basic
- Use Basic authentication for Elasticsearch clusters outside of Amazon OpenSearch Service. With Basic authentication, the connection passes the Elasticsearch user name and password.
- AWS Signature V4
- Use
AWS Signature V4 authentication for Elasticsearch clusters within Amazon
OpenSearch Service. The connection must sign HTTP requests with
Amazon Web Services credentials. For details, see the Amazon OpenSearch Service
documentation. Use one of the following methods to sign with AWS
credentials:
- Instance profile
- When the execution engine - Data Collector or Transformer - runs on an Amazon EC2 instance that has an associated instance profile, the engine uses the instance profile credentials to automatically authenticate with AWS.
- AWS access key pair
- When the execution engine does not run on an Amazon EC2 instance or when the EC2 instance doesn’t have an instance profile, you must specify the Access Key ID and Secret Access Key properties.
Elasticsearch Connection Properties
When creating an Elasticsearch connection, configure the following properties on the Elasticsearch tab:
Elasticsearch Property | Description |
---|---|
HTTP URLs | Comma-separated list of HTTP or HTTPS URLs used to connect to each
Elasticsearch server in the cluster. Use the following format:
You can specify a port number in the URLs to override the default port defined in the HTTP Port property, as follows:
When a port number is defined in both this property and in the HTTP Port property, the port in this property takes precedence. For example, if you define this property as follows:
And you define the default HTTP Port property as 9200, then server1 uses the default port of 9200 and server2 uses the port 1234. |
HTTP Port | Default port number to use for URLs that do not include a port. The default HTTP port is 9200. The default HTTPS port is 443. |
Use Security | Specifies whether security is enabled on the Elasticsearch cluster. |
Configure the following properties on the Security tab:
Security Property | Description |
---|---|
Mode | Authentication method to use:
|
User Name | Elasticsearch user name. Available when using Basic authentication. |
Password | Password for the user account. Available when using Basic authentication. |
Region | Amazon Web Services region that hosts the Elasticsearch
domain. Available when using AWS Signature V4 authentication. |
Access Key ID | AWS access key ID. Required when
not using instance profile credentials. Available when using AWS Signature V4 authentication. |
Secret Access Key | AWS secret access key.
Required when not using instance profile credentials. Available when using AWS Signature V4 authentication. |
Enable SSL | Enables the use of SSL. |
SSL Truststore Path | Location of the truststore file. Configuring this property is the equivalent to configuring the shield.ssl.truststore.path Elasticsearch property. Not necessary for Elastic Cloud clusters. |
SSL Truststore Password | Password for the truststore file. Configuring this property is the equivalent to configuring the shield.ssl.truststore.password Elasticsearch property. Not necessary for Elastic Cloud clusters. |