Elasticsearch
Available when using an authoring Data Collector version 4.0.0 or later.
streamsets-datacollector-elasticsearch_<version>-lib
, must be
installed on the selected authoring Data Collector.For a description of the Elasticsearch connection properties, see Elasticsearch Connection Properties.
Engine | Stages and Locations |
---|---|
Data Collector 4.0.0 or later |
|
Transformer 4.0.0 or later | Elasticsearch destination |
Elasticsearch Security
When security is enabled for the Elasticsearch cluster, you must specify the authentication method:
- Basic
- Use Basic authentication for Elasticsearch clusters outside of Amazon Elasticsearch Service. With Basic authentication, the connection passes the Elasticsearch user name and password.
- AWS Signature V4
- Use AWS Signature V4 authentication for
Elasticsearch clusters within Amazon Elasticsearch Service. The
connection must sign HTTP requests with Amazon Web Services credentials. For
details, see the Amazon Elasticsearch Service
documentation. Use one of the following methods to sign with AWS credentials:
- Instance profile
- When the execution engine - Data Collector or Transformer - runs on an Amazon EC2 instance that has an associated instance profile, the engine uses the instance profile credentials to automatically authenticate with AWS.
- AWS access key pair
- When the execution engine does not run on an Amazon EC2 instance or when the EC2 instance doesn’t have an instance profile, you must specify the Access Key ID and Secret Access Key properties.
Elasticsearch Connection Properties
When creating an Elasticsearch connection, configure the following properties on the Elasticsearch tab:
Elasticsearch Property | Description |
---|---|
HTTP URLs | Comma-separated list of HTTP or HTTPS URLs used to connect to each
Elasticsearch server in the cluster. Use the following format:
You can specify a port number in the URLs to override the default port defined in the HTTP Port property, as follows:
When a port number is defined in both this property and in the HTTP Port property, the port in this property takes precedence. For example, if you define this property as follows:
And you define the default HTTP Port property as 9200, then server1 uses the default port of 9200 and server2 uses the port 1234. |
HTTP Port | Default port number to use for URLs that do not include a port. The default HTTP port is 9200. The default HTTPS port is 443. |
Use Security | Specifies whether security is enabled on the Elasticsearch cluster. |
Configure the following properties on the Security tab:
Security Property | Description |
---|---|
Mode | Authentication method to use:
|
User Name | Elasticsearch user name. Available when using Basic authentication. |
Password | Password for the user account. Available when using Basic authentication. |
Region | Amazon Web Services region that hosts the Elasticsearch
domain. Available when using AWS Signature V4 authentication. |
Access Key ID | AWS access key ID. Required when
not using instance profile credentials. Available when using AWS Signature V4 authentication. |
Secret Access Key | AWS secret access key.
Required when not using instance profile credentials. Available when using AWS Signature V4 authentication. |
Enable SSL | Enables the use of SSL. |
SSL Truststore Path | Location of the truststore file. Configuring this property is the equivalent to configuring the shield.ssl.truststore.path Elasticsearch property. Not necessary for Elastic Cloud clusters. |
SSL Truststore Password | Password for the truststore file. Configuring this property is the equivalent to configuring the shield.ssl.truststore.password Elasticsearch property. Not necessary for Elastic Cloud clusters. |