Accessing Vault Secrets with Vault Functions (Deprecated)

Data Collector can use Vault functions to access information, a.k.a. secrets, stored in Hashicorp Vault. However, the Vault functions are now deprecated and will be removed in a future release. We recommend using the credential functions available with the Vault credential store integration.

You can use Vault secrets in place of username and password properties, similar properties such as AWS access key IDs and secret access keys, and in HTTP headers and bodies when using HTTPS.

To access Vault secrets using the Vault functions, configure properties for Data Collector to access Vault. Then, use Vault functions in the pipeline to access Vault at run time.

To access information stored in Vault, perform the following steps:

Configure the Vault connection properties.
In the pipeline, use Vault functions to access Vault secrets.

Note: This documentation includes details about Hashicorp Vault to simplify the installation and configuration process. For more information, see the Vault documentation.

Step 1. Configure Vault Properties

To enable Data Collector to connect to Vault, configure the following files located in the $SDC_CONF directory:

Vault properties file

The vault.properties file contains a single Vault EL property that you must uncomment to use the deprecated Vault functions.

Credential stores properties file

The credential-stores.properties file contains the remaining Vault properties. The Vault server URL, Role ID, and Secret ID are required properties. Configure other properties as needed.

Note: The credentialStore.vault.def and credentialStore.vault.config.pathKey.separator properties are not used by the Vault functions. You can ignore these properties.

Configure the following properties in the file:

Vault Property	Description
credentialStore.vault.config.addr	Required. Vault server URL entered in the following format: https://<host name>:<port number> Use HTTPS to avoid unencrypted communication.
credentialStore.vault.config.role.id	Required. Vault Role ID that Data Collector uses to authenticate with Vault. The Role ID is configured within Vault by your Vault administrator. The Data Collector Vault integration relies on Vault's App Role authentication backend. Important: The App ID authentication backend has been deprecated by Hashicorp and will be removed in a future release. As a result, do not configure the `credentialStore.vault.config.app.id` property for new installations.
credentialStore.vault.config.secret.id	Required. Vault Secret ID that Data Collector uses to authenticate with Vault. The Secret ID is configured within Vault by your Vault administrator. Enter one of the following: Secret ID value. File that contains the Secret ID value. For increased security, store the Secret ID in a separate file and reference the file in the `$SDC_CONF/credential-stores.properties` file as follows: ${file("<filename>")}. By default, the file name is `vault-secret-id` and expected in the `$SDC_CONF` directory. For more information, see Referencing Sensitive Values in Files.
credentialStore.vault.config.lease.renewal.interval.sec	Optional. Seconds to wait before checking for leases that need renewal. Default is 60.
credentialStore.vault.config.lease.expiration.buffer.sec	Optional. Buffer for expiring leases. Data Collector renews leases that expire in less than the specified number of seconds. Default is 120.
credentialStore.vault.config.open.timeout	Optional. Timeout to establish an HTTP connection to Vault in milliseconds. Default is 0 for no limit.
credentialStore.vault.config.proxy.address	Optional. Proxy URL. Configure to use a proxy to access Vault.
credentialStore.vault.config.proxy.port	Optional. Proxy port. Configure to use a proxy to access Vault.
credentialStore.vault.config.proxy.username	Optional. Proxy username. Configure to use a proxy to access Vault.
credentialStore.vault.config.proxy.password	Optional. Proxy password. Configure to use a proxy to access Vault.
credentialStore.vault.config.read.timeout	Optional. Milliseconds to wait for data before timing out. Default is 0 for no limit.
credentialStore.vault.config.ssl.enabled.protocols	Optional. SSL/TLS-enabled protocols. Versions TLSv1.2 or later are recommended. Default is TLSv1.2,TLSv1.3.
credentialStore.vault.config.ssl.truststore.file	Optional. Path to a Java truststore file. Required when using a private CA or certificates not trusted by the Java default truststore.
credentialStore.vault.config.ssl.truststore.password	Optional. Password for the truststore file.
credentialStore.vault.config.ssl.verify	Optional. Whether to verify that the Vault server hostname matches its certificate. Default is true. False is not recommended.
credentialStore.vault.config.ssl.timeout	Optional. Timeout for the SSL/TLS handshake in milliseconds. Default is 0 for no limit.
credentialStore.vault.config.timeout	Optional. Timeout to read from Vault in milliseconds, after a connection has been established. Default is 0 for no limit.

Restart Data Collector to enable the changes to both files.

Step 2. Call Vault from the Pipeline

After enabling Data Collector to access Vault, use the Vault functions in pipeline and stage properties to access Vault secrets.

The expression language provides Vault functions to return Vault secrets. You can use Vault functions in username, password, and similar properties such as AWS access key IDs and secret access keys. You can also use the functions in HTTP headers and bodies when using HTTPS.

You can use the following functions to access Vault secrets:

vault:read(): Use to return the value for the Vault path and key that you provide. Typically, you'll use this function to access secrets.
vault:readWithDelay(): Use to return the value for the specified Vault path and key with a delay. Use this function to incorporate a delay in the response to allow time for other processes to complete.; For example, you should use this function when using the Vault AWS secret backend to generate AWS access credentials based on IAM policies. According to Vault documentation, you might need a delay of 10 seconds or more before the credentials can be used successfully.

For more information, see Miscellaneous Functions.