Group Access to Secrets

As an additional layer of security, you can employ user groups to further limit access to the secrets defined in credential stores.

Data Collector provides two methods to limit access with user groups:
Required group argument in credential functions

Credential functions include a group argument that defines the user group that can access the secret. The group argument ensures that the user who attempts to preview, validate, or start a pipeline that includes a credential function belongs to the group specified in the function. The user must also have execute permission on the pipeline.

When working only with Data Collector, simply specify the group name, such as devops. When working with Control Hub, sSpecify the group using the required naming convention: <group ID>@<organization ID>. For example, devops@MyCompany.

If you do not want to restrict access to a secret, specify the default all group when working only with Data Collector. When working with Control Hub and Data Collector version 3.16.0 or later, you can specify the default group using all or all@<organization ID>. StreamSets recommends using all so that you do not need to modify credential functions when migrating pipelines from Data Collector to Control Hub.
Note: When working with Control Hub and a Data Collector version earlier than 3.16.0, you must use the default all@<organization ID> group.

If Data Collector shuts down while running a pipeline that uses a credential function, Data Collector restarts the pipeline without checking the group access.

Optional group secrets in the credential store

In addition to using the group argument in credential functions, you can configure Data Collector to require group secrets for a credential store.

To require the use of group secrets, in the $SDC_CONF/credential-stores.properties fileData Collector credential store configuration properties, set the credentialStore.<cstore ID>.config.enforceEntryGroup property to true.

A group secret is a secret defined in the credential store that contains a comma-delimited list of Data Collector user groups permitted to access the associated secret.

When the credential store ID requires group secrets, you must define a group secret for every secret that Data Collector accesses in that credential store. The name of the group secret is based on the secret name, as follows: 
<secret name>-groups
When you configure a credential function to call a secret, the user group specified in the credential function must be listed in the associated group secret that is defined in the credential store.
For example, say you enable Data Collector to require group secrets for Azure Key Vault. Then, in a Kafka Multitopic Consumer origin, you use the following expression to access a Base64-encoded keytab in the azure credential store for the origin to use:
${credential:get("azure", "kafkaprod@MyCompany", readkeytab)}
When you run the pipeline, Data Collector validates all of the following:
  • The user who starts the pipeline is in the kafkaprod user group.
  • The readkeytab secret has an associated readkeytab-groups secret defined in the credential store.
  • The readkeytab-groups secret includes the kafkaprod user group.

When Data Collector is not configured to require group secrets, Data Collector validates only the first point, verifying that the user belongs to the specified group.