Web Client

Available when using an authoring Data Collector version 5.11.0 or later.

To create a Web Client connection, the appropriate Web Client stage library must be installed on the selected authoring Data Collector:
  • Data Collector 5.11.0 or later - Requires the OkHttp Web Client stage library, streamsets-datacollector-webclient-impl-okhttp-lib.

For a description of the Web Client connection properties, see Web Client Connection Properties.

After you create a Web Client connection, you can use the connection in the following stages:
Engine Stages
Data Collector 5.11.0 or later
  • Web Client origin
  • Web Client processor
  • Web Client destination

Web Client Connection Properties

When creating a Web Client connection, configure the following properties on the Connection tab:

Connection Property Description
Base URL Fixed base URL or request endpoint of the requested resource. For example, http://localhost:8000 or http://localhost:8000/basic/authenticate.
Authentication Scheme

Determines the authentication type used to connect to the server:

  • None - Performs no authentication.

  • Basic - Uses basic authentication. Requires a username and password.

    Use with HTTPS to avoid passing unencrypted credentials.

  • Digest - Uses digest authentication. Requires a username and password.

  • Bearer - Uses bearer authentication. Requires a username and password.

  • OAuth 1 - Uses OAuth 1.0 authentication. Requires OAuth credentials.

  • OAuth 2 - Uses OAuth 2.0 authentication. Requires OAuth credentials.

Keystore Management Determines the authentication and encryption used to connect to the HTTP server.
  • Automatic (for most HTTP and HTTPS requests) - Uses automatically-generated keystore and truststore configurations.
  • Manual (for manually configured HTTPS requests) - Manually configure the keystore and truststore to use.
Keystore Location

Where to load the keystore from:

  • Local - Loads the keystore from a local file.

  • Remote - Loads the keystore from the provided key and certificate chain.

Required for manual keystore management.

Keystore File

Path to the local keystore file. Enter an absolute path to the file or enter the following expression to define the file stored in the Data Collector resources directory:

${runtime:resourcesDirPath()}/keystore.jks

Available for manual keystore management.

Keystore Type Type of keystore to use. Use one of the following types:
  • Java Keystore File (JKS)
  • PKCS #12 (p12 file)

Default is Java Keystore File (JKS).

Keystore Password

Password to the keystore file. A password is optional, but recommended.

Private Key Private key used in the remote keystore. Enter a credential function that returns the key or enter the contents of the key.
Certificate Chain Each PEM certificate used in the remote keystore. Enter a credential function that returns the certificate or enter the contents of the certificate.

Using simple or bulk edit mode, click the Add icon to add additional certificates.

Keystore Key Algorithm

Algorithm to manage the keystore.

Default is SunX509.

Truststore File

Path to the local truststore file. Enter an absolute path to the file or enter the following expression to define the file stored in the Data Collector resources directory:

${runtime:resourcesDirPath()}/truststore.jks

By default, no truststore is used.

Truststore Type
Type of truststore to use. Use one of the following types:
  • Java Keystore File (JKS)
  • PKCS #12 (p12 file)

Default is Java Keystore File (JKS).

Truststore Password

Password to the truststore file. A password is optional, but recommended.

Truststore Key Algorithm

Algorithm to manage the truststore.

Default is SunX509.

Required for manual keystore management.

Default Protocol Versions

Use only modern default secure protocol versions.

Available for manual keystore management.

Default Cipher Suites

Use only modern default cipher suites.

Available for manual keystore management.

Use Proxy Server Enables using a proxy server to connect to the system.
Proxy Server Proxy server endpoint.
Proxy User User name for proxy credentials.
Proxy Password Password for proxy credentials.
Test Endpoint Test endpoint of the URL of the requested resource. For example, /basic/authenticate.

The test endpoint is used only to verify the connection URL and credentials, and is not used in stages that use the connection.

Test Method HTTP method to apply to the requested resource.

The test method is used only to verify the connection URL and credentials, and is not used in stages that use the connection.

Note: The final connection URL is a combination of the base URL followed by the test endpoint if one is configured. For stages that use the connection, the final URL is a combination of the base URL configured for the connection followed by the request endpoint configured for the stage.
If you configured the connection to use Basic or Digest authentication, configure the following properties on the Connection tab:
Connection Property Description
User

Authentication username.

Password

Authentication password.

If you configured the connection to use Bearer authentication, configure the following property on the Connection tab:
Bearer Property Description
Token Authentication token to access the requested resource.
If you configured the connection to use OAuth 1 authentication, configure the following properties on the Connection tab:
OAuth 1 Property Description
Consumer Key

Name of the OAuth consumer key.

Consumer Secret

OAuth consumer secret.

Access Token

OAuth 1.0 access token.

Token Secret

OAuth 1.0 token secret.

If you configured the connection to use OAuth 2 authentication, configure the following properties on the Connection tab:
OAuth 2 Property Description
Grant Type

Grant type required by the HTTP service.

Token Endpoint

URL to request the access token.

Client ID

Client ID that the HTTP service uses to identify the HTTP client.

Enter for the client credentials grant that uses a client ID and secret for authentication. Or, for the resource owner password credentials grant that requires a client ID and secret.

Required for the Client Credentials grant.

Client Secret

Client secret that the HTTP service uses to authenticate the HTTP client.

Enter for the client credentials grant that uses a client ID and secret for authentication. Or, for the resource owner password credentials grant that requires a client ID and secret.

Required for the Client Credentials grant.

Signing Algorithm Algorithm used to sign the access token.

Required for the Access Token grant.

Signing Key Private key that the selected signing algorithm uses to sign the access token.

Required for the Access Token grant using a signing algorithm.

Token Headers Headers to include in the access token

Available for the Access Token grant.

Token Claims Claims to include in the access token. Specify in JSON format. Enter each claim required to obtain an access token. You can include the expression language in the token claims.

For example, to request an access token to read from Google service accounts, enter the following claims with the appropriate values:

{
  "iss":"my_name@my_account.iam.gserviceaccount.com",
  "scope":"https://www.googleapis.com/auth/drive",
  "aud":"https://oauth2.googleapis.com/token",
  "exp":${(time:dateTimeToMilliseconds(time:now())/1000) + 50 * 60},
  "iat":${time:dateTimeToMilliseconds(time:now())/1000}
}

Required for the Access Tokens grant.

Owner User Resource owner user name.

Required for the Owner Credentials grant.

Owner Password Resource owner password.

Required for the Owner Credentials grant.

Owner Client ID Resource owner client ID.

Available for the Owner Credentials grant.

Owner Client Secret Resource owner client secret.

Required for the Owner Credentials grant.

Additional Parameters Optional parameters to send to the token endpoint when requesting an access token. For example, you can define the OAuth 2 scope request parameter.

Click the Add icon to add additional key-value pairs.

Optionally configure the following property on the Request tab:
Request Property Description
Request Headers

Common headers to include in the request. Using simple or bulk edit mode, click Add to add additional common headers. For information about simple and bulk edit mode, see Simple and Bulk Edit Mode in the Data Collector documentation.