Web Client

Available when using an authoring Data Collector version 5.11.0 or later.

To create a Web Client connection, the appropriate Web Client stage library must be installed on the selected authoring Data Collector:

Data Collector 5.11.0 or later - Requires the OkHttp Web Client stage library, streamsets-datacollector-webclient-impl-okhttp-lib.

For a description of the Web Client connection properties, see Web Client Connection Properties.

After you create a Web Client connection, you can use the connection in the following stages:


Engine	Stages
Data Collector 5.11.0 or later	Web Client origin Web Client processor Web Client destination

Web Client Connection Properties

When creating a Web Client connection, configure the following properties on the Connection tab:


Connection Property	Description
Base URL	Fixed base URL or request endpoint of the requested resource. For example, `http://localhost:8000` or `http://localhost:8000/basic/authenticate`.
Authentication Scheme	Determines the authentication type used to connect to the server: None - Performs no authentication. Basic - Uses basic authentication. Requires a username and password. Use with HTTPS to avoid passing unencrypted credentials. Digest - Uses digest authentication. Requires a username and password. Bearer - Uses bearer authentication. Requires a username and password. OAuth 1 - Uses OAuth 1.0 authentication. Requires OAuth credentials. OAuth 2 - Uses OAuth 2.0 authentication. Requires OAuth credentials.
Keystore Management	Determines the authentication and encryption used to connect to the HTTP server. Automatic (for most HTTP and HTTPS requests) - Uses automatically-generated keystore and truststore configurations. Manual (for manually configured HTTPS requests) - Manually configure the keystore and truststore to use.
Keystore Location	Where to load the keystore from: Local - Loads the keystore from a local file. Remote - Loads the keystore from the provided key and certificate chain. Required for manual keystore management.
Keystore File	Path to the local keystore file. Enter an absolute path to the file or enter the following expression to define the file stored in the Data Collector resources directory: `${runtime:resourcesDirPath()}/keystore.jks` Available for manual keystore management.
Keystore Type	Type of keystore to use. Use one of the following types: Java Keystore File (JKS) PKCS #12 (p12 file) Default is Java Keystore File (JKS).
Keystore Password	Password to the keystore file. A password is optional, but recommended.
Private Key	Private key used in the remote keystore. Enter a credential function that returns the key or enter the contents of the key.
Certificate Chain	Each PEM certificate used in the remote keystore. Enter a credential function that returns the certificate or enter the contents of the certificate.
Keystore Key Algorithm	Algorithm to manage the keystore. Default is SunX509.
Truststore File	Path to the local truststore file. Enter an absolute path to the file or enter the following expression to define the file stored in the Data Collector resources directory: `${runtime:resourcesDirPath()}/truststore.jks` By default, no truststore is used.
Truststore Type	Type of truststore to use. Use one of the following types: Java Keystore File (JKS) PKCS #12 (p12 file) Default is Java Keystore File (JKS).
Truststore Password	Password to the truststore file. A password is optional, but recommended.
Truststore Key Algorithm	Algorithm to manage the truststore. Default is SunX509. Required for manual keystore management.
Default Protocol Versions	Use only modern default secure protocol versions. Available for manual keystore management.
Default Cipher Suites	Use only modern default cipher suites. Available for manual keystore management.
Use Proxy Server	Enables using a proxy server to connect to the system.
Proxy Server	Proxy server endpoint.
Proxy User	User name for proxy credentials.
Proxy Password	Password for proxy credentials.
Test Endpoint	Test endpoint of the URL of the requested resource. For example, `/basic/authenticate`. The test endpoint is used only to verify the connection URL and credentials, and is not used in stages that use the connection.
Test Method	HTTP method to apply to the requested resource. The test method is used only to verify the connection URL and credentials, and is not used in stages that use the connection.

Note: The final connection URL is a combination of the base URL followed by the test endpoint if one is configured. For stages that use the connection, the final URL is a combination of the base URL configured for the connection followed by the request endpoint configured for the stage.

If you configured the connection to use Basic or Digest authentication, configure the following properties on the Connection tab:


Connection Property	Description
User	Authentication username.
Password	Authentication password.

If you configured the connection to use Bearer authentication, configure the following property on the Connection tab:


Bearer Property	Description
Token	Authentication token to access the requested resource.

If you configured the connection to use OAuth 1 authentication, configure the following properties on the Connection tab:


OAuth 1 Property	Description
Consumer Key	Name of the OAuth consumer key.
Consumer Secret	OAuth consumer secret.
Access Token	OAuth 1.0 access token.
Token Secret	OAuth 1.0 token secret.

If you configured the connection to use OAuth 2 authentication, configure the following properties on the Connection tab:


OAuth 2 Property	Description
Grant Type	Grant type required by the HTTP service.
Token Endpoint	URL to request the access token.
Client ID	Client ID that the HTTP service uses to identify the HTTP client. Enter for the client credentials grant that uses a client ID and secret for authentication. Or, for the resource owner password credentials grant that requires a client ID and secret. Required for the Client Credentials grant.
Client Secret	Client secret that the HTTP service uses to authenticate the HTTP client. Enter for the client credentials grant that uses a client ID and secret for authentication. Or, for the resource owner password credentials grant that requires a client ID and secret. Required for the Client Credentials grant.
Signing Algorithm	Algorithm used to sign the access token. Required for the Access Token grant.
Signing Key	Private key that the selected signing algorithm uses to sign the access token. Required for the Access Token grant using a signing algorithm.
Token Headers	Headers to include in the access token Available for the Access Token grant.
Token Claims	Claims to include in the access token. Specify in JSON format. Enter each claim required to obtain an access token. You can include the expression language in the token claims. For example, to request an access token to read from Google service accounts, enter the following claims with the appropriate values: `{ "iss":"my_name@my_account.iam.gserviceaccount.com", "scope":"https://www.googleapis.com/auth/drive", "aud":"https://oauth2.googleapis.com/token", "exp":${(time:dateTimeToMilliseconds(time:now())/1000) + 50 * 60}, "iat":${time:dateTimeToMilliseconds(time:now())/1000} }` Required for the Access Tokens grant.
Owner User	Resource owner user name. Required for the Owner Credentials grant.
Owner Password	Resource owner password. Required for the Owner Credentials grant.
Owner Client ID	Resource owner client ID. Available for the Owner Credentials grant.
Owner Client Secret	Resource owner client secret. Required for the Owner Credentials grant.
Additional Parameters	Optional parameters to send to the token endpoint when requesting an access token. For example, you can define the OAuth 2 `scope` request parameter. Click the Add icon to add additional key-value pairs.

Optionally configure the following property on the Request tab:


Request Property	Description
Request Headers	Common headers to include in the request. Using simple or bulk edit mode, click Add to add additional common headers. For information about simple and bulk edit mode, see Simple and Bulk Edit Mode in the Data Collector documentation.