Webhooks

You can configure a pipeline to use webhooks.

Not available in Data Collector Edge pipelines.

A webhook is a user-defined HTTP callback - an HTTP request that the pipeline sends automatically when certain actions occur. You can use webhooks to automatically trigger external tasks based on an HTTP request. Tasks can be as simple as sending a message through an application API or as powerful as passing commands to the Data Collector command line interface.

You can configure the following types of webhooks:
Trigger Description Configuration Location
Alert The pipeline sends all alert webhooks each time an alert is triggered.

For example, if your text message application has a webhook API, you can have the pipeline send texts when alerts are triggered.

For details on how to configure an alert webhook, see Configuring an Alert Webhook.

 Pipeline Rules tab > Webhook tab
State notification The pipeline sends all state notification webhooks when the pipeline transitions to the selected pipeline states.

For example, you can send an HTTP request to the Data Collector REST API to start a different pipeline when the current pipeline transitions to a Finished state.

For details on how to configure a state notification webhook, see the Notifications properties in Configuring a Pipeline.

 Pipeline Configuration tab > Notifications tab

The flexibility of webhooks enables you to automatically trigger a wide range of tasks, depending on the ability of external systems to support webhooks or process HTTP requests.

Important: You must configure webhooks as expected by the receiving system. For details on how to configure incoming webhooks check the receiving system's documentation. You might also need to enable webhook usage within that system.

When you configure a webhook, you specify the URL to send the request and the HTTP method to use. Some HTTP methods allow you to include a request body or payload. In the payload, you can use parameters to include information about the cause of the trigger, such as the text of the alert or the latest pipeline state. You can also include request headers, content type, authentication type, username and password as needed.

Request Method

You can use the following methods in webhooks:
  • GET
  • PUT
  • POST
  • DELETE
  • HEAD

Payload and Parameters

You can include a request body or payload for PUT, POST, and DELETE request webhooks. Depending on the receiving system, the payload might not be used. For example, when using the Data Collector REST API, you simply include all required information in the URL.

When you configure a payload, you can use any valid content type, then specify the content type in the webhook properties. When defining a message, be sure to consider when the pipeline sends the webhook and include the information that the recipient needs.

You can use parameters in the payload to include information about the action that triggered the webhook. For example, when configuring a webhook that sends a text message when a pipeline stops, you might include the pipeline name, pipeline state, and the time parameters in the message.

Enclose parameters in double curly brackets, as follows: 
{{<parameter name>>}}
You can use the following parameters in webhooks:
state notification parameters
When configuring a state notification webhook, you can use the following parameters:
  • PIPELINE_TITLE - The pipeline title or name.
  • PIPELINE_URL - The direct URL to the pipeline.
  • PIPELINE_STATE - The current state of the pipeline.
  • TIME - The time of the triggered request.
  • PIPELINE_STATE_MESSAGE - The pipeline state and any related information, such as the last-saved offset or the error message for error states.
  • PIPELINE_INPUT_RECORDS_COUNT - The number of records read.
  • PIPELINE_OUTPUT_RECORDS_COUNT - The number of records written.
  • PIPELINE_ERROR_RECORDS_COUNT - The number of error records.
  • PIPELINE_ERROR_MESSAGES_COUNT - The number of error messages.
  • PIPELINE_RUNTIME_PARAMETERS - The number of pipeline parameters.
  • PIPELINE_METRICS - Metrics data for the pipeline.
For example, say you configure the pipeline to send a webhook only when the pipeline transitions to the Stopped state - that is, when someone stops the pipeline. You might use the following message in a JSON request body: 
{  
   "text":"At {{TIME}}, a user stopped the {{PIPELINE_TITLE}} pipeline. \n <To see the pipeline, click here: {{PIPELINE_URL}}"
}
However, if the pipeline is configured to send webhooks when the pipeline changes to several different states, you might use a more generic message and include the pipeline state in the message. For example:
{  
   "text":"The '{{PIPELINE_TITLE}}' pipeline changed state to {{PIPELINE_STATE}} at {{TIME}}. \n Pipeline status message: {{PIPELINE_STATE_MESSAGE}} \n <{{PIPELINE_URL}}|Click here for details.>"
}
alert parameters
When configuring an alert webhook, you can use the following parameters:
  • ALERT_CONDITION - The condition of the rule associated with the alert.
  • ALERT_NAME - The alert label or name.
  • ALERT_TEXT - The text configured for the alert.
  • ALERT_VALUE - The value that triggered the condition. For example, if the alert is configured to trigger upon reaching 1000 error records, the alert value will be 1000.
  • PIPELINE_TITLE - The pipeline title.
  • PIPELINE_URL - The direct URL to the pipeline.
  • TIME - Time of the triggered request.
For example, say you configure a pipeline to send a webhook alert each time an alert triggers. To include some key information in the JSON request body, you might try this:
{  
   "text":"{{ALERT_TEXT}}: At {{TIME}}, {{ALERT_NAME}} was triggered by {{ALERT_VALUE}} for the following condition: {{ALERT_CONDITION}}. \n This is for the {{PIPELINE_TITLE}} pipeline, at {{PIPELINE_URL}}"
}

Examples

Here are some examples of how you might use webhooks:

Send webhooks to a Slack channel
You can configure a pipeline to send webhooks to a Slack channel. For example, you could have all alerts sent to an Operations Slack channel so someone on your operations team can respond to the alert.
To do this, you perform the following steps:
  1. Configure Slack for incoming webhooks for the channel you want to use.

    At the time of writing, you can enable webhooks in Slack starting from this page. When you enable webhooks, Slack generates a URL for you to use. If you have already enabled webhooks, check your account information for the Slack URL.

  2. Copy the URL that Slack generates and use this to configure the pipeline.
    The URL looks something like this: 
    https://hooks.slack.com/services/<random string>/<random string>/<random string>

    Slack also offers instructions on how to configure the payload. At this time, they suggest a text field with the contents of the message, like Data Collector default payload.

  3. Configure the webhook in the pipeline.

    For Slack, you can just enter the URL and accept the defaults for everything else.

    Here's how a Slack alert webhook might look:

Start another pipeline
You can start a pipeline after the first pipeline completes all processing using dataflow triggers, the Pipeline Finisher executor, and a state notification webhook.
For example, say you have a JDBC Query Consumer origin that performs a full query to process all legacy data in some database tables.
You configure the origin to generate an event when it completes processing all available data, and you connect the event stream to the Pipeline Finisher executor. When the Pipeline Finisher executor receives the event from the JDBC Query Consumer, it transitions the pipeline to a Finished state. For more information on using the Pipeline Finisher executor with dataflow triggers, see Stopping a Pipeline After Processing All Available Data.
To use this state change to start a second pipeline, configure a webhook that triggers when the pipeline state changes to Finished:
  1. In the pipeline properties, click the Notifications tab and click Add to add a webhook.
  2. For the URL, enter the command to start the pipeline.

    The command looks something like this:

    <http|https>://<system ip>.<http port>/rest/v1/pipeline/<pipeline id>/start

    For example: http://localhost:18630/rest/v1/pipeline/MyPipelinefbb6894c-08ec-421e-9e3c-4e6bbb2f5baf/start

  3. Using simple or bulk edit mode, click the Add icon to add a request header, and add the following header: X-Requested-By

    And set the value to: sdc.

    You can use the default method and authentication type, and keep or delete the payload, since they aren't used by the REST API.

    This is how the webhook might look: