Draft Runs

Draft Runs Overview

A draft run is the execution of a draft pipeline. Use draft runs for development purposes only.

While editing a pipeline in the pipeline canvas, you can start a draft run to quickly test the pipeline logic. You can run a draft run as long as you'd like. This allows you to monitor the draft run over the course of hours or days, as needed, before you publish the pipeline for use in a production job.

The first time that you start a draft run of a pipeline, Control Hub automatically creates a job for the draft run, using the following naming convention: Draft Run for <pipeline name>. The draft run job runs the pipeline on the authoring engine currently selected in the pipeline canvas. When you stop the draft run, the job stops the pipeline and then transitions to an inactive status.

When you start another draft run of the same pipeline, Control Hub updates the existing draft run job with the pipeline name, commit ID, and currently selected authoring engine ID. The job then transitions to an active status, starting another draft run of the pipeline. The job maintains the run history of all draft runs for that pipeline.
Important: Jobs automatically created for draft runs use limited job functionality. They do not provide the full range of functionality as production jobs that you create for published pipelines.

Only one draft run can exist for each pipeline. A draft run exists until you manually delete the draft run from the Draft Runs view or until you delete the pipeline.

You cannot grant permissions on draft runs. The access level that users have on the draft run is determined by pipeline permissions. For more information, see Pipeline Permissions.

Comparing Draft Runs with Data Preview

As you build a pipeline, you can use both draft runs and data preview to test your pipeline logic.

Both functionalities are tools for development, but are used for different purposes:
Data preview
Preview a pipeline as you begin building a pipeline in the canvas. You can preview draft pipelines that are complete or incomplete. You can also preview data for published pipelines.
Use preview to review how a single batch of source data passes and changes through each stage. By default, pipeline preview does not write data to destination systems. However, you can configure a preview to do so.
After running preview for a pipeline, you can view the input and output schema for each stage. Control Hub uses the schema extracted from the last data preview to list available field paths in the pipeline canvas, making it easier for you to continue building the pipeline.
Preview functions slightly differently for each engine type.
Draft runs
Start a draft run as you finish building a draft pipeline in the canvas. You can start a draft run only when a draft pipeline is complete and fully configured. You cannot start a draft run for a published pipeline.
A draft run executes the complete pipeline, including writing data to the destination system. Use a draft run as a final test of the pipeline logic, so that you can review how the pipeline processes multiple batches of data. You can monitor the progress of a draft run over multiple hours or days, before you publish the pipeline for use in a production job.
Tip: Use pipeline parameters to easily switch the external systems that a pipeline connects to. When you start a draft run of the pipeline, specify parameter values so that the pipeline writes to a development destination system used for testing purposes. Then when you publish the pipeline and add it to a production job, specify different parameter values so that the pipeline writes to the production destination system.

Comparing Draft Runs with Production Jobs

When you start a draft run, Control Hub automatically creates a job for the draft pipeline. Draft run jobs function similarly to production jobs that you create for published pipelines. For example, they both run a complete pipeline on an engine, maintain a run history, and transition through the same job statuses. In addition, Control Hub ensures that engines have not exceeded any resource thresholds before running a pipeline for both job types.

However, draft run jobs are for development purposes only and have limited job functionality. They do not provide the full range of functionality as production jobs that you create for published pipelines.

Production jobs provide the following additional functionality:
  • Pipeline failover - Enable pipeline failover to minimize downtime due to unexpected failures.
  • Scalability - For Data Collector jobs, manually scale out pipeline processing by increasing the number of pipeline instances that Control Hub runs for a job.
  • Scheduled tasks - Schedule jobs to start, stop, or upgrade at a particular time.
  • Job templates - Use job templates to create one job definition that can be used to start multiple job instances. Job templates hide the complexity of production job details from business analysts.

Working with Draft Runs

The Draft Runs view lists all draft runs that you have access to.

You can complete the following tasks in the Draft Runs view:
  • View draft run details, including the pipeline version and the job and pipeline status.
  • Search for draft runs.
  • View draft run history.
  • Acknowledge draft run errors.
  • Stop draft runs.
  • Delete draft runs.
Important: To start a draft run, you must edit a draft pipeline in the pipeline canvas.

The following image shows a list of draft runs in the Draft Runs view. Each draft run is listed with the draft run name, pipeline name, pipeline version, job status, and pipeline status:

Engine Version Requirements

You can start a draft run of a pipeline when using any supported authoring engine version.

However for Data Collector pipelines, to start and immediately capture a snapshot for a draft run or to view captured snapshots for the previous draft run, you must start the draft run on an authoring Data Collector 5.0.0 or later. For more information about capturing snapshots for draft runs, see Snapshots.

Note: Snapshots are not available for draft runs or job runs for other pipeline types.

Managing Draft Runs

You start a draft run while editing a pipeline from the pipeline canvas.

You can then filter, stop, and delete all draft runs that you have access to from the Draft Runs view.

Starting a Draft Run

While editing a pipeline in the pipeline canvas, you can start a draft run to quickly test the pipeline logic.

You can start a draft run of a fully configured draft pipeline. The Draft Run menu becomes active in the pipeline canvas when a draft pipeline is complete.

You cannot start a draft run of a published pipeline. To run a published pipeline, you must first create a job for the published pipeline and then start the job.

  1. While viewing a completed draft pipeline in the pipeline canvas, click the Draft Run menu in the toolbar, and then select one of the following options:
    • Start Pipeline - Start a draft run of the pipeline.
    • Reset Origin and Start - Reset the origin and then start a draft run of the pipeline.
    • Start with Parameters - Specify the parameter values to use and then start a draft run of the pipeline.

    When the draft run starts, statistics for the draft run display in the Monitor panel.

  2. Monitor the draft run of the pipeline, including viewing real-time statistics and error information.

    To access the history of previous draft runs for the pipeline, click the Draft Run History tab. The draft run history includes the start and end time of previous draft runs and also the input, output, and error record count for each draft run.

    Click View Summary for a specific run to view a summary of the metrics for that draft run.

    For more details, see Monitoring Draft Runs.

  3. To stop the draft run, click the Stop icon: .
    You can continue building the pipeline and performing additional draft runs until you decide that the pipeline is complete, and then publish the pipeline.

Filtering Draft Runs

In the Draft Runs view, you can filter the list of displayed draft runs by engine type, or by status or status color of the job automatically created for the draft run. You can also search for draft runs by name.
Note: By default, filtering is replaced by the new search functionality. If needed, you can disable search and revert to the original filtering functionality.
  1. In the Navigation panel, click Run > Draft Runs.
  2. If the Filter column does not display, click the Toggle Filter Column icon: .

    The following image displays the Filter column in the Draft Runs view:

  3. To search for draft runs by name, enter text in the search field, and then press Return.
  4. Select an engine type, job status, or job status color to additionally filter the list of draft runs.
  5. Select the Keep Filter Persistent checkbox to retain the filter when you return to the view.
    Tip: To share the applied filter, copy the URL and send it to another user in your organization.

Stopping Draft Runs

Stop a draft run when you want to stop processing data for the draft pipeline. When you stop a draft run, the job automatically created for the draft run stops the pipeline and then transitions to an inactive status.

When stopping a draft run, Control Hub waits for the pipeline to gracefully complete all tasks for the in-progress batch. In some situations, this can take several minutes.

For example, if a scripting processor includes code with a timed wait, Control Hub waits for the scripting processor to complete its task. Then, Control Hub waits for the rest of the pipeline to complete all tasks before stopping the pipeline.

When you stop a draft run that includes an origin that can be reset, Control Hub maintains the last-saved offset.

Tip: While monitoring an engine, you can also quickly stop all draft runs currently running on the engine.
  1. In the Navigation panel, click Run > Draft Runs.
  2. Select draft runs with an active job status, and then click the Stop Jobs icon: .

    Or from the draft run monitoring view, click the Stop icon: .

  3. In the confirmation dialog box that appears, click OK.

    Depending on the pipeline complexity, the draft run might take some time to stop.

  4. Click Close.

Deleting Draft Runs

You can delete draft runs with an inactive job status. When you delete a draft run, you lose the history of draft runs for that pipeline.

When you start another draft run for the same pipeline, Control Hub automatically creates another job for the draft run.

  1. In the Navigation panel, click Run > Draft Runs.
  2. Select draft runs in the list, and then click the Delete icon: .
  3. In the confirmation dialog box that appears, click OK.

Monitoring Draft Runs

After you start a draft run in the pipeline canvas, Control Hub displays the Monitoring panel. Use the Monitoring panel to view statistics, error information, and logs about the running draft pipeline.

Click the canvas to view statistics for the entire draft run. Select a stage in the canvas to view statistics for the stage.

While monitoring draft runs for Data Collector pipelines, you can also capture and review a snapshot of the data being processed.

Because Control Hub automatically creates a job for a draft run, you monitor a draft run in the same way that you monitor a production job. For details, see Monitoring Jobs.