Troubleshooting

Use the following tips for help with general pipeline validation errors:

The pipeline has the following set of validation errors for a stage:

CONTAINER_0901 - Could not find stage definition for <stage library name>:<stage name>.
CREATION_006 - Stage definition not found. Library <stage library name>. Stage <stage name>. 
Version <version>
VALIDATION_0006 - Stage definition does not exist, library <stage library name>, 
name <stage name>, version <version>

The pipeline uses a stage that is not installed on the Data Collector. This might happen if you imported a pipeline from a different version of the Data Collector and the current Data Collector is not enabled to use the stage.

If the Data Collector uses a different version of the stage, you might delete the invalid version and replace it with a local valid version. For example, if the pipeline uses an older version of the Hadoop FS destination, you might replace it with a version used by this Data Collector.

If you need to use a stage that is not installed on the Data Collector, install the related stage library. For information about installing additional drivers, see Install External Libraries.

Why isn't the Directory origin reading all of my files?

The Directory origin reads a set of files based on the configured file name pattern, read order, and first file to process. If new files arrive after the Directory origin has passed their position in the read order, the Directory origin does not read the files unless you reset the origin.

When using the last-modified timestamp read order, arriving files should have timestamps that are later than the files in the directory.

Similarly, when using the lexicographically ascending file name read order, make sure the naming convention for the files are lexicographically ascending. For example, filename-1.log, filename-2.log, etc., works fine until filename-10.log. If filename-10.log arrives after the Directory origin completes reading filename-2.log, then the Directory origin does not read filename-10.log since it is lexicographically earlier than filename-2.log.

For more information, see Read Order.

My MySQL JDBC Driver 5.0 fails to validate the query in my JBDC Query Consumer origin.

This can occur when you use a LIMIT clause in your query.

Workaround: Upgrade to version 5.1.

I'm using a JDBC origin to read MySQL data. Why are datetime value set to zero being treated like error records?

MySQL treats invalid dates as an exception, so both the JDBC Query Consumer and the JDBC Multitable Consumer create error records for invalid dates.

You can override this behavior by setting a JDBC configuration property in the origin. Add the zeroDateTimeBehavior property and set the value to "convertToNull".

For more information about this and other MySQL-specific JDBC configuration properties, see http://dev.mysql.com/doc/connector-j/en/connector-j-reference-configuration-properties.html.

A pipeline using the JDBC Query Consumer origin keeps stopping with the following error:

JDBC_77 <db error message> attempting to execute query '<query>'. Giving up 
after <error count> errors as per stage configuration. First error: <first db error>.

This occurs when the origin cannot successfully execute a query. To handle transient connection or network errors, try increasing the value for the Number of Retries Upon Query Error property on the JDBC tab of the origin.

My pipeline using a JDBC origin generates an out-of-memory error when reading a large table.

When the Auto Commit property is enabled in a JDBC origin, some drivers ignore the fetch-size restriction, configured by the Max Batch Size property in the origin. This can lead to an out-of-memory error when reading a large table that cannot entirely fit in memory.

To resolve, disable the Auto Commit property on the Advanced tab of the origin.

Data preview continually times out for my Oracle CDC Client pipeline.

Pipelines that use the Oracle CDC Client can take longer than expected to initiate for data preview. If preview times out, try increasing the Preview Timeout property incrementally.

For more information about using preview with this origin, see Data Preview with Oracle CDC Client.

My Oracle CDC Client pipeline has paused processing during a daylight saving time change.

If the origin is configured to use a database time zone that uses daylight saving time, then the pipeline pauses processing during the time change window to ensure that all data is correctly processed. After the time change completes, the pipeline resumes processing at the last-saved offset.

For more information, see Database Time Zone.

A PostgreSQL CDC Client pipeline generates the following error:

com.streamsets.pipeline.api.StageException: JDBC_606 - Wal Sender is not active

This can occur when the Status Interval property configured for the origin is larger than the wal_sender_timeout property in the PostgreSQL postgresql.conf configuration file.

The Status Interval property should be less than the wal_sender_timeout property. Ideally, it should be set to half of the value of the wal_sender_timeout property.

For example, you can use the default status interval of 30 seconds with the default wal_sender_timeout value of 60000 milliseconds, or 1 minute.

A pipeline generates a buffering capacity error

When pipelines with a Salesforce origin fail due to a buffering capacity error, such as Buffering capacity 1048576 exceeded, increase the buffer size by editing the Streaming Buffer Size property on the Subscribe tab.

A pipeline fails to stop when users click the Stop icon

Scripts must include code that stops the script when users stop the pipeline. In the script, use the sdc.isStopped method to check whether the pipeline has been stopped.

A Jython script does not proceed beyond import lock

Pipelines freeze if Jython scripts do not release the import lock upon a failure or error. When a script does not release an import lock, you must restart Data Collector to release the lock. To avoid the problem, use a try statement with a finally block in the Jython script. For more information, see Thread Safety in Jython Scripts.

An Azure Data Lake Storage destination seems to be causing out of memory errors, with the following object using all available memory:

com.streamsets.pipeline.stage.destination.hdfs.writer.ActiveRecordWriters

This can occur due to a Hadoop known issue, which can affect the Azure Data Lake Storage Gen2 destination.

For a description of a workaround, see the documentation for the Gen2 destination.

Why is the pipeline failing entire batches when only a few records have a problem?

Due to Cassandra requirements, when you write to a Cassandra cluster, batches are atomic. This means than an error in a one or more records causes the entire batch to fail.

Why is all of my data being sent to error? Every batch is failing.

When every batch fails, you might have a data type mismatch. Cassandra requires the data type of the data to exactly match the data type of the Cassandra column.

To determine the issue, check the error messages associated with the error records. If you see a message like the following, you have a data type mismatch. The following error message indicates that data type mismatch is for Integer data being unsuccessfully written to a Varchar column:

CASSANDRA_06 - Could not prepare record 'sdk:': 
Invalid type for value 0 of CQL type varchar, expecting class java.lang.String but class java.lang. 
Integer provided`

To correct the problem, you might use a Field Type Converter processor to convert field data types. In this case, you would convert the integer data to string.

A pipeline with an Elasticsearch destination fails to start with an SSL/TLS error, such as the following:

ELASTICSEARCH_43 - Could not connect to the server(s) <SSL/TLS error details>

This message can display due to many different SSL/TLS issues. Review the details of related messages to determine the corrective measures to take.

Here are some examples of when a version of the message might display:

• If you configure the stage to use SSL/TLS, but do not specify an HTTPS-enabled port.

  To resolve the issue, specify an HTTPS-enabled port.

• If you configure the stage to use SSL/TLS and specify an HTTPS-enabled port, but configure the URL incorrectly, such as http://<host>:<port>.

  To resolve this issue, update the URL to use HTTPS, as follows: https://<host>:<port>.

• If you configure the stage to use SSL/TLS, but do not have a certificate in the specified truststore.

  To resolve the issue, place a valid certificate in the truststore.

I get the following error when validating or starting a pipeline with an HBase destination:

HBASE_06 - Cannot connect to cluster: org.apache.hadoop.hbase.MasterNotRunningException: 
com.google.protobuf.ServiceException: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: 
Call to node00.local/<IP_address>:60000 failed on local exception: 
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: 
Connection to node00.local/<IP_address>:60000 is closing. Call id=0, waitTime=58

Is your HBase master is running? If so, then you might trying to connect to a secure HBase cluster without configuring the HBase destination to use Kerberos authentication. In the HBase destination properties, select Kerberos Authentication and try again.

Can the Kafka Producer create topics?

The Kafka Producer can create a topic when all of the following are true:

• You configure the Kafka Producer to write to a topic name that does not exist.
• At least one of the Kafka brokers defined for the Kafka Producer has the auto.create.topics.enable property enabled.
• The broker with the enabled property is up and available when the Kafka Producer looks for the topic.

A pipeline that writes to Kafka keeps failing and restarting in an endless cycle.

This can happen when the pipeline tries to write message to Kafka 0.8 that is longer than the Kafka maximum message size.

Workaround: Reconfigure Kafka brokers to allow larger messages or ensure that incoming records are within the configured limit.

When I enter the name of the Impala JDBC driver in the stage, I receive an error saying that the driver is not present in the class path:

HIVE_15 - Hive JDBC Driver <driver name> not present in the class path.

When Data Collector cannot find the JDBC driver for a stage, Data Collector might generate one of the following error messages:

JDBC_00 - Cannot connect to specified database: com.streamsets.pipeline.api.StageException:
JDBC_06 - Failed to initialize connection pool: java.sql.SQLException: No suitable driver

JDBC_00 - Cannot connect to specified database: com.zaxxer.hikari.pool.PoolInitializationException:
Exception during pool initialization: The TCP/IP connection to the host 1.2.3.4, port 1234 has failed

In this case, verify that the Data Collector machine can access the database machine on the relevant port. You can use tools such as ping and netcat (nc) for this purpose. For example, to verify that the host 1.2.3.4 is accessible:

$ ping 1.2.3.4 
PING 1.2.3.4 (1.2.3.4): 56 data bytes 
64 bytes from 1.2.3.4: icmp_seq=0 ttl=57 time=12.063 ms 
64 bytes from 1.2.3.4: icmp_seq=1 ttl=57 time=11.356 ms 
64 bytes from 1.2.3.4: icmp_seq=2 ttl=57 time=11.626 ms 
^C
--- 1.2.3.4 ping statistics --- 
3 packets transmitted, 3 packets received, 0.0% packet loss 
round-trip min/avg/max/stddev = 11.356/11.682/12.063/0.291 ms

Then to verify that port 1234 can be reached:

$ nc -v -z -w2 1.2.3.4 1234 
nc: connectx to 1.2.3.4 port 1234 (tcp) failed: Connection refused