Jython Evaluator

Record by Record

The processor calls the script for each record. The processor passes the record to the script as a map and processes each record individually.

The script does not require error handling logic. Error records are passed to the processor for error handling. The processor handles error records based on the On Record Error property.

Use this mode to avoid including error handling logic in the code. Since this mode calls the script for each record, pipeline performance will be negatively affected.

Batch by Batch

The processor calls the script for each batch. The processor passes the batch to the script as a list and processes the batch at one time.

Include error handling logic in the script. Without error handling logic, a single error record sends the entire batch to the processor for error handling. The processor handles error records based on the On Record Error property.

Use this mode to improve performance by processing a batch of data at a time.

records

A collection of records to process. The records object includes different elements based on the processing mode that you use:

• Record by Record - The records array includes one record element. A record includes a single value element. The value element contains the data for the record.
• Batch by Batch - The records array includes all records in the batch.

Map and list record fields are mapped to Jython dictionaries and lists, respectively.

Data types are preserved for values that are not modified. For values that are modified, the scripting engine might change the original type.

state

An object to store information between invocations of the init, main, and destroy scripts. A state is a map object that includes a collection of key/value pairs. You can use the state object to cache data such as lookups, counters, or a connection to an external system.

The state object functions much like a member variable:

• The information is transient and is lost when the pipeline stops or restarts.
• The state object is available only for the instance of the processor stage it is defined in. If the pipeline executes in cluster mode, the state object is not shared across nodes.

The same instance of the state object is available to all three scripts. For example, you might use the init script to open a connection to a database and then store a reference to that connection in the state object. In the main script, you can access the open connection using the state object. Then in the destroy script, you can close the connection using the state object.

Warning: The state object is best used for a fixed or static set of data. Adding to the cache on every record or batch can quickly consume the memory allocated to Data Collector and cause out-of-memory exceptions.

log

An object that writes messages to the log4j log. Use sdc.log to access the object configured for the stage. The object includes methods that correspond to the level in the log file:

• info(<message template>, <arguments>...)
• warn(<message template>, <arguments>...)
• error(<message template>, <arguments>...)
• trace(<message template>, <arguments>...)

The message template can contain positional variables, indicated by curly brackets: { }. When writing messages, the method replaces each variable with the argument in the corresponding position - that is, the method replaces the first { } occurrence with the first argument, and so on.

output

An object that writes the record to the output batch. Use sdc.output to access the object configured for the stage. The object includes a write(<record>) method.

error

An object that passes error records for error handling. Use sdc.error to access the object configured for the stage. The object includes a write(<record>, <String message>) method.

sdc

A wrapper object that accesses constants, methods, and objects available to the scripts.

The sdc object contains the following constant:

• userParams - Dictionary that contains the script parameters and values configured on the Advanced tab with the Parameters in Script property.

The sdc object contains the following methods:

• createRecord(<String record ID>) - Returns a new record with the passed ID. Pass a string that uniquely identifies the record and includes enough information to track the record source.
• isPreview() - Returns a Boolean value that indicates whether the pipeline is in preview mode.
• getFieldNull(<record>, <String field path>) - Returns one of the following:
  • The value of the field at the specified path if the value is not null
  • The null object defined for the field type, such as NULL_INTEGER or NULL_STRING, if the value is null
  • The unassigned null object NULL if there is no field at the specified path
• createMap(<Boolean list-map>) - Returns a map for use as a field in a record. Pass true to create a list-map field, or false to create a map field.
• createEvent(<String type>, <Integer version>) - Returns a new event record with the specified event type and version. Verify that the stage enables event generation before implementing event methods.
• toEvent(<event record>) - Sends an event record to the event output stream. Verify that the stage enables event generation before implementing event methods.
• pipelineParameters() - Returns a map of all runtime parameters defined for the pipeline.

sdcFunctions

An object that runs functions that evaluate or modify data.

Important: The sdcFunctions object is now deprecated and will be removed in a future release. To evaluate and modify data, use the methods in the sdc object.

The sdcFunctions object includes the following methods:

• getFieldNull(<record>, <String field path>) - Returns one of the following:
  • The value of the field at the specified path if the value is not null
  • The null object defined for the field type, such as NULL_INTEGER or NULL_STRING, if the value is null
  • The unassigned null object NULL if there is no field at the specified path
• createRecord(<String record ID>) - Returns a new record with the passed ID. Pass a string that uniquely identifies the record and includes enough information to track the record source.
• createMap(<Boolean list-map>) - Returns a map for use as a field in a record. Pass true to create a list-map field, or false to create a map field.
• createEvent(<String type>, <Integer version>) - Returns a new event record with the specified event type and version. Verify that the stage enables event generation before implementing event methods.
• toEvent(<event record>) - Sends an event record to the event output stream. Verify that the stage enables event generation before implementing event methods.
• isPreview() - Returns a Boolean value that indicates whether the pipeline is in preview mode.

• Create a String field named new and set its value to new-value.
• Update the existing field named old to set the value of the attr attribute to attr-value.

Data type of null values

You can associate null values with a data type. For example, if the script assigns a null value to an Integer field, the field is returned to the pipeline as an integer with a null value.

Use constants in the Jython code to create a new field of a specific data type with a null value. For example, you can create a new String field with a null value by assigning the NULL_STRING type constant to the field as follows:

record.value['new_field'] = sdc.NULL_STRING

Date fields

Use the String data type to create a new field to store a date with a specific format. For example, the following sample code creates a new String field that stores the current date using the format YYYY-MM-dd:

# Define a date object to record the current date
import datetime as dt
newDate = dt.datetime.utcnow().strftime("%Y-%m-%d")

for record in records:
  try:
    # Create a string field to store the current date with the specified format
    record.value["date"] = newDate
    
    # Write record to the output
    sdc.output.write(record)

  except Exception as e:
    # Send record to error
    sdc.error.write(record, str(e))

Data type of modified values

Values that are not modified by the stage retain their original type.

Numeric data that is modified becomes a Double, other types of data retain their original type.

1. On the General tab, select the Produce Events property.

   This enables the event output stream for use.

2. Include both of the following methods in the script:
   • sdc.createEvent(<String type>, <Integer version>) - Creates an event record with the specified event type and version number. You can create a new event type or use an existing event type. Existing event types are documented in other event-generating stages.

     The event record contains no record fields. Generate record fields as needed.

   • sdc.toEvent(<record>) - Use to pass events to the event output stream.

• record.<header name> - Use to return the value of a header attribute.
• record.attributes - Use to return a map of custom record header attributes, or to create or update a specific record header attribute.

input_stream = record.value['fileRef'].getInputStream()
input_stream.read()

input_stream.close()

from <package> import <class name>

from org.bouncycastle.jcajce.provider.digest.SHA3 import DigestSHA3

1. Install the Python module on the Data Collector machine using a Python package installer, such as pip.

   When you install the module, note the installation location – you need to specify the path to the module in your Jython script. The base directory is typically defined in your installer preferences.

2. To import the Python module, specify the file path and name of the module in the following code.

   Updating the sys.path variable from multiple threads can lead to inconsistencies. This code uses the Data Collector import lock to ensure that only one thread updates the sys.path variable at one time:

      try:
          sdc.importLock()
          import sys
          sys.path.append('<path to module>')
          import <module name>
      finally:
          sdc.importUnlock()

try:
  sdc.importLock()
  import sys
  sys.path.append('/python/sdc-packages')
  import FixEverything
finally:
  sdc.importUnlock()