XStream

A user-defined Scala operator

Scala is the preferred language for new XStream operators. Start with the JVM SDK templates and keep processing, structure inference and bounded samples in the same operator implementation. Java uses the same JVM contracts; Python, scripts and native commands remain supported for existing code and language-specific libraries.

Prepare the development dependencies and JVM SDK as described in the SDK guide. SimpleIdentity uses the recommended template; ScalaIdentity demonstrates the lower-level protocol.

This standalone example implements scala_identity@1.0 using the XStream JVM SDK. It produces a thin JAR, sharing Scala, Arrow, Jackson, and the JVM from xstream/operator-jvm:1.1. The operator preserves the original input bytes; replace its batch processing with your transformation.

Example files:

Build and publish

From the project root, with JDK 17, sbt, and Docker access:

# Build the shared environment once (skip when already installed).
scripts/build-jvm-runtime.sh
(cd examples/operators/scala && sbt package)
(cd "$REPOSITORY_HOME" && sbt stage)
${REPOSITORY_HOME}/target/universal/stage/bin/xstream-operator-repository seed --operators examples/operators/scala/operators --root .operator-catalog
${REPOSITORY_HOME}/target/universal/stage/bin/xstream-operator-repository serve --root .operator-catalog --port 8820

Configure the execution node:

remote_operator_library_urls:
  - http://localhost:8820

Run in the connected shell:

run scala_identity@1.0 --input source=/events

The operator also works locally: set local_operator_library: ../examples/operators/scala/operators when launching from dacp-server. Build the JAR and install the runtime image first.

Entry point and stream contract

The JVM launcher passes operator /input /input/params.json to example.ScalaIdentity.main. Read named input metadata (port, file, representation, format) from /input/manifest.json. The example accepts one bytes port named source, represented by an Arrow IPC stream containing one binary data column. Stop at IPC EOS, including for FIFO inputs.

Write start/data/end frames to stdout for the result port; diagnostics belong on stderr. The shell consumes output incrementally, without publishing a saved dataset. The complete source demonstrates framing and resource cleanup. See the execution contract.

Keep the artifact small

build.sbt marks runtime libraries Provided, so they stay out of the operator JAR. Match Scala and dependency versions to the shared runtime. If your operator needs another library, build a new shared runtime with that library under /opt/lib; package private libraries in a self-contained JAR only when appropriate. Reuse existing runtime digests for code-only updates.

operator.yaml selects the runtime image, executor, main class, and JAR path. Publish a distinct name/entrypoint for another operator. Several operators can reference the same JAR containing different main classes. Adding them requires no core registry edits, server rebuild, or client changes.

To exercise this JAR and both Python artifact types in real containers, build the Python environment as described in the Python example, then run from the project root:

(cd plugins/xstream-plugin-operator-executors && XSTREAM_TEST_OPERATOR_PACKAGES=1 sbt 'testOnly xstream.operator.executors.OperatorPackageSpec')

See operator packaging for executor plugins, validation, caching, resource limits, and platform restrictions.

For complete source examples, wheel packaging, remote file transfer, administrator publication, and execution instructions, see Write and upload custom operators.

The repository accepts source only. Local builds in these examples are for development verification; use the source submission and review workflow for public publication.

REPOSITORY_HOME is the path to the independent operator repository checkout.