XStream

XStream Engine

Standalone and embeddable Scala execution of streaming YAML Jobs. One JVM shares one DataCatalog → Dataset → Item hierarchy. Embedding applications share their DataCatalog directly; local execution needs no server.

Requires JDK 17 and sbt. From the repository root:

cd xstream-engine
sbt test stage
cd ..
xstream-engine/target/universal/stage/bin/xstream-runner \
  examples/xstream/engine.yaml examples/xstream/read.yaml

xstream-runner is a program shipped by this module, not a separate module. Records are printed as JSONL; bytes are streamed directly to stdout. The launcher accepts CONFIG.yaml JOB.yaml [VARIABLES.yaml] [POLICY.yaml].

Use item_source.params.uri: context:/dataset/item for the bound DataCatalog, or an explicit dacp://host:port/dataset/item URI. Configure remote connections and credentials in nodes. Operators come from approved public repositories or explicit local source directories; isolated runtimes and compatible-operator fusion are supported.

See the full guide for Scala embedding, plugin registration, cancellation, whole-Job recovery, catalog lifecycle and designer configuration. .

Components and scope

Local execution reads Items directly from the JVM catalog. Explicit remote URIs use configured data connectors and feed local steps. The DACP client provides remote Job submission; Union provides automatic compute placement. Stream connections do not require intermediate files. Materialization and replay are explicit execution-policy choices; operator packages, runtimes and execution history may still use disk.

The example configures an operator repository at http://localhost:8820; item_source@1.0 and client_sink@1.0 are resolved from that repository or a verified cache. Jobs using repository operators require the configured repository or a verified local cache. Run the launcher from the repository root so its relative data and storage paths resolve correctly. It selects exactly one Job output and writes that stream to stdout. The embedded API supports multiple outputs.

See example configuration, example Job, and Job syntax.

Execution history records the actual execution units selected by the engine. Execution units include the runtime model/version and dependency versions supplied by the repository catalog for the pinned image. Shared-process units list their member steps and runtime image; separate steps and host stream sources are recorded separately. Viewers must not infer fusion from matching runtime images.

Local execution and remote data.

Runner sends final bytes/JSONL to stdout as they arrive, without a temporary output file. Output is provisional until Job success. On discard, Runner resets its port state, writes **** XSTREAM: ATTEMPT DISCARDED **** on a separate line to stdout, and reports the reason on stderr; raw stdout cannot retract bytes already written. For retry-aware programmatic consumption, use JobEngine.runToSink with a ResultSink whose onDiscard resets the receiving destination. See embedding.