XStream

XStream

XStream

XStream provides Dataset management and Job execution through a Scala data SDK, an embeddable streaming engine and a browser-based visual designer. Steps consume and produce streams; intermediate data need not be saved as files. Explicit execution policies enable whole-Job recovery when required.

Scala is the preferred language for new XStream operators. Start with the JVM SDK templates and keep processing, structure inference and bounded samples in the same operator implementation. Java uses the same JVM contracts; Python, scripts and native commands remain supported for existing code and language-specific libraries.

XStream executes every Job step locally. Remote item_source URIs provide data streams to local operators. DACP clients submit Jobs to remote servers; Union owns automatic placement across its members.

User guide · Development guide

Components

Module Purpose
xstream-data DataCatalog → Dataset → Item, typed DataView/DataStream contracts, plugin discovery, file and scientific-format implementations and network codecs
xstream-operator-sdk JVM/Python operator templates and metadata contracts
xstream-operator-inference Reusable on-demand structure and sample workers
xstream-operator-repository Source submissions, isolated builds, review, public operator and runtime downloads
xstream-engine Local embedded execution, cancellation, execution policies and the xstream-runner launcher
xstream-designer Visual Job editing, catalogs, variables, schema and sample previews, chat proposals and YAML import/export

context:/dataset/item resolves through the catalog bound to the current JVM. Explicit dacp://host:port/dataset/item URIs use configured remote connections. Scientific arrays, images and sequences keep their structures rather than being flattened into records. Format and backend implementations register through plugin contracts.

Run your first Job, then use the documentation index for data management, operator development and embedding APIs.

Build

Use JDK 17 and sbt 1.10.x. Building requires only this checkout, without DACP source or DACP artifacts. The bootstrap script publishes XStream development artifacts to the local repository in dependency order.

git clone git@github.com:rdcn-link/xstream-codex.git
cd xstream-codex
scripts/bootstrap-local.sh
sbt test
(cd xstream-engine && sbt stage)
(cd xstream-designer && sbt stage)

Read remote Items using DACP URIs, or use context: for the configured local catalog. See local execution and remote data.

Run

From the XStream repository root:

xstream-engine/target/universal/stage/bin/xstream-runner \
  examples/xstream/engine.yaml examples/xstream/read.yaml
xstream-designer/target/universal/stage/bin/xstream-designer \
  xstream-designer/example.yaml

Edit configuration paths, service addresses and credentials first. The runner prints its selected output stream to stdout; the designer listens on http://127.0.0.1:8840/ by default. Operators use approved public repositories, verified package caches, or explicit local source libraries for standalone development. item_source@1.0 and client_sink@1.0 must be available in the configured repository or verified package cache.

Documentation

The vector logo and symbol depict crossing streams forming an X. The designer uses the symbol as its favicon.

Complete Scala examples: DataView and view streams.

Architecture

The local catalog, engine and designer are usable independently. Remote services are optional connectors.

flowchart LR
  Designer["Visual Designer"] --> Job["Job YAML + policy"]
  Runner["Runner / embedded host"] --> Engine["XStream engine"]
  Job --> Engine
  Catalog["DataCatalog"] --> View["Item / DataView"]
  View --> Arrow["Arrow streams"] --> Engine
  Repository["Operator repository"] --> Engine
  Engine --> Result["Result stream"]
  Remote["Optional remote connector"] -.-> View

Visual identity

The canonical visual assets are assets/xstream-mark.svg and assets/xstream-logo.svg. The X symbol sits above five parallel, continuous small waves with short wavelengths and low amplitude. Keep the symbol geometry, dark background and cyan/blue/purple/amber palette consistent across applications.

Dependency declarations, locked environments, execution units and binary wrappers: dependency environment guide.

Performance benchmarks

Mixed-language Airflow, model tools and XStream benchmark. Programs and reproduction.