XStream
XStream provides Dataset management and Job execution through a Scala data SDK, an embeddable streaming engine and a browser-based visual designer. Steps consume and produce streams; intermediate data need not be saved as files. Explicit execution policies enable whole-Job recovery when required.
Scala is the preferred language for new XStream operators. Start with the JVM SDK templates and keep processing, structure inference and bounded samples in the same operator implementation. Java uses the same JVM contracts; Python, scripts and native commands remain supported for existing code and language-specific libraries.
XStream executes every Job step locally. Remote item_source URIs provide data streams to local operators. DACP clients submit Jobs to remote servers; Union owns automatic placement across its members.
User guide · Development guide
Components
| Module | Purpose |
|---|---|
| xstream-data | DataCatalog → Dataset → Item, typed DataView/DataStream contracts, plugin discovery, file and scientific-format implementations and network codecs |
| xstream-operator-sdk | JVM/Python operator templates and metadata contracts |
| xstream-operator-inference | Reusable on-demand structure and sample workers |
| xstream-operator-repository | Source submissions, isolated builds, review, public operator and runtime downloads |
| xstream-engine | Local embedded execution, cancellation, execution policies and the xstream-runner launcher |
| xstream-designer | Visual Job editing, catalogs, variables, schema and sample previews, chat proposals and YAML import/export |
context:/dataset/item resolves through the catalog bound to the current JVM. Explicit dacp://host:port/dataset/item URIs use configured remote connections. Scientific arrays, images and sequences keep their structures rather than being flattened into records. Format and backend implementations register through plugin contracts.
Run your first Job, then use the documentation index for data management, operator development and embedding APIs.
Build
Use JDK 17 and sbt 1.10.x. Building requires only this checkout, without DACP source or DACP artifacts. The bootstrap script publishes XStream development artifacts to the local repository in dependency order.
git clone git@github.com:rdcn-link/xstream-codex.git
cd xstream-codex
scripts/bootstrap-local.sh
sbt test
(cd xstream-engine && sbt stage)
(cd xstream-designer && sbt stage)
Read remote Items using DACP URIs, or use context: for the configured local catalog. See local execution and remote data.
Run
From the XStream repository root:
xstream-engine/target/universal/stage/bin/xstream-runner \
examples/xstream/engine.yaml examples/xstream/read.yaml
xstream-designer/target/universal/stage/bin/xstream-designer \
xstream-designer/example.yaml
Edit configuration paths, service addresses and credentials first. The runner prints its selected output stream to stdout; the designer listens on http://127.0.0.1:8840/ by default. Operators use approved public repositories, verified package caches, or explicit local source libraries for standalone development. item_source@1.0 and client_sink@1.0 must be available in the configured repository or verified package cache.
Documentation
- Write an operator
- Define an operator inside a local Job
- Submit source and review
- Engine and embedding
- Typed streams and scientific views
- Arrow stream contracts and representations
- Scientific authoring and previews
- Plugin contracts and registration
- Job YAML
Logo
The vector logo and symbol depict crossing streams forming an X. The designer uses the symbol as its favicon.
Complete Scala examples: DataView and view streams.
Architecture
The local catalog, engine and designer are usable independently. Remote services are optional connectors.
flowchart LR
Designer["Visual Designer"] --> Job["Job YAML + policy"]
Runner["Runner / embedded host"] --> Engine["XStream engine"]
Job --> Engine
Catalog["DataCatalog"] --> View["Item / DataView"]
View --> Arrow["Arrow streams"] --> Engine
Repository["Operator repository"] --> Engine
Engine --> Result["Result stream"]
Remote["Optional remote connector"] -.-> View
Visual identity
The canonical visual assets are assets/xstream-mark.svg and assets/xstream-logo.svg. The X symbol sits above five parallel, continuous small waves with short wavelengths and low amplitude. Keep the symbol geometry, dark background and cyan/blue/purple/amber palette consistent across applications.
Dependency declarations, locked environments, execution units and binary wrappers: dependency environment guide.
Performance benchmarks
Mixed-language Airflow, model tools and XStream benchmark. Programs and reproduction.