XStream

STREAM-ORIENTED COMPUTATION

Keep your data
in motion.

Read structured data. Connect operators. Deliver a result stream. XStream brings typed data access, an embeddable engine and visual Job design into one Scala toolkit.

Powered byApache Arrow

Open development · Scala / JVM · Python & native operators

A JOB IN MOTION STREAMING
01
Read an Itemcontext:/weather/observations.csv
↗
records ↓
02
Transform the streamTyped inputs → typed outputs
◇
records ↓
03
Deliver resultsConsume output as it arrives
↗

Stream edges connect the work.
Materialize selected boundaries when required.

SEE XSTREAM IN ACTION

From a visual flow to a result stream.

Design in your browser. Execute from your terminal. These captures show a real weather-data example.

MEASURED EXECUTION

Compare the work, not just the diagram.

Compare Airflow file handoffs, LangGraph model tool scheduling and XStream streams using identical Scala/Python operators. The reports include all timings and measured completion counts.

View performance results →

ONE CONNECTED TOOLKIT

From a data Item to a running flow.

01

Typed data access

DataCatalog → Dataset → Item. Explicit views preserve records, bytes, arrays and sequences without flattening scientific structures.

Read the guide ↗
02

An embeddable engine

Run YAML Jobs locally or embed the engine in a JVM application. Stream dependencies connect steps; execution policies control materialization and replay.

Read the guide ↗
03

Visual Job design

Browse catalogs, connect ports and edit typed parameters. Preview output structures and bounded samples, then export a YAML Job.

Read the guide ↗
04

An operator ecosystem

Develop JVM, Python and native operators. Publish through the independent operator repository for builds and review; public packages include approved runtime metadata.

Explore operator repositories →
05

Local execution, remote data

Process local data and remote Item streams in the same Job. Browse catalogs, choose a source and keep downstream processing on the local engine.

Read the guide ↗
06

Extend through plugins

Add formats and backends through shared extension contracts. Built-in and contributed implementations follow the same discovery model.

Read the guide ↗

EXPLICIT BY DESIGN

Streams by default.
Persistence by policy.

Intermediate data can flow without saving files between steps. Materialization and replay are explicit execution choices. Operator packages, execution history and runtime storage may still use disk; streaming is not a promise of zero disk activity.