XStream

Run your first Job

Start here to read a local JSONL file as a record stream. No data server or operator download is needed for this example.

1. Build the runner

Use JDK 17 and sbt 1.10.x. Run from the repository root:

git clone https://github.com/rdcn-link/xstream-codex.git
cd xstream-codex
scripts/bootstrap-local.sh
(cd xstream-engine && sbt stage)

2. Check the catalog

engine.yaml maps Dataset weather to examples/data. The supplied file is examples/data/weather-observations.jsonl. Relative filesystem paths resolve against the process working directory.

datasets:
  weather:
    provider: directory
    source:
      root: examples/data

3. Read the Item

read.yaml contains:

name: Local catalog read
steps:
  - id: read
    operator: item_source@1.0
    inputs: {}
    params:
      uri: context:/weather/weather-observations.jsonl
      view: records
  - id: client_sink
    operator: client_sink@1.0
    inputs:
      result: read

context: selects the current catalog, weather selects its Dataset, and the remaining path selects an Item. outputs.result selects the step output printed by the runner.

xstream-engine/target/universal/stage/bin/xstream-runner \
  examples/xstream/engine.yaml examples/xstream/read.yaml

The terminal prints JSONL records as they arrive. A missing Item usually means the command ran outside the repository root or the catalog root was changed. A bytes view prints original bytes instead.

Continue by task