Data model: catalogs, Items and streams
XStream separates where data is stored, how it is addressed and how it is consumed. A DataCatalog lists Datasets. A Dataset exposes a tree of logical Items. Select an Item's View to read an Arrow stream. The same Job can read a CSV file, a compressed file's member, a database table or a scientific variable without embedding a filesystem reader in each operator.
The five concepts
| Concept | Responsibility | Example |
|---|---|---|
| DataCatalog | Discover named Datasets in the current execution context or a configured service | The catalog bound to the Runner JVM |
| Dataset | Provide one logical namespace through a backend plugin | weather, backed by /srv/weather |
| Item | Identify a folder, readable object or browsable file container within a Dataset | /observations.csv.gz/observations.csv |
| View | Choose a supported representation and its selection parameters | records with a row limit, or bytes |
| ArrowDataStream | Deliver a schema and ordered, bounded Arrow batches | Records flowing into the next processing step |
A Dataset name is an address component. Its description is a JSON object containing a nonempty string title, a string description and optional application fields. For example, weather can have the human-readable title “Station observations.” An Item's name is its final path component; its path identifies it within the Dataset. Dataset names, Item paths and physical storage paths serve different purposes.
flowchart LR
C["DataCatalog"] --> D["Dataset: weather"]
D --> I["Item: observations.csv"]
I --> R["records View + Scan"]
I --> B["bytes View + AllBytes"]
R --> S["ArrowDataStream"]
B --> S
S --> O["Operator input port"]
A logical tree over physical storage
A directory Dataset maps files and folders under its configured root into logical paths. It also delegates supported archive and scientific-file structures to plugins. The children of such a file are virtual Items: they need not exist as separate files beside the original.
Suppose /srv/weather contains these physical files:
A browsable ZIP/TAR root has kind: folder and views: [bytes, items]. bytes returns the original archive; items enumerates member references without reading their content. Internal folders advertise items; ordinary members retain their own Views. folder_items_source and csv_items_source produce reference collections from directories and CSV manifests respectively. Folder records requests are rejected. See Item reference streams.
/srv/weather/
├── observations.csv
├── observations.csv.gz
├── reports.tar.gz # contains daily/east.csv and daily/west.csv
├── stations.zip # contains metadata/stations.csv
├── climate.nc # contains a variable named temperature
├── genome.fa
└── station.jpg
With the relevant plugins installed, the catalog exposes:
DataCatalog
└── weather Dataset
├── observations.csv Item: bytes, records
├── observations.csv.gz browsable Item: bytes of the GZIP file
│ └── observations.csv virtual Item: decompressed bytes, records
├── reports.tar.gz browsable Item: bytes of the TAR.GZ file
│ └── daily/ virtual folder
│ ├── east.csv virtual Item: bytes, records
│ └── west.csv virtual Item: bytes, records
├── stations.zip browsable Item: bytes of the ZIP file
│ └── metadata/ virtual folder
│ └── stations.csv virtual Item: bytes, records
├── climate.nc browsable Item: original bytes
│ └── temperature variable Item: array View
├── genome.fa browsable Item: original bytes
│ └── sequences Item: sequence View
└── station.jpg Item: bytes
The scientific children shown here depend on the actual file contents. A NetCDF variable may be inside groups. FITS exposes HDUs: image HDUs provide arrays and table HDUs provide records. JPEG currently exposes bytes; decode the image inside an operator. See View examples for Scala consumers.
Browsing and reading are independent. items, bytes, records, array and sequence are peer stream representations; items carries resource references while the others carry opened content. Operators iterate collections internally; Jobs have no map/reduce control structures. Scientific groups advertise only plugin-implemented Views; kind alone does not imply readability.
How GZIP becomes an Item
GZIP compresses a byte stream; it does not itself describe a directory tree. The gzip container plugin exposes one child. Its name is the public Item's final component with .gz removed, ignoring case. The optional filename in the GZIP header is not used. If no useful name can be derived, the child is named content.
For the Dataset weather:
| Address | Meaning |
|---|---|
context:/weather/observations.csv.gz with view: bytes |
Original compressed file |
context:/weather/observations.csv.gz/observations.csv with view: bytes |
Decompressed CSV bytes |
context:/weather/observations.csv.gz/observations.csv with view: records |
Rows parsed from the decompressed CSV |
Removing the suffix creates a child path, not a second sibling. If both /observations.csv and /observations.csv.gz/observations.csv appear, the first comes from the separate physical CSV and the second is derived from GZIP. The child format reader is selected from its filename; a generic payload.gz does not promise a records View. GZIP child size and source timestamps remain unknown when the source cannot supply them reliably.
flowchart LR
G["observations.csv.gz"] -->|"list children"| V["observations.csv: virtual child"]
G -->|"bytes View"| RAW["Compressed bytes in Arrow batches"]
V -->|"bytes View"| CSV["Decompressed CSV bytes in Arrow batches"]
V -->|"CSV parser + records View"| ROWS["Typed columns in Arrow batches"]
ZIP, TAR and TAR.GZ
| File | Virtual namespace |
|---|---|
stations.zip |
Archive paths under /stations.zip/, including intermediate folders |
reports.tar |
Archive paths under /reports.tar/ |
reports.tar.gz or reports.tgz |
TAR entries directly under the compressed archive's Item path |
observations.csv.gz |
One child named observations.csv |
/reports.tar.gz/daily/east.csv is the correct path for the example. There is no automatically inserted /reports.tar.gz/reports.tar/ layer. A sibling /reports.tar exists only if a separate file or explicit publication creates it. The compound TAR.GZ reader takes precedence over generic GZIP handling.
Archive member paths are logical paths; missing intermediate directories are synthesized. Current readers reject invalid paths, duplicate members and file/folder conflicts. TAR links, sparse files and special entries are unsupported. Nested ZIP/TAR/GZIP members remain binary Items and are not recursively expanded. For example, stations.zip/nested.zip can be read as bytes, but does not automatically expose stations.zip/nested.zip/stations.csv.
Discovery, extraction and disk use
Virtual Items do not mean permanent extraction. ZIP browsing reads its index. GZIP browsing exposes its single child after opening the compressed header. TAR and TAR.GZ browsing scans archive entries; TAR.GZ requires sequential decompression, so browsing a large archive may take time.
When reading an archive member, current ZIP, TAR and GZIP plugins stage the requested member as a temporary local file, then pass it to the appropriate reader. Cleanup runs when the operation exits, including on failures. TAR extraction scans the archive, while ZIP can select an entry through its index. Schema or count queries for a child can also require extraction. Ordinary browsing does not create a permanently extracted directory.
The current limits are 10,000 entries, 512 MiB per extracted member and 2 GiB of declared total member sizes for ZIP/TAR, with an additional bound on decompressed TAR headers and padding. GZIP's single extracted child is bounded to 512 MiB. These are plugin limits, not promises that arbitrary large archives are supported.
Input plugins may stage archives in temporary files. This is separate from final result publication and does not create a recovery checkpoint. Steps consume bounded Arrow batches without collecting the complete result in memory. Failure recovery uses whole-Job retries.
Configure and use the catalog
The following fragment defines a Runner catalog:
datasets:
weather:
description: '{"title":"Station observations","description":"CSV observations and archived reports."}'
provider: directory
source:
root: /srv/weather
For Designer configuration, place the same datasets mapping under local_catalog. Catalog roots appear in its Catalogs tab; expand a Dataset and then its folders or archive Items to load children. Selecting a readable View creates a item_source step. Local Items use context: URIs. Expanding the tree does not run a Job.
Archive browsing requires xstream-data-archives and its runtime dependencies, in addition to the directory and format plugins. The Designer distribution bundles archive support. The Runner distribution does not currently bundle that module: put the archive plugin and its dependencies on the runtime classpath or in the configured plugin_directory before using these archive examples. Plugins register through META-INF/services/xstream.data.ContainerReader; the archive plugin supplies zip, gzip, tar and tar-gzip. A CSV child additionally needs the CSV format reader. See plugin discovery and deployment.
This Job reads decompressed observations as records:
name: Read compressed station observations
steps:
- id: read
title: Read observations from GZIP
operator: item_source@1.0
inputs: {}
params:
uri: context:/weather/observations.csv.gz/observations.csv
view: records
- id: client_sink
operator: client_sink@1.0
inputs:
result: read
To read an archived report, replace the URI with context:/weather/reports.tar.gz/daily/east.csv. To read the original archive itself, use context:/weather/reports.tar.gz and view: bytes. context: is resolved by the execution environment's DataCatalog; it is not an operating-system path. The same namespace can be browsed and read through an optional configured remote data service.
What crosses an operator port
An Item is an addressable source; a View supplies its representation. A step input receives the stream produced by an upstream step, not a Dataset object or a filename to reopen. item_source connects the catalog to the processing graph. Subsequent steps reference upstream outputs, for example inputs: {source: read}.
| View | Content of a bounded Arrow batch |
|---|---|
bytes |
Binary chunks in a data column; a chunk need not end at a line or UTF-8 character boundary |
records |
Business columns and typed rows, preserving supported nested values |
array |
Multidimensional blocks with origin, shape, stride and typed values |
sequence |
Named sequence chunks with symbol offsets and final markers |
Every View emits one schema followed by ordered batches; even an empty stream has a schema. Arrays keep their dimensions instead of silently becoming tabular rows. Same-process consumers may use Arrow batches directly; process boundaries use Arrow IPC. This common stream contract does not make every View interchangeable: the operator's input must accept the selected representation.
For API ownership, backpressure and cancellation, read Data views and Arrow streams and Arrow stream contracts. For complete Job syntax, see Jobs.
For built-in reader options, container limits and publication metadata, see the file format and container reference.
Record schema metadata is optional when inference fails. A file with conflicting record field types can still expose its original bytes view; selecting records retains schema validation and reports the parsing error. Native bytes do not require successful record inference.