XStream

Data views and Arrow streams

A DataCatalog contains Datasets; each Dataset exposes logical Items. An Item may be browsable, readable, or both. A file container can expose original bytes and children simultaneously. Folder status alone does not determine readability.

Public interface

DatasetAccess.view(path, DataViewType[S]) discovers an optional DataView[S]. open(selection, StreamContext) returns a single-use ArrowDataStream. All views expose the same output interface: exactly one start(schema, dictionaries) callback followed by zero or more accept(ArrowBatch) callbacks. Empty streams still supply their schema. The schema remains constant during consumption.

val stream = access.requireView("/observations", DataViews.Records)
  .open(Scan(), StreamContext(allocator, () => cancellation.check()))
try stream.consume(new ArrowConsumer {
  def start(schema: org.apache.arrow.vector.types.pojo.Schema,
            dictionaries: org.apache.arrow.vector.dictionary.DictionaryProvider): Unit =
    println(schema)
  def accept(batch: ArrowBatch): Unit =
    println(batch.root.getRowCount)
}) finally stream.close()

Synchronous callbacks provide backpressure. Roots, vectors and dictionaries are borrowed until the callback returns; producers may reuse them. Use batch.copyTo(allocator) when a consumer needs an independently owned snapshot, and close that snapshot. Reference counting alone does not prevent mutation of a reused buffer. The caller owns the allocator. Producers acquire resources during consumption, release them in finally and call context.check() before reads and emissions. Closing during consumption prevents further callbacks; arbitrary blocking third-party I/O still requires its own cancellation mechanism.

The complete schema, representation, ownership and IPC contract is documented in Arrow view streams.

Standard views

Token Selection Arrow output
DataViews.Bytes AllBytes Binary chunks in data rows
DataViews.Records Scan Ordinary typed columns
ScientificViews.Arrays ArraySelection Multidimensional blocks: origin, shape, stride and typed values
ScientificViews.Sequences SequenceSelection Named sequence chunks with offsets and final markers

Arrays preserve rank and dimensions rather than becoming scalar records. An absent array section selects the whole variable; blocks always carry concrete coordinates. Sequence offsets count symbols. DataViews.blocks and ArrowRepresentations.writer let parsers emit bounded ByteChunk, RecordBatch, ArrayChunk or SequenceChunk values; the resulting public view still emits Arrow. ArrowBlocks is a convenience projection for these standard representations, not a second transport format.

Use Scan for projection, filtering and a global row limit. JDBC and Parquet retain their source optimizations; other readers use bounded filtering. Plain records preserve dictionaries. Scans over dictionary-encoded records remain explicitly unsupported. BatchDatasetAccess and BatchItemAccess adapt existing Arrow producers through BatchReceiver; they do not require a network listener or an IPC round trip.

Original file bytes take precedence. SQL tables expose Arrow IPC bytes. For records-only Items, ItemByteViews derives bounded UTF-8 JSONL bytes (application/x-ndjson), preserving nested values and decoding top-level dictionaries. A byte chunk need not align with a row or UTF-8 character. Errors do not flush buffered partial JSON. View conversion does not implicitly materialize output.

Scientific formats

Format Browsing and views Preservation
CSV / JSONL records and original bytes Typed columns, nulls and nested values
NetCDF groups and variables; variable array view Rank, dimensions, coordinate references, units, fill values and attributes
FITS HDUs; table records and image arrays HDU identity, headers, axes, scaling and coordinate metadata
FASTA original bytes and sequence view Sequence identity, symbol offset, final marker and bounded chunks
JPEG original bytes Decode images inside an operator

Array values are typed Arrow lists, one row per multidimensional block. Dense values use row-major order. Coordinate Items remain logical references, not copied coordinate arrays in every block. Raw numeric values and scaling/missing-value attributes remain explicit; no automatic flattening or silent conversion occurs.

Plugins and network access

Implement DatasetProvider, FormatHandler or StructuredFileHandler and register the public no-argument class in META-INF/services/<interface>. Keep third-party dependencies and configuration validation in the plugin module. A structured reader returns a DatasetAccess over its logical children; storage paths must not leak into the logical namespace. See plugins.

datasets:
  climate:
    provider: my-netcdf
    source:
      root: /srv/climate

my-netcdf is an example external provider. It owns the source settings. A custom representation defines a shared DataViewType[S] token and emits its schema and Arrow batches directly; adding it does not require a core registry or an output codec. Tokens match by identity, not only by their names.

Optional DataViewSupport plugins parse/render request selections and provide bounded schema/sample presentation. Register them through META-INF/services/xstream.data.DataViewSupport. Their output is also ordinary Arrow; they do not encode or decode payload bytes. Built-in array and sequence support lives in xstream-data-scientific-views. Flight carries these typed columns directly, including descriptor metadata and empty schemas. Python as_view, as_array and as_sequences return an ArrowViewReader that iterates PyArrow RecordBatches.

Contract tests cover empty schemas, stable schemas, bounds, borrowed lifetimes, explicit snapshots, failure/cancellation cleanup, dictionary IPC, selection, nested browsing and exact scientific structure. See also designer scientific views.

Complete Scala examples: DataView and view streams.

From Item to stream

Each view selects a representation of one Item. Every selected representation exposes the same Arrow stream contract.

flowchart TD
  Catalog["DataCatalog"] --> Dataset["Dataset"] --> Item["Item"]
  Item --> Bytes["bytes: AllBytes"]
  Item --> Records["records: Scan"]
  Item --> Arrays["array: ArraySelection"]
  Item --> Sequences["sequence: SequenceSelection"]
  Bytes --> Stream["ArrowDataStream"]
  Records --> Stream
  Arrays --> Stream
  Sequences --> Stream
  Stream --> Schema["start: schema + dictionaries"] --> Batches["accept: bounded Arrow batches"]
  Batches --> Consumer["Operator / application"]

Item timestamp metadata

Item.creationTimeMs and lastModifiedTimeMs are optional UTC Unix epoch milliseconds. None means unknown; Some(0L) is the known Unix epoch. Files and physical folders use filesystem-reported values. ZIP entries retain available creation/modification times; TAR entries retain available modification times. SQL objects, scientific-file children and virtual folders depend on their plugins. Never substitute registration, extraction or current time for a source timestamp. Folder size and record count remain unknown. Timestamp queries do not read content or traverse descendants.

View-to-stream lifecycle

flowchart LR
    Item[Item] --> View[Select an advertised View]
    View --> Open[Open with selection and cancellation context]
    Open --> Schema[One schema callback]
    Schema --> Batches[Zero or more ordered Arrow batches]
    Batches --> Consumer[Downstream consumer]
    Consumer --> Close[Close stream and release resources]

Bytes, records and arrays share the stream interface but retain different schemas and semantics. In-process consumers can use Arrow batches directly; crossing a process or network boundary requires transport encoding. A View does not require collecting the entire Item into memory.