XStream

xstream-data

Typed data access for JVM applications, independent of the job engine and network services. Public contracts live in xstream.data; implementations are separate ServiceLoader plugins. See typed streams and plugin development.

Modules

Build and use

cd xstream-data
sbt compile

Applications depend on core at compile time and selected implementations at runtime. Runner and Designer distributions bundle the scientific plugins. Register formats in META-INF/services/xstream.data.FormatHandler and view support in META-INF/services/xstream.data.DataViewSupport; entries contain fully qualified implementation class names. External JARs and their dependencies can use the existing plugin directory; core registries need no edits.

Configure a file item with format: netcdf, format: fits or format: fasta. NetCDF/FITS accept chunk_bytes (8–16777216, default 65536). FASTA accepts chunk_symbols (1–1048576, default 65536) and alphabet (dna, rna, protein, unknown). Format parameters belong to the item's options map.

NetCDF children use group/variable paths; FITS children use /hdu0, /hdu1, etc.; FASTA exposes /sequences. The file root retains original bytes. Numeric values retain raw encoding and scaling/null metadata. No implicit scientific-to-records flattening occurs. NetCDF/FITS libraries require local random access: remote ByteSources are explicitly staged to a temporary file and deleted on access close; local files are not copied. NetCDF-4/HDF5 support depends on the installed NetCDF backend and is not promised by the bundled pure Java reader. Compressed FITS images and unsupported column types do not imply supported scientific views.

Scientific network adapters preserve typed descriptors and bounded blocks through nodes, Union proxies and clients. item_source supports scientific selections; the designer provides structure and sample previews. Downstream operators must explicitly support the representation. See designer scientific views.

Run parser and Arrow contract tests with sbt test from this module root. See the Arrow representation contract.

Configuration example

datasets:
  science:
    provider: mixed
    items:
      /climate:
        file: /srv/science/climate.nc
        format: netcdf
        options: {chunk_bytes: 65536}
      /sky:
        file: /srv/science/sky.fits
        format: fits
        options: {chunk_bytes: 65536}
      /genome:
        file: /srv/science/genome.fa
        format: fasta
        options: {alphabet: dna, chunk_symbols: 65536}

Complete Scala examples: DataView and view streams.

Public dataset descriptions

DatasetInfo.description and DatasetDefinition.description are JSON object strings requiring a nonempty string title and a string description; additional fields remain extensible. DatasetInfo("weather") generates {"title":"weather","description":""}. Static configuration uses the dataset name as title when the description is omitted. Explicit descriptions are not rewritten or reserialized; aggregating hosts should retain the original string when renaming datasets.

Geospatial and HDF5 extensions

Optional plugins: GDAL raster and vector formats and HDF5 hierarchies and arrays. Both use the same format extension contracts and document installation, configuration, capability limits and native contract tests. Native dependencies belong to plugins rather than core.