xstream-data
Typed data access for JVM applications, independent of the job engine and network services. Public contracts live in xstream.data; implementations are separate ServiceLoader plugins. See typed streams and plugin development.
Modules
xstream-data-core: DataCatalog → Dataset → Item, DataView/ArrowDataStream, selections, plugin lifecycle and standard Arrow representations.xstream-data-files,xstream-data-catalog-static,xstream-data-mounted: file/JDBC, configured and mounted catalogs.xstream-data-formats-basic,xstream-data-parquet,xstream-data-excel,xstream-data-archives: existing readers and structured browsing.xstream-data-netcdf: NetCDF groups and numeric variables, bounded strided array selections. Non-numeric variables remain browsable without an array view.xstream-data-fits: HDUs, bounded array tiles and supported fixed-size binary tables; variable-size and complex table cells are not advertised as records.xstream-data-fasta: bounded sequence chunks with IDs, symbol offsets and final markers; ID/range selection scans the source sequentially.xstream-data-scientific-views: array/sequence request selection, schema presentation and bounded Arrow batch samples.
Build and use
cd xstream-data
sbt compile
Applications depend on core at compile time and selected implementations at runtime. Runner and Designer distributions bundle the scientific plugins. Register formats in META-INF/services/xstream.data.FormatHandler and view support in META-INF/services/xstream.data.DataViewSupport; entries contain fully qualified implementation class names. External JARs and their dependencies can use the existing plugin directory; core registries need no edits.
Configure a file item with format: netcdf, format: fits or format: fasta. NetCDF/FITS accept chunk_bytes (8–16777216, default 65536). FASTA accepts chunk_symbols (1–1048576, default 65536) and alphabet (dna, rna, protein, unknown). Format parameters belong to the item's options map.
NetCDF children use group/variable paths; FITS children use /hdu0, /hdu1, etc.; FASTA exposes /sequences. The file root retains original bytes. Numeric values retain raw encoding and scaling/null metadata. No implicit scientific-to-records flattening occurs. NetCDF/FITS libraries require local random access: remote ByteSources are explicitly staged to a temporary file and deleted on access close; local files are not copied. NetCDF-4/HDF5 support depends on the installed NetCDF backend and is not promised by the bundled pure Java reader. Compressed FITS images and unsupported column types do not imply supported scientific views.
Scientific network adapters preserve typed descriptors and bounded blocks through nodes, Union proxies and clients. item_source supports scientific selections; the designer provides structure and sample previews. Downstream operators must explicitly support the representation. See designer scientific views.
Run parser and Arrow contract tests with sbt test from this module root. See the Arrow representation contract.
Configuration example
datasets:
science:
provider: mixed
items:
/climate:
file: /srv/science/climate.nc
format: netcdf
options: {chunk_bytes: 65536}
/sky:
file: /srv/science/sky.fits
format: fits
options: {chunk_bytes: 65536}
/genome:
file: /srv/science/genome.fa
format: fasta
options: {alphabet: dna, chunk_symbols: 65536}
Complete Scala examples: DataView and view streams.
Public dataset descriptions
DatasetInfo.description and DatasetDefinition.description are JSON object strings requiring a nonempty string title and a string description; additional fields remain extensible. DatasetInfo("weather") generates {"title":"weather","description":""}. Static configuration uses the dataset name as title when the description is omitted. Explicit descriptions are not rewritten or reserialized; aggregating hosts should retain the original string when renaming datasets.
Geospatial and HDF5 extensions
Optional plugins: GDAL raster and vector formats and HDF5 hierarchies and arrays. Both use the same format extension contracts and document installation, configuration, capability limits and native contract tests. Native dependencies belong to plugins rather than core.