XStream

Adding a format handler

The file backend resolves file mappings through each plugin environment’s Formats. SqlAdapter handles SQL separately. Detection precedence is an explicit configuration override, compound container suffixes, provider signature matches (at most eight bytes), provider extension matches, then binary fallback. Extensions are lowercase and case-insensitive on filenames. IDs match [a-z][a-z0-9-]*; sql is reserved. Duplicate IDs or extensions fail registry initialization.

Implement xstream.data.FormatHandler in a JAR compiled against the standalone xstream-data-core with a Provided dependency. Scala providers use its trait directly; Java implementations can implement the generated JVM interface. Scala trait defaults are easiest to reuse from Scala. Compile the plugin and register it as shown below; keep parser dependencies in that plugin module. Providers must have a public no-argument constructor.

package example

import xstream.data.{ByteSource, Records, FileOptions, FormatHandler}
import scala.jdk.CollectionConverters._

class LineProvider extends FormatHandler {
  val id = "lines"
  val extensions = Set("lines")
  override val records = true
  override def rows(source: ByteSource, options: FileOptions): Records.Rows =
    new Records.Rows {
      private val input = new java.io.BufferedReader(
        new java.io.InputStreamReader(source.open(() => ()), java.nio.charset.StandardCharsets.UTF_8))
      val names = Seq("value")
      val iterator = input.lines().iterator().asScala.map(line => Seq(line))
      def close(): Unit = input.close()
    }
}

Add META-INF/services/xstream.data.FormatHandler to the provider JAR:

example.LineProvider

Put the JAR and its private dependencies in the configured plugin_directory and restart. The registry discovers it through ServiceLoader. A Java provider must implement Scala collection signatures; a small Scala adapter can delegate parsing to any Java library, as Excel delegates to Apache POI.

rows supplies ordered column names, a lazy iterator, and an idempotent close. Shared default methods infer a schema from a bounded sample, stream record batches, and compute explicit counts. Override schema, stream, or count for formats such as Parquet that have specialized readers. cheapSchema and cheapCount run during listing and must avoid full data scans. Detection, listing, and reading must agree on the format ID and advertised capability. A format with records = false exposes bytes only.

Inputs use reopenable ByteSource, without requiring local Path. A source supplies a public filename, optional size/version, streams, and optional range reads. Source plugins own connections, authentication and consistent object versions within an operation. Callers close opened streams. CSV/NDJSON read streams directly; Parquet uses size and ranges; Excel explicitly uses ByteSource.materialize and cleans temporary files when its reader closes. localPath is only an optional optimization. Never put credentials or private URIs in name.

The shared file reader sends original bytes without conversion. Opening records must not alter the source. Implementations must close parser resources on success, failure, and cancellation, and call the supplied demand callback before reading each record batch. Third-party exceptions must be mapped to FormatFailure with a fixed public message, without filenames, sheet names, data values, or library diagnostics. SDKs translate these into FORMAT_ERROR.

FileOptions carries private format, ordered Arrow columns, and a parameters: Map[String,Object] populated from YAML options. Override validateOptions to validate handler-owned keys, types, and values. The default rejects all parameters. Excel owns options.sheet and options.header_row; shared code contains no Excel-specific option validation. Nested YAML values retain Java map/list representations. The options stay host-side: public tickets contain only the dataset, logical Item path, and view. Schema is unavailable in ordinary CSV/NDJSON/Excel listings; opening records infers it or validates configured columns. Folder properties remain unknown and never visit descendants.

Excel uses HSSF events for XLS and SAX/XSSF events for XLSX, a 16-row queue, and cancellation checks between parser events. Shared-string dictionaries and individual rows/cells still consume memory. Numeric dates remain Excel serial values. Only cached formula results are read; formulas are never recalculated. Multiple sheets require a configured selection. This is read support, without workbook editing or spreadsheet rendering.

Container readers

ContainerReader is the separate interface for enumerating children and staging an entry as a seekable file. Register a reader in META-INF/services/xstream.data.ContainerReader. Declare its id, extensions, and optional compound filename matcher; the format registry automatically creates the matching format entry. Do not also register a format with the same ID. Built-in and external readers share the same interface. Duplicate IDs/extensions and ambiguous filename matches are rejected.

NamespaceAdapter publishes mapped containers as folders with a bytes view returning the original file, infers parent folders, and passes staged children to existing format readers. Folder size and record count stay unknown; available source timestamps remain optional. Nested containers remain binary data. GZIP's one child is named from the public alias, ignoring private filenames. Compound .tar.gz and .tgz suffixes resolve before ordinary extension detection.

ZIP listings use the archive directory without decompressing payloads. TAR and compressed TAR scan sequentially, with decompression for the latter; GZIP listings only read the header. Entry reads stage one file and clean it on success, failure, or cancellation. TAR scans validate the whole archive, including entries after the requested one, before exposing staged content. Scans and decompression must honor the supplied demand/cancellation callback.

Limits are 10,000 entries, 512 MiB per decompressed entry, and 2 GiB total declared size. TAR also bounds total decompressed bytes to that size plus 16 MiB of metadata/padding. Duplicate entries, invalid paths, and file/parent conflicts are rejected. TAR links, sparse files, and special entries are unsupported. Container roots reject records; their data children advertise views through the regular format registry.

Native structured files

Implement StructuredFileHandler (a FormatHandler) and register it in the format service file. openStructure(source, options, context, check) returns a request-scoped DatasetAccess rooted at /. It can expose logical objects without extracting files. List public absolute paths relative to that root; the backend prefixes the published file alias. Implement both recursive list() and immediate-child list(path), plus describe, count, and typed view. A view opens a synchronous ArrowDataStream, with its descriptor announced even when empty. Child Items must explicitly advertise their views when using native formats that do not have a file handler. Record schema must match the selected records view. Scientific Views preserve their structure in standard Arrow schemas and descriptor metadata instead of flattening arrays into business rows. The server closes access handles on success and failure. Honor check during discovery and StreamContext.check before reading and emitting blocks, and use close to release file handles even after cancellation.

The file root is a folder with original-file bytes and unknown properties. Folder metadata does not open its structure or compute descendants. Nested logical folders also have unknown properties. Native child bytes have handler-defined semantics that must be documented; the root bytes always preserve the physical file. Ordinary listing must not compute expensive counts. Startup counting skips structured roots.

Use PluginChecks.listing(access) for metadata contract tests. Also test streaming, cancellation, sanitized errors and resource cleanup independently of a service. XStream NetCDF and FITS plugins expose groups, variables or HDUs through this interface and stream numeric arrays as typed Arrow batches. Specify any record projection explicitly rather than silently flattening arrays.

Records scans

FormatHandler.scan supplies bounded batch filtering, projection, and early limit termination for every records handler. Override it to optimize the source reader; directory-backed and mixed datasets invoke the same contract, including for archive children. Native structured children use DataViews.Records. See records scan contracts.

For built-in reader options, container limits and publication metadata, see the file format and container reference.

Geospatial and HDF5 extensions

Optional plugins: GDAL raster and vector formats and HDF5 hierarchies and arrays. Both use the same format extension contracts and document installation, configuration, capability limits and native contract tests. Native dependencies belong to plugins rather than core.