XStream

Development guide

Develop operators and extend data access or execution through public plugin contracts. Choose the extension matching your task, then follow its examples and reference.

flowchart TD
  Goal["What are you adding?"] --> Operator["Processing logic"]
  Goal --> Data["A data source or representation"]
  Goal --> Host["Host integration"]
  Operator --> SDK["Operator SDK + inference + samples"]
  Data --> Catalog["Catalog / Dataset provider"]
  Data --> Format["Format / container / structured reader"]
  Data --> View["DataView + Arrow batches"]
  Host --> Executor["Executor / build / storage / Chat plugin"]

1. Operator development

Scala is the preferred language for new XStream operators. Start with the JVM SDK templates and keep processing, structure inference and bounded samples in the same operator implementation. Java uses the same JVM contracts; Python, scripts and native commands remain supported for existing code and language-specific libraries.

Task Tutorial, examples and reference
Write your first operator Complete authoring tutorial
Use Scala templates (preferred) JVM SDK, Scala examples
Implement the same JVM interfaces in Java JVM SDK
Use Python templates Python SDK, Python examples
Wrap scripts or third-party commands Script examples, native command adapters
Declare parameters, defaults and validation Parameter contract
Implement output inference and bounded samples Structure inference
Handle ports, batches, backpressure, cancellation and termination Stream contract, Arrow ownership
Declare dependencies and package code Dependency environments, packaging reference, runtime selection
Test locally Local operator library, CLI
Define an operator inside one local Job Job-local operators: SDK, command and script
Submit source for publication Submission workflow, provenance, icons

Processing, structure inference and bounded output samples are part of an operator's implementation. Use the SDK to manage stream framing and resource lifetime.

2. Data extensions

Task Contract and example
Set up a plugin and register it Plugin guide
Discover Datasets dynamically DataCatalogProvider
Implement a Dataset or Item backend DatasetProvider, access API
Parse a file format FormatHandler
Browse containers or scientific file structures ContainerReader and StructuredFileHandler
Add a representation and its selection parameters DataView API, Arrow schemas
Optimize filtering, projection and limits Records scan implementation
Connect a remote data service RemoteItemProvider

Dataset discovery belongs to catalog providers; Items and their views belong to Dataset backends. Preserve scientific groups, dimensions and metadata. Register implementations through ServiceLoader and keep parser dependencies in their plugin modules.

3. Execution and tooling extensions

Task Contract and example
Add an executor or local builder Executor plugin, local build extensions
Implement execution-record storage Execution history API
Develop a staged Sink Sink lifecycle
Integrate a model API into Designer Chat Chat configuration and adapters
Add source analysis or build recipes Build extensions

4. Contract checks

Follow the tests documented by each extension point. Cover discovery, invalid configuration, bounded output, cancellation, failure and resource release. Data extensions also test recursive and immediate-child listing, stable schemas and empty streams; scientific readers test exact structure. Storage plugins test commit/abort, quotas, retention, active leases and corrupted data.

For ordinary catalog configuration and application embedding, use the user guide.