Development guide
Develop operators and extend data access or execution through public plugin contracts. Choose the extension matching your task, then follow its examples and reference.
flowchart TD
Goal["What are you adding?"] --> Operator["Processing logic"]
Goal --> Data["A data source or representation"]
Goal --> Host["Host integration"]
Operator --> SDK["Operator SDK + inference + samples"]
Data --> Catalog["Catalog / Dataset provider"]
Data --> Format["Format / container / structured reader"]
Data --> View["DataView + Arrow batches"]
Host --> Executor["Executor / build / storage / Chat plugin"]
1. Operator development
Scala is the preferred language for new XStream operators. Start with the JVM SDK templates and keep processing, structure inference and bounded samples in the same operator implementation. Java uses the same JVM contracts; Python, scripts and native commands remain supported for existing code and language-specific libraries.
| Task | Tutorial, examples and reference |
|---|---|
| Write your first operator | Complete authoring tutorial |
| Use Scala templates (preferred) | JVM SDK, Scala examples |
| Implement the same JVM interfaces in Java | JVM SDK |
| Use Python templates | Python SDK, Python examples |
| Wrap scripts or third-party commands | Script examples, native command adapters |
| Declare parameters, defaults and validation | Parameter contract |
| Implement output inference and bounded samples | Structure inference |
| Handle ports, batches, backpressure, cancellation and termination | Stream contract, Arrow ownership |
| Declare dependencies and package code | Dependency environments, packaging reference, runtime selection |
| Test locally | Local operator library, CLI |
| Define an operator inside one local Job | Job-local operators: SDK, command and script |
| Submit source for publication | Submission workflow, provenance, icons |
Processing, structure inference and bounded output samples are part of an operator's implementation. Use the SDK to manage stream framing and resource lifetime.
2. Data extensions
| Task | Contract and example |
|---|---|
| Set up a plugin and register it | Plugin guide |
| Discover Datasets dynamically | DataCatalogProvider |
| Implement a Dataset or Item backend | DatasetProvider, access API |
| Parse a file format | FormatHandler |
| Browse containers or scientific file structures | ContainerReader and StructuredFileHandler |
| Add a representation and its selection parameters | DataView API, Arrow schemas |
| Optimize filtering, projection and limits | Records scan implementation |
| Connect a remote data service | RemoteItemProvider |
Dataset discovery belongs to catalog providers; Items and their views belong to Dataset backends. Preserve scientific groups, dimensions and metadata. Register implementations through ServiceLoader and keep parser dependencies in their plugin modules.
3. Execution and tooling extensions
| Task | Contract and example |
|---|---|
| Add an executor or local builder | Executor plugin, local build extensions |
| Implement execution-record storage | Execution history API |
| Develop a staged Sink | Sink lifecycle |
| Integrate a model API into Designer Chat | Chat configuration and adapters |
| Add source analysis or build recipes | Build extensions |
4. Contract checks
Follow the tests documented by each extension point. Cover discovery, invalid configuration, bounded output, cancellation, failure and resource release. Data extensions also test recursive and immediate-child listing, stable schemas and empty streams; scientific readers test exact structure. Storage plugins test commit/abort, quotas, retention, active leases and corrupted data.
For ordinary catalog configuration and application embedding, use the user guide.