Item collection operators
Scala SDK implementations of folder_items_source, csv_items_source, and item_inventory, item_byte_count. Sources emit Item references, not file contents. They use the execution host's private Item access service and require a runtime containing the matching SDK, execution API and data-core versions.
| Operator | Role | Input | Output |
|---|---|---|---|
| folder_items_source | source | URI parameter, recursive flag | items collection |
| csv_items_source | source | CSV URI parameter, uri_column | items collection |
| item_inventory | transform | items | text bytes containing URI/kind/format |
| item_byte_count | transform | items | URI and byte count, reading one member at a time |
Build and test
cd examples/operators/item-sources
sbt test package
The JVM SDK must be available in your configured artifact repository. *.json files illustrate execution and port declarations; repository submission additionally requires runtime pins, source provenance, icon, parameter help/examples and complete usage documentation. Output ports are inferred by operator methods, never declared at manifest top level. See operator submission.
Use locally
Configure an engine inline_runtimes.jvm_tools using a locally installed matching JVM image. Add the following definition to a local Job and reference it as operator: {inline: folder_items}:
operators:
folder_items:
type: sdk
role: source
runtime: jvm_tools
program:
path: ./target/scala-2.13/xstream-item-sources_2.13-1.0.jar
entrypoint: xstream.examples.items.FolderItemsSource
inputs: {}
The step parameters are {uri: 'context:/science/labels', recursive: true}. For CSV use CsvItemsSource and {uri: 'context:/science/manifest.csv', uri_column: uri}. CSV manifests preserve row order, duplicates and other columns as metadata; member metadata can be unknown until opening. Relative paths refer to the manifest's logical parent. Raw filesystem paths and URI credentials are not accepted.
Collection consumers declare representations: [items] and use ctx.input("source").foreachItem. They may open a member with ctx.openItem(item, "bytes") { stream => ... }; consume incrementally and release it inside the callback. See Item reference streams. Remote Jobs require approved repository references; inline definitions are local only.
For a single Item, use the view parameter of item_source to read its contents directly. Collection Sources emit references for explicit access by collection consumers.
Publish
Register runtime jvm-items@1.0 with the matching SDK, then run:
python3 publish.py --url http://localhost:8820 \
--token-file /path/to/submitter.token \
--approve-password-file /path/to/admin-password
Omit the approval password option to await administrator review. item_byte_count skips directories without bytes, preserves duplicates and ordering, and fails the Job if a member cannot be read.
End-to-end verification
After building the matching runtime image and runner, run:
python3 scripts/test-items-stream.py --image xstream/operator-jvm-items:1.0
Run from the project root with Docker access and a published client_sink@1.0. The test uses isolated containers to verify ZIP folders, duplicate CSV entries, lazy reads, missing-member failure and traversal rejection.