Scala Source and Sink examples
Four small operators demonstrate graph roles independently of implementation type. All four use type: sdk and the Scala JVM SDK. They process Arrow-backed bytes streams in bounded chunks; no file is written to the server and no external service is contacted.
Each implementation uses the base Operator interface; the published manifest declares its Source or Sink role and port contract. The dedicated SourceOperator and SinkOperator interfaces are optional role markers.
| Operator | Role | Behavior |
|---|---|---|
text_source@1.0 |
Source | Emits UTF-8 text repeated repeat times, with an 8 MiB output bound |
range_source@1.0 |
Source | Emits count signed integers from start, one per line; at most 100,000 values |
contains_text_sink@1.0 |
Sink | Consumes bytes and fails the Job unless text occurs, including across chunk boundaries |
min_lines_sink@1.0 |
Sink | Consumes bytes and fails the Job unless at least minimum lines occur |
Source steps have no inputs. Sink steps consume an input and have no output port. For example:
name: Check generated lines
steps:
- id: generate
operator: range_source@1.0
inputs: {}
params: {start: 1, count: 100}
- id: check
operator: min_lines_sink@1.0
inputs: {source: generate}
params: {minimum: 100}
To return a Source stream to a caller, connect it to client_sink@1.0 instead of a validation Sink. Source and Sink examples do not replace item_source@1.0, which reads catalog Items, or dataset publication, which persists data products.
Run contract tests with sbt test in this directory. Submit and approve on a local repository with:
python3 publish.py --url http://127.0.0.1:8820 \
--token-file /secure/submitter.token \
--approve-password-file /secure/admin-password
Omit --approve-password-file to leave submissions pending for administrator review. The script packages only Scala source, pom.xml, documentation and license; the repository builds the thin JAR. Runtime jvm-arrow@2.1 must already be registered. See operator development and submission.