In-process operator fusion
XStream automatically checks Job placement, operator environments and executor capabilities. When eligible and selected by the configured policy, an entire node subjob uses one isolated container and one JVM or Python interpreter, running steps concurrently in the same process. Internal edges use bounded memory streams and backpressure without passing through the server or Flight and without intermediate result files. Job inputs and final results retain the existing Flight streaming protocol; submissions need no new switch.
Configure the engine or DACP node with operator_fusion (default auto):
operator_fusion: auto # isolated | fused | auto
isolated: execute steps in separate execution units.fused: prefer compatible groups without comparing estimated costs. Runtime/dependency compatibility, executor support, Sink boundaries, retries and checkpoints still constrain fusion; incompatible steps remain isolated.auto: compare structural cost estimates for each compatible candidate. The initial model counts 100 cost units per avoided process startup and 1 per internal edge. Each additional fan-in/fan-out branch adds100 × candidate step count + internal edge countunits of parallelism risk. Only connected linear candidates with positive estimated savings are selected. These are uncalibrated relative units, not milliseconds or a prediction of actual speed. No input is sampled and no operator is executed during selection. Input size, working-set memory and historical performance are not yet modeled.
The setting applies to local Jobs and node-side Union sub-Jobs. Fusion decisions record reasons and, for local candidate groups, structural_v1 estimates with low confidence. Eligibility failures retain their specific reasons. Automatic retries or explicit checkpoints currently keep isolated execution boundaries even in fused mode. The setting belongs to engine/node startup configuration, not the per-Job execution policy. The value must be one of the three mode strings.
Eligibility
- A subjob needs at least two steps. All resolved runtime image SHA-256 IDs must match exactly; matching tags or JDK major versions is insufficient.
- Every step must use the same executor supporting
OperatorExecutor.groupCommand. Mixed JVM, Python and native commands do not fuse. - JVM operators must implement
xstream.operator.Operator, including ScalaBytesOperator,ByteStreamOperatorandRecordsOperatortemplates. Eligibility inspects JAR class metadata without loading user classes in the server JVM. - Python operators must explicitly define a thread-safe
process(ctx). Steps may use different.py, wheel or repository ZIP code bundles. Each bundle uses a private module namespace, including imports insideprocess(ctx), so identical entrypoint and helper module names do not overwrite one another. Repeated references to one bundle reuse its modules; runtime dependencies and the group SDK remain shared in the common image. - Job-local command and script adapters, ordinary container entrypoints and native executables run as isolated steps.
When all steps of an ordinary Job share one node, that node attempts fusion. Union evaluates each member's subjob independently without moving raw data or changing placement. The coordinator first checks the graph obtained by contracting node regions: A → B → A introduces a region cycle and disables fusion, preventing downstream-input waits from blocking upstream startup.
An identical image is necessary but does not prove arbitrary programs can safely share an interpreter. Operators must not depend on process-global arguments, modify shared environment variables or write protocol stdout. Python process(ctx) must not modify sys.argv, sys.stdout, the working directory or mutable module state shared with other steps. Calling System.exit or os._exit terminates the entire subjob. Programs requiring process isolation should use an executor that does not advertise fusion capability.
JVM operators
Existing SDK templates work directly. After building and publishing the Scala common-operator examples, scala_text_uppercase@1.0 and scala_text_strip@1.0 reference the same image and implement the SDK contract. They are not in the current six-operator collaboration repository; publish them before using this example:
name: clean-text
steps:
- id: read_uppercase_source
operator: item_source@1.0
inputs: {}
params:
uri: dacp://localhost:18861/data/message
view: bytes
node: dacp://localhost:18861
- id: uppercase
title: Uppercase text
node: dacp://localhost:18861
operator: scala_text_uppercase@1.0
inputs:
source: read_uppercase_source
params: {}
- id: trim
title: Trim whitespace
node: dacp://localhost:18861
operator: scala_text_strip@1.0
inputs:
source: uppercase
params: {}
- id: client_sink
operator: client_sink@1.0
inputs:
result: trim
The target node must configure the operators' repository and the input Item must exist and be authorized. The fusion launcher carries the current SDK and scheduling code; Arrow, Jackson and Scala dependencies remain supplied by the image. Existing images need not preinstall the new fusion launcher.
Python operators
Expose a module-level process(ctx) business function while retaining an independent execution entrypoint:
from xstream_operator import OperatorContext
def process(ctx):
output = ctx.bytes_output('result', 'text')
for chunk in ctx.input('source').bytes():
output.write(chunk.upper() if ctx.params['mode'] == 'upper' else chunk.lower())
if __name__ == '__main__':
with OperatorContext() as ctx:
process(ctx)
Steps can pass params: {mode: upper} and params: {mode: lower}. Each function receives its own context, fully consumes named inputs, and declares all output metadata before writing data. Output interfaces maintain successful end events. Package imports must have no execution side effects or launch steps. Packages exposing only command-line main are not forced into fusion; publish a new version with process(ctx) to enable it. Repository versions cannot be overwritten, so regenerated bundles require a new version identifier.
Status, limits and resources
Execution records and Union node snapshots include execution_mode: shared_process or isolated_steps, with fusion_reason explaining ineligibility. An ordinary coordinator delegating a complete remote Job records delegated_job; the actual mode is recorded at the target node. Step states and port traffic remain tracked separately, with zero remote payload for internal edges.
Fusion does not promise zero-copy execution: the SDK input adapter still uses Arrow IPC, processed within the same process rather than separate operator processes. Internal byte-channel buffers are 256 KiB and event queues hold at most two frames. Individual Arrow record-batch sizes remain operator-controlled. Item size limits of document operators remain unchanged.
Cancellation or any step failure terminates the entire shared environment, closes pipes and stops unfinished steps. Already executed operators are not retried, avoiding duplicate side effects. The server retains isolation, disabled networking, read-only filesystems and container resource limits. Union authorization and step-slot reservations still count steps; fusion does not loosen permissions or concurrency quotas.
Plugin contributors can implement OperatorExecutor.groupCommand(OperatorGroup). Returning None declines fusion for that combination. A command may carry read-only startup resources in files and code mounts under /operator/. Core handles placement, pinned environment identity, lifecycle, external-port checks and cancellation. Plugins own language loading, internal port contracts, backpressure and step events. Registration still uses META-INF/services/xstream.execution.OperatorExecutor, without core registry changes.
Validation
(cd plugins/xstream-plugin-operator-executors && sbt test)
(cd xstream-operator-sdk && sbt test)
.venv/bin/python scripts/test-python-group.py
.venv/bin/python scripts/test-operator-sdk.py
Tests cover streams larger than internal buffers, same-process PIDs, Arrow records and nulls, port-contract failures and unsupported-entrypoint fallback.
Performance comparison
See the streaming benchmark operators for a reproducible large-data comparison. The node configuration switch compares both strategies using identical images, code and Jobs, and retains every raw sample.