Galaxy streaming operator adapters
The example repository provides 200 distinct Galaxy operators; the extension families are listed below. Exact public versions and tested artifact hashes are recorded in the catalog audit. Each uses Arrow bytes inputs (source, and reference for paired adapters) and a result bytes output; no filesystem path or intermediate file is required. A stream interface does not imply constant memory: whole-stream sorting and reversal use bounded buffers; biological tools may buffer one bounded record. All normalized text output uses LF and JVM Unicode lexical order where sorting is involved.
| Operator | Purpose | Supported scope |
|---|---|---|
galaxy_reverse_lines@1.0 |
Reverse the order of newline-separated text records. | Newline records only; a bounded 8 MiB-character buffer is required. No custom separators. |
galaxy_number_lines@1.0 |
Number nonempty text lines with configurable starting number, increment and alignment. | Empty lines are unnumbered. Logical-page escape sequences are treated as literal text. |
galaxy_sort_lines@1.0 |
Stable sort of text rows using a selected tab-separated key. | One full-field sort key; numeric mode requires finite decimal/scientific values. No locale, random or human-size sorting. Bounded 8 MiB-character row buffer. |
galaxy_sort_fields@1.0 |
Sort the tab-separated fields within each row alphabetically. | As upstream, strips surrounding row whitespace before sorting fields. |
galaxy_unique_lines@1.0 |
Sort text lines and retain one occurrence of each comparison key. | Whole line or one selected field; stable representative, bounded 8 MiB-character buffer. Header lines pass unchanged. |
galaxy_unique_consecutive@1.0 |
Collapse adjacent equal text records, optionally counting or selecting duplicate groups. | Only consecutive duplicates are grouped; no arbitrary field skipping or blank-delimited grouping. |
galaxy_unfold_column@1.0 |
Expand a delimited list in one tab-separated field into multiple rows. | Literal nonempty separator; at most 10000 output rows and 16 MiB estimated characters per input row. Fixes upstream last-column newline handling. |
galaxy_replace_text_in_column@1.0 |
Replace literal text within one tab-separated field. | Literal matching only: no regular expressions, shell commands, sed programs or replacement backreferences. Case-insensitive matching is locale-independent. Missing selected columns fail. |
galaxy_replace_text_in_line@1.0 |
Replace literal text within complete lines. | Literal matching only: no regular expressions, shell commands, sed programs or replacement backreferences. Case-insensitive matching is locale-independent. Missing selected columns fail. |
galaxy_find_and_replace@1.0 |
Replace literal text within complete lines. | Literal matching only: no regular expressions, shell commands, sed programs or replacement backreferences. Case-insensitive matching is locale-independent. Missing selected columns fail. |
galaxy_grep_lines@1.0 |
Select lines containing a literal search string, with optional inversion and case folding. | Literal text search; no regex, context-line expansion or HTML output. |
galaxy_arrange_columns@1.0 |
Reorder table columns by their header names, optionally discarding unselected columns. | Unique header names and requested names are required. Ragged rows and unknown headers fail. |
galaxy_add_line@1.0 |
Add a literal header or footer line to a text stream. | Existing data lines are normalized to LF; this avoids concatenating a footer onto an unterminated final row. |
galaxy_recurring_lines@1.0 |
Generate a configured line repeatedly after draining the input stream. | One explicit repetition group, bounded 16 MiB output expansion; input is drained, not used to infer repetitions. |
galaxy_remove_columns_by_header@1.0 |
Remove named table columns or keep only the named columns. | Tab-delimited UTF-8; optional leading-character stripping on first header field. Retains source column order. Unknown requested headers fail. |
galaxy_sort_columns_by_header@1.0 |
Alphabetically reorder table columns by header, optionally pinning one column first. | Zero sorts every column; otherwise a one-based column stays first. Headers must be unique. |
galaxy_add_input_name@1.0 |
Add an input label as a new table column, with optional header and prepending. | The input label is explicit configuration; no local path or filename is required. |
galaxy_fixed_width_to_tabular@1.0 |
Convert explicitly configured fixed-width fields into trimmed tab-separated columns. | Explicit character widths only; no automatic boundary detection, FORTRAN descriptors, reports or encoding conversion. Long rows or unpadded short rows fail. |
galaxy_colorspace_to_fasta@1.0 |
Convert FASTA-like SOLiD color-space sequences to nucleotide FASTA. | A/C/G/T primer followed only by 0..3; wrapped sequence lines accepted. Buffers at most one 8 MiB-character sequence. |
galaxy_split_paired_reads@1.0 |
Extract one mate from concatenated equal-length paired FASTQ records. | Select mate 1 or 2; validates four-line FASTQ, optional matching + title, equal sequence/quality lengths and even sequence length. No silent truncation. An empty + title remains empty so emitted FASTQ remains valid. |
galaxy_pileup_interval@1.0 |
Collapse consecutive pileup positions meeting a minimum depth into BED-like sequence intervals. | One-based pileup to zero-based half-open intervals. Configurable columns cover six-column, legacy ten-column and custom rows. Active interval limited to 8 MiB bases; blank lines are skipped rather than terminating all input. |
galaxy_divide_pg_snp@1.0 |
Expand one or two pgSnp alleles, frequencies and scores into separate columns. | Rows with more than two alleles are skipped. Homozygous second allele is N or the configured reference column; extra input columns are retained. |
galaxy_vcf_to_pgsnp@1.0 |
Convert called VCF SNP genotypes to pgSnp allele counts. | Zero selects all samples; otherwise use a one-based VCF column >=10. Only single-base A/C/G/T SNPs with PASS or missing FILTER are converted. GT position, haploidy, phased calls and all allele indices are handled; missing alleles ignored. |
galaxy_reverse_complement@1.0 |
Reverse-complement FASTA nucleotide sequences using the IUPAC alphabet. | FASTA input/output only; DNA or RNA IUPAC symbols, case preserved, invalid characters rejected. One sequence buffered up to 8 MiB characters. |
galaxy_translate@1.0 |
Translate nucleotide FASTA to protein FASTA with the standard genetic code. | Standard code table 1, FASTA only. Ambiguous complete codons emit X; terminal partial codons fail. CDS requires ATG start, terminal stop and no internal stop; final stop omitted. Alternate genetic codes are not advertised. |
galaxy_codon_count@1.0 |
Count all 64 unambiguous DNA codons over FASTA sequences in frame zero. | Input lengths must be multiples of three. Only A/C/G/T codons are accepted; output is deterministic lexicographic codon/count TSV, including zero counts. |
galaxy_base_distribution@1.0 |
Report per-sequence A/C/G/T counts, sequence length and GC fraction. | FASTA input. GC denominator includes ambiguous symbols; zero-length records use GC fraction zero. Output CG% is the upstream fraction, not a percentage. |
galaxy_find_subsequences@1.0 |
Find IUPAC nucleotide motifs in FASTA and emit zero-based half-open BED6 locations. | Uses reverse complement for reverse-strand search, correcting the upstream complement-only bug. Overlapping hits are consistently included. Maximum one million candidate comparisons per record times pattern length is bounded to 64 million. |
galaxy_fasta_stats@1.0 |
Compute scaffold, contig and base-composition summary statistics from FASTA. | Statistics TSV output only, no separate gap BED. N-runs split contigs; internal N-runs count as gaps. Uses standard N50 threshold >= half, correcting upstream strict-greater edge. At most 100000 sequences/contigs and one 8 MiB-character record. |
galaxy_motif_flank_counts@1.0 |
Count motif-bearing windows upstream and downstream of each indel. | Input has an odd number of tab-separated windows with literal indel in the middle. Motifs are literal case-insensitive DNA strings configured by name=sequence or sequence. Output is row-major then motif order to stream without replaying input; upstream uses motif-major order. Regex motifs are not supported. |
Develop, verify and run
cd examples/operators/galaxy
python3 extend-catalog.py
python3 make-catalog.py
sbt package 'Test/runMain xstream.galaxy.ExtendedSafetyCheck'
python3 verify-extended.py --only reverse_complement --report verification-reverse-complement.json
cd ../../..
xstream-engine/target/universal/stage/bin/xstream-runner examples/operators/galaxy/engine.yaml examples/operators/galaxy/jobs-extended/reverse_complement.yaml
Use Python with PyArrow and PyYAML installed and Docker access to the configured shared JVM image. Every jobs-extended/<name>.yaml has a corresponding data/extended/<name>.txt. The full example includes input loading; the repository detail page presents only the current operator step. Parameters, defaults, complete examples and exact regression expectations are in extra-specs.json. extend-catalog.py regenerates contracts, fixtures and example Jobs. Zero-copy or fully streaming behavior is not claimed for global sorting/reversal or per-record sequence processing.
Provenance and review
Original reviewed sources, XML definitions, fixtures and license notices are retained under upstream-extended/{devteam,community,iuc}/. upstream-extended-manifest.json records repository URLs, pinned revisions and SHA-256 hashes. Adaptations deliberately expose a documented subset where upstream features require executable expressions, external programs, replay or additional files. Unsupported options are rejected; these packages are not drop-in replacements for every upstream command line. Source-only repository publication builds a thin Scala JAR against jvm-arrow@2.1, tests that artifact using chunked Arrow IPC plus inference/preview, then approves it. The administrator trust score is an assessment, not a safety guarantee.
Compose and audit published tools
The sequence pipeline decodes SOLiD colors, computes a reverse complement, measures the sequence and appends an explicit input label. Its output is seq1\t4\trun1\n.
flowchart LR
I[Item bytes: A013] --> D[Color-space decode: AACG]
D --> R[Reverse complement: CGTT]
R --> L[FASTA length: 4]
L --> N[Append label: run1]
N --> O[TSV bytes stream]
Run from the XStream repository root after publication:
xstream-engine/target/universal/stage/bin/xstream-runner \
examples/operators/galaxy/engine-extended.yaml \
examples/operators/galaxy/job-extended-pipeline.yaml
python3 examples/operators/galaxy/verify-extended-jobs.py
python3 examples/operators/galaxy/audit-publication.py --url http://127.0.0.1:8820
Public catalog audit checks 200 distinct names, exact versions, complete port/parameter documentation, usage examples, anonymously downloadable source/artifact hashes, entrypoint classes and embedded license notices. The recorded audit also matches every artifact to its passed repository-build test report. Extended Job checks compare each of the 180 extension Jobs with independently specified fixture output. Pipeline verification covers composition with the original FASTA length adapter. The safety report records four passing safety suites; ExtendedSafetyCheck exercises 2039 cases, including cancellation, I/O failures, invalid UTF-8, line limits, upstream golden fixtures and malformed biological input. Administrator trust scores do not replace these supported-scope and resource limits.
Coordinate and publication-format adapters
These adapters preserve text data and use Arrow bytes transport. GFF3 hierarchy is explicitly rejected by the flat Circos converters. They do not render plots or require Python/Biopython at execution.
| Operator | Purpose and scope |
|---|---|
galaxy_circos_karyotype_fasta@1.0 |
Generate Circos chromosome definitions from FASTA sequence lengths. FASTA text; sequence IDs must be unique. Colors are indexed from zero. No image rendering. |
galaxy_circos_karyotype_lengths@1.0 |
Generate Circos chromosome definitions from a sequence-length table. Unquoted tab-separated ID and positive length; unlike upstream, malformed rows fail explicitly. |
galaxy_circos_tiles_bed@1.0 |
Convert BED intervals to Circos tile records. BED3 or BED6+; zero-based half-open coordinates. Extra BED fields are ignored. |
galaxy_circos_text_bed@1.0 |
Convert labeled BED intervals to Circos text records. BED6+ required; zero-based half-open coordinates. Extra BED fields are ignored. |
galaxy_circos_tiles_gff3@1.0 |
Convert flat GFF3 features to sorted Circos tile records. Flat GFF3 only: Parent hierarchies are rejected rather than flattened. Input chromosome order is preserved; features sort by start. One-based inclusive GFF becomes zero-based half-open. Up to 100000 rows and 8 MiB characters. |
galaxy_circos_text_gff3@1.0 |
Convert named flat GFF3 features to sorted Circos text records. Flat GFF3 only; Parent hierarchies rejected. Features without the selected attribute are skipped. Coordinates converted to zero-based half-open; at most 100000 rows and 8 MiB characters. |
galaxy_remove_terminal_stop_codons@1.0 |
Remove trailing standard-code stop codons from nucleotide FASTA. NCBI translation table 1 only; uppercase normalization and U-to-T conversion. All terminal stops removed; partial terminal codons retained. At most 8 MiB characters per sequence. |
galaxy_windowmasker_to_bed@1.0 |
Convert WindowMasker inclusive intervals to BED4. Text interval mode only. Zero-based inclusive input becomes half-open BED; global wm_N labels begin at zero. |
galaxy_count_gff_features@1.0 |
Count valid feature records in a GFF stream. Counts syntactically valid nine-column GFF records, not logical exon groups. Malformed rows fail rather than silently skipping. FASTA sections are rejected. |
galaxy_resize_coordinate_window@1.0 |
Resize GFF intervals around their midpoint or existing boundaries. Nine-column GFF; uses a common chromosome-length parameter rather than a chromosome-length file. Midpoint uses floor division; comments skipped, malformed rows rejected. |
Interval, FASTQ, sequence and table analysis
All interfaces remain source → result Arrow bytes streams. FASTQ is four-line nucleotide text with matching sequence/quality lengths and optional plus titles; color-space and operator-internal compression are not supported. Interval tools accept positive-length zero-based half-open BED; malformed rows fail. Text readers reject malformed UTF-8, enforce a 1 MiB-character line limit, and poll cancellation. Sequence algorithms have a 64-million-visit comparison budget per sequence where applicable.
| Operator | Purpose | Supported scope |
|---|---|---|
galaxy_bed_sort@1.0 |
Sort BED records lexically by chromosome and numerically by start. | Stable start sort, no genome-order file. Buffer at most 100000 rows or 8 MiB characters. BED input is zero-based half-open with positive interval length. Comments are skipped; invalid records fail. |
galaxy_bed_merge@1.0 |
Merge sorted overlapping or nearby BED intervals. | Sorted input required, output BED3. Distance zero merges book-ended intervals; no strand or extra-column aggregations. BED input is zero-based half-open with positive interval length. Comments are skipped; invalid records fail. |
galaxy_bed_cluster@1.0 |
Assign connected-component IDs to sorted BED intervals. | Sorted input required, global IDs start at one; unstranded mode, original columns retained. BED input is zero-based half-open with positive interval length. Comments are skipped; invalid records fail. |
galaxy_bed_spacing@1.0 |
Append the distance from each BED interval to its predecessor. | Sorted input required. First interval per chromosome uses dot; any overlap uses -1; touching uses zero. BED input is zero-based half-open with positive interval length. Comments are skipped; invalid records fail. |
galaxy_bed_slop@1.0 |
Extend BED intervals with chromosome-boundary clipping. | Integer base distances and optional strand awareness. Chromosome lengths are explicit parameters, not a local file. BED input is zero-based half-open with positive interval length. Comments are skipped; invalid records fail. |
galaxy_bed_flank@1.0 |
Emit left and right flanking intervals around each BED feature. | Left flank then right flank; zero-length flanks omitted. Integer distances, optional strand awareness, explicit chromosome lengths. BED input is zero-based half-open with positive interval length. Comments are skipped; invalid records fail. |
galaxy_bed_complement@1.0 |
Emit genome intervals not covered by BED features. | Sorts and unions input, reports all configured chromosomes in parameter order. Buffer at most 100000 rows or 8 MiB characters. BED input is zero-based half-open with positive interval length. Comments are skipped; invalid records fail. |
galaxy_bed_genome_coverage@1.0 |
Compute run-length encoded BED coverage as bedGraph. | BED interval coverage, no BAM/CIGAR or split blocks. Emits bedGraph rather than histograms. At most 100000 rows or 8 MiB characters, plus two events per interval. BED input is zero-based half-open with positive interval length. Comments are skipped; invalid records fail. |
galaxy_interval_overlap@1.0 |
Append signed overlap or distance between two intervals on each row. | Coordinates are taken from configured columns on the same row; chromosome identity is the caller responsibility, matching upstream. Positive overlap, zero touching, negative separation. Comments are skipped; invalid records fail. |
galaxy_expand_columns@1.0 |
Expand synchronized comma-separated lists across selected table columns. | UTF-8 TSV, no BED coordinate interpretation. Columns must be unique and increasing; no Cartesian product. At most 10000 expansions and 16 MiB estimated output per row. Comments are skipped; invalid records fail. |
galaxy_group_table@1.0 |
Aggregate consecutive tabular groups using a numeric or textual operation. | One grouping key and one value column. Input must group equal keys consecutively; no automatic global sorting. Text aggregation bounded to 8 MiB characters and 100000 values per group. Comments are skipped; invalid records fail. |
galaxy_fastq_to_tabular@1.0 |
Convert FASTQ titles, sequences and quality strings to TSV. | Title tabs become spaces; short descriptions are padded to exactly the requested width. Four-line nucleotide FASTQ only, at most 1 MiB characters per line; sequence and quality lengths and optional + titles must match. No color-space or compression inside the operator. |
galaxy_tabular_to_fastq@1.0 |
Build validated FASTQ records from selected TSV columns. | Malformed rows fail instead of being silently skipped. Output Phred+33. Four-line nucleotide FASTQ only, at most 1 MiB characters per line; sequence and quality lengths and optional + titles must match. No color-space or compression inside the operator. |
galaxy_fastq_mask_quality@1.0 |
Mask FASTQ bases selected by a quality-score comparison. | ASCII Phred+33. Quality strings are retained unchanged. Four-line nucleotide FASTQ only, at most 1 MiB characters per line; sequence and quality lengths and optional + titles must match. No color-space or compression inside the operator. |
galaxy_fastq_filter_quality@1.0 |
Filter FASTQ reads by length and count of out-of-range qualities. | ASCII Phred+33, whole-read ranges only; no generated Python predicates or per-window expressions. Four-line nucleotide FASTQ only, at most 1 MiB characters per line; sequence and quality lengths and optional + titles must match. No color-space or compression inside the operator. |
galaxy_fastq_join_mates@1.0 |
Concatenate adjacent paired FASTQ reads into one read. | Adjacent /1 and /2 mates with equal base IDs, either order; no identifier-indexed second input. Unmatched or misordered pairs fail. Does not reverse-complement mate 2. Four-line nucleotide FASTQ only, at most 1 MiB characters per line; sequence and quality lengths and optional + titles must match. No color-space or compression inside the operator. |
galaxy_fastq_deinterlace@1.0 |
Select paired or unpaired mate records from interleaved FASTQ. | Slash /1 and /2 IDs only. Matches mates regardless of adjacency using a bounded 8 MiB-character / 100000-read buffer. Duplicate full IDs fail; original first-pair encounter order is preserved. Four-line nucleotide FASTQ only, at most 1 MiB characters per line; sequence and quality lengths and optional + titles must match. No color-space or compression inside the operator. |
galaxy_fastq_groom@1.0 |
Convert FASTQ quality encoding and normalize the plus line. | Sanger Phred+33, Illumina Phred+64, Solexa log-odds+64; scores are rounded and clipped to the output convention. Existing titles must agree; no automatic repair. Four-line nucleotide FASTQ only, at most 1 MiB characters per line; sequence and quality lengths and optional + titles must match. No color-space or compression inside the operator. |
galaxy_fastq_quality_stats@1.0 |
Summarize per-position FASTQ quality quartiles and nucleotide counts. | Normalized TSV columns documented by the header, not the upstream report layout. Linear-interpolated quartiles, ASCII Phred+33, at most 10000 positions; fixed 94-bin quality histograms avoid retaining reads. Four-line nucleotide FASTQ only, at most 1 MiB characters per line; sequence and quality lengths and optional + titles must match. No color-space or compression inside the operator. |
galaxy_seqtk_comp@1.0 |
Compute nucleotide, ambiguity, CpG and substitution-class composition. | Full sequences only; no BED subset. CpG counts both contributing positions, including C/Y and G/R ambiguity contexts. FASTA text only unless explicitly stated; one sequence is limited to 8 MiB characters. Output is LF-normalized and unwrapped. |
galaxy_seqtk_listhet@1.0 |
List one-based positions of two-allele IUPAC nucleotide codes. | Only two-base ambiguity symbols qualify; original character case is retained. FASTA text only unless explicitly stated; one sequence is limited to 8 MiB characters. Output is LF-normalized and unwrapped. |
galaxy_seqtk_cutN@1.0 |
Split FASTA sequences at scored N-rich tracts. | Scored forward/backward N-tract extension; no gap-only channel. Output names use one-based inclusive segment endpoints. Leading gaps are removed consistently. FASTA text only unless explicitly stated; one sequence is limited to 8 MiB characters. Output is LF-normalized and unwrapped. |
galaxy_seqtk_randbase@1.0 |
Resolve two-allele IUPAC ambiguities using a reproducible random choice. | Only two-allele symbols are resolved; other ambiguity symbols remain. JVM RNG intentionally differs from seqtk drand48; case preserved. FASTA text only unless explicitly stated; one sequence is limited to 8 MiB characters. Output is LF-normalized and unwrapped. |
galaxy_seqtk_sample@1.0 |
Reservoir-sample a bounded number of FASTA records. | Single-pass count-based sampling; no fraction or two-pass mode. JVM RNG differs from seqtk. Reservoir output order; total retained characters limited to 8 MiB. FASTA text only unless explicitly stated; one sequence is limited to 8 MiB characters. Output is LF-normalized and unwrapped. |
galaxy_seqtk_subseq@1.0 |
Extract configured half-open sequence regions from FASTA. | Coordinates are zero-based half-open, names contain one-based inclusive endpoints. Out-of-bounds matching regions fail; absent sequence IDs produce no output. Regions are emitted in input sequence order then configuration order. FASTA text only unless explicitly stated; one sequence is limited to 8 MiB characters. Output is LF-normalized and unwrapped. |
galaxy_seqtk_seq@1.0 |
Normalize FASTA case, wrapping and sequence-selection criteria. | Subset of seqtk seq: case normalization, comments, minimum length, ambiguous-sequence removal and wrapping. No region masks, quality transformation or random subsampling. FASTA text only unless explicitly stated; one sequence is limited to 8 MiB characters. Output is LF-normalized and unwrapped. |
galaxy_seqtk_hety@1.0 |
Estimate heterozygosity in sliding nucleotide windows. | seqtk sliding-window labeling: later rows start at the previous emission boundary, even for overlapping windows. Two-base IUPAC codes are heterozygous; three/four-base codes uncallable. No division by zero; at most 64 million window-base visits. FASTA text only unless explicitly stated; one sequence is limited to 8 MiB characters. Output is LF-normalized and unwrapped. |
galaxy_seqtk_telo@1.0 |
Identify terminal telomere-repeat tracts from FASTA. | Terminal motif rotations with seqtk 5-prime/3-prime scoring and reverse-complement search. Motif length 1..31; BED4 contains sequence length, no profile channel. FASTA text only unless explicitly stated; one sequence is limited to 8 MiB characters. Output is LF-normalized and unwrapped. |
galaxy_seqtk_trimfq@1.0 |
Trim FASTQ by maximum quality-score segment or fixed end distances. | Four-line nucleotide FASTQ with Phred+33. Maximum scoring segment and best-quality fixed-window fallback; fully trimmed records are omitted rather than empty FASTQ. FASTA text only unless explicitly stated; one sequence is limited to 8 MiB characters. Output is LF-normalized and unwrapped. |
galaxy_seqtk_dropse@1.0 |
Remove unmatched reads from an adjacent-pair FASTQ stream. | Four-line FASTQ, Phred+33. Adjacent equal IDs or equal IDs after removing /digit form pairs, matching seqtk. A displaced or unmatched record is discarded; no global mate search. FASTA text only unless explicitly stated; one sequence is limited to 8 MiB characters. Output is LF-normalized and unwrapped. |
galaxy_seqtk_fqchk@1.0 |
Report per-position FASTQ composition, average quality and error quality. | Four-line FASTQ, Phred+33, at most 10000 positions. seqtk probability floor for Q0..Q3; exact -10log10(mean error), without the upstream small smoothing term; normalized table omits the summary preamble and q=0 histogram mode. FASTA text only unless explicitly stated; one sequence is limited to 8 MiB characters. Output is LF-normalized and unwrapped. |
galaxy_transpose_table@1.0 |
Transpose a rectangular TSV matrix. | Strict rectangular, unquoted UTF-8 TSV; empty cells preserved. At most 8 MiB characters and 100000 rows; no split files. |
galaxy_reverse_columns@1.0 |
Reverse column order within each TSV row. | Unquoted UTF-8 TSV; rows may have different widths. Preserves empty fields, without reversing row order. |
galaxy_numeric_summary@1.0 |
Summarize a numeric TSV column with descriptive statistics. | One global column, no grouping. Reports count/sum/min/max/mean/median/population and sample SD in a normalized table, not the datamash CLI layout. Finite magnitudes <=1e100, at most 100000 values. |
galaxy_annovar_replace_na@1.0 |
Replace NA values in columns classified as numeric. | Matches upstream header-inclusive numeric/string voting, with known numeric headers 1000g2012apr_all and esp6500si_all. Numeric ties choose numeric. Rectangular TSV buffered up to 8 MiB characters and 100000 rows. |
galaxy_gff_to_bedgraph@1.0 |
Convert GFF feature intervals to zero-valued bedGraph rows. | Nine-column GFF; one-based inclusive to zero-based half-open. Corrects upstream end-minus-one error; preserves the inclusive end as the BED exclusive endpoint. Malformed records fail. |
galaxy_circos_gc_skew@1.0 |
Compute windowed FASTA GC skew as bedGraph. | FASTA nucleotide input, bedGraph text rather than upstream BigWig. (G-C)/(G+C), zero when denominator is zero. Final interval clipped to sequence length; one sequence up to 8 MiB characters. |
galaxy_correlation_matrix@1.0 |
Compute Pearson or Spearman pairwise column correlations. | Strict finite numeric rows, no NA imputation or Kendall method. Rectangular output, six decimals. Constant columns reject undefined correlation. At most 100000 rows, 16 columns and 8 MiB-character buffer. |
galaxy_circos_cytobands@1.0 |
Convert RGB BED9 bands into Circos band definitions. | BED9+ with validated 0..255 RGB triplets. Uses inline RGB values so the bytes result is self-contained, unlike upstream separate stderr color definitions. Malformed rows fail. |
galaxy_category_totals@1.0 |
Sum positive criterion values by configured element categories. | First TSV row contains criterion names only; following rows contain an element ID then numeric values. Only positive values contribute. Unknown members fail; each member belongs to exactly one category; at most 128 criteria and 128 configured categories. |
Verification and provenance
extended-regressions.json includes upstream golden cases and malformed-input cases. upstream-golden-cases.json records outputs produced by the pinned seqtk and BEDTools reference programs; upstream-dependencies-manifest.json records the actual seqtk and Galaxy sequence utility source hashes and licenses in addition to Galaxy wrappers. Every published thin JAR is built by the repository and tested over real chunked Arrow IPC before approval. The publication audit matches public downloads to their artifact test hashes. The complete engine-job report covers the 180 extension examples.
flowchart LR
A["Item bytes"] --> B["Arrow IPC"]
B --> C["Intervals / FASTQ / FASTA / TSV"]
C --> D["Validated Scala operator"]
D --> E["Arrow bytes result"]
Composed pipeline verification also checks the five-step FASTQ quality pipeline and the four-step interval coverage pipeline against their expected output. All example YAML uses explicit inputs and contains no aliases.
To replay the original reference outputs, build seqtk and BEDTools at the revisions in upstream-golden-cases.json, then run:
cd examples/operators/galaxy
python3 verify-upstream-reference.py --seqtk /path/to/seqtk --bedtools /path/to/bedtools
VCF variant stream adapters
These adapters consume uncompressed UTF-8 VCF bytes streams and produce VCF or TSV bytes streams. They independently implement documented BCFtools subsets, without BCF, indexing or arbitrary filter expressions. Records allow at most 1024 samples and ploidy eight; phase is preserved and malformed input fails. Pinned source evidence, the MIT license option and hashes are retained under upstream-dependencies/bcftools/.
| Operator | Function |
|---|---|
galaxy_vcf_list_samples |
List VCF sample names in header order. |
galaxy_vcf_genotype_dosage |
Compute alternate-allele dosage from GT calls. |
galaxy_vcf_missing_to_ref |
Replace missing genotype alleles with reference alleles. |
galaxy_vcf_fill_allele_counts |
Recalculate INFO AN and per-ALT AC from GT calls. |
galaxy_vcf_variant_counts |
Count VCF samples, sites and variant classes. |
galaxy_vcf_select_samples |
Select VCF sample columns by explicit names. |
galaxy_vcf_rename_samples |
Rename VCF samples using explicit name pairs. |
galaxy_vcf_remove_annotations |
Remove selected INFO annotations and declarations. |
galaxy_vcf_filter_quality |
Apply a soft minimum-QUAL filter to VCF records. |
galaxy_vcf_set_missing_genotypes |
Set called genotypes to missing while preserving ploidy. |
galaxy_vcf_fix_ploidy |
Convert GT calls to a requested constant ploidy. |
galaxy_vcf_genotype_table |
Extract sample genotypes into long-form TSV. |
Dosage is the total count of all nonreference alleles, with . for any missing allele, rather than upstream per-ALT output. Sample selection and GT changes do not recalculate INFO counts; chain galaxy_vcf_fill_allele_counts when needed. Ploidy conversion changes GT only and does not remap PL/GL. Runnable examples are examples/operators/galaxy/jobs-extended/vcf_*.yaml; see extra-specs.json for parameters and exact scope.
SAM alignment stream adapters
Inputs are SAM text bytes streams, without BAM files or indexes. Mapped records must match the @SQ reference dictionary; coordinates, CIGAR, sequence lengths and qualities are validated. Sorting, grouping and coverage use bounded buffers; depth output is capped at one million positions.
| Operator | Function |
|---|---|
galaxy_sam_flagstat |
Count alignment flag categories with QC-pass/fail totals. |
galaxy_sam_reference_counts |
Count mapped and unmapped segments by reference. |
galaxy_sam_depth |
Compute per-position depth from aligned CIGAR blocks. |
galaxy_sam_coverage |
Summarize reference breadth and mean depth. |
galaxy_sam_alignment_stats |
Summarize primary alignment lengths and mapping quality. |
galaxy_sam_sort |
Sort SAM records by coordinates or query name. |
galaxy_sam_collate |
Group SAM records sharing the same query name. |
galaxy_sam_to_fastq |
Export primary SAM sequences to FASTQ in original orientation. |
galaxy_sam_reheader |
Replace SAM metadata while preserving the reference dictionary. |
galaxy_sam_calmd |
Recalculate MD and NM tags against an explicit reference. |
Coverage counts M/=/X, ignores D/N, filters FLAG 0x704, and counts overlapping mates independently. Flag statistics use normalized TSV; query-name sorting is lexical and collation preserves first-seen name order. sam_calmd accepts an explicit small reference parameter, without files, N skips or BAQ. sam_reheader requires an identical reference dictionary. Exact parameters, formats, resource limits and examples are in extra-specs.json and jobs-extended/sam_*.yaml. Native samtools/BCFtools oracle inputs, commands and outputs are preserved in upstream-variant-alignment-cases.json; native binaries are test tools, not runtime dependencies.
Two-input interval analysis
These operators have source query intervals and reference features as separate Arrow bytes input ports. Outputs are BED or TSV bytes streams. Coordinates are positive-length, zero-based half-open intervals. Each execution input is capped at 8 MiB bytes; parsed buffers allow at most 8 MiB characters and 100000 rows. Query × reference comparisons are capped at 16000000. The two-stream interface uses explicitly bounded buffers rather than constant memory.
| Operator | Function |
|---|---|
galaxy_bed_intersect |
Emit pairwise clipped overlaps between two BED streams. |
galaxy_bed_subtract |
Subtract a union of reference intervals from each query interval. |
galaxy_bed_window |
Report query/reference pairs within a genomic window. |
galaxy_bed_closest |
Report all nearest same-chromosome reference features per query. |
galaxy_bed_coverage |
Report overlap count and covered query bases. |
galaxy_bed_map |
Aggregate a numeric reference column over query overlaps. |
galaxy_bed_annotate |
Annotate queries with reference overlap count and coverage fraction. |
galaxy_bed_jaccard |
Compute base-pair Jaccard similarity of two interval sets. |
Closest preserves all minimum-distance ties; book-ended intervals have distance one. Jaccard unions intervals within each set first and defines empty/empty as zero. bed_map supports sum, mean, min and max, with . for missing overlaps. These tools do not expand BED12 blocks, consume BAM or filter strands. Runnable two-input Jobs are in jobs-extended/bed_*.yaml; native oracle evidence is upstream-paired-interval-cases.json.
flowchart LR
A[Source BED bytes] --> C[Paired interval operator]
B[Reference BED bytes] --> C
C --> D[BED or TSV bytes]
Two-input sequence processing
| Operator | source / reference | Behavior |
|---|---|---|
galaxy_seqtk_mergefa |
FASTA / FASTA | IUPAC unions of equal-length sequences with matching IDs and order; uppercase only when both inputs are uppercase; emits IDs only. |
galaxy_seqtk_mergepe |
FASTQ / FASTQ | Interleaves corresponding mates; checks names after /1 or /2 removal and equal record counts; normalizes plus lines to +. |
galaxy_seqtk_mutfa |
FASTA / mutation TSV | Columns are ID, one-based position, ignored field and replacement IUPAC base; rejects duplicate mutations, unknown IDs and out-of-range coordinates. |
Each execution input is capped at 8 MiB bytes, and parsing allows at most 8 MiB characters and 100000 records. These are explicit seqtk subsets: mergefa accepts FASTA union mode only, mergepe four-line FASTQ only, and mutfa does not accept FASTQ. Replay saved two-input native oracle cases with verify-paired-reference.py --bedtools /path/to/bedtools --seqtk /path/to/seqtk.
Genomic intervals and sequence views
| Operator | Inputs and behavior |
|---|---|
galaxy_bed12_to_bed6 |
source BED12; expands discrete BED6 blocks, validating count, order, lengths and containment. |
galaxy_bed_extract_fasta |
source BED, reference FASTA; extracts subsequences, optionally reverse-complementing BED6 minus strands. |
galaxy_bed_mask_fasta |
source FASTA, reference BED; masks covered bases with N or lowercase, retaining complete titles. |
galaxy_bed_nucleotide_content |
source BED, reference FASTA; reports AT/GC fractions, ACGTN/other counts and length. |
galaxy_bed_union_bedgraph |
Two BED4 inputs; shared segmentation with original values, default missing filler 0; rejects within-input overlaps. |
galaxy_bed_multi_intersect |
Two BED3+ inputs; reports membership segments, set counts and presence flags; omits gaps absent from both sets. |
Coordinates are zero-based half-open. Each execution input is capped at 8 MiB bytes; parsing allows at most 8 MiB characters and 100000 lines or records. FASTA IDs must be unique and intervals must lie within reference lengths. Composition fractions use the full subsequence length as denominator, count ambiguous IUPAC bases as other, and use headerless six-decimal output. Exact scopes and runnable examples are in each contract and jobs-extended/; native BEDTools oracle evidence is upstream-genome-cases.json.
EMBOSS sequence processing
These independent Scala implementations follow pinned Galaxy emboss_5 wrappers. Inputs are FASTA bytes streams; outputs are normalized FASTA or TSV bytes streams, without invoking native EMBOSS. They implement explicit subsets; other upstream formats and CLI options are outside these contracts. Each sequence is capped at 8 MiB characters, with at most 100000 records. Region parameters use one-based inclusive coordinates and reject out-of-range values.
| Operator | Function |
|---|---|
galaxy_emboss_biosed |
Replace literal sequence motifs in FASTA. |
galaxy_emboss_checktrans |
Report stop-codon markers in translated protein FASTA. |
galaxy_emboss_cutseq |
Remove a specified inclusive sequence region. |
galaxy_emboss_degapseq |
Remove alignment gap characters from FASTA. |
galaxy_emboss_descseq |
Replace or append FASTA description text. |
galaxy_emboss_extractseq |
Concatenate configured inclusive sequence regions. |
galaxy_emboss_getorf |
Translate stop-delimited ORFs in forward reading frames. |
galaxy_emboss_maskseq |
Mask configured FASTA coordinate ranges. |
galaxy_emboss_notseq |
Exclude FASTA records by exact identifiers. |
galaxy_emboss_nthseq |
Select one FASTA sequence by ordinal. |
galaxy_emboss_shuffleseq |
Shuffle residues reproducibly within each FASTA record. |
galaxy_emboss_skipseq |
Skip initial FASTA sequence records. |
galaxy_emboss_splitter |
Split sequences into overlapping FASTA windows. |
galaxy_emboss_textsearch |
Find literal text in FASTA identifiers and descriptions. |
galaxy_emboss_trimest |
Trim exact terminal poly-A tails from nucleotide FASTA. |
galaxy_emboss_trimseq |
Trim ambiguous N residues at nucleotide sequence ends. |
galaxy_emboss_transeq |
Translate six standard-code reading frames. |
galaxy_emboss_fuzzpro |
Find protein motifs with bounded substitution mismatches. |
galaxy_emboss_compseq |
Count observed k-mer composition in each DNA sequence. |
galaxy_emboss_freak |
Compute windowed frequency of selected nucleotide residues. |
Important scopes: getorf is forward-only, standard-code, stop-to-stop and emits frame order, omitting partial terminal codons. transeq defines reverse frames as offsets 0/1/2 of the reverse complement, differing from native EMBOSS reverse-frame numbering, and omits partial terminal codons. fuzzpro supports letters, X wildcards and substitution mismatches, without PROSITE expressions or indels. trimest handles exact 3′ poly-A only; trimseq handles terminal N only. compseq emits observed ACGT words only, excluding ambiguous windows from the denominator. shuffleseq uses an explicit JVM seed and differs from native EMBOSS RNG; splitter avoids repeating a tail already covered by the final full window. Parameters, budgets and runnable examples are in extra-specs.json and jobs-extended/emboss_*.yaml.
Native EMBOSS 6.6.0 oracle inputs, commands and outputs are preserved in upstream-emboss-cases.json, with explicit formatting, ORF-order, reverse-frame and partial-codon normalization. Native binaries are local test references and are not distributed with these operators.
Protein properties and processing
Inputs are FASTA bytes streams containing nonempty sequences of the standard 20 amino acids; ambiguity codes, stop markers and gaps are rejected. Each sequence is capped at 8 MiB characters and there are at most 100000 records. Properties use normalized TSV; digestion and back-translation produce FASTA.
| Operator | Function |
|---|---|
galaxy_emboss_pepstats |
Summarize protein mass, charge, pI and composition properties. |
galaxy_emboss_iep |
Calculate protein isoelectric point using Henderson-Hasselbalch charge. |
galaxy_emboss_charge |
Calculate protein charge at a configured pH. |
galaxy_emboss_hmoment |
Calculate sliding Eisenberg hydrophobic moments. |
galaxy_emboss_pepwindow |
Calculate sliding Kyte-Doolittle protein hydropathy. |
galaxy_emboss_pepinfo |
Report per-residue mass and hydrophobicity properties. |
galaxy_emboss_antigenic |
Compute Kolaskar-Tongaonkar seven-residue antigenic propensity. |
galaxy_emboss_digest |
Generate complete trypsin digestion peptides. |
galaxy_emboss_backtranseq |
Construct a deterministic standard-code DNA encoding for a protein. |
galaxy_emboss_oddcomp |
Find protein windows meeting specified composition thresholds. |
Charge and pI use Henderson–Hasselbalch with both termini: pKa N-terminus7.5, C-terminus3.6, C8.5, D3.9, E4.1, H6.5, K10.8, R12.5 and Y10.1. pI uses bisection over pH0–14. ph_x100: 700 means pH7. Mass sums average residue masses plus18.01528 Da water; reduced extinction is5500×W+1490×Y. Hydrophobicity uses Kyte–Doolittle and Eisenberg consensus tables.
antigenic reports Kolaskar–Tongaonkar seven-residue propensity only, without biologically validated epitope calls. digest uses a zero-missed-cleavage K/R rule unless the next residue is P. backtranseq chooses the lexically first standard codon, without host preference optimization. oddcomp accepts residue count thresholds, without upstream pair-composition files. See contracts for exact limits and examples. Five native EMBOSS pI cases are preserved in upstream-protein-cases.json and validated with tolerance matching upstream four-decimal precision.
Pairwise DNA alignment
| Operator | Function |
|---|---|
galaxy_emboss_needle |
Compute global affine-gap pairwise DNA alignment. |
galaxy_emboss_water |
Compute local affine-gap pairwise DNA alignment. |
galaxy_emboss_diffseq |
Report substitution and indel blocks from global DNA alignment. |
galaxy_emboss_dottup |
Find all exact word matches between two DNA sequences. |
galaxy_emboss_dotmatcher |
Find similar sequence windows using a bounded dot-plot score. |
galaxy_emboss_wordmatch |
Find maximal exact common DNA substrings. |
galaxy_emboss_merger |
Assemble two DNA sequences at their longest exact suffix-prefix overlap. |
Exactly one nonempty ACGT FASTA per input. Global/local alignment uses Gotoh, gap(k)=open+(k−1)×extend, with penalized global terminal gaps and at most one million DP cells. Alignment outputs FASTA with score and one-based coordinates; differences use zero-based event offsets. Word/dot adapters emit normalized TSV with comparison and100000-hit limits. Merger requires an exact forward overlap, without mismatch consensus. Ten native EMBOSS score cases are preserved in upstream-alignment-cases.json.
DNA motifs and codons
| Operator | Function |
|---|---|
galaxy_emboss_palindrome |
Find exact inverted repeats with bounded intervening loops. |
galaxy_emboss_equicktandem |
Locate maximal exact tandem repeats of primitive DNA units. |
galaxy_emboss_fuzznuc |
Search nucleotide IUPAC motifs allowing bounded substitutions. |
galaxy_emboss_fuzztran |
Search a protein motif across all six translated DNA frames. |
galaxy_emboss_cpgreport |
Summarize CpG abundance and observed-to-expected ratio. |
galaxy_emboss_cpgplot |
Call CpG-rich islands from thresholded sliding windows. |
galaxy_emboss_newcpgseek |
Find CpG-rich segments using a resettable dinucleotide score walk. |
galaxy_emboss_syco |
Measure synonymous codon diversity for each encoded amino acid. |
galaxy_emboss_wobble |
Compute sliding third-codon-base composition. |
galaxy_emboss_cai |
Calculate codon adaptation from explicitly provided relative weights. |
Nonempty ACGT FASTA, up to one million bases per sequence. Searches allow at most64 million comparisons and100000 hits. Palindrome finds exact inverted stems; equicktandem finds exact primitive repeats. Fuzznuc allows substitutions only; fuzztran searches standard-code six frames and omits partial codons. CpG O/E=count×length/(C×G), zero for a zero denominator. Cpgplot merges accepted complete windows; newcpgseek rewards adjacent CG by cpg_score, otherwise subtracts1, resetting nonpositive scores. Syco reports synonymous-codon Shannon entropy; wobble compares selected bases at position3 versus1+2; cai takes the geometric mean of explicit caller weights, excluding stops. These three require lengths divisible by3. Documented subsets exclude full native pattern syntax, smoothing and graphical output.
Codon distribution comparison
| Operator | Function |
|---|---|
galaxy_emboss_codcmp |
Compare two coding sequences by normalized codon distributions. |
Exactly one ACGT coding sequence per input, lengths divisible by3 and at most one million bases each. Compares64-codon frequencies including stops using symmetric Jensen–Shannon divergence in bits; not a complete native codon-usage file comparator.
Statistical analysis
| Operator | Function |
|---|---|
galaxy_linear_regression |
Fit multivariate ordinary least-squares regression with an intercept. |
galaxy_partial_r_square |
Measure each predictor contribution conditional on all other predictors. |
galaxy_pca |
Compute centered covariance principal-component scores. |
galaxy_histogram |
Count numeric observations in fixed-width histogram bins. |
galaxy_scatterplot |
Render paired numeric observations into a standalone SVG scatter plot. |
galaxy_logistic_regression |
Fit binary logistic regression with ridge-stabilized IRLS. |
galaxy_statistical_hypothesis_testing |
Perform a two-sided one-sample Student t test. |
Strict headerless numeric TSV, at most10000 rows,128 columns and8 MiB characters; finite values with magnitude<=1e12, missing values rejected. Regression includes intercept, at most16 predictors, reorthogonalized QR and rank checks; partial R² drops one predictor at a time. PCA uses centered sample covariance and Jacobi eigenvectors, sorted by eigenvalue with largest loading nonnegative. Logistic regression standardizes predictors, applies L2 regularization and damped Newton; nonconvergence after100 steps fails, no VIF. The t test uses sample variance and incomplete-beta tails, requiring positive variance. Histogram final bins include their upper edge; scatterplot emits script-free SVG. Normalized numeric/SVG outputs replace native R diagnostic PDFs and do not expose all upstream options.
Table joins
| Operator | Function |
|---|---|
galaxy_join_files_by_id |
Join two tables by exact ID with bounded output. |
galaxy_join_files_on_column_fuzzy |
Join numeric table keys within an absolute tolerance. |
Two headerless rectangular TSV inputs, each capped at100000 rows,128 columns and8 MiB characters; output capped at100000 rows and16 MiB characters. Exact joins retain both keys, fill missing values with dots and expand duplicate-key matches; full joins append unmatched references. Fuzzy means absolute numeric tolerance, not edit distance, with at most16 million comparisons; nearest ties select the first reference row. No ppm mode.
DNA insertion
| Operator | Function |
|---|---|
galaxy_emboss_pasteseq |
Insert one DNA sequence at a specified boundary of another. |
Exactly one nonempty ACGT FASTA per input, up to one million bases each. Position counts consumed source bases; zero prepends. Out-of-range positions fail. Source title is retained.
Codon-preserving alignment
| Operator | Function |
|---|---|
galaxy_emboss_tranalign |
Project aligned proteins into a codon-preserving nucleotide alignment. |
Inputs must have identical unique ID sets. Source is complete frame1 standard-code DNA; reference is aligned protein with dash gaps. Every nongap residue must match its source codon translation; each gap becomes three dashes. Each input is limited to one million total characters and10000 records; all aligned proteins must have equal width; no ORF search or alternative translation tables.
Protein presentation
| Operator | Function |
|---|---|
galaxy_emboss_octanol |
Calculate sliding Wimley-White octanol and interface transfer energies. |
galaxy_emboss_pepwheel |
Render a protein sequence as an SVG helical wheel. |
Standard20-residue FASTA. Octanol uses factual EMBOSS Ewhite-wimley table values (charged D/E, neutral H) and emits complete-window mean octanol/interface/difference in kcal/mol, with100000 residues per record and100000 total windows. Pepwheel accepts one1–36-residue protein, producing400×400 SVG with configurable angular advance; no native graphics or embedded input scripts.
Additional reference checks
verification-variant-alignment-reference.json records27 BCFtools/SAMtools native cases; verification-paired-reference.json records39 BEDTools/seqtk paired cases; verification-scientific-reference.json records46 genome, EMBOSS sequence/protein/alignment and nucleotide native cases. Reproduce the latter with python3 verify-scientific-reference.py --bedtools /path/to/bedtools --emboss-root /path/to/emboss-prefix from the Galaxy example directory. Native binaries are test tools and are not packaged or called by operators. upstream-numerical-cases.json contains30 independent NumPy/SciPy comparisons for least squares, partial R², PCA, logistic regression and t tests; generate-numerical-reference.py regenerates these deterministic cases using NumPy and SciPy. ExtendedSafetyCheck validates all expected normalized results. The7 composed Jobs in verification-pipelines.json cover cross-package stream chains, including paired inputs, genotype normalization, sequence edits, codon projection and joined regression.