One biosample's full sequencing, harmonized against model/schema/ber_feature_model.yaml
Built 2026-09-18. Replaces an earlier worked example (examples/one-workflow-run/, removed before the layout migration) that
turned out to be a poor choice: it was built on a run selected only because two of its files
happened to already be vendored in corpus/index.yaml, and those had been picked by that file's own
stated method, "the smallest real file of its kind out of roughly 600 sampled." A 2-contig,
813-bp assembly is not representative of a real metagenome, and using it made every design
decision look more trivial than it is. Mark caught this by asking whether it was small because the
underlying biosample was poor quality; it wasn't, the biosample and study are entirely legitimate,
the file was just a deliberately minimal example picked for a different purpose.
Source
Data generation nmdc:omprc-11-2t8ft192, "Montane desert soil microbial communities from Mustang
Region, Nepal - Schmidt56.soil.Nepal.3", PI Janet Jansson (PNNL), GOLD sequencing project
Gp0452617, NCBI BioProject PRJEB42019. Picked this time by filtering the NMDC public API
directly for a moderate file-size band (file_size_bytes between 3,000 and 15,000 for Annotation
Enzyme Commission files) rather than sampling blindly, after an earlier attempt at a 75th-percentile
pick landed on a 27MB file. This run was the only size-appropriate candidate found with an actual,
non-empty CRISPR array, which mattered enough to Mark to be worth the extra searching.
One sequencing, seven workflow executions
Querying the NMDC public API for every DataObject whose URL contains this data generation's id
(not scoped to one workflow) surfaced the complete chain: read QC (wfrqc-11-y4x3c425.1), assembly
(wfmgas-11-19jh9v28.1), annotation (wfmgan-11-5xxrm214.1 and .2, two processing versions),
read-based taxonomic classification (wfrbt-11-k9jkd382.1), and MAG binning
(wfmag-11-n2d0ma07.2 and .3). 80 DataObjects total; 76 fetched (about 4.7MB), 4 skipped by
size and recorded by metadata only: a 1.18GB assembly coverage BAM, a 1.03GB Centrifuge
classification, an 804MB Kraken2 classification, and a 1.13GB filtered-reads FASTQ. None of the
four skipped files are feature tables.
Of the five workflow types, exactly one produces feature-table content: metagenome annotation
(wfmgan). Assembly (wfmgas) is upstream and adjacent, it produces the Contig this schema's
features sit on, plus the AGP format (a tabular, positional description of how contigs join into
scaffolds, worth noting as GFF-adjacent but not covered here). Read QC, read-based taxonomy, and
MAG binning produce read-level, taxon-level, and bin-level data respectively, none of it shaped
like a feature table. This matters for anyone extending fetch tooling in this repo: point it at
wfmgan DataObjects, not at the other four workflow types.
A workflow execution id does not name both a Contig and its Features. seqid values in every
GFF file from the annotation workflow reference the ASSEMBLY workflow's id
(nmdc:wfmgas-11-19jh9v28.1_scf_N_cM), not the annotation workflow's own id
(wfmgan-11-5xxrm214.2). A Contig and the Features on it come from two different workflow
executions. Both classes now carry generated_by; every record in this example populates it.
The former SequenceFeature-only predicted_by name was replaced to cover assembly as well as annotation.
source_files lists the artifact URLs used for the record, including lineage and other sidecars.
These are record-level sources, not a claim that every field came from generated_by.
This run has two annotation versions, .1 and .2, and they are not identical. Their Product
Names outputs differ by a few bytes (104,223 vs 104,195). This example uses only .2, so as not to
mix two provenance records under one label.
What is represented here, and why nothing is duplicated
Three contigs, fifteen features. Not the full ~1,121 genes this run actually has; a representative slice chosen to exercise every part of the schema with real content, which a full harmonization would not make any clearer.
scf_1_c1_104_853: a gene where Prodigal and GeneMark called the identical 104-853 boundary independently. Recorded once, from Prodigal (the roll-up's choice), with GeneMark's agreement noted as an attribute rather than a second, redundant SequenceFeature row, since the coordinates are identical, not a competing claim. Carries five real functional-evidence hits: COG, CATH-FunFam, Pfam, SUPERFAMILY, and a combined KO/EC hit.- The KO/EC hit is one row, not three.
ec.tsv,ko.tsv, andKO_EC Annotation GFFall report the same 252 underlyinglastalalignments for this run;ec.tsv(88 rows) is a filtered subset of the same 252 hitsko.tsvandko_ec.gffboth carry in full.ko_ec.gffis the richest of the three (both labels combined, e.g.KO:K00059__EC:1.1.1.100, plus full evidence), so it's the only one used here. Confirmed by checking three EC/KO row pairs by hand: identical coordinates, e-value, and every numeric field in each pair. scf_2_c1_3_866vs.scf_2_c1_3_740: a genuine Prodigal/GeneMark disagreement. Same start, different end (866 vs. 740), so these are two different boundary claims for one locus, not a duplicate. Both are kept,is_selecteddistinguishes them.scf_1_c1_3_65: the case that complicatedis_selectedfurther. This Prodigal call has no competing GeneMark call at all, and is still absent from Structural Annotation GFF and from Annotation Amino Acid FASTA.is_selected=falsehere cannot mean "lost to a competitor"; it means something filtered this call out regardless. Flagged as an open question in the schema.scf_2_c1_887_1104: an RFAM/INFERNAL ncRNA hit (a Group II catalytic intron), included directly in Structural Annotation GFF the same way Prodigal's CDS calls are, confirming the roll-up duplicates non-CDS structural calls too, not just gene calls. Kept once, sourced fromrfam.gff. Notable fortype: this one uses the generic termmisc_featurerather than a bare accession, the opposite problem from the Pfam/COG/CATH pattern documented elsewhere in this repository, here the specific identity is inattributes(model,accession) and the positional column holds a placeholder.- The CRISPR array,
scf_344_c1_76_183, plus its threerepeat_unitchildren linked byparent. This is the run's populatedCRT Annotation GFFarray. The corpus's newly addedCrispr Terms(nmdc_wfmgan-11-vypv6e87.1_crt.crisprs) example contains three populated, six-field records: each has a sequence identifier, numeric fields, a repeated sequence, a variable sequence (absent on the last row), and a finalcfield. The exact column meanings remain unverified, as recorded incorpus/index.yaml. The older vendoredCRT Annotation GFF(nmdc_wfmgan-11-bvg4py20.1_crt.gff) contains four feature rows. These are different runs and different formats; neither artifact is empty.
None of these three contigs are MAG scaffolds. Confirmed 2026-09-18 by unzipping both MAG
binning outputs for this biosample (nmdc:wfmag-11-n2d0ma07.2 and .3): every one of the four
bin archives (hqmq_bin.zip and lq_bin.zip, both versions) contains only an empty marker file,
no_hqmq_mags.txt or no_lq_mags.txt. This assembly produced zero MAGs, of either quality
category, in either binning run. scf_1_c1, scf_2_c1, and scf_344_c1 are plain metagenome
assembly contigs, never binned into anything. Recorded here because an early description of this
case study (in meeting notes, not in this repository) called them "MAG scaffolds," which this
directly contradicts.
Not represented, and why: Product Names duplicates product/product_source exactly (checked
for both genes here); Gene Phylogeny and Scaffold Lineage are used but folded into the SequenceFeature and
Contig they describe rather than kept as separate rows; the tiny gene's absence from Annotation
Amino Acid FASTA is recorded as an absence of translated_sequence, not invented.
Reproducing
# Every DataObject for the whole data generation, any workflow execution:
filter = {"url": {"$regex": "omprc-11-2t8ft192"}}
# https://api.microbiomedata.org/nmdcschema/data_object_set?filter=<url-encoded filter>&max_page_size=200
Raw files are not committed here (76 files, ~4.7MB, disproportionate for one worked example); they
were downloaded into this worktree's gitignored local/ directory, one subdirectory per workflow
execution id, and are reproducible from the query above plus the DataObject urls cited inline in
harmonized.yaml. The source manifest records their DataObject IDs,
workflow executions, and NMDC-reported MD5 checksums. On 2026-09-21 each cited local source
was checked against that checksum before adding the URLs. CI checks the manifest's coverage;
it does not download or re-hash the upstream files on every run.
The separate multiple-Pfam example uses another gene from the same annotation version to exercise the actual multiple-Pfam query. This example's five evidence databases are not five Pfams, and its CRISPR repeat children are not protein domains.
Status
Draft. Not reviewed by Sierra Moxon, AJ Ireland, or anyone else on the BER data modeling team.