Skip to content

How each vendored file got here

Written 2026-09-11. Every step below was run, not planned.

The 14 NMDC files

  1. Read src/schema/basic_slots.yaml in microbiomedata/nmdc-schema and found 16 distinct data_object_type values whose descriptions say GFF3.
  2. Queried the NMDC public API once per type, same query shape for all 16. Fourteen returned records. Two returned none, Misc Annotation GFF and TMRNA Annotation GFF, so those two absences are real rather than a broken query.
  3. For each of the 14, paged up to 600 records and selected the smallest by file_size_bytes. The selection is recorded per entry in corpus/index.yaml under selection, including how many records were sampled.
  4. Downloaded each file and compared its MD5 against the md5_checksum field in the NMDC record. All 14 matched. That comparison is recorded per entry as md5_matches_source_record: true.

The smallest files are small because those workflow runs produced few features, not because the files are truncated. Each is a complete file as NMDC serves it.

The 2 NCBI files

Fetched from the RefSeq FTP tree by assembly accession. Both returned HTTP 200 and were complete gzipped GFF3, then decompressed for storage here because this repository carries no binary files. The recorded MD5 is therefore of the decompressed text, not of the gzip the URL serves.

A third candidate, tobacco mosaic virus at GCF_000862205.1, returned 404 and was discarded rather than kept as a 990 byte error page.

NCBI publishes no per-file checksum at those URLs, so the MD5 in corpus/index.yaml is of the bytes as fetched on 2026-09-11 and establishes only that the file has not changed since, not that it matches an upstream record.

What was searched and not used

See docs/background.md, which records the four existing models and their staleness, the seven normalization efforts and why none of them is a data model, the eleven named format flavors with their attributions, and the approaches that were considered and rejected. Also the why_not_vendored field on each linked entry.

The 2 NCBI GTF files

Same accessions as the GFF3 files, fetched from the _genomic.gtf.gz sibling on the same FTP path, then decompressed. Both declare #gtf-version 2.2 in their own first line, so the flavor is the file's own claim rather than an assumption.

The 2 vendored specifications

corpus/specifications/chado_1.4_feature_tables.sql. Downloaded the 2.2 MB schemas/1.4/default_schema.sql from branch 1.4 of GMOD/Chado, then extracted the seven create table blocks whose names begin feature. The repository default branch is 1.4, not master; a raw URL built on master returns 404, which is how the first attempt failed.

corpus/specifications/kbase_cdm_bioentity.yaml. Taken whole from src/linkml/cdm_bioentity.yaml on main. The Feature class is in that module, not in cdm_components.yaml; the first attempt fetched the wrong file and a grep for Feature: matched an enum value inside it, which looked like success.

The 9 derived files

Built by scripts/make_malformed.py from the vendored phiX174 GFF3, six in corpus/fixtures/malformed/ and three in corpus/fixtures/edge-cases/. Each output is that source preserved line for line with one change to one data row, plus provenance comments, which are ordinary single-hash lines appended after the source's own ### terminator. They are not pragmas and they are not in the header, and earlier wording in this file called them header pragmas, which was wrong.

no_version_pragma is the one exception to both halves of that: its single change removes a header directive, so its change is not to a data row and its header is deliberately not byte-identical.

The three files in corpus/fixtures/edge-cases/ are valid GFF3 rather than malformed. multiple_parents is permitted by the specification and refused by many tools. cds_phase_biologically_wrong carries a legal phase value that is biologically wrong, so a syntax-only validator passes it while the translated protein changes. unescaped_semicolon_silent demonstrates a legal extra key/value pair created by an unescaped separator, silently changing the intended annotation.

All nine are labeled derived in the index and are not evidence about any real producer. The earlier eight-file account preceded the addition of unescaped_semicolon_silent in commit c41ba55 on 2026-09-11. The current index records all nine; dated correction notes below describe intermediate states in that acquisition history.

EMSL, searched and not found

Searched GitHub repository and code search across ber-data and microbiomedata for BASALT, and the open web for MONet data access. MONet is real, published without embargo on EMSL Science Central, and recorded. BASALT is not public anywhere that could be found on 2026-09-11. Recorded as not located so the gap is visible rather than forgotten.

Corrections after the second review, 2026-09-11

The first version of the derived files was wrong in ways the review caught, and the corrections changed the files themselves rather than only their descriptions.

The generator discarded every source pragma. It rebuilt each fixture from data rows alone and wrote a fresh header, so #!genome-build, ##sequence-region, ##species and the ### terminator were all lost. Each file therefore differed from its source in many header semantics while claiming one documented change. The generator now preserves the source line for line and appends provenance comments, single-hash lines after the terminator, so the header region is byte-identical. These are comments rather than pragmas and they sit outside the header, which earlier wording in this file got wrong. The one exception is no_version_pragma, whose single change is the removal of that one header line.

Two fixtures carried a second, undocumented defect. The semicolon case overwrote a product attribute that already existed, adding a duplicate-attribute defect beside the intended one; it now uses Note. The multiple-parent case named a second parent that was defined nowhere, making it a dangling-parent fixture as well; it now uses gene-phiX174p06, which the source defines.

The circular-genome rationale was backwards. The earlier text said a swapped start and end was ambiguous rather than wrong because phiX174 is circular. GFF3 requires start no greater than end on every feature, circular included, and expresses wraparound by extending end past the landmark length. The case is a plain coordinate-rule violation. Corrected in the generator, the index and the README.

One valid file was filed as invalid. The multiple-parent case is legal GFF3 that many tools refuse, and it sat in derived-malformed/ labeled invalid. It now lives in derived-edge-cases/, and every derived entry carries a validity field so the distinction cannot be lost again.

The index was indexing one artifact twice. The Sequence Ontology GFF3 specification appeared as both so-gff3-spec and spec-gff3-so, which inflated the entry and specification counts. The duplicate is removed, and the verifier now rejects a repeated origin_url, exempting derived entries because they legitimately share the generator that produces them.

Three checks could not fail. The verifier did not require origin_url on a vendored entry though the README calls it part of the contract; it raised KeyError instead of a clean nonzero report when an entry it had just rejected lacked path or tier; and the CI fixture check used git diff, which ignores untracked files, so a fixture deleted from a pull request would have been silently regenerated and the step would have passed. All three are fixed, and CI now injects five separate defects and fails the build if the verifier accepts any of them or rejects one without naming it.

The README counts were wrong. Twelve specifications, not eleven, and ten linked, not nine. The byte total still said 80 KB after four files were added.

Corrections after the balanced review, 2026-09-11

Two fixtures were classified as specification violations without checking the specification, which is the same mistake as the multiple_parents misclassification corrected earlier the same day.

A changed CDS phase is not a violation. GFF3 permits 0, 1 and 2, so altering 0 to 1 produces a biologically wrong annotation that stays valid. That file moved to derived-edge-cases/ as cds_phase_biologically_wrong, and a genuine violation was added as cds_phase_illegal, which sets phase to 3. The pair is now the most useful thing in the derived set: one is caught by a validator and the other is not, and the one that is not is the failure AgBioData leads with.

A duplicated row is not a duplicate ID. GFF3 permits a discontinuous feature to span several lines under a shared ID. The earlier fixture inserted a row with the same seqid, source, type, strand and attributes, which is exactly a legal discontinuous feature. The ID now collides across a gene and a sequence_alteration, which cannot describe one feature, so the violation is real.

Descriptions that contradicted the files. The README stated "one change to one data row" and a byte-identical header as blanket claims, while no_version_pragma is the exception to both. That exception is now stated in the README and in that entry's own notes rather than the entry repeating the generic text.

Wording. The provenance lines were called header pragmas. They are single-hash comments appended after the source's ### terminator, outside the header and not pragmas at all.

September 21, 2026: cross-format conversion evidence

Added two unchanged, redistributable examples from Biopython 1.85, commit 668de08f73fca7f8336049dfd78716d7dd095f21: Tests/Blat/psl_34_004.bed and Tests/GenBank/cor6_6.gb, under corpus/sources/biopython/. The upstream license is retained there. The index records source versions, retrieval date, URLs, MD5/SHA-256 and attribution; its total is now 60 entries, 25 vendored.

The BED12 artifact supplies optional display fields, strand and ordered blocks from BLAT v34 real-sequence alignments. It does not declare the target assembly; the query label hg19_dna is not treated as an assembly declaration. The GenBank artifact supplies six accession-versioned plant records, with joined and partial locations and repeated qualifiers. It is not a five-column submission table.

The conversion report distinguishes supported GFF3/BED12 mappings from unsupported INSDC/GTF or location constructs. Existing source and derived fixture bytes were not changed.

September 21, 2026: protein-relative conversion companions

Retained the full Pfam GFF, structural GFF and protein FASTA for nmdc:wfmgan-11-5xxrm214.2, previously downloaded for the curated model examples. Each file's MD5 was checked against the NMDC DataObject value already recorded in model/examples/source-artifacts.yaml before copying it into the corpus. This reuses that retrieval; it is not a new NMDC-wide sample or API census. The index records the original download URLs, DataObject IDs, byte counts, MD5 and SHA-256, with NMDC CC BY 4.0 attribution. All three files are unchanged.

These companions make exact protein-ID joins, sequence-length bounds and CDS relationships reproducible for 416 Pfam hits on 397 proteins. The generated context retains only participating CDSs; this selection does not alter the three source files or imply that other CDSs were biologically filtered. The corpus now contains 63 entries, 28 vendored.

September 21, 2026: a reproducible BGC query example

Retained the complete NCBI annotwriter response for NC_003888.3, retrieved through the indexed sviewer URL. Despite from/to parameters, that response contains the whole linear chromosome annotation (4,691,954 bytes). The original is unchanged; the date and checksums identify the annotation snapshot separately from sequence accession version. NCBI's data availability policy is linked in the index.

scripts/bgc_example.py selects the actinorhodin genes SCO5071–SCO5092, identified by the original genome publication, and their direct CDS children. The generated 44-row excerpt is indexed as derived and stored under corpus/derived-examples/, with explicit selection provenance. Row bytes, order and absolute coordinates are preserved. The selection does not claim that omitted genes were biologically filtered. Both original and excerpt pass GenomeTools 1.6.6. The corpus now has 65 entries, 29 vendored and 10 derived; nine derived artifacts remain deliberately altered fixtures. See the query exercise.

September 21 retrieval, modeled September 22: circular INSDC evidence

Retained the complete NCBI RefSeq GenBank response for NC_001422.1, phiX174, unchanged from the indexed public endpoint. The index records its retrieval date, 26,238 bytes, MD5/SHA-256, accession version and NCBI reuse policy. This is a 5,386-base circular reference with 32 annotated features, six of which cross the origin using joined locations. It complements the previously retained GFF3 from the same reference; no automatic annotation equivalence between snapshots is assumed.

The bounded INSDC profile tests this record and the six already retained plant records. It reconstructs their modeled locations and qualifiers separately from byte recovery; independent Biopython parsing checks the reconstructed feature meaning. No pre-existing source bytes were changed. The corpus now has 66 entries, 30 vendored and 10 derived.

September 24 and 25, 2026: JGI IMG isolate annotation sets

Every vendored IMG file before this came from an NMDC metagenome workflow. For https://github.com/turbomam/feature-table-corpus/issues/53, two public isolate records from the JGI Data Portal are now retained under corpus/sources/jgi-img/<record>/, with the portal's file names:

  • Clostridium acetobutylicum DJ311 IMG_AP-1268149 (Ga0423362): all 19 Ga0423362_*.gff files, the per-method outputs and structural and functional roll-ups of the newer IMG pipeline. Downloaded 2026-09-25 after a tape restore requested on 2026-09-24.
  • Bacillus sp. V-88 IMG_AP-1121004: 106476.assembled.gff from IMG pipeline 4.14.0 (the record's 106476.pipeline_version.info, read 2026-09-25), 2708743150.gff from img_core_v400, and that taxon's seven .tab.txt side tables. Downloaded 2026-09-24. The 8,587,401-byte 106476.assembled.gbk is not retained: 5,541,445 of its bytes are ORIGIN sequence blocks (measured 2026-09-25), the corpus already holds GenBank feature evidence, and it alone would have grown corpus/ by another 8.6 MB.

Methanococcus maripaludis S1 IMG_AP-1257706 came from the same tape restore and is left out: it is an archaeon, and the issue scopes this addition to the two bacterial records.

Downloading needs a JGI login, so these files were fetched in a logged-in browser as described in docs/restricted-sources.md, placed under local/jgi/, and checked with just jgi-verify. Each file's size and md5 match the value the anonymous search API reports, recorded in model/examples/jgi-inputs.yaml, and were checked again after copying. The index records each file's portal file_id, its download URL, MD5 and SHA-256. No bytes were changed.

Reuse terms, checked 2026-09-25: both records are data_utilization_status: Unrestricted and visibility: public in the search API. Their proposals were accepted on 2015-12-10 (Clostridium, BrcProposal) and 2013-09-20 (Bacillus, Csp2014Proposal), before FY22, so the JGI Legacy Data Policy governs them, not the current JGI Data Policy, which covers projects accepted FY22 and later. The first version of this section cited the current policy; review on https://github.com/turbomam/feature-table-corpus/pull/63 caught it. The legacy policy names no license and asks publications to carry JGI's acknowledgment. Each entry records the license as "JGI Legacy Data Policy (proposal accepted before FY22); data_utilization_status Unrestricted; attribution requested by this repository", with the acknowledgment and the dataset DOI (10.25585/1488021 and 10.25585/1488085).

The first version also used the download folder's short name, BacspinBacteria_29, for the Bacillus record. The portal title, and the name used here, is Bacillus sp. V-88.

The 28 files total 13,392,577 bytes. By git ls-tree -r -l, corpus/ is 5,690,300 bytes before https://github.com/turbomam/feature-table-corpus/pull/63 and 19,135,385 bytes after it; the second figure also counts the growth of index.yaml and this file. The corpus now has 94 entries, 58 vendored. The 21 GFF files have measured validator cases in model/validation/; see the validation report.

September 30, 2026: a Populus trichocarpa v4.1 excerpt

For https://github.com/turbomam/feature-table-corpus/issues/107, the Phytozome gene_exons dialect now has a real input in the repository: corpus/derived-examples/populus-v4.1-chr01-200-genes.gene_exons.gff3, the first 200 Chr01 genes of Populus trichocarpa v4.1 (Phytozome-533), made by scripts/populus_excerpt.py. The full GFF3, downloaded by Mark with a JGI login on 2026-09-29, is 112.8 MB uncompressed, so it is recorded as the restricted entry jgi-phytozome-populus-v4.1-gene-exons with its JGI file ID and md5, and the excerpt as derived from it. The derived tier's definition now allows a restricted source whose md5 is recorded. Terms and citations are in the derived-examples README. The corpus now has 96 entries: 58 vendored, 22 linked, 4 restricted, 1 not located and 11 derived.

October 1, 2026: the Populus annotation_info excerpt

For https://github.com/turbomam/feature-table-corpus/issues/144, the 293 annotation_info rows of the 200 excerpted Populus genes are now in corpus/derived-examples/populus-v4.1-chr01-200-genes.annotation_info.txt, derived from the restricted entry jgi-phytozome-populus-v4.1-annotation-info. With the gene_exons excerpt they make the annotation_info dialect's round trip clean in CI. The corpus now has 98 entries: 58 vendored, 22 linked, 5 restricted, 1 not located and 12 derived.

October 1, 2026: the Populus excerpts cite their sources

For https://github.com/turbomam/feature-table-corpus/issues/151, both Populus excerpts now carry a fourth provenance line, # cite:, with the two publications Phytozome asks users to cite (Tuskan et al. 2006, Goodstein et al. 2012) and JGI's legacy-policy acknowledgment, which is kept although whether that policy governs this genome is undetermined (docs/restricted-sources.md), so the files name them and not only corpus/derived-examples/README.md. The Phytozome dialects require the line in an excerpt's provenance block, and the converters' --header JSON keeps it. scripts/populus_excerpt.py regenerated both files from the same source md5s; the only change is that line.