Repository map
The repository groups artifacts by their role. Paths below describe this branch's actual layout; the model remains a draft, regardless of its directory name.
corpus/ Evidence collected from producers and prior art
index.yaml Inventory, provenance, licenses, and checksums
PROVENANCE.md Dated acquisition and correction history
sources/{nmdc,ncbi-refseq,biopython,jgi-img}/ Preserved upstream source text
specifications/ Prior-art schemas and specification extracts
fixtures/{malformed,edge-cases}/ Deliberately altered, indexed test cases
derived-examples/ Explicit selections of unchanged real source rows
model/ This project's proposed contracts and instances
schema/ LinkML schemas and validation guide
profiles/ Versioned, executable source conversion contracts
dialects/ LinkML schemas for a source's own rows, validated before mapping
transforms/ linkml-map specifications from a dialect to the model
flat/ Scalar-only profile derived from the schema (docs/flat-profile.md)
validation/ Expected validator rules for each contract or dialect
examples/ Worked examples, source-artifact and JGI input manifests
analyses/ Source-specific measurements and selection reports
nmdc-data-objects/ NMDC categorical-slot and URL-host counts
nmdc-selection/ Saved sampling report for the original GFF files
conversion-roundtrips/ Case manifest and reproducible preservation outcomes
bgc-query/ Source-grounded actinorhodin order/proximity exercise
format-validation/ Independent GFF3 verdicts and retained diagnostics
docs/ Comparisons, interpretation, and design decisions
scripts/ Acquisition, generation, and validation code
tests/ Executable checks and small test-only fixtures
local/ Gitignored inputs, caches, and generated outputs
The layout decision records the old-to-new path mapping. Each artifact has one maintained location; there are no compatibility copies at the old paths. Commands below run from the repository root.
Running tasks
Use Python 3.11 or later, uv, and just 1.27 or later. The justfile uses
groups and explicit help descriptions
to keep the command list readable. Run just or just --list from the repository
root; just --show RECIPE displays a task's implementation. Named defaults such as
feature_example keep long paths out of the list; just --evaluate displays them.
| Group | Main tasks | Effect |
|---|---|---|
| Validation | check, test, schema lint and example validation |
Check retained artifacts; test scratch goes under local/ |
| Conversions | conversion-import, conversion-export, conversion-validate, conversion-check, conversion-report |
Create new conversion outputs, check preserved fields/bytes, or explicitly regenerate the tracked report |
| Corpus | verify, verify-links, fixtures-generate, pr-validation |
Link checks and the default PR audit use the network; fixture generation rewrites tracked derived files |
| Model | diagram, flat-profile-audit [schema] |
Print results; a schema URL may require network access |
| Queries | build-duckdb, query-duckdb, query-attribute, query-overlap, query-neighbors |
Build/update a database, then query it read-only |
| Queries | lakehouse-export |
Write one Parquet file per collection through linkml-store under local/ (guide) |
| Queries | bgc-check, bgc-report |
Check the real BGC exercise offline, or explicitly regenerate its selected excerpt and query report |
| Source documents | source-parse, source-validate, source-replay, validate-source-example |
Parse/replay into new files; validate consistency or compare with a retained original |
| NMDC | nmdc-collect, nmdc-render, nmdc-sample |
Explicit live collection/sampling or offline regeneration of source-specific reports |
| JGI | jgi-urls, jgi-verify [dir], jgi-collect |
Print login-only download URLs, check local copies under local/jgi/, or refresh the file list from the search API (guide) |
| Documentation | docs-build, docs-check, docs-serve [port] |
Stage tracked public files, render and check the site, or preview on loopback |
just check keeps its validation scope: it does not collect live NMDC records, sample
files, regenerate corpus fixtures, or publish reports. uv can still need the network
to obtain dependencies; UV_OFFLINE=1 uses cached dependencies when available.
Run just validity-install once to install checksum-pinned GenomeTools and
GFF3toolkit under local/tools/. validity-check checks the exact rule expectations
in model/validation/ and compares independent GFF3 verdicts and diagnostics with the
retained report. validity-report regenerates it only if the expectations match.
AGAT's macOS arm64 installation blocker is recorded in the
validation report documentation.
The optional pinned-upstream audit regression requires a retained schema supplied
through PINNED_GFF_SCHEMA; otherwise that test reports a skip.
Conversion commands and regression tests use requirements-conversion.txt to pin
the measured toolchain. conversion-check is read-only; conversion-report is the
explicit report-writing task.
The source-document guide, query examples,
and nearby NMDC reports document task arguments. Recipe arguments are forwarded as
quoted shell positional parameters: quote values with spaces at the command line.
Source-document recipes accept trailing CLI options such as --profile prodigal,
--source-uri URI, --strict, and --original PATH where the underlying operation
supports them. Their errors and exit statuses are preserved.
Recurring user and contributor operations get recipes. A recipe's existence does not
make it a dependency of check. The specialized
Bakta key extractor intentionally remains a direct
command taking the two chosen Bakta source files; its module documentation gives the
pinned retrieval and invocation commands. Individual test modules are reached through
just test rather than separate recipes. There is no blanket one-recipe-per-script
requirement. Add concise help and a task group when introducing a new public recipe.
Choose an artifact
| Question | Start here | Authority and reproduction |
|---|---|---|
| Which real files can I reuse? | Corpus index, acquisition history, then source files | The index governs the corpus inventory. Each entry records provenance and license; paths are repository-root-relative. just verify checks inventory invariants and stored checksums. RefSeq gzip downloads are stored decompressed. |
| Which cases were constructed here? | Malformed and edge-case fixtures | Nine indexed mutations of the public-domain phiX174 source, each with a mutation description and appended provenance. just fixtures-generate regenerates them; CI checks exact reproduction. Independent validation measures the validity labels separately from conversion support. |
| How do prior models differ? | Comparison, columns and producer discretion, and specification evidence | Chado SQL is an extracted subset under Artistic-2.0; the KBase YAML is an upstream module under MIT. Their indexed bytes and acquisition scope are recorded. These files are evidence for comparison. |
| What model are we proposing? | Schema guide, attribute semantics, and source-document guide | Reusable draft LinkML contracts live in model/schema/. Generic attributes are shared by biological records and source records. just check runs schema, semantic, corpus, and regression checks. |
| Which instances demonstrate those contracts? | Example crosswalk below | Harmonized examples are curated transformations; the parsed source-document example is generated. Their guides identify exact sources and validation commands. |
| Which conversions can round-trip? | Profile contracts, protein-relative example and measured outcomes | GFF3/BED12, NMDC Pfam and bounded INSDC adapters require explicit profiles and reference context; exact byte recovery and mapped-field reconstruction are tested separately. Joined, partial and circular locations have explicit semantics; unsupported GTF and richer location constructs remain visible. |
| What do the NMDC counts measure? | DataObject analysis | A generated catalogue and counts of NMDC metadata records only. The report identifies pinned schema inputs, collection timestamps, hashes, and data-use terms. Refresh all report/JSON/CSV outputs together using the commands below. |
| How were the original NMDC GFF files selected? | Selection report guide and saved selection | This sampled selection is distinct from the full DataObject analysis. just nmdc-sample writes a new live sample to local/nmdc-selection/selection.json; inspect errors and changed selections before replacing the saved report. |
| Where does other work overlap or conflict with this model? | Overlaps and conflicts and downstream copies | A running list, each row marked measured, observed or read with a date. Copies in bridge-central-schema and nmdc-lakehouse are listed with the version they follow and how they differ. |
| What should I read beyond one format's advocates? | Cross-format reading guide | Sources are dated and their perspectives identified. Literature and tool documentation motivate tests; they do not establish conversion fidelity. |
| Who will judge the format, and by what test? | Acceptance tests by discipline | Predicted tests and metrics for each discipline, with what this repository has today, and unusual gene structures from https://github.com/cmdcolin/oddgenes as representation tests. |
corpus/PROVENANCE.md is a historical account, so its intermediate counts can differ
from the current index. Likewise, a fresh upstream download or API traversal is a new
observation, not automatically a reproduction of an earlier snapshot.
Examples and their contracts
| Example | Contract | Source and reproduction |
|---|---|---|
| One biosample's sequencing | Harmonized feature model, importing generic attributes | Guide, transformation notes, and source manifest. The curated selection uses a different biosample from the standalone NMDC corpus files. just validate-example-closed validates the instance. |
| One CDS with three Pfams | The same harmonized feature and attribute schemas | Guide and the same source manifest identify rows, FASTA records, attribution, and checksums. just validate-example-closed model/examples/multiple-pfams/harmonized.yaml validates the instance. |
| Actinorhodin walkthrough | Harmonized feature model through gff3-contig/4.0.0 |
One RefSeq excerpt as source GFF3, Dataset, Parquet and BERDL table. Gives the conversion, export and query commands; the BERDL load is recorded but its script is not in this repository. |
| Converted BED12 document | Versioned BED12 conversion bundle combining Dataset, SourceDocument and mapping records | Guide reproduces and validates the generated instance from the pinned BED source; tests require byte agreement and reconstruction of ordered blocks. |
| Parsed Prodigal document | Source-document model, importing generic attributes | Guide supplies the exact parse command for the unchanged nmdc-prodigal corpus entry. just validate-source-example checks shape, internal consistency, and byte agreement with the retained corpus source. |
| Actinorhodin Dataset and Parquet | Harmonized feature model through gff3-contig/4.0.0, exported by just lakehouse-export |
The RefSeq actinorhodin excerpt as a Dataset and three Parquet files. features.parquet and contigs.parquet are the bytes loaded into BERDL on 2026-10-02; the empty contig_collections.parquet was not loaded. just actinorhodin-check, part of just check, requires the Dataset to reproduce byte for byte and the Parquet files row for row. |
| Clean round trips | Each dialect under model/dialects/ with one vendored real input and its mapping, or a gap with its issue |
tests/test_clean_round_trips.py fails if a dialect has no entry or a listed input doesn't map to the feature model and back byte for byte with nothing skipped (https://github.com/turbomam/feature-table-corpus/issues/107). |
| IMG core crosswalk | SSSOM mappings from model slots to IMG img_core_v400 columns |
Guide gives the source commit and what doesn't line up; tests/test_img_core_crosswalk.py checks every subject is a slot of its class. |
| Restricted-source checklist | When a derived excerpt may come from a login-only source, and the redistribution_basis its index entry needs |
scripts/verify.py refuses a derived entry from a restricted source without a complete redistribution_basis; tests/test_verify_derived_sources.py covers each part. |
The first two represent biological records. The third preserves ordered source text, including comments, directives, and feature rows. Source replay alone is not evidence of a reversible conversion through the harmonized model; executable conversion-profile guarantees are tracked in #16.
Parser test fixtures stay under tests/
because they are small constructed inputs for unit tests, not additional observed
producer outputs. Corpus fixtures are separately indexed evidence for validator
comparisons and remain under corpus/fixtures/.
Record granularity
- An NMDC DataObject describes an artifact. Its category and URL-host frequencies count NMDC metadata records, not genomic features or format use across genomics.
- A source artifact identifies bytes and provenance. A source document represents the ordered contents of one physical text artifact.
- A harmonized Dataset can combine several artifacts and workflow executions. Contig and SequenceFeature instances are biological/model records.
- An Attribute is an attached key/value entry. The containing object supplies its scope; GFF column 9 is one use case.
Generated outputs and local work
The schema diagram is regenerated by just diagram, with freshness
checked by CI. just build-duckdb writes an experimental physical mapping under
gitignored local/build/. Database binaries do not define the model.
just lakehouse-export writes Parquet under local/ with the toolchain pinned in
requirements-lakehouse.txt; see the Parquet export guide.
To refresh the NMDC DataObject analysis with Python 3.11 or later:
just nmdc-collect
just nmdc-render
nmdc-collect traverses the public API and writes local inputs plus published aggregates.
nmdc-render requires checksum-verified inputs under local/nmdc-profile/ and cached
dependencies; a fresh clone cannot reproduce the dated counts offline. An API traversal
is not a transactional snapshot. See the report's refresh safeguards.
Keep task inputs, retained downloads, and scratch output under local/<task>/, with
source URLs and checksums where needed. These gitignored files are excluded from the
GitHub Pages build.