Skip to content

NMDC DataObject profile

This catalogue describes NMDC DataObjects only. NMDC is one source of evidence for the broader feature/attribute modeling work. These fields, vocabularies, hosts, and frequencies are not requirements of the proposed general model.

Collected 2026-09-21T15:34:56+00:00 from the public NMDC API. Coverage: 303,977 DataObjects, all 304 pages of data_object_set, without a feature-file or workflow filter. Unique IDs and the collection counts before/after agree. This traversal is not a transactional snapshot; unchanged totals cannot prove that no individual record changed during collection.

Schema release: 11.23.0, commit 84651fe32b2e. The release's merged YAML embeds version 0.0.0; the release, commit, URL, and SHA-256 in the JSON provenance identify the actual source used.

Categorical slots

Schema enums and booleans are discovered from DataObject's induced slots, including inheritance. compression_type and the type class discriminator are explicitly included as categorical strings; neither is falsely labeled an enum. Free-text names, descriptions, identifiers, references, sizes, and checksums are not categorical slots.

Slot Range Required Allowed enum values Observed distinct values Present Missing Null Empty string Invalid type
compression_type string false not enumerated 0 0 303977 0 0 0
data_category DataCategoryEnum true 3 3 303977 0 0 0 0
data_object_type FileTypeEnum true 106 82 303977 0 0 0 0
type uriorcurie true not enumerated 1 303977 0 0 0 0

categorical-values.csv lists each value and its DataObject count, including permitted-but-unobserved enum values with zero counts. Unknown enum values remain visible. Missing, null, empty, and invalid types are separate states in counts.json; these are not counted as distinct categorical values.

Counts measure records carrying a value; duplicate values in a multivalued slot would count once per record. A multivalued slot's frequencies need not sum to the record total. Empty lists and lists containing empty strings are recorded as empty_list and empty_item; neither contributes categorical counts. All slots in this release are scalar.

URL hosts

Hosts are parsed from the URL authority after the scheme and before the path, then lowercased. Ports and user information are excluded; IPv6 hosts are supported. Missing, null, empty, malformed, and hostless values remain separate. Bare paths or bare domains are not silently treated as HTTP URLs. Counts describe metadata records, not files fetched, host availability, or distinct URLs.

Hostname DataObjects
data.microbiomedata.org 256,359
nmdcdemo.emsl.pnnl.gov 26,423
storage.neonscience.org 5,646
massive.ucsd.edu 2,733
portal.nersc.gov 231

Distinct hosts: 5.

URL states: missing: 12,585, present: 291,392.

url-hosts.csv provides the same host counts as CSV.

Reproduction and interpretation

Run from the repository root:

just nmdc-collect
just nmdc-render

nmdc-collect refreshes the public metadata projection and verifies completeness before publishing aggregates. nmdc-render recreates them offline from the checksum-verified files under gitignored local/nmdc-profile/. No data files referenced by the URLs are downloaded. The script is pinned to the API schema release above and refuses a different release until its schema source is updated. A later collection may have different counts; each report carries its own timestamps and content hashes.

These recipes require Python 3.11+, uv, and just 1.27+. Both accept optional work and output directories: just nmdc-render WORK_DIR OUTPUT_DIR. Use a separate output directory when inspecting a retained snapshot before updating this report.

Run just nmdc-collect online once to obtain the projection and schema and populate uv's dependency cache. The just nmdc-render recipe uses uv's --offline mode as well as making no API calls; it needs those cached dependencies.

The catalogue is derived from the NMDC schema source, licensed CC0. NMDC's data use policy applies to the source metadata. This is descriptive source profiling, not validation of every NMDC schema constraint or a universal feature-table vocabulary.