Parquet export with linkml-store
just lakehouse-export DATASET DIR validates a Dataset with
validate_closed.py, loads it into
linkml-store's DuckDB backend, and writes one
Parquet file per collection to DIR. It was written for
https://github.com/turbomam/feature-table-corpus/issues/57. The code is
scripts/lakehouse_export.py and the pinned toolchain is
requirements-lakehouse.txt (linkml 1.11.1, linkml-store
0.3.2 plus the scalar-type fix described below, DuckDB 1.5.5, pyarrow 25.0.1).
just map-img-functional local/jgi/IMG_AP-1268149/Ga0423362_functional_annotation.gff local/ga0423362.json
just lakehouse-export local/ga0423362.json local/lakehouse/ga0423362
DIR must not exist yet and must be under local/, so Parquet files never sit beside
tracked files. Nothing appears at DIR unless the three checks below pass.
Concurrency contract. The export writes a new directory under local/. Other processes
writing to the same path at the same time are not supported. Within that:
- Everything is written first to a private staging directory, created with a unique name
beside
DIR. On any failure, that staging directory is the only thing removed. DIRis claimed withmkdirand each file is linked in withos.link; both fail if the name already exists, so nothing another process creates is replaced.- Missing parent directories are created one at a time, and only those whose
mkdirsucceeded count as this run's. - The export never deletes or replaces anything in the target path. On failure it may
leave newly created parent directories,
DIR, and files already linked intoDIR; each is named on its own line ("left behind, remove if unwanted: PATH") so you can remove it.
Layout
The collections are the inlined list slots of the schema's tree root, read from
ber_feature_model.yaml, so the export writes
contig_collections.parquet, contigs.parquet and features.parquet. contig_collections
was added by https://github.com/turbomam/feature-table-corpus/pull/62, merged 2026-09-26,
and got its own file without a change to the export; that pull request also gave
build_duckdb.py the table the row-count check needs. A collection the schema adds later
works the same way. Each file has one column per slot of its class, in schema order. A
collection with no rows still gets a file with every column, so the files for different
Datasets have the same columns.
Types
linkml-store creates each table from the schema. Released linkml-store 0.3.2 maps three
scalar ranges to types that lose values, so requirements-lakehouse.txt pins commit
22bc4d7
on branch fix/duckdb-scalar-types of https://github.com/turbomam/linkml-store, which maps them without loss.
That is a personal fork, not a linkml-store release. The problem is reported in
https://github.com/linkml/linkml-store/issues/85 and the fix proposed in
https://github.com/linkml/linkml-store/pull/86; the pin goes back to a released version once
that is merged and released. The export checks the installed type table before it writes
anything and stops on a lossy one. It then casts every column to the type the schema gives
when writing, which turns inlined objects into nested Parquet types.
| Slot range | linkml-store 0.3.2 | Pinned commit | Written to Parquet |
|---|---|---|---|
float (score, lineage_confidence) |
4-byte FLOAT |
DOUBLE |
double |
integer (start, end, length_bp, phase) |
4-byte INTEGER |
BIGINT |
int64 |
boolean (is_selected) |
VARCHAR holding "true" |
BOOLEAN |
bool |
inlined class (attributes, location) |
JSON or JSON[] text |
same | struct or list<struct> |
reference to a class (seqid, parent), string, enum |
VARCHAR |
same | string |
With the 0.3.2 types, a score of 239.1 in the committed
one-biosample example reads
back as 239.10000610351562. Measured on 2026-09-25, those types changed the score of
4,222 of 4,439 features in Ga0423362 and 1,916 of 2,005 in Ga0416744; with the pinned
commit on 2026-09-28, Ga0423362 exported with every value read back equal. Tests keep the
0.3.2 types as negative controls. The 4-byte integer holds positions up to 2,147,483,647,
which no current input exceeds, but 0.3.2 refuses a larger one with "out of range"; a test
exports a feature at 2^33 and checks that the 0.3.2 mapping refuses it.
linkml-store's own Parquet routes were tried first, on 2026-09-25:
Database.export_database(path, Format.PARQUET)raises "Unsupported output format: Format.PARQUET" and leaves an empty file.- The filesystem backend with
file_format: parquetwrites through pandas, so types come from the data rather than the schema.phasebecamedoublebecause some rows have no phase, and slots that no row uses (topology,locationin the example) got no column.
So the export keeps linkml-store's DuckDB tables and writes them with DuckDB's
COPY ... (FORMAT PARQUET) on the same connection.
Nested attributes
attributes comes out as list<struct<key: string, value: string>>, one entry per value
in file order. It is not flattened into a separate table. One UNNEST gives a key/value
row per attribute:
SELECT feature_id,
generate_subscripts(attributes, 1) AS ordinal,
unnest(attributes).key AS key,
unnest(attributes).value AS value
FROM 'local/lakehouse/ga0423362/features.parquet';
On 2026-09-25 this returned 53,609 rows for Ga0423362 and 24,507 for Ga0416744, the
attribute counts the linkml-map round trip
reports for those files. location is a struct with a parts list of structs.
That works for engines that read nested Parquet. A catalog limited to scalar columns, as the BRIDGE catalog prototype is described in https://github.com/turbomam/feature-table-corpus/issues/45, still needs the derived child tables proposed there; this export does not produce them.
Checks
All three run before DIR is created, and a failure leaves no Parquet output (at most
the reported parent directories):
- Column types. Each file's column names and types, as DuckDB reads them, must equal
those the schema gives. The value check can't see this: an
is_selectedrewritten asint64compares equal, becauseTrue == 1in Python. DuckDB's DESCRIBE does not show Parquet-level differences such aslarge_stringagainststringor nullability; that is enough here because DuckDB's ownCOPYwrites the files and cannot produce those. - Row counts. Each collection's row count must equal the matching table that
build_duckdb.pybuilds from the same Dataset (the table name is the range class in snake case, such assequence_feature). A table with no matching collection is also an error. https://github.com/turbomam/feature-table-corpus/issues/57 asked for this check. The value check already compares row counts against the input, and both paths read the file with the sameload_validated, so this adds no independent reading. It is kept because it is the only check that ties the export to the existing DuckDB mapping: a collection one has and the other lacks is reported. It is also most of the check time: 5.37 s onGa0423362, measured 2026-09-25, against 0.01 s for types and 0.09 s for values. - Values. Each file is read back with pyarrow, and every row must equal the validated input, with null and empty lists treated as absent.
tests/test_lakehouse_export.py reads the files back
for the committed example, the constructed IMG fixture and the vendored phiX174 RefSeq
record (joined and origin-crossing locations). Its negative controls change one attribute
value, drop one row (with the value check off, so only the row count can catch it),
rewrite is_selected as int64, export with linkml-store 0.3.2's types, and export with a
lossy type table installed; each must be reported.
Measurements
One run each on 2026-09-25, after the column-type check was added. "Checks" is mostly
build_duckdb.py validating the Dataset again and inserting row by row.
| Dataset | Features | Dataset JSON | features.parquet |
Validate | Load and write | Checks | Wall clock |
|---|---|---|---|---|---|---|---|
| One-biosample example | 15 | YAML | 9,737 B | 0.06 s | 0.03 s | 0.05 s | not timed |
| Constructed IMG fixture | 10 | 7,908 B | 5,895 B | 0.06 s | 0.03 s | 0.05 s | 0.79 s |
Ga0416744 (IMG_AP-1257706) |
2,005 | 2,426,519 B | 274,639 B | 2.03 s | 0.15 s | 2.45 s | 5.30 s |
Ga0423362 (IMG_AP-1268149) |
4,439 | 5,371,707 B | 595,593 B | 4.46 s | 0.30 s | 5.35 s | 10.78 s |
For comparison, just build-duckdb on the Ga0423362 Dataset took 9.03 s and wrote a
2,109,440-byte database. The contig files were under 1 KB for the IMG Datasets and 2,161 B
for the example.