Skip to content

Parquet export with linkml-store

just lakehouse-export DATASET DIR validates a Dataset with validate_closed.py, loads it into linkml-store's DuckDB backend, and writes one Parquet file per collection to DIR. It was written for https://github.com/turbomam/feature-table-corpus/issues/57. The code is scripts/lakehouse_export.py and the pinned toolchain is requirements-lakehouse.txt (linkml 1.11.1, linkml-store 0.3.2 plus the scalar-type fix described below, DuckDB 1.5.5, pyarrow 25.0.1).

just map-img-functional local/jgi/IMG_AP-1268149/Ga0423362_functional_annotation.gff local/ga0423362.json
just lakehouse-export local/ga0423362.json local/lakehouse/ga0423362

DIR must not exist yet and must be under local/, so Parquet files never sit beside tracked files. Nothing appears at DIR unless the three checks below pass.

Concurrency contract. The export writes a new directory under local/. Other processes writing to the same path at the same time are not supported. Within that:

  • Everything is written first to a private staging directory, created with a unique name beside DIR. On any failure, that staging directory is the only thing removed.
  • DIR is claimed with mkdir and each file is linked in with os.link; both fail if the name already exists, so nothing another process creates is replaced.
  • Missing parent directories are created one at a time, and only those whose mkdir succeeded count as this run's.
  • The export never deletes or replaces anything in the target path. On failure it may leave newly created parent directories, DIR, and files already linked into DIR; each is named on its own line ("left behind, remove if unwanted: PATH") so you can remove it.

Layout

The collections are the inlined list slots of the schema's tree root, read from ber_feature_model.yaml, so the export writes contig_collections.parquet, contigs.parquet and features.parquet. contig_collections was added by https://github.com/turbomam/feature-table-corpus/pull/62, merged 2026-09-26, and got its own file without a change to the export; that pull request also gave build_duckdb.py the table the row-count check needs. A collection the schema adds later works the same way. Each file has one column per slot of its class, in schema order. A collection with no rows still gets a file with every column, so the files for different Datasets have the same columns.

Types

linkml-store creates each table from the schema. Released linkml-store 0.3.2 maps three scalar ranges to types that lose values, so requirements-lakehouse.txt pins commit 22bc4d7 on branch fix/duckdb-scalar-types of https://github.com/turbomam/linkml-store, which maps them without loss. That is a personal fork, not a linkml-store release. The problem is reported in https://github.com/linkml/linkml-store/issues/85 and the fix proposed in https://github.com/linkml/linkml-store/pull/86; the pin goes back to a released version once that is merged and released. The export checks the installed type table before it writes anything and stops on a lossy one. It then casts every column to the type the schema gives when writing, which turns inlined objects into nested Parquet types.

Slot range linkml-store 0.3.2 Pinned commit Written to Parquet
float (score, lineage_confidence) 4-byte FLOAT DOUBLE double
integer (start, end, length_bp, phase) 4-byte INTEGER BIGINT int64
boolean (is_selected) VARCHAR holding "true" BOOLEAN bool
inlined class (attributes, location) JSON or JSON[] text same struct or list<struct>
reference to a class (seqid, parent), string, enum VARCHAR same string

With the 0.3.2 types, a score of 239.1 in the committed one-biosample example reads back as 239.10000610351562. Measured on 2026-09-25, those types changed the score of 4,222 of 4,439 features in Ga0423362 and 1,916 of 2,005 in Ga0416744; with the pinned commit on 2026-09-28, Ga0423362 exported with every value read back equal. Tests keep the 0.3.2 types as negative controls. The 4-byte integer holds positions up to 2,147,483,647, which no current input exceeds, but 0.3.2 refuses a larger one with "out of range"; a test exports a feature at 2^33 and checks that the 0.3.2 mapping refuses it.

linkml-store's own Parquet routes were tried first, on 2026-09-25:

  • Database.export_database(path, Format.PARQUET) raises "Unsupported output format: Format.PARQUET" and leaves an empty file.
  • The filesystem backend with file_format: parquet writes through pandas, so types come from the data rather than the schema. phase became double because some rows have no phase, and slots that no row uses (topology, location in the example) got no column.

So the export keeps linkml-store's DuckDB tables and writes them with DuckDB's COPY ... (FORMAT PARQUET) on the same connection.

Nested attributes

attributes comes out as list<struct<key: string, value: string>>, one entry per value in file order. It is not flattened into a separate table. One UNNEST gives a key/value row per attribute:

SELECT feature_id,
       generate_subscripts(attributes, 1) AS ordinal,
       unnest(attributes).key AS key,
       unnest(attributes).value AS value
FROM 'local/lakehouse/ga0423362/features.parquet';

On 2026-09-25 this returned 53,609 rows for Ga0423362 and 24,507 for Ga0416744, the attribute counts the linkml-map round trip reports for those files. location is a struct with a parts list of structs.

That works for engines that read nested Parquet. A catalog limited to scalar columns, as the BRIDGE catalog prototype is described in https://github.com/turbomam/feature-table-corpus/issues/45, still needs the derived child tables proposed there; this export does not produce them.

Checks

All three run before DIR is created, and a failure leaves no Parquet output (at most the reported parent directories):

  • Column types. Each file's column names and types, as DuckDB reads them, must equal those the schema gives. The value check can't see this: an is_selected rewritten as int64 compares equal, because True == 1 in Python. DuckDB's DESCRIBE does not show Parquet-level differences such as large_string against string or nullability; that is enough here because DuckDB's own COPY writes the files and cannot produce those.
  • Row counts. Each collection's row count must equal the matching table that build_duckdb.py builds from the same Dataset (the table name is the range class in snake case, such as sequence_feature). A table with no matching collection is also an error. https://github.com/turbomam/feature-table-corpus/issues/57 asked for this check. The value check already compares row counts against the input, and both paths read the file with the same load_validated, so this adds no independent reading. It is kept because it is the only check that ties the export to the existing DuckDB mapping: a collection one has and the other lacks is reported. It is also most of the check time: 5.37 s on Ga0423362, measured 2026-09-25, against 0.01 s for types and 0.09 s for values.
  • Values. Each file is read back with pyarrow, and every row must equal the validated input, with null and empty lists treated as absent.

tests/test_lakehouse_export.py reads the files back for the committed example, the constructed IMG fixture and the vendored phiX174 RefSeq record (joined and origin-crossing locations). Its negative controls change one attribute value, drop one row (with the value check off, so only the row count can catch it), rewrite is_selected as int64, export with linkml-store 0.3.2's types, and export with a lossy type table installed; each must be reported.

Measurements

One run each on 2026-09-25, after the column-type check was added. "Checks" is mostly build_duckdb.py validating the Dataset again and inserting row by row.

Dataset Features Dataset JSON features.parquet Validate Load and write Checks Wall clock
One-biosample example 15 YAML 9,737 B 0.06 s 0.03 s 0.05 s not timed
Constructed IMG fixture 10 7,908 B 5,895 B 0.06 s 0.03 s 0.05 s 0.79 s
Ga0416744 (IMG_AP-1257706) 2,005 2,426,519 B 274,639 B 2.03 s 0.15 s 2.45 s 5.30 s
Ga0423362 (IMG_AP-1268149) 4,439 5,371,707 B 595,593 B 4.46 s 0.30 s 5.35 s 10.78 s

For comparison, just build-duckdb on the Ga0423362 Dataset took 9.03 s and wrote a 2,109,440-byte database. The contig files were under 1 KB for the IMG Datasets and 2,161 B for the example.