DuckDB compatibility for exported datasets #19

New Issue

2026-04-12T16:52:19-04:00

aalexmmaldonado commented

2026-04-12 16:52:19 -04:00

(Migrated from github.com)

For the eventual website and for power users, datasets should be queryable without loading the entire file into memory. Parquet files are natively queryable by DuckDB, which runs SQL directly on Parquet with zero configuration.

Scope

Make sure the Parquet files from Issue #18 are partitioned sensibly for large annotations, by source_name or CID range, for instance.

Add a query_dataset(path: Path, sql: str) -> pl.DataFrame function using the duckdb Python package. Document example queries in the README or docs: "Get all pKa values for caffeine (CID 2519)," "Get high-confidence pKa values between 3.0 and 5.0," "Count data points per source."

Add duckdb as an optional dependency. The core package must work without it.

Definition of done

query_dataset works on exported Parquet files. Example queries documented. The core package works without DuckDB installed.

For the eventual website and for power users, datasets should be queryable without loading the entire file into memory. Parquet files are natively queryable by DuckDB, which runs SQL directly on Parquet with zero configuration. ### Scope Make sure the Parquet files from Issue #18 are partitioned sensibly for large annotations, by `source_name` or CID range, for instance. Add a `query_dataset(path: Path, sql: str) -> pl.DataFrame` function using the `duckdb` Python package. Document example queries in the README or docs: "Get all pKa values for caffeine (CID 2519)," "Get high-confidence pKa values between 3.0 and 5.0," "Count data points per source." Add `duckdb` as an optional dependency. The core package must work without it. ### Definition of done `query_dataset` works on exported Parquet files. Example queries documented. The core package works without DuckDB installed.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: data/pcdigitizer#19