Parquet export and dataset distribution #18

New Issue

2026-04-12T16:51:02-04:00

aalexmmaldonado commented

2026-04-12 16:51:02 -04:00

(Migrated from github.com)

The end goal of pcdigitizer is ML-ready datasets. Right now, the only way to get data is by calling the API at runtime. For reproducibility, sharing, and downstream tooling, we need to export complete, pre-processed datasets as Parquet files.

Scope

Create pcdigitizer/export.py with an export_annotation function that fetches all pages for a given annotation, processes them, concatenates the resulting DataFrames, and writes a single Parquet file using polars.DataFrame.write_parquet() with zstd or snappy compression.

Add a CLI entry point: python -m pcdigitizer export --annotation "Dissociation Constants" --output pka_dataset.parquet. Use argparse, no extra dependencies.

Embed metadata in the Parquet schema: pcdigitizer_version, annotation, export_date (ISO timestamp), and total_pages. Add a read_dataset convenience function that reads the Parquet back and validates the schema.

Definition of done

A CLI command that exports a complete annotation to Parquet. Metadata embedded in the file. Round-trip test: export, read, verify schema, and row count.

The end goal of pcdigitizer is ML-ready datasets. Right now, the only way to get data is by calling the API at runtime. For reproducibility, sharing, and downstream tooling, we need to export complete, pre-processed datasets as Parquet files. ### Scope Create `pcdigitizer/export.py` with an `export_annotation` function that fetches all pages for a given annotation, processes them, concatenates the resulting DataFrames, and writes a single Parquet file using `polars.DataFrame.write_parquet()` with zstd or snappy compression. Add a CLI entry point: `python -m pcdigitizer export --annotation "Dissociation Constants" --output pka_dataset.parquet`. Use `argparse`, no extra dependencies. Embed metadata in the Parquet schema: `pcdigitizer_version`, `annotation`, `export_date` (ISO timestamp), and `total_pages`. Add a `read_dataset` convenience function that reads the Parquet back and validates the schema. ### Definition of done A CLI command that exports a complete annotation to Parquet. Metadata embedded in the file. Round-trip test: export, read, verify schema, and row count.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: data/pcdigitizer#18