Automatic pagination in PubChemAPI.get_data #21

New Issue

2026-04-12T16:57:10-04:00

aalexmmaldonado commented

2026-04-12 16:57:10 -04:00

(Migrated from github.com)

PubChemAPI.get_data claims to fetch all data when page=None, but it doesn't actually paginate. It makes a single request without a page parameter, and PubChem returns only page 1 by default. The response includes Page and TotalPages fields that tell you exactly how many pages exist. We're ignoring them.

Scope

When page=None, fetch page 1, read TotalPages from the response, then fetch pages 2 through TotalPages and concatenate the annotation lists. Respect rate limits (Issue 13). Add a max_pages parameter to cap pagination for testing and for large annotations. Log progress: "Fetching page {n}/{total} for {annotation}".

Definition of done

PubChemAPI.get_data(Annotation.DISSOCIATION_CONSTANTS) with no page argument returns all pages. max_pages works. Tested with mock responses simulating multi-page results.

`PubChemAPI.get_data` claims to fetch all data when `page=None`, but it doesn't actually paginate. It makes a single request without a page parameter, and PubChem returns only page 1 by default. The response includes `Page` and `TotalPages` fields that tell you exactly how many pages exist. We're ignoring them. ### Scope When `page=None`, fetch page 1, read `TotalPages` from the response, then fetch pages 2 through `TotalPages` and concatenate the annotation lists. Respect rate limits (Issue 13). Add a `max_pages` parameter to cap pagination for testing and for large annotations. Log progress: `"Fetching page {n}/{total} for {annotation}"`. ### Definition of done `PubChemAPI.get_data(Annotation.DISSOCIATION_CONSTANTS)` with no page argument returns all pages. `max_pages` works. Tested with mock responses simulating multi-page results.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: data/pcdigitizer#21

Automatic pagination in PubChemAPI.get_data #21

Scope

Definition of done

Automatic pagination in `PubChemAPI.get_data` #21