Design a confidence ranking framework #16

New Issue

2026-04-12T16:48:25-04:00

aalexmmaldonado commented

2026-04-12 16:48:25 -04:00

(Migrated from github.com)

PubChem data comes from depositors with wildly different levels of rigor. ChEBI and HSDB curate from primary literature. EPISuite deposits automated predictions. Some submissions are bare floats with no citation, no temperature, no context. An ML-ready dataset should carry a quality signal so users can filter by confidence.

This is a design issue. The deliverable is a written document, not code.

Scope

The design should address four questions.

First: what signals are available? Source reputation, the presence of a literature reference, whether conditions (temperature, solvent) are reported, internal consistency across sources for the same compound, and whether a PCLID is present.

Second: how should confidence be represented? A single numeric score (0–100)? A categorical tier ("high", "medium", "low", "predicted")? Multiple independent flags (has_reference, has_temperature, is_curated_source)?

Third: where does it live in the codebase? A post-processing step applied to any processor's DataFrame? Built into each processor? A separate module in pcdigitizer/quality/?

Fourth: How should source reputation be maintained? A static YAML file mapping source names to tiers? Fetched dynamically from PubChem's source table?

Consider at least two alternative designs and compare them.

Definition of done

A markdown document covering all four questions with a recommended approach and rationale. Review and sign off from the maintainer before implementation begins.

PubChem data comes from depositors with wildly different levels of rigor. ChEBI and HSDB curate from primary literature. EPISuite deposits automated predictions. Some submissions are bare floats with no citation, no temperature, no context. An ML-ready dataset should carry a quality signal so users can filter by confidence. This is a design issue. The deliverable is a written document, not code. ### Scope The design should address four questions. First: what signals are available? Source reputation, the presence of a literature reference, whether conditions (temperature, solvent) are reported, internal consistency across sources for the same compound, and whether a PCLID is present. Second: how should confidence be represented? A single numeric score (0–100)? A categorical tier ("high", "medium", "low", "predicted")? Multiple independent flags (`has_reference`, `has_temperature`, `is_curated_source`)? Third: where does it live in the codebase? A post-processing step applied to any processor's DataFrame? Built into each processor? A separate module in `pcdigitizer/quality/`? Fourth: How should source reputation be maintained? A static YAML file mapping source names to tiers? Fetched dynamically from PubChem's source table? Consider at least two alternative designs and compare them. ### Definition of done A markdown document covering all four questions with a recommended approach and rationale. Review and sign off from the maintainer before implementation begins.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: data/pcdigitizer#16