93 lines
6.7 KiB
Markdown
93 lines
6.7 KiB
Markdown
# Overview
|
|
|
|
## The idea
|
|
|
|
Most tools for scientific literature aggregate: they index hundreds of millions of papers and answer questions over all of them, shallowly.
|
|
This project does the opposite.
|
|
A team working on a focused question cares more about fifty papers it understands completely than about the world's literature.
|
|
Those fifty papers deserve to be converted once, corrected once, split down to the sentence, annotated by the people who know them, linked to checked claims, and kept.
|
|
|
|
What that work produces answers two questions scientists ask every day.
|
|
|
|
1. **Does the cited passage say what is claimed?**
|
|
A citation to a real paper can still be inaccurate, incomplete or beside the point, and a reader needs to see the exact sentence it rests on.
|
|
2. **Where did this number come from, and how far was it checked?**
|
|
A force constant in a table, a value read off a plot, a parameter in a model: each should lead back to the cell or sentence it came from, with a record of who compared it with the page.
|
|
|
|
A model can summarize a paper, but it cannot produce a signed statement that a person checked a passage, IDs that never move, or a store its owner controls.
|
|
Those are what these formats are for, and they are worth most at the scale of one team.
|
|
|
|
## Who it is for
|
|
|
|
| User | Example |
|
|
| --- | --- |
|
|
| One researcher | A thesis student, a science writer or fact-checker, a Wikipedia editor checking a citation, a scientist keeping a deep personal library. |
|
|
| An academic lab | A research group, a thesis committee, a journal club reading one annotated paper together. |
|
|
| A course | An instructor whose students learn to check whether a source supports a claim, with answers that are addresses and can be graded. |
|
|
| An industrial team | A computational chemistry or materials group that needs to show which passage of which paper each parameter came from. |
|
|
|
|
"Team" throughout means roughly five to twenty people with a few dozen to a few hundred papers that matter.
|
|
Everything is designed for that scale: hundreds of documents and millions of sentences, not hundreds of millions of papers.
|
|
|
|
## The parts
|
|
|
|
The formats in this repository are the meeting point of several programs.
|
|
No program imports another's code; they meet only through these files.
|
|
|
|
```text
|
|
spec (this repository)
|
|
│
|
|
┌─────────────────┼──────────────────┐
|
|
core core-ts typst-scienting
|
|
┌──────┬──┴─────┬────────┐ │
|
|
paperpack digitize lode varve varve-web
|
|
```
|
|
|
|
| Part | Does |
|
|
| --- | --- |
|
|
| **spec** | Defines every format: prose, schemas, tables, test vectors and fixtures. |
|
|
| **core** | `scienting-core`, the reference implementation of the shared rules in Python, standard library only: canonical JSON, hashing, IDs, addresses, anchors, packages, receipts and rights checks. |
|
|
| **core-ts** | The same rules in TypeScript, for the browser. |
|
|
| **paperpack** | The converter. It turns a PDF, Supporting Information, JATS XML, LaTeX, HTML, EPUB or a Typst project into a package, raises doubtful readings as suspects, and applies corrections only when they pass its gate. |
|
|
| **digitize** | Reads values off plots: calibrated axes, traced series, an overlay a person confirms. |
|
|
| **Lode** | The store of checked claims. Its extracts point at sentences through anchors, so a claim can verify itself against the words it rests on. |
|
|
| **Varve** | The team workspace: a store a team can clone, a server, a reader, an editor and an MCP server that lets a team's own models quote the corpus by address. |
|
|
| **typst-scienting** | A Typst package that lets a draft cite sentences, claims and datasets. |
|
|
|
|
A varve is one year's pair of sediment layers on a lake bed; geologists count them because each layer is laid down once and kept.
|
|
That is the product's idea: every reading a team does leaves a layer, dated and signed, over a text that never has to be redone.
|
|
|
|
## How a paper moves through the system
|
|
|
|
1. **Acquire.** A witness arrives (the publisher's PDF, say) with a rights record that says how the team holds it and what that allows.
|
|
2. **Read and reconcile.** paperpack reads it with one or more backends, compares the readings with each other and with the PDF's text layer, and fingerprints every page so that differently stamped downloads are recognized as the same pages.
|
|
3. **Segment and mint.** Text is split into sentences, and every section, paragraph, sentence, figure, panel, table, row, cell, equation, footnote and reference gets a permanent ID and five content hashes.
|
|
4. **Review.** Disagreements become suspects with page crops; a person or a model session answers them as corrections, and the gate refuses any correction that changes a digit without image evidence.
|
|
5. **Package.** The result is a package: a folder of canonical JSON files with a manifest whose hash pins the whole.
|
|
6. **Use.** The package loads into a database, a reader renders it with layers of annotations, Lode anchors claims to it, datasets anchor values to its cells, and drafts cite its sentences.
|
|
7. **Check and sign.** People proofread what matters and sign receipts; second reviewers spot-check them; every level counts only at the current text.
|
|
|
|
[The walkthrough](walkthrough.md) follows one paper through every step.
|
|
|
|
## Principles
|
|
|
|
| Principle | Means |
|
|
| --- | --- |
|
|
| Quality over quantity | A small corpus processed to a stated and checkable fidelity beats a large one processed to an unknown one. |
|
|
| Do the work once | Conversion, correction, segmentation and digitization are recorded and kept; nothing is reparsed to be fixed again. |
|
|
| Address everything | Every section, paragraph, sentence, figure, panel, caption, table, cell, equation, footnote and reference has a permanent ID and a content hash. |
|
|
| Keep source and commentary apart | What a source says is stored as the source said it; everything people or models add is stored beside it and points at it. |
|
|
| Count only what was done | A level says how far something was checked, and it falls back when the checked content changes. |
|
|
| Models propose, people sign | Models draft, segment, gloss and digitize; only a named person confirms. |
|
|
| The team owns its data | Every store is a database the team can clone, and every format is documented and readable without the studio's code. |
|
|
|
|
## What it is not
|
|
|
|
It is not a search engine over all literature, a citation index, a recommendation feed or a chat-with-any-PDF tool.
|
|
Tools may import from Crossref, OpenAlex, PubMed Central or arXiv, but never depend on them to work.
|
|
|
|
## Where next
|
|
|
|
- [Concepts](concepts.md) explains each idea with an example.
|
|
- [The specification](../spec/README.md) defines each format exactly.
|