papers/paperspec
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: papers/paperspec:2fd330714785ea98d1fc1fca67d8da9bcfca000e
papers/paperspec:main papers/paperspec:dev
..
compare: papers/paperspec:dev
papers/paperspec:dev papers/paperspec:main
12 Commits
2fd3307147 .. dev
Author SHA1 Message Date
alexmandClaude 7890fa1ad7 feat: trace annotation body for traced citations; paperspec.traces 
A trace follows one citation of a sentence of a citing paper to a passage of the
cited paper (spec/18 section 18.5). Minor change: a new body kind, optional
properties and two rules, with no valid file made invalid.

- annotation schema branch (source, selector, reference, note, verdict, proposer;
  proposer required for a Software creator); body_kind in sql/store.sql; the note
  and proposer terms of the JSON-LD context; rules layer.trace.source and
  layer.trace.target.
- paperspec.traces: resolve(annotation, citing_state, cited_state, same_as) gives
  current, changed, stale, unlinked or misresolved with open and cites; problems,
  disputes, disputed.
- fixtures/records/traces/ on writer2027fixed-ynr7.v3 and vectors/traces.jsonl
  (29 cases); check.py runs the record checks over both layers; counts, glossary,
  examples and CHANGELOG updated.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uy4ZYbWRoCdUtWLEfigysY
 2026-10-17 20:12:04 -04:00
alexmandClaude 44a8e3f07c docs: the converter is named paperform 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SarHTuHhV9cS85XWy2NrLJ
 2026-10-17 10:39:56 -04:00
alexmandClaude c7f54320dd refactor!: claim references are claim:TAG 
lode is renamed paperclaim. A claim reference is claim:TAG or
claim:host/TAG and a pin claim:TAG@hex6 (claimRef and claimPin in
common.schema.json, spec/18 §18.4, spec/02, spec/20); the eight-character
tags do not change. A project names its claim namespace as
claims.paperclaim, and the main example's targeted receipts take the
scope targeted:paperclaim. No old prefix or name is accepted.

paperspec build regenerated the fixtures and vectors: the claim link's and
the symbol's record IDs, the layer receipt, project.json, the two targeted
receipts and Ben's spot-check, the QA report, the inclusion proof, the log's
checkpoint and tiles, and the signature, spot-check, VRF and checkpoint
vectors. The walkthrough and examples quote the new receipt IDs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SarHTuHhV9cS85XWy2NrLJ
 2026-10-17 10:38:53 -04:00
alexmandClaude 568e0271d3 docs: scienting-math/1 tree kinds prescript and mark 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017R7sgDNvUGWY2zFPCq2JH6
 2026-10-17 07:15:55 -04:00
alexmandClaude 17880686fa feat: scienting-math/1 sources and derived/math.jsonl; lode references; stress fixes 
Math content may say its source is in scienting-math/1's canonical TeX (form,
in no hash); derived/math.jsonl holds papermath's tree, symbols and issues,
and the validator now checks derived files. A symbol annotation names its
symbol by key; lode claim references use lode's tag and pin syntax. The zip
reader refuses entries zipfile cannot read, and checkpoint lines end in LF only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017R7sgDNvUGWY2zFPCq2JH6
 2026-10-17 06:44:28 -04:00
alexmandClaude 8b5ee87275 feat: comment categories; emend, relink and resolve corrections 
scienting-layer/1: an optional, fixed category on comment, caveat and
question bodies (comment, context, critique, question, correction,
feedback, resource).

Corrections: emend fixes the authors' own slip and keeps what was printed
(a run's printed), never lowering fidelity or a witness match; relink sets
or removes what a citation names; resolve sets a reference's resolution.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017R7sgDNvUGWY2zFPCq2JH6
 2026-10-17 06:17:10 -04:00
alexmandClaude 4d59b29b63 chore!: the converter is named papervert (short form pvt) 
paperstract becomes papervert, the paper converter: package, command,
PAPERVERT_* variables, the [papervert] settings section, tools/pvt.py
and the .pvt- CSS classes. No old name is kept as an alias.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017R7sgDNvUGWY2zFPCq2JH6
 2026-10-17 05:44:51 -04:00
alexmandClaude 3836a6c885 feat: alignment pairs a misread sentence with its other reading; a suspect names its witness 
spec/05 §5.6 gains a substitution step: an unpaired sentence beside one at
least 0.8 alike (2L/(|a|+|b|) over the lossy projections' code points) is
paired with it, so a misread is one number, equation or word suspect rather
than a dropped and an added one. §5.7 and suspect.schema.json gain an
optional `witness`, checked by doc.suspect.witness. alignment.jsonl gains a
`substitutions` case; the first case is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 22:41:15 -04:00
alexm 9e24ca2306 Merge branch 'main' into dev 2026-10-15 09:53:38 -04:00
alexm 78101dc0db Reorganize paperspec commands and pin dependency versions (#2)
check / check (push) Successful in 52s
Details
site / site (push) Successful in 1m45s
Details 
Reviewed-on: infra/paperspec#2
 2026-10-15 09:53:05 -04:00
alexmandClaude ad65fcc014 Merge branch 'main' into dev
check / check (pull_request) Successful in 51s
Details 
main holds one commit beyond the initial one, 4fa326d "First implementation
(#1)", the squash merge of pull request 1. Its tree is identical to dev's
0dc0c2b (`git diff 4fa326d 0dc0c2b` is empty), which dev already contains,
so main brings no change dev lacks.

The merge uses the `ours` strategy and keeps dev's tree as it is. A
three-way merge against the only shared base, the initial commit 34c0abb,
would count every file of the squash as added by main, and so bring back
tools/oracle/*.py and tools/*.py, which dev has since moved to
src/paperspec/, without a conflict to show it.

With main as a parent, the pull request from dev to main merges cleanly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-15 09:49:46 -04:00
alexm 4fa326dc76 First implementation (#1)
check / check (push) Successful in 42s
Details
site / site (push) Successful in 3m4s
Details 
Reviewed-on: infra/paperspec#1
 2026-10-15 05:47:05 -04:00
122 changed files with 2547 additions and 11643 deletions
Expand all files Collapse all files

No files matched your search

.gitea/workflows/check.yml
-5
View File
@@ -18,9 +18,4 @@ jobs:
- name: Check the repository - name: Check the repository
run: pixi run check run: pixi run check
- name: Stress the reference implementation - name: Stress the reference implementation
<<<<<<< HEAD
run: pixi run stress run: pixi run stress
||||||| parent of 4fa326d (First implementation (#1))
=======
run: pixi run python tools/stress.py
>>>>>>> 4fa326d (First implementation (#1))
.gitignore
-760
View File
@@ -3,7 +3,6 @@ __pycache__/
*.db *.db
*.db.assets/ *.db.assets/
<<<<<<< HEAD
# the package version, written by setuptools-scm when the package is built or installed # the package version, written by setuptools-scm when the package is built or installed
/src/paperspec/_version.py /src/paperspec/_version.py
@@ -763,765 +762,6 @@ FodyWeavers.xsd
# End of https://www.toptal.com/developers/gitignore/api/osx,python,pycharm,windows,visualstudio,visualstudiocode,zsh,jetbrains,jupyternotebooks # End of https://www.toptal.com/developers/gitignore/api/osx,python,pycharm,windows,visualstudio,visualstudiocode,zsh,jetbrains,jupyternotebooks
# The Visual Studio rules above ignore every packages/ and log/ folder; these hold fixtures that `paperspec check` # The Visual Studio rules above ignore every packages/ and log/ folder; these hold fixtures that `paperspec check`
||||||| parent of 4fa326d (First implementation (#1))
=======
# the website, built by tools/site.py
/_site/
# pixi environments
.pixi/*
!.pixi/config.toml
# Created by https://www.toptal.com/developers/gitignore/api/osx,python,pycharm,windows,visualstudio,visualstudiocode,zsh,jetbrains,jupyternotebooks
# Edit at https://www.toptal.com/developers/gitignore?templates=osx,python,pycharm,windows,visualstudio,visualstudiocode,zsh,jetbrains,jupyternotebooks
### JetBrains ###
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
# User-specific stuff
.idea/**/workspace.xml
.idea/**/tasks.xml
.idea/**/usage.statistics.xml
.idea/**/dictionaries
.idea/**/shelf
# AWS User-specific
.idea/**/aws.xml
# Generated files
.idea/**/contentModel.xml
# Sensitive or high-churn files
.idea/**/dataSources/
.idea/**/dataSources.ids
.idea/**/dataSources.local.xml
.idea/**/sqlDataSources.xml
.idea/**/dynamic.xml
.idea/**/uiDesigner.xml
.idea/**/dbnavigator.xml
# Gradle
.idea/**/gradle.xml
.idea/**/libraries
# CMake
cmake-build-*/
# Mongo Explorer plugin
.idea/**/mongoSettings.xml
# File-based project format
*.iws
# IntelliJ
out/
# mpeltonen/sbt-idea plugin
.idea_modules/
# JIRA plugin
atlassian-ide-plugin.xml
# Cursive Clojure plugin
.idea/replstate.xml
# SonarLint plugin
.idea/sonarlint/
# Crashlytics plugin (for Android Studio and IntelliJ)
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
fabric.properties
# Editor-based Rest Client
.idea/httpRequests
# Android studio 3.1+ serialized cache file
.idea/caches/build_file_checksums.ser
### JetBrains Patch ###
# Sonarlint plugin
# https://plugins.jetbrains.com/plugin/7973-sonarlint
.idea/**/sonarlint/
# SonarQube Plugin
# https://plugins.jetbrains.com/plugin/7238-sonarqube-community-plugin
.idea/**/sonarIssues.xml
# Markdown Navigator plugin
# https://plugins.jetbrains.com/plugin/7896-markdown-navigator-enhanced
.idea/**/markdown-navigator.xml
.idea/**/markdown-navigator-enh.xml
.idea/**/markdown-navigator/
# Cache file creation bug
# See https://youtrack.jetbrains.com/issue/JBR-2257
.idea/$CACHE_FILE$
# CodeStream plugin
# https://plugins.jetbrains.com/plugin/12206-codestream
.idea/codestream.xml
# Azure Toolkit for IntelliJ plugin
# https://plugins.jetbrains.com/plugin/8053-azure-toolkit-for-intellij
.idea/**/azureSettings.xml
### JupyterNotebooks ###
# gitignore template for Jupyter Notebooks
# website: http://jupyter.org/
.ipynb_checkpoints
*/.ipynb_checkpoints/*
# IPython
profile_default/
ipython_config.py
### OSX ###
# General
.DS_Store
.AppleDouble
.LSOverride
# Icon must end with two \r
Icon
# Thumbnails
._*
# Files that might appear in the root of a volume
.DocumentRevisions-V100
.fseventsd
.Spotlight-V100
.TemporaryItems
.Trashes
.VolumeIcon.icns
.com.apple.timemachine.donotpresent
# Directories potentially created on remote AFP share
.AppleDB
.AppleDesktop
Network Trash Folder
Temporary Items
.apdisk
### Python ###
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
.claude
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
### Python Patch ###
# Poetry local configuration file - https://python-poetry.org/docs/configuration/#local-configuration
poetry.toml
# ruff
.ruff_cache/
# LSP config files
pyrightconfig.json
### VisualStudioCode ###
.vscode/*
!.vscode/settings.json
!.vscode/tasks.json
!.vscode/launch.json
!.vscode/extensions.json
!.vscode/*.code-snippets
# Local History for Visual Studio Code
.history/
# Built Visual Studio Code Extensions
*.vsix
### VisualStudioCode Patch ###
# Ignore all local history of files
.history
.ionide
### Windows ###
# Windows thumbnail cache files
Thumbs.db
Thumbs.db:encryptable
ehthumbs.db
ehthumbs_vista.db
# Dump file
*.stackdump
# Folder config file
[Dd]esktop.ini
# Recycle Bin used on file shares
$RECYCLE.BIN/
# Windows Installer files
*.cab
*.msi
*.msix
*.msm
*.msp
# Windows shortcuts
*.lnk
### Zsh ###
# Zsh compiled script + zrecompile backup
*.zwc
*.zwc.old
# Zsh completion-optimization dumpfile
*zcompdump*
# Zsh history
.zsh_history
# Zsh sessions
.zsh_sessions
# Zsh zcalc history
.zcalc_history
# A popular plugin manager's files
._zinit
.zinit_lstupd
# zdharma/zshelldoc tool's files
zsdoc/data
# robbyrussell/oh-my-zsh/plugins/per-directory-history plugin's files
# (when set-up to store the history in the local directory)
.directory_history
# MichaelAquilina/zsh-autoswitch-virtualenv plugin's files
# (for Zsh plugins using Python)
# Zunit tests' output
/tests/_output/*
!/tests/_output/.gitkeep
### VisualStudio ###
## Ignore Visual Studio temporary files, build results, and
## files generated by popular Visual Studio add-ons.
##
## Get latest from https://github.com/github/gitignore/blob/main/VisualStudio.gitignore
# User-specific files
*.rsuser
*.suo
*.user
*.userosscache
*.sln.docstates
# User-specific files (MonoDevelop/Xamarin Studio)
*.userprefs
# Mono auto generated files
mono_crash.*
# Build results
[Dd]ebug/
[Dd]ebugPublic/
[Rr]elease/
[Rr]eleases/
x64/
x86/
[Ww][Ii][Nn]32/
[Aa][Rr][Mm]/
[Aa][Rr][Mm]64/
bld/
[Bb]in/
[Oo]bj/
[Ll]og/
[Ll]ogs/
# Visual Studio 2015/2017 cache/options directory
.vs/
# Uncomment if you have tasks that create the project's static files in wwwroot
#wwwroot/
# Visual Studio 2017 auto generated files
Generated\ Files/
# MSTest test Results
[Tt]est[Rr]esult*/
[Bb]uild[Ll]og.*
# NUnit
*.VisualState.xml
TestResult.xml
nunit-*.xml
# Build Results of an ATL Project
[Dd]ebugPS/
[Rr]eleasePS/
dlldata.c
# Benchmark Results
BenchmarkDotNet.Artifacts/
# .NET Core
project.lock.json
project.fragment.lock.json
artifacts/
# ASP.NET Scaffolding
ScaffoldingReadMe.txt
# StyleCop
StyleCopReport.xml
# Files built by Visual Studio
*_i.c
*_p.c
*_h.h
*.ilk
*.meta
*.obj
*.iobj
*.pch
*.pdb
*.ipdb
*.pgc
*.pgd
*.rsp
*.sbr
*.tlb
*.tli
*.tlh
*.tmp
*.tmp_proj
*_wpftmp.csproj
*.tlog
*.vspscc
*.vssscc
.builds
*.pidb
*.svclog
*.scc
# Chutzpah Test files
_Chutzpah*
# Visual C++ cache files
ipch/
*.aps
*.ncb
*.opendb
*.opensdf
*.sdf
*.cachefile
*.VC.db
*.VC.VC.opendb
# Visual Studio profiler
*.psess
*.vsp
*.vspx
*.sap
# Visual Studio Trace Files
*.e2e
# TFS 2012 Local Workspace
$tf/
# Guidance Automation Toolkit
*.gpState
# ReSharper is a .NET coding add-in
_ReSharper*/
*.[Rr]e[Ss]harper
*.DotSettings.user
# TeamCity is a build add-in
_TeamCity*
# DotCover is a Code Coverage Tool
*.dotCover
# AxoCover is a Code Coverage Tool
.axoCover/*
!.axoCover/settings.json
# Coverlet is a free, cross platform Code Coverage Tool
coverage*.json
coverage*.xml
coverage*.info
# Visual Studio code coverage results
*.coverage
*.coveragexml
# NCrunch
_NCrunch_*
.*crunch*.local.xml
nCrunchTemp_*
# MightyMoose
*.mm.*
AutoTest.Net/
# Web workbench (sass)
.sass-cache/
# Installshield output folder
[Ee]xpress/
# DocProject is a documentation generator add-in
DocProject/buildhelp/
DocProject/Help/*.HxT
DocProject/Help/*.HxC
DocProject/Help/*.hhc
DocProject/Help/*.hhk
DocProject/Help/*.hhp
DocProject/Help/Html2
DocProject/Help/html
# Click-Once directory
publish/
# Publish Web Output
*.[Pp]ublish.xml
*.azurePubxml
# Note: Comment the next line if you want to checkin your web deploy settings,
# but database connection strings (with potential passwords) will be unencrypted
*.pubxml
*.publishproj
# Microsoft Azure Web App publish settings. Comment the next line if you want to
# checkin your Azure Web App publish settings, but sensitive information contained
# in these scripts will be unencrypted
PublishScripts/
# NuGet Packages
*.nupkg
# NuGet Symbol Packages
*.snupkg
# The packages folder can be ignored because of Package Restore
**/[Pp]ackages/*
# except build/, which is used as an MSBuild target.
!**/[Pp]ackages/build/
# Uncomment if necessary however generally it will be regenerated when needed
#!**/[Pp]ackages/repositories.config
# NuGet v3's project.json files produces more ignorable files
*.nuget.props
*.nuget.targets
# Microsoft Azure Build Output
csx/
*.build.csdef
# Microsoft Azure Emulator
ecf/
rcf/
# Windows Store app package directories and files
AppPackages/
BundleArtifacts/
Package.StoreAssociation.xml
_pkginfo.txt
*.appx
*.appxbundle
*.appxupload
# Visual Studio cache files
# files ending in .cache can be ignored
*.[Cc]ache
# but keep track of directories ending in .cache
!?*.[Cc]ache/
# Others
ClientBin/
~$*
*~
*.dbmdl
*.dbproj.schemaview
*.jfm
*.pfx
*.publishsettings
orleans.codegen.cs
# Including strong name files can present a security risk
# (https://github.com/github/gitignore/pull/2483#issue-259490424)
#*.snk
# Since there are multiple workflows, uncomment next line to ignore bower_components
# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)
#bower_components/
# RIA/Silverlight projects
Generated_Code/
# Backup & report files from converting an old project file
# to a newer Visual Studio version. Backup files are not needed,
# because we have git ;-)
_UpgradeReport_Files/
Backup*/
UpgradeLog*.XML
UpgradeLog*.htm
ServiceFabricBackup/
*.rptproj.bak
# SQL Server files
*.mdf
*.ldf
*.ndf
# Business Intelligence projects
*.rdl.data
*.bim.layout
*.bim_*.settings
*.rptproj.rsuser
*- [Bb]ackup.rdl
*- [Bb]ackup ([0-9]).rdl
*- [Bb]ackup ([0-9][0-9]).rdl
# Microsoft Fakes
FakesAssemblies/
# GhostDoc plugin setting file
*.GhostDoc.xml
# Node.js Tools for Visual Studio
.ntvs_analysis.dat
node_modules/
# Visual Studio 6 build log
*.plg
# Visual Studio 6 workspace options file
*.opt
# Visual Studio 6 auto-generated workspace file (contains which files were open etc.)
*.vbw
# Visual Studio 6 auto-generated project file (contains which files were open etc.)
*.vbp
# Visual Studio 6 workspace and project file (working project files containing files to include in project)
*.dsw
*.dsp
# Visual Studio 6 technical files
# Visual Studio LightSwitch build output
**/*.HTMLClient/GeneratedArtifacts
**/*.DesktopClient/GeneratedArtifacts
**/*.DesktopClient/ModelManifest.xml
**/*.Server/GeneratedArtifacts
**/*.Server/ModelManifest.xml
_Pvt_Extensions
# Paket dependency manager
.paket/paket.exe
paket-files/
# FAKE - F# Make
.fake/
# CodeRush personal settings
.cr/personal
# Python Tools for Visual Studio (PTVS)
*.pyc
# Cake - Uncomment if you are using it
# tools/**
# !tools/packages.config
# Tabs Studio
*.tss
# Telerik's JustMock configuration file
*.jmconfig
# BizTalk build output
*.btp.cs
*.btm.cs
*.odx.cs
*.xsd.cs
# OpenCover UI analysis results
OpenCover/
# Azure Stream Analytics local run output
ASALocalRun/
# MSBuild Binary and Structured Log
*.binlog
# NVidia Nsight GPU debugger configuration file
*.nvuser
# MFractors (Xamarin productivity tool) working folder
.mfractor/
# Local History for Visual Studio
.localhistory/
# Visual Studio History (VSHistory) files
.vshistory/
# BeatPulse healthcheck temp database
healthchecksdb
# Backup folder for Package Reference Convert tool in Visual Studio 2017
MigrationBackup/
# Ionide (cross platform F# VS Code tools) working folder
.ionide/
# Fody - auto-generated XML schema
FodyWeavers.xsd
# VS Code files for those working on multiple tools
*.code-workspace
# Local History for Visual Studio Code
# Windows Installer files from build outputs
# JetBrains Rider
*.sln.iml
### VisualStudio Patch ###
# Additional files built by Visual Studio
# End of https://www.toptal.com/developers/gitignore/api/osx,python,pycharm,windows,visualstudio,visualstudiocode,zsh,jetbrains,jupyternotebooks
# The Visual Studio rules above ignore every packages/ and log/ folder; these hold fixtures that tools/check.py
>>>>>>> 4fa326d (First implementation (#1))
# compares byte for byte, so keep them. # compares byte for byte, so keep them.
!/fixtures/packages/* !/fixtures/packages/*
!/fixtures/records/log/ !/fixtures/records/log/
CHANGELOG.md
+68 -5
View File
@@ -7,7 +7,7 @@ A change to any computed value is a breaking change and needs a new major versio
The first release. The first release.
- The specification, in chapters 0 to 23, covering every format: 32 formats, of which 19 are `candidate` and 13 `provisional`. - The specification, in chapters 0 to 23, covering every format: 32 formats, of which 19 are `candidate` and 13 `provisional`.
- 42 JSON Schemas, using a keyword subset that independent validators read alike. - 43 JSON Schemas, using a keyword subset that independent validators read alike.
- Tables for SQLite, Dolt and PostgreSQL, a team's store, and the SQLite bundle with full-text search. - Tables for SQLite, Dolt and PostgreSQL, a team's store, and the SQLite bundle with full-text search.
- The JSON-LD context for layers, and registries of formats, rules, download-stamp rules and default stage requirements. - The JSON-LD context for layers, and registries of formats, rules, download-stamp rules and default stage requirements.
- Test vectors for every computed value, from canonical JSON to anchor resolution, grading and the target set of targeted F3. - Test vectors for every computed value, from canonical JSON to anchor resolution, grading and the target set of targeted F3.
@@ -19,7 +19,6 @@ The first release.
- Guides for newcomers: an overview, the concepts, a walkthrough of the example paper, runnable examples, the rationale, a FAQ and a glossary. - Guides for newcomers: an overview, the concepts, a walkthrough of the example paper, runnable examples, the rationale, a FAQ and a glossary.
- Tools that build and check everything, validate packages and zipped packages and run proposed corrections through the gate, load packages into SQLite or a bundle, and stress the reference implementation with random inputs. - Tools that build and check everything, validate packages and zipped packages and run proposed corrections through the gate, load packages into SQLite or a bundle, and stress the reference implementation with random inputs.
- A pixi workspace pinned to Python 3.14, with tasks for checking, building, linting, validating and loading. - A pixi workspace pinned to Python 3.14, with tasks for checking, building, linting, validating and loading.
<<<<<<< HEAD
### 0.1.1 ### 0.1.1
@@ -36,6 +35,70 @@ Packaging, commands and lazy imports only; the formats stay at 0.1.0, and no com
- `src/` is Apache-2.0, as `tools/` was, and the package metadata states `Apache-2.0 AND CC0-1.0` for the code and the data it carries. - `src/` is Apache-2.0, as `tools/` was, and the package metadata states `Apache-2.0 AND CC0-1.0` for the code and the data it carries.
- The pixi workspace installs the package editable, and a `dev` environment builds the sdist and the wheel with `pixi run -e dev dist`. - The pixi workspace installs the package editable, and a `dev` environment builds the sdist and the wheel with `pixi run -e dev dist`.
- Pushing a version tag publishes the package to the Gitea package registry through `.gitea/workflows/publish.yml` ([deploy/](deploy/README.md#publish-the-python-package)); a release tag versions the package, and [CONTRIBUTING.md](CONTRIBUTING.md#releasing) says how that relates to the format version. - Pushing a version tag publishes the package to the Gitea package registry through `.gitea/workflows/publish.yml` ([deploy/](deploy/README.md#publish-the-python-package)); a release tag versions the package, and [CONTRIBUTING.md](CONTRIBUTING.md#releasing) says how that relates to the format version.
||||||| parent of 4fa326d (First implementation (#1))
======= ### Unreleased
>>>>>>> 4fa326d (First implementation (#1))
Two changes asked for by the first producer that compares a converted reading with a second witness (paperform with a JATS file).
Both are minor changes in the sense of [spec/00](spec/00-introduction.md): no valid file becomes invalid, and no hash, ID or key changes; the next release numbers them.
- Alignment has a substitution step ([spec/05 §5.6](spec/05-hashes.md#56-alignment), step 3): in each stretch between pairs, an unpaired sentence of the first reading is paired with the first unpaired sentence of the second whose similarity (2*L* / (|*a*| + |*b*|) over code points of the lossy projections, *L* their longest common subsequence) is at least 0.8, both of prose length 20 or more.
A sentence one reading misread is now one `number`, `equation` or `word` suspect rather than a `dropped` and an `added` one, so a producer re-aligning the same two readings can get different suspects than before; the pairs and suspects of readings with no such sentence are unchanged, as the first alignment vector is.
`paperspec.align` gains `lcs_length` (bit-parallel, Hyyrö 2004) and `similarity`, and `alignment.jsonl` a second case, `substitutions`: a misread number, a misread word, a sentence too short to be paired this way and two sentences too unlike to be one.
- A suspect may name the witness its second reading came from (`witness`, a witness ID; [spec/05 §5.7](spec/05-hashes.md#57-suspects), `suspect.schema.json`), and a package whose suspect names a witness it does not have breaks `doc.suspect.witness`.
No broken package exercises the rule yet: adding one changes the count of broken packages that the 0.1.0 entry above states, so it waits for the release.
Four more, asked for by editors who correct and discuss papers rather than readings.
They are minor changes too: new operations, enumeration values and optional properties, with no valid file made invalid, no value of an existing file computed differently, and no existing vector case or package changed.
- Annotation categories ([spec/18 §18.4](spec/18-layers.md#184-bodies), `annotation.schema.json`, the `category` term of the JSON-LD context): a `comment`, `caveat` or `question` body may carry a `category` from the fixed list `comment`, `context`, `critique`, `question`, `correction`, `feedback` and `resource`, which a producer that supports categories SHOULD set on every such body.
The store's `annotations` table gains a nullable `category` column with its CHECK ([sql/store.sql](sql/store.sql)).
The example layer's comment, question and reply gain categories, which changes their record IDs, and it gains a caveat; no other fixture changes.
- `emend`, a correction of the authors' own typo, spelling, grammar, punctuation or wording, where the reading matches the page and the page is wrong ([spec/11 §11.6](spec/11-corrections.md#116-emendations)).
The emended text keeps what was printed in a new optional run property, `printed` ([spec/03 §3.5](spec/03-nodes.md#35-inline-runs), `runs.schema.json`), so a leaf as printed is recovered from the package alone ([3.6](spec/03-nodes.md#36-the-plain-projection), `paperspec.hashing.printed` and `as_printed`), and runs with `printed` are never joined.
The strict and lossy projections are of the runs as printed, so an emend changes the leaf's `text_hash`, `own_hash` and `node_hash` but not its `match_hash` or `align_key`, and alignment, corroboration and suspects never report an emendation ([spec/05 §5.1](spec/05-hashes.md#51-the-five-values)).
The gate refuses an emend in math, across runs, or that changes a digit or edits beside one, and refuses a `replace`, text `insert` or `split` inside an emended run (`correction.emend`).
- `relink` makes a run a citation of other references, a text run a cite run, or a cite run text, without changing its text, and `resolve` sets a reference's resolution in `references.jsonl`, keeping the one before in `from` ([spec/11 §11.7](spec/11-corrections.md#117-citations-and-references)); a package whose last resolve of a reference is not its resolution breaks `doc.correction.resolution`.
Like an emend, neither lowers a leaf's level, and a receipt keeps counting across all three ([spec/08 §8.1–8.2](spec/08-fidelity.md#82-levels), `receipts.carried_hashes`).
- The rules `correction.emend`, `correction.relink`, `correction.resolve` and `doc.correction.resolution` are registered, the `corrections` table's CHECK takes the three operations ([sql/doc.sql](sql/doc.sql)), `corrections.State` carries the reference records, and `paperspec stress` walks the three operations with their invariants.
`corrections.jsonl` gains thirteen cases and `projections.jsonl` and `projection-groups.jsonl` two each; no existing case changes.
Four more, asked for by papermath, which defines the math form `scienting-math/1`, and by paperclaim, the claim store whose claims and symbols layers point at.
The first two are minor changes: an optional property, a derived file and its schema, with no hash, ID or key of an existing file changed.
The last two make some files that were valid invalid, which [spec/00](spec/00-introduction.md) calls breaking; they are made now because no release of `scienting-doc/1` or `scienting-layer/1` has been tagged, and a reference worth changing is changed before one is ([CONTRIBUTING.md](CONTRIBUTING.md#releasing), item 4).
- Math forms ([spec/03 §3.7](spec/03-nodes.md#37-math-forms), [spec/09 §9.4](spec/09-figures-tables-equations.md#94-the-math-form), `node.schema.json`, `runs.schema.json`, `mathForm` in `common.schema.json`): an equation's content and a `math` run may carry `form`, whose one value, `scienting-math/1`, says that the source is canonical TeX; the notation is then `tex`.
A consumer may rely on the source being canonical and idempotent, carrying no label, and on everything else being derived from it.
`form` enters no content value ([spec/05 §5.2](spec/05-hashes.md#52-own-content)): own content leaves it out of math runs (`paperspec.hashing`), an equation's own content never held more than its notation and source, and no projection reads it, so adding `form` to a source already in the form changes no hash, ID or anchor, while respelling a source changes it as any edit does.
Because a complex formula's key compares sources, a producer aligning a reading in a form with another reading respells the other's math first.
`hashes.jsonl` gains a case, `math-form`, whose values `paperspec check` also computes without `form`; the rule `doc.math.form` is registered for validators that implement the form, which the reference validator does not.
- `derived/math.jsonl` ([spec/12 §12.7](spec/12-package.md#127-derived-files), [spec/09 §9.4](spec/09-figures-tables-equations.md#94-the-math-form), `math.schema.json`): one line per formula, `{node, run, tool, form, tree, symbols, complete, issues, label?}`, papermath's reading without the canonical TeX, which is the source; `run` is null for an equation, as in `derived/mathml.jsonl`, and `label` is omitted when the source has no `\tag`.
A tree node is an array whose first item is its kind; the schema takes any lowercase kind and issue code, so the form can add some in a minor version, and lists the 49 kinds and six codes of version 1.
The validator now checks derived files: each line against its schema, that it names a formula of the package (`doc.derived.node`), that a math line's formula is TeX and its `complete` and `symbols` agree with its tree (`doc.derived.math`), and that a checks-only package carries none (`doc.policy.withheld`), since a tree or MathML reads the withheld text.
The main example gains `derived/math.jsonl`, papermath 0.1.0's reading of its equation and of two math runs, copied by hand because the oracle does not implement the form; `paperspec check` validates fourteen variants of it, and no fixture's hashes change.
There are now 43 schemas, which the 0.1.0 entry above, held to the repository by `paperspec check`, states too.
- Claim references ([spec/18 §18.4](spec/18-layers.md#184-bodies), [spec/02 §2.4](spec/02-model.md#24-work-and-document-keys), [spec/20](spec/20-projects.md), `claimRef` and `claimPin` in `common.schema.json`): the `source` of a `claim-link` or `symbol` body, a `claim:` identifier in `same_as`, and a project's claim step are paperclaim references, `claim:<tag>` or `claim:<host>/<tag>` with the tag eight Crockford base32 capitals, and a `pin` is the reference, `@` and six lowercase hex digits of the claim's statement hash, as paperclaim writes them; a project names its claim namespace as `claims.paperclaim`.
A reference by slug, such as `amber/k-bond`, which paperclaim may rename, or under any other prefix, was valid before and is not now.
The fixtures' slug reference and `illustrative-pin` become `claim:272XTX7C` and `claim:272XTX7C@02f47b`, the claim `amber/k-bond` of paperclaim's example store, which changes the claim link's and the symbol's record IDs, the layer receipt and `project.json`.
The main example's two targeted receipts take the scope `targeted:paperclaim`, which changes their IDs and signatures, Ben's spot-check of the second (its seed and its sample of ten, still without errors), the QA report, the inclusion proof, the transparency log's checkpoint and tiles, and the signature, spot-check, VRF and checkpoint vectors built from them; no node's level or the report's counts change.
- Symbol keys ([spec/18 §18.4](spec/18-layers.md#184-bodies), `annotation.schema.json`, the `key` term of the JSON-LD context, `sct:symbolKey`): a `symbol` body names its symbol by `key`, its `scienting-math/1` key, besides the span its target selects, so a symbol is found across papers and matched to its claim by key; a `symbol` body without `key` was valid before and is not now.
The example layer gains a proposed symbol annotation on `$K_r$`, with the key `K_{r}`.
- Three failures of `paperspec stress` with the seeds 1, 2 and 7, in code the earlier changes did not touch.
The zip reader let `zipfile`'s own errors escape on an archive it cannot read, such as `NotImplementedError` for an entry that needs zip version 9.8; `paperspec.package.read_zip` now refuses every such archive with `package.zip.entry` ([spec/12 §12.2](spec/12-package.md#122-paths)), and `paperspec check` crafts one of each kind.
A checkpoint note was split into lines with `str.splitlines`, which also breaks at U+000B and other separators and so accepted a note whose last newline was replaced by one; [spec/23 §23.2](spec/23-log.md#232-checkpoints) now says every line ends in U+000A alone, `paperspec.log.parse_checkpoint` splits on it, and `paperspec check` tries ten other separators.
The third was the stress harness's own: a truncation could cut a file at its full length, leaving the package unchanged and valid, so it now always shortens the file.
So was a fourth, with 1000 rounds: the random walk of corrections could delete the body section, leaving nothing to correct and too few corrections accepted, so a delete now leaves at least eight nodes.
One more, asked for by paperdesk, which lets a reader follow a citation to the passage it rests on.
It is a minor change in the sense of [spec/00](spec/00-introduction.md): a new body kind, optional properties and two rules, with no valid file made invalid, no computed value changed and no existing vector case or package changed.
- Traced citations ([spec/18 §18.5](spec/18-layers.md#185-traced-citations), `annotation.schema.json`, the `note` and `proposer` terms of the JSON-LD context): a `trace` body follows one citation of a sentence of a citing paper to a passage of the cited paper.
The target is a leaf or span anchor on the citing sentence, which must cite the body's `reference`; the body names the cited document in `source`, the passage in `selector` (an `AnchorSelector` with its pin and span, then a `TextQuoteSelector` when the quote is plain), and optionally a `note`, a `verdict` and a `proposer` (`{model, run, score?, rank?}`, required when the creator is `Software`).
A `crosslink` stays what it was, a symmetric correspondence of two spans with no room for a note, the reference meant or a proposer's provenance.
The state of a trace is the annotation's `status`, and a trace is disputed while an `assessing` reply to it stands, which is computed from the replies.
The `body_kind` CHECK of the store's `annotations` table takes `trace` ([sql/store.sql](sql/store.sql)), and `paperspec check` holds it to the schema's list of body kinds.
- `paperspec.traces` and the rules `layer.trace.source` and `layer.trace.target` ([spec/22 §22.4](spec/22-conformance.md#224-rules)).
`resolve(annotation, citing_state, cited_state, same_as)` resolves both anchors by [spec/06 §6.5](spec/06-anchors.md#65-resolution) and returns `current`, `changed`, `stale` (with the side), `unlinked` (a `relink` removed the citation) or `misresolved` (a `resolve` points the reference at another work), with `open`, the passage's current address, and `cites`, the citations inside it, so a chain of citations is followed one trace at a time.
`problems` reports the two rules, and `disputes` and `disputed` read the replies.
- `fixtures/records/traces/` is a second layer, on the Typst draft, with a model's confirmed trace to the main example's abstract and its receipt, a rejected candidate, a person's trace to Table 1 and a reply that disputes the first, and `vectors/traces.jsonl` has 29 cases that correct either paper and show every outcome and both rules.
`paperspec check` runs both layers through the record checks and the store.
The example project holds the new layer, which changes `project.json`; no other fixture changes.
CONTRIBUTING.md
-48
View File
@@ -13,7 +13,6 @@ The schemas, SQL, registries, vectors and fixtures then follow the chapter ([spe
2. **Change the schemas, SQL and registries** that the rule shapes. 2. **Change the schemas, SQL and registries** that the rule shapes.
A new refusal gets a rule ID in [registry/rules.json](registry/rules.json). A new refusal gets a rule ID in [registry/rules.json](registry/rules.json).
3. **Compute, never type, the expected values.** 3. **Compute, never type, the expected values.**
<<<<<<< HEAD
Change the oracle in [src/paperspec/](src/paperspec/) to apply the rule, and add the case that shows it to [src/paperspec/cli/build.py](src/paperspec/cli/build.py): a vector, a fixture or a broken package in `fixtures/invalid/`. Change the oracle in [src/paperspec/](src/paperspec/) to apply the rule, and add the case that shows it to [src/paperspec/cli/build.py](src/paperspec/cli/build.py): a vector, a fixture or a broken package in `fixtures/invalid/`.
Then run `pixi run build`, which rewrites `fixtures/` and `vectors/`. Then run `pixi run build`, which rewrites `fixtures/` and `vectors/`.
4. **Run the checks.** 4. **Run the checks.**
@@ -61,50 +60,3 @@ The keys in `fixtures/keys/` are public test keys and must never sign anything e
## Licenses ## Licenses
Each path keeps the license that [LICENSE.md](LICENSE.md) gives it: prose CC-BY-4.0, the normative data CC0-1.0, and the code in `src/` and `tools/` Apache-2.0. Each path keeps the license that [LICENSE.md](LICENSE.md) gives it: prose CC-BY-4.0, the normative data CC0-1.0, and the code in `src/` and `tools/` Apache-2.0.
||||||| parent of 4fa326d (First implementation (#1))
=======
Change the oracle in [tools/oracle/](tools/oracle/) to apply the rule, and add the case that shows it to [tools/build.py](tools/build.py): a vector, a fixture or a broken package in `fixtures/invalid/`.
Then run `pixi run build`, which rewrites `fixtures/` and `vectors/`.
4. **Run the checks.**
`pixi run check` must end with `All checks pass.`
Without pixi, run `python3 tools/check.py`, with Python 3.14 and two packages, `jsonschema` and `cryptography` installed.
5. **Record the change** in [CHANGELOG.md](CHANGELOG.md).
## Versions
A change that makes a valid file invalid, or that changes a computed value, is a breaking change and needs a new major version of the format, with a migration.
Adding an optional property, a node kind, a run kind or an enumeration value is a minor change ([spec/00 §0.3](spec/00-introduction.md#03-format-identifiers-and-versions)).
A rebuild that changes any file under `vectors/` or any hash in `fixtures/` is a computed value changing, so say in the pull request why it must.
## New node kinds and run kinds
Propose a kind with the vector or fixture that shows it.
A node kind needs its row in [chapter 3](spec/03-nodes.md), the node schema, and its entry in the `CHILDREN` table of `tools/oracle/model.py`.
A run kind needs its row in [section 3.5](spec/03-nodes.md#35-inline-runs) and the node schema, and its text in the plain projection ([3.6](spec/03-nodes.md#36-the-plain-projection)) and in the strict and lossy projections ([5.3](spec/05-hashes.md#53-the-strict-projection), [5.5](spec/05-hashes.md#55-the-lossy-projection)).
## Releasing
A release is a tag on `main` whose version is the top heading of [CHANGELOG.md](CHANGELOG.md).
Before tagging:
1. `pixi run check` passes on Linux and macOS, which the workflow in `.gitea/workflows/` runs on every push.
The check also confirms that every count the prose states (formats, schemas, vectors, broken packages) matches the repository, and that `pixi.toml`, the registries and `tools/build.py` state the same version.
2. `python3 tools/stress.py` passes with its default seed, and with one or two others.
3. The website is published from the release ([deploy/](deploy/README.md)), and the JSON-LD context at `https://paperspec.scient.ing/context/scienting-1.jsonld` and a schema at its `$id` both answer with their media types.
Layers claim to be W3C Web Annotations, and a JSON-LD processor that cannot fetch the context cannot read them.
4. The decisions that freeze on release are made: enumeration values and format identifiers never change meaning afterwards ([spec/22 §22.5](spec/22-conformance.md#225-stability)), so a name worth changing is changed now.
5. The CHANGELOG entry describes the release in the terms of the chapters it touches.
## Writing
The checker enforces two rules on every Markdown file: one sentence per line, and US spelling.
`pixi run lint` runs markdownlint with the repository's settings.
Define each term where it first appears, because the guides and READMEs are written for newcomers as well as implementers.
All fixture text is synthetic, written for the fixtures.
The keys in `fixtures/keys/` are public test keys and must never sign anything else.
## Licenses
Each path keeps the license that [LICENSE.md](LICENSE.md) gives it: prose CC-BY-4.0, the normative data CC0-1.0, and `tools/` Apache-2.0.
>>>>>>> 4fa326d (First implementation (#1))
LICENSE.md
-5
View File
@@ -4,12 +4,7 @@
| --- | --- | --- | | --- | --- | --- |
| `spec/`, `docs/`, `README.md`, `CONTRIBUTING.md`, `CHANGELOG.md` and the other prose | Creative Commons Attribution 4.0 International | `CC-BY-4.0` | | `spec/`, `docs/`, `README.md`, `CONTRIBUTING.md`, `CHANGELOG.md` and the other prose | Creative Commons Attribution 4.0 International | `CC-BY-4.0` |
| `schemas/`, `sql/`, `context/`, `registry/`, `vectors/`, `fixtures/` | Creative Commons Zero 1.0 Universal | `CC0-1.0` | | `schemas/`, `sql/`, `context/`, `registry/`, `vectors/`, `fixtures/` | Creative Commons Zero 1.0 Universal | `CC0-1.0` |
<<<<<<< HEAD
| `src/` (the `paperspec` package and its commands), `tools/` | Apache License 2.0 | `Apache-2.0` | | `src/` (the `paperspec` package and its commands), `tools/` | Apache License 2.0 | `Apache-2.0` |
||||||| parent of 4fa326d (First implementation (#1))
=======
| `tools/` | Apache License 2.0 | `Apache-2.0` |
>>>>>>> 4fa326d (First implementation (#1))
The legal texts are at <https://creativecommons.org/licenses/by/4.0/legalcode>, <https://creativecommons.org/publicdomain/zero/1.0/legalcode> and <https://www.apache.org/licenses/LICENSE-2.0>. The legal texts are at <https://creativecommons.org/licenses/by/4.0/legalcode>, <https://creativecommons.org/publicdomain/zero/1.0/legalcode> and <https://www.apache.org/licenses/LICENSE-2.0>.
README.md
-37
View File
@@ -61,7 +61,6 @@ The address grammar is in [spec/06 §6.1](spec/06-anchors.md#61-addresses), and
| [registry/](registry/README.md) | The list of formats, the rule IDs validators report, download-stamp rules, and the default rights each processing stage requires. | | [registry/](registry/README.md) | The list of formats, the rule IDs validators report, download-stamp rules, and the default rights each processing stage requires. |
| [vectors/](vectors/README.md) | Test vectors: inputs and the exact outputs every implementation must compute. | | [vectors/](vectors/README.md) | Test vectors: inputs and the exact outputs every implementation must compute. |
| [fixtures/](fixtures/README.md) | Example packages and records, deliberately broken packages, and the test keys that signed them. | | [fixtures/](fixtures/README.md) | Example packages and records, deliberately broken packages, and the test keys that signed them. |
<<<<<<< HEAD
| [tools/](tools/README.md) | How to run the commands that build and check everything above. | | [tools/](tools/README.md) | How to run the commands that build and check everything above. |
| [src/paperspec/](src/paperspec/) | The reference implementation of every rule and those commands, installable as the Python package `paperspec`. | | [src/paperspec/](src/paperspec/) | The reference implementation of every rule and those commands, installable as the Python package `paperspec`. |
| [docs/](docs/README.md) | Guides for people getting to know the project. | | [docs/](docs/README.md) | Guides for people getting to know the project. |
@@ -87,34 +86,6 @@ The run ends with `All checks pass.`
The commands need Python 3.14 and the package's `tools` extra (`jsonschema`, `referencing` and `cryptography`), which pixi installs along with markdownlint for the Markdown. The commands need Python 3.14 and the package's `tools` extra (`jsonschema`, `referencing` and `cryptography`), which pixi installs along with markdownlint for the Markdown.
Without pixi, install the checkout as the package with `pip install -e ".[tools]"` on Python 3.14, then run `paperspec check`. Without pixi, install the checkout as the package with `pip install -e ".[tools]"` on Python 3.14, then run `paperspec check`.
`paperspec --help` lists the commands, and each command's `--help` gives advice and examples ([tools/](tools/README.md)). `paperspec --help` lists the commands, and each command's `--help` gives advice and examples ([tools/](tools/README.md)).
||||||| parent of 4fa326d (First implementation (#1))
Version 0.1.0.
The formats of milestone M1 are `candidate`; datasets, digitizations, views, projects, bundles and the draft formats are `provisional` until their milestones; see [registry/formats.json](registry/formats.json).
=======
| [tools/](tools/README.md) | The programs that build and check everything above. |
| [docs/](docs/README.md) | Guides for people getting to know the project. |
| [deploy/](deploy/README.md) | How the website at <https://paperspec.scient.ing> is built and served. |
## Try it
With [pixi](https://pixi.sh), one command sets up a pinned environment and checks everything:
```sh
pixi run check
```
The checker validates every schema, record and package.
It confirms that every broken package is refused for the stated reason.
It recomputes every test vector, and it rebuilds everything and compares it byte for byte.
Last, it loads every package into SQLite and back, and checks the prose and the links.
Pixi lints the Markdown first.
The run ends with `All checks pass.`
`pixi task list` shows the other tasks: `quick`, `build`, `lint`, `load`, `validate` and `stress`.
`pixi run python tools/site.py` builds the website into `_site/` ([deploy/](deploy/README.md)).
The tools need Python 3.14 and two packages, `jsonschema` and `cryptography`, which pixi installs along with markdownlint for the Markdown.
Without pixi, install those two with pip and run `python3 tools/check.py`.
>>>>>>> 4fa326d (First implementation (#1))
To explore a worked package, load the example paper (a synthetic chemistry paper with 12 corrections) into SQLite and query it; [docs/examples.md](docs/examples.md) has queries to start from. To explore a worked package, load the example paper (a synthetic chemistry paper with 12 corrections) into SQLite and query it; [docs/examples.md](docs/examples.md) has queries to start from.
@@ -162,13 +133,5 @@ Table columns and figure-text items have IDs and are not nodes ([spec/04 §4.2](
The specification and other prose are CC BY 4.0, so anyone may implement the formats. The specification and other prose are CC BY 4.0, so anyone may implement the formats.
Schemas, SQL, the JSON-LD context, the registries, the vectors and the fixtures are CC0 1.0. Schemas, SQL, the JSON-LD context, the registries, the vectors and the fixtures are CC0 1.0.
<<<<<<< HEAD
The code, the `paperspec` package in `src/` with its commands, is Apache-2.0. The code, the `paperspec` package in `src/` with its commands, is Apache-2.0.
||||||| parent of 4fa326d (First implementation (#1))
The chapters and other prose are licensed CC BY 4.0.
Schemas, SQL, the JSON-LD context, the registries, the vectors and the fixtures are dedicated to the public domain under CC0 1.0, so anyone may implement the formats.
The tools are licensed Apache-2.0.
=======
The tools are Apache-2.0.
>>>>>>> 4fa326d (First implementation (#1))
See [LICENSE.md](LICENSE.md). See [LICENSE.md](LICENSE.md).
context/README.md
+15 -17
View File
@@ -33,7 +33,7 @@ The repository's name, paperspec, is a separate matter.
## An example from the fixtures ## An example from the fixtures
[`fixtures/records/layer/annotations.jsonl`](../fixtures/records/layer/annotations.jsonl) holds nine annotations on the example paper `hale2026harmonic-damb`. [`fixtures/records/layer/annotations.jsonl`](../fixtures/records/layer/annotations.jsonl) holds twelve annotations on the example paper `hale2026harmonic-damb`.
The first is a `gloss`, an explanation for a novice, of the phrase "bond force constant". The first is a `gloss`, an explanation for a novice, of the phrase "bond force constant".
Shortened, it reads: Shortened, it reads:
@@ -57,14 +57,14 @@ This context defines `kind`, `status`, `audience`, `AnchorSelector` and `anchor`
## What it defines ## What it defines
The file defines 23 terms. The file defines 27 terms.
Twenty-two map to IRIs under `https://paperspec.scient.ing/vocab#`: `format`, `kind`, `status`, `receipt`, `layer`, `audience`, `visibility`, `purpose`, `project`, `docs`, `pin`, `role`, `symbol`, `verdict`, `reason`, `reference`, `for`, `generator`, `exercise`, `anchor`, `rights_use` and the class `AnchorSelector`. Twenty-six map to IRIs under `https://paperspec.scient.ing/vocab#`: `format`, `kind`, `status`, `receipt`, `layer`, `audience`, `visibility`, `purpose`, `project`, `docs`, `pin`, `role`, `symbol`, `key`, `verdict`, `reason`, `reference`, `note`, `proposer`, `for`, `category`, `generator`, `exercise`, `anchor`, `rights_use` and the class `AnchorSelector`.
Three are renamed: `for` is `sct:questionFor`, `generator` is `sct:generatedBy`, and `rights_use` is `sct:rightsUse`. Five are renamed: `key`, the key of a `symbol` body's symbol, is `sct:symbolKey`, `for` is `sct:questionFor`, `generator` is `sct:generatedBy`, `proposer`, the tool that found a trace, is `sct:proposedBy`, and `rights_use` is `sct:rightsUse`.
The twenty-third, `total`, maps to `totalItems` in the ActivityStreams namespace. The twenty-seventh, `total`, maps to `totalItems` in the ActivityStreams namespace.
Two terms carry a type rule. Two terms carry a type rule.
A processor reads `layer` as an IRI, and it always reads `docs` as a set, so a one-element list stays a list. A processor reads `layer` as an IRI, and it always reads `docs` as a set, so a one-element list stays a list.
The file declares `anchor` and `exercise` as JSON literals (`"@type": "@json"`, a JSON-LD 1.1 feature, which is why the file sets `"@version": 1.1`). The file declares `anchor`, `exercise` and `proposer` as JSON literals (`"@type": "@json"`, a JSON-LD 1.1 feature, which is why the file sets `"@version": 1.1`).
A JSON-LD processor then keeps a `scienting-anchor/1` record intact instead of expanding it. A JSON-LD processor then keeps a `scienting-anchor/1` record intact instead of expanding it.
An anchor holds hashes (fixed-length fingerprints of text) and a `span`, a pair of character offsets whose order carries meaning. An anchor holds hashes (fixed-length fingerprints of text) and a `span`, a pair of character offsets whose order carries meaning.
Expansion would drop the anchor's unmapped field names ([JSON-LD 1.1 API, expansion step 13.3](https://www.w3.org/TR/json-ld11-api/#alg-expand-each-key-value)) and would give its arrays no order ([JSON-LD 1.1 §4.3](https://www.w3.org/TR/json-ld11/#sets-and-lists)). Expansion would drop the anchor's unmapped field names ([JSON-LD 1.1 API, expansion step 13.3](https://www.w3.org/TR/json-ld11-api/#alg-expand-each-key-value)) and would give its arrays no order ([JSON-LD 1.1 §4.3](https://www.w3.org/TR/json-ld11/#sets-and-lists)).
@@ -88,7 +88,14 @@ Such a tool still reads the annotation's body and its `TextQuoteSelector`; the c
That is the repository's statement; no Web Annotation client has been tested against these files. That is the repository's statement; no Web Annotation client has been tested against these files.
The fallback is not always there. The fallback is not always there.
Three of the nine fixture annotations have only an `AnchorSelector` (the section-level question, the `generated` annotation on a figure and the exercise on a table), because the anchor points at a container and has no quote. Three of the eleven fixture annotations have only an `AnchorSelector` (the section-level question, the `generated` annotation on a figure and the exercise on a table), because the anchor points at a container and has no quote.
## A body that points at another paper
A `trace` body ([spec/18 §18.5](../spec/18-layers.md#185-traced-citations)) follows a citation in the annotated sentence to a passage of the cited paper, so its `source` and `selector` name the cited paper, not the annotated one.
[`fixtures/records/traces/annotations.jsonl`](../fixtures/records/traces/annotations.jsonl) holds four annotations on the draft `writer2027fixed-ynr7.v3`, which cites the example paper.
The Web Annotation context defines `source` and `selector` for a body as it does for a target, so a trace's selector list puts the `AnchorSelector` first and the `TextQuoteSelector` after it, as a target's does.
This context adds the rest of the body: `reference`, the reference of the draft that the sentence cites, `note`, `verdict` and `proposer`, which a processor keeps whole, as it keeps an anchor, because its model's hashes and its score are values and not terms.
## Using the context ## Using the context
@@ -96,21 +103,12 @@ Three of the nine fixture annotations have only an `AnchorSelector` (the section
- Reader: Follow [chapter 6](../spec/06-anchors.md) to resolve an anchor. - Reader: Follow [chapter 6](../spec/06-anchors.md) to resolve an anchor.
The file `vectors/resolve.jsonl` resolves seventeen anchors against `fixtures/packages/hale2026harmonic-damb` and gives every outcome. The file `vectors/resolve.jsonl` resolves seventeen anchors against `fixtures/packages/hale2026harmonic-damb` and gives every outcome.
- JSON-LD processor: Use a JSON-LD 1.1 processor; the context relies on `@json` and `@version` 1.1. - JSON-LD processor: Use a JSON-LD 1.1 processor; the context relies on `@json` and `@version` 1.1.
<<<<<<< HEAD
- Checker: `schemas/annotation.schema.json` and `schemas/layer.schema.json` validate the files. - Checker: `schemas/annotation.schema.json` and `schemas/layer.schema.json` validate the files.
`paperspec check` also checks that each annotation's ID derives from its content and that the receipt at `fixtures/records/layer/receipt.json` covers each `confirmed` annotation. `paperspec check` also checks that each annotation's ID derives from its content and that the receipt of its layer, at `fixtures/records/layer/receipt.json` or `fixtures/records/traces/receipt.json`, covers each `confirmed` annotation.
The traps: The traps:
- No test vector pins the context, and nothing here runs a JSON-LD processor on it; `paperspec check` and `paperspec site` read only its term IRIs. - No test vector pins the context, and nothing here runs a JSON-LD processor on it; `paperspec check` and `paperspec site` read only its term IRIs.
||||||| parent of 4fa326d (First implementation (#1))
=======
- Checker: `schemas/annotation.schema.json` and `schemas/layer.schema.json` validate the files. `tools/check.py` also checks that each annotation's ID derives from its content and that the receipt at `fixtures/records/layer/receipt.json` covers each `confirmed` annotation.
The traps:
- No test vector pins the context, and nothing in `tools/` reads `scienting-1.jsonld`.
>>>>>>> 4fa326d (First implementation (#1))
The schemas require only that `@context` be an array of at least two non-empty strings; they do not check the URLs. The schemas require only that `@context` be an array of at least two non-empty strings; they do not check the URLs.
A file with a wrong context can pass the checker and still expand wrongly. A file with a wrong context can pass the checker and still expand wrongly.
- `@context` is part of an annotation's content hash (the hash covers everything except `status`, `receipt` and `modified`). - `@context` is part of an annotation's content hash (the hash covers everything except `status`, `receipt` and `modified`).
context/scienting-1.jsonld
+7
View File
@@ -21,10 +21,17 @@
"pin": "sct:pin", "pin": "sct:pin",
"role": "sct:role", "role": "sct:role",
"symbol": "sct:symbol", "symbol": "sct:symbol",
"key": "sct:symbolKey",
"verdict": "sct:verdict", "verdict": "sct:verdict",
"reason": "sct:reason", "reason": "sct:reason",
"reference": "sct:reference", "reference": "sct:reference",
"note": "sct:note",
"proposer": {
"@id": "sct:proposedBy",
"@type": "@json"
},
"for": "sct:questionFor", "for": "sct:questionFor",
"category": "sct:category",
"generator": "sct:generatedBy", "generator": "sct:generatedBy",
"exercise": { "exercise": {
"@id": "sct:exercise", "@id": "sct:exercise",
deploy/Caddyfile
-5
View File
@@ -1,9 +1,4 @@
<<<<<<< HEAD
# Serves the paperspec website built by `paperspec site` (`pixi run site`, the `_site/` folder), copied to /srv/www/paperspec. # Serves the paperspec website built by `paperspec site` (`pixi run site`, the `_site/` folder), copied to /srv/www/paperspec.
||||||| parent of 4fa326d (First implementation (#1))
=======
# Serves the paperspec website built by `python3 tools/site.py` (the `_site/` folder), copied to /srv/www/paperspec.
>>>>>>> 4fa326d (First implementation (#1))
# The site mirrors the repository, so every schema is served at its $id and the JSON-LD context at the address # The site mirrors the repository, so every schema is served at its $id and the JSON-LD context at the address
# layers name; this file sets the media types those addresses need and lets any page fetch them. # layers name; this file sets the media types those addresses need and lets any page fetch them.
paperspec.scient.ing { paperspec.scient.ing {
deploy/README.md
-59
View File
@@ -1,4 +1,3 @@
<<<<<<< HEAD
# Deploying the website and the Python package # Deploying the website and the Python package
This folder holds what it takes to serve the paperspec website at <https://paperspec.scient.ing>. This folder holds what it takes to serve the paperspec website at <https://paperspec.scient.ing>.
@@ -117,64 +116,6 @@ If the organization or the package is private, the installer also needs a token
A git pin to a commit, as `pixi.toml` and `pyproject.toml` can also express, keeps working and needs no registry. A git pin to a commit, as `pixi.toml` and `pyproject.toml` can also express, keeps working and needs no registry.
## Distributing document packages ## Distributing document packages
||||||| parent of 4fa326d (First implementation (#1))
=======
# Deploying the website
This folder holds what it takes to serve the paperspec website at <https://paperspec.scient.ing>.
The site is static: `tools/site.py` builds it into `_site/`, and any web server can serve that folder.
## What the site serves
The site mirrors the repository, so a file at `schemas/anchor.schema.json` in the repository is served at `https://paperspec.scient.ing/schemas/anchor.schema.json`.
Three addresses that the formats themselves name depend on that:
| Address | What is there |
| --- | --- |
| `https://paperspec.scient.ing/schemas/<file>` | Each JSON Schema, at the address in its `$id`. |
| `https://paperspec.scient.ing/context/scienting-1.jsonld` | The JSON-LD context every layer and annotation lists. |
| `https://paperspec.scient.ing/vocab` | The vocabulary page, whose fragments are the IRIs of the context's terms. |
Beside every Markdown file the builder writes a rendered page: `README.md` becomes its folder's `index.html`, and `X.md` becomes `X.html`.
A folder without a README gets a listing page, so every fixture and vector can be browsed.
The pages need no server-side code; search runs in the browser over a small index of headings, `search.json`.
## Build it
```sh
pixi run python tools/site.py # writes _site/
pixi run python tools/site.py --out /tmp/paperspec-site
```
`pixi run check` builds the site into a temporary folder and checks that every Markdown file has a page, that every internal link and fragment resolves, that no Markdown is left unrendered, and that each schema's `$id`, the context's address and the vocabulary's IRIs are files of the site.
To look at it locally, serve the folder with any static server, for example:
```sh
pixi run python -m http.server --directory _site 8000
```
## Serve it with Caddy
[Caddyfile](Caddyfile) serves `/srv/www/paperspec` at `paperspec.scient.ing`.
It sets the media types the data needs (`application/ld+json` for the context, `application/schema+json` for schemas, `application/jsonl` for vectors and package files), allows every file to be fetched from any origin, and serves `404.html` for missing paths.
Add it to the server's Caddy configuration, for example with an `import` line, and reload Caddy.
## Publish from CI
The workflow `.gitea/workflows/site.yml` runs on every push to `main`.
It installs the environment from `pixi.lock`, runs `pixi run quick`, builds the site, and copies it with `rsync` to `/srv/www/paperspec`, the folder Caddy serves.
A failed check stops the publish, so the site never serves a release that does not pass its own checks.
The copy is local: the `linux-base` runner starts each job in a container with the host's `/srv/www` mounted at the same path, so no key or secret is involved.
On a runner without that mount the workflow still builds and checks the site, and says that it did not publish.
`SITE_ROOT` and `SITE_SLUG` at the top of the workflow name the folder.
Only changed files are copied, so every other file keeps its `Last-Modified` and `ETag` and caches stay valid across releases.
The environment in `.pixi/envs` is cached between runs under the hash of `pixi.lock`, so a fresh container installs it only when the lock file changes.
## Distributing packages
>>>>>>> 4fa326d (First implementation (#1))
A zipped package is an ordinary file, and the site serves the fixtures as such. A zipped package is an ordinary file, and the site serves the fixtures as such.
For a team's own packages, the Gitea instance speaks the OCI distribution protocol, so packages can be pushed to and pulled from its package registry as OCI artifacts with ORAS, and signed there with cosign; the media type of the artifact is the package's, `application/vnd.scienting.doc+zip`. For a team's own packages, the Gitea instance speaks the OCI distribution protocol, so packages can be pushed to and pulled from its package registry as OCI artifacts with ORAS, and signed there with cosign; the media type of the artifact is the package's, `application/vnd.scienting.doc+zip`.
docs/comparison.md
+1 -7
View File
@@ -428,14 +428,8 @@ Keep using it as a reading and distribution format.
*Status.* *Status.*
The specification is version 0.1.0. The specification is version 0.1.0.
`registry/formats.json` lists 32 formats: 19 are `candidate` (complete, with vectors and fixtures, ready to implement) and 13 are `provisional` (complete; expected to change as implementations gain experience): project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint ([spec/00 §0.4](../spec/00-introduction.md#04-status)). `registry/formats.json` lists 32 formats: 19 are `candidate` (complete, with vectors and fixtures, ready to implement) and 13 are `provisional` (complete; expected to change as implementations gain experience): project, bundle, dataset, digitization, view, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint ([spec/00 §0.4](../spec/00-introduction.md#04-status)).
<<<<<<< HEAD The repository holds 24 chapters, 43 JSON Schemas, 26 vector files, five valid fixture packages, one of them also zipped, and 31 invalid ones, each naming the rule it breaks, and a reference checker, `paperspec check`.
The repository holds 24 chapters, 42 JSON Schemas, 25 vector files, five valid fixture packages, one of them also zipped, and 31 invalid ones, each naming the rule it breaks, and a reference checker, `paperspec check`.
Licenses: prose CC BY 4.0; schemas, SQL, the JSON-LD context, registries, vectors and fixtures CC0 1.0; the code in `src/` Apache-2.0 ([README.md](../README.md#licenses)). Licenses: prose CC BY 4.0; schemas, SQL, the JSON-LD context, registries, vectors and fixtures CC0 1.0; the code in `src/` Apache-2.0 ([README.md](../README.md#licenses)).
||||||| parent of 4fa326d (First implementation (#1))
=======
The repository holds 24 chapters, 42 JSON Schemas, 25 vector files, five valid fixture packages, one of them also zipped, and 31 invalid ones, each naming the rule it breaks, and a reference checker, `tools/check.py`.
Licenses: prose CC BY 4.0; schemas, SQL, the JSON-LD context, registries, vectors and fixtures CC0 1.0; tools Apache-2.0 ([README.md](../README.md#licenses)).
>>>>>>> 4fa326d (First implementation (#1))
*No adoption claim.* *No adoption claim.*
The repository names no outside adopter and no independent implementation, and this page claims none. The repository names no outside adopter and no independent implementation, and this page claims none.
docs/concepts.md
+8 -3
View File
@@ -212,7 +212,9 @@ Every fix is a correction: an operation with its evidence, its author and a reas
``` ```
A document's history is its first reading plus its corrections, and its revision is the index of the last one. A document's history is its first reading plus its corrections, and its revision is the index of the last one.
The operations are `replace`, `insert`, `delete`, `split`, `merge`, `move`, `reclassify`, `relabel` and `set`. The operations are `replace`, `insert`, `delete`, `split`, `merge`, `move`, `reclassify`, `relabel`, `set`, `emend`, `relink` and `resolve`.
A `replace` fixes what the converter misread; an `emend` fixes a typo the authors printed, and keeps the printed text, so the sentence still agrees with the PDF and keeps its level.
A `relink` changes which references a citation names, and a `resolve` what a reference resolves to.
Evidence is a crop (an image of the page region), a witness, a source file by SHA-256, or a named person. Evidence is a crop (an image of the page region), a witness, a source file by SHA-256, or a named person.
A gate refuses any correction that would change a digit without a crop of the page as evidence, or that would let a digit change anywhere it did not declare. A gate refuses any correction that would change a digit without a crop of the page as evidence, or that would let a digit change anywhere it did not declare.
@@ -371,10 +373,13 @@ A statement true independently of the paper is a claim, which is checked and lev
A statement about reading the paper belongs in a layer, where it is attributed and signed and not checked. A statement about reading the paper belongs in a layer, where it is attributed and signed and not checked.
Claims are not defined in this repository. Claims are not defined in this repository.
A comment, caveat or question also says what it is about with a `category` from a fixed list: a general `comment`, `context`, a `critique`, a `question`, a `correction` of an error in the paper, `feedback` for the authors, or a `resource`.
Model-written text is always a `generated` annotation, never source, and a model's annotation stays `proposed` until a person confirms it. Model-written text is always a `generated` annotation, never source, and a model's annotation stays `proposed` until a person confirms it.
Whether a generic Web Annotation client displays an `AnchorSelector` has not been tested; the fallback exists so that it can still use the quote. Whether a generic Web Annotation client displays an `AnchorSelector` has not been tested; the fallback exists so that it can still use the quote.
A `trace` annotation follows a citation to the passage it rests on: it targets the citing sentence and its body names the reference, the cited paper and the passage in it.
Whether a trace still leads there is computed from both papers rather than stored: `current`, `changed`, `stale`, `unlinked` or `misresolved`.
Layers are `candidate`. Layers are `candidate`.
Pinned by the layer records in `fixtures/records/layer/`. Pinned by the layer records in `fixtures/records/layer/` and `fixtures/records/traces/`, and by [vectors/traces.jsonl](../vectors/traces.jsonl).
See [chapter 18](../spec/18-layers.md). See [chapter 18](../spec/18-layers.md).
## Exercises ## Exercises
@@ -431,7 +436,7 @@ hale2026harmonic-damb/
provenance.json tools, versions and where each stage ran provenance.json tools, versions and where each stage ran
assets/ images, by SHA-256 assets/ images, by SHA-256
datasets/ datasets reported from the document datasets/ datasets reported from the document
derived/ MathML and other renderings, rebuildable derived/ MathML, math trees and other renderings, rebuildable
``` ```
Only `MANIFEST.json`, `document.json`, `nodes.jsonl`, `witnesses.json` and `provenance.json` are required, and an optional file with no records is left out; the example package omits `suspects.jsonl`. Only `MANIFEST.json`, `document.json`, `nodes.jsonl`, `witnesses.json` and `provenance.json` are required, and an optional file with no records is left out; the example package omits `suspects.jsonl`.
docs/examples.md
+36 -154
View File
@@ -9,7 +9,6 @@ The formats are version 0.1.0, and each is `candidate` or `provisional` in `regi
Run every command from the root of the repository, <https://git.scient.ing/infra/paperspec>. Run every command from the root of the repository, <https://git.scient.ing/infra/paperspec>.
With [pixi](https://pixi.sh), `pixi run <task>` sets up the environment first. With [pixi](https://pixi.sh), `pixi run <task>` sets up the environment first.
<<<<<<< HEAD
Every command also works without pixi once the checkout is installed with `pip install -e ".[tools]"` on Python 3.14, given standard command-line tools (the `sed` command in the last section is the GNU version of `sed`). Every command also works without pixi once the checkout is installed with `pip install -e ".[tools]"` on Python 3.14, given standard command-line tools (the `sed` command in the last section is the GNU version of `sed`).
## Run the checks ## Run the checks
@@ -156,155 +155,6 @@ writer2027fixed-ynr7.v3: 15 nodes
``` ```
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `paperspec check` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL. The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `paperspec check` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
||||||| parent of 4fa326d (First implementation (#1))
=======
Every command also works without pixi, with Python 3.14 and two packages, `jsonschema` and `cryptography` installed and standard command-line tools (the `sed` command in the last section is the GNU version of `sed`).
## Run the checks
This shows how to confirm that the repository's specification, fixtures and test vectors agree with each other.
```sh
pixi run check
```
`pixi run check` lints the Markdown, then runs `python tools/check.py`, the repository's reference checker.
Each section prints how many checks it ran, and the last line is `All checks pass.`
`pixi run quick`, or `python3 tools/check.py --quick`, skips the slowest part, which rebuilds everything and compares it byte for byte.
Without pixi, the Markdown lint is a separate step (`markdownlint-cli2`); `tools/check.py` does not run it.
## Recompute a hash by hand
This shows that a hash in a package is not magic: you can reproduce it with a standard tool.
A hash is a fixed-length fingerprint of some text; change one character and the fingerprint changes.
A sentence's `text_hash` is the SHA-256 of its text after two steps.
The text is converted to Unicode NFC (a standard form that writes each accented or composed character one way), and each run of whitespace is collapsed to one space, as [chapter 1](../spec/01-conventions.md#14-text-normalization-for-hashing) defines.
For a sentence with math or citations, the text is its plain projection ([chapter 5](../spec/05-hashes.md#51-the-five-values)).
For a sentence of plain text, the whole computation is a one-liner:
```sh
printf '%s' 'The two parameter sets agree to within 2% for every bond.' | sha256sum
```
```text
175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a -
```
The same value is in the package, in the `nodes.jsonl` of `hale2026harmonic-damb`:
```sh
grep -o '"id":"v3063qv".*' fixtures/packages/hale2026harmonic-damb/nodes.jsonl | grep -o '"text_hash":"[0-9a-f]*"'
```
```text
"text_hash":"175bfc8867816a9761fd5bb9ffc956df6d9e7f203870961a5485fc1912cb663a"
```
## Check a package's manifest
This shows how a reader detects that a package's files have changed.
A package is a folder of JSON (JavaScript Object Notation, a plain-text format for structured data) files describing one paper.
`MANIFEST.json` lists every other file of a package with its SHA-256 and size, and the SHA-256 of the manifest itself pins the whole package ([chapter 12](../spec/12-package.md#123-the-manifest)).
```sh
cd fixtures/packages/okafor2025solvent-acth
sha256sum nodes.jsonl
grep -o '"path":"nodes.jsonl","sha256":"[0-9a-f]*"' MANIFEST.json
```
```text
09c9b99eed97987e51dd0f85461ad9ad8768a0769ba7facfb7e0879d742660a3 nodes.jsonl
"path":"nodes.jsonl","sha256":"09c9b99eed97987e51dd0f85461ad9ad8768a0769ba7facfb7e0879d742660a3"
```
## Read a package with nothing but a JSON parser
This shows that a package needs no special software to read.
JSON Lines is a text format with one JSON object per line, here one node (a section, sentence, figure, table or equation) per line of `nodes.jsonl`.
Save this script as `outline.py`.
It prints a document's sections, figures, tables and equations.
```python
import json, sys
pkg = sys.argv[1]
nodes = [json.loads(line) for line in open(pkg + "/nodes.jsonl", encoding="utf-8")]
by_id = {n["id"]: n for n in nodes}
children = {}
for n in nodes:
children.setdefault(n["parent"], []).append(n)
def text(n):
"""The plain projection of a node with runs."""
out = []
for r in n["content"]:
if r["kind"] == "math":
out.append("$" + r["source"] + "$")
elif r["kind"] == "equation":
out.append("$$" + by_id[r["node"]]["content"]["source"] + "$$")
else:
out.append(r["text"])
return "".join(out)
for n in nodes:
if n["kind"] == "section":
heading = children[n["id"]][0]
print(n["id"], n.get("label", ""), " ".join(text(s) for s in children[heading["id"]]))
elif n["kind"] in ("figure", "table", "equation"):
print(n["id"], " ", n["kind"], n.get("label", ""))
```
```sh
python3 outline.py fixtures/packages/hale2026harmonic-damb
```
```text
p36k8t6 1 Introduction
tkxsmts equation (1)
d400rz3 2 Methods
v7cx6qs figure Figure 1
hj4s7pb table Table 1
```
## Follow an address
This shows how a citation points inside a sentence.
An address names a node, and optionally a span of its text ([chapter 6](../spec/06-anchors.md#61-addresses)).
`hale2026harmonic-damb#hca6dc5[29:44]` names code points 29 to 44 of sentence `hca6dc5`.
With `text()` from the script above:
```python
print(text(by_id["hca6dc5"])[29:44])
```
```text
first test case
```
Offsets count Unicode code points, one per character; in Python, string indexes already do.
JavaScript strings index UTF-16 units, so convert first.
For text with characters outside the Basic Multilingual Plane the two counts differ, and [vectors/offsets.jsonl](../vectors/offsets.jsonl) gives both.
## Load packages into SQLite
This shows that the same packages can live in a database and be queried with SQL.
```sh
pixi run load --sqlite lab.db fixtures/packages/hale2026harmonic-damb fixtures/packages/okafor2025solvent-acth fixtures/packages/writer2027fixed-ynr7.v3
```
Without pixi, run `python3 tools/load.py` with the same arguments.
```text
hale2026harmonic-damb: 75 nodes
okafor2025solvent-acth: 15 nodes
writer2027fixed-ynr7.v3: 15 nodes
```
The tables are those of [sql/doc.sql](../sql/doc.sql) and [sql/store.sql](../sql/store.sql), which run unchanged on Dolt and PostgreSQL as well ([chapter 13](../spec/13-database.md)); `tools/check.py` exercises them on SQLite only, and this repository has not yet run them on Dolt or PostgreSQL.
>>>>>>> 4fa326d (First implementation (#1))
Here are four queries, in Python's `sqlite3` so that nothing else is needed. Here are four queries, in Python's `sqlite3` so that nothing else is needed.
```python ```python
@@ -343,12 +193,12 @@ for (row,) in rows:
('a4jrz2t', 'F0') ('a4jrz2t', 'F0')
('xjns57z', 'F0') ('xjns57z', 'F0')
('hca6dc5',) ('hca6dc5',)
('ana', 'targeted:lode', 3) ('ana', 'targeted:paperclaim', 3)
('ana', 'targeted:lode', 23) ('ana', 'targeted:paperclaim', 23)
('ana', 'section:p36k8t6', 2) ('ana', 'section:p36k8t6', 2)
('cara', 'page:2', 3) ('cara', 'page:2', 3)
('ana', 'targeted:captions', 3) ('ana', 'targeted:captions', 3)
('ben', 'qa:y2zw8ctg5s8dp', 10) ('ben', 'qa:fhhvg9qrzqp0n', 10)
('ben', 'qa:gx2s84z9jf9xv', 3) ('ben', 'qa:gx2s84z9jf9xv', 3)
Bond parameters Bond parameters
CT–CT | 310.0 | 1.526 | 305.2 | 1.5260 CT–CT | 310.0 | 1.526 | 305.2 | 1.5260
@@ -360,7 +210,39 @@ The first query reads the level the converter recorded, not the final one.
A receipt is a signed record that a reviewer checked certain nodes against the page. A receipt is a signed record that a reviewer checked certain nodes against the page.
Receipts by `ana` raise `ec5jc93` and `a4jrz2t` to F3; `xjns57z` stays at F0 because the only receipt that covered it, by `cara`, was revoked. Receipts by `ana` raise `ec5jc93` and `a4jrz2t` to F3; `xjns57z` stays at F0 because the only receipt that covered it, by `cara`, was revoked.
To compute the final level, check each receipt's signature, revocations and spot-checks, as [chapter 8](../spec/08-fidelity.md#82-levels) describes. To compute the final level, check each receipt's signature, revocations and spot-checks, as [chapter 8](../spec/08-fidelity.md#82-levels) describes.
The `scope` column of the third query shows the value `targeted:lode`, which is data in the fixture, copied here exactly as the query prints it. The `scope` column of the third query shows the value `targeted:paperclaim`, which is data in the fixture, copied here exactly as the query prints it.
## Follow a traced citation
This shows how a citation leads to the passage it rests on, and how to tell that it still does.
A trace is an annotation from the sentence that cites a paper to a passage of that paper, with the reference it follows ([chapter 18](../spec/18-layers.md#185-traced-citations)).
The draft in the fixtures cites the main example once, and its layer of traces holds three: a model's, a person's, and a candidate a person rejected.
The Python package `paperspec.traces` judges each against the current state of both papers:
```python
import json
from paperspec import package, traces
draft = package.read_state("fixtures/packages/writer2027fixed-ynr7.v3")
paper = package.read_state("fixtures/packages/hale2026harmonic-damb")
same_as = json.load(open("fixtures/packages/hale2026harmonic-damb/document.json"))["work"]["same_as"]
annotations = [json.loads(line) for line in open("fixtures/records/traces/annotations.jsonl", encoding="utf-8")]
for a in annotations:
if a["body"]["kind"] == "trace":
r = traces.resolve(a, draft, paper, same_as)
print(a["status"], r["result"], r["open"], traces.disputed(a["id"], annotations))
```
```text
confirmed current smithlab/hale2026harmonic-damb#mhp0at8@ee42c4[48:60] True
rejected current smithlab/hale2026harmonic-damb#kea5026@b836d5[81:105] False
confirmed current smithlab/hale2026harmonic-damb#hj4s7pb@6c6fba False
```
Each line gives the trace's status, its outcome, the current address of the passage it leads to, and whether a reply disputes it.
The first trace is disputed by a reply that says the table is the better passage, which is where the third trace leads.
The outcomes here are all `current`; [vectors/traces.jsonl](../vectors/traces.jsonl) corrects the draft or the paper and shows the others: `changed` when the passage was reworded but is still found, `stale` when it is gone, `unlinked` when the sentence no longer cites the reference, and `misresolved` when the reference now resolves to another paper.
Resolving a trace also lists the citations inside the passage, so a chain of "as previously reported" is followed one trace at a time.
## Test your implementation against a vector ## Test your implementation against a vector
docs/faq.md
+1 -27
View File
@@ -107,8 +107,7 @@ A `candidate` format is complete, with vectors and fixtures, and ready to implem
A `provisional` one is complete and expected to change as implementations gain experience. A `provisional` one is complete and expected to change as implementations gain experience.
The provisional formats are dataset, digitization, view, project, bundle, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint. The provisional formats are dataset, digitization, view, project, bundle, snapshot, draft-ids, exercise, submission, grade, checklist, log-entry and checkpoint.
<<<<<<< HEAD The repository holds a reference checker (`paperspec check`), 26 vector files, five valid packages, one of them also as a zip, and 31 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
The repository holds a reference checker (`paperspec check`), 25 vector files, five valid packages, one of them also as a zip, and 31 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
## What does it not do? ## What does it not do?
@@ -131,28 +130,3 @@ See [CONTRIBUTING.md](../CONTRIBUTING.md).
`paperspec check` recomputes and checks every one, and `paperspec load` loads packages into SQLite. `paperspec check` recomputes and checks every one, and `paperspec load` loads packages into SQLite.
The oracle's modules in `src/paperspec/` compute every expected value, so that none is typed by hand. The oracle's modules in `src/paperspec/` compute every expected value, so that none is typed by hand.
The other repositories of the project install them as the Python package `paperspec`; independent implementations follow the specification ([tools/README.md](../tools/README.md)). The other repositories of the project install them as the Python package `paperspec`; independent implementations follow the specification ([tools/README.md](../tools/README.md)).
||||||| parent of 4fa326d (First implementation (#1))
=======
The repository holds a reference checker (`tools/check.py`), 25 vector files, five valid packages, one of them also as a zip, and 31 deliberately broken packages ([spec/00 §0.4](../spec/00-introduction.md#04-status), [spec/22](../spec/22-conformance.md)).
## What does it not do?
It is built for a team-scale corpus of hundreds of documents, not hundreds of millions of papers.
It does not replace publisher interchange formats; it reads JATS as a witness.
It does not fix what a converter reads wrongly.
It records the reading and raises disagreements as suspects, and a correction is checkable.
It is not a search engine, a citation index or a recommendation feed ([docs/overview.md](overview.md)).
## How do I add a node kind or a run kind?
A node kind is a type of document part, such as `sentence`; a run kind is a type of inline content within a sentence, such as `math`.
Propose either with the vector or fixture that shows it.
A new kind is a minor version ([spec/00 §0.3](../spec/00-introduction.md#03-format-identifiers-and-versions)).
See [CONTRIBUTING.md](../CONTRIBUTING.md).
## What are the files in `tools/` for?
`build.py` builds the fixtures and vectors. `check.py` recomputes and checks every one, and `load.py` loads packages into SQLite.
The `oracle/` modules compute every expected value, so that none is typed by hand.
They are not a library; implementations follow the specification ([tools/README.md](../tools/README.md)).
>>>>>>> 4fa326d (First implementation (#1))
docs/glossary.md
+17
View File
@@ -62,6 +62,10 @@ Drift
: The amount that has changed below a container since an anchor on it was made, counted from the corrections: the share of the container's leaves touched by later corrections. : The amount that has changed below a container since an anchor on it was made, counted from the corrections: the share of the container's leaves touched by later corrections.
It is reported as `none`, `low` or `high`, with a default threshold of a quarter. ([6.6](../spec/06-anchors.md#66-drift)) It is reported as `none`, `low` or `high`, with a default threshold of a quarter. ([6.6](../spec/06-anchors.md#66-drift))
Emend
: A correction of the authors' own typo, spelling, grammar, punctuation or wording, where the reading matches the page and the page is wrong; a `replace` fixes a misreading instead.
The emended run keeps the text as printed, so the leaf as printed, which is what witnesses are compared with, stays what it was, and so do its match hash and its level. ([11.6](../spec/11-corrections.md#116-emendations), [3.6](../spec/03-nodes.md#36-the-plain-projection))
Exercise Exercise
: An annotation whose body holds a prompt, an answer key and a grading rule. : An annotation whose body holds a prompt, an answer key and a grading rule.
The key is anchors, a verdict, a chain, a value or planted errors, by exercise kind. ([19.1](../spec/19-exercises.md#191-exercises)) The key is anchors, a verdict, a chain, a value or planted errors, by exercise kind. ([19.1](../spec/19-exercises.md#191-exercises))
@@ -96,6 +100,11 @@ Match hash
Two readings agree only when their match hashes are equal and their complex-notation keys are equal. Two readings agree only when their match hashes are equal and their complex-notation keys are equal.
A complex-notation key is `notation:source` for a piece of math too complex for the formats to normalize, with the source's whitespace removed ([5.3](../spec/05-hashes.md#53-the-strict-projection)). ([5.1](../spec/05-hashes.md#51-the-five-values), [5.3](../spec/05-hashes.md#53-the-strict-projection)) A complex-notation key is `notation:source` for a piece of math too complex for the formats to normalize, with the source's whitespace removed ([5.3](../spec/05-hashes.md#53-the-strict-projection)). ([5.1](../spec/05-hashes.md#51-the-five-values), [5.3](../spec/05-hashes.md#53-the-strict-projection))
Math form
: A standard spelling of formulas, so that one formula is written alike in every paper and claim.
`scienting-math/1` is canonical TeX, with a tree read from it and a key for each symbol; papermath implements it.
An equation or math run in the form says so with `form`, which enters no hash, and its tree lives in `derived/math.jsonl`. ([3.7](../spec/03-nodes.md#37-math-forms), [9.4](../spec/09-figures-tables-equations.md#94-the-math-form))
ML-DSA-65 ML-DSA-65
: Outside term. : Outside term.
The lattice-based signature scheme of FIPS 204, believed to resist quantum computers. The lattice-based signature scheme of FIPS 204, believed to resist quantum computers.
@@ -168,6 +177,10 @@ Strict projection
Suspect Suspect
: A doubtful reading raised for a person or a review session to answer. ([5.7](../spec/05-hashes.md#57-suspects)) : A doubtful reading raised for a person or a review session to answer. ([5.7](../spec/05-hashes.md#57-suspects))
Symbol key
: The name of a symbol in a formula, its canonical TeX with `\mathrm` spelled `\text`: `K_{r}` for `K_r`.
A paper's symbols and a paperclaim claim's have one key, so a `symbol` annotation names its symbol by it. ([9.4](../spec/09-figures-tables-equations.md#94-the-math-form), [18.4](../spec/18-layers.md#184-bodies))
Targeted F3 Targeted F3
: Proofreading the nodes a team's claims, quotations and datasets rest on, rather than every node. : Proofreading the nodes a team's claims, quotations and datasets rest on, rather than every node.
The target set also includes the leaves of every anchored container, every equation and, under the project's digit rule (`all`, `anchored-sections` or `none`), sentences and cells with a digit in a text or math run (an inline stretch of plain text, or of TeX or Typst math, within the node; [3.5](../spec/03-nodes.md#35-inline-runs)). ([8.10](../spec/08-fidelity.md#810-targeted-f3)) The target set also includes the leaves of every anchored container, every equation and, under the project's digit rule (`all`, `anchored-sections` or `none`), sentences and cells with a digit in a text or math run (an inline stretch of plain text, or of TeX or Typst math, within the node; [3.5](../spec/03-nodes.md#35-inline-runs)). ([8.10](../spec/08-fidelity.md#810-targeted-f3))
@@ -181,6 +194,10 @@ Text policy
Tile Tile
: One file of a transparency log served statically: 256 hashes of one level of the tree, or 256 entries, so a client fetches only the subtrees it needs. ([23.4](../spec/23-log.md#234-serving-a-log)) : One file of a transparency log served statically: 256 hashes of one level of the tree, or 256 entries, so a client fetches only the subtrees it needs. ([23.4](../spec/23-log.md#234-serving-a-log))
Trace
: An annotation that follows one citation of a sentence to the passage of the cited paper it rests on, naming the reference it follows and, when a tool found the passage, the tool, its run, score and rank.
Whether it still leads there is computed from both papers as current, changed, stale, unlinked or misresolved, and it is disputed while a reply assesses it. ([18.5](../spec/18-layers.md#185-traced-citations))
Verdict Verdict
: A judgment of whether a cited passage supports a citing sentence: supports, partly, does not, or cannot tell. ([19.2](../spec/19-exercises.md#192-verdicts)) : A judgment of whether a cited passage supports a citing sentence: supports, partly, does not, or cannot tell. ([19.2](../spec/19-exercises.md#192-verdicts))
docs/overview.md
+2 -26
View File
@@ -52,12 +52,11 @@ It appears inside hashed data and test vectors, and it is separate from the repo
| Folder | Holds | | Folder | Holds |
| --- | --- | | --- | --- |
| [spec/](../spec/README.md) | The specification, in 24 chapters (00 to 23). Normative. | | [spec/](../spec/README.md) | The specification, in 24 chapters (00 to 23). Normative. |
| [schemas/](../schemas/README.md) | 42 JSON (plain-text data format) Schemas for the files and records. | | [schemas/](../schemas/README.md) | 43 JSON (plain-text data format) Schemas for the files and records. |
| [registry/](../registry/README.md) | The list of formats, the rule IDs a checker reports, download-stamp rules, and default rights requirements. | | [registry/](../registry/README.md) | The list of formats, the rule IDs a checker reports, download-stamp rules, and default rights requirements. |
<<<<<<< HEAD
| [tools/](../tools/README.md) | How to run the reference commands: `paperspec check` checks everything, `paperspec build` rebuilds the fixtures and vectors, and `paperspec load` loads packages into SQLite. | | [tools/](../tools/README.md) | How to run the reference commands: `paperspec check` checks everything, `paperspec build` rebuilds the fixtures and vectors, and `paperspec load` loads packages into SQLite. |
| [src/paperspec/](../src/paperspec/) | The oracle, a compact implementation of every rule, which the other repositories of the project install as the Python package `paperspec`. | | [src/paperspec/](../src/paperspec/) | The oracle, a compact implementation of every rule, which the other repositories of the project install as the Python package `paperspec`. |
| [vectors/](../vectors/README.md) | 25 files of test vectors: inputs and the exact outputs every implementation must compute. | | [vectors/](../vectors/README.md) | 26 files of test vectors: inputs and the exact outputs every implementation must compute. |
| [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 31 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. | | [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 31 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. |
| [sql/](../sql/README.md) | Tables for SQLite, Dolt and PostgreSQL, and a SQLite bundle with full-text search. | | [sql/](../sql/README.md) | Tables for SQLite, Dolt and PostgreSQL, and a SQLite bundle with full-text search. |
| [context/](../context/README.md) | The JSON-LD context that makes annotation layers W3C Web Annotations. | | [context/](../context/README.md) | The JSON-LD context that makes annotation layers W3C Web Annotations. |
@@ -76,29 +75,6 @@ The repository ships reference tools only for the checker and store roles.
| Converter | Turns a witness (a PDF, JATS XML, the article format publishers use, a LaTeX source archive, an HTML page, an EPUB or a Typst project) into a package and raises doubtful readings as suspects (flagged disagreements awaiting review). | | Converter | Turns a witness (a PDF, JATS XML, the article format publishers use, a LaTeX source archive, an HTML page, an EPUB or a Typst project) into a package and raises doubtful readings as suspects (flagged disagreements awaiting review). |
| Checker | Recomputes hashes, validates files against their schemas and reports broken rules by rule ID. `paperspec check` does this for the repository. | | Checker | Recomputes hashes, validates files against their schemas and reports broken rules by rule ID. `paperspec check` does this for the repository. |
| Store | Keeps packages and records in a database a team can clone. `paperspec load` loads packages into SQLite. | | Store | Keeps packages and records in a database a team can clone. `paperspec load` loads packages into SQLite. |
||||||| parent of 4fa326d (First implementation (#1))
=======
| [tools/](../tools/README.md) | Reference programs: `check.py` checks everything, `build.py` rebuilds the fixtures and vectors, `load.py` loads packages into SQLite, and `oracle/` is a compact implementation of every rule. |
| [vectors/](../vectors/README.md) | 25 files of test vectors: inputs and the exact outputs every implementation must compute. |
| [fixtures/](../fixtures/README.md) | Five valid example packages, one of them also as a zip, 31 deliberately broken ones that each break one named rule, example records, and the test keys that signed them. All text is synthetic. |
| [sql/](../sql/README.md) | Tables for SQLite, Dolt and PostgreSQL, and a SQLite bundle with full-text search. |
| [context/](../context/README.md) | The JSON-LD context that makes annotation layers W3C Web Annotations. |
| [docs/](README.md) | Guides, including this page. |
## Roles
The formats serve five roles.
A role is a job someone could implement.
The repository ships reference tools only for the checker and store roles.
| Role | Does |
| --- | --- |
| Reader | Opens a package, resolves addresses and anchors, and shows text, annotations and each passage's fidelity level. |
| Writer | Produces files in these formats: nodes, corrections, receipts, annotations, datasets. |
| Converter | Turns a witness (a PDF, JATS XML, the article format publishers use, a LaTeX source archive, an HTML page, an EPUB or a Typst project) into a package and raises doubtful readings as suspects (flagged disagreements awaiting review). |
| Checker | Recomputes hashes, validates files against their schemas and reports broken rules by rule ID. `tools/check.py` does this for the repository. |
| Store | Keeps packages and records in a database a team can clone. `tools/load.py` loads packages into SQLite. |
>>>>>>> 4fa326d (First implementation (#1))
The [specification's reading paths](../spec/README.md) say which chapters each role needs. The [specification's reading paths](../spec/README.md) say which chapters each role needs.
docs/rationale.md
-5
View File
@@ -112,12 +112,7 @@ Only the three text values, `text_hash`, `match_hash` and `align_key`, normalize
### The character data is pinned to one Unicode version ### The character data is pinned to one Unicode version
Normalization and General Categories change between Unicode versions, and both enter every hash. Normalization and General Categories change between Unicode versions, and both enter every hash.
<<<<<<< HEAD
The chapters pin Unicode 16.0, the reference implementation refuses to import on another version, and moving to a later one is a change to every text-hashing format ([spec/01 §1.4](../spec/01-conventions.md#14-text-normalization-for-hashing)). The chapters pin Unicode 16.0, the reference implementation refuses to import on another version, and moving to a later one is a change to every text-hashing format ([spec/01 §1.4](../spec/01-conventions.md#14-text-normalization-for-hashing)).
||||||| parent of 4fa326d (First implementation (#1))
=======
The chapters pin Unicode 16.0, the reference tools refuse to run on another version, and moving to a later one is a change to every text-hashing format ([spec/01 §1.4](../spec/01-conventions.md#14-text-normalization-for-hashing)).
>>>>>>> 4fa326d (First implementation (#1))
### Hashes that leave the team are keyed ### Hashes that leave the team are keyed
docs/walkthrough.md
+4 -162
View File
@@ -13,7 +13,6 @@ It has nothing to do with the repository's name.
Open [fixtures/packages/hale2026harmonic-damb/](../fixtures/packages/hale2026harmonic-damb/) beside this page, because every ID below is in it. Open [fixtures/packages/hale2026harmonic-damb/](../fixtures/packages/hale2026harmonic-damb/) beside this page, because every ID below is in it.
A package is a folder of files, mostly JSON Lines (`.jsonl`: one JSON record per line). A package is a folder of files, mostly JSON Lines (`.jsonl`: one JSON record per line).
<<<<<<< HEAD
The hand-written source of the first reading is [fixtures/src/hale2026harmonic.json](../fixtures/src/hale2026harmonic.json); no converter ran on it, and [`paperspec build`](../src/paperspec/cli/build.py) expands it into the package. The hand-written source of the first reading is [fixtures/src/hale2026harmonic.json](../fixtures/src/hale2026harmonic.json); no converter ran on it, and [`paperspec build`](../src/paperspec/cli/build.py) expands it into the package.
Each step says what a converter, a person or a checker does. Each step says what a converter, a person or a checker does.
@@ -144,11 +143,11 @@ It lifts a node to F3, the top fidelity level (F0 is one reading; F1, every doub
One signature covers a Merkle tree, a hash over everything reviewed in a sitting, built as in RFC 6962 (a Request for Comments standard; replaced by [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162), which defines the same tree). One signature covers a Merkle tree, a hash over everything reviewed in a sitting, built as in RFC 6962 (a Request for Comments standard; replaced by [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162), which defines the same tree).
The receipts are in [`receipts.jsonl`](../fixtures/packages/hale2026harmonic-damb/receipts.jsonl), and the keys that signed them in [`roster.json`](../fixtures/packages/hale2026harmonic-damb/roster.json). The receipts are in [`receipts.jsonl`](../fixtures/packages/hale2026harmonic-damb/receipts.jsonl), and the keys that signed them in [`roster.json`](../fixtures/packages/hale2026harmonic-damb/roster.json).
- **Before the corrections**, `ana` had signed receipt `npc15pn6675z5` for three cells at revision 0. - **Before the corrections**, `ana` had signed receipt `vskq18t5sxydz` for three cells at revision 0.
Two of them never changed, so the receipt still counts for them. Two of them never changed, so the receipt still counts for them.
For the corrected cell it no longer counts, because the cell's hash moved on. For the corrected cell it no longer counts, because the cell's hash moved on.
- After the corrections, `ana` signs `y2zw8ctg5s8dp` at F3 for 23 nodes: the cells of Table 1's three data rows (15), its four column headers, the equation, and three sentences (`ec5jc93`, `hca6dc5` and `rr20tev`, the one holding the equation). - After the corrections, `ana` signs `fhhvg9qrzqp0n` at F3 for 23 nodes: the cells of Table 1's three data rows (15), its four column headers, the equation, and three sentences (`ec5jc93`, `hca6dc5` and `rr20tev`, the one holding the equation).
`ben` spot-checks it with receipt `mg18668nzxvap`. `ben` spot-checks it with receipt `ksx5sc327xp97`.
His signature, made with his separate spot-check key `ben-spot`, picks 10 nodes nobody could have chosen, and he finds no errors. His signature, made with his separate spot-check key `ben-spot`, picks 10 nodes nobody could have chosen, and he finds no errors.
- `ana` signs `mvjmb94nz7rdh` for two sentences of section 1, including the new `a4jrz2t`. - `ana` signs `mvjmb94nz7rdh` for two sentences of section 1, including the new `a4jrz2t`.
- `cara` signs `q465jpwz8afhe` for three sentences on page 2, then revokes it herself (`pcbhtpwvysgq6`) because she had compared them with the crops of another paper. - `cara` signs `q465jpwz8afhe` for three sentences on page 2, then revokes it herself (`pcbhtpwvysgq6`) because she had compared them with the crops of another paper.
@@ -156,7 +155,7 @@ His signature, made with his separate spot-check key `ben-spot`, picks 10 nodes
- `ana` signs `gx2s84z9jf9xv` for the figure's three caption sentences. `ben`'s spot-check `yt70fcc3qm0xb` finds a missing period in `caehhp6`, which suspends the receipt until she reviews the sitting again and signs a new one. - `ana` signs `gx2s84z9jf9xv` for the figure's three caption sentences. `ben`'s spot-check `yt70fcc3qm0xb` finds a missing period in `caehhp6`, which suspends the receipt until she reviews the sitting again and signs a new one.
The document's report, in [`document.json`](../fixtures/packages/hale2026harmonic-damb/document.json), counts the result over 45 leaves: 27 at F3, 13 at F2, 4 at F1 and 1 at F0. The document's report, in [`document.json`](../fixtures/packages/hale2026harmonic-damb/document.json), counts the result over 45 leaves: 27 at F3, 13 at F2, 4 at F1 and 1 at F0.
The 27 are the 23 nodes of `y2zw8ctg5s8dp`, the 2 of `mvjmb94nz7rdh`, and 2 of the 3 caption sentences. The 27 are the 23 nodes of `fhhvg9qrzqp0n`, the 2 of `mvjmb94nz7rdh`, and 2 of the 3 caption sentences.
Those two still count because Ben's spot-check is itself a valid receipt, and he compared them with the page and found them right. Those two still count because Ben's spot-check is itself a valid receipt, and he compared them with the page and found them right.
The third, `caehhp6`, stays at F2: a spot-check never raises a node it found in error ([spec/08 §8.2](../spec/08-fidelity.md#82-levels)). The third, `caehhp6`, stays at F2: a spot-check never raises a node it found in error ([spec/08 §8.2](../spec/08-fidelity.md#82-levels)).
@@ -168,163 +167,6 @@ Its CSV keeps every digit as printed (`1.526` and `1.5260` stay different), and
The descriptor types the fields as numbers; how a generic Data Package tool treats `1.5260` was not verified. The descriptor types the fields as numbers; how a generic Data Package tool treats `1.5260` was not verified.
All twelve value anchors resolve as `current`. All twelve value anchors resolve as `current`.
The reference checker, [`paperspec check`](../src/paperspec/cli/check.py), confirms each CSV value equals its cell's text. The reference checker, [`paperspec check`](../src/paperspec/cli/check.py), confirms each CSV value equals its cell's text.
||||||| parent of 4fa326d (First implementation (#1))
=======
The hand-written source of the first reading is [fixtures/src/hale2026harmonic.json](../fixtures/src/hale2026harmonic.json); no converter ran on it, and [`tools/build.py`](../tools/build.py) expands it into the package.
Each step says what a converter, a person or a checker does.
## 1. A PDF arrives
The team downloads the publisher's PDF.
The file becomes a witness, identified by its SHA-256 (a 64-digit hash that changes if any byte changes), with the short ID `qptsanv` ([spec/02](../spec/02-model.md)).
A person records its rights, meaning what the team may do with the file ([spec/14](../spec/14-rights.md)).
The library's subscription allows reading, text mining, and AI processing on the team's own machines for internal use, until August 31, 2027.
Whether it allows processing in the team's cloud is unknown.
The publisher stamped the download in the margin with the institution and the time, so two downloads rarely share a SHA-256.
A page fingerprint avoids the problem of downloads that differ in their stamps.
It is a hash of a page's text, size and images after the stamp lines are removed ([spec/07](../spec/07-geometry.md)), and the rules `downloaded-via` and `sharing-guidelines` fired on both pages (see [`pages.jsonl`](../fixtures/packages/hale2026harmonic-damb/pages.jsonl)).
Page 101's fingerprint is `6272e6f3…` in this copy, stamped October 11, 2026, and in a second download from another institution, stamped March 2, 2027 ([vectors/fingerprints.jsonl](../vectors/fingerprints.jsonl)).
## 2. The first reading, with its mistakes
A converter reads the PDF into sections, paragraphs, sentences, a figure, a table, an equation, a footnote and references.
Each of these is a node, and each node gets a permanent ID: seven characters minted once from the document key, the node's kind, a hash of its text and an occurrence count, then frozen ([spec/04](../spec/04-identity.md)).
An ID encodes no page, order or kind, so it keeps naming the node after the text is corrected.
Like every reading, this one has mistakes:
| What the reading did | What the page says |
| --- | --- |
| A cell read `3l0.0`. | `310.0`, with the digit 1. |
| A sentence read `Values below 10-3 kcal mol⁻¹…`. | A superscript: 10⁻³. |
| Two sentences ran together: "…treated as zero. Table 1 lists…". | Two sentences. |
| One sentence was cut in two at a line break: "…and the energies were" and "fitted by least squares." | One sentence. |
| A paragraph was split at a column break. | One paragraph. |
| The running head "J. Example Chem. 2026, 14, 101–109" became a paragraph. | A running head, not text. |
| The equation's number read `(l)` and the figure's `Figure l`. | `(1)` and `Figure 1`. |
| The CT–HC row merged two cells into "338.7 1.0900". | Two cells. |
| The OW–HW row was missing. | It is there. |
| "Set B" spanned one column. | It spans two. |
The converter does not guess.
Each doubtful place becomes a suspect, with a crop (an image cut from the page) for a person to answer ([spec/05 §5.7](../spec/05-hashes.md)).
Unresolved suspects travel in `suspects.jsonl`, which this package does not need.
Nothing changes until a correction passes the gate ([spec/11](../spec/11-corrections.md)).
## 3. Twelve corrections
Each fix is appended to [`corrections.jsonl`](../fixtures/packages/hale2026harmonic-damb/corrections.jsonl), numbered, with its evidence (a crop, or a person who read the page), its author (`ana` throughout) and its reason.
| # | Operation | On | What it did |
| --- | --- | --- | --- |
| 1 | `replace` | cell `c20cna6` | `3l0.0` to `310.0`. A digit changed, so the gate required a crop, and it has one. |
| 2 | `replace` | sentence `ec5jc93` | `10-3` to the runs `10` and superscript `−3`. |
| 3 | `split` | sentence `ec5jc93` | Split before "Table 1"; the first part keeps `ec5jc93`, the second is new: `a4jrz2t`. |
| 4 | `merge` | sentence `xjns57z` | Joined with `ayea71r`, which is retired with `xjns57z` as its successor. |
| 5 | `merge` | paragraph `gwzbj3s` | Joined with `ncx2v4f`, retired the same way. |
| 6 | `delete` | paragraph `v8e4z6z` | The running head is gone; its IDs are retired with no successor. |
| 7 | `relabel` | equation `tkxsmts` | `(l)` to `(1)`. |
| 8 | `delete` | row `rncchma` | The garbled CT–HC row and its four cells. |
| 9 | `insert` | row `kx2kz2y` | The CT–HC row, read again: five new cells. |
| 10 | `insert` | row `ke0jxpn` | The missing OW–HW row. |
| 11 | `set` | cell `xyy7x6c` | Set B's `colspan` from 1 to 2. |
| 12 | `relabel` | figure `v7cx6qs` | `Figure l` to `Figure 1`. |
The document is now at revision 12.
No uncorrected cell, sentence or row changed its ID or its hashes.
Only corrected nodes and their ancestors got new `node_hash` values; the sentence `rr20tev` is one of the ancestors, because it holds the relabeled equation.
Inserting a row adds a row node.
The table's own hash (`own_hash`) covers only its label and column definitions, so every other cell, and every value already taken from them, is untouched.
## 4. IDs that never move
The sentence about water is still `hca6dc5` after twelve corrections, the corrected cell is still `c20cna6`, and the sentence that lost its superscript is still `ec5jc93`.
The retired IDs are recorded in [`ids.jsonl`](../fixtures/packages/hale2026harmonic-damb/ids.jsonl), so anything that pointed at them can follow the trail.
Legacy aliases are recorded there too: `b-p01-text-003`, the ID an older tool gave the water sentence, still leads to `hca6dc5`.
A fresh conversion of the same input mints the same IDs.
A reading by a different converter is aligned with this document sentence by sentence ([spec/05 §5.6](../spec/05-hashes.md), pinned by [vectors/alignment.jsonl](../vectors/alignment.jsonl)).
Sentences the alignment pairs keep their IDs, and every difference becomes a suspect rather than a silent change.
Sentences it cannot pair are recorded as dropped or added.
## 5. What happened to the notes
An anchor is the stored pointer behind a note or citation.
It holds a node's ID, the node's hashes when the anchor was made, the revision, and, for part of a sentence, the quoted words with context ([spec/06](../spec/06-anchors.md)).
The team had already made seventeen anchors pointing into the first reading at revision 0, before any correction.
Resolving an anchor gives one of five results:
- `current`: nothing changed.
- `changed-below`: the node is unchanged, and something under it changed.
- `moved`: the node was merged into a successor that still holds the quote.
- `reattached`: the node changed, and the quote or the cell's headers now find exactly one place.
- `stale`: nothing is found, so a person must look.
A `changed-below` result carries a drift, the share of the leaves under the node, counting retired ones, that corrections touched.
Above a quarter the drift is `high`.
The table shows how each anchor resolves at revision 12 ([vectors/resolve.jsonl](../vectors/resolve.jsonl) records them all).
| Anchor on | Result | Why |
| --- | --- | --- |
| "H$_2$O" in the water sentence | `current` | Nothing in that sentence changed. |
| The cell `1.5260` | `current` | Untouched, though its table changed around it. |
| The water sentence, through the alias `b-p01-text-003` | `current` | The alias leads to the same node. |
| The cell that read `3l0.0` | `stale` | Its text changed and "3l0.0" is no longer in it; a person must look again. |
| "treated as zero" | `reattached` | The sentence changed twice, and the quote occurs in it exactly once, so the anchor is re-pinned. |
| "fitted constants" | `stale` | After the split, those words are in the new sentence `a4jrz2t`, not in `ec5jc93`. |
| "least squares", in the half-sentence `ayea71r` | `moved` | That ID was merged into `xjns57z`, which still holds the quote. |
| The paragraph `ncx2v4f` | `moved` | Merged into `gwzbj3s`, which holds all of its sentences. |
| The running head | `stale` | Deleted, with no successor. |
| The equation | `reattached` | Only its number changed. |
| The figure | `reattached` | Only its label changed, by a `relabel`. |
| The CT–HC cell `1.090` in the garbled row | `reattached` | Its row was deleted, and exactly one cell in Table 1 has the same row and column headers and the same text: the new `b5gd0h1`. |
| Section 2 | `changed-below`, drift high | The section itself is unchanged, and 53% of what it holds or held was touched. |
| Table 1 | `changed-below`, drift high | Rows were replaced and added. |
| The table body | `changed-below`, drift high | 75% of the cells it holds or held were touched. |
| The paragraph whose broken sentence was rejoined | `changed-below`, drift high | Two of the three sentences it has held were touched. |
| The CT–CT row | `changed-below`, drift low | One of its five cells was corrected. |
A reader shows each note by its result.
A checker of exact words accepts only `current`, `moved` and `reattached`.
The [W3C Web Annotation Data Model](https://www.w3.org/TR/annotation-model/) is the general W3C Recommendation for annotating web resources. paperspec's annotation layers build on it, with a custom `AnchorSelector` followed by a standard `TextQuoteSelector` as a fallback.
It has no permanent IDs for sentences inside the source, no hashes of the target text, and no results such as `moved` or drift.
See [comparison.md](comparison.md).
## 6. Review and signatures
A receipt is a signed record that a person compared a set of nodes with the page.
It lifts a node to F3, the top fidelity level (F0 is one reading; F1, every doubtful place answered; F2, an independent witness agrees) ([spec/08](../spec/08-fidelity.md)).
One signature covers a Merkle tree, a hash over everything reviewed in a sitting, built as in RFC 6962 (a Request for Comments standard; replaced by [RFC 9162](https://www.rfc-editor.org/rfc/rfc9162), which defines the same tree).
The receipts are in [`receipts.jsonl`](../fixtures/packages/hale2026harmonic-damb/receipts.jsonl), and the keys that signed them in [`roster.json`](../fixtures/packages/hale2026harmonic-damb/roster.json).
- **Before the corrections**, `ana` had signed receipt `npc15pn6675z5` for three cells at revision 0.
Two of them never changed, so the receipt still counts for them.
For the corrected cell it no longer counts, because the cell's hash moved on.
- After the corrections, `ana` signs `y2zw8ctg5s8dp` at F3 for 23 nodes: the cells of Table 1's three data rows (15), its four column headers, the equation, and three sentences (`ec5jc93`, `hca6dc5` and `rr20tev`, the one holding the equation).
`ben` spot-checks it with receipt `mg18668nzxvap`.
His signature, made with his separate spot-check key `ben-spot`, picks 10 nodes nobody could have chosen, and he finds no errors.
- `ana` signs `mvjmb94nz7rdh` for two sentences of section 1, including the new `a4jrz2t`.
- `cara` signs `q465jpwz8afhe` for three sentences on page 2, then revokes it herself (`pcbhtpwvysgq6`) because she had compared them with the crops of another paper.
Those sentences fall back to their earlier levels, which is why `xjns57z` is still F0.
- `ana` signs `gx2s84z9jf9xv` for the figure's three caption sentences. `ben`'s spot-check `yt70fcc3qm0xb` finds a missing period in `caehhp6`, which suspends the receipt until she reviews the sitting again and signs a new one.
The document's report, in [`document.json`](../fixtures/packages/hale2026harmonic-damb/document.json), counts the result over 45 leaves: 27 at F3, 13 at F2, 4 at F1 and 1 at F0.
The 27 are the 23 nodes of `y2zw8ctg5s8dp`, the 2 of `mvjmb94nz7rdh`, and 2 of the 3 caption sentences.
Those two still count because Ben's spot-check is itself a valid receipt, and he compared them with the page and found them right.
The third, `caehhp6`, stays at F2: a spot-check never raises a node it found in error ([spec/08 §8.2](../spec/08-fidelity.md#82-levels)).
## 7. Data out of the table
Table 1 becomes a dataset, `ds-hale-bond-params`, inside the package ([spec/15](../spec/15-datasets.md)).
It is a [Frictionless Data Package](https://datapackage.org/standard/data-package/) with units, uncertainty and an origin added, and the format is `provisional`.
Its CSV keeps every digit as printed (`1.526` and `1.5260` stay different), and every value has an anchor to the cell that prints it, so a claim that rests on a value can show the cell.
The descriptor types the fields as numbers; how a generic Data Package tool treats `1.5260` was not verified.
All twelve value anchors resolve as `current`.
The reference checker, [`tools/check.py`](../tools/check.py), confirms each CSV value equals its cell's text.
>>>>>>> 4fa326d (First implementation (#1))
## 8. Sharing without the text ## 8. Sharing without the text
fixtures/README.md
+3 -81
View File
@@ -25,7 +25,7 @@ A conforming reader accepts each one and recomputes every hash it contains.
| Package | What it is for | | Package | What it is for |
| --- | --- | | --- | --- |
| `packages/hale2026harmonic/` | The main example, followed step by step in [the walkthrough](../docs/walkthrough.md): a chemistry paper that was read once with mistakes and then fixed by twelve corrections. It has a table with two levels of column headers and grouped rows, an equation inside a sentence, a footnote, a figure with text inside its panels and a real image, references, legacy aliases, boxes on fingerprinted pages, receipts in every state (counting, partly invalidated, spot-checked, revoked and suspended), the roster of public keys that checks their signatures, a dataset taken from the table, and derived MathML. | | `packages/hale2026harmonic/` | The main example, followed step by step in [the walkthrough](../docs/walkthrough.md): a chemistry paper that was read once with mistakes and then fixed by twelve corrections. It has a table with two levels of column headers and grouped rows, an equation inside a sentence, a footnote, a figure with text inside its panels and a real image, references, legacy aliases, boxes on fingerprinted pages, receipts in every state (counting, partly invalidated, spot-checked, revoked and suspended), the roster of public keys that checks their signatures, a dataset taken from the table, derived MathML, and the derived reading of three formulas in the math form `scienting-math/1`, one of which the form's tree cannot read ([spec/09 section 9.4](../spec/09-figures-tables-equations.md#94-the-math-form)). |
| `packages/hale2026harmonic.checks-only/` | The same document prepared for sharing within the team: text withheld, every hash an HMAC (a hash computed under a secret key) under the team key. The team key is derived from the team secret in `keys/` ([spec/05](../spec/05-hashes.md), [spec/12 section 12.5](../spec/12-package.md#125-text-policies)). It shows how a team can share its checks on a paywalled paper without sharing the paper. | | `packages/hale2026harmonic.checks-only/` | The same document prepared for sharing within the team: text withheld, every hash an HMAC (a hash computed under a secret key) under the team key. The team key is derived from the team secret in `keys/` ([spec/05](../spec/05-hashes.md), [spec/12 section 12.5](../spec/12-package.md#125-text-policies)). It shows how a team can share its checks on a paywalled paper without sharing the paper. |
| `packages/okafor2025open/` | An open-access paper established from a JATS file (an XML format for journal articles) and a PDF that agree, so every leaf is at fidelity F2, corroborated by an independent witness. | | `packages/okafor2025open/` | An open-access paper established from a JATS file (an XML format for journal articles) and a PDF that agree, so every leaf is at fidelity F2, corroborated by an independent witness. |
| `packages/smithlab2027charges/` | A short draft written in Typst (a typesetting language). One of its sentences holds a cross-reference whose target is the address `hale2026harmonic#a4jrz2t`, a sentence of the main example. | | `packages/smithlab2027charges/` | A short draft written in Typst (a typesetting language). One of its sentences holds a cross-reference whose target is the address `hale2026harmonic#a4jrz2t`, a sentence of the main example. |
@@ -35,7 +35,6 @@ A conforming reader accepts each one and recomputes every hash it contains.
## Records ## Records
`records/` holds a valid example of every other format that is a JSON file of its own. `records/` holds a valid example of every other format that is a JSON file of its own.
<<<<<<< HEAD
Three formats have their example elsewhere: the stage requirements (`scienting-stages/1`) and the stamp rules (`scienting-stamps/1`) are in [registry/](../registry/README.md), and a bundle (`scienting-bundle/1`) is a SQLite file that `paperspec load --bundle` builds from the packages. Three formats have their example elsewhere: the stage requirements (`scienting-stages/1`) and the stamp rules (`scienting-stamps/1`) are in [registry/](../registry/README.md), and a bundle (`scienting-bundle/1`) is a SQLite file that `paperspec load --bundle` builds from the packages.
Each record is a single file, or a folder where a format needs several files. Each record is a single file, or a folder where a format needs several files.
@@ -48,7 +47,8 @@ Each record is a single file, or a folder where a format needs several files.
| `backend-docling.json`, `backend-examplevendor.json` | The code and weight licenses of a conversion tool and the conditions attached to them. | | `backend-docling.json`, `backend-examplevendor.json` | The code and weight licenses of a conversion tool and the conditions attached to them. |
| `profile-acme.json`, `profile-smithlab.json` | What those conditions ask about a team: its purpose, its funding and whom it competes with. | | `profile-acme.json`, `profile-smithlab.json` | What those conditions ask about a team: its purpose, its funding and whom it competes with. |
| `roster.json` | The keys that may sign, the role of each key's holder in the team, and what each key may sign. Ana is the owner, so she may revoke anyone's receipt. | | `roster.json` | The keys that may sign, the role of each key's holder in the team, and what each key may sign. Ana is the owner, so she may revoke anyone's receipt. |
| `layer/` | A layer of annotations on the main example (`layer.json`, `annotations.jsonl`) and the receipt that confirms them (`receipt.json`). | | `layer/` | A layer of annotations on the main example (`layer.json`, `annotations.jsonl`) and the receipt that confirms them (`receipt.json`). Its claim link and its symbol name the claim `amber/k-bond` of paperclaim's example store as paperclaim references it, `claim:272XTX7C`, pinned at `claim:272XTX7C@02f47b`; the symbol names `K_r` by its key, `K_{r}`. |
| `traces/` | A layer of traced citations on the Typst draft, which cites the main example (`layer.json`, `annotations.jsonl`, `receipt.json`): a trace a model proposed from the draft's citation of "a two-step fit" to the abstract of the main example and a person confirmed, with its receipt; a second candidate, the introduction's sentence, that a person rejected; a person's trace of the same sentence to Table 1; and a person's reply that disputes the first ([spec/18 §18.5](../spec/18-layers.md#185-traced-citations)). `vectors/traces.jsonl` resolves them after corrections. |
| `submissions/`, `grades/` | A student's answers to exercises in the layer, and the grader's result for each. | | `submissions/`, `grades/` | A student's answers to exercises in the layer, and the grader's result for each. |
| `suspect.json` | A doubtful reading raised for review: the converter's reading of an exponent in the main example, before the first correction answered it. | | `suspect.json` | A doubtful reading raised for review: the converter's reading of an exponent in the main example, before the first correction answered it. |
| `qa-report.json` | Reviewer error bounds for a delivery that covers two of the main example's receipts, signed by the team's owner key. | | `qa-report.json` | Reviewer error bounds for a delivery that covers two of the main example's receipts, signed by the team's owner key. |
@@ -111,81 +111,3 @@ Use them for nothing else.
`src/` holds the hand-written descriptions of the three main documents, as first read; see [src/README.md](src/README.md). `src/` holds the hand-written descriptions of the three main documents, as first read; see [src/README.md](src/README.md).
Everything else here is generated by `paperspec build`, so change the sources or the builder, never the generated files. Everything else here is generated by `paperspec build`, so change the sources or the builder, never the generated files.
||||||| parent of 4fa326d (First implementation (#1))
=======
Three formats have their example elsewhere: the stage requirements (`scienting-stages/1`) and the stamp rules (`scienting-stamps/1`) are in [registry/](../registry/README.md), and a bundle (`scienting-bundle/1`) is a SQLite file that `tools/load.py --bundle` builds from the packages.
Each record is a single file, or a folder where a format needs several files.
| Files | What they are for |
| --- | --- |
| `anchor-span.json`, `anchor-leaf.json`, `anchor-cell.json`, `anchor-paragraph.json` | An anchor is an address pinned to a hash, with a quotation and context, so it can be found again after the text changes. These four anchors point at a span of a sentence, a sentence, a table cell and a paragraph of the main example. |
| `anchor-keyed.json` | An anchor that leaves the team, as in a public layer: its table context is stored as keyed hashes, so the labels cannot be read from it ([spec/06 section 6.3](../spec/06-anchors.md#63-keyed-anchors)). |
| `rights-open.json`, `rights-subscription.json`, `rights-collective.json` | A witness's license and what uses it allows, under an open license, a subscription and a collective license. |
| `policy.json` | A signed team decision that settles a use the rights record leaves unknown. |
| `backend-docling.json`, `backend-examplevendor.json` | The code and weight licenses of a conversion tool and the conditions attached to them. |
| `profile-acme.json`, `profile-smithlab.json` | What those conditions ask about a team: its purpose, its funding and whom it competes with. |
| `roster.json` | The keys that may sign, the role of each key's holder in the team, and what each key may sign. Ana is the owner, so she may revoke anyone's receipt. |
| `layer/` | A layer of annotations on the main example (`layer.json`, `annotations.jsonl`) and the receipt that confirms them (`receipt.json`). |
| `submissions/`, `grades/` | A student's answers to exercises in the layer, and the grader's result for each. |
| `suspect.json` | A doubtful reading raised for review: the converter's reading of an exponent in the main example, before the first correction answered it. |
| `qa-report.json` | Reviewer error bounds for a delivery that covers two of the main example's receipts, signed by the team's owner key. |
| `checklist/` | A conformance checklist (`checklist.json`) for an accessible HTML export of the main example, checked against WCAG 2.1 AA criterion by criterion, and the receipt that covers it at its hash (`receipt.json`). The format is `provisional`. |
| `proof-inclusion.json`, `proof-consistency.json` | Two RFC 9162 proofs: that one node was covered by the main example's first receipt without showing the rest, and that revision 12 of its correction history extends revision 8 rather than rewriting it. |
| `checkpoint.json` | A signed checkpoint of the team's transparency log over the main example's receipts, revocations and roster, as the parsed form of its signed note, cosigned by a witness. The format is `provisional`. |
| `log/` | The same log as the static files a server would publish: `checkpoint` and the hash and entries tiles ([spec/23 §23.4](../spec/23-log.md#234-serving-a-log)). |
| `receipt-ml-dsa.json` | A receipt signed with ML-DSA-65 (FIPS 204) by `dana`, whose roster key is post-quantum. |
| `project.json`, `digitization.json`, `digitization-confirmation.json`, `datasets/`, `view.vl.json`, `snapshot.json`, `draft-ids.jsonl` | A team project; values read off the main example's figure by hand, the receipt that confirms them, and the dataset they yield; a Vega-Lite chart pinned to that dataset; what the draft cites, pinned; and the sentence IDs of the draft's Typst project. These formats are `provisional`. |
## Broken packages
Each folder in `invalid/` holds a `package/` folder, or a `package.sct.zip` archive, that breaks exactly one rule, built from `packages/tiny2026example/` by one edit, and an `expect.json` that names the rule.
A conforming implementation refuses every one of them and should report that rule.
`registry/rules.json` lists every rule ID with its meaning, and a validator reports each failure as `path:line: rule: message` ([spec/22 section 22.2](../spec/22-conformance.md#222-validating-a-package)).
| Case | What is wrong | Rule |
| --- | --- | --- |
| `alias-dangling/` | An alias whose node does not exist. | `doc.id.successor` |
| `bad-id/` | An ID that uses a letter Crockford base32 lacks (i, l, o or u). | `package.schema` |
| `checks-only-plain/` | A checks-only package that still carries text and plain hashes. | `doc.policy.withheld` |
| `cite-not-reference/` | A cite run names a node that is not a reference. | `doc.run.cite` |
| `counts-wrong/` | document.json's counts are not the number of nodes and leaves. | `doc.counts` |
| `crlf/` | A file whose lines end in CR LF, which is not the canonical form. | `package.file.canonical` |
| `empty-optional-file/` | An optional JSONL file with no records. | `package.file.empty` |
| `hash-mismatch/` | A node_hash that does not recompute. | `doc.hash.node-hash` |
| `heading-not-first/` | A heading must be a section's first child. | `doc.tree.child` |
| `history-wrong/` | document.json's history is not the root over the corrections. | `doc.history` |
| `key-suffix/` | A work key whose hash part is not the hash of the work's DOI. | `doc.key.suffix` |
| `key-work/` | A document key whose suffix is not the version's. | `doc.key.work` |
| `location-unknown-page/` | A location on a page no witness has. | `doc.location.page` |
| `manifest-hash/` | A file changed after the manifest was written. | `package.manifest.hash` |
| `manifest-key/` | The manifest names another key than document.json. | `package.manifest.key` |
| `missing-required-file/` | A required file is missing. | `package.file.required` |
| `not-canonical/` | A line that is valid JSON but not canonical JSON (the fixed form of RFC 8785, which gives equal data identical bytes). | `package.file.canonical` |
| `receipt-root/` | A receipt whose Merkle root (one hash that summarizes a list of hashes) does not recompute. | `doc.receipt.root` |
| `receipt-signature/` | A receipt changed after it was signed. | `doc.receipt.signature` |
| `receipt-unsorted-nodes/` | A receipt whose nodes are not sorted by ID. | `doc.receipt.nodes` |
| `retired-live/` | An ID that is both live and retired. | `doc.id.retired-live` |
| `rev-without-corrections/` | document.json's rev is not the number of corrections. | `doc.correction.rev` |
| `root-mismatch/` | document.json's root is not the root node's node_hash. | `doc.hash.root` |
| `signer-mismatch/` | A receipt whose signature was made with another key than the signer it names. | `doc.receipt.signature` |
| `style-sup-sub/` | A run cannot be both a subscript and a superscript. | `doc.run.style` |
| `text-edited/` | Text changed without its hashes. | `doc.hash.text-hash` |
| `unknown-kind/` | Kinds outside the list are refused. | `package.schema` |
| `unlisted-file/` | A file the manifest does not list. | `package.manifest.unlisted` |
| `unsafe-path/` | The manifest lists a path that leaves the package. | `package.path.unsafe` |
| `zip-symlink/` | A zipped package with an entry that is a symbolic link. | `package.zip.entry` |
| `zip-traversal/` | A zipped package with an entry whose path leaves the package folder. | `package.zip.entry` |
## Keys
`keys/` holds the Ed25519 secret keys (`ana`, `ben`, `cara`, `wit`, the witness, and `ben-spot`, whose VRF proof picks the nodes of a spot-check), the seed of `dana`'s ML-DSA-65 key, which signed the fixtures' receipts, and the team secret of the checks-only package.
Ed25519 is a signature scheme: the secret key makes a signature, and the public key in a roster verifies it.
These are published test keys.
Anyone can reproduce the signatures, so a signature made with these keys proves nothing about who made it.
Use them for nothing else.
## Sources
`src/` holds the hand-written descriptions of the three main documents, as first read; see [src/README.md](src/README.md).
Everything else here is generated by `python3 tools/build.py`, so change the sources or the builder, never the generated files.
>>>>>>> 4fa326d (First implementation (#1))
fixtures/packages/hale2026harmonic-damb/MANIFEST.json
+1 -1
View File
@@ -1 +1 @@
{"doc":"scienting-doc/1","files":[{"path":"assets/586f92512b294a1c325bfc16207771fb9acc7bb67bb206f25a429a055134056f.png","present":false,"sha256":"586f92512b294a1c325bfc16207771fb9acc7bb67bb206f25a429a055134056f","size":18213},{"path":"assets/a18e1eb7f434c85ee180399dc55d2ee78c5ceb915c4a004c5dd0e490399d4c75.png","sha256":"a18e1eb7f434c85ee180399dc55d2ee78c5ceb915c4a004c5dd0e490399d4c75","size":73},{"path":"corrections.jsonl","sha256":"026b493baaecba29c15a834e29933b840f306956d377fe85f350372313cab18f","size":8310},{"path":"datasets/ds-hale-bond-params/anchors.jsonl","sha256":"cfacea27f25c0ca912b82882c1f9221b693ae6bcf91c888628b780243ba7e625","size":6046},{"path":"datasets/ds-hale-bond-params/data.csv","sha256":"dbde72e4526e469e1535b87272eb50683160b2fe7c40b7f064e6592341f754a8","size":206},{"path":"datasets/ds-hale-bond-params/datapackage.json","sha256":"d8bbf8bd7c1b0207ada817d10580e835bda3a8691ad3e14fc1910519f4341974","size":574},{"path":"derived/mathml.jsonl","sha256":"47981f878391f3c733072081a0f6d1ad59ba8fa37507c8a4d774feb780c7df6c","size":348},{"path":"document.json","sha256":"f9d3fd85dec8b32378be5b045d0734ec05f3745ec88737bd0554749290e2fbbd","size":1088},{"path":"ids.jsonl","sha256":"d19c4716a605352d4cd84f81381d99a5e2db6b59c88e926611dd13a5a16e4155","size":852},{"path":"locations.jsonl","sha256":"37a3f73ff3dc81082e98b4a02afff535fbdf0f1439cf56fdb39566d28076c6e2","size":7396},{"path":"nodes.jsonl","sha256":"87eeac4b781f67872da36f44f74b859fe5a7c528e3c46276ec0b7dd4f5e2cb71","size":45649},{"path":"pages.jsonl","sha256":"948177c6b7e5e361ac1da0ecff6b52406d3513f14d58cd27cd84f17e6f1c054e","size":438},{"path":"provenance.json","sha256":"4bc7649bf6d3113f6c72b4016713654f57c8f03069632eabd1e6025c05cb9dac","size":226},{"path":"receipts.jsonl","sha256":"da6a4d9cf8af852dff7a175585867d6f2a224c664bd2935f4d47733cb76d1f5d","size":13150},{"path":"references.jsonl","sha256":"25c7a54848e0c3167475236846b85bcea1a35698f6ae5b24a31b6a3690078697","size":567},{"path":"roster.json","sha256":"3c71b0c007924cbacd3b4af02485214ab904db8ff8fca133ab32e67c5c76d481","size":6941},{"path":"witnesses.json","sha256":"99ae654df4fb461a424bb51edd476a89041182c7df9c98653a81b78031f92ada","size":1006}],"format":"scienting-package/1","key":"hale2026harmonic-damb"} {"doc":"scienting-doc/1","files":[{"path":"assets/586f92512b294a1c325bfc16207771fb9acc7bb67bb206f25a429a055134056f.png","present":false,"sha256":"586f92512b294a1c325bfc16207771fb9acc7bb67bb206f25a429a055134056f","size":18213},{"path":"assets/a18e1eb7f434c85ee180399dc55d2ee78c5ceb915c4a004c5dd0e490399d4c75.png","sha256":"a18e1eb7f434c85ee180399dc55d2ee78c5ceb915c4a004c5dd0e490399d4c75","size":73},{"path":"corrections.jsonl","sha256":"026b493baaecba29c15a834e29933b840f306956d377fe85f350372313cab18f","size":8310},{"path":"datasets/ds-hale-bond-params/anchors.jsonl","sha256":"cfacea27f25c0ca912b82882c1f9221b693ae6bcf91c888628b780243ba7e625","size":6046},{"path":"datasets/ds-hale-bond-params/data.csv","sha256":"dbde72e4526e469e1535b87272eb50683160b2fe7c40b7f064e6592341f754a8","size":206},{"path":"datasets/ds-hale-bond-params/datapackage.json","sha256":"d8bbf8bd7c1b0207ada817d10580e835bda3a8691ad3e14fc1910519f4341974","size":574},{"path":"derived/math.jsonl","sha256":"7c2e892825f2eb503187d6ac683d4a5f50b99e1d186ddd9ce64597eb0ba3493c","size":774},{"path":"derived/mathml.jsonl","sha256":"47981f878391f3c733072081a0f6d1ad59ba8fa37507c8a4d774feb780c7df6c","size":348},{"path":"document.json","sha256":"f9d3fd85dec8b32378be5b045d0734ec05f3745ec88737bd0554749290e2fbbd","size":1088},{"path":"ids.jsonl","sha256":"d19c4716a605352d4cd84f81381d99a5e2db6b59c88e926611dd13a5a16e4155","size":852},{"path":"locations.jsonl","sha256":"37a3f73ff3dc81082e98b4a02afff535fbdf0f1439cf56fdb39566d28076c6e2","size":7396},{"path":"nodes.jsonl","sha256":"87eeac4b781f67872da36f44f74b859fe5a7c528e3c46276ec0b7dd4f5e2cb71","size":45649},{"path":"pages.jsonl","sha256":"948177c6b7e5e361ac1da0ecff6b52406d3513f14d58cd27cd84f17e6f1c054e","size":438},{"path":"provenance.json","sha256":"4bc7649bf6d3113f6c72b4016713654f57c8f03069632eabd1e6025c05cb9dac","size":226},{"path":"receipts.jsonl","sha256":"ece8011e110d663b91a86b828d9c66b2396b3e03a5fd10554a1bfd3f4b53de20","size":13162},{"path":"references.jsonl","sha256":"25c7a54848e0c3167475236846b85bcea1a35698f6ae5b24a31b6a3690078697","size":567},{"path":"roster.json","sha256":"3c71b0c007924cbacd3b4af02485214ab904db8ff8fca133ab32e67c5c76d481","size":6941},{"path":"witnesses.json","sha256":"99ae654df4fb461a424bb51edd476a89041182c7df9c98653a81b78031f92ada","size":1006}],"format":"scienting-package/1","key":"hale2026harmonic-damb"}
fixtures/packages/hale2026harmonic-damb/derived/math.jsonl
+3
View File
@@ -0,0 +1,3 @@
{"complete":true,"form":"scienting-math/1","issues":[],"node":"tkxsmts","run":null,"symbols":["E_{\\text{bond}}","K_{r}","r","r_{\\text{eq}}"],"tool":{"name":"papermath","version":"0.1.0"},"tree":["rel",["sym","E_{\\text{bond}}"],"=",["mul",["frac",["num","1"],["num","2"]],["sym","K_{r}"],["pow",["add",["sym","r"],["neg",["sym","r_{\\text{eq}}"]]],["num","2"]]]]}
{"complete":true,"form":"scienting-math/1","issues":[],"node":"rr20tev","run":3,"symbols":["K_{r}"],"tool":{"name":"papermath","version":"0.1.0"},"tree":["sym","K_{r}"]}
{"complete":false,"form":"scienting-math/1","issues":[{"at":"_{2}","code":"opaque","message":"a part after the formula's end"}],"node":"hca6dc5","run":1,"symbols":[],"tool":{"name":"papermath","version":"0.1.0"},"tree":["opaque","_{2}"]}
fixtures/packages/hale2026harmonic-damb/receipts.jsonl
+3 -3
View File
@@ -1,8 +1,8 @@
{"at":"2026-10-01T16:00:00Z","by":"ana","doc":"hale2026harmonic-damb","evidence":[{"kind":"crop","node":"bnyvykm","sha256":"2c560caeaf914f3521fd0e3fca433922fcb9408c184c9b72bcfa869adf5ed409"},{"kind":"crop","node":"c20cna6","sha256":"bdbf7a4d9e86609675e3fc124c21ce808053a52d6bc0f35b15e1ba39027f13af"},{"kind":"crop","node":"ztww0g8","sha256":"902e8ca35e1a313a81130c4c100aeb2233fd53854677b3a992fb16f744fe3151"},{"kind":"witness","sha256":"23b6b2aaece9cb0a828a2f0265539500fcf34d5195bf720b934b8bfdeccb9834"}],"format":"scienting-receipt/1","id":"npc15pn6675z5","level":"F3","nodes":[["bnyvykm","13b0c5e01b52e467de5975acc898c2387fcc015d222517a865810a330a336c94"],["c20cna6","7252bb19f33a3be672d74a8aa8323750311b5c7d2b8f6cf21e07f17f2b27f3a1"],["ztww0g8","bda85639faacb77e504c891c035012602836171c009ecfd21584de4d6461ba93"]],"rev":0,"root":"5aeacf3cbd00a10320411c374b3b503583d1786f10c31fa04cc55117eba266ea","scope":"targeted:lode","signature":{"key":"ana","scheme":"ed25519","value":"IVRJwo1fygoctDfalPkrwvd5G5JARIqlV2FqzdPmL/BMvdVSKU9sRFJeLKHR+lXDVy3DY7GJgrjPh6VDdJAkCg=="}} {"at":"2026-10-01T16:00:00Z","by":"ana","doc":"hale2026harmonic-damb","evidence":[{"kind":"crop","node":"bnyvykm","sha256":"2c560caeaf914f3521fd0e3fca433922fcb9408c184c9b72bcfa869adf5ed409"},{"kind":"crop","node":"c20cna6","sha256":"bdbf7a4d9e86609675e3fc124c21ce808053a52d6bc0f35b15e1ba39027f13af"},{"kind":"crop","node":"ztww0g8","sha256":"902e8ca35e1a313a81130c4c100aeb2233fd53854677b3a992fb16f744fe3151"},{"kind":"witness","sha256":"23b6b2aaece9cb0a828a2f0265539500fcf34d5195bf720b934b8bfdeccb9834"}],"format":"scienting-receipt/1","id":"vskq18t5sxydz","level":"F3","nodes":[["bnyvykm","13b0c5e01b52e467de5975acc898c2387fcc015d222517a865810a330a336c94"],["c20cna6","7252bb19f33a3be672d74a8aa8323750311b5c7d2b8f6cf21e07f17f2b27f3a1"],["ztww0g8","bda85639faacb77e504c891c035012602836171c009ecfd21584de4d6461ba93"]],"rev":0,"root":"5aeacf3cbd00a10320411c374b3b503583d1786f10c31fa04cc55117eba266ea","scope":"targeted:paperclaim","signature":{"key":"ana","scheme":"ed25519","value":"OQmqG6r3mrl7bYq1pIUX+zoF8MprYLEmVu3rPoe632s7NZgH/Go2tIW+TGHcg+BuBStaBJCW/L47/5Z6OyayCQ=="}}
{"at":"2026-10-11T15:02:11Z","by":"ana","doc":"hale2026harmonic-damb","evidence":[{"kind":"crop","node":"avbhhey","sha256":"942c23ea73a297e69c2ae8a82abe88d688de2532cb3f6d3543d1107c54f65b2e"},{"kind":"crop","node":"b5gd0h1","sha256":"a0b6f761fe95f8f29ef6dc37901fa85c7a62857b50b0163ca67b0aab048e4d17"},{"kind":"crop","node":"bnyvykm","sha256":"2c560caeaf914f3521fd0e3fca433922fcb9408c184c9b72bcfa869adf5ed409"},{"kind":"crop","node":"c20cna6","sha256":"bdbf7a4d9e86609675e3fc124c21ce808053a52d6bc0f35b15e1ba39027f13af"},{"kind":"crop","node":"d1mhw8b","sha256":"3104c57d9f8fb38302b057ead3bc44456da94f7c17d7c5c8f6db8732d4812c31"},{"kind":"crop","node":"ec5jc93","sha256":"321a926653b9e097987d255d9286880893d71abf895d73f1649403ed9ac0877d"},{"kind":"crop","node":"exms5gc","sha256":"cfc0e42e734470df5cfb741f92080526ef796bf48bcf05847d44f7091cfaaab1"},{"kind":"crop","node":"ganq1ns","sha256":"81d88697cd002fdfd770ee11413c550e4e4ffaa464d35841648311f278fc13bc"},{"kind":"crop","node":"hca6dc5","sha256":"a26619c3a9824bc19fddcf89ee618d9a4792d69110439f121eb717b6f98e88a6"},{"kind":"crop","node":"j6y45v6","sha256":"e689deda15dde9ad24433eb946200417473d273a1bf3136df67bf572256a483f"},{"kind":"crop","node":"jv1741a","sha256":"de60579bacfab769f614fa74d26d20c7b1aa02caf60af1479dc46b98ddcbc106"},{"kind":"crop","node":"k6a6616","sha256":"37b29d7ba293b27f913868690a785591d89ba9a4f47da8adaf764a4d5ae9675b"},{"kind":"crop","node":"m1p1n2g","sha256":"369c69758d2d09cc3d42b2424723c4fbc75f9ceb6817da81ce183be7d36f045f"},{"kind":"crop","node":"rr20tev","sha256":"f57e2cfe704d52c712b01e993e492b121ffe8c4d0174b11a0b0fa21fc5b9930b"},{"kind":"crop","node":"ryhrskg","sha256":"e6f816d93cfff8418dbb771ba03a09fb89a3b2438f97fabfc4ebef7f78af0dbf"},{"kind":"crop","node":"tkxsmts","sha256":"5d8e205660c2c59dcaff119b5671ac11650974b5287d7d7b239a9118e9171a85"},{"kind":"crop","node":"vd76637","sha256":"389742a44f1291e2dc7d7a69acd09410c92ed8a305718b2ab86b0b09b23253f4"},{"kind":"crop","node":"w0gr97q","sha256":"588a38407463e5674714899e6cf359810ccfed98fefdd2c1a90e8cb2ed0a9d64"},{"kind":"crop","node":"w264b74","sha256":"0babb0cef51933b8744f5b22d075069d51e431d3bdc08cb5e35d7822097b5953"},{"kind":"crop","node":"x3ns64d","sha256":"6bd7423a5933eb89572ded1d40d2f5baf6f6fa2f3c46791de8210f201a94b9d1"},{"kind":"crop","node":"ycwgbx1","sha256":"5ee72aca6b32e45f961f9cb2ff70408e71fc9ed7fccd28e91eb86e872041fed3"},{"kind":"crop","node":"yvkd7xr","sha256":"62fd5beecb3e76f430dfd142f443158b84657196af17173a976345a0cdd74449"},{"kind":"crop","node":"ztww0g8","sha256":"902e8ca35e1a313a81130c4c100aeb2233fd53854677b3a992fb16f744fe3151"},{"kind":"witness","sha256":"23b6b2aaece9cb0a828a2f0265539500fcf34d5195bf720b934b8bfdeccb9834"}],"format":"scienting-receipt/1","id":"y2zw8ctg5s8dp","level":"F3","nodes":[["avbhhey","c51730c0b1d7e11b51b317163ae42838f1be808b3690f196923482c9021f97c0"],["b5gd0h1","9635a750ac2acf526a4db28746fd3ed84659f4e384f5a73fef85d002b4e9308b"],["bnyvykm","13b0c5e01b52e467de5975acc898c2387fcc015d222517a865810a330a336c94"],["c20cna6","c8e6a553f5ae1377776e86dde6ee7d48a923dea7de417408841ea33b81563272"],["d1mhw8b","557a47334329925d191e772172fddbc1bd1a035eb2217240c476ee6bc43c92ab"],["ec5jc93","5421e9cf0ef3a995792963d0371b79e11a8efad519b02efb5cb88311a896d0cc"],["exms5gc","b7c47f8360eb7272d2ac0943e8bf1aec9d71cb4ea4e5e1d02f8acee5b430a038"],["ganq1ns","40d46db2088057debc3b1d5b4a3c972ff3a94fc3d5463565f98876d0927f1e67"],["hca6dc5","8cfed5a34aab28517e89e8a164f0f96c14361de83ebc800439116e83ef479f34"],["j6y45v6","00c4b92e220303ebc2bfb36aa557ac80ae4dba1675e9ed47adb82f6aeff10634"],["jv1741a","f644190803b5ea36aa24738e7f168a7fdf0d05cb5123c14a814b7628dab71f1e"],["k6a6616","90aa63a43587bdcee8e428c7dc7a95ac1502bdbcb9f49cae05c433db6d18348b"],["m1p1n2g","0847b5d75c2259257c3619e761c64d5571d3ca3932e96a96ee07e8ce68cc63be"],["rr20tev","76e06b4811a688357ac4b912172283463cfaf1b7a1f0795d274fc733a2309ba7"],["ryhrskg","525583990e7871a2ba55739e8206abac2df0f19d30c86ad74f6cafdd7c3cbe25"],["tkxsmts","cc0bf73dd84e298da0d36c4b363c8d2bdc9b6d5738d1923d14c4a9028dca08d6"],["vd76637","33810e4da42f8e65a224b5369c06fe8db254890294b7b6760808fe6e039da21a"],["w0gr97q","f738a008336ac59b2f1d45cda2f045dc96d34f0f1378c431ce6ff8e434feb8a7"],["w264b74","2ba9cea0f5c07ee9f7b47c5d541c85e92465709322869fb6c0d994ab61c56cad"],["x3ns64d","6bfc0a055148d877dbaf28b7f62565de7b6926137bf032c382282c39a4876797"],["ycwgbx1","167d1551ca64cad5b1143431c5af6d3fc24530887470b7fd265df88d1f54f267"],["yvkd7xr","aab8791a24bc4067c7e37fc5f0244d2128205e0a169e65c2c0fad977aabce313"],["ztww0g8","bda85639faacb77e504c891c035012602836171c009ecfd21584de4d6461ba93"]],"rev":12,"root":"373a7d860a14b58e5bfa9b9ad3bc916091bcb6db7556f0dbc9c71972a612494d","scope":"targeted:lode","signature":{"key":"ana","scheme":"ed25519","value":"2sraHCMlItocGta02EhvPlDW+F8rB1ZYMXBzA5lmNmtH0h+AlrNOPTbtykVSAWQXfKT2DXuB81/6x5jyNsIVBA=="}} {"at":"2026-10-11T15:02:11Z","by":"ana","doc":"hale2026harmonic-damb","evidence":[{"kind":"crop","node":"avbhhey","sha256":"942c23ea73a297e69c2ae8a82abe88d688de2532cb3f6d3543d1107c54f65b2e"},{"kind":"crop","node":"b5gd0h1","sha256":"a0b6f761fe95f8f29ef6dc37901fa85c7a62857b50b0163ca67b0aab048e4d17"},{"kind":"crop","node":"bnyvykm","sha256":"2c560caeaf914f3521fd0e3fca433922fcb9408c184c9b72bcfa869adf5ed409"},{"kind":"crop","node":"c20cna6","sha256":"bdbf7a4d9e86609675e3fc124c21ce808053a52d6bc0f35b15e1ba39027f13af"},{"kind":"crop","node":"d1mhw8b","sha256":"3104c57d9f8fb38302b057ead3bc44456da94f7c17d7c5c8f6db8732d4812c31"},{"kind":"crop","node":"ec5jc93","sha256":"321a926653b9e097987d255d9286880893d71abf895d73f1649403ed9ac0877d"},{"kind":"crop","node":"exms5gc","sha256":"cfc0e42e734470df5cfb741f92080526ef796bf48bcf05847d44f7091cfaaab1"},{"kind":"crop","node":"ganq1ns","sha256":"81d88697cd002fdfd770ee11413c550e4e4ffaa464d35841648311f278fc13bc"},{"kind":"crop","node":"hca6dc5","sha256":"a26619c3a9824bc19fddcf89ee618d9a4792d69110439f121eb717b6f98e88a6"},{"kind":"crop","node":"j6y45v6","sha256":"e689deda15dde9ad24433eb946200417473d273a1bf3136df67bf572256a483f"},{"kind":"crop","node":"jv1741a","sha256":"de60579bacfab769f614fa74d26d20c7b1aa02caf60af1479dc46b98ddcbc106"},{"kind":"crop","node":"k6a6616","sha256":"37b29d7ba293b27f913868690a785591d89ba9a4f47da8adaf764a4d5ae9675b"},{"kind":"crop","node":"m1p1n2g","sha256":"369c69758d2d09cc3d42b2424723c4fbc75f9ceb6817da81ce183be7d36f045f"},{"kind":"crop","node":"rr20tev","sha256":"f57e2cfe704d52c712b01e993e492b121ffe8c4d0174b11a0b0fa21fc5b9930b"},{"kind":"crop","node":"ryhrskg","sha256":"e6f816d93cfff8418dbb771ba03a09fb89a3b2438f97fabfc4ebef7f78af0dbf"},{"kind":"crop","node":"tkxsmts","sha256":"5d8e205660c2c59dcaff119b5671ac11650974b5287d7d7b239a9118e9171a85"},{"kind":"crop","node":"vd76637","sha256":"389742a44f1291e2dc7d7a69acd09410c92ed8a305718b2ab86b0b09b23253f4"},{"kind":"crop","node":"w0gr97q","sha256":"588a38407463e5674714899e6cf359810ccfed98fefdd2c1a90e8cb2ed0a9d64"},{"kind":"crop","node":"w264b74","sha256":"0babb0cef51933b8744f5b22d075069d51e431d3bdc08cb5e35d7822097b5953"},{"kind":"crop","node":"x3ns64d","sha256":"6bd7423a5933eb89572ded1d40d2f5baf6f6fa2f3c46791de8210f201a94b9d1"},{"kind":"crop","node":"ycwgbx1","sha256":"5ee72aca6b32e45f961f9cb2ff70408e71fc9ed7fccd28e91eb86e872041fed3"},{"kind":"crop","node":"yvkd7xr","sha256":"62fd5beecb3e76f430dfd142f443158b84657196af17173a976345a0cdd74449"},{"kind":"crop","node":"ztww0g8","sha256":"902e8ca35e1a313a81130c4c100aeb2233fd53854677b3a992fb16f744fe3151"},{"kind":"witness","sha256":"23b6b2aaece9cb0a828a2f0265539500fcf34d5195bf720b934b8bfdeccb9834"}],"format":"scienting-receipt/1","id":"fhhvg9qrzqp0n","level":"F3","nodes":[["avbhhey","c51730c0b1d7e11b51b317163ae42838f1be808b3690f196923482c9021f97c0"],["b5gd0h1","9635a750ac2acf526a4db28746fd3ed84659f4e384f5a73fef85d002b4e9308b"],["bnyvykm","13b0c5e01b52e467de5975acc898c2387fcc015d222517a865810a330a336c94"],["c20cna6","c8e6a553f5ae1377776e86dde6ee7d48a923dea7de417408841ea33b81563272"],["d1mhw8b","557a47334329925d191e772172fddbc1bd1a035eb2217240c476ee6bc43c92ab"],["ec5jc93","5421e9cf0ef3a995792963d0371b79e11a8efad519b02efb5cb88311a896d0cc"],["exms5gc","b7c47f8360eb7272d2ac0943e8bf1aec9d71cb4ea4e5e1d02f8acee5b430a038"],["ganq1ns","40d46db2088057debc3b1d5b4a3c972ff3a94fc3d5463565f98876d0927f1e67"],["hca6dc5","8cfed5a34aab28517e89e8a164f0f96c14361de83ebc800439116e83ef479f34"],["j6y45v6","00c4b92e220303ebc2bfb36aa557ac80ae4dba1675e9ed47adb82f6aeff10634"],["jv1741a","f644190803b5ea36aa24738e7f168a7fdf0d05cb5123c14a814b7628dab71f1e"],["k6a6616","90aa63a43587bdcee8e428c7dc7a95ac1502bdbcb9f49cae05c433db6d18348b"],["m1p1n2g","0847b5d75c2259257c3619e761c64d5571d3ca3932e96a96ee07e8ce68cc63be"],["rr20tev","76e06b4811a688357ac4b912172283463cfaf1b7a1f0795d274fc733a2309ba7"],["ryhrskg","525583990e7871a2ba55739e8206abac2df0f19d30c86ad74f6cafdd7c3cbe25"],["tkxsmts","cc0bf73dd84e298da0d36c4b363c8d2bdc9b6d5738d1923d14c4a9028dca08d6"],["vd76637","33810e4da42f8e65a224b5369c06fe8db254890294b7b6760808fe6e039da21a"],["w0gr97q","f738a008336ac59b2f1d45cda2f045dc96d34f0f1378c431ce6ff8e434feb8a7"],["w264b74","2ba9cea0f5c07ee9f7b47c5d541c85e92465709322869fb6c0d994ab61c56cad"],["x3ns64d","6bfc0a055148d877dbaf28b7f62565de7b6926137bf032c382282c39a4876797"],["ycwgbx1","167d1551ca64cad5b1143431c5af6d3fc24530887470b7fd265df88d1f54f267"],["yvkd7xr","aab8791a24bc4067c7e37fc5f0244d2128205e0a169e65c2c0fad977aabce313"],["ztww0g8","bda85639faacb77e504c891c035012602836171c009ecfd21584de4d6461ba93"]],"rev":12,"root":"373a7d860a14b58e5bfa9b9ad3bc916091bcb6db7556f0dbc9c71972a612494d","scope":"targeted:paperclaim","signature":{"key":"ana","scheme":"ed25519","value":"wTBxOxNPgseU/6ENyaRVcQKc6AChkmfqnkNbXfTEwQkIXlcSnh9GIHEerGOYUy2sTzm146sL3PQMnfU3LkucBA=="}}
{"at":"2026-10-11T16:30:00Z","by":"ana","doc":"hale2026harmonic-damb","evidence":[{"kind":"crop","node":"a4jrz2t","sha256":"46790e75ae8cff068515a73b05642be78939e4dd8bad40a080e0d135ba838799"},{"kind":"crop","node":"kea5026","sha256":"1ee24fc6dec45a1372134888b70ad35e4823bdfe7d3e57d2dab08e9d7d329356"},{"kind":"witness","sha256":"23b6b2aaece9cb0a828a2f0265539500fcf34d5195bf720b934b8bfdeccb9834"}],"format":"scienting-receipt/1","id":"mvjmb94nz7rdh","level":"F3","nodes":[["a4jrz2t","f64c6a5040f9c773d5ac4e559b58465aeae30bba79a82fc38b14ac7b61826ba8"],["kea5026","b836d5872ca37471fa996be04c538254a8b1e37ca12e8f1b4a40837b503afa33"]],"rev":12,"root":"5e3134a4583e5ea26e575da2a363a0520a7ee111f4da782c641e8c24070f8895","scope":"section:p36k8t6","signature":{"key":"ana","scheme":"ed25519","value":"tprithiSRx7h9KjqJrVCq7zK3ZufvNmACZNFORVde8I0sQUZgq6RTx9orJP3okDZj1I9/gGwJ9Fv1cCVBCYEAw=="}} {"at":"2026-10-11T16:30:00Z","by":"ana","doc":"hale2026harmonic-damb","evidence":[{"kind":"crop","node":"a4jrz2t","sha256":"46790e75ae8cff068515a73b05642be78939e4dd8bad40a080e0d135ba838799"},{"kind":"crop","node":"kea5026","sha256":"1ee24fc6dec45a1372134888b70ad35e4823bdfe7d3e57d2dab08e9d7d329356"},{"kind":"witness","sha256":"23b6b2aaece9cb0a828a2f0265539500fcf34d5195bf720b934b8bfdeccb9834"}],"format":"scienting-receipt/1","id":"mvjmb94nz7rdh","level":"F3","nodes":[["a4jrz2t","f64c6a5040f9c773d5ac4e559b58465aeae30bba79a82fc38b14ac7b61826ba8"],["kea5026","b836d5872ca37471fa996be04c538254a8b1e37ca12e8f1b4a40837b503afa33"]],"rev":12,"root":"5e3134a4583e5ea26e575da2a363a0520a7ee111f4da782c641e8c24070f8895","scope":"section:p36k8t6","signature":{"key":"ana","scheme":"ed25519","value":"tprithiSRx7h9KjqJrVCq7zK3ZufvNmACZNFORVde8I0sQUZgq6RTx9orJP3okDZj1I9/gGwJ9Fv1cCVBCYEAw=="}}
{"at":"2026-10-11T17:00:00Z","by":"cara","doc":"hale2026harmonic-damb","evidence":[{"kind":"crop","node":"qwgfg51","sha256":"886b5234b8240653d2a1f9551552592d3e64792658980b06b2d38f1af61b51d8"},{"kind":"crop","node":"vya7t4n","sha256":"4ef9dce7bed5821302a3fe118dbe32395df49cab231723cc1fe1d2b3b15093f1"},{"kind":"crop","node":"xjns57z","sha256":"57d07288058274f0e54c57899476eec6ddfc2662043ec609b5b91315bf0a4786"},{"kind":"witness","sha256":"23b6b2aaece9cb0a828a2f0265539500fcf34d5195bf720b934b8bfdeccb9834"}],"format":"scienting-receipt/1","id":"q465jpwz8afhe","level":"F3","nodes":[["qwgfg51","4a3be03766323b3c88d5b43591b0f573823b89fe582f2c3a6f0b43474685a30d"],["vya7t4n","5e7362eb5c7cd2db425e557982083a51c2607521b6b1f73d79d98956616965df"],["xjns57z","42b1de7852cdc68730984fff789fe248ecd5b43198c0e7d671931ff3fe561dc3"]],"rev":12,"root":"36aeaeab7720b10ffbbe12d07751ac3273c77fc642355e00e795cbe7db73b3b0","scope":"page:2","signature":{"key":"cara","scheme":"ed25519","value":"QY2hQLVWKyOLkSCQNzTOskn5suI4s1WmflgrQDvB43IyBx5AOP4qXdvyKdkvb1YJ+puHdpDi53/4Ad92jKYPCQ=="}} {"at":"2026-10-11T17:00:00Z","by":"cara","doc":"hale2026harmonic-damb","evidence":[{"kind":"crop","node":"qwgfg51","sha256":"886b5234b8240653d2a1f9551552592d3e64792658980b06b2d38f1af61b51d8"},{"kind":"crop","node":"vya7t4n","sha256":"4ef9dce7bed5821302a3fe118dbe32395df49cab231723cc1fe1d2b3b15093f1"},{"kind":"crop","node":"xjns57z","sha256":"57d07288058274f0e54c57899476eec6ddfc2662043ec609b5b91315bf0a4786"},{"kind":"witness","sha256":"23b6b2aaece9cb0a828a2f0265539500fcf34d5195bf720b934b8bfdeccb9834"}],"format":"scienting-receipt/1","id":"q465jpwz8afhe","level":"F3","nodes":[["qwgfg51","4a3be03766323b3c88d5b43591b0f573823b89fe582f2c3a6f0b43474685a30d"],["vya7t4n","5e7362eb5c7cd2db425e557982083a51c2607521b6b1f73d79d98956616965df"],["xjns57z","42b1de7852cdc68730984fff789fe248ecd5b43198c0e7d671931ff3fe561dc3"]],"rev":12,"root":"36aeaeab7720b10ffbbe12d07751ac3273c77fc642355e00e795cbe7db73b3b0","scope":"page:2","signature":{"key":"cara","scheme":"ed25519","value":"QY2hQLVWKyOLkSCQNzTOskn5suI4s1WmflgrQDvB43IyBx5AOP4qXdvyKdkvb1YJ+puHdpDi53/4Ad92jKYPCQ=="}}
{"at":"2026-10-11T17:20:00Z","by":"cara","format":"scienting-revocation/1","id":"pcbhtpwvysgq6","reason":"Signed against the crops of another paper.","receipt":"q465jpwz8afhe","signature":{"key":"cara","scheme":"ed25519","value":"hObSv+xgW3GLZPkkR08CB+9LdxPP+16j/C/eI4OnqeuvbamQwDv7uZJmHBNBG1GkU+TcVM0V4IbDNfpuYyO4DQ=="}} {"at":"2026-10-11T17:20:00Z","by":"cara","format":"scienting-revocation/1","id":"pcbhtpwvysgq6","reason":"Signed against the crops of another paper.","receipt":"q465jpwz8afhe","signature":{"key":"cara","scheme":"ed25519","value":"hObSv+xgW3GLZPkkR08CB+9LdxPP+16j/C/eI4OnqeuvbamQwDv7uZJmHBNBG1GkU+TcVM0V4IbDNfpuYyO4DQ=="}}
{"at":"2026-10-11T18:00:00Z","by":"ana","doc":"hale2026harmonic-damb","evidence":[{"kind":"crop","node":"caehhp6","sha256":"03932451473e6756936d196da5db53b9027220ab90402f32c212ed02a3e0cb69"},{"kind":"crop","node":"nms5qjg","sha256":"87a410886dac6569211ac21d5787953e6cb538b691d1cee916eb9ae72061ebba"},{"kind":"crop","node":"z86b2za","sha256":"3bd36b4ee0d38edfcd3e8feb546e24693cd8034e6ce3c11f01d2be8871f38c76"},{"kind":"witness","sha256":"23b6b2aaece9cb0a828a2f0265539500fcf34d5195bf720b934b8bfdeccb9834"}],"format":"scienting-receipt/1","id":"gx2s84z9jf9xv","level":"F3","nodes":[["caehhp6","9f9c9a412cd08d0634daa3f23c63f5125c1270c13834c45c045c1a9a7452af86"],["nms5qjg","e2f0b1d093c2f38f329c16edae255aed6e913d489e730a9d4857a3c3b5040f63"],["z86b2za","9a1c438b0f413d63b7e9f1ed7037cc6ad1665efd335baee2ef31d3d232c79552"]],"rev":12,"root":"1d4c44e1018708ff37d9e4b6b722b55388f627a2a9d00c32ea4dcbb0dad4a976","scope":"targeted:captions","signature":{"key":"ana","scheme":"ed25519","value":"nsms22ca/Cff9M6lamEOyn81qL/+OJg9uSDcXVny91XFQAlKZ3s3hS/Xf+8snrhCsuNEOqEfHYpg0nnVAFnwAQ=="}} {"at":"2026-10-11T18:00:00Z","by":"ana","doc":"hale2026harmonic-damb","evidence":[{"kind":"crop","node":"caehhp6","sha256":"03932451473e6756936d196da5db53b9027220ab90402f32c212ed02a3e0cb69"},{"kind":"crop","node":"nms5qjg","sha256":"87a410886dac6569211ac21d5787953e6cb538b691d1cee916eb9ae72061ebba"},{"kind":"crop","node":"z86b2za","sha256":"3bd36b4ee0d38edfcd3e8feb546e24693cd8034e6ce3c11f01d2be8871f38c76"},{"kind":"witness","sha256":"23b6b2aaece9cb0a828a2f0265539500fcf34d5195bf720b934b8bfdeccb9834"}],"format":"scienting-receipt/1","id":"gx2s84z9jf9xv","level":"F3","nodes":[["caehhp6","9f9c9a412cd08d0634daa3f23c63f5125c1270c13834c45c045c1a9a7452af86"],["nms5qjg","e2f0b1d093c2f38f329c16edae255aed6e913d489e730a9d4857a3c3b5040f63"],["z86b2za","9a1c438b0f413d63b7e9f1ed7037cc6ad1665efd335baee2ef31d3d232c79552"]],"rev":12,"root":"1d4c44e1018708ff37d9e4b6b722b55388f627a2a9d00c32ea4dcbb0dad4a976","scope":"targeted:captions","signature":{"key":"ana","scheme":"ed25519","value":"nsms22ca/Cff9M6lamEOyn81qL/+OJg9uSDcXVny91XFQAlKZ3s3hS/Xf+8snrhCsuNEOqEfHYpg0nnVAFnwAQ=="}}
{"at":"2026-10-12T09:00:00Z","by":"ben","doc":"hale2026harmonic-damb","evidence":[{"kind":"crop","node":"bnyvykm","sha256":"2c560caeaf914f3521fd0e3fca433922fcb9408c184c9b72bcfa869adf5ed409"},{"kind":"crop","node":"c20cna6","sha256":"bdbf7a4d9e86609675e3fc124c21ce808053a52d6bc0f35b15e1ba39027f13af"},{"kind":"crop","node":"d1mhw8b","sha256":"3104c57d9f8fb38302b057ead3bc44456da94f7c17d7c5c8f6db8732d4812c31"},{"kind":"crop","node":"exms5gc","sha256":"cfc0e42e734470df5cfb741f92080526ef796bf48bcf05847d44f7091cfaaab1"},{"kind":"crop","node":"ganq1ns","sha256":"81d88697cd002fdfd770ee11413c550e4e4ffaa464d35841648311f278fc13bc"},{"kind":"crop","node":"m1p1n2g","sha256":"369c69758d2d09cc3d42b2424723c4fbc75f9ceb6817da81ce183be7d36f045f"},{"kind":"crop","node":"ryhrskg","sha256":"e6f816d93cfff8418dbb771ba03a09fb89a3b2438f97fabfc4ebef7f78af0dbf"},{"kind":"crop","node":"w264b74","sha256":"0babb0cef51933b8744f5b22d075069d51e431d3bdc08cb5e35d7822097b5953"},{"kind":"crop","node":"yvkd7xr","sha256":"62fd5beecb3e76f430dfd142f443158b84657196af17173a976345a0cdd74449"},{"kind":"crop","node":"ztww0g8","sha256":"902e8ca35e1a313a81130c4c100aeb2233fd53854677b3a992fb16f744fe3151"}],"format":"scienting-receipt/1","id":"mg18668nzxvap","level":"F3","nodes":[["bnyvykm","13b0c5e01b52e467de5975acc898c2387fcc015d222517a865810a330a336c94"],["c20cna6","c8e6a553f5ae1377776e86dde6ee7d48a923dea7de417408841ea33b81563272"],["d1mhw8b","557a47334329925d191e772172fddbc1bd1a035eb2217240c476ee6bc43c92ab"],["exms5gc","b7c47f8360eb7272d2ac0943e8bf1aec9d71cb4ea4e5e1d02f8acee5b430a038"],["ganq1ns","40d46db2088057debc3b1d5b4a3c972ff3a94fc3d5463565f98876d0927f1e67"],["m1p1n2g","0847b5d75c2259257c3619e761c64d5571d3ca3932e96a96ee07e8ce68cc63be"],["ryhrskg","525583990e7871a2ba55739e8206abac2df0f19d30c86ad74f6cafdd7c3cbe25"],["w264b74","2ba9cea0f5c07ee9f7b47c5d541c85e92465709322869fb6c0d994ab61c56cad"],["yvkd7xr","aab8791a24bc4067c7e37fc5f0244d2128205e0a169e65c2c0fad977aabce313"],["ztww0g8","bda85639faacb77e504c891c035012602836171c009ecfd21584de4d6461ba93"]],"qa":{"errors":[],"key":"ben-spot","minimum":10,"of":"y2zw8ctg5s8dp","rate":"0.05","sample":["bnyvykm","c20cna6","d1mhw8b","exms5gc","ganq1ns","m1p1n2g","ryhrskg","w264b74","yvkd7xr","ztww0g8"],"seed":"3vORmZaL4zpWgtbZM29fKrWxgKOHhtXspIkutCyqgZgWUaBvmYCQfHtmCID+YNW2M+LIr8uMlpq/LD5cIp0omQRDdKSTIdrxoCxQHFgcwg0="},"rev":12,"root":"1819b7c2f0e597c440ac79096630f9e2e3cf915bedde34a0dbacae09e4ffee13","scope":"qa:y2zw8ctg5s8dp","signature":{"key":"ben","scheme":"ed25519","value":"iQZEJAY4djubmhFAFBq/wu4N0zoZly8ahIuVwQtra10NCxqSce2v+NCxu8dj+5xTWuOObdUHFVnbAVDdRBFJDQ=="}} {"at":"2026-10-12T09:00:00Z","by":"ben","doc":"hale2026harmonic-damb","evidence":[{"kind":"crop","node":"b5gd0h1","sha256":"a0b6f761fe95f8f29ef6dc37901fa85c7a62857b50b0163ca67b0aab048e4d17"},{"kind":"crop","node":"ec5jc93","sha256":"321a926653b9e097987d255d9286880893d71abf895d73f1649403ed9ac0877d"},{"kind":"crop","node":"hca6dc5","sha256":"a26619c3a9824bc19fddcf89ee618d9a4792d69110439f121eb717b6f98e88a6"},{"kind":"crop","node":"j6y45v6","sha256":"e689deda15dde9ad24433eb946200417473d273a1bf3136df67bf572256a483f"},{"kind":"crop","node":"jv1741a","sha256":"de60579bacfab769f614fa74d26d20c7b1aa02caf60af1479dc46b98ddcbc106"},{"kind":"crop","node":"m1p1n2g","sha256":"369c69758d2d09cc3d42b2424723c4fbc75f9ceb6817da81ce183be7d36f045f"},{"kind":"crop","node":"tkxsmts","sha256":"5d8e205660c2c59dcaff119b5671ac11650974b5287d7d7b239a9118e9171a85"},{"kind":"crop","node":"w0gr97q","sha256":"588a38407463e5674714899e6cf359810ccfed98fefdd2c1a90e8cb2ed0a9d64"},{"kind":"crop","node":"x3ns64d","sha256":"6bd7423a5933eb89572ded1d40d2f5baf6f6fa2f3c46791de8210f201a94b9d1"},{"kind":"crop","node":"ztww0g8","sha256":"902e8ca35e1a313a81130c4c100aeb2233fd53854677b3a992fb16f744fe3151"}],"format":"scienting-receipt/1","id":"ksx5sc327xp97","level":"F3","nodes":[["b5gd0h1","9635a750ac2acf526a4db28746fd3ed84659f4e384f5a73fef85d002b4e9308b"],["ec5jc93","5421e9cf0ef3a995792963d0371b79e11a8efad519b02efb5cb88311a896d0cc"],["hca6dc5","8cfed5a34aab28517e89e8a164f0f96c14361de83ebc800439116e83ef479f34"],["j6y45v6","00c4b92e220303ebc2bfb36aa557ac80ae4dba1675e9ed47adb82f6aeff10634"],["jv1741a","f644190803b5ea36aa24738e7f168a7fdf0d05cb5123c14a814b7628dab71f1e"],["m1p1n2g","0847b5d75c2259257c3619e761c64d5571d3ca3932e96a96ee07e8ce68cc63be"],["tkxsmts","cc0bf73dd84e298da0d36c4b363c8d2bdc9b6d5738d1923d14c4a9028dca08d6"],["w0gr97q","f738a008336ac59b2f1d45cda2f045dc96d34f0f1378c431ce6ff8e434feb8a7"],["x3ns64d","6bfc0a055148d877dbaf28b7f62565de7b6926137bf032c382282c39a4876797"],["ztww0g8","bda85639faacb77e504c891c035012602836171c009ecfd21584de4d6461ba93"]],"qa":{"errors":[],"key":"ben-spot","minimum":10,"of":"fhhvg9qrzqp0n","rate":"0.05","sample":["b5gd0h1","ec5jc93","hca6dc5","j6y45v6","jv1741a","m1p1n2g","tkxsmts","w0gr97q","x3ns64d","ztww0g8"],"seed":"28DienSItno55u/F/YFtP1uuY8zgyDpx/5AzTw98mEzXjKJ3MnG6NgX5c7FDY3iSYkuKqRGUkKmy3eRt7AWoCIQFQW6prIZQnDC4+slwKgE="},"rev":12,"root":"acdb858e72b9911e01809b400e3c6e8a5c8fd0d8da808ebd4078e7ea60db0865","scope":"qa:fhhvg9qrzqp0n","signature":{"key":"ben","scheme":"ed25519","value":"e3p1wQgF7KCKJunNOT7zersemSK3UXzYxJolH6yCZzLfBenpPi+937Uz+10x3gG/+/9WqF33lLDn+xFLEeKEAQ=="}}
{"at":"2026-10-12T09:30:00Z","by":"ben","doc":"hale2026harmonic-damb","evidence":[{"kind":"crop","node":"caehhp6","sha256":"03932451473e6756936d196da5db53b9027220ab90402f32c212ed02a3e0cb69"},{"kind":"crop","node":"nms5qjg","sha256":"87a410886dac6569211ac21d5787953e6cb538b691d1cee916eb9ae72061ebba"},{"kind":"crop","node":"z86b2za","sha256":"3bd36b4ee0d38edfcd3e8feb546e24693cd8034e6ce3c11f01d2be8871f38c76"}],"format":"scienting-receipt/1","id":"yt70fcc3qm0xb","level":"F3","nodes":[["caehhp6","9f9c9a412cd08d0634daa3f23c63f5125c1270c13834c45c045c1a9a7452af86"],["nms5qjg","e2f0b1d093c2f38f329c16edae255aed6e913d489e730a9d4857a3c3b5040f63"],["z86b2za","9a1c438b0f413d63b7e9f1ed7037cc6ad1665efd335baee2ef31d3d232c79552"]],"qa":{"errors":[{"kind":"word","node":"caehhp6","note":"The page prints a period the reading lacks."}],"key":"ben-spot","minimum":10,"of":"gx2s84z9jf9xv","rate":"0.05","sample":["caehhp6","nms5qjg","z86b2za"],"seed":"OGWvNMD0Wsd3kUTzvby0kVC6P1h4UKOMD+vVidd489IfuQIsp68CVEYuGwovPKNdHuIvBCYZL+2WoHbY5ClR8yH0r4t4JXV+2JGsGPbb+gs="},"rev":12,"root":"1d4c44e1018708ff37d9e4b6b722b55388f627a2a9d00c32ea4dcbb0dad4a976","scope":"qa:gx2s84z9jf9xv","signature":{"key":"ben","scheme":"ed25519","value":"9j18am+VrPqloaPVfPZoh0nbyGB4d1rWDwQdsb+NjD3+TMjNQbcVv0CRtgwWCLTw9ibMtk4XNRqDN7ImqpV0CA=="}} {"at":"2026-10-12T09:30:00Z","by":"ben","doc":"hale2026harmonic-damb","evidence":[{"kind":"crop","node":"caehhp6","sha256":"03932451473e6756936d196da5db53b9027220ab90402f32c212ed02a3e0cb69"},{"kind":"crop","node":"nms5qjg","sha256":"87a410886dac6569211ac21d5787953e6cb538b691d1cee916eb9ae72061ebba"},{"kind":"crop","node":"z86b2za","sha256":"3bd36b4ee0d38edfcd3e8feb546e24693cd8034e6ce3c11f01d2be8871f38c76"}],"format":"scienting-receipt/1","id":"yt70fcc3qm0xb","level":"F3","nodes":[["caehhp6","9f9c9a412cd08d0634daa3f23c63f5125c1270c13834c45c045c1a9a7452af86"],["nms5qjg","e2f0b1d093c2f38f329c16edae255aed6e913d489e730a9d4857a3c3b5040f63"],["z86b2za","9a1c438b0f413d63b7e9f1ed7037cc6ad1665efd335baee2ef31d3d232c79552"]],"qa":{"errors":[{"kind":"word","node":"caehhp6","note":"The page prints a period the reading lacks."}],"key":"ben-spot","minimum":10,"of":"gx2s84z9jf9xv","rate":"0.05","sample":["caehhp6","nms5qjg","z86b2za"],"seed":"OGWvNMD0Wsd3kUTzvby0kVC6P1h4UKOMD+vVidd489IfuQIsp68CVEYuGwovPKNdHuIvBCYZL+2WoHbY5ClR8yH0r4t4JXV+2JGsGPbb+gs="},"rev":12,"root":"1d4c44e1018708ff37d9e4b6b722b55388f627a2a9d00c32ea4dcbb0dad4a976","scope":"qa:gx2s84z9jf9xv","signature":{"key":"ben","scheme":"ed25519","value":"9j18am+VrPqloaPVfPZoh0nbyGB4d1rWDwQdsb+NjD3+TMjNQbcVv0CRtgwWCLTw9ibMtk4XNRqDN7ImqpV0CA=="}}
fixtures/records/checkpoint.json
+1 -1
View File
@@ -1 +1 @@
{"format":"scienting-checkpoint/1","note":"scient.ing/log/smithlab\n9\nGa0n103p3Gl136z6XWmM92kGvsvMYGXlk+ijjX4KWiA=\n\n— ana dBv7wtCAdNnczkCO5KwtrmDvtWUVPRusBSuLPeBDQoLZi9Hm34IHuNjvnjyPEar5afNf7xqcDtokk2XZEP31uJMVuwE=\n— wit h7ghDsJGkhWLaPO7+TIZQUenslhiQh/E/TKWU4ddphpFcuylQBZM2kG/o8H1nyPfUT0MkbvnFqPFfHPH3OaN5W1EwAI=\n","origin":"scient.ing/log/smithlab","root":"19ad27d74de9dc6975dfacfa5d698cf76906becbcc6065e593e8a38d7e0a5a20","signatures":[{"key_hash":"741bfbc2","name":"ana","value":"0IB02dzOQI7krC2uYO+1ZRU9G6wFK4s94ENCgtmL0ebfgge42O+ePI8Rqvlp81/vGpwO2iSTZdkQ/fW4kxW7AQ=="},{"key_hash":"87b8210e","name":"wit","value":"wkaSFYto87v5MhlBR6eyWGJCH8T9MpZTh12mGkVy7KVAFkzaQb+jwfWfI99RPQyRu+cWo8V8c8fc5o3lbUTAAg=="}],"size":9} {"format":"scienting-checkpoint/1","note":"scient.ing/log/smithlab\n9\nWQxnIeoDxqOJ/rIz5b0uJ3pfpq4S1kUQMNvbG55bPEM=\n\n— ana dBv7wrLxQN8XFJOwKZO4L98UosXwyw3jVbYTM3a+dZWQdtZkPqd6fM8LHwgbIyBT+OEq18meE0JkqJKoYeZ2DMP7eAo=\n— wit h7ghDsbQuWAI5VIGw+jiEAMHeSpDVlubOR46YIEuxL5TWkxU1xAtrp4gwgKxCV+q72A9mZC6APgyaCAUi78euiV/SA8=\n","origin":"scient.ing/log/smithlab","root":"590c6721ea03c6a389feb233e5bd2e277a5fa6ae12d6451030dbdb1b9e5b3c43","signatures":[{"key_hash":"741bfbc2","name":"ana","value":"svFA3xcUk7Apk7gv3xSixfDLDeNVthMzdr51lZB21mQ+p3p8zwsfCBsjIFP44SrXyZ4TQmSokqhh5nYMw/t4Cg=="},{"key_hash":"87b8210e","name":"wit","value":"xtC5YAjlUgbD6OIQAwd5KkNWW5s5HjpggS7EvlNaTFTXEC2uniDCArEJX6rvYD2ZkLoA+DJoIBSLvx66JX9IDw=="}],"size":9}
fixtures/records/layer/annotations.jsonl
+10 -8
View File
@@ -1,10 +1,12 @@
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"audience":"novice","format":"text/markdown","kind":"gloss","language":"en","type":"TextualBody","value":"A *harmonic* term treats a bond like a spring: the energy grows with the square of the stretch."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/f4s8pc1hdhd9d","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"describing","receipt":"r10e5mzvw6tja","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"rr20tev","node_hash":"76e06b4811a688357ac4b912172283463cfaf1b7a1f0795d274fc733a2309ba7","own_hash":"2d7df7b8b233eab6f96d5db0d29c99062aaa94b55cf4e388c49b9995ce7548d5","quote":{"exact":"bond force constant","prefix":"rm{eq}})^2$$ where $K_r$ is the ","suffix":" and $r_\\mathrm{eq}$ the equilib"},"rev":12,"span":[111,130],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"bond force constant","prefix":"rm{eq}})^2$$ where $K_r$ is the ","suffix":" and $r_\\mathrm{eq}$ the equilib","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"audience":"novice","format":"text/markdown","kind":"gloss","language":"en","type":"TextualBody","value":"A *harmonic* term treats a bond like a spring: the energy grows with the square of the stretch."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/f4s8pc1hdhd9d","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"describing","receipt":"w9j3dwgydy2e0","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"rr20tev","node_hash":"76e06b4811a688357ac4b912172283463cfaf1b7a1f0795d274fc733a2309ba7","own_hash":"2d7df7b8b233eab6f96d5db0d29c99062aaa94b55cf4e388c49b9995ce7548d5","quote":{"exact":"bond force constant","prefix":"rm{eq}})^2$$ where $K_r$ is the ","suffix":" and $r_\\mathrm{eq}$ the equilib"},"rev":12,"span":[111,130],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"bond force constant","prefix":"rm{eq}})^2$$ where $K_r$ is the ","suffix":" and $r_\\mathrm{eq}$ the equilib","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"}
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"claim-link","pin":"illustrative-pin","role":"states","source":"lode:amber/k-bond","type":"SpecificResource"},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/nzwtq23khdx4s","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"linking","receipt":"r10e5mzvw6tja","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"rr20tev","node_hash":"76e06b4811a688357ac4b912172283463cfaf1b7a1f0795d274fc733a2309ba7","own_hash":"2d7df7b8b233eab6f96d5db0d29c99062aaa94b55cf4e388c49b9995ce7548d5","quote":{"exact":"The bond energy is given by $$E_{\\mathrm{bond}} = \\frac{1}{2} K_r (r - r_{\\mathrm{eq}})^2$$ where $K_r$ is the bond force constant and $r_\\mathrm{eq}$ the equilibrium bond length."},"rev":12,"store":"smithlab"},"type":"AnchorSelector"},{"exact":"The bond energy is given by $$E_{\\mathrm{bond}} = \\frac{1}{2} K_r (r - r_{\\mathrm{eq}})^2$$ where $K_r$ is the bond force constant and $r_\\mathrm{eq}$ the equilibrium bond length.","prefix":"","suffix":"","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"claim-link","pin":"claim:272XTX7C@02f47b","role":"states","source":"claim:272XTX7C","type":"SpecificResource"},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/m2vbd164hfkc3","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"linking","receipt":"w9j3dwgydy2e0","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"rr20tev","node_hash":"76e06b4811a688357ac4b912172283463cfaf1b7a1f0795d274fc733a2309ba7","own_hash":"2d7df7b8b233eab6f96d5db0d29c99062aaa94b55cf4e388c49b9995ce7548d5","quote":{"exact":"The bond energy is given by $$E_{\\mathrm{bond}} = \\frac{1}{2} K_r (r - r_{\\mathrm{eq}})^2$$ where $K_r$ is the bond force constant and $r_\\mathrm{eq}$ the equilibrium bond length."},"rev":12,"store":"smithlab"},"type":"AnchorSelector"},{"exact":"The bond energy is given by $$E_{\\mathrm{bond}} = \\frac{1}{2} K_r (r - r_{\\mathrm{eq}})^2$$ where $K_r$ is the bond force constant and $r_\\mathrm{eq}$ the equilibrium bond length.","prefix":"","suffix":"","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"}
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"comment","type":"TextualBody","value":"The step size matters more than the text admits."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/m9e0g8r5ebnkf","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"commenting","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"xjns57z","node_hash":"42b1de7852cdc68730984fff789fe248ecd5b43198c0e7d671931ff3fe561dc3","own_hash":"4dc28d1ad57c74d044f87264dfbc744ae9d5e9644638db976dbc346361442486","quote":{"exact":"steps of 0.01","prefix":"Each bond was scanned in ","suffix":" Å around its minimum, and the e"},"rev":12,"span":[25,38],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"steps of 0.01","prefix":"Each bond was scanned in ","suffix":" Å around its minimum, and the e","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"category":"critique","kind":"comment","type":"TextualBody","value":"The step size matters more than the text admits."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/sxj4eq8415s92","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"commenting","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"xjns57z","node_hash":"42b1de7852cdc68730984fff789fe248ecd5b43198c0e7d671931ff3fe561dc3","own_hash":"4dc28d1ad57c74d044f87264dfbc744ae9d5e9644638db976dbc346361442486","quote":{"exact":"steps of 0.01","prefix":"Each bond was scanned in ","suffix":" Å around its minimum, and the e"},"rev":12,"span":[25,38],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"steps of 0.01","prefix":"Each bond was scanned in ","suffix":" Å around its minimum, and the e","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"}
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"for":"group","kind":"question","type":"TextualBody","value":"Why fit to a scan first and to frequencies second?"},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/x0emxkxem54hw","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"questioning","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"section","node":"d400rz3","node_hash":"e5a8eeb3c112f2dc3341da8f433540a3fc5a7e7f631e72c4dd36c6933aae808d","own_hash":"c3d50f266b05d7889c001aef16662cecd4e99ac4a2747ec331c1044e6adc8e3e","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"category":"question","for":"group","kind":"question","type":"TextualBody","value":"Why fit to a scan first and to frequencies second?"},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/fsg960szdyy6q","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"questioning","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"section","node":"d400rz3","node_hash":"e5a8eeb3c112f2dc3341da8f433540a3fc5a7e7f631e72c4dd36c6933aae808d","own_hash":"c3d50f266b05d7889c001aef16662cecd4e99ac4a2747ec331c1044e6adc8e3e","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"}
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"citation-context","reason":"value","reference":"vxsdvzq","type":"TextualBody","value":"Cited for the band position; the reference is not in the library.","verdict":"cannot-tell"},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/gbj7nn87ndfd8","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"assessing","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"hca6dc5","node_hash":"8cfed5a34aab28517e89e8a164f0f96c14361de83ebc800439116e83ef479f34","own_hash":"c270571517825ceed31ca912b80ac76203effe47f1c84f8ee8962800d0afc92f","quote":{"exact":"1595","prefix":" case, with a bending band near ","suffix":" cm⁻¹.2"},"rev":12,"span":[71,75],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"1595","prefix":" case, with a bending band near ","suffix":" cm⁻¹.2","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"citation-context","reason":"value","reference":"vxsdvzq","type":"TextualBody","value":"Cited for the band position; the reference is not in the library.","verdict":"cannot-tell"},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/gbj7nn87ndfd8","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"assessing","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"hca6dc5","node_hash":"8cfed5a34aab28517e89e8a164f0f96c14361de83ebc800439116e83ef479f34","own_hash":"c270571517825ceed31ca912b80ac76203effe47f1c84f8ee8962800d0afc92f","quote":{"exact":"1595","prefix":" case, with a bending band near ","suffix":" cm⁻¹.2"},"rev":12,"span":[71,75],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"1595","prefix":" case, with a bending band near ","suffix":" cm⁻¹.2","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"}
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"generator":{"model":{"name":"example-model","parameters":"998b5161e39e01c8bc43ef81ec89141e15b7ea53b2e936707920a6097dbcb2c5","processor":"local","prompt":"9cc338fba7c62db7266f530cb555a5ee36910d41a0120a3fe7fd7de1944d0931","provider":"Example Models (illustrative)","version":"2026-09","weights":"504b2d340045b92c22682b60020c21103aae7ff41172bb632fbe305eb4af9b5e"},"run":"illustrative-run-1"},"kind":"generated","type":"TextualBody","value":"Two panels: an energy scan and its fit residual."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:software:example-model","name":"A model (illustrative)","type":"Software"},"id":"tag:scient.ing,2026:annotation:smithlab/szb6k3fj9z7aw","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"describing","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"figure","node":"v7cx6qs","node_hash":"f76864779843485fa9e2f3cc3cd1c76ad222e1649c56f4cface33bf20b8f33b6","own_hash":"a1e46cb40201a17903dfd6f128db1e707b0e9f5bf45fcc32ec7ca772011edeae","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"generator":{"model":{"name":"example-model","parameters":"998b5161e39e01c8bc43ef81ec89141e15b7ea53b2e936707920a6097dbcb2c5","processor":"local","prompt":"9cc338fba7c62db7266f530cb555a5ee36910d41a0120a3fe7fd7de1944d0931","provider":"Example Models (illustrative)","version":"2026-09","weights":"504b2d340045b92c22682b60020c21103aae7ff41172bb632fbe305eb4af9b5e"},"run":"illustrative-run-1"},"kind":"generated","type":"TextualBody","value":"Two panels: an energy scan and its fit residual."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:software:example-model","name":"A model (illustrative)","type":"Software"},"id":"tag:scient.ing,2026:annotation:smithlab/szb6k3fj9z7aw","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"describing","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"figure","node":"v7cx6qs","node_hash":"f76864779843485fa9e2f3cc3cd1c76ad222e1649c56f4cface33bf20b8f33b6","own_hash":"a1e46cb40201a17903dfd6f128db1e707b0e9f5bf45fcc32ec7ca772011edeae","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"}
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"exercise":{"exercise":"anchor","format":"scienting-exercise/1","key":{"elements":[{"anchors":[{"context":{"column_path":["Set A","$K_r$ (kcal mol⁻¹ Å⁻²)"],"row_path":["Bond parameters","CT–CT"],"table_label":"Table 1"},"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"cell","node":"c20cna6","node_hash":"c8e6a553f5ae1377776e86dde6ee7d48a923dea7de417408841ea33b81563272","own_hash":"45bf6ab427d435f28b82cb2b64d93c502968921507fc9397abd5d5170731c6ab","quote":{"exact":"310.0"},"rev":12,"store":"smithlab"}],"feedback":{"hit":"That is the cell.","miss":"Look in Table 1, row CT–CT."},"points":"1"}]},"prompt":"Find the table cell that gives the Set A bond constant for the CT–CT bond.","rule":{"match":"same-node"}},"kind":"exercise","type":"TextualBody","value":"Find the table cell that gives the Set A bond constant for the CT–CT bond."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/whcypxav2cqbx","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"assessing","receipt":"r10e5mzvw6tja","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"table","node":"hj4s7pb","node_hash":"6c6fba200faf41cd06fa8b64e06e2aa29ff8848568b859ae42d1902596348643","own_hash":"66310a9ab1c92568f4c6cae012e35d4245d4ca4b89d2542b7c2113a31ad379ed","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"exercise":{"exercise":"anchor","format":"scienting-exercise/1","key":{"elements":[{"anchors":[{"context":{"column_path":["Set A","$K_r$ (kcal mol⁻¹ Å⁻²)"],"row_path":["Bond parameters","CT–CT"],"table_label":"Table 1"},"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"cell","node":"c20cna6","node_hash":"c8e6a553f5ae1377776e86dde6ee7d48a923dea7de417408841ea33b81563272","own_hash":"45bf6ab427d435f28b82cb2b64d93c502968921507fc9397abd5d5170731c6ab","quote":{"exact":"310.0"},"rev":12,"store":"smithlab"}],"feedback":{"hit":"That is the cell.","miss":"Look in Table 1, row CT–CT."},"points":"1"}]},"prompt":"Find the table cell that gives the Set A bond constant for the CT–CT bond.","rule":{"match":"same-node"}},"kind":"exercise","type":"TextualBody","value":"Find the table cell that gives the Set A bond constant for the CT–CT bond."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/whcypxav2cqbx","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"assessing","receipt":"w9j3dwgydy2e0","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"table","node":"hj4s7pb","node_hash":"6c6fba200faf41cd06fa8b64e06e2aa29ff8848568b859ae42d1902596348643","own_hash":"66310a9ab1c92568f4c6cae012e35d4245d4ca4b89d2542b7c2113a31ad379ed","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"}
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"exercise":{"exercise":"check","format":"scienting-exercise/1","key":{"elements":[{"anchors":[{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"kea5026","node_hash":"b836d5872ca37471fa996be04c538254a8b1e37ca12e8f1b4a40837b503afa33","own_hash":"d673d236c03d9583ecf6e01fa819765ec324dc684bd9bbc6967fa50f216e34d2","quote":{"exact":"the most common description","prefix":"Harmonic bond terms remain ","suffix":" of covalent stretching in class"},"rev":12,"span":[27,54],"store":"smithlab"}],"points":"1"}],"verdict":{"points":"1","value":"supports"}},"prompt":"Does the cited passage support the claim that harmonic terms are the most common choice?","rule":{"match":"overlap","min_overlap":"0.5"}},"kind":"exercise","type":"TextualBody","value":"Does the cited passage support the claim that harmonic terms are the most common choice?"},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/y7bd3he0jwjbx","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"assessing","receipt":"r10e5mzvw6tja","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"kea5026","node_hash":"b836d5872ca37471fa996be04c538254a8b1e37ca12e8f1b4a40837b503afa33","own_hash":"d673d236c03d9583ecf6e01fa819765ec324dc684bd9bbc6967fa50f216e34d2","quote":{"exact":"Harmonic bond terms remain the most common description of covalent stretching in classical force fields.1"},"rev":12,"store":"smithlab"},"type":"AnchorSelector"},{"exact":"Harmonic bond terms remain the most common description of covalent stretching in classical force fields.1","prefix":"","suffix":"","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"exercise":{"exercise":"check","format":"scienting-exercise/1","key":{"elements":[{"anchors":[{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"kea5026","node_hash":"b836d5872ca37471fa996be04c538254a8b1e37ca12e8f1b4a40837b503afa33","own_hash":"d673d236c03d9583ecf6e01fa819765ec324dc684bd9bbc6967fa50f216e34d2","quote":{"exact":"the most common description","prefix":"Harmonic bond terms remain ","suffix":" of covalent stretching in class"},"rev":12,"span":[27,54],"store":"smithlab"}],"points":"1"}],"verdict":{"points":"1","value":"supports"}},"prompt":"Does the cited passage support the claim that harmonic terms are the most common choice?","rule":{"match":"overlap","min_overlap":"0.5"}},"kind":"exercise","type":"TextualBody","value":"Does the cited passage support the claim that harmonic terms are the most common choice?"},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/y7bd3he0jwjbx","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"assessing","receipt":"w9j3dwgydy2e0","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"kea5026","node_hash":"b836d5872ca37471fa996be04c538254a8b1e37ca12e8f1b4a40837b503afa33","own_hash":"d673d236c03d9583ecf6e01fa819765ec324dc684bd9bbc6967fa50f216e34d2","quote":{"exact":"Harmonic bond terms remain the most common description of covalent stretching in classical force fields.1"},"rev":12,"store":"smithlab"},"type":"AnchorSelector"},{"exact":"Harmonic bond terms remain the most common description of covalent stretching in classical force fields.1","prefix":"","suffix":"","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"}
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"description","language":"en","role":"alt","type":"TextualBody","value":"Energy along the C–C bond stretch (panel a) and the fit residual (panel b)."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/gb5tmv3hq08vz","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"describing","receipt":"r10e5mzvw6tja","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"figure","node":"v7cx6qs","node_hash":"f76864779843485fa9e2f3cc3cd1c76ad222e1649c56f4cface33bf20b8f33b6","own_hash":"a1e46cb40201a17903dfd6f128db1e707b0e9f5bf45fcc32ec7ca772011edeae","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"description","language":"en","role":"alt","type":"TextualBody","value":"Energy along the C–C bond stretch (panel a) and the fit residual (panel b)."},"created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/gb5tmv3hq08vz","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"describing","receipt":"w9j3dwgydy2e0","status":"confirmed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"figure","node":"v7cx6qs","node_hash":"f76864779843485fa9e2f3cc3cd1c76ad222e1649c56f4cface33bf20b8f33b6","own_hash":"a1e46cb40201a17903dfd6f128db1e707b0e9f5bf45fcc32ec7ca772011edeae","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"}
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"comment","type":"TextualBody","value":"Agreed; see the residual panel."},"created":"2026-10-11T19:05:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/j3vr50xm97cp7","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"replying","status":"proposed","target":{"source":"tag:scient.ing,2026:annotation:smithlab/m9e0g8r5ebnkf","type":"SpecificResource"},"type":"Annotation"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"category":"comment","kind":"comment","type":"TextualBody","value":"Agreed; see the residual panel."},"created":"2026-10-11T19:05:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/t5cv4rec7j0mt","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"replying","status":"proposed","target":{"source":"tag:scient.ing,2026:annotation:smithlab/sxj4eq8415s92","type":"SpecificResource"},"type":"Annotation"}
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"category":"context","kind":"caveat","type":"TextualBody","value":"The band position is cited, not measured here; reference 2 is not in the library."},"created":"2026-10-11T19:10:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/g7ngzk3jkdy40","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"commenting","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"hca6dc5","node_hash":"8cfed5a34aab28517e89e8a164f0f96c14361de83ebc800439116e83ef479f34","own_hash":"c270571517825ceed31ca912b80ac76203effe47f1c84f8ee8962800d0afc92f","quote":{"exact":"1595","prefix":" case, with a bending band near ","suffix":" cm⁻¹.2"},"rev":12,"span":[71,75],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"1595","prefix":" case, with a bending band near ","suffix":" cm⁻¹.2","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"}
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"key":"K_{r}","kind":"symbol","pin":"claim:272XTX7C@02f47b","source":"claim:272XTX7C","symbol":"K_r","type":"SpecificResource"},"created":"2026-10-11T19:15:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/f2603vdj2e3tr","layer":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","motivation":"identifying","status":"proposed","target":{"selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"rr20tev","node_hash":"76e06b4811a688357ac4b912172283463cfaf1b7a1f0795d274fc733a2309ba7","own_hash":"2d7df7b8b233eab6f96d5db0d29c99062aaa94b55cf4e388c49b9995ce7548d5","quote":{"exact":"$K_r$","prefix":"(r - r_{\\mathrm{eq}})^2$$ where ","suffix":" is the bond force constant and "},"rev":12,"span":[98,103],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"$K_r$","prefix":"(r - r_{\\mathrm{eq}})^2$$ where ","suffix":" is the bond force constant and ","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"type":"Annotation"}
fixtures/records/layer/layer.json
+1 -1
View File
@@ -1 +1 @@
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"audience":"student","created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"docs":["smithlab/hale2026harmonic-damb"],"format":"scienting-layer/1","id":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","label":"Smith lab reading of Hale and Osei","license":{"spdx":"CC-BY-4.0"},"project":"smithlab-charges","purpose":"Glosses, claim links and exercises for new students.","total":10,"type":"AnnotationCollection","visibility":"team"} {"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"audience":"student","created":"2026-10-11T19:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"docs":["smithlab/hale2026harmonic-damb"],"format":"scienting-layer/1","id":"tag:scient.ing,2026:layer:smithlab/kp4m2x7q","label":"Smith lab reading of Hale and Osei","license":{"spdx":"CC-BY-4.0"},"project":"smithlab-charges","purpose":"Glosses, claim links and exercises for new students.","total":12,"type":"AnnotationCollection","visibility":"team"}
fixtures/records/layer/receipt.json
+1 -1
View File
@@ -1 +1 @@
{"at":"2026-10-11T19:30:00Z","by":"ana","doc":"hale2026harmonic-damb","evidence":[],"format":"scienting-receipt/1","id":"r10e5mzvw6tja","level":"F3","nodes":[],"records":[["annotation","f4s8pc1hdhd9d","9db553d2150a912e4ba3027d002706e08ccaded843d3e49c79b95a2e0581b6f1"],["annotation","gb5tmv3hq08vz","3b776d37af21d1bdfd3a0130c9bfa26affa484fb4d76e41d83cbd8bb02b4975c"],["annotation","nzwtq23khdx4s","8ea94b7a3d266f51e11dd31399784019acd1b1b53ed093e15c199417327992ab"],["annotation","whcypxav2cqbx","11845db3ef685b0b9bd13895cc5b8ade5b345ee7d72737a630be60d690a2ba5d"],["annotation","y7bd3he0jwjbx","0286ede04d8bf548432e34dd7beeee0fb044bd43022a8661a292018dd2ef5578"]],"rev":12,"root":"246581b0926273458b78ece15b465f0567c150a1e071f72322ce41895f9502e1","scope":"targeted:layer","signature":{"key":"ana","scheme":"ed25519","value":"WMDdARYwB0ETltF57ks1MxJ6ny/c4XziLvYUBd8PoW7MwcfhPKmDTqNGxRNg4z5g4c/bHnWS4ftZLJiONTL/Cg=="}} {"at":"2026-10-11T19:30:00Z","by":"ana","doc":"hale2026harmonic-damb","evidence":[],"format":"scienting-receipt/1","id":"w9j3dwgydy2e0","level":"F3","nodes":[],"records":[["annotation","f4s8pc1hdhd9d","9db553d2150a912e4ba3027d002706e08ccaded843d3e49c79b95a2e0581b6f1"],["annotation","gb5tmv3hq08vz","3b776d37af21d1bdfd3a0130c9bfa26affa484fb4d76e41d83cbd8bb02b4975c"],["annotation","m2vbd164hfkc3","47f154185dc5440157d81e93779b60dca9d58d6a9a35e2b4da2e1a1083d7e276"],["annotation","whcypxav2cqbx","11845db3ef685b0b9bd13895cc5b8ade5b345ee7d72737a630be60d690a2ba5d"],["annotation","y7bd3he0jwjbx","0286ede04d8bf548432e34dd7beeee0fb044bd43022a8661a292018dd2ef5578"]],"rev":12,"root":"9c6936a337abfd0732a8629568aef125953041fa9d2403991b7a47bef35b604e","scope":"targeted:layer","signature":{"key":"ana","scheme":"ed25519","value":"huZ9eQ3PZfC9Kq/yXtvIhdK9FGgPVH7WVQn+YHFgOVXPDCSSjlpLr+eqsHEdl0XsNM/ku8uXS1XiZiRzSWitBg=="}}
fixtures/records/log/checkpoint
+3 -3
View File
@@ -1,6 +1,6 @@
scient.ing/log/smithlab scient.ing/log/smithlab
9 9
Ga0n103p3Gl136z6XWmM92kGvsvMYGXlk+ijjX4KWiA= WQxnIeoDxqOJ/rIz5b0uJ3pfpq4S1kUQMNvbG55bPEM=
— ana dBv7wtCAdNnczkCO5KwtrmDvtWUVPRusBSuLPeBDQoLZi9Hm34IHuNjvnjyPEar5afNf7xqcDtokk2XZEP31uJMVuwE= — ana dBv7wrLxQN8XFJOwKZO4L98UosXwyw3jVbYTM3a+dZWQdtZkPqd6fM8LHwgbIyBT+OEq18meE0JkqJKoYeZ2DMP7eAo=
— wit h7ghDsJGkhWLaPO7+TIZQUenslhiQh/E/TKWU4ddphpFcuylQBZM2kG/o8H1nyPfUT0MkbvnFqPFfHPH3OaN5W1EwAI= — wit h7ghDsbQuWAI5VIGw+jiEAMHeSpDVlubOR46YIEuxL5TWkxU1xAtrp4gwgKxCV+q72A9mZC6APgyaCAUi78euiV/SA8=
fixtures/records/log/tile/0/000.p/9
BIN
View File
Binary file not shown.
fixtures/records/log/tile/entries/000.p/9
+3 -3
View File
@@ -1,9 +1,9 @@
["receipt","npc15pn6675z5","37b3025b54c6397e5f8d400912b5788cd6ce203059cd9143e7d521c2cdbf2917"] ["receipt","vskq18t5sxydz","7fccee1468b9ef9bf712279f10d13f5544abb79db03e33b122a2504a3cf25789"]
["receipt","y2zw8ctg5s8dp","ae17f8866a05ca1b671d96b7758984e11ac694acd6743835f107cb513c7cc600"] ["receipt","fhhvg9qrzqp0n","f78c7704df1fbd815ac3ccf9114c82a189f9032ffcd75023e10b27b924f406b5"]
["receipt","mvjmb94nz7rdh","a4dca8b492bf3e1b190f22e09de77713523e345526fd1b01c1186d0fe80d1cf4"] ["receipt","mvjmb94nz7rdh","a4dca8b492bf3e1b190f22e09de77713523e345526fd1b01c1186d0fe80d1cf4"]
["receipt","q465jpwz8afhe","39218b2b73e853e2e66d8d4ce68cf7e155ca001f05ee4da5942a8d9b63b216bc"] ["receipt","q465jpwz8afhe","39218b2b73e853e2e66d8d4ce68cf7e155ca001f05ee4da5942a8d9b63b216bc"]
["revocation","pcbhtpwvysgq6","9062e3ab737ecc2e6d487d2076ee27f764be60f06ad01f1ad103d708d466e56a"] ["revocation","pcbhtpwvysgq6","9062e3ab737ecc2e6d487d2076ee27f764be60f06ad01f1ad103d708d466e56a"]
["receipt","gx2s84z9jf9xv","5ee8b2827d327a7bbcacb08fd19b9523fad83ebf6c29350d5ce7ca30bf5ce388"] ["receipt","gx2s84z9jf9xv","5ee8b2827d327a7bbcacb08fd19b9523fad83ebf6c29350d5ce7ca30bf5ce388"]
["receipt","mg18668nzxvap","4c80506322bfeed5649cfab303959e4ccf0aa0e594995ee6fd5ad59f9d3ee6a7"] ["receipt","ksx5sc327xp97","1fcf4b960c47ed927b2f9599bc7e54221e35b2e1c001d46d92fc18505f219395"]
["receipt","yt70fcc3qm0xb","2ad1c0f63077a03ab1cce01e6235734577d0b072b90ebab2f7db7fd481ccee2c"] ["receipt","yt70fcc3qm0xb","2ad1c0f63077a03ab1cce01e6235734577d0b072b90ebab2f7db7fd481ccee2c"]
["roster","smithlab","782c5e68f4e4f9918e69b7a44b108d9d70c89a289981acff3d25300ec8373c15"] ["roster","smithlab","782c5e68f4e4f9918e69b7a44b108d9d70c89a289981acff3d25300ec8373c15"]
fixtures/records/project.json
+1 -1
View File
@@ -1 +1 @@
{"claims":{"lode":"amber"},"datasets":["ds-hale-bond-params","ds-hale-ctct-scan"],"documents":[{"doc":"smithlab/hale2026harmonic-damb","reason":"The bond constants we start from.","status":{"Ben Example":"read","Cara Example":"reading"},"tags":["parameters"]},{"doc":"okafor2025solvent-acth","reason":"Solvent shifts for the validation set.","tags":["spectra"]}],"drafts":["smithlab/writer2027fixed-ynr7.v3"],"format":"scienting-project/1","key":"smithlab-charges","layers":["tag:scient.ing,2026:layer:smithlab/kp4m2x7q"],"members":[{"keys":["ana"],"person":"Ana Example","role":"owner"},{"keys":["ben","ben-spot"],"person":"Ben Example","role":"editor"},{"keys":["cara"],"person":"Cara Example","role":"reader"}],"name":"Charges and bonds","paths":[{"id":"onboarding","steps":[{"note":"Read the introduction.","ref":"smithlab/hale2026harmonic-damb","type":"document"},{"note":"Then the table.","ref":"smithlab/hale2026harmonic-damb#hj4s7pb","type":"node"},{"ref":"lode:amber/k-bond","type":"claim"}],"title":"First week"}],"settings":{"digit_rule":"anchored-sections","drift_threshold":"0.25","log":{"origin":"scient.ing/log/smithlab","threshold":1,"witnesses":["wit"]},"rights_unknown":"refuse","spot_check":{"minimum":10,"rate":"0.05"},"stages":"scienting-stages-default"},"store":"smithlab","views":["ctct-scan"]} {"claims":{"paperclaim":"amber"},"datasets":["ds-hale-bond-params","ds-hale-ctct-scan"],"documents":[{"doc":"smithlab/hale2026harmonic-damb","reason":"The bond constants we start from.","status":{"Ben Example":"read","Cara Example":"reading"},"tags":["parameters"]},{"doc":"okafor2025solvent-acth","reason":"Solvent shifts for the validation set.","tags":["spectra"]}],"drafts":["smithlab/writer2027fixed-ynr7.v3"],"format":"scienting-project/1","key":"smithlab-charges","layers":["tag:scient.ing,2026:layer:smithlab/kp4m2x7q","tag:scient.ing,2026:layer:smithlab/tr6c3q8m"],"members":[{"keys":["ana"],"person":"Ana Example","role":"owner"},{"keys":["ben","ben-spot"],"person":"Ben Example","role":"editor"},{"keys":["cara"],"person":"Cara Example","role":"reader"}],"name":"Charges and bonds","paths":[{"id":"onboarding","steps":[{"note":"Read the introduction.","ref":"smithlab/hale2026harmonic-damb","type":"document"},{"note":"Then the table.","ref":"smithlab/hale2026harmonic-damb#hj4s7pb","type":"node"},{"note":"The bond force constant, amber/k-bond.","ref":"claim:272XTX7C","type":"claim"}],"title":"First week"}],"settings":{"digit_rule":"anchored-sections","drift_threshold":"0.25","log":{"origin":"scient.ing/log/smithlab","threshold":1,"witnesses":["wit"]},"rights_unknown":"refuse","spot_check":{"minimum":10,"rate":"0.05"},"stages":"scienting-stages-default"},"store":"smithlab","views":["ctct-scan"]}
fixtures/records/proof-inclusion.json
+1 -1
View File
@@ -1 +1 @@
{"format":"scienting-proof/1","index":3,"item":["node","c20cna6","c8e6a553f5ae1377776e86dde6ee7d48a923dea7de417408841ea33b81563272"],"kind":"inclusion","path":["1344d74efcc8182f2a1ac7501fc4a3b888bfd753b1cc9c4c02562d7d51b91a1d","be61b225edd17fe256a73b79b26c2be1e2cc43d370ba4c9e7f0536b12c305650","abe0ffd73517576e3c64411f54f854fb008b13a828695d1f9e3059cfb9bfd2cd","0497554cb70abf31d1bf919cf74b62652e21a5c4ed1875c951971ddada26bc3e","c352582a32865ce563529803845821c0227a85e06490dc069cb945a80fc9cfa9"],"receipt":"y2zw8ctg5s8dp","root":"373a7d860a14b58e5bfa9b9ad3bc916091bcb6db7556f0dbc9c71972a612494d","size":23} {"format":"scienting-proof/1","index":3,"item":["node","c20cna6","c8e6a553f5ae1377776e86dde6ee7d48a923dea7de417408841ea33b81563272"],"kind":"inclusion","path":["1344d74efcc8182f2a1ac7501fc4a3b888bfd753b1cc9c4c02562d7d51b91a1d","be61b225edd17fe256a73b79b26c2be1e2cc43d370ba4c9e7f0536b12c305650","abe0ffd73517576e3c64411f54f854fb008b13a828695d1f9e3059cfb9bfd2cd","0497554cb70abf31d1bf919cf74b62652e21a5c4ed1875c951971ddada26bc3e","c352582a32865ce563529803845821c0227a85e06490dc069cb945a80fc9cfa9"],"receipt":"fhhvg9qrzqp0n","root":"373a7d860a14b58e5bfa9b9ad3bc916091bcb6db7556f0dbc9c71972a612494d","size":23}
fixtures/records/qa-report.json
+1 -1
View File
@@ -1 +1 @@
{"at":"2026-10-12T10:00:00Z","by":"ana","delivery":"smithlab evidence pilot (fictional)","docs":["smithlab/hale2026harmonic-damb"],"format":"scienting-qa-report/1","id":"b6dt4h4pv0x98","receipts":["y2zw8ctg5s8dp","gx2s84z9jf9xv"],"reviewers":[{"bound":"0.316340","confidence":"0.95","errors":1,"key":"ana","method":"clopper-pearson","sampled":13}],"signature":{"key":"ana","scheme":"ed25519","value":"7nFBiJ2csiX8Y0lgn3Tp5+7IKlMUMRfVEN3B3pdrgw3rCdEG2VXJaafoJ2lClb19Io2bMpfww3k0epkKfef/Dw=="}} {"at":"2026-10-12T10:00:00Z","by":"ana","delivery":"smithlab evidence pilot (fictional)","docs":["smithlab/hale2026harmonic-damb"],"format":"scienting-qa-report/1","id":"whqq47rxzzkp6","receipts":["fhhvg9qrzqp0n","gx2s84z9jf9xv"],"reviewers":[{"bound":"0.316340","confidence":"0.95","errors":1,"key":"ana","method":"clopper-pearson","sampled":13}],"signature":{"key":"ana","scheme":"ed25519","value":"mLB9WGnX0DbRzug2n946waD7c7oWi+MAR4TB1n/ezpeflZBMaP5onaH9vtkm4ztI13rb/Ej04IDIUpBonTq9AA=="}}
fixtures/records/traces/annotations.jsonl
+4
View File
@@ -0,0 +1,4 @@
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"trace","note":"The abstract states the two steps: a scan first, then vibrational frequencies.","proposer":{"model":{"name":"example-model","parameters":"998b5161e39e01c8bc43ef81ec89141e15b7ea53b2e936707920a6097dbcb2c5","processor":"local","prompt":"9cc338fba7c62db7266f530cb555a5ee36910d41a0120a3fe7fd7de1944d0931","provider":"Example Models (illustrative)","version":"2026-09","weights":"504b2d340045b92c22682b60020c21103aae7ff41172bb632fbe305eb4af9b5e"},"rank":1,"run":"illustrative-run-2","score":"0.8731"},"reference":"ebcq5cy","selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"mhp0at8","node_hash":"ee42c4780724a411697e3a95daf182d37c792d4689a7172f32929785d5fd968e","own_hash":"24cc437784bf358d639755dbf59984ad2c0c7a0e7f9448b23e965039b67f759b","quote":{"exact":"in two steps","prefix":"bond constants for twelve bonds ","suffix":", first to a scan and then to vi"},"rev":12,"span":[48,60],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"in two steps","prefix":"bond constants for twelve bonds ","suffix":", first to a scan and then to vi","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource","verdict":"supports"},"created":"2026-10-12T09:00:00Z","creator":{"id":"tag:scient.ing,2026:software:example-model","name":"A model (illustrative)","type":"Software"},"id":"tag:scient.ing,2026:annotation:smithlab/jh0e4st2c0kpt","layer":"tag:scient.ing,2026:layer:smithlab/tr6c3q8m","motivation":"linking","receipt":"qvckzaatz5d1v","status":"confirmed","target":{"selector":[{"anchor":{"doc":"writer2027fixed-ynr7.v3","format":"scienting-anchor/1","kind":"sentence","node":"qgh1k31","node_hash":"93a39490870e694299e88e6f475d42c38078d1442959e555bdbdf58488dcf187","own_hash":"e004436e7b28ff3dc41e0056f231ba707d9be985694e68acb911345a547842ed","quote":{"exact":"two-step fit","prefix":"Its constants come from a ","suffix":" (Hale and Osei, Table 1).1"},"rev":0,"span":[26,38],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"two-step fit","prefix":"Its constants come from a ","suffix":" (Hale and Osei, Table 1).1","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/writer2027fixed-ynr7.v3","type":"SpecificResource"},"type":"Annotation"}
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"trace","note":"Says harmonic terms are common, not how the constants were fitted.","proposer":{"model":{"name":"example-model","parameters":"998b5161e39e01c8bc43ef81ec89141e15b7ea53b2e936707920a6097dbcb2c5","processor":"local","prompt":"9cc338fba7c62db7266f530cb555a5ee36910d41a0120a3fe7fd7de1944d0931","provider":"Example Models (illustrative)","version":"2026-09","weights":"504b2d340045b92c22682b60020c21103aae7ff41172bb632fbe305eb4af9b5e"},"rank":2,"run":"illustrative-run-2","score":"0.6410"},"reference":"ebcq5cy","selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"sentence","node":"kea5026","node_hash":"b836d5872ca37471fa996be04c538254a8b1e37ca12e8f1b4a40837b503afa33","own_hash":"d673d236c03d9583ecf6e01fa819765ec324dc684bd9bbc6967fa50f216e34d2","quote":{"exact":"classical force fields.1","prefix":"ption of covalent stretching in "},"rev":12,"span":[81,105],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"classical force fields.1","prefix":"ption of covalent stretching in ","suffix":"","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource"},"created":"2026-10-12T09:00:00Z","creator":{"id":"tag:scient.ing,2026:software:example-model","name":"A model (illustrative)","type":"Software"},"id":"tag:scient.ing,2026:annotation:smithlab/arm3zmj6be9fz","layer":"tag:scient.ing,2026:layer:smithlab/tr6c3q8m","motivation":"linking","status":"rejected","target":{"selector":[{"anchor":{"doc":"writer2027fixed-ynr7.v3","format":"scienting-anchor/1","kind":"sentence","node":"qgh1k31","node_hash":"93a39490870e694299e88e6f475d42c38078d1442959e555bdbdf58488dcf187","own_hash":"e004436e7b28ff3dc41e0056f231ba707d9be985694e68acb911345a547842ed","quote":{"exact":"two-step fit","prefix":"Its constants come from a ","suffix":" (Hale and Osei, Table 1).1"},"rev":0,"span":[26,38],"store":"smithlab"},"type":"AnchorSelector"},{"exact":"two-step fit","prefix":"Its constants come from a ","suffix":" (Hale and Osei, Table 1).1","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/writer2027fixed-ynr7.v3","type":"SpecificResource"},"type":"Annotation"}
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"kind":"trace","note":"Table 1 lists the fitted constants of both parameter sets.","reference":"ebcq5cy","selector":[{"anchor":{"doc":"hale2026harmonic-damb","format":"scienting-anchor/1","kind":"table","node":"hj4s7pb","node_hash":"6c6fba200faf41cd06fa8b64e06e2aa29ff8848568b859ae42d1902596348643","own_hash":"66310a9ab1c92568f4c6cae012e35d4245d4ca4b89d2542b7c2113a31ad379ed","rev":12,"store":"smithlab"},"type":"AnchorSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/hale2026harmonic-damb","type":"SpecificResource","verdict":"supports"},"created":"2026-10-12T09:10:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/fhnynfb8jx0wc","layer":"tag:scient.ing,2026:layer:smithlab/tr6c3q8m","motivation":"linking","receipt":"qvckzaatz5d1v","status":"confirmed","target":{"selector":[{"anchor":{"doc":"writer2027fixed-ynr7.v3","format":"scienting-anchor/1","kind":"sentence","node":"qgh1k31","node_hash":"93a39490870e694299e88e6f475d42c38078d1442959e555bdbdf58488dcf187","own_hash":"e004436e7b28ff3dc41e0056f231ba707d9be985694e68acb911345a547842ed","quote":{"exact":"Its constants come from a two-step fit (Hale and Osei, Table 1).1"},"rev":0,"store":"smithlab"},"type":"AnchorSelector"},{"exact":"Its constants come from a two-step fit (Hale and Osei, Table 1).1","prefix":"","suffix":"","type":"TextQuoteSelector"}],"source":"tag:scient.ing,2026:doc:smithlab/writer2027fixed-ynr7.v3","type":"SpecificResource"},"type":"Annotation"}
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"body":{"category":"critique","kind":"comment","type":"TextualBody","value":"The abstract names the two steps but not the constants; the table is the better passage."},"created":"2026-10-12T10:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ben","name":"Ben Example","type":"Person"},"id":"tag:scient.ing,2026:annotation:smithlab/g4fm8f93czwkk","layer":"tag:scient.ing,2026:layer:smithlab/tr6c3q8m","motivation":"assessing","status":"proposed","target":{"source":"tag:scient.ing,2026:annotation:smithlab/jh0e4st2c0kpt","type":"SpecificResource"},"type":"Annotation"}
fixtures/records/traces/layer.json
+1
View File
@@ -0,0 +1 @@
{"@context":["http://www.w3.org/ns/anno.jsonld","https://paperspec.scient.ing/context/scienting-1.jsonld"],"audience":"expert","created":"2026-10-12T09:00:00Z","creator":{"id":"tag:scient.ing,2026:key:ana","name":"Ana Example","type":"Person"},"docs":["smithlab/writer2027fixed-ynr7.v3"],"format":"scienting-layer/1","id":"tag:scient.ing,2026:layer:smithlab/tr6c3q8m","label":"Where the draft's citations lead","license":{"spdx":"CC-BY-4.0"},"project":"smithlab-charges","purpose":"Traces from the draft's citations of Hale and Osei to the passages they rest on.","total":4,"type":"AnnotationCollection","visibility":"team"}
fixtures/records/traces/receipt.json
+1
View File
@@ -0,0 +1 @@
{"at":"2026-10-12T09:30:00Z","by":"ana","doc":"writer2027fixed-ynr7.v3","evidence":[],"format":"scienting-receipt/1","id":"qvckzaatz5d1v","level":"F3","nodes":[],"records":[["annotation","fhnynfb8jx0wc","a1115243473b190f7ed8877780f29c935f29801d6c9fe0b423601247ebf078f8"],["annotation","jh0e4st2c0kpt","b774528344f096051420b6df73c089fdce4d0e1093e4ea3d845fff023c17dee7"]],"rev":0,"root":"d1b96ae5b9887ae843cd209d7da5ea9bb454f1bab77b93d05d38fbfc2b21e826","scope":"targeted:traces","signature":{"key":"ana","scheme":"ed25519","value":"d2xj2bQY3AoDEVV7yzs31mQpMoGre+V919NUM/Ba/qj+ySb1QKs7D9ZidM6+ksl1xlNsCLNU2rdeclUduoZZBA=="}}
fixtures/src/README.md
-5
View File
@@ -1,11 +1,6 @@
# Fixture sources # Fixture sources
<<<<<<< HEAD
Each source describes a document as a converter first read it, mistakes included, in a shorthand that `paperspec build` expands. Each source describes a document as a converter first read it, mistakes included, in a shorthand that `paperspec build` expands.
||||||| parent of 4fa326d (First implementation (#1))
=======
Each source describes a document as a converter first read it, mistakes included, in a shorthand that `tools/build.py` expands.
>>>>>>> 4fa326d (First implementation (#1))
The build mints IDs, applies the listed corrections, lays out synthetic geometry, signs the listed receipts and writes the package. The build mints IDs, applies the listed corrections, lays out synthetic geometry, signs the listed receipts and writes the package.
The shorthand is tooling, not a format. The shorthand is tooling, not a format.
fixtures/src/hale2026harmonic.json
+2 -2
View File
@@ -153,8 +153,8 @@
"paragraph-moved": {"node": "@para7"} "paragraph-moved": {"node": "@para7"}
}, },
"receipts": [ "receipts": [
{"name": "r0", "by": "ana", "level": "F3", "scope": "targeted:lode", "at": "2026-10-01T16:00:00Z", "rev": 0, "nodes": ["@cc_ak", "@cc_br", "@cc_ar"], "crops": true}, {"name": "r0", "by": "ana", "level": "F3", "scope": "targeted:paperclaim", "at": "2026-10-01T16:00:00Z", "rev": 0, "nodes": ["@cc_ak", "@cc_br", "@cc_ar"], "crops": true},
{"name": "r1", "by": "ana", "level": "F3", "scope": "targeted:lode", "at": "2026-10-11T15:02:11Z", "nodes": ["@cc", "@cc_ak", "@cc_ar", "@cc_bk", "@cc_br", "@ch2", "@ch2_ak", "@ch2_ar", "@ch2_bk", "@ch2_br", "@ow", "@ow_ak", "@ow_ar", "@ow_bk", "@ow_br", "@h_ak", "@h_ar", "@h_bk", "@h_br", "@s2", "@eq1", "@s3", "@s4"], "crops": true}, {"name": "r1", "by": "ana", "level": "F3", "scope": "targeted:paperclaim", "at": "2026-10-11T15:02:11Z", "nodes": ["@cc", "@cc_ak", "@cc_ar", "@cc_bk", "@cc_br", "@ch2", "@ch2_ak", "@ch2_ar", "@ch2_bk", "@ch2_br", "@ow", "@ow_ak", "@ow_ar", "@ow_bk", "@ow_br", "@h_ak", "@h_ar", "@h_bk", "@h_br", "@s2", "@eq1", "@s3", "@s4"], "crops": true},
{"name": "q1", "qa_of": "r1", "by": "ben", "key": "ben-spot", "at": "2026-10-12T09:00:00Z", "errors": []}, {"name": "q1", "qa_of": "r1", "by": "ben", "key": "ben-spot", "at": "2026-10-12T09:00:00Z", "errors": []},
{"name": "r2", "by": "ana", "level": "F3", "scope": "section:@sec1", "at": "2026-10-11T16:30:00Z", "nodes": ["@s1", "@s5"], "crops": true}, {"name": "r2", "by": "ana", "level": "F3", "scope": "section:@sec1", "at": "2026-10-11T16:30:00Z", "nodes": ["@s1", "@s5"], "crops": true},
{"name": "r3", "by": "cara", "level": "F3", "scope": "page:2", "at": "2026-10-11T17:00:00Z", "nodes": ["@s6a", "@s7", "@fns"], "crops": true}, {"name": "r3", "by": "cara", "level": "F3", "scope": "page:2", "at": "2026-10-11T17:00:00Z", "nodes": ["@s6a", "@s7", "@fns"], "crops": true},
pixi.toml
-15
View File
@@ -20,7 +20,6 @@ markdownlint-cli2 = ">=0.23.3"
nodejs = ">=22" nodejs = ">=22"
cryptography = ">=46" cryptography = ">=46"
<<<<<<< HEAD
# The reference implementation in src/paperspec, installed editable so that `import paperspec` and the # The reference implementation in src/paperspec, installed editable so that `import paperspec` and the
# `paperspec` command work in every pixi environment; without pixi, `pip install -e ".[tools]"` does the same. # `paperspec` command work in every pixi environment; without pixi, `pip install -e ".[tools]"` does the same.
[pypi-dependencies] [pypi-dependencies]
@@ -52,17 +51,3 @@ load = { cmd = "paperspec load", description = "Load packages into SQLite: pixi
validate = { cmd = "paperspec validate", description = "Validate packages, or run proposed corrections through the gate" } validate = { cmd = "paperspec validate", description = "Validate packages, or run proposed corrections through the gate" }
stress = { cmd = "paperspec stress", description = "Stress the oracle with random inputs: pixi run stress --seed N" } stress = { cmd = "paperspec stress", description = "Stress the oracle with random inputs: pixi run stress --seed N" }
site = { cmd = "paperspec site", description = "Build the website into _site/" } site = { cmd = "paperspec site", description = "Build the website into _site/" }
||||||| parent of 4fa326d (First implementation (#1))
=======
[tasks]
check = { cmd = "python tools/check.py", depends-on = [
"lint",
], description = "Check schemas, records, packages, vectors, a byte-for-byte rebuild, the SQL round trip, prose and links" }
quick = { cmd = "python tools/check.py --quick", description = "Check everything except the byte-for-byte rebuild" }
build = { cmd = "python tools/build.py", description = "Rebuild fixtures/ and vectors/ from fixtures/src/ and the oracle" }
lint = { cmd = "markdownlint-cli2", description = "Lint every Markdown file with markdownlint" }
load = { cmd = "python tools/load.py", description = "Load packages into SQLite: pixi run load --sqlite lab.db PACKAGE..." }
validate = { cmd = "python tools/validate.py", description = "Validate packages, or run proposed corrections through the gate" }
stress = { cmd = "python tools/stress.py", description = "Stress the oracle with random inputs: pixi run stress -- --seed N" }
site = { cmd = "python tools/site.py", description = "Build the website into _site/" }
>>>>>>> 4fa326d (First implementation (#1))
registry/README.md
-5
View File
@@ -43,12 +43,7 @@ A rule's meaning never changes, and a retired rule's ID is never reused, so tool
A tool can match on the `rule` field, whose meaning is looked up in `rules.json`, and treat the message as text for people. A tool can match on the `rule` field, whose meaning is looked up in `rules.json`, and treat the message as text for people.
[Chapter 22, section 22.4](../spec/22-conformance.md#224-rules) lists the ID prefixes. [Chapter 22, section 22.4](../spec/22-conformance.md#224-rules) lists the ID prefixes.
Each directory in `fixtures/invalid/` has an `expect.json` that names the rule it breaks. Each directory in `fixtures/invalid/` has an `expect.json` that names the rule it breaks.
<<<<<<< HEAD
The reference checker (`paperspec check`) fails if that rule is not in `rules.json`. The reference checker (`paperspec check`) fails if that rule is not in `rules.json`.
||||||| parent of 4fa326d (First implementation (#1))
=======
The reference checker (`tools/check.py`) fails if that rule is not in `rules.json`.
>>>>>>> 4fa326d (First implementation (#1))
## Adding stamp rules ## Adding stamp rules
registry/rules.json
+52 -2
View File
@@ -5,9 +5,14 @@
"rules": [ "rules": [
{ {
"id": "correction.digits", "id": "correction.digits",
"description": "A digit changes without crop evidence, or a leaf's digits change undeclared.", "description": "A digit changes without crop evidence, an emend changes a digit or edits beside one, or a leaf's digits change undeclared.",
"chapter": "spec/22-conformance.md" "chapter": "spec/22-conformance.md"
}, },
{
"id": "correction.emend",
"description": "An emend targets no runs, falls in math or across runs, or another operation changes an emended run or writes `printed`.",
"chapter": "spec/11-corrections.md"
},
{ {
"id": "correction.evidence", "id": "correction.evidence",
"description": "An insert has no crop evidence.", "description": "An insert has no crop evidence.",
@@ -48,11 +53,21 @@
"description": "The kind takes no label.", "description": "The kind takes no label.",
"chapter": "spec/22-conformance.md" "chapter": "spec/22-conformance.md"
}, },
{
"id": "correction.relink",
"description": "A relink names no text or cite run of a node with runs, or unlinks a run that is not a cite.",
"chapter": "spec/11-corrections.md"
},
{ {
"id": "correction.replace", "id": "correction.replace",
"description": "Runs replace text outside a text run, or include an equation run.", "description": "Runs replace text outside a text run, or include an equation run.",
"chapter": "spec/22-conformance.md" "chapter": "spec/22-conformance.md"
}, },
{
"id": "correction.resolve",
"description": "A resolve names a node that is not a reference, or a resolution that is not doi, document or none.",
"chapter": "spec/11-corrections.md"
},
{ {
"id": "correction.schema", "id": "correction.schema",
"description": "A proposed correction does not match the correction schema.", "description": "A proposed correction does not match the correction schema.",
@@ -88,11 +103,26 @@
"description": "Correction indexes do not count from 1 without gaps.", "description": "Correction indexes do not count from 1 without gaps.",
"chapter": "spec/22-conformance.md" "chapter": "spec/22-conformance.md"
}, },
{
"id": "doc.correction.resolution",
"description": "The last resolve of a reference does not end at its resolution in references.jsonl.",
"chapter": "spec/11-corrections.md"
},
{ {
"id": "doc.correction.rev", "id": "doc.correction.rev",
"description": "document.json's rev is not the number of corrections.", "description": "document.json's rev is not the number of corrections.",
"chapter": "spec/22-conformance.md" "chapter": "spec/22-conformance.md"
}, },
{
"id": "doc.derived.math",
"description": "A line of derived/math.jsonl names a formula that is not TeX, says it is complete when its tree has an opaque node or the reverse, or lists symbols that are not the sym keys of its tree.",
"chapter": "spec/09-figures-tables-equations.md"
},
{
"id": "doc.derived.node",
"description": "A line of a derived file names no equation, equation line or math run of the package.",
"chapter": "spec/12-package.md"
},
{ {
"id": "doc.fidelity.report", "id": "doc.fidelity.report",
"description": "document.json's fidelity distribution does not recompute.", "description": "document.json's fidelity distribution does not recompute.",
@@ -188,6 +218,11 @@
"description": "A location's page_fp is not a page of any witness.", "description": "A location's page_fp is not a page of any witness.",
"chapter": "spec/22-conformance.md" "chapter": "spec/22-conformance.md"
}, },
{
"id": "doc.math.form",
"description": "A source with a form is not that form's canonical spelling. Only a validator that implements the form can check it; the reference validator does not.",
"chapter": "spec/03-nodes.md"
},
{ {
"id": "doc.node.attrs", "id": "doc.node.attrs",
"description": "A section has no level.", "description": "A section has no level.",
@@ -330,7 +365,7 @@
}, },
{ {
"id": "doc.run.canonical", "id": "doc.run.canonical",
"description": "Runs are not canonical: an empty text run, or two adjacent text runs with the same styles.", "description": "Runs are not canonical: an empty text run, two adjacent text runs with the same styles and no `printed`, or a `printed` equal to its run's text.",
"chapter": "spec/22-conformance.md" "chapter": "spec/22-conformance.md"
}, },
{ {
@@ -363,6 +398,11 @@
"description": "A same-document xref names no node.", "description": "A same-document xref names no node.",
"chapter": "spec/22-conformance.md" "chapter": "spec/22-conformance.md"
}, },
{
"id": "doc.suspect.witness",
"description": "A suspect names a witness the package does not have.",
"chapter": "spec/22-conformance.md"
},
{ {
"id": "doc.table.header", "id": "doc.table.header",
"description": "A column or row header path names a node that is not a cell.", "description": "A column or row header path names a node that is not a cell.",
@@ -408,6 +448,16 @@
"description": "The exercise kind is unknown.", "description": "The exercise kind is unknown.",
"chapter": "spec/22-conformance.md" "chapter": "spec/22-conformance.md"
}, },
{
"id": "layer.trace.source",
"description": "A trace's passage is not in the document its source names, its source is the citing document, its selector has no AnchorSelector, or its TextQuoteSelector does not repeat the anchor's plain quote.",
"chapter": "spec/18-layers.md"
},
{
"id": "layer.trace.target",
"description": "A trace's target is not a leaf with runs of the citing document selected by an AnchorSelector, or its sentence does not cite the reference the trace names.",
"chapter": "spec/18-layers.md"
},
{ {
"id": "package.file.canonical", "id": "package.file.canonical",
"description": "A JSON or JSON Lines record is not in canonical JSON.", "description": "A JSON or JSON Lines record is not in canonical JSON.",
schemas/README.md
+4 -8
View File
@@ -1,6 +1,6 @@
# Schemas # Schemas
This folder holds the JSON (JavaScript Object Notation) Schemas (draft 2020-12) for every file and record that the paperspec formats (paperspec is the specification this repository holds) define: 42 schema files. This folder holds the JSON (JavaScript Object Notation) Schemas (draft 2020-12) for every file and record that the paperspec formats (paperspec is the specification this repository holds) define: 43 schema files.
A JSON Schema is a file, itself written in JSON, that states what a valid JSON file looks like. A JSON Schema is a file, itself written in JSON, that states what a valid JSON file looks like.
Newcomers can read it as a form: it lists the fields a record must have and what kind of value each field takes. Newcomers can read it as a form: it lists the fields a record must have and what kind of value each field takes.
Implementers will find the loading recipe and the traps under "Validating" and "Writing schemas". Implementers will find the loading recipe and the traps under "Validating" and "Writing schemas".
@@ -20,7 +20,7 @@ The number after each name is the chapter of `spec/` that defines the file.
| Topic | Schemas (chapter) | | Topic | Schemas (chapter) |
| --- | --- | | --- | --- |
| Shared definitions | `common.schema.json` (1): hashes, IDs, keys, addresses, times, decimals, boxes, licenses, signatures, evidence, quotes. | | Shared definitions | `common.schema.json` (1): hashes, IDs, keys, addresses, times, decimals, boxes, licenses, signatures, evidence, quotes. |
| A package | `manifest.schema.json` (12), `document.schema.json` (2, 12), `node.schema.json` (3), `runs.schema.json` (3), `witnesses.schema.json` (2), `provenance.schema.json` (12), `page.schema.json` (7), `location.schema.json` (7), `reference.schema.json` (10), `id-entry.schema.json` (4), `correction.schema.json` (11), `receipt.schema.json` (8), `revocation.schema.json` (8), `roster.schema.json` (8), `suspect.schema.json` (5), `mathml.schema.json` (12) | | A package | `manifest.schema.json` (12), `document.schema.json` (2, 12), `node.schema.json` (3), `runs.schema.json` (3), `witnesses.schema.json` (2), `provenance.schema.json` (12), `page.schema.json` (7), `location.schema.json` (7), `reference.schema.json` (10), `id-entry.schema.json` (4), `correction.schema.json` (11), `receipt.schema.json` (8), `revocation.schema.json` (8), `roster.schema.json` (8), `suspect.schema.json` (5), `mathml.schema.json` (12), `math.schema.json` (9, 12) |
| Pointing into documents | `anchor.schema.json` (6) | | Pointing into documents | `anchor.schema.json` (6) |
| Rights | `rights.schema.json` (14), `policy.schema.json` (14), `stages.schema.json` (14), `backend.schema.json` (14), `profile.schema.json` (14), `stamps.schema.json` (7) | | Rights | `rights.schema.json` (14), `policy.schema.json` (14), `stages.schema.json` (14), `backend.schema.json` (14), `profile.schema.json` (14), `stamps.schema.json` (7) |
| Review | `qa-report.schema.json` (8), `spot-check.schema.json` (8), `checklist.schema.json` (8), `proof.schema.json` (8, 11) | | Review | `qa-report.schema.json` (8), `spot-check.schema.json` (8), `checklist.schema.json` (8), `proof.schema.json` (8, 11) |
@@ -37,7 +37,8 @@ The Data schemas for datasets, digitizations and views, the Teams and drafts sch
The rest belong to `candidate` formats or to no format of their own. The rest belong to `candidate` formats or to no format of their own.
The bundle has no JSON Schema: its registry entry points to `sql/bundle-sqlite.sql`. The bundle has no JSON Schema: its registry entry points to `sql/bundle-sqlite.sql`.
Of the 42 schemas, 11 have no format of their own because each describes a line or part of a larger file: `common`, `node`, `runs`, `witnesses`, `provenance`, `location`, `reference`, `id-entry`, `mathml`, `annotation` and `value-anchor`. Of the 43 schemas, 12 have no format of their own because each describes a line or part of a larger file: `common`, `node`, `runs`, `witnesses`, `provenance`, `location`, `reference`, `id-entry`, `mathml`, `math`, `annotation` and `value-anchor`.
A line of `derived/math.jsonl` carries `form`, the math form `scienting-math/1` it reads the formula in; that names a spelling of TeX, which papermath implements ([spec/09 §9.4](../spec/09-figures-tables-equations.md#94-the-math-form)), not a file format, so the registry does not list it.
Of the schemas that do have one, 27 require a `format` property; `spot-check.schema.json` describes the input a VRF proof is made over rather than a file. Of the schemas that do have one, 27 require a `format` property; `spot-check.schema.json` describes the input a VRF proof is made over rather than a file.
The others (`page`, `draft-ids`, `view` and `log-entry`) describe a line of a file, a Vega-Lite specification (a JSON description of a chart) or an item of a tree, and the record itself does not need to carry the identifier. The others (`page`, `draft-ids`, `view` and `log-entry`) describe a line of a file, a Vega-Lite specification (a JSON description of a chart) or an item of a tree, and the record itself does not need to carry the identifier.
@@ -64,12 +65,7 @@ The `Registry` maps that name to the file's contents, so nothing is fetched over
Never load a schema alone. Never load a schema alone.
Any draft 2020-12 validator works the same way: give it every schema under its `$id`, then validate against the one you need. Any draft 2020-12 validator works the same way: give it every schema under its `$id`, then validate against the one you need.
<<<<<<< HEAD
`paperspec check` validates every fixture with `jsonschema` this way, and checks that every schema keeps to the keyword subset. `paperspec check` validates every fixture with `jsonschema` this way, and checks that every schema keeps to the keyword subset.
||||||| parent of 4fa326d (First implementation (#1))
=======
`python3 tools/check.py` validates every fixture with `jsonschema` this way, and checks that every schema keeps to the keyword subset.
>>>>>>> 4fa326d (First implementation (#1))
## What a schema does not check ## What a schema does not check
schemas/annotation.schema.json
+190 -7
View File
@@ -3,6 +3,64 @@
"$id": "https://paperspec.scient.ing/schemas/annotation.schema.json", "$id": "https://paperspec.scient.ing/schemas/annotation.schema.json",
"title": "Annotation", "title": "Annotation",
"description": "One line of annotations.jsonl in scienting-layer/1: a W3C Web Annotation; spec/18-layers.md.", "description": "One line of annotations.jsonl in scienting-layer/1: a W3C Web Annotation; spec/18-layers.md.",
"$defs": {
"category": {
"description": "What a comment, caveat or question is about (spec/18 §18.4).",
"enum": [
"comment",
"context",
"critique",
"question",
"correction",
"feedback",
"resource"
]
},
"selector": {
"description": "A selector of a passage: the AnchorSelector, or the standard TextQuoteSelector that repeats the anchor's plain quote for tools that do not know ours (spec/18 §18.3).",
"oneOf": [
{
"type": "object",
"properties": {
"type": {
"const": "AnchorSelector"
},
"anchor": {
"$ref": "anchor.schema.json"
}
},
"required": [
"type",
"anchor"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"type": {
"const": "TextQuoteSelector"
},
"exact": {
"type": "string",
"minLength": 1
},
"prefix": {
"type": "string"
},
"suffix": {
"type": "string"
}
},
"required": [
"type",
"exact"
],
"additionalProperties": false
}
]
}
},
"type": "object", "type": "object",
"properties": { "properties": {
"@context": { "@context": {
@@ -97,15 +155,13 @@
"const": "SpecificResource" "const": "SpecificResource"
}, },
"source": { "source": {
"type": "string", "$ref": "common.schema.json#/$defs/claimRef"
"minLength": 1
}, },
"kind": { "kind": {
"const": "claim-link" "const": "claim-link"
}, },
"pin": { "pin": {
"type": "string", "$ref": "common.schema.json#/$defs/claimPin"
"minLength": 1
}, },
"role": { "role": {
"enum": [ "enum": [
@@ -130,17 +186,21 @@
"const": "SpecificResource" "const": "SpecificResource"
}, },
"source": { "source": {
"type": "string", "$ref": "common.schema.json#/$defs/claimRef"
"minLength": 1
}, },
"kind": { "kind": {
"const": "symbol" "const": "symbol"
}, },
"pin": { "pin": {
"$ref": "common.schema.json#/$defs/claimPin"
},
"key": {
"description": "The symbol's scienting-math/1 key, which the symbol claim's notation has too (spec/09 §9.4, spec/18 §18.4).",
"type": "string", "type": "string",
"minLength": 1 "minLength": 1
}, },
"symbol": { "symbol": {
"description": "The symbol as the span writes it.",
"type": "string", "type": "string",
"minLength": 1 "minLength": 1
} }
@@ -148,7 +208,8 @@
"required": [ "required": [
"type", "type",
"kind", "kind",
"source" "source",
"key"
], ],
"additionalProperties": false "additionalProperties": false
}, },
@@ -173,6 +234,9 @@
}, },
"kind": { "kind": {
"const": "comment" "const": "comment"
},
"category": {
"$ref": "#/$defs/category"
} }
}, },
"required": [ "required": [
@@ -203,6 +267,9 @@
}, },
"kind": { "kind": {
"const": "caveat" "const": "caveat"
},
"category": {
"$ref": "#/$defs/category"
} }
}, },
"required": [ "required": [
@@ -240,6 +307,9 @@
"group", "group",
"authors" "authors"
] ]
},
"category": {
"$ref": "#/$defs/category"
} }
}, },
"required": [ "required": [
@@ -397,6 +467,79 @@
], ],
"additionalProperties": false "additionalProperties": false
}, },
{
"type": "object",
"properties": {
"type": {
"const": "SpecificResource"
},
"kind": {
"const": "trace"
},
"source": {
"description": "The cited document, as a target's source names a document.",
"type": "string",
"pattern": "^tag:scient\\.ing,2026:doc:[a-z0-9][a-z0-9._-]*/[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}(\\.[a-z0-9][a-z0-9-]*)*$"
},
"selector": {
"description": "The passage of the cited document: an AnchorSelector, then a TextQuoteSelector when the anchor's quote is plain.",
"type": "array",
"items": {
"$ref": "#/$defs/selector"
},
"minItems": 1
},
"reference": {
"description": "The reference node of the citing document that the target's sentence cites and the trace is about.",
"$ref": "common.schema.json#/$defs/nodeId"
},
"note": {
"type": "string",
"minLength": 1
},
"verdict": {
"enum": [
"supports",
"partly",
"does-not",
"cannot-tell"
]
},
"proposer": {
"description": "The tool that proposed the trace, required when the annotation's creator is Software (spec/18 §18.4).",
"type": "object",
"properties": {
"model": {
"$ref": "common.schema.json#/$defs/model"
},
"run": {
"type": "string",
"minLength": 1
},
"score": {
"$ref": "common.schema.json#/$defs/decimal"
},
"rank": {
"type": "integer",
"minimum": 1
}
},
"required": [
"model",
"run"
],
"additionalProperties": false
}
},
"required": [
"type",
"kind",
"source",
"selector",
"reference"
],
"additionalProperties": false
},
{ {
"type": "object", "type": "object",
"properties": { "properties": {
@@ -738,6 +881,46 @@
"receipt" "receipt"
] ]
} }
},
{
"description": "A tool's trace names the tool and the run that proposed it (spec/18 §18.4).",
"if": {
"properties": {
"creator": {
"properties": {
"type": {
"const": "Software"
}
},
"required": [
"type"
]
},
"body": {
"properties": {
"kind": {
"const": "trace"
}
},
"required": [
"kind"
]
}
},
"required": [
"creator",
"body"
]
},
"then": {
"properties": {
"body": {
"required": [
"proposer"
]
}
}
}
} }
] ]
} }
schemas/common.schema.json
+16
View File
@@ -71,6 +71,22 @@
"type": "string", "type": "string",
"pattern": "^([a-z0-9][a-z0-9._-]*/)?[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}(\\.[a-z0-9][a-z0-9-]*)*#[a-z0-9][a-z0-9._-]*(@[0-9a-f]{6})?(\\[(0|[1-9][0-9]*):(0|[1-9][0-9]*)\\])?$" "pattern": "^([a-z0-9][a-z0-9._-]*/)?[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}(\\.[a-z0-9][a-z0-9-]*)*#[a-z0-9][a-z0-9._-]*(@[0-9a-f]{6})?(\\[(0|[1-9][0-9]*):(0|[1-9][0-9]*)\\])?$"
}, },
"claimRef": {
"type": "string",
"pattern": "^claim:([a-z0-9.-]+\\.[a-z]{2,}/)?[0-9A-HJKMNP-TV-Z]{8}$",
"description": "A claim in a paperclaim store: claim:TAG, or claim:host/TAG for another store, TAG eight Crockford base32 capitals (spec/18 §18.4)."
},
"claimPin": {
"type": "string",
"pattern": "^claim:([a-z0-9.-]+\\.[a-z]{2,}/)?[0-9A-HJKMNP-TV-Z]{8}@[0-9a-f]{6}$",
"description": "A claim pinned to its statement: the claim's reference, @ and the first six hex digits of its statement hash (spec/18 §18.4)."
},
"mathForm": {
"enum": [
"scienting-math/1"
],
"description": "The form a math source is spelled in; scienting-math/1 is canonical TeX (spec/03 §3.7, spec/09 §9.4)."
},
"time": { "time": {
"type": "string", "type": "string",
"pattern": "^[0-9]{4}-[0-1][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]Z$" "pattern": "^[0-9]{4}-[0-1][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]Z$"
schemas/correction.schema.json
+109 -1
View File
@@ -3,6 +3,18 @@
"$id": "https://paperspec.scient.ing/schemas/correction.schema.json", "$id": "https://paperspec.scient.ing/schemas/correction.schema.json",
"title": "Correction", "title": "Correction",
"description": "scienting-correction/1: a line of corrections.jsonl, or a proposal; spec/11-corrections.md.", "description": "scienting-correction/1: a line of corrections.jsonl, or a proposal; spec/11-corrections.md.",
"$defs": {
"emendKind": {
"enum": [
"typo",
"spelling",
"grammar",
"punctuation",
"wording",
"other"
]
}
},
"type": "object", "type": "object",
"properties": { "properties": {
"format": { "format": {
@@ -25,7 +37,10 @@
"move", "move",
"reclassify", "reclassify",
"relabel", "relabel",
"set" "set",
"emend",
"relink",
"resolve"
] ]
}, },
"target": { "target": {
@@ -71,8 +86,39 @@
"type": "string" "type": "string"
}, },
"kind": { "kind": {
"description": "A reclassify's new node kind, or what an emend corrects.",
"anyOf": [
{
"$ref": "common.schema.json#/$defs/nodeKind" "$ref": "common.schema.json#/$defs/nodeKind"
}, },
{
"$ref": "#/$defs/emendKind"
}
]
},
"note": {
"type": "string",
"minLength": 1
},
"refs": {
"description": "A relink's references: reference node IDs, or null to unlink.",
"anyOf": [
{
"type": "array",
"items": {
"$ref": "common.schema.json#/$defs/nodeId"
},
"minItems": 1,
"uniqueItems": true
},
{
"type": "null"
}
]
},
"resolution": {
"$ref": "reference.schema.json#/properties/resolution"
},
"label": { "label": {
"anyOf": [ "anyOf": [
{ {
@@ -287,12 +333,74 @@
} }
}, },
"then": { "then": {
"properties": {
"kind": {
"$ref": "common.schema.json#/$defs/nodeKind"
}
},
"required": [ "required": [
"target", "target",
"kind" "kind"
] ]
} }
}, },
{
"if": {
"properties": {
"op": {
"const": "emend"
}
}
},
"then": {
"properties": {
"replace": {
"type": "string",
"minLength": 1
},
"kind": {
"$ref": "#/$defs/emendKind"
}
},
"required": [
"target",
"find",
"replace",
"kind"
]
}
},
{
"if": {
"properties": {
"op": {
"const": "relink"
}
}
},
"then": {
"required": [
"target",
"run",
"refs"
]
}
},
{
"if": {
"properties": {
"op": {
"const": "resolve"
}
}
},
"then": {
"required": [
"target",
"resolution"
]
}
},
{ {
"if": { "if": {
"properties": { "properties": {
schemas/document.schema.json
+8 -1
View File
@@ -47,8 +47,15 @@
"same_as": { "same_as": {
"type": "array", "type": "array",
"items": { "items": {
"anyOf": [
{
"type": "string", "type": "string",
"pattern": "^(doi|arxiv|isbn|pmid|pmcid|openalex|url|lode):[^ ]+$" "pattern": "^(doi|arxiv|isbn|pmid|pmcid|openalex|url):[^ ]+$"
},
{
"$ref": "common.schema.json#/$defs/claimRef"
}
]
}, },
"uniqueItems": true "uniqueItems": true
} }
schemas/math.schema.json
+172
View File
@@ -0,0 +1,172 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://paperspec.scient.ing/schemas/math.schema.json",
"title": "Derived math",
"description": "One line of derived/math.jsonl: a formula of the package read in the form scienting-math/1 by a pinned tool, its tree, symbol keys and issues; never authoritative; spec/09-figures-tables-equations.md §9.4, spec/12-package.md §12.7.",
"$defs": {
"kind": {
"description": "What a tree node is. The kinds of scienting-math/1 are num, sym, text, const, oper, dots, empty, add, neg, pm, mp, mul, times, div, op, frac, binom, sqrt, root, pow, sub, prime, fact, prescript, mark, apply, lim, sum, prod, coprod, int, iint, iiint, oint, bigcup, bigcap, abs, norm, angle, floor, ceil, braces, interval, rel, list, lines, cases, matrix and opaque (spec/09 §9.4). The schema takes any lowercase name, so that a later minor version of the form may add a kind; a consumer that meets a kind it does not know treats that node as it treats opaque.",
"type": "string",
"pattern": "^[a-z][a-z0-9-]*$",
"examples": [
"sym",
"frac",
"rel",
"opaque"
]
},
"node": {
"description": "A tree node: an array whose first item is its kind and whose other items are its parts.",
"type": "array",
"minItems": 1,
"prefixItems": [
{
"$ref": "#/$defs/kind"
}
],
"items": {
"$ref": "#/$defs/part"
}
},
"group": {
"description": "Parts that are not a node: a case's value and condition, or a matrix's rows and cells. Its first item, when it has one, is a node, a group or null, never a string, so it is never mistaken for a node.",
"type": "array",
"prefixItems": [
{
"anyOf": [
{
"type": "array"
},
{
"type": "null"
}
]
}
],
"items": {
"$ref": "#/$defs/part"
}
},
"part": {
"description": "A part of a node: a string (digits, a symbol key, words, a command, a relation such as =), null (a script or condition that is absent), a node, or a group.",
"anyOf": [
{
"type": "string"
},
{
"type": "null"
},
{
"$ref": "#/$defs/node"
},
{
"$ref": "#/$defs/group"
}
]
},
"issue": {
"description": "Something a reader of the formula should look at; the formula is read all the same.",
"type": "object",
"properties": {
"code": {
"description": "unbalanced, unknown-command, label-or-power, script-on-group, opaque or infix-fraction in scienting-math/1 (spec/09 §9.4); a later minor version may add a code.",
"type": "string",
"pattern": "^[a-z][a-z0-9-]*$"
},
"message": {
"type": "string",
"minLength": 1
},
"at": {
"description": "The TeX the issue is about, in canonical form; empty when it is about the whole formula.",
"type": "string"
}
},
"required": [
"code",
"message",
"at"
],
"additionalProperties": false
}
},
"type": "object",
"properties": {
"node": {
"$ref": "common.schema.json#/$defs/nodeId"
},
"run": {
"description": "The index of the math run in the node's runs, or null for an equation or equation-line node, as in derived/mathml.jsonl.",
"anyOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "null"
}
]
},
"tool": {
"type": "object",
"properties": {
"name": {
"type": "string",
"minLength": 1
},
"version": {
"type": "string",
"minLength": 1
}
},
"required": [
"name",
"version"
],
"additionalProperties": false
},
"form": {
"enum": [
"scienting-math/1"
]
},
"tree": {
"$ref": "#/$defs/node"
},
"symbols": {
"description": "The key of every symbol in the tree, each once, in the order the formula first writes it.",
"type": "array",
"items": {
"type": "string",
"minLength": 1
},
"uniqueItems": true
},
"complete": {
"description": "True exactly when the tree has no opaque node.",
"type": "boolean"
},
"issues": {
"type": "array",
"items": {
"$ref": "#/$defs/issue"
}
},
"label": {
"description": "The printed number the tool lifted out of a \\tag in the source; omitted when the source has none.",
"type": "string",
"minLength": 1
}
},
"required": [
"node",
"run",
"tool",
"form",
"tree",
"symbols",
"complete",
"issues"
],
"additionalProperties": false
}
schemas/node.schema.json
+16 -1
View File
@@ -329,13 +329,28 @@
"source": { "source": {
"type": "string", "type": "string",
"minLength": 1 "minLength": 1
},
"form": {
"$ref": "common.schema.json#/$defs/mathForm"
} }
}, },
"required": [ "required": [
"notation", "notation",
"source" "source"
], ],
"additionalProperties": false "additionalProperties": false,
"if": {
"required": [
"form"
]
},
"then": {
"properties": {
"notation": {
"const": "tex"
}
}
}
} }
} }
} }
schemas/project.schema.json
+17 -3
View File
@@ -131,7 +131,21 @@
"type", "type",
"ref" "ref"
], ],
"additionalProperties": false "additionalProperties": false,
"if": {
"properties": {
"type": {
"const": "claim"
}
}
},
"then": {
"properties": {
"ref": {
"$ref": "common.schema.json#/$defs/claimRef"
}
}
}
}, },
"minItems": 1 "minItems": 1
} }
@@ -172,13 +186,13 @@
"claims": { "claims": {
"type": "object", "type": "object",
"properties": { "properties": {
"lode": { "paperclaim": {
"type": "string", "type": "string",
"minLength": 1 "minLength": 1
} }
}, },
"required": [ "required": [
"lode" "paperclaim"
], ],
"additionalProperties": false "additionalProperties": false
}, },
schemas/runs.schema.json
+38 -1
View File
@@ -3,6 +3,13 @@
"$id": "https://paperspec.scient.ing/schemas/runs.schema.json", "$id": "https://paperspec.scient.ing/schemas/runs.schema.json",
"title": "Inline runs", "title": "Inline runs",
"description": "The runs of a sentence, cell, reference, authors or keywords node; spec/03-nodes.md.", "description": "The runs of a sentence, cell, reference, authors or keywords node; spec/03-nodes.md.",
"$defs": {
"printed": {
"description": "The run's text as the paper prints it, kept by an emend correction (spec/03 §3.5, spec/11 §11.1).",
"type": "string",
"minLength": 1
}
},
"type": "array", "type": "array",
"minItems": 1, "minItems": 1,
"items": { "items": {
@@ -31,6 +38,9 @@
}, },
"minItems": 1, "minItems": 1,
"uniqueItems": true "uniqueItems": true
},
"printed": {
"$ref": "#/$defs/printed"
} }
}, },
"required": [ "required": [
@@ -54,6 +64,9 @@
"source": { "source": {
"type": "string", "type": "string",
"minLength": 1 "minLength": 1
},
"form": {
"$ref": "common.schema.json#/$defs/mathForm"
} }
}, },
"required": [ "required": [
@@ -61,7 +74,19 @@
"notation", "notation",
"source" "source"
], ],
"additionalProperties": false "additionalProperties": false,
"if": {
"required": [
"form"
]
},
"then": {
"properties": {
"notation": {
"const": "tex"
}
}
}
}, },
{ {
"type": "object", "type": "object",
@@ -79,6 +104,9 @@
"text": { "text": {
"type": "string", "type": "string",
"minLength": 1 "minLength": 1
},
"printed": {
"$ref": "#/$defs/printed"
} }
}, },
"required": [ "required": [
@@ -110,6 +138,9 @@
"text": { "text": {
"type": "string", "type": "string",
"minLength": 1 "minLength": 1
},
"printed": {
"$ref": "#/$defs/printed"
} }
}, },
"required": [ "required": [
@@ -138,6 +169,9 @@
"text": { "text": {
"type": "string", "type": "string",
"minLength": 1 "minLength": 1
},
"printed": {
"$ref": "#/$defs/printed"
} }
}, },
"required": [ "required": [
@@ -159,6 +193,9 @@
"text": { "text": {
"type": "string", "type": "string",
"minLength": 1 "minLength": 1
},
"printed": {
"$ref": "#/$defs/printed"
} }
}, },
"required": [ "required": [
schemas/suspect.schema.json
+3
View File
@@ -60,6 +60,9 @@
} }
] ]
}, },
"witness": {
"$ref": "common.schema.json#/$defs/nodeId"
},
"crop": { "crop": {
"$ref": "common.schema.json#/$defs/hash" "$ref": "common.schema.json#/$defs/hash"
}, },
spec/00-introduction.md
-9
View File
@@ -82,18 +82,9 @@ The other folders help people use the specification.
| `registry/` | Formats, rule IDs, stamp rules and the default stage requirements. | Yes | | `registry/` | Formats, rule IDs, stamp rules and the default stage requirements. | Yes |
| `vectors/` | Test vectors, one case per line. | Yes | | `vectors/` | Test vectors, one case per line. | Yes |
| `fixtures/` | Valid packages and records, invalid packages, and the keys that signed them. | Yes | | `fixtures/` | Valid packages and records, invalid packages, and the keys that signed them. | Yes |
<<<<<<< HEAD
| `src/` | The Python package `paperspec`: the reference implementation, and the `paperspec` command that computes and checks the vectors and fixtures and loads packages into SQLite. | No | | `src/` | The Python package `paperspec`: the reference implementation, and the `paperspec` command that computes and checks the vectors and fixtures and loads packages into SQLite. | No |
| `tools/` | How to run those commands. | No | | `tools/` | How to run those commands. | No |
| `docs/` | Guides for people getting to know the formats. | No | | `docs/` | Guides for people getting to know the formats. | No |
The reference implementation in `src/paperspec/` computes every expected value, so that none is typed by hand. The reference implementation in `src/paperspec/` computes every expected value, so that none is typed by hand.
Other repositories of the project import it, but implementations conform to the chapters and the vectors, not to it. Other repositories of the project import it, but implementations conform to the chapters and the vectors, not to it.
||||||| parent of 4fa326d (First implementation (#1))
=======
| `tools/` | The programs that compute and check the vectors and fixtures, and load packages into SQLite. | No |
| `docs/` | Guides for people getting to know the formats. | No |
The programs in `tools/` compute every expected value, so that none is typed by hand.
They are not a library, and implementations conform to the chapters and the vectors.
>>>>>>> 4fa326d (First implementation (#1))
spec/01-conventions.md
-5
View File
@@ -46,12 +46,7 @@ Stored text is kept exactly as witnessed; normalization applies only when a valu
The vectors in `normalize.jsonl` cover the cases, including the angstrom sign, which NFC turns into Å. The vectors in `normalize.jsonl` cover the cases, including the angstrom sign, which NFC turns into Å.
Normalization forms, General Categories and the `White_Space` property are those of Unicode 16.0. Normalization forms, General Categories and the `White_Space` property are those of Unicode 16.0.
<<<<<<< HEAD
A later Unicode version can change a decomposition or a category, and with it a hash, so an implementation states the Unicode version its character data comes from, the reference implementation refuses to import on another, and moving the formats to a later version is a change to every format that hashes text ([0.3](00-introduction.md#03-format-identifiers-and-versions)). A later Unicode version can change a decomposition or a category, and with it a hash, so an implementation states the Unicode version its character data comes from, the reference implementation refuses to import on another, and moving the formats to a later version is a change to every format that hashes text ([0.3](00-introduction.md#03-format-identifiers-and-versions)).
||||||| parent of 4fa326d (First implementation (#1))
=======
A later Unicode version can change a decomposition or a category, and with it a hash, so an implementation states the Unicode version its character data comes from, the reference tools refuse to run on another, and moving the formats to a later version is a change to every format that hashes text ([0.3](00-introduction.md#03-format-identifiers-and-versions)).
>>>>>>> 4fa326d (First implementation (#1))
## 1.5 Character offsets ## 1.5 Character offsets
spec/02-model.md
+3 -2
View File
@@ -4,7 +4,7 @@
| Object | Is | Identified by | | Object | Is | Identified by |
| --- | --- | --- | | --- | --- | --- |
| Work | The intellectual work, across all its versions. | A work key ([2.4](#24-work-and-document-keys)), and `same_as` identifiers: `doi:`, `arxiv:`, `isbn:`, `pmid:`, `pmcid:`, `openalex:`, `url:` or `lode:`. | | Work | The intellectual work, across all its versions. | A work key ([2.4](#24-work-and-document-keys)), and `same_as` identifiers: `doi:`, `arxiv:`, `isbn:`, `pmid:`, `pmcid:`, `openalex:`, `url:` or `claim:`. |
| Version | One expression of the work: a preprint, an accepted manuscript, the version of record, an erratum, Supporting Information, a draft. | The work's key and a version type, with an optional sequence number and date. | | Version | One expression of the work: a preprint, an accepted manuscript, the version of record, an erratum, Supporting Information, a draft. | The work's key and a version type, with an optional sequence number and date. |
| Witness | One file of a version: a PDF, JATS XML, a LaTeX source archive, an HTML page, an EPUB, a Typst project. | The SHA-256 of its bytes. | | Witness | One file of a version: a PDF, JATS XML, a LaTeX source archive, an HTML page, an EPUB, a Typst project. | The SHA-256 of its bytes. |
| Document | The text established for one version from one or more witnesses. | A key, and a root hash per revision. | | Document | The text established for one version from one or more witnesses. | A key, and a root hash per revision. |
@@ -83,7 +83,8 @@ The stop words are `a an the of on in for to and or at by with from as into onto
**The identity** is the work's first identifier among `doi`, `arxiv`, `isbn`, `pmid`, `pmcid` and `openalex` in `same_as`, in that order, written `<kind>:<value>` with a DOI's ASCII letters lower-cased. **The identity** is the work's first identifier among `doi`, `arxiv`, `isbn`, `pmid`, `pmcid` and `openalex` in `same_as`, in that order, written `<kind>:<value>` with a DOI's ASCII letters lower-cased.
A work with none has the identity `meta:<title words>|<year>|<family>` from the same folded parts. A work with none has the identity `meta:<title words>|<year>|<family>` from the same folded parts.
`url:` and `lode:` identifiers name copies and claims, not the work, and never enter the identity. `url:` and `claim:` identifiers name copies and claims, not the work, and never enter the identity.
A `claim:` identifier is a claim reference as a layer writes one, `claim:<tag>` or `claim:<host>/<tag>` ([18.4](18-layers.md#184-bodies)).
**The hash part** is the leading bits of SHA-256 of `scienting-key/1:` followed by the identity, in UTF-8, written in lowercase Crockford base32, five bits a character: four characters, or, when another work in the store already holds that key, six, eight and so on until the key is free. **The hash part** is the leading bits of SHA-256 of `scienting-key/1:` followed by the identity, in UTF-8, written in lowercase Crockford base32, five bits a character: four characters, or, when another work in the store already holds that key, six, eight and so on until the key is free.
A key is minted once, when the work enters a store, and never recomputed: a corrected title or a year revised later leaves it alone, while the hash of an identifier still shows which work the key names. A key is minted once, when the work enters a store, and never recomputed: a corrected title or a year revised later leaves it alone, while the hash of an identifier still shows which work the key names.
spec/03-nodes.md
+30 -4
View File
@@ -79,11 +79,11 @@ Page breaks, running heads and columns are not nodes: they are geometry ([7](07-
| Content | Kinds | Shape | | Content | Kinds | Shape |
| --- | --- | --- | | --- | --- | --- |
| Runs | `sentence`, `cell`, `reference`, `authors`, `keywords` | A non-empty list of inline runs ([3.5](#35-inline-runs)). | | Runs | `sentence`, `cell`, `reference`, `authors`, `keywords` | A non-empty list of inline runs ([3.5](#35-inline-runs)). |
| Math | `equation`, `equation-line` | `{notation, source}`: `tex` or `typst`, and the source as written. | | Math | `equation`, `equation-line` | `{notation, source, form?}`: `tex` or `typst`, the source as written or as a form spells it, and that form ([3.7](#37-math-forms)). |
| Verbatim | `code`, `algorithm` | `{language?, text}`. | | Verbatim | `code`, `algorithm` | `{language?, text}`. |
Other kinds have no `content`. Other kinds have no `content`.
Derived renderings, such as MathML, are not content; they live in `derived/` ([12.7](12-package.md#127-derived-files)). Derived renderings and readings, such as MathML and a formula's tree, are not content; they live in `derived/` ([12.7](12-package.md#127-derived-files)).
## 3.4 Attributes ## 3.4 Attributes
@@ -105,14 +105,17 @@ No other kind carries `attrs`, and each kind carries only its own attributes.
| Run | Properties | Plain projection | | Run | Properties | Plain projection |
| --- | --- | --- | | --- | --- | --- |
| `text` | `text`, and optional `style`: a sorted list of distinct values from `em`, `strong`, `sup`, `sub`, `sc`, `mono`, never both `sup` and `sub`. | The text. | | `text` | `text`, and optional `style`: a sorted list of distinct values from `em`, `strong`, `sup`, `sub`, `sc`, `mono`, never both `sup` and `sub`. | The text. |
| `math` | `notation` (`tex` or `typst`), `source`. | `$` + source + `$` | | `math` | `notation` (`tex` or `typst`), `source`, and optional `form` ([3.7](#37-math-forms)). | `$` + source + `$` |
| `cite` | `refs`: reference node IDs; `text`: the printed form, such as `23`, `[1,2]` or `(Smith, 2019)`. | The printed form. | | `cite` | `refs`: reference node IDs; `text`: the printed form, such as `23`, `[1,2]` or `(Smith, 2019)`. | The printed form. |
| `xref` | `target`: a node ID in this document, an address in another, or null when not yet resolved; `text`. | The printed form. | | `xref` | `target`: a node ID in this document, an address in another, or null when not yet resolved; `text`. | The printed form. |
| `fnmark` | `footnote`: a footnote's ID, or null; `text`: the mark. | The mark. | | `fnmark` | `footnote`: a footnote's ID, or null; `text`: the mark. | The mark. |
| `link` | `url`, `text`. | The text. | | `link` | `url`, `text`. | The text. |
| `equation` | `node`: the ID of an equation child of the sentence. | `$$` + that equation's source + `$$` | | `equation` | `node`: the ID of an equation child of the sentence. | `$$` + that equation's source + `$$` |
Runs are canonical: no `text` run is empty, and no two adjacent `text` runs have the same style list. A `text`, `link`, `cite`, `xref` or `fnmark` run MAY carry `printed`: its text as the paper prints it, where an `emend` corrected the authors ([11.6](11-corrections.md#116-emendations)).
The run's `text` is then the emended text, and only an emend writes `printed`.
Runs are canonical: no `text` run is empty, no two adjacent `text` runs have the same style list unless one of them carries `printed`, and no run's `printed` equals its `text`.
A sentence's `equation` runs name its `equation` children, each once, in the children's order, and a sentence has no other children. A sentence's `equation` runs name its `equation` children, each once, in the children's order, and a sentence has no other children.
A display equation the sentence runs through is thus part of the sentence, while keeping its own ID and hashes. A display equation the sentence runs through is thus part of the sentence, while keeping its own ID and hashes.
@@ -125,3 +128,26 @@ That of any other node is the plain projections of its text leaves joined with o
A node's text leaves are found by walking its subtree in document order and stopping at `sentence`, `cell`, `reference`, `authors`, `keywords`, `code` and `algorithm` nodes, and at `equation` nodes whose parent is not a sentence; an `equation-line` is never a text leaf. A node's text leaves are found by walking its subtree in document order and stopping at `sentence`, `cell`, `reference`, `authors`, `keywords`, `code` and `algorithm` nodes, and at `equation` nodes whose parent is not a sentence; an `equation-line` is never a text leaf.
Offsets, quotations and the text hash are all defined on the plain projection as stored, before normalization. Offsets, quotations and the text hash are all defined on the plain projection as stored, before normalization.
A node's runs as printed are its runs with each run's `printed` in place of its `text`, and each `text` run that carried one joined with the adjacent `text` runs of the same style list, as the run was before an emend split it.
Its projection as printed is the plain projection of its runs as printed; an equation's or a verbatim node's is its plain projection, and any other node's joins its text leaves' projections as printed with one U+0020.
It is the text a witness of the paper holds, recovered from `nodes.jsonl` alone; for a node no emend touched it is the plain projection.
The strict and lossy projections are taken over the runs as printed ([5.3](05-hashes.md#53-the-strict-projection)), and `projections.jsonl` gives an emended sentence's projections with its projection as printed.
## 3.7 Math forms
A math form is a standard spelling of formulas, so that one formula is written alike whichever paper, converter or claim it comes from.
An equation's or equation line's content and a `math` run MAY carry `form`, which names the form its `source` is written in; a source without `form` is as the witness wrote it.
The one form is `scienting-math/1`, canonical TeX, which [9.4](09-figures-tables-equations.md#94-the-math-form) describes; a node or run with `form` has the notation `tex`.
A consumer MAY rely on these properties of a source that carries `form`:
1. It is canonical: it is the form's own spelling of the formula the witness wrote, which renders as that formula did, apart from presentation such as spacing, `\displaystyle` and delimiter sizes.
2. It is idempotent: spelling it in the form again gives it back unchanged, so a tool that rewrites it to check it, or a converter run twice, changes nothing.
3. It carries no label: the form lifts a `\tag` out of the source, and an equation keeps it as its `label`; `\label`, `\nonumber` and `\notag` are gone.
4. Everything else is derived from it: the tree, the symbol keys and the issues of [9.4](09-figures-tables-equations.md#94-the-math-form) are read from the source by a pinned tool into `derived/math.jsonl` ([12.7](12-package.md#127-derived-files)), are never content, and are the same for the same source under the same form.
A producer writes `form` only on a source it has spelled in that form.
A correction that changes such a source writes the new source in the form too, since the gate does not implement the form and cannot check it.
A validator that implements the form reports a source with `form` that is not its own canonical spelling as `doc.math.form`.
`form` enters no content value, so adding it to a source already in the form changes no hash, ID or anchor, while respelling a source in the form changes them as any edit of the source does ([5.2](05-hashes.md#52-own-content)).
spec/05-hashes.md
+27 -8
View File
@@ -17,6 +17,9 @@ In this chapter *H* is SHA-256 written as hexadecimal; under a keyed scheme, *H*
| `node_hash` | *H*(canonical JSON of `{"own": own_hash, "children": [each child's node_hash, in order]}`) | Pins on leaves; the root hash; knowing that something below has changed. | | `node_hash` | *H*(canonical JSON of `{"own": own_hash, "children": [each child's node_hash, in order]}`) | Pins on leaves; the root hash; knowing that something below has changed. |
Normalization is that of [1.4](01-conventions.md#14-text-normalization-for-hashing). Normalization is that of [1.4](01-conventions.md#14-text-normalization-for-hashing).
The plain projection is of the runs as stored, and the strict and lossy projections of the runs as printed ([3.6](03-nodes.md#36-the-plain-projection)).
So an `emend`, which corrects the authors' own slip ([11.6](11-corrections.md#116-emendations)), changes a leaf's `text_hash`, `own_hash` and `node_hash`, because the text that is read, quoted and pinned is the emended one.
It leaves the leaf's `match_hash` and `align_key` as they were, because the witnesses they compare the leaf with hold the printed text, and a value computed from the emended text would report every emendation as a disagreement.
Because each `node_hash` covers its children's, the root's `node_hash` covers the whole tree (it is a hash over canonical JSON of `{own, children}`, not an RFC 6962 Merkle root, which [8.5](08-fidelity.md#85-merkle-roots) defines for receipts): a correction to one sentence changes that sentence's values and the `node_hash` of each ancestor, and nothing else. Because each `node_hash` covers its children's, the root's `node_hash` covers the whole tree (it is a hash over canonical JSON of `{own, children}`, not an RFC 6962 Merkle root, which [8.5](08-fidelity.md#85-merkle-roots) defines for receipts): a correction to one sentence changes that sentence's values and the `node_hash` of each ancestor, and nothing else.
`hashes.jsonl` gives every value for every node of a small document, with each node's own content. `hashes.jsonl` gives every value for every node of a small document, with each node's own content.
@@ -27,8 +30,8 @@ A node's own content is an object with its `kind` and these members, and nothing
| Kinds | Members | | Kinds | Members |
| --- | --- | | --- | --- |
| Kinds that take a label | `label`: the label, or null. | | Kinds that take a label | `label`: the label, or null. |
| `sentence`, `cell`, `reference`, `authors`, `keywords` | `runs`: the runs as stored, except that each `equation` run is reduced to `{"kind": "equation"}`, since an ID is never content. | | `sentence`, `cell`, `reference`, `authors`, `keywords` | `runs`: the runs as stored, except that each `equation` run is reduced to `{"kind": "equation"}`, since an ID is never content, and each `math` run is without its `form`. |
| `equation`, `equation-line` | `notation`, `source`. | | `equation`, `equation-line` | `notation`, `source`; never `form`. |
| `code`, `algorithm` | `language` (or null), `text`. | | `code`, `algorithm` | `language` (or null), `text`. |
| `section` | `level`, and `heading`: the `text_hash` of its heading, or null. | | `section` | `level`, and `heading`: the `text_hash` of its heading, or null. |
| `table` | `columns`, as stored. | | `table` | `columns`, as stored. |
@@ -39,10 +42,18 @@ So paragraphs, rows, lists, items and the other containers without content have
Own content covers stored values exactly; it is not normalized. Own content covers stored values exactly; it is not normalized.
A table's columns name header cells by ID, so fixing a header cell changes that cell, not the table. A table's columns name header cells by ID, so fixing a header cell changes that cell, not the table.
A math form ([3.7](03-nodes.md#37-math-forms)) enters no content value.
`form` says how the source is spelled, which the source itself shows, so it is left out of own content, and no projection reads it: a source in canonical TeX is projected as the TeX it is, by the rules of [5.3](#53-the-strict-projection) and [5.4](#54-simple-notation).
So a producer that adds `form` to sources already in the form leaves every hash, ID, pin and anchor as it was, and a reading with `form` and one without agree whenever their sources do.
Respelling a source in the form is an edit of the source: it changes the `text_hash`, `own_hash` and `node_hash` of the node, and, for a complex source, its complex key ([5.3](#53-the-strict-projection)).
Simple sources such as `K_r` and `K_{r}` have one canonical linear form, so their `match_hash` and `align_key` stay; a respelling that turns a simple character into a command [5.4](#54-simple-notation) does not read, as `ħ` becomes `\hbar`, makes the source complex and changes them.
Because the complex key compares sources as stored, a producer that aligns a reading in a form with a second reading ([5.6](#56-alignment)) spells the second reading's math in the same form first, or each complex formula it respelled becomes an `equation` suspect.
`hashes.jsonl` gives the values of a small document whose equation and math runs carry `form`, which are those of the same document without it.
## 5.3 The strict projection ## 5.3 The strict projection
The strict projection folds notation to one canonical linear form and keeps structure, so that it can certify agreement. The strict projection folds notation to one canonical linear form and keeps structure, so that it can certify agreement.
It is computed over a node's runs, or its math, as follows; for other nodes it is the strict projections of the text leaves joined with U+0020, then step 3. It is computed over a node's runs as printed ([3.6](03-nodes.md#36-the-plain-projection)), or its math, as follows; for other nodes it is the strict projections of the text leaves joined with U+0020, then step 3.
1. Apply NFC to every string, then project each run: 1. Apply NFC to every string, then project each run:
@@ -142,25 +153,33 @@ A node's prose length is the number of code points in its lossy projection other
Two readings of a document are aligned as whole sentence sequences, never paragraph by paragraph, because backends disagree about paragraphs at page and column breaks. Two readings of a document are aligned as whole sentence sequences, never paragraph by paragraph, because backends disagree about paragraphs at page and column breaks.
Each reading is a list of sentences in document order, each with its strict projection, lossy projection, complex keys, container and context (the kind of its container: paragraph, heading, caption and so on). Each reading is a list of sentences in document order, each with its strict projection, lossy projection, complex keys, container and context (the kind of its container: paragraph, heading, caption and so on).
Those projections are of the runs as printed ([5.3](#53-the-strict-projection)), so a corrected document aligned with a witness again shows no emendation as a suspect.
1. **Anchors.** A sentence is a candidate when its lossy projection occurs exactly once in each reading and its prose length is at least 20. 1. **Anchors.** A sentence is a candidate when its lossy projection occurs exactly once in each reading and its prose length is at least 20.
Take the candidates in the order of the first reading, and choose the longest subsequence whose positions in the second reading increase; among equally long ones, the one whose sequence of positions in the first reading is lexicographically smallest. Take the candidates in the order of the first reading, and choose the longest subsequence whose positions in the second reading increase; among equally long ones, the one whose sequence of positions in the first reading is lexicographically smallest.
2. **Between anchors**, and before the first and after the last, align by the longest common subsequence of lossy projections. 2. **Between anchors**, and before the first and after the last, align by the longest common subsequence of lossy projections.
Ties are broken deterministically: walking both segments from their start, pair two sentences whose lossy projections are equal; otherwise skip the first reading's sentence when the remainder without it has a common subsequence at least as long as without the second's, and skip the second's otherwise. Ties are broken deterministically: walking both segments from their start, pair two sentences whose lossy projections are equal; otherwise skip the first reading's sentence when the remainder without it has a common subsequence at least as long as without the second's, and skip the second's otherwise.
3. **Compare** each pair. 3. **Substitutions.** A sentence one reading misread has a different lossy projection, so steps 1 and 2 leave it unpaired on both sides; this step pairs it with its other reading.
Steps 1 and 2 leave stretches between consecutive pairs, and before the first and after the last, with unpaired sentences on either side.
In each stretch, take the first reading's unpaired sentences in order, and pair each whose prose length is at least 20 with the first of the second reading's unpaired sentences in the same stretch that comes after the last one paired in this step, has a prose length of at least 20, and has a similarity to it of at least 0.8.
The similarity of two lossy projections *a* and *b* is 2*L* / (|*a*| + |*b*|), where *L* is the length of their longest common subsequence of code points and |*a*| and |*b*| are their lengths in code points.
A pair made here never agrees, since its lossy projections differ, so step 4 makes it a `number`, `equation` or `word` suspect rather than a `dropped` and an `added` one.
4. **Compare** each pair.
They agree when their strict projections and complex keys are equal. They agree when their strict projections and complex keys are equal.
Otherwise the pair is a suspect: of kind `number` when their numeric skeletons differ, else `equation` when their complex keys differ, else `word`. Otherwise the pair is a suspect: of kind `number` when their numeric skeletons differ, else `equation` when their complex keys differ, else `word`.
The numeric skeleton keeps only digits, U+2212, `.`, `,`, `+`, `_{`, `^{` and `}`. The numeric skeleton keeps only digits, U+2212, `.`, `,`, `+`, `_{`, `^{` and `}`.
4. **Structure.** A pair whose contexts differ is also a `structure` suspect; two consecutive pairs that share a container in one reading but not in the other are a `join` suspect. 5. **Structure.** A pair whose contexts differ is also a `structure` suspect; two consecutive pairs that share a container in one reading but not in the other are a `join` suspect.
5. **Unplaced** sentences are `dropped` (in the first reading only) or `added` (in the second only). 6. **Unplaced** sentences are `dropped` (in the first reading only) or `added` (in the second only).
Pairs are listed in the first reading's order, whichever step made them.
Paired sentences keep the first reading's ID even when they disagree, so a reading that lost a superscript is caught rather than accepted. Paired sentences keep the first reading's ID even when they disagree, so a reading that lost a superscript is caught rather than accepted.
`alignment.jsonl` aligns two readings that exercise every rule, including two prose-thin sentences aligned by their order. `alignment.jsonl` aligns two readings that exercise every rule, including two prose-thin sentences aligned by their order, and a second pair of readings with substitutions: a misread number, a misread word, a sentence too short to be paired this way, and two sentences too unlike to be one.
## 5.7 Suspects ## 5.7 Suspects
A suspect, `scienting-suspect/1`, records a doubtful reading for review: its `id`, its `kind` (`word`, `number`, `equation`, `structure`, `join`, `added`, `dropped`, `boundary` for an uncertain sentence boundary, or `reading` for anything else), the node, the readings compared, an optional crop by SHA-256 and a note. A suspect, `scienting-suspect/1`, records a doubtful reading for review: its `id`, its `kind` (`word`, `number`, `equation`, `structure`, `join`, `added`, `dropped`, `boundary` for an uncertain sentence boundary, or `reading` for anything else), the node, the readings compared, an optional crop by SHA-256 and a note.
Suspects may travel in a package's `suspects.jsonl`, sorted by ID, until they are resolved. When the second reading, `b`, is another witness's reading of the node (a PDF's text layer, a JATS file, a LaTeX source), `witness` names that witness by its ID in `witnesses.json`, so a reviewer knows what the node was compared with.
Suspects may travel in a package's `suspects.jsonl`, sorted by ID, until they are resolved; in a package, a suspect's `witness` is one of the package's witnesses.
## 5.8 Keyed hashes ## 5.8 Keyed hashes
spec/06-anchors.md
+2
View File
@@ -105,6 +105,8 @@ The result is `{result, node, node_hash?, span?, drift?, via?, reason?}`: the ou
A `moved` or `reattached` anchor is re-pinned to the live node, and the move is recorded. A `moved` or `reattached` anchor is re-pinned to the live node, and the move is recorded.
Checks are stricter than readers: an extract is a span or leaf anchor and never accepts `changed-below`, because a check is about exact words. Checks are stricter than readers: an extract is a span or leaf anchor and never accepts `changed-below`, because a check is about exact words.
An `emend` changes a leaf's `node_hash` and plain projection as a `replace` does ([11.6](11-corrections.md#116-emendations)), so an anchor on the leaf is judged as after a replace: a quote outside the emended text that occurs once is `reattached` with its new position, and a quote that spans the emended text is `stale`.
A `relink` changes the leaf's `node_hash` but not its plain projection, so an anchor on it is `reattached` by its quote; a `resolve` changes no node, so an anchor on the reference stays `current`.
`resolve.jsonl` resolves seventeen anchors made at revision 0 against `fixtures/packages/hale2026harmonic-damb` at revision 12, with every outcome. `resolve.jsonl` resolves seventeen anchors made at revision 0 against `fixtures/packages/hale2026harmonic-damb` at revision 12, with every outcome.
## 6.6 Drift ## 6.6 Drift
spec/08-fidelity.md
+9 -2
View File
@@ -12,10 +12,16 @@
Levels apply to leaves. Levels apply to leaves.
The pipeline records F0, F1 or F2 in a leaf's `fidelity`; F3 comes only from receipts. The pipeline records F0, F1 or F2 in a leaf's `fidelity`; F3 comes only from receipts.
A correction that changes a leaf drops it back: the gate's crop evidence makes it F1 at most, until it is corroborated or proofread again. A correction that changes a leaf drops it back: the gate's crop evidence makes it F1 at most, until it is corroborated or proofread again.
An `emend`, `relink` or `resolve` ([11.1](11-corrections.md#111-operations)) never does, because a level measures the reading against its witnesses and the page.
An emend corrects the authors, not the reading, and leaves the leaf as printed ([3.6](03-nodes.md#36-the-plain-projection)) as it was; a relink or a resolve changes no text, and the references a citation names are not text a witness corroborates.
Whatever compares a leaf with a witness or with the page compares the leaf as printed: alignment and its suspects ([5.6](05-hashes.md#56-alignment)), corroboration by `match_hash`, a proofreader who signs a receipt, and a spot-check's `errors`.
So an emendation is never reported as a misreading.
## 8.2 Levels ## 8.2 Levels
A leaf's level is the highest of its recorded `fidelity` and the levels of every valid receipt ([8.6](#86-receipt-status)) that lists it at its current `node_hash`. A leaf's level is the highest of its recorded `fidelity` and the levels of every valid receipt ([8.6](#86-receipt-status)) that lists it at its current `node_hash`, or at a `node_hash` that only `emend`, `relink` and `resolve` corrections of the leaf have changed since.
To find those hashes, walk the document's corrections back from the last: one of those three that targets the leaf and whose `after` is the hash reached so far leads back to its `before`, and any other correction that names the leaf, as its `target` or in `touched`, `path`, `to_path`, `created` or `retired`, ends the walk.
A receipt at a lower level never lowers a node; only a revocation, a failed spot-check or a correction does. A receipt at a lower level never lowers a node; only a revocation, a failed spot-check or a correction does.
A spot-check's `qa:` receipt ([8.8](#88-spot-checks)) raises the nodes it sampled, except those its `errors` name. A spot-check's `qa:` receipt ([8.8](#88-spot-checks)) raises the nodes it sampled, except those its `errors` name.
A document reports, in `document.json`, the lowest level among its leaves and the count at each level. A document reports, in `document.json`, the lowest level among its leaves and the count at each level.
@@ -31,7 +37,7 @@ A paper has thousands of nodes, and nobody signs thousands of times, so people s
| `doc` | Yes | The document key. | | `doc` | Yes | The document key. |
| `rev` | Yes | The revision reviewed. | | `rev` | Yes | The revision reviewed. |
| `level` | Yes | `F1`, `F2` or `F3`. | | `level` | Yes | `F1`, `F2` or `F3`. |
| `scope` | Yes | `all`, `section:<id>`, `page:<n>`, `targeted:<name>` (such as `targeted:lode`), or `qa:<receipt id>` for a spot-check. | | `scope` | Yes | `all`, `section:<id>`, `page:<n>`, `targeted:<name>` (such as `targeted:paperclaim`), or `qa:<receipt id>` for a spot-check. |
| `nodes` | Yes | `[[id, node_hash], …]`, sorted by ID without repeats. | | `nodes` | Yes | `[[id, node_hash], …]`, sorted by ID without repeats. |
| `records` | No | `[[kind, id, hash], …]` for other things signed in the sitting: `annotation`, `digitization`, `correction`, `layer` or `checklist`. | | `records` | No | `[[kind, id, hash], …]` for other things signed in the sitting: `annotation`, `digitization`, `correction`, `layer` or `checklist`. |
| `root` | Yes | The Merkle root ([8.5](#85-merkle-roots)). | | `root` | Yes | The Merkle root ([8.5](#85-merkle-roots)). |
@@ -45,6 +51,7 @@ A paper has thousands of nodes, and nobody signs thousands of times, so people s
A receipt lists nodes, records, or both. A receipt lists nodes, records, or both.
A correction invalidates a receipt for the corrected node only, because the node's `node_hash` no longer matches; the rest still counts. A correction invalidates a receipt for the corrected node only, because the node's `node_hash` no longer matches; the rest still counts.
An emend, relink or resolve invalidates none ([8.2](#82-levels)).
A receipt shared outside the team cannot carry plain node hashes, which would leak short texts, and its signature covers them. A receipt shared outside the team cannot carry plain node hashes, which would leak short texts, and its signature covers them.
So an export re-issues it: a new receipt with `export_of`, values under the export's key, signed by the original reviewer or a team key with the `export` purpose. So an export re-issues it: a new receipt with `export_of`, values under the export's key, signed by the original reviewer or a team key with the `export` purpose.
spec/09-figures-tables-equations.md
+80 -1
View File
@@ -34,4 +34,83 @@ For people, a cell can be shown as "Table 2, row CT–CT, column K_r" from its h
An equation keeps its source as written, `tex` from a conversion or `typst` from a draft, and its printed number as its `label`, so numbering never collides with a renderer's own. An equation keeps its source as written, `tex` from a conversion or `typst` from a draft, and its printed number as its `label`, so numbering never collides with a renderer's own.
An equation that the source numbers line by line (Eq. 3a) has `equation-line` children, each with its own label and source. An equation that the source numbers line by line (Eq. 3a) has `equation-line` children, each with its own label and source.
MathML is derived by a pinned tool and recorded, with the tool's name and version, in `derived/mathml.jsonl`. A converter that respells TeX in the standard form of [9.4](#94-the-math-form) stores the respelled source and says so with `form` beside it ([3.7](03-nodes.md#37-math-forms)); the source as written stays in the witness.
MathML is derived by a pinned tool and recorded, with the tool's name and version, in `derived/mathml.jsonl`, and the form's reading of each formula in `derived/math.jsonl` ([12.7](12-package.md#127-derived-files)).
## 9.4 The math form
`scienting-math/1` is the standard form of a formula in a paper: one canonical TeX spelling, a tree read from it, a key for each symbol, and the issues a reader should look at.
A converter writes a package's equations and math runs in it, a reader renders and searches it, and a claim store reads a paper's equations and its own claims through it, so the two can be matched.
papermath is its reference implementation, and what papermath computes is what the form says; this section states what the specification relies on.
The identifier names a spelling of formulas, not a file format, so [registry/formats.json](../registry/formats.json) does not list it; its readings travel in a package's `derived/math.jsonl`.
The form reads TeX only.
**Canonical TeX** respells and never reinterprets, so it renders as the source did apart from presentation:
1. Unicode NFC, and a math character a converter handed over becomes its command: `−` is `-`, `≤` is `\leq`, `α` is `\alpha`.
2. Space is dropped, except inside text and where a command needs one before a letter (`\sin x`).
3. One spelling per meaning: `\le` is `\leq`, `\dfrac` and `\tfrac` are `\frac`, `\textrm` is `\text`, `x^{\prime}` is `x'`.
4. Presentation is dropped: `\displaystyle` and the other styles, `\big` and its sizes, `\color`, and `\left` and `\right` around ordinary delimiters.
5. Every argument and script is braced, and on one base the subscript comes first: `x_i^2` is `x_{i}^{2}`, `\frac12` is `\frac{1}{2}`.
6. `\tag` is lifted out as the label, and `\label`, `\nonumber` and `\notag` are dropped.
7. An environment goes by one name: `align`, `split` and `multline` are `aligned`, `equation` and `gather` are `gathered`.
8. A command outside the form's vocabulary is kept as written and reported.
Spelling a canonical source again gives it back unchanged.
**The tree** is the canonical TeX read as JSON.
A node is an array whose first item is a string, its kind; its other items are its parts: strings, null for a part that is absent, nodes, and groups, which are arrays whose first item is not a string.
Brackets group and leave no node.
| Kind | Parts | Is |
| --- | --- | --- |
| `num` | The digits | A number as printed, a decimal comma read as `.`. |
| `sym` | The key | A symbol. |
| `text` | The words | Words in the formula. |
| `const`, `oper` | The command | A constant, such as `\infty`, or an operator, such as `\partial`. |
| `dots`, `empty` | None | An ellipsis; nothing. |
| `add` | The terms | A sum, in which a subtracted term is a `neg` node, ± a `pm` and ∓ an `mp`, each with one part. |
| `mul` | The factors | A product, by juxtaposition or `\cdot`. |
| `times`, `div`, `op` | Two operands, after the operator's command for `op` | `\times`; `/` or `\div`; any other binary operator. |
| `frac`, `binom`, `sqrt`, `root` | Numerator and denominator; the radicand, after the index for `root` | Fractions, binomials and roots. |
| `pow`, `sub`, `prime`, `fact` | The base, and the script for `pow` and `sub` | Scripts, primes and factorials on what is not a symbol. |
| `prescript` | The lower script, the upper script, the base | Scripts before their base, as in the isotope `^{31}\text{P}`. |
| `mark` | The sign or operator | A sign or operator with no operand: a charge (`x^{+}`), the radical dot of `\cdot\text{OH}`, a formula cut by its converter. |
| `apply` | The function's command, its argument, and its subscript if any | A named function: `\cos(x)`, `\log_{2} n`, `\max_{i} x_{i}`. |
| `lim` | The command, what is under it, the body | `\lim`, `\liminf` and `\limsup`. |
| `sum`, `prod`, `coprod`, `int`, `iint`, `iiint`, `oint`, `bigcup`, `bigcap` | The lower limit, the upper limit, the body | A big operator, whose body runs to the next sign outside brackets. |
| `abs`, `norm`, `angle`, `floor`, `ceil`, `braces` | The inside | A fence. |
| `interval` | The opening and closing brackets, the inside | Brackets that do not match, as in `[0, 1)`. |
| `rel` | Operands with a relation between each two | A relation or a chain; a continued line starts with null. |
| `list` | The formulas | Formulas side by side, at `,`, `;`, `\quad` or `\qquad` outside brackets. |
| `lines` | The lines | The lines of an `aligned` or `gathered` block. |
| `cases` | A group of value and condition for each case | Cases, a condition being null when there is none. |
| `matrix` | The environment, then a group of rows, each a group of cells | A matrix or array. |
| `opaque` | Canonical TeX | What the tree does not read. |
A tree with no `opaque` node is complete.
A later minor version of the form may add a kind, and a consumer that meets a kind it does not know reads that node as it reads `opaque`; `math.schema.json` therefore takes any lowercase kind, and its description lists these.
**A symbol** is a base, which is a letter, a Greek letter, or a font or accent command on a group (`\mathbf{F}`, `\hat{H}`), with any subscript, any superscript that names rather than raises (`R^{*}`, `A^{\dagger}`, `E^{\text{LJ}}`), and primes.
A text group alone is a symbol too, unless it holds several words, is a condition in `cases`, or is the set under a big operator.
A symbol's key is its canonical TeX with `\mathrm` spelled `\text`, `\ast` spelled `*`, `\prime` spelled `'`, and no space inside text: `K_r` and `K_{r}` have the key `K_{r}`, and `r_{\mathrm{eq}}` has the key `r_{\text{eq}}`.
The key is the one a paperclaim claim store gives a symbol, so a symbol in a paper and the same symbol in a claim have one key, and a `symbol` annotation names its symbol by it ([18.4](18-layers.md#184-bodies)).
**Issues** never stop the reading; each is `{code, message, at}`, with `at` the canonical TeX it is about.
| Code | Means |
| --- | --- |
| `unbalanced` | A brace, bracket or fence does not close; with a brace, the whole formula is one `opaque` node. |
| `unknown-command` | A command outside the vocabulary, kept as written. |
| `label-or-power` | A superscript that may be a label written without `\text`, such as `E^{LJ}`, read as a power. |
| `script-on-group` | A script on a braced group rather than on a symbol. |
| `opaque` | A part the tree does not read. |
| `infix-fraction` | `\over`, `\choose` or `\atop`, rewritten as `\frac` or `\binom`. |
A later minor version may add a code.
**In a package**, the canonical source is the equation's or the math run's `source`, with `form` beside it ([3.7](03-nodes.md#37-math-forms)), and the rest is derived: one line of `derived/math.jsonl` per formula a pinned tool read, `{node, run, tool, form, tree, symbols, complete, issues, label?}`.
`run` is the index of the math run among the node's runs, or null for an equation or equation line, as in `derived/mathml.jsonl`; `symbols` lists the keys in the order the formula first writes them; `label` is the label the tool lifted from a `\tag`, omitted when the source has none, as a canonical source never does.
The schema is `math.schema.json`.
A validator checks what it can without implementing the form: that each line names a formula of the package (`doc.derived.node`), and that the formula is TeX, `complete` is true exactly when the tree has no `opaque` node, and `symbols` holds each `sym` key of the tree once (`doc.derived.math`).
`fixtures/packages/hale2026harmonic-damb/derived/math.jsonl` holds papermath's reading of the example's equation, of a math run that is one symbol, and of the run `_2` of "H₂O", which the tree cannot read alone.
spec/10-references.md
+4
View File
@@ -9,7 +9,11 @@ Each reference is a `reference` node holding its printed runs, exactly as printe
| `{type: "document", doc}` | It names a document in the same store, by key. | | `{type: "document", doc}` | It names a document in the same store, by key. |
| `{type: "none"}` | Nothing yet. | | `{type: "none"}` | Nothing yet. |
A reference without a record has the resolution `none`.
A `resolve` correction sets a reference's resolution, and its record keeps the one before ([11.7](11-corrections.md#117-citations-and-references)).
Every `cite` run links a sentence to one or more references. Every `cite` run links a sentence to one or more references.
A `relink` correction changes the references a run names, makes a text run a cite run, or makes a cite run text again, without changing its text ([11.7](11-corrections.md#117-citations-and-references)).
The citation graph is therefore sentence-level: for any reference, the format gives every sentence that cites it, which is the citation's context. The citation graph is therefore sentence-level: for any reference, the format gives every sentence that cites it, which is the citation's context.
The database derives a `citations` table from the runs ([13](13-database.md)). The database derives a `citations` table from the runs ([13](13-database.md)).
An article's references and its Supporting Information's are separate documents' references and never mix. An article's references and its Supporting Information's are separate documents' references and never mix.
spec/11-corrections.md
+62 -12
View File
@@ -18,9 +18,14 @@ Each has an `index`, counted from 1 in the order corrections are applied, and th
| `reclassify` | `target`, `kind` | Changes a node's kind, such as a paragraph that is really a caption, to a kind with the same content shape. | Unchanged. | | `reclassify` | `target`, `kind` | Changes a node's kind, such as a paragraph that is really a caption, to a kind with the same content shape. | Unchanged. |
| `relabel` | `target`, `label` | Changes a printed label. | Unchanged. | | `relabel` | `target`, `label` | Changes a printed label. | Unchanged. |
| `set` | `target`, `attr`, `value` | Sets one structural attribute: a section's `level`, a table's `columns`, a cell's `column`, `rowspan`, `colspan` or `role`, a row's `header_path`, a list's `ordered`, an item's `marker` or a theorem's `variant`. | Unchanged. | | `set` | `target`, `attr`, `value` | Sets one structural attribute: a section's `level`, a table's `columns`, a cell's `column`, `rowspan`, `colspan` or `role`, a row's `header_path`, a list's `ordered`, an item's `marker` or a theorem's `variant`. | Unchanged. |
| `emend` | `target`, `find`, `replace`, `kind`, optional `run` and `note` | Corrects the authors' own slip, where the reading matches the page and the page is wrong: replaces the one occurrence of `find` among the target's prose strings with `replace`, a non-empty string, and keeps what was printed ([11.6](#116-emendations)). `kind` is `typo`, `spelling`, `grammar`, `punctuation`, `wording` or `other`. | Unchanged. |
| `relink` | `target`, `run`, `refs` | Makes run `run` of the target a `cite` run naming `refs`, a non-empty list of reference nodes, or with `refs` null makes a `cite` run a `text` run, keeping its text ([11.7](#117-citations-and-references)). | Unchanged. |
| `resolve` | `target`, `resolution` | Sets a reference node's resolution in `references.jsonl` ([11.7](#117-citations-and-references)). | Unchanged. |
A leaf's editable strings are the `text` of its `text`, `link`, `cite`, `xref` and `fnmark` runs and the `source` of its `math` runs, each searched on its own; an equation's `source`; a verbatim node's `text`. A leaf's editable strings are the `text` of its `text`, `link`, `cite`, `xref` and `fnmark` runs and the `source` of its `math` runs, each searched on its own; an equation's `source`; a verbatim node's `text`.
Its prose strings, which an emend searches, are the `text` of those five run kinds alone: never a math or equation source, and never a verbatim node's text.
`set` exists because readings drop spans, roles and header paths; see the [rationale](../docs/rationale.md#corrections). `set` exists because readings drop spans, roles and header paths; see the [rationale](../docs/rationale.md#corrections).
`emend` exists because a faithful reading of a wrong page is not a misreading: it corrects the paper while keeping what was printed ([11.6](#116-emendations)).
Runs are made canonical after every operation, and the result MUST satisfy every rule of [3](03-nodes.md). Runs are made canonical after every operation, and the result MUST satisfy every rule of [3](03-nodes.md).
## 11.2 Every correction records ## 11.2 Every correction records
@@ -43,9 +48,9 @@ When a correction is applied, its record gains:
| `before`, `after` | The target's `node_hash` before and after, or null; for an inserted subtree, `target` is its root. | | `before`, `after` | The target's `node_hash` before and after, or null; for an inserted subtree, `target` is its root. |
| `path` | The target's ancestors when it was applied, root first; for an insert, the new subtree's. | | `path` | The target's ancestors when it was applied, root first; for an insert, the new subtree's. |
| `to_path` | For a move, the new ancestors. | | `to_path` | For a move, the new ancestors. |
| `touched` | The leaves whose content the correction changed, sorted: the target of a `replace`, text `insert`, `relabel` or `set` on a leaf; the new leaves of a node `insert`; the deleted leaves of a `delete`; both parts of a `split`; both nodes of a run `merge`, or the moved-in leaves of a container `merge`; the moved leaves of a `move`; the leaves of a reclassified node. | | `touched` | The leaves whose content the correction changed, sorted: the target of a `replace`, text `insert`, `relabel` or `set` on a leaf, and of an `emend` or `relink`; the new leaves of a node `insert`; the deleted leaves of a `delete`; both parts of a `split`; both nodes of a run `merge`, or the moved-in leaves of a container `merge`; the moved leaves of a `move`; the leaves of a reclassified node; none for a `resolve`. |
| `created`, `retired` | IDs minted or retired. | | `created`, `retired` | IDs minted or retired. |
| `from` | The label, kind or attribute value before a `relabel`, `reclassify` or `set`. | | `from` | The label, kind or attribute value before a `relabel`, `reclassify` or `set`; the references the run named before a `relink`, or null for a text run; the resolution before a `resolve`. |
Retirements are also entered in `ids.jsonl` with the correction's index as `rev`. Retirements are also entered in `ids.jsonl` with the correction's index as `rev`.
Drift ([6.6](06-anchors.md#66-drift)) is computed from `path`, `to_path` and `touched` alone, so it needs no replay. Drift ([6.6](06-anchors.md#66-drift)) is computed from `path`, `to_path` and `touched` alone, so it needs no replay.
@@ -58,24 +63,22 @@ A correction is refused, naming the rule ([22](22-conformance.md)), when:
2. a target, parent or sibling is not live, or the root would be deleted, moved, merged or reclassified; 2. a target, parent or sibling is not live, or the root would be deleted, moved, merged or reclassified;
3. `find` is empty or does not occur exactly once; 3. `find` is empty or does not occur exactly once;
4. an `insert` has no crop evidence, or names a run or offset its target does not have; 4. an `insert` has no crop evidence, or names a run or offset its target does not have;
5. a digit changes without crop evidence: in a `replace` whose `find` and replacement have different digits, a `delete` of leaves that hold digits, or a `relabel` whose labels have different digits; 5. a digit changes without crop evidence: in a `replace` whose `find` and replacement have different digits, a `delete` of leaves that hold digits, or a `relabel` whose labels have different digits; or an `emend` changes, adds or removes a digit, or edits beside one, whatever its evidence ([11.6](#116-emendations));
6. any leaf outside `touched` has different digits afterwards; 6. any leaf outside `touched` has different digits afterwards;
7. a `split` falls outside the sentence, inside a run other than text, or leaves a part empty; 7. a `split` falls outside the sentence, inside a run other than text or inside an emended run, or leaves a part empty;
8. a `merge` joins nodes that differ in kind, are not mergeable, or are not adjacent siblings; 8. a `merge` joins nodes that differ in kind, are not mergeable, or are not adjacent siblings;
9. a `move` would put a node inside itself; 9. a `move` would put a node inside itself;
10. a `reclassify` changes the content shape, or a `relabel` or `set` targets a kind that does not take it; or 10. a `reclassify` changes the content shape, or a `relabel` or `set` targets a kind that does not take it;
11. the result breaks a rule of [3](03-nodes.md). 11. the result breaks a rule of [3](03-nodes.md);
12. an `emend` targets a node without runs, or finds its text only in a math or equation source or only across runs; or a `replace` or text `insert` would change an emended run, or a correction other than an emend writes `printed`;
13. a `relink` names a run that is neither text nor a cite, or unlinks a run that is not a cite; or
14. a `resolve` names a node that is not a reference.
The gate needs only the document's current state, so a tool can run it on a proposal before applying anything, and get back the record the correction would append or the rule it breaks. The gate needs only the document's current state, so a tool can run it on a proposal before applying anything, and get back the record the correction would append or the rule it breaks.
<<<<<<< HEAD
`paperspec validate --correction` does this against a package. `paperspec validate --correction` does this against a package.
||||||| parent of 4fa326d (First implementation (#1))
=======
`tools/validate.py --correction` does this against a package.
>>>>>>> 4fa326d (First implementation (#1))
Digits are characters of General Category Nd in the plain projection. Digits are characters of General Category Nd in the plain projection.
A package's corrections are also checked as a whole: indexes run from 1 without gaps, each `id` derives from its record, `rev` equals their number, every retired ID is in `ids.jsonl`, and the last correction of each leaf ends at its current `node_hash`. A package's corrections are also checked as a whole: indexes run from 1 without gaps, each `id` derives from its record, `rev` equals their number, every retired ID is in `ids.jsonl`, the last correction of each leaf ends at its current `node_hash`, and the last `resolve` of each reference ends at its resolution in `references.jsonl`.
`corrections.jsonl` in the vectors applies each operation, and each refusal, to a small document. `corrections.jsonl` in the vectors applies each operation, and each refusal, to a small document.
## 11.5 The history root ## 11.5 The history root
@@ -85,3 +88,50 @@ Because corrections are only appended, a consistency proof (RFC 9162 §2.1.4) be
The proof is a `scienting-proof/1` with `kind` `consistency`, holding the `doc`, `from_rev`, `to_rev`, `from_root`, `to_root` and the `path` as hex hashes; a verifier applies RFC 9162 §2.1.4.2. The proof is a `scienting-proof/1` with `kind` `consistency`, holding the `doc`, `from_rev`, `to_rev`, `from_root`, `to_root` and the `path` as hex hashes; a verifier applies RFC 9162 §2.1.4.2.
A checks-only package carries no corrections, so its history is the empty root. A checks-only package carries no corrections, so its history is the empty root.
`proofs.jsonl` gives consistency proofs for every prefix of small trees and for the example paper between revisions 8 and 12; `fixtures/records/proof-consistency.json` is the latter. `proofs.jsonl` gives consistency proofs for every prefix of small trees and for the example paper between revisions 8 and 12; `fixtures/records/proof-consistency.json` is the latter.
## 11.6 Emendations
A `replace` fixes the reading: the converter misread the page, and the page is right.
An `emend` fixes the paper: the reading matches the page, and the authors printed a typo, a misspelling, a slip of grammar or punctuation, or a wording an editor corrects for readers.
The two claim different things, so different things follow from them.
**What is kept.**
In a `text` run, the found text becomes a run of its own, with the style of the run it was in, whose `text` is the replacement and whose `printed` is the found text.
In a `link`, `cite`, `xref` or `fnmark` run, or a text run emended before, the run's text changes and `printed` keeps its whole text as it was first printed.
The record keeps `find`, the printed string, and `replace`, the emended one, so the history says what each emendation was, while the runs say where it is after any later split, merge or move.
A leaf as printed ([3.6](03-nodes.md#36-the-plain-projection)) is therefore recovered from `nodes.jsonl` alone: each run's `printed` in place of its text.
**The gate.**
An emend searches the target's prose strings for exactly one occurrence, as a `replace` searches its editable strings, but never a math or equation source, which `replace` corrects.
The replacement is a string, never runs, and is not empty; to delete a word, an editor finds it with a neighbor and keeps the neighbor.
The edit stays inside one run, so a `find` that occurs only across runs is refused.
No digit may change, be added or removed, and the edit may not touch one: setting aside the longest common prefix of the run's string before and after, and then the longest common suffix of the rest, what remains holds no digit in either string, and the unchanged character on each side of it is not a digit.
So an emend never turns "3,5" into "3.5": a number is data, corrected by `replace` with crop evidence, even where the authors printed it wrong.
Only an emend writes `printed`.
A `replace` or text `insert` that would change an emended run, a `split` inside one, and replacement runs or inserted nodes that carry `printed` are refused, and a second emend of an emended run keeps what was first printed.
**What follows.**
1. The leaf keeps its level ([8.1](08-fidelity.md#81-levels-of-fidelity), [8.2](08-fidelity.md#82-levels)): a level measures the reading against its witnesses and the page, and the leaf as printed is what it was.
2. Whatever compares the leaf with a witness or the page compares it as printed, so an emendation is never reported as a misreading ([8.1](08-fidelity.md#81-levels-of-fidelity)).
3. The leaf's `text_hash`, `own_hash` and `node_hash` change, as for any change of text, because the text read, quoted and pinned is the emended one; its `match_hash` and `align_key`, which are of the runs as printed, do not ([5.1](05-hashes.md#51-the-five-values)).
4. Anchors on the leaf relocate by their quote, as after a `replace` ([6.5](06-anchors.md#65-resolution)).
`corrections.jsonl` emends a word and a cross-reference's printed form, and refuses a changed digit, a `find` across runs, a `find` in math, an emend of a node without runs and a `replace` of an emended run; each emend's expectation adds the leaf as printed.
## 11.7 Citations and references
A `relink` changes what a citation names, never its text, so it needs no crop.
With `refs` a non-empty list of reference nodes, run `run` becomes a `cite` run naming them, or stays one, and keeps its text: a text run's whole text becomes the cite's printed form, and its style is dropped, since a cite run has none ([3.5](03-nodes.md#35-inline-runs)).
With `refs` null, a cite run becomes a text run with the same text.
Anything an emend kept in `printed` stays.
Every reference must be a `reference` node, as for any cite run (rule 11 of [11.4](#114-the-gate)), and `from` keeps the references the run named before, or null for a text run.
A relink changes no plain projection, so the leaf keeps its level.
Turning a text run into a cite, or back, still changes the leaf's `match_hash` and `align_key`, because the strict projection marks a citation ([5.3](05-hashes.md#53-the-strict-projection)); changing only which references a cite run names changes neither.
To cite part of a text run, a `replace` gives that part as runs that include the cite run ([11.1](#111-operations)).
A `resolve` sets a reference node's resolution in `references.jsonl` ([10](10-references.md)) to any valid one: `doi`, `document` or `none`.
The record keeps the resolution before in `from`, which is `{type: "none"}` for a reference without a record; the resolve then adds the record, in document order.
No content value covers `references.jsonl`, so a resolve changes no node's values: its `before` and `after` are the reference's unchanged `node_hash`, and `touched` is empty.
It changes `references.jsonl`, that file's entry in the manifest, the revision and the history root, as every correction does ([11.5](#115-the-history-root)).
`corrections.jsonl` links a citation printed as a superscript, unlinks a cite run, resolves a reference to a DOI, and refuses a relink of a cross-reference, a cite of a node that is not a reference and a resolve of a sentence.
spec/12-package.md
+13 -3
View File
@@ -32,6 +32,7 @@ Paths use `/`, are relative, have no empty, `.` or `..` segment, no backslash an
No two paths are the same once case-folded, so a package reads alike on every file system. No two paths are the same once case-folded, so a package reads alike on every file system.
A package folder carries no symbolic links, and a consumer refuses one that does, as it refuses a package whose files exceed its own size limit. A package folder carries no symbolic links, and a consumer refuses one that does, as it refuses a package whose files exceed its own size limit.
In a zip, entries are UTF-8 named, sit under one top-level folder named by the key, are stored or deflated, carry no symbolic links, and are unique; a consumer refuses an archive that breaks any of these, or whose expanded size exceeds its own limit. In a zip, entries are UTF-8 named, sit under one top-level folder named by the key, are stored or deflated, carry no symbolic links, and are unique; a consumer refuses an archive that breaks any of these, or whose expanded size exceeds its own limit.
It refuses in the same way (`package.zip.entry`) an archive it cannot read, such as one whose entries need a later version of the zip format than it implements, or whose deflated data does not inflate.
A producer that wants byte-identical archives writes entries sorted by path, dated 1980-01-01 00:00, with no extra fields; `fixtures/packages/tiny2026tiny-j4qm.sct.zip` is such an archive. A producer that wants byte-identical archives writes entries sorted by path, dated 1980-01-01 00:00, with no extra fields; `fixtures/packages/tiny2026tiny-j4qm.sct.zip` is such an archive.
## 12.3 The manifest ## 12.3 The manifest
@@ -50,7 +51,7 @@ Every stage that uses a model or an outside service records where it ran ([14](1
| Policy | Means | | Policy | Means |
| --- | --- | | --- | --- |
| `full` | Every leaf keeps its content. | | `full` | Every leaf keeps its content. |
| `checks-only` | Every leaf has `withheld: true` and no `content`; panels drop their figure text; references keep their resolution but no CSL; there are no corrections, suspects or datasets; and the hashing scheme is keyed: `hmac-team` between the team's machines and `hkdf-pages` outside the team ([5.8](05-hashes.md#58-keyed-hashes)). | | `checks-only` | Every leaf has `withheld: true` and no `content`; panels drop their figure text; references keep their resolution but no CSL; there are no corrections, suspects, datasets or derived files; and the hashing scheme is keyed: `hmac-team` between the team's machines and `hkdf-pages` outside the team ([5.8](05-hashes.md#58-keyed-hashes)). |
A checks-only package is how a team shares its annotations on a paywalled paper without sharing the paper. A checks-only package is how a team shares its annotations on a paywalled paper without sharing the paper.
Its text lives in the team's private asset store, and only key holders can check its values. Its text lives in the team's private asset store, and only key holders can check its values.
@@ -63,8 +64,17 @@ Bytes that live only in the asset store are listed in the manifest with `present
## 12.7 Derived files ## 12.7 Derived files
`derived/` holds what a pinned tool makes from the package, such as `derived/mathml.jsonl`, whose lines are `{node, run, tool, mathml}` with `run` null for an equation node. `derived/` holds what a pinned tool makes from the package.
Nothing derived is content, and a consumer may delete and rebuild it. Each line of a derived file names a formula by `node` and `run`, the index of a math run among the node's runs, or null for an equation or equation line, and the `tool` (`{name, version}`) that made it.
| Path | Lines | Schema |
| --- | --- | --- |
| `derived/mathml.jsonl` | `{node, run, tool, mathml}`: the formula rendered as MathML. | `mathml.schema.json` |
| `derived/math.jsonl` | `{node, run, tool, form, tree, symbols, complete, issues, label?}`: the formula read in a math form ([9.4](09-figures-tables-equations.md#94-the-math-form)). | `math.schema.json` |
Nothing derived is content, and a consumer may delete and rebuild it; a validator does not recompute a derived file, since it does not run the tool, but checks its lines against their schema and that each names a formula of the package (`doc.derived.node`), and that a math line's formula is TeX and its `complete` and `symbols` agree with its tree (`doc.derived.math`).
A derived file reads the text, so a checks-only package carries none ([12.5](#125-text-policies)).
A correction can delete a formula, change it or move its run, so a producer that applies one rebuilds the derived files or leaves them out.
## 12.8 Media type and extension ## 12.8 Media type and extension
spec/13-database.md
+1 -1
View File
@@ -39,7 +39,7 @@ Unloading gives the package back byte for byte, and its manifest hash with it; t
## 13.4 The store tables ## 13.4 The store tables
`store.sql` adds what a team adds to its documents: `roster_keys`, `projects`, `members`, `project_documents`, `reading_paths`, `layers`, `annotations`, `annotation_targets` (derived, so a reader finds the annotations on a node), `datasets`, `dataset_values`, `digitizations`, `views`, `policies`, `submissions` and `grades`. `store.sql` adds what a team adds to its documents: `roster_keys`, `projects`, `members`, `project_documents`, `reading_paths`, `layers`, `annotations`, `annotation_targets` (derived, so a reader finds the annotations on a node), `datasets`, `dataset_values`, `digitizations`, `views`, `policies`, `submissions` and `grades`.
Each keeps its record whole in a `record` or `descriptor` column, with the columns a query needs beside it. Each keeps its record whole in a `record` or `descriptor` column, with the columns a query needs beside it, such as an annotation's `body_kind` and its `category`, which is NULL when the body has none.
## 13.5 The bundle ## 13.5 The bundle
spec/18-layers.md
+77 -6
View File
@@ -43,21 +43,92 @@ Replies are annotations that target annotations, so discussions are threads anch
| `kind` | W3C type | Adds | Says | | `kind` | W3C type | Adds | Says |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| `gloss` | `TextualBody` | `audience` | An explanation at `novice`, `student` or `expert` level. | | `gloss` | `TextualBody` | `audience` | An explanation at `novice`, `student` or `expert` level. |
| `claim-link` | `SpecificResource` | `source` (`lode:<tag>`), `pin`, `role` | That the span `states`, `uses` or `disputes` a claim. | | `claim-link` | `SpecificResource` | `source`, `pin`, `role` | That the span `states`, `uses` or `disputes` a claim. |
| `symbol` | `SpecificResource` | `source`, `pin`, `symbol` | That this occurrence of a symbol is a symbol claim. | | `symbol` | `SpecificResource` | `source`, `pin`, `key`, `symbol` | That this occurrence of a symbol, the one whose key is `key`, is the symbol a symbol claim defines. |
| `comment` | `TextualBody` | | A remark, opinion or interpretation. | | `comment` | `TextualBody` | `category` | A remark, opinion or interpretation. |
| `caveat` | `TextualBody` | | A warning a reader should see. | | `caveat` | `TextualBody` | `category` | A warning a reader should see. |
| `question` | `TextualBody` | `for` | A question for a `reader`, a `group` or the `authors`. | | `question` | `TextualBody` | `for`, `category` | A question for a `reader`, a `group` or the `authors`. |
| `exercise` | `TextualBody` | `exercise` | A task with a key and a rule ([19](19-exercises.md)). | | `exercise` | `TextualBody` | `exercise` | A task with a key and a rule ([19](19-exercises.md)). |
| `citation-context` | `TextualBody` | `reference`, `verdict`, `reason` | Why a sentence cites a reference, and whether it supports it. | | `citation-context` | `TextualBody` | `reference`, `verdict`, `reason` | Why a sentence cites a reference, and whether it supports it. |
| `prerequisite` | `SpecificResource` | `source` | What to read or know first. | | `prerequisite` | `SpecificResource` | `source` | What to read or know first. |
| `crosslink` | `SpecificResource` | `source`, `selector` | That the span corresponds to a span in another document. | | `crosslink` | `SpecificResource` | `source`, `selector` | That the span corresponds to a span in another document. |
| `trace` | `SpecificResource` | `source`, `selector`, `reference`, `note`, `verdict`, `proposer` | That the target's sentence cites `reference` on the strength of a passage of the cited document ([18.5](#185-traced-citations)). |
| `translation` | `TextualBody` | `language` | The span in another language. | | `translation` | `TextualBody` | `language` | The span in another language. |
| `description` | `TextualBody` | `role` | A description of a figure, panel or table for readers who cannot see it: `alt`, a short text equivalent, or `long`, a full description. | | `description` | `TextualBody` | `role` | A description of a figure, panel or table for readers who cannot see it: `alt`, a short text equivalent, or `long`, a full description. |
| `generated` | `TextualBody` | `generator` | Text a model wrote about the span; never mistaken for source. `generator` is `{model, run}` as a correction's model author is ([11.2](11-corrections.md#112-every-correction-records)). | | `generated` | `TextualBody` | `generator` | Text a model wrote about the span; never mistaken for source. `generator` is `{model, run}` as a correction's model author is ([11.2](11-corrections.md#112-every-correction-records)). |
A textual body has a `value`, and optionally a `format` (`text/plain` or `text/markdown`) and a `language`. A textual body has a `value`, and optionally a `format` (`text/plain` or `text/markdown`) and a `language`.
The `source` of a `claim-link` or `symbol` body is a claim in a paperclaim claim store, written as paperclaim writes a reference: `claim:` and the claim's tag, eight Crockford base32 capitals (the digits and the letters without I, L, O and U), as in `claim:272XTX7C`, or `claim:<host>/<tag>` for a claim in another store, `<host>` being a domain name.
A tag never changes, while a claim's slug, such as `amber/k-bond`, may be renamed, so a slug is never a reference.
The optional `pin` fixes the claim as the annotator read it: the same reference, `@`, and the first six lowercase hex digits of the claim's statement hash, as in `claim:272XTX7C@02f47b`.
`common.schema.json` defines both, as `claimRef` and `claimPin`.
A `symbol` body names its symbol twice, as the target's span and by `key`, the symbol's key in the math form `scienting-math/1` ([9.4](09-figures-tables-equations.md#94-the-math-form)), which a claim store gives the symbol a symbol claim defines: `K_{r}` for a span `$K_r$`.
So a reader finds every occurrence of a symbol across a library by its key, however its source spaced or braced it, and a claim store matches the annotation to its claim without parsing the span.
The span is still required, since the key says which symbol and the span says where; `symbol`, optional, is the symbol as the span writes it.
A `comment`, `caveat` or `question` body MAY carry a `category`, which says what it is about, from a fixed list:
| `category` | Says |
| --- | --- |
| `comment` | A general remark. |
| `context` | Background, related work, or what a reader should know to follow the span. |
| `critique` | A flaw in the method, the analysis or the argument. |
| `question` | A question. |
| `correction` | An error of fact, number, citation or reference in the paper. |
| `feedback` | A suggestion addressed to the authors. |
| `resource` | Data, code or a related paper. |
A producer that supports categories SHOULD set one on every comment, caveat and question; the property is optional only so that layers written before it stay valid.
The category is independent of the kind, so a caveat may be a `correction` and a question a `critique`.
A `correction` category says that a reader found an error in the paper; correcting the document's reading of it is a correction of [11](11-corrections.md).
The store's `annotations` table keeps the category beside the body's kind ([13.4](13-database.md#134-the-store-tables)).
Nothing a model writes is ever stored as source text. Nothing a model writes is ever stored as source text.
A model may propose a `description`, and like any annotation it stays `proposed` until a person confirms it; an accessible export uses confirmed descriptions only, and a figure with none has no description. A model may propose a `description`, and like any annotation it stays `proposed` until a person confirms it; an accessible export uses confirmed descriptions only, and a figure with none has no description.
Other text a model writes about a span, such as a summary, is a `generated` annotation, which names its `generator`. Other text a model writes about a span, such as a summary, is a `generated` annotation, which names its `generator`.
`fixtures/records/layer/` holds a layer with one annotation of each common kind and the receipt that confirms some of them. `fixtures/records/layer/` holds a layer with one annotation of each common kind, each comment, caveat and question with its category, a claim link and a symbol that reference a paperclaim claim and pin it, and the receipt that confirms some of them.
## 18.5 Traced citations
"As previously reported [12]": a reader who finds where the cited paper reports it marks the passage, and the `trace` body keeps the mark.
A `crosslink` says only that two spans correspond, in both directions, whereas a trace goes from one sentence's citation to one passage, and says which reference it follows, why, and who or what found it.
The annotation's target is a leaf or span anchor ([6.4](06-anchors.md#64-four-kinds-of-anchor)) on the citing sentence, which MUST be a leaf with runs, a sentence or a cell, and MUST cite the reference the body names: one of its `cite` runs lists `reference` ([3.5](03-nodes.md#35-inline-runs)).
`reference` is the ID of a `reference` node of the citing document, because a sentence may cite several references at once, as in `[3, 12]`, and the target says only which sentence.
`source` is the cited document, written as a target's source is, `tag:scient.ing,2026:doc:<store>/<key>`, and never the citing document itself.
`selector` selects the passage in it: an `AnchorSelector` whose anchor holds the passage's pin (its `node_hash`) and, for part of a leaf, its `span`, then a `TextQuoteSelector` when the anchor's quote is plain ([18.3](#183-the-annotation)).
The passage may be a container, such as a table, whose anchor has no quote.
| Property | Required | Holds |
| --- | --- | --- |
| `reference` | Yes | The reference node of the citing document that the sentence cites and the trace follows. |
| `source`, `selector` | Yes | The cited document and the passage in it. |
| `note` | No | What the passage reports, in a person's or a tool's words. |
| `verdict` | No | `supports`, `partly`, `does-not` or `cannot-tell`, as a `citation-context` body has it ([19.2](19-exercises.md#192-verdicts)). |
| `proposer` | When the creator is `Software` | `{model, run, score?, rank?}`: the model and run that found the passage as a candidate, as a `generated` body's `generator` names them, the candidate's `score` as a decimal string, and its `rank`, from 1, among the candidates the run proposed for this citation. |
The state of a trace is the annotation's `status`.
A trace a tool proposes is `proposed` until a person confirms it, and `rejected` when a person turns the candidate down; a rejected candidate is kept, so that the tool is not offered it again.
A trace is *disputed* while a reply to it, an annotation whose target is the trace and whose motivation is `assessing`, is `proposed` or `confirmed`.
The dispute is computed from the replies and never written on the trace, so a reply can be withdrawn.
Whether a trace still leads where it did is also computed, from the current state of both documents.
Resolve the target's anchor against the citing document and the body's anchor against the cited document ([6.5](06-anchors.md#65-resolution)).
The first row that applies gives the outcome:
| Outcome | When |
| --- | --- |
| `stale` | An anchor is stale, on the citing side, the cited side or both. |
| `unlinked` | The sentence no longer cites `reference`: a `relink` removed the citation ([11.7](11-corrections.md#117-citations-and-references)), or the reference was deleted. |
| `misresolved` | A `resolve` made the reference name a DOI or a document that is not the cited work. A resolution of `none` names nothing and contradicts nothing ([10](10-references.md)). |
| `changed` | Both anchors resolve and at least one is not `current`. |
| `current` | Both anchors are `current`, and the reference still resolves to the cited work or to nothing. |
A resolution names the cited work when its document is the cited document, another version of the same work (a key with the same work key, [2.4](02-model.md#24-work-and-document-keys)), or one the caller knows to be the work; and a `doi` resolution does when the work is known by that DOI (`same_as`, [2.4](02-model.md#24-work-and-document-keys)).
Besides the outcome, a resolution gives the passage's current address, `key#node@pin[start:end]`, while its anchor resolves, and the citations inside the passage.
A chain of "as previously reported" is then followed one trace at a time: the reader opens the passage, and a citation in it that has a trace of its own leads on.
A trace breaks `layer.trace.source` when its selector has no `AnchorSelector`, an anchor is in another document than `source` names, `source` is the citing document, or the `TextQuoteSelector` does not repeat the anchor's plain quote.
It breaks `layer.trace.target` when the target is not a leaf with runs of the citing document selected by an `AnchorSelector`, or the sentence does not cite `reference` ([22.4](22-conformance.md#224-rules)).
`vectors/traces.jsonl` resolves traces against the fixture packages after corrections, with every outcome and every refusal, and `fixtures/records/traces/` holds a layer of four annotations on `writer2027fixed-ynr7.v3` that cites `hale2026harmonic-damb`: a trace a model proposed to the paper's abstract and a person confirmed with a receipt, a candidate a person rejected, a person's trace to Table 1, and a reply that disputes the first.
spec/20-projects.md
+2 -2
View File
@@ -7,9 +7,9 @@ A project, `scienting-project/1`, is the unit a team works in.
| `key`, `store`, `name`, `description` | Its identity. | | `key`, `store`, `name`, `description` | Its identity. |
| `members` | People with a `role` (`owner`, `editor`, `reader` or `guest`) and the roster keys they sign with. | | `members` | People with a `role` (`owner`, `editor`, `reader` or `guest`) and the roster keys they sign with. |
| `documents` | The project's documents, each with the `reason` it is included, `tags`, and a reading `status` per member: `unread`, `reading` or `read`. | | `documents` | The project's documents, each with the `reason` it is included, `tags`, and a reading `status` per member: `unread`, `reading` or `read`. |
| `paths` | Reading paths: ordered steps through documents, nodes, claims and layers, for onboarding a student or preparing a reading group. | | `paths` | Reading paths: ordered steps through documents, nodes, claims and layers, for onboarding a student or preparing a reading group; a claim step's `ref` is a claim reference, `claim:<tag>` ([18.4](18-layers.md#184-bodies)). |
| `layers`, `datasets`, `views`, `drafts` | What the project holds. | | `layers`, `datasets`, `views`, `drafts` | What the project holds. |
| `claims` | The claim namespace or bundle it uses, as the `lode` setting. | | `claims` | The claim namespace or bundle it uses, as the `paperclaim` setting. |
| `settings` | The digit rule for targeted F3 ([8.10](08-fidelity.md#810-targeted-f3)), the drift threshold ([6.6](06-anchors.md#66-drift)), the spot-check rate and minimum ([8.8](08-fidelity.md#88-spot-checks)), what happens when a right is unknown (`refuse`, the default, or `warn`), the stage requirements in use, as the `id` of their stage file ([14](14-rights.md)), and the project's transparency `log`: its `origin`, the roster keys that `witness` it, and the `threshold` of witnesses a checkpoint needs ([23](23-log.md)). | | `settings` | The digit rule for targeted F3 ([8.10](08-fidelity.md#810-targeted-f3)), the drift threshold ([6.6](06-anchors.md#66-drift)), the spot-check rate and minimum ([8.8](08-fidelity.md#88-spot-checks)), what happens when a right is unknown (`refuse`, the default, or `warn`), the stage requirements in use, as the `id` of their stage file ([14](14-rights.md)), and the project's transparency `log`: its `origin`, the roster keys that `witness` it, and the `threshold` of witnesses a checkpoint needs ([23](23-log.md)). |
Projects are small on purpose: everything about them is designed for hundreds of documents, not millions. Projects are small on purpose: everything about them is designed for hundreds of documents, not millions.
spec/22-conformance.md
+8 -8
View File
@@ -6,12 +6,7 @@
| --- | --- | --- | | --- | --- | --- |
| `vectors/*.jsonl` | Inputs and expected outputs, one case per line. | Computes every expected output from its input. | | `vectors/*.jsonl` | Inputs and expected outputs, one case per line. | Computes every expected output from its input. |
| `fixtures/packages/` | Valid packages: a converted chemistry paper with corrections, receipts, a multi-level-header table and a dataset; its checks-only export; an open-access paper with two witnesses; a Typst draft; a tiny document, as a folder and as the zip of [12.2](12-package.md#122-paths). | Accepts each, and recomputes every value it contains. | | `fixtures/packages/` | Valid packages: a converted chemistry paper with corrections, receipts, a multi-level-header table and a dataset; its checks-only export; an open-access paper with two witnesses; a Typst draft; a tiny document, as a folder and as the zip of [12.2](12-package.md#122-paths). | Accepts each, and recomputes every value it contains. |
<<<<<<< HEAD | `fixtures/records/` | A valid record of every other format that is a JSON file of its own, and a second layer, of traced citations ([18.5](18-layers.md#185-traced-citations)), whose traces resolve as `vectors/traces.jsonl` says; the stage requirements and stamp rules are in `registry/`, and a bundle is built by `paperspec load --bundle`. | Accepts each. |
| `fixtures/records/` | A valid record of every other format that is a JSON file of its own; the stage requirements and stamp rules are in `registry/`, and a bundle is built by `paperspec load --bundle`. | Accepts each. |
||||||| parent of 4fa326d (First implementation (#1))
=======
| `fixtures/records/` | A valid record of every other format that is a JSON file of its own; the stage requirements and stamp rules are in `registry/`, and a bundle is built by `tools/load.py --bundle`. | Accepts each. |
>>>>>>> 4fa326d (First implementation (#1))
| `fixtures/invalid/` | Invalid packages, each with `expect.json` naming the rule it breaks. | Refuses each, and SHOULD name that rule. | | `fixtures/invalid/` | Invalid packages, each with `expect.json` naming the rule it breaks. | Refuses each, and SHOULD name that rule. |
| `fixtures/keys/` | The test keys that signed the fixtures and the team secret of the checks-only export. | Never uses them for anything else. | | `fixtures/keys/` | The test keys that signed the fixtures and the team secret of the checks-only export. | Never uses them for anything else. |
@@ -27,7 +22,7 @@ A validator checks, in order, and reports each failure as `path:line: rule: mess
3. The tree ([3](03-nodes.md)), runs and references between nodes, the counts, and the keys ([2.4](02-model.md#24-work-and-document-keys)). 3. The tree ([3](03-nodes.md)), runs and references between nodes, the counts, and the keys ([2.4](02-model.md#24-work-and-document-keys)).
4. The policy and hashing scheme ([12.5](12-package.md#125-text-policies)). 4. The policy and hashing scheme ([12.5](12-package.md#125-text-policies)).
5. The five values of every node, the root and `meta_hash`, when the text is present and the key, if any, is known. 5. The five values of every node, the root and `meta_hash`, when the text is present and the key, if any, is known.
6. The ID table, the corrections and the history root, the witnesses, pages, locations and references, and every file's order. 6. The ID table, the corrections and the history root, the witnesses, pages, locations and references, the formulas derived files name, and every file's order.
7. Receipts: IDs, roots, documents, node order, signatures against the package's roster, spot-check keys, seeds and samples, and the fidelity distribution. 7. Receipts: IDs, roots, documents, node order, signatures against the package's roster, spot-check keys, seeds and samples, and the fidelity distribution.
## 22.3 Schemas ## 22.3 Schemas
@@ -44,7 +39,8 @@ Rule IDs are stable: a rule's meaning never changes, and a retired rule's ID is
| Prefix | Covers | | Prefix | Covers |
| --- | --- | | --- | --- |
| `package.` | The manifest, paths, files and schemas. | | `package.` | The manifest, paths, files and schemas. |
| `doc.tree.`, `doc.node.`, `doc.run.`, `doc.table.` | The tree, node content and runs. | | `doc.tree.`, `doc.node.`, `doc.run.`, `doc.table.`, `doc.math.` | The tree, node content and runs. |
| `doc.derived.` | Derived files ([12.7](12-package.md#127-derived-files)). |
| `doc.hash.` | The five values, the root and the metadata hash. | | `doc.hash.` | The five values, the root and the metadata hash. |
| `doc.id.`, `doc.order.` | The ID table and file order. | | `doc.id.`, `doc.order.` | The ID table and file order. |
| `doc.policy.` | Text policies and keyed hashing. | | `doc.policy.` | Text policies and keyed hashing. |
@@ -53,6 +49,10 @@ Rule IDs are stable: a rule's meaning never changes, and a retired rule's ID is
| `doc.receipt.`, `doc.qa.`, `doc.fidelity.` | Receipts, spot-checks and levels. | | `doc.receipt.`, `doc.qa.`, `doc.fidelity.` | Receipts, spot-checks and levels. |
| `doc.witness.`, `doc.page.`, `doc.location.`, `doc.reference.` | Witnesses and geometry. | | `doc.witness.`, `doc.page.`, `doc.location.`, `doc.reference.` | Witnesses and geometry. |
| `grade.` | Grading. | | `grade.` | Grading. |
| `layer.` | Annotations that refer to documents: `layer.trace.source` and `layer.trace.target` ([18.5](18-layers.md#185-traced-citations)). |
A package is validated by [22.2](#222-validating-a-package), but a layer is not part of one.
The `layer.` rules are reported for an annotation, in the form `annotation: rule: message`, by a validator that has the annotation and the states of the documents it names; the reference implementation does so for traces in `paperspec.traces.problems`.
## 22.5 Stability ## 22.5 Stability
spec/23-log.md
+1
View File
@@ -35,6 +35,7 @@ Q2xlYXIgdGV4dCBpcyBub3QgYSByb290IGhhc2guLi4uLi4uLi4=
The origin names the log, and a team's logs are named `scient.ing/log/<store>` or under a domain the team controls. The origin names the log, and a team's logs are named `scient.ing/log/<store>` or under a domain the team controls.
A signature line is an em dash (U+2014), a space, the signer's roster key, a space, and Base64 of the signer's four-byte key hash followed by the signature. A signature line is an em dash (U+2014), a space, the signer's roster key, a space, and Base64 of the signer's four-byte key hash followed by the signature.
The key hash is the first four bytes of SHA-256 of the signer's name, a newline, the byte 0x01 and the Ed25519 public key; the signature is Ed25519 over the note's first three lines, each ending in a newline. The key hash is the first four bytes of SHA-256 of the signer's name, a newline, the byte 0x01 and the Ed25519 public key; the signature is Ed25519 over the note's first three lines, each ending in a newline.
Every line of a note, the last signature line included, ends in one U+000A LINE FEED; a note that ends a line with anything else, such as U+000B, U+0085 or U+2028, which some libraries also take for line breaks, or that lacks its final U+000A, is malformed.
A note parsed into JSON has `format`, `origin`, `size`, `root` as hex, `signatures` as `{name, key_hash, value}` and the `note` text itself; the schema is `checkpoint.schema.json`. A note parsed into JSON has `format`, `origin`, `size`, `root` as hex, `signatures` as `{name, key_hash, value}` and the `note` text itself; the schema is `checkpoint.schema.json`.
`checkpoint.jsonl` gives a checkpoint over the example paper's receipts and its verification. `checkpoint.jsonl` gives a checkpoint over the example paper's receipts and its verification.
sql/README.md
-31
View File
@@ -31,7 +31,6 @@ dolt sql < sql/doc.sql && dolt sql < sql/store.sql
The PostgreSQL database `lab` and the Dolt repository must already exist. The PostgreSQL database `lab` and the Dolt repository must already exist.
The `sqlite3` line runs as shown; the `psql` and `dolt` lines follow the portability rules of [chapter 13](../spec/13-database.md#131-portability) but have not been run on those engines in this repository. The `sqlite3` line runs as shown; the `psql` and `dolt` lines follow the portability rules of [chapter 13](../spec/13-database.md#131-portability) but have not been run on those engines in this repository.
<<<<<<< HEAD
To load packages into SQLite without pixi, install the package with its `tools` extra on Python 3.14, then run `paperspec load --sqlite lab.db PACKAGE…`. To load packages into SQLite without pixi, install the package with its `tools` extra on Python 3.14, then run `paperspec load --sqlite lab.db PACKAGE…`.
The file [docs/examples.md](../docs/examples.md#load-packages-into-sqlite) has queries to try. The file [docs/examples.md](../docs/examples.md#load-packages-into-sqlite) has queries to try.
The loader copies files that have no table (assets, dataset files, derived files) by SHA-256 into `lab.db.assets/`. The loader copies files that have no table (assets, dataset files, derived files) by SHA-256 into `lab.db.assets/`.
@@ -60,36 +59,6 @@ On `fixtures/packages/hale2026harmonic-damb` it returns one row, `hca6dc5`: the
- Round trip: each row keeps every property of its record, so unloading restores the same bytes and the same manifest hash. - Round trip: each row keeps every property of its record, so unloading restores the same bytes and the same manifest hash.
Unloading copies the files with no table from the asset store, which must travel with the database. Unloading copies the files with no table from the asset store, which must travel with the database.
`paperspec check` tests this for every fixture package. `paperspec check` tests this for every fixture package.
||||||| parent of 4fa326d (First implementation (#1))
=======
To load packages into SQLite without pixi, run `python3 tools/load.py --sqlite lab.db PACKAGE…` with Python 3.14 and two packages, `jsonschema` and `cryptography` installed.
The file [docs/examples.md](../docs/examples.md#load-packages-into-sqlite) has queries to try.
The loader copies files that have no table (assets, dataset files, derived files) by SHA-256 into `lab.db.assets/`.
## A first query
The query below finds every sentence that cites a given reference, which is the citation's context.
It returns sentence IDs, and `nodes.content` holds each sentence's text.
```sql
SELECT c.sentence
FROM citations c
JOIN refs r ON r.doc_key = c.doc_key AND r.node = c.ref_node
WHERE c.doc_key = 'hale2026harmonic-damb' AND r.label = '2';
```
On `fixtures/packages/hale2026harmonic-damb` it returns one row, `hca6dc5`: the sentence ending "served as the first test case, with a bending band near 1595 cm⁻¹."
## Conventions
- Foreign keys: none.
Validators check references, so the same file runs everywhere.
- Types: enumerations are `VARCHAR` with a `CHECK`, JSON values are canonical JSON in `TEXT`, and booleans are `INTEGER` 0 or 1.
- Names: no table, column or index name is reserved in SQLite, PostgreSQL or MySQL (Dolt is MySQL-compatible).
The formats' `key`, `index`, `at`, `by`, `before` and `after` become `doc_key`, `idx`, `made_at`, `signer`, `before_hash` and `after_hash`.
- Round trip: each row keeps every property of its record, so unloading restores the same bytes and the same manifest hash.
Unloading copies the files with no table from the asset store, which must travel with the database. `tools/check.py` tests this for every fixture package.
>>>>>>> 4fa326d (First implementation (#1))
- Derived tables: `citations`, `receipt_nodes` and `annotation_targets` are built on load for queries and are not part of the package. - Derived tables: `citations`, `receipt_nodes` and `annotation_targets` are built on load for queries and are not part of the package.
## Status ## Status
sql/doc.sql
+1 -1
View File
@@ -128,7 +128,7 @@ CREATE TABLE corrections (
doc_key VARCHAR(128) NOT NULL, doc_key VARCHAR(128) NOT NULL,
idx INTEGER NOT NULL CHECK (idx >= 1), idx INTEGER NOT NULL CHECK (idx >= 1),
id VARCHAR(32) NOT NULL, id VARCHAR(32) NOT NULL,
op VARCHAR(16) NOT NULL CHECK (op IN ('replace', 'insert', 'delete', 'split', 'merge', 'move', 'reclassify', 'relabel', 'set')), op VARCHAR(16) NOT NULL CHECK (op IN ('replace', 'insert', 'delete', 'split', 'merge', 'move', 'reclassify', 'relabel', 'set', 'emend', 'relink', 'resolve')),
target VARCHAR(16), target VARCHAR(16),
evidence TEXT NOT NULL, evidence TEXT NOT NULL,
author TEXT NOT NULL, author TEXT NOT NULL,
sql/store.sql
+2 -1
View File
@@ -68,7 +68,8 @@ CREATE TABLE layers (
CREATE TABLE annotations ( CREATE TABLE annotations (
id VARCHAR(256) NOT NULL, id VARCHAR(256) NOT NULL,
layer VARCHAR(256) NOT NULL, layer VARCHAR(256) NOT NULL,
body_kind VARCHAR(24) NOT NULL CHECK (body_kind IN ('gloss', 'claim-link', 'symbol', 'comment', 'caveat', 'question', 'exercise', 'citation-context', 'prerequisite', 'crosslink', 'translation', 'description', 'generated')), body_kind VARCHAR(24) NOT NULL CHECK (body_kind IN ('gloss', 'claim-link', 'symbol', 'comment', 'caveat', 'question', 'exercise', 'citation-context', 'prerequisite', 'crosslink', 'trace', 'translation', 'description', 'generated')),
category VARCHAR(12) CHECK (category IN ('comment', 'context', 'critique', 'question', 'correction', 'feedback', 'resource')),
status VARCHAR(10) NOT NULL CHECK (status IN ('proposed', 'confirmed', 'rejected', 'withdrawn')), status VARCHAR(10) NOT NULL CHECK (status IN ('proposed', 'confirmed', 'rejected', 'withdrawn')),
receipt VARCHAR(32), receipt VARCHAR(32),
creator VARCHAR(256) NOT NULL, creator VARCHAR(256) NOT NULL,
src/paperspec/align.py
+52
View File
@@ -9,6 +9,7 @@ import unicodedata
from .hashing import prose_length from .hashing import prose_length
MIN_PROSE = 20 MIN_PROSE = 20
MIN_SIMILAR = 0.8
def _numeric_skeleton(strict): def _numeric_skeleton(strict):
@@ -82,6 +83,56 @@ def _lcs(A, B, a0, a1, b0, b1):
return out return out
def lcs_length(a, b):
"""Length of the longest common subsequence of two strings' code points (bit-parallel, Hyyrö 2004)."""
if len(a) < len(b):
a, b = b, a
masks = {}
for i, ch in enumerate(a):
masks[ch] = masks.get(ch, 0) | (1 << i)
full = (1 << len(a)) - 1
v = full
for ch in b:
u = v & masks.get(ch, 0)
v = ((v + u) | (v - u)) & full
return len(a) - v.bit_count()
def similarity(a, b):
"""2L / (|a| + |b|) over code points, L the longest common subsequence; 1 for two empty strings."""
if not a and not b:
return 1.0
return 2 * lcs_length(a, b) / (len(a) + len(b))
def _similar(a, b):
"""Whether two lossy projections are at least MIN_SIMILAR alike; exact, with two cheap upper bounds first."""
total = len(a) + len(b)
if 2 * min(len(a), len(b)) < MIN_SIMILAR * total:
return False
common = sum(min(n, b.count(ch)) for ch, n in {ch: a.count(ch) for ch in set(a)}.items())
if 2 * common < MIN_SIMILAR * total:
return False
return 2 * lcs_length(a, b) >= MIN_SIMILAR * total
def _substitutions(A, B, pairs):
"""Step 3: in each stretch between pairs, an unpaired sentence beside a nearly equal one is paired with it."""
out = []
bounds = [(-1, -1)] + pairs + [(len(A), len(B))]
for (a0, b0), (a1, b1) in zip(bounds, bounds[1:]):
nxt = b0 + 1
for i in range(a0 + 1, a1):
if prose_length(A[i]["lossy"]) < MIN_PROSE:
continue
for j in range(nxt, b1):
if prose_length(B[j]["lossy"]) >= MIN_PROSE and _similar(A[i]["lossy"], B[j]["lossy"]):
out.append((i, j))
nxt = j + 1
break
return out
def align(A, B): def align(A, B):
"""Align reading B to reading A. Returns {"pairs": [...], "suspects": [...]}.""" """Align reading B to reading A. Returns {"pairs": [...], "suspects": [...]}."""
count_a, count_b = {}, {} count_a, count_b = {}, {}
@@ -102,6 +153,7 @@ def align(A, B):
if ai < len(A): if ai < len(A):
pairs.append((ai, bi)) pairs.append((ai, bi))
prev_a, prev_b = ai + 1, bi + 1 prev_a, prev_b = ai + 1, bi + 1
pairs = sorted(pairs + _substitutions(A, B, pairs))
out_pairs, suspects = [], [] out_pairs, suspects = [], []
for ai, bi in pairs: for ai, bi in pairs:
kind = classify(A[ai], B[bi]) kind = classify(A[ai], B[bi])
src/paperspec/cli/build.py
+364 -34
View File
@@ -26,10 +26,10 @@ from paperspec import align as AL # noqa: E402
from paperspec import anchors as AN # noqa: E402 from paperspec import anchors as AN # noqa: E402
from paperspec import corrections as CO # noqa: E402 from paperspec import corrections as CO # noqa: E402
from paperspec import crypto, fingerprint as FP, grade as GR, hashing, jcs, receipts as RC, rights as RI # noqa: E402 from paperspec import crypto, fingerprint as FP, grade as GR, hashing, jcs, receipts as RC, rights as RI # noqa: E402
from paperspec import keys as K, log as LG, mldsa, targets as TG, vrf # noqa: E402 from paperspec import keys as K, log as LG, mldsa, targets as TG, traces as TR, vrf # noqa: E402
from paperspec.model import LEAVES, MATH_KINDS, RUN_KINDS, VERBATIM_KINDS, Doc # noqa: E402 from paperspec.model import LEAVES, MATH_KINDS, RUN_KINDS, VERBATIM_KINDS, Doc # noqa: E402
from paperspec.notation import fold_math # noqa: E402 from paperspec.notation import fold_math # noqa: E402
from paperspec.package import ZIP_EXTENSION, history_items, jsonl, line, witness_id, write, write_zip # noqa: E402 from paperspec.package import ZIP_EXTENSION, history_items, jsonl, line, read_state, witness_id, write, write_zip # noqa: E402
from paperspec.schema import Store # noqa: E402 from paperspec.schema import Store # noqa: E402
from paperspec.text import letter_first, mint_node_id, norm, record_id, sha256_hex # noqa: E402 from paperspec.text import letter_first, mint_node_id, norm, record_id, sha256_hex # noqa: E402
from paperspec.tree import Refusal, _flatten, mint_ids, validate_structure # noqa: E402 from paperspec.tree import Refusal, _flatten, mint_ids, validate_structure # noqa: E402
@@ -448,7 +448,7 @@ def build_document(src, used=None, extra_files=None):
nodes, side = establish(key, spec, names, used) nodes, side = establish(key, spec, names, used)
aliases = [{"id": a, "kind": "alias", "successor": names.ids[t]} aliases = [{"id": a, "kind": "alias", "successor": names.ids[t]}
for a, t in sorted(src.get("legacy_aliases", {}).items())] for a, t in sorted(src.get("legacy_aliases", {}).items())]
state = CO.State(key, nodes, aliases) state = CO.State(key, nodes, aliases, references=reference_records(Doc(key, nodes), side, names))
states = {0: state} states = {0: state}
records = [] records = []
for c in src.get("corrections", []): for c in src.get("corrections", []):
@@ -478,11 +478,12 @@ def build_document(src, used=None, extra_files=None):
names.ids[new_name] = rec["created"][0] names.ids[new_name] = rec["created"][0]
records.append(rec) records.append(rec)
states[state.rev] = state states[state.rev] = state
# A corrected leaf was checked against the page image: it is F1 at most until corroborated again. # A corrected leaf was checked against the page image: it is F1 at most until corroborated again. An emend,
# relink or resolve leaves what the witness read as it was, so it keeps the leaf's level (spec/08 §8.1).
final = state final = state
touched = set() touched = set()
for rec in records: for rec in records:
if any(e["kind"] == "crop" for e in rec["evidence"]): if any(e["kind"] == "crop" for e in rec["evidence"]) and rec["op"] not in CO.KEEPS_LEVEL:
touched |= set(rec["touched"]) | set(rec.get("created", [])) touched |= set(rec["touched"]) | set(rec.get("created", []))
for n in final.nodes: for n in final.nodes:
if n["id"] in touched and n["kind"] in LEAVES: if n["id"] in touched and n["kind"] in LEAVES:
@@ -504,6 +505,22 @@ def _strip_private(spec, names):
return out return out
def reference_records(doc, side, names, resolved=None):
"""The records of references.jsonl: each reference node in document order, with its label, its parsed CSL and
its resolution, which is the source's unless a resolve correction set it (`resolved`, by node)."""
refs = []
for nid in doc.preorder():
s = side.get(nid, {})
if doc.nodes[nid]["kind"] == "reference":
r = {"node": nid, "label": doc.nodes[nid].get("label")}
if "_csl" in s:
r["csl"] = s["_csl"]
res = s.get("_resolution", {"type": "none"})
r["resolution"] = (resolved or {}).get(nid) or {k: names.resolve(v) for k, v in res.items()}
refs.append({k: v for k, v in r.items() if v is not None})
return refs
def package_files(src, key, names, side, final, records, receipts=None, roster_rec=None, witnesses=None, def package_files(src, key, names, side, final, records, receipts=None, roster_rec=None, witnesses=None,
pages=None, boxes=None, extra=None, absent=()): pages=None, boxes=None, extra=None, absent=()):
doc = final.doc() doc = final.doc()
@@ -523,16 +540,7 @@ def package_files(src, key, names, side, final, records, receipts=None, roster_r
locs.sort(key=lambda l: (pos[l["node"]], l["page"], l["page_fp"])) locs.sort(key=lambda l: (pos[l["node"]], l["page"], l["page_fp"]))
if locs: if locs:
files["locations.jsonl"] = jsonl(locs) files["locations.jsonl"] = jsonl(locs)
refs = [] refs = reference_records(doc, side, names, {r["node"]: r["resolution"] for r in final.references})
for nid in doc.preorder():
s = side.get(nid, {})
if doc.nodes[nid]["kind"] == "reference":
r = {"node": nid, "label": doc.nodes[nid].get("label")}
if "_csl" in s:
r["csl"] = s["_csl"]
res = s.get("_resolution", {"type": "none"})
r["resolution"] = {k: names.resolve(v) for k, v in res.items()}
refs.append({k: v for k, v in r.items() if v is not None})
if refs: if refs:
files["references.jsonl"] = jsonl(refs) files["references.jsonl"] = jsonl(refs)
if records: if records:
@@ -551,7 +559,7 @@ def package_files(src, key, names, side, final, records, receipts=None, roster_r
"tools": [{"name": "paperspec tools/build.py", "version": VERSION}], "tools": [{"name": "paperspec tools/build.py", "version": VERSION}],
"stages": [{"stage": "hand", "processor": "local", "tool": "paperspec tools/build.py"}], "stages": [{"stage": "hand", "processor": "local", "tool": "paperspec tools/build.py"}],
"note": "Built by hand for the conformance suite; every reading is synthetic."}) + "\n").encode() "note": "Built by hand for the conformance suite; every reading is synthetic."}) + "\n").encode()
levels = RC.effective_levels(nodes, vals, receipts or [], ROSTER_BY_ID if roster_rec else {}) levels = RC.effective_levels(nodes, vals, receipts or [], ROSTER_BY_ID if roster_rec else {}, records or [])
d = dict(src["document"]) d = dict(src["document"])
d = {"format": "scienting-doc/1", **d} d = {"format": "scienting-doc/1", **d}
d.update({"hashing": {"scheme": "plain", "algorithm": "sha-256"}, "root": vals[doc.root]["node_hash"], "rev": final.rev, d.update({"hashing": {"scheme": "plain", "algorithm": "sha-256"}, "root": vals[doc.root]["node_hash"], "rev": final.rev,
@@ -625,11 +633,29 @@ def build_hale(out):
mathml = ('<math display="block"><msub><mi>E</mi><mtext>bond</mtext></msub><mo>=</mo><mfrac><mn>1</mn><mn>2' mathml = ('<math display="block"><msub><mi>E</mi><mtext>bond</mtext></msub><mo>=</mo><mfrac><mn>1</mn><mn>2'
'</mn></mfrac><msub><mi>K</mi><mi>r</mi></msub><msup><mrow><mo>(</mo><mi>r</mi><mo>−</mo><msub><mi>r' '</mn></mfrac><msub><mi>K</mi><mi>r</mi></msub><msup><mrow><mo>(</mo><mi>r</mi><mo>−</mo><msub><mi>r'
'</mi><mtext>eq</mtext></msub><mo>)</mo></mrow><mn>2</mn></msup></math>') '</mi><mtext>eq</mtext></msub><mo>)</mo></mrow><mn>2</mn></msup></math>')
# The scienting-math/1 reading of three formulas (spec/09 §9.4): what papermath 0.1.0 prints for each source,
# copied here because the oracle does not implement the form, and checked by the validator only for what it
# can check without it (spec/12 §12.7). The run `_2` of "H₂O" has no base, so its tree is one opaque node.
math_tool = {"name": "papermath", "version": "0.1.0"}
s2_runs, s3_runs = final_doc.nodes[names.ids["@s2"]]["content"], final_doc.nodes[names.ids["@s3"]]["content"]
math = [
{"node": eq, "run": None, "tree": [
"rel", ["sym", "E_{\\text{bond}}"], "=",
["mul", ["frac", ["num", "1"], ["num", "2"]], ["sym", "K_{r}"],
["pow", ["add", ["sym", "r"], ["neg", ["sym", "r_{\\text{eq}}"]]], ["num", "2"]]]],
"symbols": ["E_{\\text{bond}}", "K_{r}", "r", "r_{\\text{eq}}"], "complete": True, "issues": []},
{"node": names.ids["@s2"], "run": next(i for i, r in enumerate(s2_runs) if r.get("source") == "K_r"),
"tree": ["sym", "K_{r}"], "symbols": ["K_{r}"], "complete": True, "issues": []},
{"node": names.ids["@s3"], "run": next(i for i, r in enumerate(s3_runs) if r.get("source") == "_2"),
"tree": ["opaque", "_{2}"], "symbols": [], "complete": False,
"issues": [{"code": "opaque", "message": "a part after the formula's end", "at": "_{2}"}]},
]
extra = { extra = {
"assets/%s.png" % image_sha: image, "assets/%s.png" % image_sha: image,
"datasets/%s/datapackage.json" % ds_id: (jcs.dumps(datapackage) + "\n").encode(), "datasets/%s/datapackage.json" % ds_id: (jcs.dumps(datapackage) + "\n").encode(),
"datasets/%s/data.csv" % ds_id: ("\n".join(csv) + "\n").encode(), "datasets/%s/data.csv" % ds_id: ("\n".join(csv) + "\n").encode(),
"datasets/%s/anchors.jsonl" % ds_id: jsonl(anchors), "datasets/%s/anchors.jsonl" % ds_id: jsonl(anchors),
"derived/math.jsonl": jsonl([dict(m, tool=math_tool, form="scienting-math/1") for m in math]),
"derived/mathml.jsonl": jsonl([{"node": eq, "run": None, "tool": {"name": "hand", "version": "0"}, "derived/mathml.jsonl": jsonl([{"node": eq, "run": None, "tool": {"name": "hand", "version": "0"},
"mathml": mathml}]), "mathml": mathml}]),
} }
@@ -659,7 +685,7 @@ def build_checks_only(out, hale):
n["attrs"].pop("figure_text", None) n["attrs"].pop("figure_text", None)
nodes.append(node_record(n, keyed)) nodes.append(node_record(n, keyed))
f = dict(hale["files"]) f = dict(hale["files"])
for p in ("corrections.jsonl", "receipts.jsonl", "roster.json", "derived/mathml.jsonl"): for p in ("corrections.jsonl", "receipts.jsonl", "roster.json", "derived/math.jsonl", "derived/mathml.jsonl"):
f.pop(p, None) f.pop(p, None)
for p in [p for p in f if p.startswith("datasets/")]: for p in [p for p in f if p.startswith("datasets/")]:
f.pop(p) f.pop(p)
@@ -786,6 +812,9 @@ def projection_case(cid, runs, note=None):
plain = hashing.plain(doc, s) plain = hashing.plain(doc, s)
c = {"id": cid, "runs": runs, "plain": plain, "normalized": norm(plain), "strict": strict, "lossy": lossy, c = {"id": cid, "runs": runs, "plain": plain, "normalized": norm(plain), "strict": strict, "lossy": lossy,
"complex": cx, "text_hash": H(norm(plain)), "match_hash": H(strict), "align_key": H(lossy)} "complex": cx, "text_hash": H(norm(plain)), "match_hash": H(strict), "align_key": H(lossy)}
printed = hashing.printed(doc, s)
if printed != plain: # runs an emend touched: the projection as printed, which the strict one is made from
c["printed"] = printed
if note: if note:
c["note"] = note c["note"] = note
return c return c
@@ -878,6 +907,11 @@ def vectors(out, hale, tiny):
projection_case("hyphen-before-digit", [T("pages 10-20 and CT\u2013CT with 6-31G*")]), projection_case("hyphen-before-digit", [T("pages 10-20 and CT\u2013CT with 6-31G*")]),
projection_case("en-dash-before-digit", [T("pages 10\u201320 and CT-CT with 6\u201331G*")]), projection_case("en-dash-before-digit", [T("pages 10\u201320 and CT-CT with 6\u201331G*")]),
projection_case("astral", [T("Let "), M("\U0001d465"), T(" vary.")]), projection_case("astral", [T("Let "), M("\U0001d465"), T(" vary.")]),
# An emend keeps what was printed (spec/03 §3.5): the plain projection reads the emended text, the strict
# and lossy ones the printed text, so the emended sentence still agrees with a witness that prints the slip.
projection_case("emended", [T("The "), dict(T("bond", style=["em"]), printed="bnod"), T(" was rigid.")],
"The authors printed 'bnod'; an emend reads 'bond'."),
projection_case("as-printed", [T("The "), T("bnod", style=["em"]), T(" was rigid.")]),
] ]
groups = [ groups = [
{"id": "h2o", "same": "match_hash", {"id": "h2o", "same": "match_hash",
@@ -893,6 +927,8 @@ def vectors(out, hale, tiny):
{"id": "cite-structure", "differ": "match_hash", "cases": ["cite-mark", "cite-as-text"]}, {"id": "cite-structure", "differ": "match_hash", "cases": ["cite-mark", "cite-as-text"]},
{"id": "dashes-align", "same": "align_key", "cases": ["hyphen-before-digit", "en-dash-before-digit"]}, {"id": "dashes-align", "same": "align_key", "cases": ["hyphen-before-digit", "en-dash-before-digit"]},
{"id": "dashes-text", "differ": "text_hash", "cases": ["hyphen-before-digit", "en-dash-before-digit"]}, {"id": "dashes-text", "differ": "text_hash", "cases": ["hyphen-before-digit", "en-dash-before-digit"]},
{"id": "emended-match", "same": "match_hash", "cases": ["emended", "as-printed"]},
{"id": "emended-text", "differ": "text_hash", "cases": ["emended", "as-printed"]},
] ]
byid = {c["id"]: c for c in pc} byid = {c["id"]: c for c in pc}
for g in groups: for g in groups:
@@ -917,8 +953,23 @@ def vectors(out, hale, tiny):
tdoc = Doc(tiny["key"], tnodes) tdoc = Doc(tiny["key"], tnodes)
tv = hashing.compute(tdoc) tv = hashing.compute(tdoc)
own = {nid: hashing.own_content(tdoc, nid, lambda x: tv[x]["text_hash"]) for nid in tdoc.order} own = {nid: hashing.own_content(tdoc, nid, lambda x: tv[x]["text_hash"]) for nid in tdoc.order}
# A sentence whose display equation and math run carry the form their canonical TeX is spelled in (spec/03 §3.7):
# `form` is in no own content and no projection, so the values are those of the same nodes without it.
form = "scienting-math/1"
fnodes = [{"id": "aaaaaaa", "kind": "document", "parent": None, "ord": 0},
{"id": "bbbbbbb", "kind": "section", "parent": "aaaaaaa", "ord": 0, "attrs": {"level": 1}},
{"id": "ccccccc", "kind": "paragraph", "parent": "bbbbbbb", "ord": 0},
{"id": "ddddddd", "kind": "sentence", "parent": "ccccccc", "ord": 0, "content": [
T("The energy is "), {"kind": "equation", "node": "eeeeeee"}, T(" where "),
dict(M("K_{r}"), form=form), T(" is the force constant.")]},
{"id": "eeeeeee", "kind": "equation", "parent": "ddddddd", "ord": 0,
"content": {"notation": "tex", "source": "E=\\frac{1}{2}K_{r}x^{2}", "form": form}}]
fdoc = Doc("vectors", fnodes)
fv = hashing.compute(fdoc)
fown = {nid: hashing.own_content(fdoc, nid, lambda x: fv[x]["text_hash"]) for nid in fdoc.order}
wv(out, "hashes.jsonl", [{"id": "tiny2026example", "doc": tiny["key"], "nodes": tnodes, wv(out, "hashes.jsonl", [{"id": "tiny2026example", "doc": tiny["key"], "nodes": tnodes,
"own_content": own, "values": tv}]) "own_content": own, "values": tv},
{"id": "math-form", "doc": "vectors", "nodes": fnodes, "own_content": fown, "values": fv}])
# Addresses. The example keys are real work keys: Cornell et al. (1995) by its DOI, a fictional preprint. # Addresses. The example keys are real work keys: Cornell et al. (1995) by its DOI, a fictional preprint.
cornell = "cornell1995second-" + K.suffix("doi:10.1021/ja00124a002") cornell = "cornell1995second-" + K.suffix("doi:10.1021/ja00124a002")
dong = "dong2026machine-" + K.suffix("doi:10.5555/example.dong.2026") dong = "dong2026machine-" + K.suffix("doi:10.5555/example.dong.2026")
@@ -1126,7 +1177,22 @@ def vectors(out, hale, tiny):
item("b5", [T("where "), M("\\frac{c}{e}"), T(" is the bond length.")], "q2"), item("b5", [T("where "), M("\\frac{c}{e}"), T(" is the bond length.")], "q2"),
item("b6", [T("A sentence only the second reading has, also long.")], "q2"), item("b6", [T("A sentence only the second reading has, also long.")], "q2"),
item("b7", [T("The last sentence of the section is unique and long enough.")], "q2")] item("b7", [T("The last sentence of the section is unique and long enough.")], "q2")]
wv(out, "alignment.jsonl", [{"id": "two-readings", "a": A, "b": B, "expect": AL.align(A, B)}]) # Substitutions (step 3): a misread number and a misread word pair with their other readings; a short
# sentence and two sentences too unlike to be one stay dropped and added.
SA = [item("c1", [T("The force constants were fitted to ab initio scans of every bond.")], "p1"),
item("c2", [T("The bond constant was 310.0 kcal/mol for every aliphatic carbon.")], "p1"),
item("c3", [T("This prototypal set was later extended to nucleic acids.")], "p1"),
item("c4", [T("See below.")], "p1"),
item("c5", [T("Torsions were refit against a second, larger set of conformers.")], "p1"),
item("c6", [T("The charges came from a two-stage restrained fit to the potential.")], "p2")]
SB = [item("d1", [T("The force constants were fitted to ab initio scans of every bond.")], "q1"),
item("d2", [T("The bond constant was 3l0.0 kcal/mol for every aliphatic carbon.")], "q1"),
item("d3", [T("This prototypical set was later extended to nucleic acids.")], "q1"),
item("d4", [T("See bclow.")], "q1"),
item("d5", [T("Only the second reading prints this line about water models.")], "q1"),
item("d6", [T("The charges came from a two-stage restrained fit to the potential.")], "q2")]
wv(out, "alignment.jsonl", [{"id": "two-readings", "a": A, "b": B, "expect": AL.align(A, B)},
{"id": "substitutions", "a": SA, "b": SB, "expect": AL.align(SA, SB)}])
# Corrections, op by op, on the tiny document. # Corrections, op by op, on the tiny document.
wv(out, "corrections.jsonl", corrections_vectors()) wv(out, "corrections.jsonl", corrections_vectors())
# Anchor resolution against the hale package. # Anchor resolution against the hale package.
@@ -1182,23 +1248,73 @@ def corrections_vectors():
# A model proposed this one, so its author names the model, by weights and prompt, and the run. # A model proposed this one, so its author names the model, by weights and prompt, and the run.
("replace-by-model", {"op": "replace", "target": nm["@c"], "find": "second", "replace": "later", "evidence": person, ("replace-by-model", {"op": "replace", "target": nm["@c"], "find": "second", "replace": "later", "evidence": person,
"author": {"model": MODEL, "run": "illustrative-run-7"}}), "author": {"model": MODEL, "run": "illustrative-run-7"}}),
# Emend (spec/11 §11.1): the authors' own slips. The found text becomes a run that keeps what was printed, so
# the match_hash and align_key stay those of the printed text, and the expectation adds the leaf as printed.
("emend", {"op": "emend", "target": nm["@b"], "find": "the note", "replace": "the notes", "kind": "grammar",
"note": "The sentence points at two notes below the table.", "evidence": person}),
("emend-xref", {"op": "emend", "target": nm["@b"], "find": "Table 1", "replace": "Tab. 1", "kind": "wording",
"evidence": person}),
("emend-digit", {"op": "emend", "target": nm["@a"], "find": "42.0", "replace": "42,0", "kind": "punctuation",
"evidence": crop}),
("emend-across-runs", {"op": "emend", "target": nm["@b"], "find": "1 and", "replace": "1, and",
"kind": "punctuation", "evidence": person}),
("emend-not-runs", {"op": "emend", "target": nm["@t"], "find": "x", "replace": "X", "kind": "typo",
"evidence": person}),
# Relink (spec/11 §11.1): the cite run becomes text with the same text, which joins the text before it.
("relink-unlink", {"op": "relink", "target": nm["@b"], "run": 3, "refs": None, "evidence": person}),
("relink-xref", {"op": "relink", "target": nm["@b"], "run": 1, "refs": [nm["@r"]], "evidence": person}),
("relink-not-reference", {"op": "relink", "target": nm["@b"], "run": 3, "refs": [nm["@a"]], "evidence": person}),
# Resolve (spec/11 §11.1): the reference's record in references.jsonl, which the state before and after holds.
("resolve-doi", {"op": "resolve", "target": nm["@r"], "resolution": {"type": "doi", "doi": "10.5555/example.ref.2020"},
"evidence": [{"kind": "person", "by": "ana", "note": "Matched by title and year."}]}),
("resolve-not-reference", {"op": "resolve", "target": nm["@a"], "resolution": {"type": "none"},
"evidence": person}),
] ]
# Cases on other states: a math run, a citation read as a superscript, and a run already emended.
st_math, _ = CO.apply(st0, dict(base, **dict(cases)["replace-runs"]), store=SCHEMAS)
st_emended, _ = CO.apply(st0, dict(base, **dict(cases)["emend"]), store=SCHEMAS)
sup = [dict(n, content=[T("The value is 42.0 at 300 K."), T("1", style=["sup"])]) if n["id"] == nm["@a"] else n
for n in st0.nodes]
others = {
"emend-math": (CO.State(st0.key, st_math.nodes), {"op": "emend", "target": nm["@a"], "find": "mathrm",
"replace": "mathup", "kind": "typo", "evidence": person}),
"relink-superscript": (CO.State(st0.key, sup), {"op": "relink", "target": nm["@a"], "run": 1, "refs": [nm["@r"]],
"evidence": person}),
"replace-emended": (CO.State(st0.key, st_emended.nodes), {"op": "replace", "target": nm["@b"], "find": "notes",
"replace": "note", "evidence": crop}),
}
out = [] out = []
vals0 = hashing.compute(st0.doc()) for cid, c in cases + [(cid, c) for cid, (_, c) in others.items()]:
before = [dict(n, **vals0[n["id"]]) for n in st0.nodes] st = others[cid][0] if cid in others else st0
for cid, c in cases: vals0 = hashing.compute(st.doc())
c = dict(base, **c) c = dict(base, **c)
case = {"id": cid, "doc": st0.key, "before": {"nodes": before, "ids": [], "rev": 0}, "correction": c} before = {"nodes": [dict(n, **vals0[n["id"]]) for n in st.nodes], "ids": [], "rev": 0}
if c["op"] == "resolve":
st = CO.State(st.key, st.nodes, references=st0.references)
before["references"] = st.references
case = {"id": cid, "doc": st0.key, "before": before, "correction": c}
try: try:
st1, rec = CO.apply(st0, c, store=SCHEMAS) st1, rec = CO.apply(st, c, store=SCHEMAS)
v1 = hashing.compute(st1.doc()) case["expect"] = correction_expect(st1, rec, "references" in before)
case["expect"] = {"record": rec, "nodes": [dict(n, **v1[n["id"]]) for n in st1.nodes], "ids": st1.ids}
except Refusal as e: except Refusal as e:
case["expect"] = {"refusal": e.rule} case["expect"] = {"refusal": e.rule}
out.append(case) out.append(case)
return out return out
def correction_expect(st1, rec, references=False):
"""What a correction vector expects: the record, the nodes with their values and the ID table; the reference
records too when the case gives them; and for an emend, its leaf as printed (spec/03 §3.6)."""
d1 = st1.doc()
v1 = hashing.compute(d1)
out = {"record": rec, "nodes": [dict(n, **v1[n["id"]]) for n in st1.nodes], "ids": st1.ids}
if references:
out["references"] = st1.references
if rec["op"] == "emend":
out["printed"] = {nid: hashing.printed(d1, nid) for nid in rec["touched"]}
return out
def resolve_vectors(hale): def resolve_vectors(hale):
src, names, states, final = hale["src"], hale["names"], hale["states"], hale["final"] src, names, states, final = hale["src"], hale["names"], hale["states"], hale["final"]
s0 = states[0] s0 = states[0]
@@ -1276,6 +1392,10 @@ def ann_hash(a):
TAG = "tag:scient.ing,2026:" TAG = "tag:scient.ing,2026:"
# The claim amber/k-bond of paperclaim's example store, by its tag; paperclaim prints its pin as claim:272XTX7C@02f47b.
CLAIM_K_BOND = "claim:272XTX7C"
# The layer of traced citations on the draft (spec/18 §18.5), which the project holds beside the example layer.
TRACE_LAYER = TAG + "layer:smithlab/tr6c3q8m"
# A model's provenance: enough to name the weights and the instructions it ran with (all illustrative). # A model's provenance: enough to name the weights and the instructions it ran with (all illustrative).
MODEL = {"name": "example-model", "provider": "Example Models (illustrative)", "version": "2026-09", MODEL = {"name": "example-model", "provider": "Example Models (illustrative)", "version": "2026-09",
"weights": sha256_hex("illustrative weights"), "prompt": sha256_hex("illustrative review prompt"), "weights": sha256_hex("illustrative weights"), "prompt": sha256_hex("illustrative review prompt"),
@@ -1394,11 +1514,15 @@ def records(out, hale):
ann({"type": "TextualBody", "kind": "gloss", "audience": "novice", "format": "text/markdown", "language": "en", ann({"type": "TextualBody", "kind": "gloss", "audience": "novice", "format": "text/markdown", "language": "en",
"value": "A *harmonic* term treats a bond like a spring: the energy grows with the square of the stretch."}, "value": "A *harmonic* term treats a bond like a spring: the energy grows with the square of the stretch."},
target(anchor("@s2", "bond force constant")), "describing", "confirmed"), target(anchor("@s2", "bond force constant")), "describing", "confirmed"),
ann({"type": "SpecificResource", "kind": "claim-link", "source": "lode:amber/k-bond", "pin": "illustrative-pin", # paperclaim's references (spec/18 §18.4): the claim amber/k-bond by its tag, pinned to its statement hash.
ann({"type": "SpecificResource", "kind": "claim-link", "source": CLAIM_K_BOND, "pin": CLAIM_K_BOND + "@02f47b",
"role": "states"}, target(anchor("@s2")), "linking", "confirmed"), "role": "states"}, target(anchor("@s2")), "linking", "confirmed"),
ann({"type": "TextualBody", "kind": "comment", "value": "The step size matters more than the text admits."}, # Comments, caveats and questions carry the category a producer that supports categories always sets
# (spec/18 §18.4).
ann({"type": "TextualBody", "kind": "comment", "category": "critique",
"value": "The step size matters more than the text admits."},
target(anchor("@s6a", "steps of 0.01")), "commenting"), target(anchor("@s6a", "steps of 0.01")), "commenting"),
ann({"type": "TextualBody", "kind": "question", "for": "group", ann({"type": "TextualBody", "kind": "question", "category": "question", "for": "group",
"value": "Why fit to a scan first and to frequencies second?"}, "value": "Why fit to a scan first and to frequencies second?"},
target(AN.make_anchor(doc, vals, I["@sec2"], rev, store=STORE)), "questioning"), target(AN.make_anchor(doc, vals, I["@sec2"], rev, store=STORE)), "questioning"),
ann({"type": "TextualBody", "kind": "citation-context", "reference": I["@r2"], "verdict": "cannot-tell", ann({"type": "TextualBody", "kind": "citation-context", "reference": I["@r2"], "verdict": "cannot-tell",
@@ -1416,8 +1540,16 @@ def records(out, hale):
"value": "Energy along the C–C bond stretch (panel a) and the fit residual (panel b)."}, "value": "Energy along the C–C bond stretch (panel a) and the fit residual (panel b)."},
target(AN.make_anchor(doc, vals, I["@fig1"], rev, store=STORE)), "describing", "confirmed"), target(AN.make_anchor(doc, vals, I["@fig1"], rev, store=STORE)), "describing", "confirmed"),
] ]
anns.append(ann({"type": "TextualBody", "kind": "comment", "value": "Agreed; see the residual panel."}, anns.append(ann({"type": "TextualBody", "kind": "comment", "category": "comment",
"value": "Agreed; see the residual panel."},
{"type": "SpecificResource", "source": anns[2]["id"]}, "replying", created="2026-10-11T19:05:00Z")) {"type": "SpecificResource", "source": anns[2]["id"]}, "replying", created="2026-10-11T19:05:00Z"))
anns.append(ann({"type": "TextualBody", "kind": "caveat", "category": "context",
"value": "The band position is cited, not measured here; reference 2 is not in the library."},
target(anchor("@s3", "1595")), "commenting", created="2026-10-11T19:10:00Z"))
# A symbol: the span $K_r$ is the symbol the claim defines, named by its scienting-math/1 key (spec/09 §9.4).
anns.append(ann({"type": "SpecificResource", "kind": "symbol", "source": CLAIM_K_BOND, "pin": CLAIM_K_BOND + "@02f47b",
"key": "K_{r}", "symbol": "K_r"},
target(anchor("@s2", "$K_r$")), "identifying", created="2026-10-11T19:15:00Z"))
confirmed = [a for a in anns if a["status"] == "confirmed"] confirmed = [a for a in anns if a["status"] == "confirmed"]
lr = RC.seal({"format": "scienting-receipt/1", "doc": HALE, "rev": rev, "level": "F3", lr = RC.seal({"format": "scienting-receipt/1", "doc": HALE, "rev": rev, "level": "F3",
"scope": "targeted:layer", "nodes": [], "scope": "targeted:layer", "nodes": [],
@@ -1469,9 +1601,9 @@ def records(out, hale):
"paths": [{"id": "onboarding", "title": "First week", "steps": [ "paths": [{"id": "onboarding", "title": "First week", "steps": [
{"type": "document", "ref": QHALE, "note": "Read the introduction."}, {"type": "document", "ref": QHALE, "note": "Read the introduction."},
{"type": "node", "ref": QHALE + "#" + I["@tab1"], "note": "Then the table."}, {"type": "node", "ref": QHALE + "#" + I["@tab1"], "note": "Then the table."},
{"type": "claim", "ref": "lode:amber/k-bond"}]}], {"type": "claim", "ref": CLAIM_K_BOND, "note": "The bond force constant, amber/k-bond."}]}],
"layers": [layer_id], "datasets": ["ds-hale-bond-params", "ds-hale-ctct-scan"], "views": ["ctct-scan"], "layers": [layer_id, TRACE_LAYER], "datasets": ["ds-hale-bond-params", "ds-hale-ctct-scan"], "views": ["ctct-scan"],
"drafts": [STORE + "/" + DRAFT], "claims": {"lode": "amber"}, "drafts": [STORE + "/" + DRAFT], "claims": {"paperclaim": "amber"},
"settings": {"digit_rule": "anchored-sections", "drift_threshold": "0.25", "settings": {"digit_rule": "anchored-sections", "drift_threshold": "0.25",
"spot_check": {"rate": "0.05", "minimum": 10}, "rights_unknown": "refuse", "spot_check": {"rate": "0.05", "minimum": 10}, "rights_unknown": "refuse",
"stages": "scienting-stages-default", "stages": "scienting-stages-default",
@@ -1631,6 +1763,203 @@ def records(out, hale):
wr(out, "draft-ids.jsonl", jsonl(lines)) wr(out, "draft-ids.jsonl", jsonl(lines))
# --- traced citations ---------------------------------------------------------------
def traces(out, hale, draft):
"""The layer of traced citations on the draft, which cites the main example (spec/18 §18.5), and
vectors/traces.jsonl: each trace resolved against the two packages, after corrections that make it current,
changed, stale, unlinked or misresolved, and the traces that break `layer.trace.source` or `layer.trace.target`."""
HALE, WRITER = hale["key"], draft["key"]
QHALE, QWRITER = STORE + "/" + HALE, STORE + "/" + WRITER
OKAFOR = Names.registry["okafor2025open"].key
pkgs = os.path.join(out, "fixtures", "packages")
cited = read_state(os.path.join(pkgs, HALE))
citing = read_state(os.path.join(pkgs, WRITER))
hdoc, wdoc = cited.doc(), citing.doc()
hvals, wvals = hashing.compute(hdoc), hashing.compute(wdoc)
sentences = {hashing.plain(hdoc, n): n for n in hdoc.order if hdoc.nodes[n]["kind"] == "sentence"}
sentence = next(n for t, n in sentences.items() if t.startswith("We fit harmonic bond constants"))
intro = next(n for t, n in sentences.items() if t.startswith("Harmonic bond terms remain"))
table = hale["names"].ids["@tab1"]
cell = hale["names"].ids["@ow_ak"]
sentence_citing = draft["names"].ids["@d2"]
reference = draft["names"].ids["@dr1"]
other_sentence = draft["names"].ids["@d1"]
def anchor(doc, vals, rev, nid, quote=None):
span = None
if quote:
t = hashing.plain(doc, nid)
i = t.index(quote)
span = [i, i + len(quote)]
return AN.make_anchor(doc, vals, nid, rev, span=span, store=STORE)
def selector(a):
return [{"type": "AnchorSelector", "anchor": a}] + (
[{"type": "TextQuoteSelector", "exact": a["quote"]["exact"], "prefix": a["quote"].get("prefix", ""),
"suffix": a["quote"].get("suffix", "")}] if "quote" in a and "exact" in a["quote"] else [])
def target(a):
return {"type": "SpecificResource", "source": TAG + "doc:" + QWRITER, "selector": selector(a)}
def body(passage, note=None, verdict=None, proposer=None):
b = {"type": "SpecificResource", "kind": "trace", "source": TAG + "doc:" + QHALE, "selector": selector(passage),
"reference": reference}
for k, v in (("note", note), ("verdict", verdict), ("proposer", proposer)):
if v is not None:
b[k] = v
return b
ana = {"id": TAG + "key:ana", "type": "Person", "name": "Ana Example"}
ben = {"id": TAG + "key:ben", "type": "Person", "name": "Ben Example"}
model = {"id": TAG + "software:example-model", "type": "Software", "name": "A model (illustrative)"}
def ann(b, tgt, status, who, created, motivation="linking"):
a = {"@context": CTX, "type": "Annotation", "layer": TRACE_LAYER, "motivation": motivation, "creator": who,
"created": created, "body": b, "target": tgt}
a["id"] = TAG + "annotation:smithlab/" + record_id(H(jcs.dumpb(a)))
a["status"] = status
return a
span_citing = anchor(wdoc, wvals, citing.rev, sentence_citing, "two-step fit")
whole_citing = anchor(wdoc, wvals, citing.rev, sentence_citing)
hrev = cited.rev
run = "illustrative-run-2"
# A model found the abstract for the citation of "a two-step fit" and a person confirmed it; its second candidate,
# the introduction's sentence with a citation of its own, a person turned down.
best = ann(body(anchor(hdoc, hvals, hrev, sentence, "in two steps"),
note="The abstract states the two steps: a scan first, then vibrational frequencies.",
verdict="supports", proposer={"model": MODEL, "run": run, "score": "0.8731", "rank": 1}),
target(span_citing), "confirmed", model, "2026-10-12T09:00:00Z")
second = ann(body(anchor(hdoc, hvals, hrev, intro, "classical force fields.1"),
note="Says harmonic terms are common, not how the constants were fitted.",
proposer={"model": MODEL, "run": run, "score": "0.6410", "rank": 2}),
target(span_citing), "rejected", model, "2026-10-12T09:00:00Z")
# A person's trace for the same sentence as a whole: the table it points at.
person = ann(body(anchor(hdoc, hvals, hrev, table), note="Table 1 lists the fitted constants of both parameter sets.",
verdict="supports"),
target(whole_citing), "confirmed", ana, "2026-10-12T09:10:00Z")
reply = ann({"type": "TextualBody", "kind": "comment", "category": "critique",
"value": "The abstract names the two steps but not the constants; the table is the better passage."},
{"type": "SpecificResource", "source": best["id"]}, "proposed", ben, "2026-10-12T10:00:00Z", "assessing")
anns = [best, second, person, reply]
confirmed = [a for a in anns if a["status"] == "confirmed"]
lr = RC.seal({"format": "scienting-receipt/1", "doc": WRITER, "rev": citing.rev, "level": "F3",
"scope": "targeted:traces", "nodes": [],
"records": sorted([["annotation", a["id"].rsplit("/", 1)[1], ann_hash(a)] for a in confirmed]),
"evidence": [], "by": "ana", "at": "2026-10-12T09:30:00Z"}, secret("ana"), "ana")
for a in confirmed:
a["receipt"] = lr["id"]
wr(out, "traces/layer.json", {"@context": CTX, "format": "scienting-layer/1", "id": TRACE_LAYER,
"type": "AnnotationCollection", "label": "Where the draft's citations lead",
"purpose": "Traces from the draft's citations of Hale and Osei to the passages they rest on.",
"audience": "expert", "visibility": "team", "license": {"spdx": "CC-BY-4.0"},
"creator": ana, "created": "2026-10-12T09:00:00Z", "project": "smithlab-charges",
"docs": [QWRITER], "total": len(anns)})
wr(out, "traces/annotations.jsonl", jsonl(anns))
wr(out, "traces/receipt.json", lr)
# The vectors. A case names the two packages, the corrections that bring each to the state to judge, and the
# annotation; the expected outcome is computed here, never typed.
base = {"author": {"key": "ana"}, "reason": "Vector.", "at": "2026-10-13T12:00:00Z"}
person_ev = [{"kind": "person", "by": "ana"}]
crop_ev = [{"kind": "crop", "sha256": crop_sha(cell)}]
def corr(c):
return dict(base, **c)
def evolve(state, corrections):
for c in corrections:
state, _ = CO.apply(state, c, store=SCHEMAS)
return state
def case(cid, a, citing_c=(), cited_c=(), same_as=()):
cc, dc = [corr(c) for c in citing_c], [corr(c) for c in cited_c]
res = TR.resolve(a, evolve(citing, cc), evolve(cited, dc), same_as)
return {"id": cid, "op": "resolve", "citing": "fixtures/packages/" + WRITER, "cited": "fixtures/packages/" + HALE,
"citing_corrections": cc, "cited_corrections": dc, "same_as": list(same_as), "annotation": a,
"expect": res}
def broken(cid, a):
# The cited state is given when the passage is in the cited package; a trace that names the draft itself
# is checked without it.
passage = _anchors_of(a["body"])
use = cited if passage and passage[0]["doc"] == HALE else None
return {"id": cid, "op": "problems", "citing": "fixtures/packages/" + WRITER,
"cited": "fixtures/packages/" + HALE if use else None, "annotation": a,
"expect": TR.problems(a, citing, use)}
unlink = {"op": "relink", "target": sentence_citing, "run": 3, "refs": None, "evidence": person_ev}
def resolve_to(res):
return {"op": "resolve", "target": reference, "resolution": res, "evidence": person_ev}
def replace(target_id, find, new, ev=person_ev):
return {"op": "replace", "target": target_id, "find": find, "replace": new, "evidence": ev}
hale_doi = "doi:10.5555/example.hale.2026"
stale_cited = replace(sentence, "in two steps", "in three stages")
stale_citing = replace(sentence_citing, "two-step", "two-stage")
cases = [
case("current-abstract", best),
case("current-table", person),
case("current-candidate-with-citation", second),
case("changed-passage-moved", best, cited_c=[replace(sentence, "twelve bonds", "all bonds")]),
case("changed-sentence", best, citing_c=[replace(sentence_citing, "Its constants", "The constants")]),
case("changed-table", person, cited_c=[replace(cell, "553.0", "552.0", crop_ev)]),
case("stale-cited", best, cited_c=[stale_cited]),
case("stale-citing", best, citing_c=[stale_citing]),
case("stale-both", best, citing_c=[stale_citing], cited_c=[stale_cited]),
case("stale-citing-whole-sentence", person, citing_c=[stale_citing]),
case("unlinked", best, citing_c=[unlink]),
case("misresolved-doi", best, citing_c=[resolve_to({"type": "doi", "doi": "10.5555/example.other.2020"})],
same_as=[hale_doi]),
case("misresolved-document", best, citing_c=[resolve_to({"type": "document", "doc": STORE + "/" + OKAFOR})],
same_as=[hale_doi]),
case("misresolved-doi-without-same-as", best, citing_c=[resolve_to({"type": "doi", "doi": "10.5555/example.hale.2026"})]),
case("current-doi-in-same-as", best, citing_c=[resolve_to({"type": "doi", "doi": "10.5555/Example.Hale.2026"})],
same_as=[hale_doi]),
case("current-another-version", best, citing_c=[resolve_to({"type": "document", "doc": QHALE + ".v2"})]),
case("current-reference-unresolved", best, citing_c=[resolve_to({"type": "none"})]),
case("unlinked-before-misresolved", best, citing_c=[unlink, resolve_to({"type": "doi", "doi": "10.5555/example.other.2020"})]),
]
# Traces that break a rule: each is the confirmed model trace changed in one way.
tag_okafor = TAG + "doc:" + STORE + "/" + OKAFOR
plain_anchor = _anchors_of(best["body"])[0]
cites_nothing = anchor(wdoc, wvals, citing.rev, other_sentence)
paragraph = anchor(wdoc, wvals, citing.rev, wdoc.nodes[sentence_citing]["parent"])
def edited(body_edit=None, target_edit=None):
a = copy.deepcopy(best)
a["body"].update(body_edit or {})
a["target"] = target_edit if target_edit is not None else a["target"]
return a
quote_differs = copy.deepcopy(best["body"]["selector"])
quote_differs[1]["exact"] = "in three steps"
unknown_node = copy.deepcopy(plain_anchor)
unknown_node["node"] = "zzzzzzz"
broken_cases = [
broken("problems-none", best),
broken("problems-table", person),
broken("source-is-another-document", edited({"source": tag_okafor})),
broken("source-is-the-citing-document", edited({"source": TAG + "doc:" + QWRITER, "selector": selector(span_citing)})),
broken("source-without-anchor", edited({"selector": [best["body"]["selector"][1]]})),
broken("source-quote-differs", edited({"selector": [best["body"]["selector"][0], quote_differs[1]]})),
broken("source-node-unknown", edited({"selector": [{"type": "AnchorSelector", "anchor": unknown_node}]})),
broken("target-is-a-paragraph", edited(target_edit=target(paragraph))),
broken("target-cites-nothing", edited(target_edit=target(cites_nothing))),
broken("target-reference-is-a-sentence", edited({"reference": sentence_citing})),
broken("target-is-a-reply", edited(target_edit={"type": "SpecificResource", "source": second["id"]})),
]
wv(out, "traces.jsonl", cases + broken_cases)
def _anchors_of(body):
return [s["anchor"] for s in body["selector"] if s["type"] == "AnchorSelector"]
# --- invalid packages --------------------------------------------------------------- # --- invalid packages ---------------------------------------------------------------
@@ -1841,13 +2170,14 @@ def main(out):
hale = build_hale(out) hale = build_hale(out)
build_checks_only(out, hale) build_checks_only(out, hale)
build_simple(out, "okafor2025open") build_simple(out, "okafor2025open")
build_simple(out, "smithlab2027charges") draft = build_simple(out, "smithlab2027charges")
tiny = tiny_package(None) tiny = tiny_package(None)
tiny_dir = os.path.join(out, "fixtures", "packages", tiny["key"]) tiny_dir = os.path.join(out, "fixtures", "packages", tiny["key"])
write(tiny_dir, tiny["files"]) write(tiny_dir, tiny["files"])
# The same package zipped as spec/12 §12.2 says a producer writes a byte-identical archive. # The same package zipped as spec/12 §12.2 says a producer writes a byte-identical archive.
write_zip(tiny_dir, tiny_dir + ZIP_EXTENSION) write_zip(tiny_dir, tiny_dir + ZIP_EXTENSION)
records(out, hale) records(out, hale)
traces(out, hale, draft)
vectors(out, hale, tiny) vectors(out, hale, tiny)
invalid(out, tiny) invalid(out, tiny)
src/paperspec/cli/check.py
+279 -13
View File
@@ -27,7 +27,7 @@ from paperspec import anchors as AN # noqa: E402
from paperspec import corrections as CO # noqa: E402 from paperspec import corrections as CO # noqa: E402
from paperspec import crypto, db, fingerprint as FP, grade as GR, hashing, jcs, receipts as RC, rights as RI # noqa: E402 from paperspec import crypto, db, fingerprint as FP, grade as GR, hashing, jcs, receipts as RC, rights as RI # noqa: E402
from paperspec import package as PK # noqa: E402 from paperspec import package as PK # noqa: E402
from paperspec import keys as K, log as LG, mldsa, targets as TG, vrf # noqa: E402 from paperspec import keys as K, log as LG, mldsa, targets as TG, traces as TR, vrf # noqa: E402
import unicodedata # noqa: E402 import unicodedata # noqa: E402
from paperspec.model import Doc # noqa: E402 from paperspec.model import Doc # noqa: E402
from paperspec.notation import fold_math # noqa: E402 from paperspec.notation import fold_math # noqa: E402
@@ -77,6 +77,7 @@ def fixture_key(stem):
HALE = fixture_key("hale2026harmonic") HALE = fixture_key("hale2026harmonic")
DRAFT = fixture_key("smithlab2027charges")
TINY_KEY = fixture_key("tiny2026example") TINY_KEY = fixture_key("tiny2026example")
SCHEMA_FILES = sorted(f for f in os.listdir(p("schemas")) if f.endswith(".schema.json")) SCHEMA_FILES = sorted(f for f in os.listdir(p("schemas")) if f.endswith(".schema.json"))
FORMAT_SCHEMA = {f["format"]: f["schema"] for f in js(p("registry", "formats.json"))["formats"]} FORMAT_SCHEMA = {f["format"]: f["schema"] for f in js(p("registry", "formats.json"))["formats"]}
@@ -194,10 +195,18 @@ def check_records():
stage_ids = {js(p("registry", "stages", f))["id"] for f in os.listdir(p("registry", "stages")) if f.endswith(".json")} stage_ids = {js(p("registry", "stages", f))["id"] for f in os.listdir(p("registry", "stages")) if f.endswith(".json")}
stages = js(os.path.join(R, "project.json")).get("settings", {}).get("stages") stages = js(os.path.join(R, "project.json")).get("settings", {}).get("stages")
ok("records", stages is None or stages in stage_ids, "project settings.stages names no stage file: %s" % stages) ok("records", stages is None or stages in stage_ids, "project settings.stages names no stage file: %s" % stages)
# Confirmed annotations are covered by the layer receipt at their content hash. # Confirmed annotations are covered by the layer receipt at their content hash, in both layers: the example
lr = js(os.path.join(R, "layer", "receipt.json")) # layer on the main example and the layer of traces on the draft.
for layer in ("layer", "traces"):
lj = js(os.path.join(R, layer, "layer.json"))
lr = js(os.path.join(R, layer, "receipt.json"))
items = {(k, i): h for k, i, h in lr["records"]} items = {(k, i): h for k, i, h in lr["records"]}
for a in jl(os.path.join(R, "layer", "annotations.jsonl")): lines = jl(os.path.join(R, layer, "annotations.jsonl"))
ok("records", lj["total"] == len(lines) and all(a["layer"] == lj["id"] for a in lines),
"%s: the layer's total or an annotation's layer differs" % layer)
ok("records", any(d.endswith("/" + lr["doc"]) for d in lj["docs"]),
"%s: the receipt is for a document the layer does not annotate" % layer)
for a in lines:
sel = a["target"].get("selector", []) sel = a["target"].get("selector", [])
anchors = [x["anchor"] for x in sel if x.get("type") == "AnchorSelector"] anchors = [x["anchor"] for x in sel if x.get("type") == "AnchorSelector"]
for x in sel: for x in sel:
@@ -212,7 +221,7 @@ def check_records():
if a["status"] == "confirmed": if a["status"] == "confirmed":
ok("records", items.get(("annotation", a["id"].rsplit("/", 1)[1])) == sha256_hex(jcs.dumpb(body)), ok("records", items.get(("annotation", a["id"].rsplit("/", 1)[1])) == sha256_hex(jcs.dumpb(body)),
"a confirmed annotation is not covered at its hash: " + a["id"]) "a confirmed annotation is not covered at its hash: " + a["id"])
ok("records", lr["root"] == crypto.merkle_root(crypto.receipt_items(lr)), "layer receipt root") ok("records", lr["root"] == crypto.merkle_root(crypto.receipt_items(lr)), "%s: layer receipt root" % layer)
# Every figure of the main example has a confirmed description, which is what an accessible export needs. # Every figure of the main example has a confirmed description, which is what an accessible export needs.
anns = jl(os.path.join(R, "layer", "annotations.jsonl")) anns = jl(os.path.join(R, "layer", "annotations.jsonl"))
described = {x["anchor"]["node"] for a in anns if a["body"]["kind"] == "description" and a["status"] == "confirmed" described = {x["anchor"]["node"] for a in anns if a["body"]["kind"] == "description" and a["status"] == "confirmed"
@@ -222,6 +231,27 @@ def check_records():
% sorted(figures - described)) % sorted(figures - described))
ok("records", any(a["body"]["kind"] == "generated" and a["status"] == "proposed" for a in anns), ok("records", any(a["body"]["kind"] == "generated" and a["status"] == "proposed" for a in anns),
"the layer has no proposed generated annotation to contrast with the confirmed description") "the layer has no proposed generated annotation to contrast with the confirmed description")
# The fixtures' producer supports categories, so it sets one on every comment, caveat and question (spec/18 §18.4).
ok("records", all("category" in a["body"] for a in anns if a["body"]["kind"] in ("comment", "caveat", "question")),
"a comment, caveat or question of the layer has no category")
# A claim's pin names the claim its source does (spec/18 §18.4), and a symbol names its symbol by its key.
claims = [a["body"] for a in anns if a["body"]["kind"] in ("claim-link", "symbol")]
ok("records", {b["kind"] for b in claims} == {"claim-link", "symbol"}, "the layer lacks a claim link or a symbol")
for b in claims:
ok("records", "pin" not in b or b["pin"].startswith(b["source"] + "@"),
"a %s pins another claim than its source: %s, %s" % (b["kind"], b["source"], b.get("pin")))
common = STORE.load("common.schema.json")["$id"]
def errors(defn, value):
return list(Draft202012Validator({"$ref": common + "#/$defs/" + defn}, registry=STORE.registry()).iter_errors(value))
for bad in ("claim:amber/k-bond", "claim:272xtx7c", "claim:272XTXIC", "claim:example/272XTX7C", "claim:272XTX7C@02f47b",
"pcl:272XTX7C", "272XTX7C"):
ok("records", errors("claimRef", bad) != [], "%s is taken for a claim reference" % bad)
for bad in ("illustrative-pin", "claim:272XTX7C", "claim:272XTX7C@02F47B", "claim:272XTX7C@02f47"):
ok("records", errors("claimPin", bad) != [], "%s is taken for a claim pin" % bad)
for good in ("claim:272XTX7C", "claim:claims.example.org/272XTX7C"):
ok("records", errors("claimRef", good) == [] and errors("claimPin", good + "@02f47b") == [],
"%s is refused as a claim reference" % good)
def state_of(pkg): def state_of(pkg):
@@ -262,6 +292,63 @@ def check_packages():
"CSV value differs from its cell: %s/%s" % (va["row"], va["field"])) "CSV value differs from its cell: %s/%s" % (va["row"], va["field"]))
for m in jl(os.path.join(hp, "derived", "mathml.jsonl")): for m in jl(os.path.join(hp, "derived", "mathml.jsonl")):
valid("packages", "mathml.schema.json", m, "derived/mathml.jsonl") valid("packages", "mathml.schema.json", m, "derived/mathml.jsonl")
for m in jl(os.path.join(hp, "derived", "math.jsonl")):
valid("packages", "math.schema.json", m, "derived/math.jsonl")
def check_derived():
"""Derived files are checked, never recomputed (spec/12 §12.7): a line names a formula of the package, and a math
line's formula is TeX and its `complete` and `symbols` agree with its tree (spec/09 §9.4); a tree kind the
schema does not list is accepted, so the form can add kinds; a checks-only package carries no derived file.
No broken package carries these, so each is a variant of the main example's derived/math.jsonl.
"""
hp = p("fixtures", "packages", HALE)
manifest = js(os.path.join(hp, "MANIFEST.json"))
files, absent = {}, [f for f in manifest["files"] if f.get("present", True) is False]
for f in manifest["files"]:
if f.get("present", True):
with open(os.path.join(hp, f["path"]), "rb") as fh:
files[f["path"]] = fh.read()
math = jl(os.path.join(hp, "derived", "math.jsonl"))
eq, run, opaque = math[0], next(m for m in math if m["run"] is not None), next(m for m in math if not m["complete"])
sentence = next(n for n in jl(os.path.join(hp, "nodes.jsonl")) if n["kind"] == "sentence")
cases = [
("as built", math, set()),
("a new tree kind", [dict(eq, tree=["rel", ["sym", "x"], "=", ["tensor", ["sym", "y"]]], symbols=["x", "y"])]
+ math[1:], set()),
("unknown node", [dict(eq, node="zzzzzzz")], {"doc.derived.node"}),
("not a math run", [dict(run, run=0)], {"doc.derived.node"}),
("run past the end", [dict(run, run=99)], {"doc.derived.node"}),
("run null on a sentence", [dict(run, node=sentence["id"], run=None)], {"doc.derived.node"}),
("complete with an opaque node", [dict(opaque, complete=True)], {"doc.derived.math"}),
("incomplete without one", [dict(eq, complete=False)], {"doc.derived.math"}),
("a symbol missing", [dict(eq, symbols=eq["symbols"][1:])], {"doc.derived.math"}),
("a symbol not in the tree", [dict(eq, symbols=eq["symbols"] + ["q"])], {"doc.derived.math"}),
("a tree that is not a node", [dict(eq, tree=[["sym", "x"]])], {"package.schema"}),
("a kind that is not a name", [dict(eq, tree=["Sym", "x"])], {"package.schema"}),
("another form", [dict(eq, form="scienting-math/2")], {"package.schema"}),
("an issue without its text", [dict(opaque, issues=[{"code": "opaque", "message": "x"}])], {"package.schema"}),
]
for name, lines, want in cases:
files["derived/math.jsonl"] = PK.jsonl(lines)
with tempfile.TemporaryDirectory() as tmp:
PK.write(tmp, files, absent)
rules = {e["rule"] for e in PK.validate(tmp, STORE).errors}
ok("packages", rules == want, "derived math, %s: expected %s, got %s" % (name, sorted(want), sorted(rules)))
# A checks-only package withholds the text, and a derived file reads it (spec/12 §12.5).
co = p("fixtures", "packages", HALE + ".checks-only")
team_key = crypto.team_key(bytes.fromhex(open(p("fixtures", "keys", "smithlab-team-secret.hex")).read().strip()))
with tempfile.TemporaryDirectory() as tmp:
pkg = os.path.join(tmp, "pkg")
shutil.copytree(co, pkg)
cm = js(os.path.join(pkg, "MANIFEST.json"))
cfiles = {f["path"]: open(os.path.join(pkg, f["path"]), "rb").read() for f in cm["files"] if f.get("present", True)}
cfiles["derived/math.jsonl"] = PK.jsonl([math[0]])
shutil.rmtree(pkg)
PK.write(pkg, cfiles, [f for f in cm["files"] if f.get("present", True) is False])
rules = {e["rule"] for e in PK.validate(pkg, STORE, team_key).errors}
ok("packages", "doc.policy.withheld" in rules, "a checks-only package with a derived file: %s" % sorted(rules))
def _zip(path, entries): def _zip(path, entries):
@@ -339,6 +426,29 @@ def check_zips():
_zip(os.path.join(tmp, "empty.zip"), []) _zip(os.path.join(tmp, "empty.zip"), [])
ok("zips", {e["rule"] for e in PK.validate_zip(os.path.join(tmp, "empty.zip"), STORE).errors} ok("zips", {e["rule"] for e in PK.validate_zip(os.path.join(tmp, "empty.zip"), STORE).errors}
== {"package.zip.entry"}, "an empty archive is refused") == {"package.zip.entry"}, "an empty archive is refused")
# Archives the reader cannot read are refused with the rule, never with the zipfile module's own error: an
# entry that needs a later zip version (NotImplementedError), and deflated bytes that do not inflate
# (zlib.error), as `paperspec stress --seed 1` found.
with zipfile.ZipFile(os.path.join(tmp, "zip-version.zip"), "w") as zf:
for name, data, _ in entries:
zi = zipfile.ZipInfo(name, date_time=(1980, 1, 1, 0, 0, 0))
zi.extract_version = 98
zf.writestr(zi, data)
with zipfile.ZipFile(os.path.join(tmp, "bad-deflate.zip"), "w", zipfile.ZIP_DEFLATED) as zf:
for name, data, _ in entries:
zf.writestr(zipfile.ZipInfo(name, date_time=(1980, 1, 1, 0, 0, 0)), data, zipfile.ZIP_DEFLATED)
with zipfile.ZipFile(os.path.join(tmp, "bad-deflate.zip")) as zf:
zi = next(i for i in zf.infolist() if i.compress_size > 8)
start = zi.header_offset + 30 + len(zi.filename.encode("utf-8")) + len(zi.extra)
with open(os.path.join(tmp, "bad-deflate.zip"), "r+b") as fh:
fh.seek(start)
fh.write(b"\xff" * 8)
for name in ("zip-version", "bad-deflate"):
try:
rules = {e["rule"] for e in PK.validate_zip(os.path.join(tmp, name + ".zip"), STORE).errors}
except Exception as e: # noqa: BLE001
rules = {"raised %r" % e}
ok("zips", rules == {"package.zip.entry"}, "%s: expected package.zip.entry, got %s" % (name, sorted(rules)))
outside = [f for f in os.listdir(tmp) if not f.endswith(".zip")] outside = [f for f in os.listdir(tmp) if not f.endswith(".zip")]
ok("zips", not outside, "reading archives wrote outside their folders: %s" % outside) ok("zips", not outside, "reading archives wrote outside their folders: %s" % outside)
@@ -550,6 +660,24 @@ def check_vectors():
ok("vectors", vals == c["values"], "hashes " + c["id"]) ok("vectors", vals == c["values"], "hashes " + c["id"])
for nid, o in c["own_content"].items(): for nid, o in c["own_content"].items():
ok("vectors", sha256_hex(jcs.dumpb(o)) == vals[nid]["own_hash"], "own content " + nid) ok("vectors", sha256_hex(jcs.dumpb(o)) == vals[nid]["own_hash"], "own content " + nid)
# A math form enters no content value (spec/05 §5.2): the values of the nodes with `form` are those without it,
# and the nodes validate; the schema takes `form` only beside the notation tex.
fc = next(c for c in V("hashes.jsonl") if c["id"] == "math-form")
def unformed(n):
c = n.get("content")
if isinstance(c, dict):
c = {k: v for k, v in c.items() if k != "form"}
elif isinstance(c, list):
c = [{k: v for k, v in r.items() if k != "form"} for r in c]
return dict(n, content=c) if "content" in n else n
formed = sum(json.dumps(n).count('"form"') for n in fc["nodes"])
ok("vectors", formed == 2 and hashing.compute(Doc(fc["doc"], [unformed(n) for n in fc["nodes"]])) == fc["values"],
"a math form changes a content value")
for n in fc["nodes"]:
valid("vectors", "node.schema.json", dict(n, **fc["values"][n["id"]]), "hashes math-form " + n["id"])
eq = next(n for n in fc["nodes"] if n["kind"] == "equation")
typst = dict(eq, content=dict(eq["content"], notation="typst"), **fc["values"][eq["id"]])
ok("vectors", schema_errors("node.schema.json", typst) != [], "the schema takes a form beside the notation typst")
for c in V("addresses.jsonl"): for c in V("addresses.jsonl"):
ok("vectors", AN.parse_address(c["address"]) == c["parsed"], "address " + c["id"]) ok("vectors", AN.parse_address(c["address"]) == c["parsed"], "address " + c["id"])
if c["parsed"]: if c["parsed"]:
@@ -610,18 +738,27 @@ def check_vectors():
"zero errors in 300 bounds the rate below 1%") "zero errors in 300 bounds the rate below 1%")
for c in V("alignment.jsonl"): for c in V("alignment.jsonl"):
ok("vectors", AL.align(c["a"], c["b"]) == c["expect"], "alignment " + c["id"]) ok("vectors", AL.align(c["a"], c["b"]) == c["expect"], "alignment " + c["id"])
from paperspec.cli.build import correction_expect
for c in V("corrections.jsonl"): for c in V("corrections.jsonl"):
st = CO.State(c["doc"], [{k: v for k, v in n.items() if k not in ( st = CO.State(c["doc"], [{k: v for k, v in n.items() if k not in (
"text_hash", "match_hash", "align_key", "own_hash", "node_hash")} for n in c["before"]["nodes"]]) "text_hash", "match_hash", "align_key", "own_hash", "node_hash")} for n in c["before"]["nodes"]],
references=c["before"].get("references"))
try: try:
st1, rec = CO.apply(st, c["correction"], store=STORE) st1, rec = CO.apply(st, c["correction"], store=STORE)
v1 = hashing.compute(st1.doc())
ok("vectors", st1.index().values == reference_values(st1.doc()), ok("vectors", st1.index().values == reference_values(st1.doc()),
"correction %s: incremental values differ from the full computation" % c["id"]) "correction %s: incremental values differ from the full computation" % c["id"])
got = {"record": rec, "nodes": [dict(n, **v1[n["id"]]) for n in st1.nodes], "ids": st1.ids} got = correction_expect(st1, rec, "references" in c["before"])
except Refusal as e: except Refusal as e:
got = {"refusal": e.rule} got = {"refusal": e.rule}
ok("vectors", got == c["expect"], "correction " + c["id"]) ok("vectors", got == c["expect"], "correction " + c["id"])
if c["correction"]["op"] == "emend" and "record" in c["expect"]:
# An emend leaves the leaf as printed, and so its match_hash and align_key, as they were (spec/11 §11.3).
nid = c["correction"]["target"]
before = {n["id"]: n for n in c["before"]["nodes"]}[nid]
after = {n["id"]: n for n in c["expect"]["nodes"]}[nid]
ok("vectors", c["expect"]["printed"][nid] == hashing.plain(st.doc(), nid)
and all(before[k] == after[k] for k in ("match_hash", "align_key")) and before["text_hash"] != after["text_hash"],
"correction %s: the emend changed the leaf as printed" % c["id"])
states = {} states = {}
for c in V("resolve.jsonl"): for c in V("resolve.jsonl"):
st = states.setdefault(c["package"], state_of(p(c["package"]))) st = states.setdefault(c["package"], state_of(p(c["package"])))
@@ -710,6 +847,116 @@ def check_vectors():
ok("vectors", sorted(LG.tiles(c["entries"])) == c["tiles"], "checkpoint %s: tile paths" % c["id"]) ok("vectors", sorted(LG.tiles(c["entries"])) == c["tiles"], "checkpoint %s: tile paths" % c["id"])
def check_traces():
"""Traced citations (spec/18 §18.5): the layer of traces on the draft, resolved against the two packages, and
the vectors, which resolve and check traces after corrections that bring every outcome and every refusal."""
R = p("fixtures", "records", "traces")
states = {}
def st(path):
return states.setdefault(path, state_of(p(path)))
def evolve(state, corrections):
for c in corrections:
state, _ = CO.apply(state, c, store=STORE)
return state
rules = {r["id"]: r for r in js(p("registry", "rules.json"))["rules"]}
for rule in ("layer.trace.source", "layer.trace.target"):
ok("traces", rule in rules and rules[rule]["chapter"] == "spec/18-layers.md", "%s is not registered" % rule)
# The fixture layer: a model's confirmed trace to the abstract, a candidate a person rejected, a person's trace to
# Table 1 and a reply that disputes the first. Every trace is current, and breaks no rule alone or with the states.
citing, cited = st("fixtures/packages/" + DRAFT), st("fixtures/packages/" + HALE)
same_as = js(p("fixtures", "packages", HALE, "document.json"))["work"]["same_as"]
anns = jl(os.path.join(R, "annotations.jsonl"))
mine = [a for a in anns if a["body"]["kind"] == "trace"]
ok("traces", len(mine) == 3 and len(anns) == 4, "the layer holds three traces and a reply")
refs = {r["node"]: r["resolution"] for r in citing.references}
for a in mine:
b = a["body"]
ok("traces", TR.problems(a) == [] and TR.problems(a, citing, cited) == [], "a trace breaks a rule: " + a["id"])
got = TR.resolve(a, citing, cited, same_as)
ok("traces", got["result"] == "current" and got["open"] is not None, "a fixture trace is not current: " + a["id"])
ok("traces", ("proposer" in b) == (a["creator"]["type"] == "Software"), "proposer iff Software: " + a["id"])
ok("traces", refs[b["reference"]] == {"type": "document", "doc": b["source"].split("doc:", 1)[1]},
"the reference does not resolve to the trace's source: " + a["id"])
tool = next(a for a in mine if a["creator"]["type"] == "Software" and a["status"] == "confirmed")
ok("traces", {a["status"] for a in mine} == {"confirmed", "rejected"}
and [a["status"] for a in mine if a["creator"]["type"] == "Software"].count("rejected") == 1
and any(a["creator"]["type"] == "Person" and a["status"] == "confirmed"
and TR._anchors(a["body"])[0]["kind"] == "table" for a in mine),
"the layer lacks a confirmed model trace, a rejected candidate or a person's trace to a table")
ok("traces", [a["id"] for a in mine if TR.disputed(a["id"], anns)] == [tool["id"]]
and len(TR.disputes(tool["id"], anns)) == 1, "only the model's confirmed trace is disputed")
reply = TR.disputes(tool["id"], anns)[0]
ok("traces", reply["motivation"] == "assessing" and reply["creator"] != tool["creator"] and reply not in mine,
"the dispute is not an assessing reply by another")
for edit in ({"status": "withdrawn"}, {"status": "rejected"}, {"motivation": "replying"}):
ok("traces", not TR.disputed(tool["id"], [dict(reply, **edit)]), "a reply with %s disputes" % edit)
ok("traces", TR.replies(tool["id"], anns) == [reply] and TR.replies(reply["id"], anns) == [], "replies are found by target")
# A model's trace names its proposer; a person's may too; the shapes are closed (spec/18 §18.4).
bare = {k: v for k, v in tool["body"].items() if k != "proposer"}
person = next(a for a in mine if a["creator"]["type"] == "Person")
for name, a, good in (
("a model's trace without a proposer", dict(tool, body=bare), False),
("a person's trace with one", dict(person, body=dict(person["body"], proposer=tool["body"]["proposer"])), True),
("a rank below 1", dict(tool, body=dict(tool["body"], proposer=dict(tool["body"]["proposer"], rank=0))), False),
("a score that is no decimal", dict(tool, body=dict(tool["body"], proposer=dict(tool["body"]["proposer"], score="high"))), False),
("an unknown verdict", dict(person, body=dict(person["body"], verdict="agrees")), False),
("no reference", dict(person, body={k: v for k, v in person["body"].items() if k != "reference"}), False),
("a source that is no document tag", dict(person, body=dict(person["body"], source="hale2026harmonic-damb")), False),
("an empty note", dict(person, body=dict(person["body"], note="")), False),
("an extra property", dict(person, body=dict(person["body"], language="en")), False)):
ok("traces", (schema_errors("annotation.schema.json", a) == []) == good, "schema, %s" % name)
# The states must be those of the documents the trace names.
for call in (lambda: TR.resolve(tool, cited, citing), lambda: TR.problems(tool, cited), lambda: TR.problems(tool, citing, citing),
lambda: TR.resolve(next(a for a in jl(os.path.join(ROOT, "fixtures", "records", "layer", "annotations.jsonl"))), citing, cited)):
try:
call()
raised = False
except ValueError:
raised = True
ok("traces", raised, "a call with the wrong states or a body that is no trace did not raise")
# The vocabulary and the store know the kind (spec/18 §18.4, sql/store.sql).
ctx = js(p("context", "scienting-1.jsonld"))["@context"]
ok("traces", ctx["note"] == "sct:note" and ctx["proposer"] == {"@id": "sct:proposedBy", "@type": "@json"},
"the context lacks the terms note and proposer")
# The vectors.
outcomes, broken = set(), set()
ids = set()
for c in V("traces.jsonl"):
ids.add(c["id"])
valid("traces", "annotation.schema.json", c["annotation"], "vector " + c["id"])
a_citing = st(c["citing"])
snapshot = json.dumps([a_citing.nodes, a_citing.references], sort_keys=True)
if c["op"] == "resolve":
got = TR.resolve(c["annotation"], evolve(a_citing, c["citing_corrections"]),
evolve(st(c["cited"]), c["cited_corrections"]), c["same_as"])
ok("traces", got == c["expect"], "traces " + c["id"])
ok("traces", got["result"] in TR.RESULTS and ("side" in got) == (got["result"] == "stale")
and ("reason" in got) == (got["result"] != "current")
and (got["open"] is None) == (got["cited"]["result"] == "stale") and (got["cites"] == [] or got["open"]),
"traces %s: the result is not shaped as traces.resolve says" % c["id"])
outcomes.add(got["result"])
# Correcting a state to judge a trace leaves the state it started from as it was.
ok("traces", json.dumps([a_citing.nodes, a_citing.references], sort_keys=True) == snapshot,
"traces %s: the state changed" % c["id"])
else:
got = TR.problems(c["annotation"], a_citing, st(c["cited"]) if c["cited"] else None)
ok("traces", got == c["expect"], "traces " + c["id"])
broken |= {x["rule"] for x in got}
ok("traces", all(x["rule"] in rules for x in got), "traces %s: an unregistered rule" % c["id"])
ok("traces", outcomes == set(TR.RESULTS), "every outcome occurs: %s" % sorted(outcomes))
ok("traces", broken == {"layer.trace.source", "layer.trace.target"}, "both rules are broken by some case")
ok("traces", {"changed-passage-moved", "stale-both", "unlinked", "misresolved-doi-without-same-as",
"current-doi-in-same-as"} <= ids, "the vectors lack a case they are known for")
cites = [c for c in V("traces.jsonl") if c["op"] == "resolve" and c["expect"]["cites"]]
ok("traces", len(cites) == 1 and cites[0]["expect"]["cites"][0]["refs"][0]["resolution"]["type"] == "doi",
"one case follows a citation inside the passage")
def check_determinism(): def check_determinism():
with tempfile.TemporaryDirectory() as tmp: with tempfile.TemporaryDirectory() as tmp:
subprocess.run([sys.executable, "-m", "paperspec", "build", "--out", tmp], check=True) subprocess.run([sys.executable, "-m", "paperspec", "build", "--out", tmp], check=True)
@@ -767,6 +1014,10 @@ def check_sql():
for m in re.finditer(r"CREATE INDEX (\w+) ON", ddl): for m in re.finditer(r"CREATE INDEX (\w+) ON", ddl):
ok("sql", m.group(1).upper() not in RESERVED, "index name " + m.group(1)) ok("sql", m.group(1).upper() not in RESERVED, "index name " + m.group(1))
ok("sql", "FOREIGN KEY" not in ddl and "REFERENCES" not in ddl, "no foreign keys") ok("sql", "FOREIGN KEY" not in ddl and "REFERENCES" not in ddl, "no foreign keys")
# The store accepts exactly the body kinds the annotation schema has.
sql_kinds = set(re.findall(r"'([a-z-]+)'", re.search(r"body_kind VARCHAR\(24\) NOT NULL CHECK \(body_kind IN \((.*?)\)\)", ddl).group(1)))
schema_kinds = {b["properties"]["kind"]["const"] for b in STORE.load("annotation.schema.json")["properties"]["body"]["oneOf"]}
ok("sql", sql_kinds == schema_kinds, "body_kind differs from the schema's body kinds: %s" % sorted(sql_kinds ^ schema_kinds))
with tempfile.TemporaryDirectory() as tmp: with tempfile.TemporaryDirectory() as tmp:
conn = sqlite3.connect(os.path.join(tmp, "bundle.db")) conn = sqlite3.connect(os.path.join(tmp, "bundle.db"))
db.create_bundle(conn, p("sql")) db.create_bundle(conn, p("sql"))
@@ -793,14 +1044,17 @@ def check_sql():
check_bundle(conn) check_bundle(conn)
# Store records satisfy the store tables' CHECK constraints. # Store records satisfy the store tables' CHECK constraints.
R = p("fixtures", "records") R = p("fixtures", "records")
lay = js(os.path.join(R, "layer", "layer.json")) for layer in ("layer", "traces"):
lay = js(os.path.join(R, layer, "layer.json"))
conn.execute("INSERT INTO layers VALUES (?,?,?,?,?,?,?,?)", conn.execute("INSERT INTO layers VALUES (?,?,?,?,?,?,?,?)",
(lay["id"], lay.get("project"), lay["label"], lay["purpose"], lay.get("audience"), (lay["id"], lay.get("project"), lay["label"], lay["purpose"], lay.get("audience"),
lay["visibility"], jcs.dumps(lay["license"]), jcs.dumps(lay))) lay["visibility"], jcs.dumps(lay["license"]), jcs.dumps(lay)))
for a in jl(os.path.join(R, "layer", "annotations.jsonl")): for a in jl(os.path.join(R, layer, "annotations.jsonl")):
conn.execute("INSERT INTO annotations VALUES (?,?,?,?,?,?,?,?)", conn.execute("INSERT INTO annotations VALUES (?,?,?,?,?,?,?,?,?)",
(a["id"], a["layer"], a["body"]["kind"], a["status"], a.get("receipt"), a["creator"]["id"], (a["id"], a["layer"], a["body"]["kind"], a["body"].get("category"), a["status"],
a["created"], jcs.dumps(a))) a.get("receipt"), a["creator"]["id"], a["created"], jcs.dumps(a)))
kinds = {k for (k,) in conn.execute("SELECT DISTINCT body_kind FROM annotations")}
ok("sql", "trace" in kinds, "the store holds no trace annotation")
prj = js(os.path.join(R, "project.json")) prj = js(os.path.join(R, "project.json"))
conn.execute("INSERT INTO projects VALUES (?,?,?,?,?,?,?)", conn.execute("INSERT INTO projects VALUES (?,?,?,?,?,?,?)",
(prj["key"], prj.get("store"), prj["name"], prj.get("description"), jcs.dumps(prj.get("claims")), (prj["key"], prj.get("store"), prj["name"], prj.get("description"), jcs.dumps(prj.get("claims")),
@@ -1043,6 +1297,16 @@ def check_log():
ok("log", not LG.accept_checkpoint(note, roster, dict(settings, origin="other"))[0], "another origin was accepted") ok("log", not LG.accept_checkpoint(note, roster, dict(settings, origin="other"))[0], "another origin was accepted")
alone = LG.sign_checkpoint(settings["origin"], entries, [("wit", __import__("paperspec.cli.build", fromlist=["secret"]).secret("wit"))]) alone = LG.sign_checkpoint(settings["origin"], entries, [("wit", __import__("paperspec.cli.build", fromlist=["secret"]).secret("wit"))])
ok("log", not LG.accept_checkpoint(alone, roster, settings)[0], "a checkpoint without the team's key was accepted") ok("log", not LG.accept_checkpoint(alone, roster, settings)[0], "a checkpoint without the team's key was accepted")
# Every line of a note ends in U+000A and nothing else (spec/23 §23.2): a note whose last line break, or the break
# between its signature lines, is another line separator is malformed, as `paperspec stress --seed 2` found.
head, _, sigs = note.partition("\n\n")
lines = sigs.split("\n")[:-1]
ok("log", len(lines) >= 2 and LG.parse_checkpoint(head + "\n\n" + "\n".join(lines) + "\n") is not None,
"the checkpoint does not have two signature lines to separate")
for sep in ("\x0b", "\x0c", "\x1c", "\x1d", "\x1e", "\x85", "
", "
", "\r", ""):
ok("log", LG.parse_checkpoint(note[:-1] + sep) is None, "a note ending in %r instead of a newline parses" % sep)
ok("log", LG.parse_checkpoint(head + "\n\n" + sep.join(lines) + "\n") is None,
"a note whose signature lines are separated by %r parses" % sep)
# The ML-DSA-65 receipt verifies under dana's key, and fails under a tampered payload. # The ML-DSA-65 receipt verifies under dana's key, and fails under a tampered payload.
rec = js(os.path.join(R, "receipt-ml-dsa.json")) rec = js(os.path.join(R, "receipt-ml-dsa.json"))
ok("log", rec["signature"]["scheme"] == "ml-dsa-65" and RC.verify_signature(rec, roster), "the ML-DSA receipt does not verify") ok("log", rec["signature"]["scheme"] == "ml-dsa-65" and RC.verify_signature(rec, roster), "the ML-DSA receipt does not verify")
@@ -1175,11 +1439,13 @@ def run(args):
check_packages() check_packages()
check_zips() check_zips()
check_owner_revocation() check_owner_revocation()
check_derived()
check_unsafe_paths() check_unsafe_paths()
check_incremental() check_incremental()
check_gate() check_gate()
check_invalid() check_invalid()
check_vectors() check_vectors()
check_traces()
if not args.quick: if not args.quick:
check_determinism() check_determinism()
check_sql() check_sql()
src/paperspec/cli/stress.py
+50 -4
View File
@@ -5,7 +5,8 @@ Usage: paperspec stress [--seed N] [--rounds N]
The vectors pin values; this program checks the invariants the chapters state, over inputs drawn from a seeded The vectors pin values; this program checks the invariants the chapters state, over inputs drawn from a seeded
random generator, so a failure is reproducible from the seed it prints. Canonical JSON round-trips and sorts keys random generator, so a failure is reproducible from the seed it prints. Canonical JSON round-trips and sorts keys
by UTF-16 units; normalization is idempotent; notation folding never raises; random corrections keep the five by UTF-16 units; normalization is idempotent; notation folding never raises; random corrections keep the five
values equal to a full recomputation, never reuse an ID and apply deterministically; anchors made before those values equal to a full recomputation, never reuse an ID and apply deterministically, and an emend, relink or
resolve leaves what a witness read, and the receipts on it, as they were; anchors made before those
corrections always resolve to one of the five outcomes; Merkle roots agree with a second implementation; corrections always resolve to one of the five outcomes; Merkle roots agree with a second implementation;
spot-check samples do not depend on input order; error bounds are the least six-decimal value with the stated tail; spot-check samples do not depend on input order; error bounds are the least six-decimal value with the stated tail;
differently stamped pages share a fingerprint; the zip reader and the package validator refuse or accept mutated differently stamped pages share a fingerprint; the zip reader and the package validator refuse or accept mutated
@@ -154,7 +155,7 @@ def random_correction(rng, st, doc):
sentences = _leaf_ids(doc, {"sentence"}) sentences = _leaf_ids(doc, {"sentence"})
cells = _leaf_ids(doc, {"cell"}) cells = _leaf_ids(doc, {"cell"})
op = rng.choice(["replace", "replace", "split", "merge", "move", "relabel", "set", "delete", "reclassify", op = rng.choice(["replace", "replace", "split", "merge", "move", "relabel", "set", "delete", "reclassify",
"insert-text", "insert-node"]) "insert-text", "insert-node", "emend", "emend", "relink", "resolve"])
evidence = [{"kind": "crop", "sha256": "0" * 64}] if rng.random() < 0.8 else [{"kind": "person", "by": "ana"}] evidence = [{"kind": "crop", "sha256": "0" * 64}] if rng.random() < 0.8 else [{"kind": "person", "by": "ana"}]
c = {"op": op.split("-")[0], "evidence": evidence} c = {"op": op.split("-")[0], "evidence": evidence}
if op == "replace" and (sentences or cells): if op == "replace" and (sentences or cells):
@@ -188,7 +189,12 @@ def random_correction(rng, st, doc):
c.update(target=rng.choice(cells), attr=rng.choice(["colspan", "rowspan", "role", "column", "level"]), c.update(target=rng.choice(cells), attr=rng.choice(["colspan", "rowspan", "role", "column", "level"]),
value=rng.choice([1, 2, 3, "2", -1, "data", "row-header", "boss", None])) value=rng.choice([1, 2, 3, "2", -1, "data", "row-header", "boss", None]))
elif op == "delete" and len(doc.order) > 8: elif op == "delete" and len(doc.order) > 8:
c.update(target=rng.choice(doc.order[1:])) # Never below eight nodes: deleting the body section would leave a document with nothing to correct, and the
# rest of the walk would only be refused.
targets = [nid for nid in doc.order[1:] if len(doc.order) - len(doc.preorder(nid)) >= 8]
if not targets:
return None
c.update(target=rng.choice(targets))
elif op == "reclassify": elif op == "reclassify":
c.update(target=rng.choice(doc.order), kind=rng.choice(["footnote", "quote", "paragraph", "caption", "section"])) c.update(target=rng.choice(doc.order), kind=rng.choice(["footnote", "quote", "paragraph", "caption", "section"]))
elif op == "insert-text" and sentences: elif op == "insert-text" and sentences:
@@ -205,6 +211,25 @@ def random_correction(rng, st, doc):
if rng.random() < 0.2: if rng.random() < 0.2:
node["foo"] = 1 node["foo"] = 1
c.update(parent=parent, place_after=rng.choice(kids + [None]), node=node) c.update(parent=parent, place_after=rng.choice(kids + [None]), node=node)
elif op == "emend" and (sentences or cells):
nid = rng.choice(sentences + cells)
texts = [r.get("text") or r.get("source", "") for r in doc.nodes[nid]["content"]]
text = rng.choice(texts + [hashing.plain(doc, nid)])
if not text:
return None
i = rng.randrange(len(text))
j = min(len(text), i + rng.randint(1, 6))
c.update(target=nid, find=text[i:j], replace=rand_text(rng, 1, 8),
kind=rng.choice(["typo", "spelling", "grammar", "punctuation", "wording", "other"]))
elif op == "relink" and sentences:
nid = rng.choice(sentences)
refs = _leaf_ids(doc, {"reference"})
c.update(target=nid, run=rng.randint(0, len(doc.nodes[nid]["content"])),
refs=rng.choice([None, refs[:1] or None, [rng.choice(doc.order)]]))
elif op == "resolve":
c.update(target=rng.choice(_leaf_ids(doc, {"reference"}) + [rng.choice(doc.order)]),
resolution=rng.choice([{"type": "none"}, {"type": "doi", "doi": "10.5555/stress.%d" % rng.randint(0, 9)},
{"type": "document", "doc": "smithlab/" + TINY_KEY}]))
else: else:
return None return None
return c return c
@@ -247,6 +272,24 @@ def stress_corrections(rng, rounds):
ok("corrections", rec["id"] == record_id(sha256_hex(jcs.dumpb({k: v for k, v in rec.items() if k != "id"}))), ok("corrections", rec["id"] == record_id(sha256_hex(jcs.dumpb({k: v for k, v in rec.items() if k != "id"}))),
"the record's id does not derive from its content") "the record's id does not derive from its content")
ok("corrections", rec["index"] == st.rev + 1 == st1.rev, "the record's index is not the next revision") ok("corrections", rec["index"] == st.rev + 1 == st1.rev, "the record's index is not the next revision")
# Emend, relink and resolve leave what a witness read as it was (spec/11 §11.3): an emend keeps the leaf as
# printed, and so its match_hash and align_key; a relink keeps its plain projection; a resolve its node_hash.
t = c.get("target")
d0, d1 = st.doc(), st1.doc()
if c["op"] == "emend":
v0, v1 = st.index().values[t], st1.index().values[t]
ok("corrections", hashing.printed(d0, t) == hashing.printed(d1, t) and v0["match_hash"] == v1["match_hash"]
and v0["align_key"] == v1["align_key"], "an emend changed the leaf as printed: %s" % json.dumps(c)[:160])
elif c["op"] == "relink":
ok("corrections", hashing.plain(d0, t) == hashing.plain(d1, t), "a relink changed the text")
elif c["op"] == "resolve":
ok("corrections", st.index().values == st1.index().values
and {r["node"]: r["resolution"] for r in st1.references}.get(t) == c["resolution"],
"a resolve changed a node or did not record its resolution")
if c["op"] in CO.KEEPS_LEVEL:
carried = RC.carried_hashes(st1.index().values, st1.corrections)
ok("corrections", st.index().values[t]["node_hash"] in carried[t],
"a receipt at the hash before an %s would no longer count" % c["op"])
st2, rec2 = CO.apply(st, copy.deepcopy(c), store=SCHEMAS) st2, rec2 = CO.apply(st, copy.deepcopy(c), store=SCHEMAS)
ok("corrections", rec2 == rec and st2.nodes == st1.nodes and st2.ids == st1.ids, "apply is not deterministic") ok("corrections", rec2 == rec and st2.nodes == st1.nodes and st2.ids == st1.ids, "apply is not deterministic")
try: try:
@@ -443,8 +486,11 @@ def stress_packages(rng, rounds):
data[k] = (data[k] + rng.randrange(1, 256)) % 256 data[k] = (data[k] + rng.randrange(1, 256)) % 256
open(full, "wb").write(data) open(full, "wb").write(data)
elif mode == "truncate": elif mode == "truncate":
# Strictly shorter: a cut at the full length would leave the file as it was, and the package valid.
data = open(full, "rb").read() data = open(full, "rb").read()
open(full, "wb").write(data[:rng.randrange(len(data) + 1)]) if not data:
continue
open(full, "wb").write(data[:rng.randrange(len(data))])
elif mode == "delete": elif mode == "delete":
os.remove(full) os.remove(full)
elif mode == "add": elif mode == "add":
src/paperspec/corrections.py
+140 -6
View File
@@ -23,17 +23,23 @@ SETTABLE = {
"theorem": {"variant"}, "theorem": {"variant"},
} }
EDITABLE_RUN_FIELD = {"text": "text", "link": "text", "cite": "text", "xref": "text", "fnmark": "text", "math": "source"} EDITABLE_RUN_FIELD = {"text": "text", "link": "text", "cite": "text", "xref": "text", "fnmark": "text", "math": "source"}
# The runs an emend searches: prose, never a math or equation source (spec/11 §11.1).
PROSE_RUNS = frozenset(["text", "link", "cite", "xref", "fnmark"])
# The operations that never lower a leaf's level: what a witness read stays what it was (spec/08 §8.1-8.2).
KEEPS_LEVEL = frozenset(["emend", "relink", "resolve"])
class State: class State:
"""The current state of a document: nodes, the ID table and the corrections applied.""" """The current state of a document: nodes, the ID table, the corrections applied and the reference records."""
def __init__(self, key, nodes, ids=None, corrections=None, indexes=None): def __init__(self, key, nodes, ids=None, corrections=None, indexes=None, references=None):
self.key = key self.key = key
self.nodes = [dict(n) for n in nodes] self.nodes = [dict(n) for n in nodes]
self.ids = [dict(e) for e in (ids or [])] self.ids = [dict(e) for e in (ids or [])]
self.corrections = [dict(c) for c in (corrections or [])] self.corrections = [dict(c) for c in (corrections or [])]
self._indexes = dict(indexes or {}) self._indexes = dict(indexes or {})
# The records of references.jsonl, which a resolve rewrites (spec/10, spec/11 §11.1).
self.references = [dict(r) for r in (references or [])]
@property @property
def rev(self): def rev(self):
@@ -85,6 +91,34 @@ def _has_crop(c):
return any(e.get("kind") == "crop" for e in c.get("evidence", [])) return any(e.get("kind") == "crop" for e in c.get("evidence", []))
def _carries_printed(v):
"""Whether runs, or a subtree to insert, carry `printed`, which only an emend writes (spec/11 §11.4)."""
if isinstance(v, list):
return any(_carries_printed(x) for x in v)
if isinstance(v, dict):
return "printed" in v or any(_carries_printed(x) for x in v.values())
return False
def _edits_by_digit(old, new):
"""Whether turning `old` into `new` changes, adds or removes a digit, or edits beside one (spec/11 §11.4).
The edit is what is left once the longest common prefix, and then the longest common suffix of the rest, are
set aside; its characters in both strings, and the unchanged character on each side of it, are checked.
"""
if old == new:
return False
p = 0
while p < len(old) and p < len(new) and old[p] == new[p]:
p += 1
s = 0
while s < len(old) - p and s < len(new) - p and old[-1 - s] == new[-1 - s]:
s += 1
edit = old[p:len(old) - s] + new[p:len(new) - s]
beside = old[max(p - 1, 0):p] + old[len(old) - s:len(old) - s + 1]
return bool(digits(edit + beside))
def _rebuild(nodes): def _rebuild(nodes):
"""Recompute document order and `ord` from parent links, keeping sibling order as listed.""" """Recompute document order and `ord` from parent links, keeping sibling order as listed."""
by_id = {n["id"]: n for n in nodes} by_id = {n["id"]: n for n in nodes}
@@ -181,6 +215,7 @@ def apply(state, c, hasher=hashing.PLAIN, store=None):
retired, created = [], [] retired, created = [], []
path = doc.ancestors(target) if target in by_id else None path = doc.ancestors(target) if target in by_id else None
touched = [] touched = []
references = state.references
if op == "replace": if op == "replace":
node = by_id[target] node = by_id[target]
@@ -195,6 +230,11 @@ def apply(state, c, hasher=hashing.PLAIN, store=None):
if len(hits) != 1: if len(hits) != 1:
raise Refusal("correction.find", "find occurs %d times, not exactly once" % len(hits), target) raise Refusal("correction.find", "find occurs %d times, not exactly once" % len(hits), target)
pth, s, i = hits[0] pth, s, i = hits[0]
# An emended run's text is the editor's, not a reading, so only another emend changes it.
if len(pth) == 2 and "printed" in node["content"][pth[0]]:
raise Refusal("correction.emend", "the run was emended; only another emend changes its text", target)
if _carries_printed(repl):
raise Refusal("correction.emend", "only an emend records what was printed", target)
if isinstance(repl, str): if isinstance(repl, str):
_set_string(node, pth, s[:i] + repl + s[i + len(find):]) _set_string(node, pth, s[:i] + repl + s[i + len(find):])
else: else:
@@ -215,6 +255,8 @@ def apply(state, c, hasher=hashing.PLAIN, store=None):
elif op == "insert" and "node" in c: elif op == "insert" and "node" in c:
if not _has_crop(c): if not _has_crop(c):
raise Refusal("correction.evidence", "an insert needs crop evidence") raise Refusal("correction.evidence", "an insert needs crop evidence")
if _carries_printed(c["node"]):
raise Refusal("correction.emend", "only an emend records what was printed")
parent = c["parent"] parent = c["parent"]
if parent not in by_id: if parent not in by_id:
raise Refusal("correction.target", "no live parent %s" % parent) raise Refusal("correction.target", "no live parent %s" % parent)
@@ -244,6 +286,8 @@ def apply(state, c, hasher=hashing.PLAIN, store=None):
f = EDITABLE_RUN_FIELD.get(r["kind"]) f = EDITABLE_RUN_FIELD.get(r["kind"])
if not f or offset > len(r[f]) or not isinstance(c.get("text"), str): if not f or offset > len(r[f]) or not isinstance(c.get("text"), str):
raise Refusal("correction.insert", "bad run or offset", target) raise Refusal("correction.insert", "bad run or offset", target)
if "printed" in r:
raise Refusal("correction.emend", "the run was emended; only another emend changes its text", target)
node["content"][run] = dict(r, **{f: r[f][:offset] + c["text"] + r[f][offset:]}) node["content"][run] = dict(r, **{f: r[f][:offset] + c["text"] + r[f][offset:]})
node["content"] = canonical_runs(node["content"]) node["content"] = canonical_runs(node["content"])
touched = [target] touched = [target]
@@ -269,12 +313,13 @@ def apply(state, c, hasher=hashing.PLAIN, store=None):
first.append(r) first.append(r)
elif pos >= k: elif pos >= k:
second.append(r) second.append(r)
elif r["kind"] == "text": elif r["kind"] == "text" and "printed" not in r:
cut = k - pos cut = k - pos
first.append(dict(r, text=r["text"][:cut])) first.append(dict(r, text=r["text"][:cut]))
second.append(dict(r, text=r["text"][cut:])) second.append(dict(r, text=r["text"][cut:]))
else: else:
raise Refusal("correction.split", "the offset falls inside a %s run" % r["kind"], target) where = "an emended run" if "printed" in r else "a %s run" % r["kind"]
raise Refusal("correction.split", "the offset falls inside " + where, target)
pos += ln pos += ln
_strip_edge(first, end=True) _strip_edge(first, end=True)
_strip_edge(second, end=False) _strip_edge(second, end=False)
@@ -375,6 +420,88 @@ def apply(state, c, hasher=hashing.PLAIN, store=None):
node["attrs"] = attrs node["attrs"] = attrs
touched = [target] if node["kind"] in LEAVES else [] touched = [target] if node["kind"] in LEAVES else []
elif op == "emend":
node = by_id[target]
if node["kind"] not in RUN_KINDS:
raise Refusal("correction.emend", "emend corrects the prose of runs; a %s is corrected by replace"
% node["kind"], target)
find, repl = c.get("find", ""), c["replace"]
if not find:
raise Refusal("correction.find", "find must not be empty")
runs = node["content"]
# `run`, when given, narrows the search to one run, as for replace.
mine = [i for i in range(len(runs)) if c.get("run", i) == i]
hits = [(i, j) for i in mine if runs[i]["kind"] in PROSE_RUNS for j in _occurrences(runs[i]["text"], find)]
if not hits:
sources = [runs[i]["source"] if runs[i]["kind"] == "math"
else doc.nodes[runs[i]["node"]]["content"]["source"]
for i in mine if runs[i]["kind"] in ("math", "equation")]
if any(find in s for s in sources) or ("run" in c and len(mine) == 1 and sources):
raise Refusal("correction.emend", "emend corrects prose, not math; a source is corrected by replace",
target)
if "run" not in c and find in hashing.plain_runs(runs, doc):
raise Refusal("correction.emend", "find crosses a run boundary; an emend stays inside one run", target)
if len(hits) != 1:
raise Refusal("correction.find", "find occurs %d times, not exactly once" % len(hits), target)
i, j = hits[0]
r = runs[i]
old = r["text"]
new = old[:j] + repl + old[j + len(find):]
if _edits_by_digit(old, new):
raise Refusal("correction.digits", "an emend changes no digit and edits beside none; a number is "
"corrected by replace, with crop evidence", target)
if r["kind"] == "text" and "printed" not in r:
# The found text becomes a run of its own, styled as before, that keeps what was printed.
style = {"style": r["style"]} if "style" in r else {}
mid = [dict({"kind": "text", "text": old[:j]}, **style),
dict({"kind": "text", "text": repl, "printed": find}, **style),
dict({"kind": "text", "text": old[j + len(find):]}, **style)]
node["content"] = runs[:i] + mid + runs[i + 1:]
else:
# Any other run, or one emended before, keeps its whole printed text, from the first emend.
node["content"][i] = dict(r, text=new, printed=r.get("printed", old))
node["content"] = canonical_runs(node["content"])
touched = [target]
elif op == "relink":
node = by_id[target]
run, refs = c.get("run"), c.get("refs")
if node["kind"] not in RUN_KINDS or not _is_index(run) or run >= len(node["content"]):
raise Refusal("correction.relink", "relink names a run of a node with runs", target)
r = node["content"][run]
if r["kind"] not in ("text", "cite") or (refs is None and r["kind"] != "cite"):
raise Refusal("correction.relink", "relink links a text or cite run, and unlinks a cite run", target)
if refs is not None and (not isinstance(refs, list) or not refs):
raise Refusal("correction.relink", "refs is a non-empty list of references, or null", target)
record["from"] = r["refs"] if r["kind"] == "cite" else None
# The printed form and anything an emend kept stay; a cite run takes no style (spec/03 §3.5).
keep = {"printed": r["printed"]} if "printed" in r else {}
if refs is None:
new_run = dict({"kind": "text", "text": r["text"]}, **keep)
else:
new_run = dict({"kind": "cite", "refs": list(refs), "text": r["text"]}, **keep)
node["content"] = canonical_runs(node["content"][:run] + [new_run] + node["content"][run + 1:])
touched = [target]
elif op == "resolve":
node = by_id[target]
if node["kind"] != "reference":
raise Refusal("correction.resolve", "resolve names a reference node, not a %s" % node["kind"], target)
res = c.get("resolution")
if not isinstance(res, dict) or res.get("type") not in ("doi", "document", "none"):
raise Refusal("correction.resolve", "the resolution is doi, document or none (spec/10)", target)
references = [dict(x) for x in state.references]
held = next((x for x in references if x["node"] == target), None)
# A reference without a record has not been resolved yet.
record["from"] = held["resolution"] if held else {"type": "none"}
if held:
held["resolution"] = res
else:
pos = {nid: k for k, nid in enumerate(doc.order)}
references.append(dict({"node": target}, **({"label": node["label"]} if "label" in node else {}),
resolution=res))
references.sort(key=lambda x: pos.get(x["node"], len(pos)))
else: else:
raise Refusal("correction.op", "unknown operation " + op) raise Refusal("correction.op", "unknown operation " + op)
@@ -407,7 +534,8 @@ def apply(state, c, hasher=hashing.PLAIN, store=None):
raise Refusal("correction.digits", "an undeclared digit change", nid) raise Refusal("correction.digits", "an undeclared digit change", nid)
for k in ("op", "target", "find", "replace", "run", "offset", "text", "parent", "place_after", "node", "with", for k in ("op", "target", "find", "replace", "run", "offset", "text", "parent", "place_after", "node", "with",
"joiner", "kind", "label", "attr", "value", "evidence", "author", "reason", "at"): "joiner", "kind", "label", "attr", "value", "refs", "resolution", "note", "evidence", "author", "reason",
"at"):
if k in c: if k in c:
record[k] = c[k] record[k] = c[k]
if op == "insert" and "node" in c: if op == "insert" and "node" in c:
@@ -429,7 +557,7 @@ def apply(state, c, hasher=hashing.PLAIN, store=None):
"rev": record["index"]}) "rev": record["index"]})
new_ids.sort(key=id_order) new_ids.sort(key=id_order)
new_state = State(state.key, [after_doc.nodes[x] for x in after_doc.preorder()], new_ids, new_state = State(state.key, [after_doc.nodes[x] for x in after_doc.preorder()], new_ids,
state.corrections + [record], {hasher.key: after_ix}) state.corrections + [record], {hasher.key: after_ix}, references)
return new_state, record return new_state, record
@@ -450,5 +578,11 @@ def _strip_edge(runs, end):
t = t[1:] t = t[1:]
if t: if t:
runs[i] = dict(r, text=t) runs[i] = dict(r, text=t)
# What an emended run printed loses the same edge whitespace, so the parts read as printed split alike.
if "printed" in r:
p = r["printed"]
while p and is_ws(p[-1 if end else 0]):
p = p[:-1] if end else p[1:]
runs[i] = dict(runs[i], printed=p) if p else {k: v for k, v in runs[i].items() if k != "printed"}
return return
del runs[i] del runs[i]
src/paperspec/hashing.py
+40 -1
View File
@@ -58,6 +58,28 @@ def plain_runs(runs, doc=None):
return "".join(out) return "".join(out)
def as_printed(runs):
"""The runs as the paper prints them (spec/03 §3.6): each run's `printed` in place of its text, and a text run
that carried one joined with an adjacent text run of the same style, as the run was before an emend split it.
Only an emend sets `printed` (spec/11 §11.1), so these are the runs with every emendation undone and every
other correction kept; runs without one are returned as they are.
"""
out, last = [], False
for r in runs:
mark = "printed" in r
if mark:
r = dict(r)
r["text"] = r.pop("printed")
if (mark or last) and out and r["kind"] == "text" and out[-1]["kind"] == "text" \
and out[-1].get("style") == r.get("style"):
out[-1] = dict(out[-1], text=out[-1]["text"] + r["text"])
else:
out.append(r)
last = mark
return out
def _strict_math(notation, source, complexes): def _strict_math(notation, source, complexes):
folded = fold_math(notation, source) folded = fold_math(notation, source)
if folded is None: if folded is None:
@@ -72,8 +94,9 @@ def _finish_strict(s):
def strict_runs_raw(runs, doc, complexes): def strict_runs_raw(runs, doc, complexes):
"""The strict projection of runs before NFKC and whitespace, over the runs as printed (spec/05 §5.3)."""
out = [] out = []
for r in runs: for r in as_printed(runs):
k = r["kind"] k = r["kind"]
if k == "text": if k == "text":
st = r.get("style", []) st = r.get("style", [])
@@ -148,6 +171,18 @@ def plain(doc, nid):
return " ".join(plain(doc, x) for x in doc.text_leaves(nid)) return " ".join(plain(doc, x) for x in doc.text_leaves(nid))
def printed(doc, nid):
"""The plain projection as printed (spec/03 §3.6): the plain projection of the runs as printed, so the text a
witness of the paper holds even where an emend corrected the authors. Equal to `plain` for a node no emend
touched; equations and verbatim text are never emended."""
n = doc.nodes[nid]
if n["kind"] in RUN_KINDS:
return plain_runs(as_printed(n["content"]), doc)
if n["kind"] in MATH_KINDS or n["kind"] in VERBATIM_KINDS:
return plain(doc, nid)
return " ".join(printed(doc, x) for x in doc.text_leaves(nid))
def strict(doc, nid): def strict(doc, nid):
"""The strict canonical projection and the node's complex math keys, in order.""" """The strict canonical projection and the node's complex math keys, in order."""
n = doc.nodes[nid] n = doc.nodes[nid]
@@ -174,10 +209,14 @@ def lossy(doc, nid):
def _runs_for_own(runs): def _runs_for_own(runs):
"""The runs as own content (spec/05 §5.2): an equation run without the ID it names, a math run without its
`form`, which says how the source is spelled and adds nothing to it (spec/03 §3.7)."""
out = [] out = []
for r in runs: for r in runs:
if r["kind"] == "equation": if r["kind"] == "equation":
out.append({"kind": "equation"}) out.append({"kind": "equation"})
elif r["kind"] == "math" and "form" in r:
out.append({k: v for k, v in r.items() if k != "form"})
else: else:
out.append(r) out.append(r)
return out return out
src/paperspec/log.py
+9 -2
View File
@@ -63,15 +63,22 @@ def sign_checkpoint(origin, entries, signers):
def parse_checkpoint(text): def parse_checkpoint(text):
"""The origin, size, root (hex) and [(name, key hash, signature)] of a note, or None when malformed.""" """The origin, size, root (hex) and [(name, key hash, signature)] of a note, or None when malformed.
Every line ends in one U+000A and nothing else (spec/23 §23.2): a note is split on U+000A alone, never with
`str.splitlines`, which also ends a line at U+000B, U+000C, U+001C to U+001E, U+0085, U+2028 and U+2029 and
would accept a note whose last line break was replaced by one of them.
"""
try: try:
body, _, sigs = text.partition("\n\n") body, _, sigs = text.partition("\n\n")
origin, size, root_b64 = body.split("\n") origin, size, root_b64 = body.split("\n")
rt = _b64(root_b64) rt = _b64(root_b64)
if rt is None or len(rt) != 32 or not size.isdigit() or str(int(size)) != size or not origin: if rt is None or len(rt) != 32 or not size.isdigit() or str(int(size)) != size or not origin:
return None return None
if sigs and not sigs.endswith("\n"):
return None
out = [] out = []
for ln in sigs.splitlines(): for ln in sigs.split("\n")[:-1]:
if not ln.startswith("— "): if not ln.startswith("— "):
return None return None
_, name, blob = ln.split(" ", 2) _, name, blob = ln.split(" ", 2)
src/paperspec/package.py
+96 -5
View File
@@ -11,9 +11,11 @@ import os
import posixpath import posixpath
import re import re
import stat import stat
import struct
import tempfile import tempfile
import unicodedata import unicodedata
import zipfile import zipfile
import zlib
from . import crypto, hashing, jcs, keys as K, receipts as R, vrf from . import crypto, hashing, jcs, keys as K, receipts as R, vrf
from .anchors import ADDRESS_RE from .anchors import ADDRESS_RE
@@ -42,6 +44,11 @@ JSON = {
"roster.json": "roster.schema.json", "roster.json": "roster.schema.json",
} }
REQUIRED = ["document.json", "nodes.jsonl", "witnesses.json", "provenance.json"] REQUIRED = ["document.json", "nodes.jsonl", "witnesses.json", "provenance.json"]
# Derived files (spec/12 §12.7): checked against their schemas and the formulas they name, never recomputed.
DERIVED = {
"derived/math.jsonl": "math.schema.json",
"derived/mathml.jsonl": "mathml.schema.json",
}
def line(rec): def line(rec):
@@ -253,6 +260,13 @@ def _validate(root, store, key, rep, limit=SIZE_LIMIT):
rep.add("package.file.empty", p, "an optional JSONL file with no records is omitted") rep.add("package.file.empty", p, "an optional JSONL file with no records is omitted")
else: else:
data[p] = [] data[p] = []
for p, sch in DERIVED.items():
data[p] = read_jsonl(os.path.join(root, p)) if p in listed else []
for i, rec in enumerate(data[p], 1):
for e in schema_validate(store, sch, rec):
rep.add("package.schema", "%s:%d" % (p, i), e)
if p in listed and not data[p]:
rep.add("package.file.empty", p, "an optional JSONL file with no records is omitted")
if not rep: if not rep:
return return
@@ -290,6 +304,8 @@ def _validate(root, store, key, rep, limit=SIZE_LIMIT):
rep.add("doc.policy.withheld", "nodes.jsonl", "a checks-only package withholds every leaf's content") rep.add("doc.policy.withheld", "nodes.jsonl", "a checks-only package withholds every leaf's content")
if scheme == "plain": if scheme == "plain":
rep.add("doc.policy.keyed", "document.json", "a checks-only package carries keyed hashes") rep.add("doc.policy.keyed", "document.json", "a checks-only package carries keyed hashes")
for p in sorted(x for x in listed if x.startswith("derived/")):
rep.add("doc.policy.withheld", p, "a checks-only package carries no derived files, which read the text")
if scheme == "hmac-team" and key is not None and document["hashing"]["key_id"] != crypto.key_id(key): if scheme == "hmac-team" and key is not None and document["hashing"]["key_id"] != crypto.key_id(key):
rep.add("doc.hash.key", "document.json", "key_id does not match the key supplied") rep.add("doc.hash.key", "document.json", "key_id does not match the key supplied")
@@ -302,6 +318,8 @@ def _validate(root, store, key, rep, limit=SIZE_LIMIT):
leaves_n = sum(1 for n in nodes if n["kind"] in LEAVES) leaves_n = sum(1 for n in nodes if n["kind"] in LEAVES)
if document.get("counts") != {"nodes": len(nodes), "leaves": leaves_n}: if document.get("counts") != {"nodes": len(nodes), "leaves": leaves_n}:
rep.add("doc.counts", "document.json", "counts do not recompute") rep.add("doc.counts", "document.json", "counts do not recompute")
for p in DERIVED:
_check_derived(doc, p, data[p], rep)
# Hashes. # Hashes.
hasher = hashing.PLAIN if scheme == "plain" else (hashing.Hasher(key) if key else None) hasher = hashing.PLAIN if scheme == "plain" else (hashing.Hasher(key) if key else None)
@@ -394,6 +412,14 @@ def _validate(root, store, key, rep, limit=SIZE_LIMIT):
for i, ref in enumerate(data["references.jsonl"], 1): for i, ref in enumerate(data["references.jsonl"], 1):
if kinds.get(ref["node"]) != "reference": if kinds.get(ref["node"]) != "reference":
rep.add("doc.reference.node", "references.jsonl:%d" % i, "names no reference node") rep.add("doc.reference.node", "references.jsonl:%d" % i, "names no reference node")
# The last resolve of each live reference ends at its resolution in references.jsonl (spec/11 §11.3), which is
# none for a reference without a record.
held = {r["node"]: r["resolution"] for r in data["references.jsonl"]}
resolved = {c["target"]: c for c in cors if c["op"] == "resolve"}
for nid, c in resolved.items():
if nid in live and held.get(nid, {"type": "none"}) != c["resolution"]:
rep.add("doc.correction.resolution", c["id"], "the last resolve of %s is not its resolution in "
"references.jsonl" % nid)
rorder = sorted(data["references.jsonl"], key=lambda r: pos.get(r["node"], -1)) rorder = sorted(data["references.jsonl"], key=lambda r: pos.get(r["node"], -1))
if rorder != data["references.jsonl"]: if rorder != data["references.jsonl"]:
rep.add("doc.order.references", "references.jsonl", "references follow document order") rep.add("doc.order.references", "references.jsonl", "references follow document order")
@@ -401,6 +427,9 @@ def _validate(root, store, key, rep, limit=SIZE_LIMIT):
sorder = sorted(data["suspects.jsonl"], key=lambda x: x["id"]) sorder = sorted(data["suspects.jsonl"], key=lambda x: x["id"])
if sorder != data["suspects.jsonl"]: if sorder != data["suspects.jsonl"]:
rep.add("doc.order.suspects", "suspects.jsonl", "suspects sort by ID") rep.add("doc.order.suspects", "suspects.jsonl", "suspects sort by ID")
for s in data["suspects.jsonl"]:
if "witness" in s and s["witness"] not in wits:
rep.add("doc.suspect.witness", s["id"], "a suspect's witness is one of the package's witnesses")
# Receipts and fidelity. # Receipts and fidelity.
roster = {k["id"]: k for k in data.get("roster.json", {}).get("keys", [])} roster = {k["id"]: k for k in data.get("roster.json", {}).get("keys", [])}
@@ -467,10 +496,66 @@ def _validate(root, store, key, rep, limit=SIZE_LIMIT):
if not withheld and (roster or not recs): if not withheld and (roster or not recs):
# Without a roster no receipt can be judged valid, so a package that carries receipts but no roster is # Without a roster no receipt can be judged valid, so a package that carries receipts but no roster is
# checked against its store instead; one with neither reports the leaves' own levels. # checked against its store instead; one with neither reports the leaves' own levels.
if R.distribution(R.effective_levels(nodes, vals, recs, roster)) != document["fidelity"]: if R.distribution(R.effective_levels(nodes, vals, recs, roster, cors)) != document["fidelity"]:
rep.add("doc.fidelity.report", "document.json", "the fidelity distribution does not recompute") rep.add("doc.fidelity.report", "document.json", "the fidelity distribution does not recompute")
def derived_formula(doc, node, run):
"""The formula a derived line names (spec/12 §12.7), as `{notation, source, form?}`, or None when it names none:
the content of an equation or equation line when `run` is null, else the math run at index `run` of a node
with runs."""
n = doc.nodes.get(node)
if n is None or n.get("withheld"):
return None
if run is None:
return n.get("content") if n["kind"] in MATH_KINDS else None
runs = n.get("content") if n["kind"] in RUN_KINDS else None
if not isinstance(runs, list) or run >= len(runs) or runs[run]["kind"] != "math":
return None
return runs[run]
def tree_facts(tree):
"""The `sym` keys of a scienting-math/1 tree in document order, and whether it has an `opaque` node (spec/09
§9.4). A node is an array whose first item is its kind; an array whose first item is not a string is a group of
parts. Walked without recursion, so a deep tree cannot exhaust the stack."""
keys, opaque, stack = [], False, [tree]
while stack:
x = stack.pop()
if not isinstance(x, list) or not x:
continue
if isinstance(x[0], str):
if x[0] == "opaque":
opaque = True
elif x[0] == "sym" and len(x) > 1 and isinstance(x[1], str):
keys.append(x[1])
stack.extend(reversed(x[1:]))
else:
stack.extend(reversed(x))
return keys, opaque
def _check_derived(doc, path, lines, rep):
"""Each line of a derived file names a formula of the package; a math line's formula is TeX, and its `complete`
and `symbols` agree with its tree (spec/12 §12.7, spec/09 §9.4). The tree itself is the tool's and not
recomputed."""
for i, rec in enumerate(lines, 1):
where = "%s:%d" % (path, i)
formula = derived_formula(doc, rec["node"], rec["run"])
if formula is None:
rep.add("doc.derived.node", where, "names no equation, equation line or math run of the package")
continue
if path != "derived/math.jsonl":
continue
if formula["notation"] != "tex":
rep.add("doc.derived.math", where, "the math form reads TeX, and the formula is %s" % formula["notation"])
keys, opaque = tree_facts(rec["tree"])
if rec["complete"] == opaque:
rep.add("doc.derived.math", where, "complete is true exactly when the tree has no opaque node")
if sorted(set(keys)) != sorted(rec["symbols"]):
rep.add("doc.derived.math", where, "symbols are not the sym keys of the tree, each once")
def read_state(root): def read_state(root):
"""The current state of the package at `root`, for the gate (spec/11 §11.4): nodes without their values, """The current state of the package at `root`, for the gate (spec/11 §11.4): nodes without their values,
the ID table and the corrections applied so far. Validate the package first.""" the ID table and the corrections applied so far. Validate the package first."""
@@ -482,7 +567,8 @@ def read_state(root):
values = ("text_hash", "match_hash", "align_key", "own_hash", "node_hash") values = ("text_hash", "match_hash", "align_key", "own_hash", "node_hash")
nodes = [{k: v for k, v in n.items() if k not in values} for n in jl("nodes.jsonl")] nodes = [{k: v for k, v in n.items() if k not in values} for n in jl("nodes.jsonl")]
return State(read_json(os.path.join(root, "document.json"))["key"], nodes, jl("ids.jsonl"), jl("corrections.jsonl")) return State(read_json(os.path.join(root, "document.json"))["key"], nodes, jl("ids.jsonl"), jl("corrections.jsonl"),
references=jl("references.jsonl"))
def n_root_hash(nodes, doc): def n_root_hash(nodes, doc):
@@ -502,6 +588,11 @@ ZIP_LIMIT = 1 << 30
_ZIP_DATE = (1980, 1, 1, 0, 0, 0) _ZIP_DATE = (1980, 1, 1, 0, 0, 0)
_FLAG_ENCRYPTED = 0x1 _FLAG_ENCRYPTED = 0x1
_FLAG_UTF8 = 0x800 _FLAG_UTF8 = 0x800
# What `zipfile` raises on an archive it cannot read: a damaged structure (BadZipFile, struct.error, EOFError), a
# version or feature it does not implement (NotImplementedError, as for "zip file version 9.8"), a name that is not
# the UTF-8 its flag claims (UnicodeDecodeError, a ValueError), or deflated bytes that do not inflate (zlib.error).
_ZIP_ERRORS = (zipfile.BadZipFile, NotImplementedError, EOFError, ValueError, RuntimeError, OSError, struct.error,
zlib.error)
def is_zip(path): def is_zip(path):
@@ -577,8 +668,8 @@ def read_zip(path, dest, limit=ZIP_LIMIT):
""" """
try: try:
zf = zipfile.ZipFile(path) zf = zipfile.ZipFile(path)
except (zipfile.BadZipFile, OSError) as e: except _ZIP_ERRORS as e:
raise Refusal("package.zip.entry", "not a zip archive: %s" % e) raise Refusal("package.zip.entry", "not a zip archive this reader can open: %s" % e)
with zf: with zf:
folder = _zip_entries(zf, limit) folder = _zip_entries(zf, limit)
root = os.path.join(dest, folder) root = os.path.join(dest, folder)
@@ -602,7 +693,7 @@ def read_zip(path, dest, limit=ZIP_LIMIT):
if n > zi.file_size or written > limit: if n > zi.file_size or written > limit:
raise Refusal("package.zip.size", "%s: holds more than its header declares" % zi.filename) raise Refusal("package.zip.size", "%s: holds more than its header declares" % zi.filename)
dst.write(chunk) dst.write(chunk)
except (zipfile.BadZipFile, RuntimeError, OSError) as e: except _ZIP_ERRORS as e:
raise Refusal("package.zip.entry", "%s: %s" % (zi.filename, e)) raise Refusal("package.zip.entry", "%s: %s" % (zi.filename, e))
return root return root
src/paperspec/receipts.py
+27 -3
View File
@@ -256,13 +256,37 @@ def receipt_status(rec, records, roster):
return statuses(records, roster).get(rec.get("id"), "invalid") return statuses(records, roster).get(rec.get("id"), "invalid")
def effective_levels(nodes, values, records, roster): def carried_hashes(values, corrections):
"""Every leaf's level: its recorded level, raised by valid receipts that list it at its current hash. """For each node, the node_hashes at which a receipt still counts for it (spec/08 §8.2): its current one, and
each earlier one that only emend, relink and resolve corrections of the node itself have changed since.
Walking the corrections back from the last, one that names the node in any way ends its chain, unless it is
one of those three, targets the node, and ends at the hash reached so far; then its `before` is reached.
"""
from .corrections import KEEPS_LEVEL
out = {nid: {v["node_hash"]} for nid, v in values.items()}
reached = {nid: v["node_hash"] for nid, v in values.items()}
for c in reversed(corrections):
named = ({c.get("target")} | set(c.get("touched", [])) | set(c.get("path") or []) | set(c.get("to_path") or [])
| set(c.get("created", [])) | set(c.get("retired", [])))
for nid in named & set(reached):
if c["op"] in KEEPS_LEVEL and c.get("target") == nid and c.get("after") == reached[nid] and c.get("before"):
reached[nid] = c["before"]
out[nid].add(c["before"])
else:
del reached[nid]
return out
def effective_levels(nodes, values, records, roster, corrections=()):
"""Every leaf's level: its recorded level, raised by valid receipts that list it at its current hash, or at a
hash only emend, relink and resolve corrections have changed since, given the document's `corrections`.
A spot-check's `qa:` receipt does not raise the nodes it found in error (spec/08 §8.2). A spot-check's `qa:` receipt does not raise the nodes it found in error (spec/08 §8.2).
""" """
level = {n["id"]: n.get("fidelity") for n in nodes if "fidelity" in n} level = {n["id"]: n.get("fidelity") for n in nodes if "fidelity" in n}
status = statuses(records, roster) status = statuses(records, roster)
carried = carried_hashes(values, corrections)
for rec in records: for rec in records:
if rec["format"] != "scienting-receipt/1": if rec["format"] != "scienting-receipt/1":
continue continue
@@ -270,7 +294,7 @@ def effective_levels(nodes, values, records, roster):
continue continue
wrong = {e["node"] for e in (rec.get("qa") or {}).get("errors", [])} wrong = {e["node"] for e in (rec.get("qa") or {}).get("errors", [])}
for nid, h in rec.get("nodes", []): for nid, h in rec.get("nodes", []):
if nid in level and nid not in wrong and values[nid]["node_hash"] == h: if nid in level and nid not in wrong and h in carried[nid]:
if LEVELS.index(rec["level"]) > LEVELS.index(level[nid]): if LEVELS.index(rec["level"]) > LEVELS.index(level[nid]):
level[nid] = rec["level"] level[nid] = rec["level"]
return level return level
src/paperspec/traces.py
+268
View File
@@ -0,0 +1,268 @@
"""Traced citations: where a citation leads, and whether it still does.
A trace is an annotation whose body has the kind `trace`: its target is a sentence of the citing document, and its
body names the reference the sentence cites and a passage of the cited document. Non-normative; see
spec/18-layers.md §18.4 for the body and spec/22-conformance.md §22.4 for the rules `layer.trace.source` and
`layer.trace.target`.
`resolve` judges a trace against the current states of both documents, `problems` checks a trace when it is
written, and `disputes` finds the replies that dispute it. Nothing here needs more than the standard library.
"""
import re
from . import anchors as AN
from . import hashing
from .model import RUN_KINDS
RESULTS = ("current", "changed", "stale", "unlinked", "misresolved")
DOC_TAG = re.compile(r"^tag:scient\.ing,2026:doc:(?P<store>[a-z0-9][a-z0-9._-]*)/(?P<key>[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}"
r"(?:\.[a-z0-9][a-z0-9-]*)*)$")
# The statuses of a reply that still stands: a withdrawn or rejected reply disputes nothing.
LIVE = ("proposed", "confirmed")
def doc_tag(store, key):
"""The tag of a document of a store, as `source` and a target's source write it."""
return "tag:scient.ing,2026:doc:%s/%s" % (store, key)
def _anchors(holder):
"""The anchors of the `AnchorSelector`s of a body or a target, in order."""
sel = holder.get("selector") if isinstance(holder, dict) else None
return [s["anchor"] for s in sel or [] if isinstance(s, dict) and s.get("type") == "AnchorSelector"]
def _bare(doc):
"""A document key without its store: `smithlab/hale2026harmonic-damb` and `hale2026harmonic-damb` agree."""
return doc.rsplit("/", 1)[-1]
def _work(doc):
"""The work key of a document key: the key before the first dot of its version suffix (spec/02 §2.4)."""
return _bare(doc).split(".", 1)[0]
def _live(state, nid):
"""The live node an ID leads to, following aliases and retirements; None when it was deleted or never was."""
return AN._follow(state, nid)[0]
def _record(state, nid):
"""The resolution of a reference: the record of references.jsonl for the node, else `none` (spec/10)."""
for r in state.references:
if r["node"] == nid:
return r["resolution"]
return {"type": "none"}
def _cites(state, nid, reference):
"""Whether the leaf `nid` has a cite run that names the reference `reference`, however either was renamed."""
n = {x["id"]: x for x in state.nodes}.get(nid)
if n is None or n["kind"] not in RUN_KINDS:
return False
want = _live(state, reference)
return want is not None and any(_live(state, x) == want for r in n["content"] if r["kind"] == "cite"
for x in r["refs"])
def _agrees(resolution, cited_key, same_as):
"""Whether a reference's resolution still names the cited work.
`none` names nothing, so it contradicts nothing. A `document` resolution agrees when its key is the cited
document's, another version of the same work (the key before the first dot), or one of `same_as`; a `doi`
resolution agrees when `doi:` and the DOI, in lower case, is one of `same_as`.
"""
known = {x.lower() for x in same_as}
kind = resolution.get("type")
if kind == "document":
doc = resolution["doc"]
return _work(doc) == _work(cited_key) or _bare(doc) in known or doc in known
if kind == "doi":
return "doi:" + resolution["doi"].lower() in known
return True
def _passage_cites(state, node, span):
"""The citations inside a passage, so a chain of 'as previously reported' can be followed one trace at a time.
One item for each `cite` run of the passage's leaves (those that overlap `span`, for a span of a leaf), in
document order: the leaf, the run's index and span in the leaf's plain projection, its printed text, and each
reference it names with that reference's resolution in the cited document.
"""
doc = state.doc()
leaves = doc.leaves(node)
out = []
for leaf in leaves:
n = doc.nodes[leaf]
if n["kind"] not in RUN_KINDS:
continue
pos = 0
for i, r in enumerate(n["content"]):
end = pos + len(hashing.plain_runs([r], doc))
if r["kind"] == "cite" and (span is None or (pos < span[1] and end > span[0])):
out.append({"node": leaf, "run": i, "span": [pos, end], "text": r["text"],
"refs": [{"node": x, "resolution": _record(state, _live(state, x) or x)} for x in r["refs"]]})
pos = end
return out
def _sides(annotation, citing_state, cited_state):
"""The citing anchor and the passage anchor of a trace, after checking that the states are those documents'."""
body = annotation.get("body", {})
if body.get("kind") != "trace":
raise ValueError("the annotation is not a trace")
mine, theirs = _anchors(annotation["target"]), _anchors(body)
if not mine or not theirs:
raise ValueError("a trace needs an AnchorSelector on its target and one in its body")
if mine[0]["doc"] != citing_state.key:
raise ValueError("citing_state is %s, but the trace's target is on %s" % (citing_state.key, mine[0]["doc"]))
if theirs[0]["doc"] != cited_state.key:
raise ValueError("cited_state is %s, but the trace's passage is in %s" % (cited_state.key, theirs[0]["doc"]))
return mine[0], theirs[0]
def resolve(annotation, citing_state, cited_state, same_as=()):
"""Judge a trace against the current state of the citing document and of the cited document.
`annotation` is the trace, an annotation with a `trace` body; `citing_state` and `cited_state` are the
`corrections.State` of the document its target is on and of the document its body's `source` names (for a
package, `package.read_state`); `same_as` is everything the cited work is known by, the `work.same_as` of its
`document.json` (`doi:…`) and the keys of its other versions, and decides whether a `doi` resolution of the
reference still names the cited work. Both anchors are resolved by spec/06 §6.5.
Returns a dict. `result` is the first that applies of:
- `stale`: an anchor does not resolve; `side` is `citing`, `cited` or `both`.
- `unlinked`: the sentence no longer cites the reference (a `relink` removed the citation, or the reference
was deleted).
- `misresolved`: the reference now resolves to a `doi` or a `document` that is not the cited work (a `resolve`).
- `changed`: both anchors resolve, and at least one is not `current`, so the passage or the sentence is not as
it was when the trace was made; a reader shows the trace with its new address and offers a review.
- `current`: both anchors are `current` and the citation links the reference to the cited work.
`citing` and `cited` are the two anchor resolutions (spec/06 §6.5). `open` is the current address of the
cited passage, `store/key#node@pin[start:end]`, whenever its anchor still resolves, whatever `result` says;
`None` when it is stale. `cites` lists the citations inside the passage, `[]` when it is stale: one item
`{node, run, span, text, refs}` for each `cite` run of the passage's leaves, in document order, where `node`
and `run` find the run in the cited document, `span` is its place in the leaf's plain projection, `text` is
what the paper prints, and `refs` lists `{node, resolution}` for each reference it names, so a chain of "as
previously reported" is followed by looking for a trace on that leaf and reference. `reference` is the live
node of the reference in the citing document, or None when it was deleted, and `resolution` the one its
record holds. `reason` says why for every result but `current`.
The annotation's `status` and its replies are not read here: a rejected candidate resolves like any other
trace, and `disputed` says whether a reply disputes it. A `ValueError` means the call is wrong, not the trace:
the annotation has no trace body, or a state is not that of the document the annotation names.
"""
body = annotation["body"]
mine, theirs = _sides(annotation, citing_state, cited_state)
citing, cited = AN.resolve(citing_state, mine), AN.resolve(cited_state, theirs)
reference = _live(citing_state, body["reference"])
resolution = _record(citing_state, reference) if reference else _record(citing_state, body["reference"])
out = {"citing": citing, "cited": cited, "reference": reference, "resolution": resolution}
if cited["result"] != "stale":
m = DOC_TAG.match(body["source"])
store = m.group("store") if m else theirs.get("store")
out["open"] = AN.format_address(store, cited_state.key, cited["node"], cited["node_hash"][:6], cited.get("span"))
out["cites"] = _passage_cites(cited_state, cited["node"], cited.get("span"))
else:
out["open"], out["cites"] = None, []
stale = [s for s, r in (("citing", citing), ("cited", cited)) if r["result"] == "stale"]
if stale:
out.update(result="stale", side="both" if len(stale) == 2 else stale[0],
reason="; ".join("the %s anchor is stale: %s" % (s, (citing if s == "citing" else cited).get("reason", ""))
for s in stale))
elif reference is None or not _cites(citing_state, citing["node"], reference):
out.update(result="unlinked", reason="the sentence no longer cites the reference")
elif not _agrees(resolution, cited_state.key, same_as):
out.update(result="misresolved", reason="the reference resolves to %s, not to the cited work"
% (resolution.get("doc") or "doi:" + resolution.get("doi", "")))
elif citing["result"] != "current" or cited["result"] != "current":
out.update(result="changed", reason="; ".join("the %s anchor is %s" % (s, r["result"]) for s, r in (
("citing", citing), ("cited", cited)) if r["result"] != "current"))
else:
out["result"] = "current"
return out
def problems(annotation, citing_state=None, cited_state=None):
"""The rules a trace breaks, as `[{"rule", "message"}]` in the order checked; empty when it breaks none.
Without states the trace is checked alone; with them, against the documents it was made on, which is how a
producer checks a trace it writes or imports. A trace later made `unlinked` by a correction breaks
`layer.trace.target` here, since the sentence it names no longer cites the reference; `resolve` is the judgment
of a trace over time. `layer.trace.source` covers the body, `layer.trace.target` the target and the reference.
"""
found = []
def add(rule, message):
found.append({"rule": rule, "message": message})
body, target = annotation["body"], annotation["target"]
mine, passage = _anchors(target), _anchors(body)
if citing_state is not None and mine and mine[0]["doc"] != citing_state.key:
raise ValueError("citing_state is not the document the trace's target is on")
if cited_state is not None and passage and passage[0]["doc"] != cited_state.key:
raise ValueError("cited_state is not the document the trace's passage is in")
# The source: a document of a store, named by its tag, and the passage inside it.
m = DOC_TAG.match(body.get("source", ""))
if not m:
add("layer.trace.source", "the source is not a document tag")
if not passage:
add("layer.trace.source", "the body's selector has no AnchorSelector")
elif m:
for a in passage:
if a["doc"] != m.group("key") or ("store" in a and a["store"] != m.group("store")):
add("layer.trace.source", "an anchor of the selector is in %s, not in the source %s/%s"
% (a["doc"], m.group("store"), m.group("key")))
break
citing_doc = target.get("source", "") if isinstance(target, dict) else ""
if citing_doc == body["source"]:
add("layer.trace.source", "the source is the citing document itself; a trace leads to another paper")
quote = passage[0].get("quote", {}) if passage else {}
for s in body.get("selector", []) if passage else []:
if isinstance(s, dict) and s.get("type") == "TextQuoteSelector":
if "exact" not in quote or any(s.get(k, "") != quote.get(k, "") for k in ("exact", "prefix", "suffix")):
add("layer.trace.source", "the TextQuoteSelector does not repeat the anchor's plain quote")
break
if cited_state is not None and passage:
node = passage[0]["node"]
if node not in {n["id"] for n in cited_state.nodes} | {e["id"] for e in cited_state.ids}:
add("layer.trace.source", "the passage's node %s is not in the cited document" % node)
# The target: a leaf with runs of the citing document, which cites the reference.
if not mine:
add("layer.trace.target", "the target is not a document selected by an AnchorSelector")
else:
a = mine[0]
kind = a.get("kind")
if (kind is not None and kind not in RUN_KINDS) or "quote" not in a:
add("layer.trace.target", "the target anchor is not on a leaf with runs, whole or in part")
elif citing_state is not None:
live = _live(citing_state, a["node"])
nodes = {n["id"]: n for n in citing_state.nodes}
ref = nodes.get(_live(citing_state, body["reference"]) or "")
if ref is None or ref["kind"] != "reference":
add("layer.trace.target", "the reference %s is not a reference node of the citing document" % body["reference"])
elif live is None or not _cites(citing_state, live, body["reference"]):
add("layer.trace.target", "the sentence does not cite the reference %s" % body["reference"])
return found
def replies(annotation_id, annotations):
"""The annotations among `annotations` that reply to the annotation `annotation_id`: those whose target is it."""
return [a for a in annotations if isinstance(a.get("target"), dict) and "selector" not in a["target"]
and a["target"].get("source") == annotation_id]
def disputes(annotation_id, annotations):
"""The replies that dispute a trace: those with the motivation `assessing` that are `proposed` or `confirmed`."""
return [a for a in replies(annotation_id, annotations) if a.get("motivation") == "assessing"
and a.get("status") in LIVE]
def disputed(annotation_id, annotations):
"""Whether a trace is disputed; a dispute is computed from the replies, never stored on the trace."""
return bool(disputes(annotation_id, annotations))
src/paperspec/tree.py
+9 -2
View File
@@ -24,10 +24,16 @@ class Refusal(Exception):
def canonical_runs(runs): def canonical_runs(runs):
"""Drop empty text runs and join adjacent text runs with the same styles.""" """Drop empty text runs and join adjacent text runs with the same styles.
A run that keeps what was printed (`printed`, spec/03 §3.5) is never joined, so the emendation stays where it
was made; a `printed` equal to the run's text records nothing and is dropped.
"""
out = [] out = []
for r in runs: for r in runs:
r = dict(r) r = dict(r)
if "printed" in r and r["printed"] == r.get("text"):
del r["printed"]
if r["kind"] == "text": if r["kind"] == "text":
if "style" in r: if "style" in r:
r["style"] = sorted(set(r["style"])) r["style"] = sorted(set(r["style"]))
@@ -35,7 +41,8 @@ def canonical_runs(runs):
del r["style"] del r["style"]
if r["text"] == "": if r["text"] == "":
continue continue
if out and out[-1]["kind"] == "text" and out[-1].get("style") == r.get("style"): if (out and out[-1]["kind"] == "text" and out[-1].get("style") == r.get("style")
and "printed" not in out[-1] and "printed" not in r):
out[-1] = dict(out[-1], text=out[-1]["text"] + r["text"]) out[-1] = dict(out[-1], text=out[-1]["text"] + r["text"])
continue continue
out.append(r) out.append(r)
tools/README.md
+2 -71
View File
@@ -4,7 +4,6 @@ This page is for anyone who wants to run the repository's checks, rebuild its ex
Newcomers get each term defined where it first appears. Newcomers get each term defined where it first appears.
Implementers get the commands and the traps. Implementers get the commands and the traps.
<<<<<<< HEAD
These commands are the repository's reference implementation. These commands are the repository's reference implementation.
They apply every rule of the specification, so the rest of the repository can be checked against them. They apply every rule of the specification, so the rest of the repository can be checked against them.
Two build and check the repository, one validates packages and proposed corrections, one loads packages into SQLite, a database stored in a single file, one stresses the reference implementation with random inputs, and one builds the website. Two build and check the repository, one validates packages and proposed corrections, one loads packages into SQLite, a database stored in a single file, one stresses the reference implementation with random inputs, and one builds the website.
@@ -76,8 +75,8 @@ A node's five values depend only on its own subtree, so after a correction the o
`paperspec check` compares the result after every correction with a slow computation that projects each node from its whole subtree. `paperspec check` compares the result after every correction with a slow computation that projects each node from its whole subtree.
Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec`, from the Gitea package registry or as a git pin ([deploy/](../deploy/README.md#install-from-the-registry)), and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code. Other repositories of the project do not reimplement the specification: they install this repository as the Python package `paperspec`, from the Gitea package registry or as a git pin ([deploy/](../deploy/README.md#install-from-the-registry)), and import the oracle from it, so that every key, identifier, canonical JSON string, package check and database form comes from one piece of code.
The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema` and `rights`, with the schemas, SQL and vectors as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike. The library surface they use is `keys`, `text`, `jcs`, `package`, `db`, `schema`, `rights` and `traces`, with the schemas, SQL and vectors as package data: `paperspec.resources.path("sql")` finds a folder in an installed wheel and in an editable install alike.
The vectors travel with the package, but `resolve.jsonl`, `grading.jsonl` and `targets.jsonl` name a fixture package, and the fixtures stay in the repository. The vectors travel with the package, but `resolve.jsonl`, `grading.jsonl`, `targets.jsonl` and `traces.jsonl` name a fixture package, and the fixtures stay in the repository.
The package needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other. The package needs Python 3.14, the only version with the Unicode 16.0 character data the specification pins, and refuses to import on any other.
Importing any of its modules needs only the standard library. Importing any of its modules needs only the standard library.
Schema validation (`schema`, and with it `package.validate` and `corrections.gate`) needs `paperspec[schema]` (`jsonschema` and `referencing`), and HKDF and Ed25519 in `crypto`, which the signature and receipt checks use, need `paperspec[crypto]`. Schema validation (`schema`, and with it `package.validate` and `corrections.gate`) needs `paperspec[schema]` (`jsonschema` and `referencing`), and HKDF and Ed25519 in `crypto`, which the signature and receipt checks use, need `paperspec[crypto]`.
@@ -85,71 +84,3 @@ Schema validation (`schema`, and with it `package.validate` and `corrections.gat
The package version comes from git tags through setuptools-scm, and `pixi run -e dev dist` builds the sdist and the wheel into `dist/`. The package version comes from git tags through setuptools-scm, and `pixi run -e dev dist` builds the sdist and the wheel into `dist/`.
When `scienting-core` exists it takes this role behind the same vectors, and the import changes without any computed value changing. When `scienting-core` exists it takes this role behind the same vectors, and the import changes without any computed value changing.
Independent implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md). Independent implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).
||||||| parent of 4fa326d (First implementation (#1))
=======
These programs are the repository's reference implementation.
They apply every rule of the specification, so the rest of the repository can be checked against them.
Two build and check the repository, one validates packages and proposed corrections, and one loads packages into SQLite, a database stored in a single file.
They need Python 3.14 and two packages, `jsonschema` and `cryptography`, and are licensed Apache-2.0.
A package is the folder of files that holds one document, such as `fixtures/packages/hale2026harmonic`.
A vector is a test case: an input and the exact output every implementation must compute (see [the vectors](../vectors/README.md)).
A fixture is an example file or package under `fixtures/`.
| Program | Does |
| --- | --- |
| `check.py` | Checks everything: schemas, records, packages and the zipped package, the broken packages, every vector, a byte-for-byte rebuild, the SQL (a database query language) round trip, the counts the prose states, and the prose. `--quick` skips the rebuild. |
| `build.py` | Builds `fixtures/` and `vectors/` from the hand-written sources in `fixtures/src/` and the cases it defines. `--out DIR` builds into another folder. |
| `validate.py` | Validates packages, as folders or as zipped packages (`.sct.zip`, unpacked under the rules of [spec/12 §12.2](../spec/12-package.md#122-paths)): `python3 tools/validate.py PACKAGE [PACKAGE ...]`. With `--correction PROPOSALS` it runs proposed corrections through the gate against one package's current state, and prints the record each would append or the rule it breaks, without writing anything ([spec/11 §11.4](../spec/11-corrections.md#114-the-gate)). |
| `load.py` | Validates packages, folders or zipped, then loads them into a SQLite database with the tables of `sql/`: `python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]`. With `--bundle --built-from REV` it builds a new bundle instead, with the full-text search index ([spec/13 §13.5](../spec/13-database.md#135-the-bundle)). |
| `site.py` | Builds the website into `_site/`: every file of the repository at its own path, and each Markdown file rendered as a page beside it, with a vocabulary page, folder listings and a search index. It uses only the standard library. `pixi run check` builds it into a temporary folder and checks every link ([deploy/](../deploy/README.md)). |
| `stress.py` | Stresses the oracle with random inputs from a seeded generator: `python3 tools/stress.py [--seed N] [--rounds N]`. It checks the invariants the chapters state rather than the vectors' values, such as canonical JSON round trips, corrections keeping the five values consistent with a full recomputation, anchors always resolving to one of the five outcomes, the zip reader never writing outside its folder, and a mutated package always being refused. |
| `oracle/` | A compact implementation of every rule. `build.py` uses it to compute expected values, and `check.py` uses it to recompute them. |
## When to run what
Each program is also a pixi task.
Pixi sets up a pinned Python and the other packages the repository uses.
The commands `pixi run check`, `pixi run quick`, `pixi run build` and `pixi run load` then work from anywhere in the repository.
`pixi run check` lints the Markdown first.
Run the programs directly from the repository root if you do not use pixi:
```sh
python3 tools/check.py # everything
python3 tools/check.py --quick # everything except the rebuild
python3 tools/build.py --out /tmp/rebuilt
python3 tools/validate.py fixtures/packages/hale2026harmonic
python3 tools/load.py --sqlite lab.db fixtures/packages/hale2026harmonic
```
A passing check ends with `All checks pass.`
The load command prints `hale2026harmonic: 75 nodes`, one line per package, and writes `lab.db` with a folder `lab.db.assets/` beside it for files that have no table of their own.
It validates every package first and loads nothing if one fails, printing each failure as `package: where: rule: message`.
The full and checks-only forms of one document share its key, so they go into separate databases.
Each program prints its options with `--help` and refuses options it does not know.
`build.py` rebuilds `fixtures/` and `vectors/` in place unless you pass `--out`.
Run `pixi run check` before every commit.
Run `pixi run build` after you change a fixture source, a rule in `oracle/` or a case in `build.py`, and commit the regenerated files with the change.
The check rebuilds into a temporary folder and compares every file byte for byte, so it fails on out-of-date files.
## About the oracle
An oracle is a reference computation: a slow, plain program whose answers count as the expected ones.
It is not a production reader or writer.
The oracle exists so that no expected value in this repository is typed by hand.
It favors clarity, and leaves the cryptography and schema validation to established packages.
Ed25519 (a digital signature scheme) and HKDF come from `cryptography`, and JSON Schema (a format that describes the shape of JSON data) validation from `jsonschema`.
The VRF of RFC 9381 (`oracle/vrf.py`) and ML-DSA-65 of FIPS 204 (`oracle/mldsa.py`) are written out in plain Python, slow and clear, and checked against the standards' own test vectors.
The oracle still refuses any schema that uses a keyword outside the subset the schemas are restricted to.
Canonical JSON goes through Python's built-in JSON encoder, and falls back to an exact RFC 8785 serializer for the values that encoder writes differently: floating-point numbers, integers beyond 2^53 and some non-ASCII member names.
Hashes are kept incrementally.
A node's five values depend only on its own subtree, so after a correction the oracle rehashes only the nodes that changed and their ancestors, and a container's projections are its children's cached projections joined.
`check.py` compares the result after every correction with a slow computation that projects each node from its whole subtree.
Do not build on it as a library.
Implementations follow the [specification](../spec/README.md) and test themselves against the [vectors](../vectors/README.md).
>>>>>>> 4fa326d (First implementation (#1))
tools/__init__.py
View File
Whitespace-only changes.
tools/build.py
-1869
View File
File diff suppressed because it is too large. Load diff
tools/check.py
-1206
View File
File diff suppressed because it is too large. Load diff
tools/load.py
-81
View File
@@ -1,81 +0,0 @@
#!/usr/bin/env python3
"""Load packages into a SQLite database with the tables of sql/doc.sql and sql/store.sql.
Non-normative tooling.
Usage: python3 tools/load.py --sqlite lab.db PACKAGE [PACKAGE ...]
python3 tools/load.py --sqlite lab.db --bundle --built-from REV PACKAGE [PACKAGE ...]
A package is a folder, or a zipped package (`.sct.zip`), unpacked under the rules of spec/12 §12.2 first.
With --bundle the new file is a scienting-bundle/1: it also gets the tables of sql/bundle-sqlite.sql,
the full-text search index and bundle_meta (spec/13 §13.5).
Each package is validated first, and nothing is loaded if any package fails.
Files without tables of their own (assets, datasets, derived files) are copied by SHA-256 into an
asset store folder beside the database, named <database>.assets/.
"""
import argparse
import os
import sqlite3
import sys
import tempfile
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, ROOT)
from tools.oracle import db, package # noqa: E402
from tools.oracle.schema import Store # noqa: E402
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--sqlite", required=True, help="the database file; created with the tables if new")
ap.add_argument("--bundle", action="store_true", help="build a new scienting-bundle/1 file with search")
ap.add_argument("--built-from", help="with --bundle: the store revision the packages come from")
ap.add_argument("--store", help="with --bundle: the store's name (default: the packages' common store)")
ap.add_argument("--max-size", metavar="BYTES", type=int, default=package.ZIP_LIMIT,
help="refuse a zipped package that expands past this many bytes (default %d)" % package.ZIP_LIMIT)
ap.add_argument("packages", nargs="+", help="package folders or zipped packages")
a = ap.parse_args()
if a.bundle and (os.path.exists(a.sqlite) or not a.built_from):
ap.error("--bundle needs --built-from and a database file that does not exist yet")
store, bad = Store(os.path.join(ROOT, "schemas")), False
with tempfile.TemporaryDirectory() as tmp:
folders = []
for i, p in enumerate(a.packages):
if package.is_zip(p):
try:
folders.append(package.read_zip(p, os.path.join(tmp, str(i)), a.max_size))
except package.Refusal as e:
print("%s: %s: %s" % (p, e.rule, e.message), file=sys.stderr)
bad = True
continue
else:
folders.append(p)
for e in package.validate(folders[-1], store).errors:
print("%s: %s: %s: %s" % (p, e["where"], e["rule"], e["message"]), file=sys.stderr)
bad = True
if bad:
sys.exit("Nothing was loaded.")
new = not os.path.exists(a.sqlite)
conn = sqlite3.connect(a.sqlite)
if a.bundle:
db.create_bundle(conn, os.path.join(ROOT, "sql"))
elif new:
db.create(conn, os.path.join(ROOT, "sql"))
for p in folders:
key = db.load(conn, p, a.sqlite + ".assets")
n = conn.execute("SELECT COUNT(*) FROM nodes WHERE doc_key = ?", (key,)).fetchone()[0]
if a.bundle:
print("%s: %d nodes, %d search rows" % (key, n, db.index(conn, key)))
else:
print("%s: %d nodes" % (key, n))
if a.bundle:
stores = {r[0] for r in conn.execute("SELECT DISTINCT store FROM documents WHERE store IS NOT NULL")}
store_name = a.store or (stores.pop() if len(stores) == 1 else None)
if store_name is None:
sys.exit("The packages name no single store; give one with --store.")
db.write_meta(conn, store_name, a.built_from)
conn.commit()
if __name__ == "__main__":
main()
tools/oracle/__init__.py
-13
View File
@@ -1,13 +0,0 @@
"""The reference computations behind every expected value in this repository; see tools/README.md."""
import importlib.util
import sys
if sys.version_info < (3, 14):
sys.exit("The paperspec tools need Python 3.14 or later; this is Python %d.%d." % sys.version_info[:2])
_missing = [m for m in ("cryptography", "jsonschema") if importlib.util.find_spec(m) is None]
if _missing:
sys.exit("The paperspec tools need %s %s; run them through pixi, or install %s with pip."
% ("the package" if len(_missing) == 1 else "the packages", " and ".join(_missing),
"it" if len(_missing) == 1 else "them"))
tools/oracle/align.py
-127
View File
@@ -1,127 +0,0 @@
"""Whole-document sentence alignment, in the manner of a patience diff.
Non-normative; see spec/05-hashes.md, "Alignment".
Each reading is a list of items: {"id", "strict", "lossy", "complex", "container", "context"}.
"""
import unicodedata
from .hashing import prose_length
MIN_PROSE = 20
def _numeric_skeleton(strict):
keep = []
i, n = 0, len(strict)
while i < n:
ch = strict[i]
if ch in "_^" and i + 1 < n and strict[i + 1] == "{":
keep.append(strict[i:i + 2])
i += 2
continue
if unicodedata.category(ch) == "Nd" or ch in "\u2212.,+}":
keep.append(ch)
i += 1
return "".join(keep)
def classify(a, b):
"""`None` when two paired sentences agree; else the suspect kind."""
if a["strict"] == b["strict"] and a["complex"] == b["complex"]:
return None
if _numeric_skeleton(a["strict"]) != _numeric_skeleton(b["strict"]):
return "number"
if a["complex"] != b["complex"]:
return "equation"
return "word"
def _lis(pairs):
"""Longest subsequence increasing in B among pairs sorted by A; lexicographically smallest A on ties."""
n = len(pairs)
if not n:
return []
best = [1] * n
nxt = [-1] * n
for i in range(n - 1, -1, -1):
for j in range(i + 1, n):
if pairs[j][1] > pairs[i][1] and best[j] + 1 > best[i]:
best[i] = best[j] + 1
nxt[i] = j
top = max(best)
i = next(k for k in range(n) if best[k] == top)
out = []
while i != -1:
out.append(pairs[i])
i = nxt[i]
return out
def _lcs(A, B, a0, a1, b0, b1):
"""LCS over align keys of A[a0:a1] and B[b0:b1], as index pairs, with the spec's tie rule."""
n, m = a1 - a0, b1 - b0
L = [[0] * (m + 1) for _ in range(n + 1)]
for i in range(n - 1, -1, -1):
for j in range(m - 1, -1, -1):
if A[a0 + i]["lossy"] == B[b0 + j]["lossy"]:
L[i][j] = L[i + 1][j + 1] + 1
else:
L[i][j] = max(L[i + 1][j], L[i][j + 1])
out = []
i = j = 0
while i < n and j < m:
if A[a0 + i]["lossy"] == B[b0 + j]["lossy"]:
out.append((a0 + i, b0 + j))
i += 1
j += 1
elif L[i + 1][j] >= L[i][j + 1]:
i += 1
else:
j += 1
return out
def align(A, B):
"""Align reading B to reading A. Returns {"pairs": [...], "suspects": [...]}."""
count_a, count_b = {}, {}
for x in A:
count_a[x["lossy"]] = count_a.get(x["lossy"], 0) + 1
for x in B:
count_b[x["lossy"]] = count_b.get(x["lossy"], 0) + 1
pos_b = {x["lossy"]: j for j, x in enumerate(B)}
cands = [
(i, pos_b[x["lossy"]]) for i, x in enumerate(A)
if count_a[x["lossy"]] == 1 and count_b.get(x["lossy"]) == 1 and prose_length(x["lossy"]) >= MIN_PROSE
]
anchors = _lis(cands)
pairs = []
prev_a, prev_b = 0, 0
for ai, bi in anchors + [(len(A), len(B))]:
pairs += _lcs(A, B, prev_a, ai, prev_b, bi)
if ai < len(A):
pairs.append((ai, bi))
prev_a, prev_b = ai + 1, bi + 1
out_pairs, suspects = [], []
for ai, bi in pairs:
kind = classify(A[ai], B[bi])
out_pairs.append({"a": A[ai]["id"], "b": B[bi]["id"], "agree": kind is None})
if kind:
suspects.append({"kind": kind, "a": A[ai]["id"], "b": B[bi]["id"]})
if A[ai]["context"] != B[bi]["context"]:
suspects.append({"kind": "structure", "a": A[ai]["id"], "b": B[bi]["id"]})
for (a1, b1), (a2, b2) in zip(pairs, pairs[1:]):
if a2 == a1 + 1 and b2 == b1 + 1:
same_a = A[a1]["container"] == A[a2]["container"]
same_b = B[b1]["container"] == B[b2]["container"]
if same_a != same_b:
suspects.append({"kind": "join", "a": A[a2]["id"], "b": B[b2]["id"]})
paired_a = {a for a, _ in pairs}
paired_b = {b for _, b in pairs}
for i, x in enumerate(A):
if i not in paired_a:
suspects.append({"kind": "dropped", "a": x["id"], "b": None})
for j, x in enumerate(B):
if j not in paired_b:
suspects.append({"kind": "added", "a": None, "b": x["id"]})
return {"pairs": out_pairs, "suspects": suspects}
tools/oracle/anchors.py
-271
View File
@@ -1,271 +0,0 @@
"""Addresses, anchors and their resolution.
Non-normative; see spec/06-anchors.md.
"""
import re
from fractions import Fraction
from . import hashing, jcs
from .model import CONTAINERS_WITH_CONTENT, LEAVES, RUN_KINDS, Doc, anchor_class
from .text import norm
ADDRESS_RE = re.compile(
r"^(?:(?P<store>[a-z0-9][a-z0-9._-]*)/)?"
r"(?P<key>[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}(?:\.[a-z0-9][a-z0-9-]*)*)"
r"#(?P<node>[a-z0-9][a-z0-9._-]*)"
r"(?:@(?P<pin>[0-9a-f]{6}))?"
r"(?:\[(?P<start>0|[1-9][0-9]*):(?P<end>0|[1-9][0-9]*)\])?$"
)
QUOTE_CAP = 200
CONTEXT_CAP = 32
def parse_address(s):
m = ADDRESS_RE.match(s)
if not m:
return None
d = m.groupdict()
out = {"store": d["store"], "key": d["key"], "node": d["node"], "pin": d["pin"], "span": None}
if d["start"] is not None:
a, b = int(d["start"]), int(d["end"])
if a > b:
return None
out["span"] = [a, b]
return out
def format_address(store, key, node, pin=None, span=None):
s = (store + "/" if store else "") + key + "#" + node
if pin:
s += "@" + pin[:6]
if span is not None:
s += "[%d:%d]" % tuple(span)
return s
def cell_context(doc, cell_id):
"""Table label and the text of the row and column header paths of a cell."""
cell = doc.nodes[cell_id]
row = doc.nodes[cell["parent"]]
table_id = doc.ancestors(cell_id)[-3]
table = doc.nodes[table_id]
cols = {c["id"]: c for c in table.get("attrs", {}).get("columns", [])}
col = cols.get(cell.get("attrs", {}).get("column"), {"header_path": []})
def txt(nid):
return norm(hashing.plain(doc, nid))
return {
"table_label": table.get("label"),
"row_path": [txt(x) for x in row.get("attrs", {}).get("header_path", [])],
"column_path": [txt(x) for x in col["header_path"]],
}
def make_anchor(doc, values, node, rev, span=None, store=None):
n = doc.nodes[node]
a = {"format": "scienting-anchor/1"}
if store:
a["store"] = store
a.update({"doc": doc.key, "node": node, "kind": n["kind"], "own_hash": values[node]["own_hash"],
"node_hash": values[node]["node_hash"]})
if n["kind"] in LEAVES:
text = hashing.plain(doc, node)
if span is not None:
a["span"] = list(span)
exact = text[span[0]:span[1]]
pre, suf = text[max(0, span[0] - CONTEXT_CAP):span[0]], text[span[1]:span[1] + CONTEXT_CAP]
else:
exact, pre, suf = text, "", ""
q = {}
if len(exact) <= QUOTE_CAP:
q["exact"] = exact
else:
q["exact_hash"] = hashing.PLAIN(norm(exact))
q["length"] = len(exact)
if pre:
q["prefix"] = pre
if suf:
q["suffix"] = suf
a["quote"] = q
if n["kind"] == "cell":
a["context"] = cell_context(doc, node)
elif anchor_class(n["kind"]) == "container-without-content":
a["leaves"] = [[x, values[x]["match_hash"]] for x in doc.leaves(node)]
a["rev"] = rev
return a
def _quote_hits(text, q):
"""Start positions where the quote matches `text`."""
if "exact" in q:
ex = q["exact"]
hits = [i for i in range(len(text) - len(ex) + 1) if text.startswith(ex, i)]
length = len(ex)
else:
length = q["length"]
hits = [i for i in range(len(text) - length + 1)
if hashing.PLAIN(norm(text[i:i + length])) == q["exact_hash"]]
if len(hits) > 1 and ("prefix" in q or "suffix" in q):
pre, suf = q.get("prefix", ""), q.get("suffix", "")
hits = [i for i in hits if text[:i].endswith(pre) and text[i + length:].startswith(suf)]
return hits, length
def _drift(state, doc, anchor, nid, threshold):
leaves = set(doc.leaves(nid))
touched = set()
count = 0
for c in state.corrections:
if c["index"] <= anchor["rev"]:
continue
inside = nid in (c.get("path") or []) or nid in (c.get("to_path") or [])
t = set(c.get("touched", []))
if inside and t:
count += 1
touched |= t
denom = len(leaves | touched)
share = Fraction(len(touched), denom) if denom else Fraction(0)
level = "none" if count == 0 else ("high" if share > threshold else "low")
return {"count": count, "share": _decimal(share), "level": level}
def _decimal(fr):
"""A share as a decimal string with four places, rounded half up."""
v = (fr * 10000 + Fraction(1, 2)).__floor__()
return "%d.%04d" % (v // 10000, v % 10000)
def _follow(state, nid):
"""Follow aliases and retirements. Returns (live id or None, chain, via)."""
live = {n["id"] for n in state.nodes}
table = {e["id"]: e for e in state.ids}
chain, via = [], None
seen = set()
while nid not in live:
if nid in seen or nid not in table:
return None, chain, via
seen.add(nid)
e = table[nid]
if e["kind"] == "retired" or via is None:
via = e["kind"]
chain.append(nid)
nid = e.get("successor")
if nid is None:
return None, chain, via
return nid, chain, via
def _relabel_reverted_own(state, doc, nid, anchor):
"""The own_hash the node had before the relabel corrections since the anchor, or None when anything else
changed the node or what is below it since (spec/06 §6.5: its *only* changes were relabel corrections)."""
since = [c for c in state.corrections if c["index"] > anchor["rev"] and (
c.get("target") == nid or nid in (c.get("path") or []) or nid in (c.get("to_path") or []))]
rel = [c for c in since if c.get("target") == nid]
if not rel or any(c["op"] != "relabel" or c.get("target") != nid for c in since):
return None
n = dict(doc.nodes[nid], label=rel[0]["from"])
tmp = Doc(doc.key, [n if x["id"] == nid else x for x in [doc.nodes[i] for i in doc.order]])
return hashing.PLAIN(jcs.dumpb(hashing.own_content(tmp, nid, lambda h: hashing.PLAIN(
norm(hashing.plain(tmp, h))))))
def resolve(state, anchor, threshold=Fraction(1, 4)):
doc = state.doc()
values = hashing.compute(doc)
kind = anchor.get("kind")
original = anchor["node"]
nid, chain, via = _follow(state, original)
live_kind = doc.nodes[nid]["kind"] if nid else None
table = {e["id"]: e for e in state.ids}
# The kind falls back to the retired entry the chain reached (spec/06 §6.5); an alias carries no kind.
retired_kind = next((table[x].get("node_kind") for x in reversed(chain)
if table.get(x, {}).get("kind") == "retired"), None)
kind = kind or live_kind or retired_kind
cls = anchor_class(kind) if kind else "leaf"
is_span = "span" in anchor
result = {"node": nid}
retired = via == "retired"
def out(res, **kw):
r = {"result": res, "node": kw.pop("node", nid)}
if r["node"]:
r["node_hash"] = values[r["node"]]["node_hash"]
if chain:
r["via"] = chain
r.update(kw)
return r
# The cell rule: a retired cell is found again by its table label and header paths.
def cell_rule():
ctx = anchor.get("context")
q = anchor.get("quote")
if kind != "cell" or not ctx or not q:
return None
found = []
for x in doc.order:
if doc.nodes[x]["kind"] != "cell":
continue
if cell_context(doc, x) != ctx:
continue
hits, ln = _quote_hits(hashing.plain(doc, x), q)
if len(hits) == 1:
found.append((x, hits[0], ln))
if len(found) == 1:
x, i, ln = found[0]
kw = {"node": x}
if is_span:
kw["span"] = [i, i + ln]
return out("reattached", **kw)
return None
if nid is None:
return cell_rule() or out("stale", node=None, reason="retired without a successor" if chain
else "unknown node")
if cls == "leaf":
text = hashing.plain(doc, nid)
if not retired and values[nid]["node_hash"] == anchor["node_hash"]:
return out("current", **({"span": anchor["span"]} if is_span else {}))
q = anchor.get("quote")
if retired:
if q:
hits, ln = _quote_hits(text, q)
if (is_span and len(hits) == 1) or (not is_span and hits):
return out("moved", **({"span": [hits[0], hits[0] + ln]} if is_span else {}))
return cell_rule() or out("stale", reason="the successor does not contain the quote")
if q:
hits, ln = _quote_hits(text, q)
if len(hits) == 1:
return out("reattached", **({"span": [hits[0], hits[0] + ln]} if is_span else {}))
own = _relabel_reverted_own(state, doc, nid, anchor)
if own is not None and own == anchor["own_hash"]:
return out("reattached", reason="relabel")
return out("stale", reason="the node changed and the quote does not occur exactly once")
if cls == "container-with-content":
if values[nid]["own_hash"] == anchor["own_hash"]:
if retired:
return out("moved")
if values[nid]["node_hash"] == anchor["node_hash"]:
return out("current")
return out("changed-below", drift=_drift(state, doc, anchor, nid, threshold))
if not retired:
own = _relabel_reverted_own(state, doc, nid, anchor)
if own is not None and own == anchor["own_hash"]:
return out("reattached", reason="relabel")
return out("stale", reason="the node's own content changed")
# Container without content.
if retired:
orig = anchor.get("leaves", [])
now = set(doc.leaves(nid))
now_match = {values[x]["match_hash"] for x in now}
held = sum(1 for i, m in orig if i in now or m in now_match)
if 2 * held >= len(orig):
return out("moved")
return out("stale", reason="the successor holds fewer than half of the original leaves")
if values[nid]["node_hash"] == anchor["node_hash"]:
return out("current")
return out("changed-below", drift=_drift(state, doc, anchor, nid, threshold))
tools/oracle/corrections.py
-454
View File
@@ -1,454 +0,0 @@
"""Corrections on an established document.
Non-normative; see spec/11-corrections.md.
"""
import copy
from . import hashing
from .model import ATTRS, LABELLED, LEAVES, MATH_KINDS, RUN_KINDS, VERBATIM_KINDS, Doc, CHILDREN
from .tree import Refusal, canonical_runs, mint_ids, validate_references, validate_structure, _flatten
from . import jcs
from .schema import validate as schema_validate
from .text import digits, record_id, sha256_hex
MERGEABLE = RUN_KINDS | {"paragraph", "list", "item", "quote", "caption", "figure-note", "table-note"}
SETTABLE = {
"section": {"level"},
"table": {"columns"},
"cell": {"column", "rowspan", "colspan", "role"},
"row": {"header_path"},
"list": {"ordered"},
"item": {"marker"},
"theorem": {"variant"},
}
EDITABLE_RUN_FIELD = {"text": "text", "link": "text", "cite": "text", "xref": "text", "fnmark": "text", "math": "source"}
class State:
"""The current state of a document: nodes, the ID table and the corrections applied."""
def __init__(self, key, nodes, ids=None, corrections=None, indexes=None):
self.key = key
self.nodes = [dict(n) for n in nodes]
self.ids = [dict(e) for e in (ids or [])]
self.corrections = [dict(c) for c in (corrections or [])]
self._indexes = dict(indexes or {})
@property
def rev(self):
return len(self.corrections)
def doc(self):
return Doc(self.key, self.nodes)
def index(self, hasher=hashing.PLAIN):
"""The node values under `hasher`, kept with the state so the next correction rehashes only what changes."""
if hasher.key not in self._indexes:
self._indexes[hasher.key] = hashing.Index.build(self.doc(), hasher)
return self._indexes[hasher.key]
def used_ids(self):
return {n["id"] for n in self.nodes} | {e["id"] for e in self.ids}
def _strings(node):
"""Editable strings of a leaf: (path, value) pairs."""
k = node["kind"]
if k in RUN_KINDS:
out = []
for i, r in enumerate(node["content"]):
f = EDITABLE_RUN_FIELD.get(r["kind"])
if f:
out.append(((i, f), r[f]))
return out
if k in MATH_KINDS:
return [(("source",), node["content"]["source"])]
if k in VERBATIM_KINDS:
return [(("text",), node["content"]["text"])]
return []
def _occurrences(s, find):
return [i for i in range(len(s) - len(find) + 1) if s.startswith(find, i)]
def _set_string(node, path, value):
if len(path) == 2:
i, f = path
node["content"][i] = dict(node["content"][i], **{f: value})
else:
node["content"] = dict(node["content"], **{path[0]: value})
def _has_crop(c):
return any(e.get("kind") == "crop" for e in c.get("evidence", []))
def _rebuild(nodes):
"""Recompute document order and `ord` from parent links, keeping sibling order as listed."""
by_id = {n["id"]: n for n in nodes}
kids = {}
root = None
for n in nodes:
if n.get("parent") is None:
root = n["id"]
else:
kids.setdefault(n["parent"], []).append(n["id"])
for p in kids:
kids[p].sort(key=lambda c: by_id[c]["ord"])
out = []
stack = [root]
while stack:
x = stack.pop()
out.append(by_id[x])
for i, c in enumerate(kids.get(x, [])):
by_id[c]["ord"] = i
stack.extend(reversed(kids.get(x, [])))
return out
def _subtree(doc, nid):
return doc.preorder(nid)
def _place(nodes, nid, parent, after):
"""Give `nid` a fractional position under `parent` after sibling `after` (None: first)."""
by_id = {n["id"]: n for n in nodes}
sibs = sorted([n for n in nodes if n.get("parent") == parent and n["id"] != nid], key=lambda n: n["ord"])
for i, s in enumerate(sibs):
s["ord"] = 2 * i + 2
if after is None:
pos = 0
else:
if after not in by_id or by_id[after].get("parent") != parent:
raise Refusal("correction.target", "`place_after` is not a child of the parent")
pos = by_id[after]["ord"] + 1
by_id[nid]["parent"] = parent
by_id[nid]["ord"] = pos
def gate(state, proposal, store, hasher=hashing.PLAIN):
"""Run the gate (spec/11 §11.4) on a proposed correction, leaving `state` unchanged.
Returns the state the correction would produce and the record it would append to corrections.jsonl, as apply
does. Raises Refusal naming the rule broken. `store` is the schema store that holds correction.schema.json.
"""
errors = schema_validate(store, "correction.schema.json", proposal)
if errors:
raise Refusal("correction.schema", "; ".join(errors[:3]))
return apply(state, proposal, hasher, store)
def _is_index(v):
return isinstance(v, int) and not isinstance(v, bool) and v >= 0
def id_order(e):
"""The order of ids.jsonl (spec/04 §4.4): aliases first by ID, then retirements by revision and ID."""
return (0 if e["kind"] == "alias" else 1, e.get("rev", 0), e["id"])
def _node_rule(error):
"""The rule a node-schema error breaks, from the property the error names."""
path = error.split(":", 1)[0]
for prefix, rule in (("/attrs", "doc.node.attrs"), ("/label", "doc.node.label"), ("/content", "doc.node.content")):
if path.startswith(prefix):
return rule
return "doc.node.schema"
def apply(state, c, hasher=hashing.PLAIN, store=None):
"""Apply correction `c` to `state` and return the recorded correction. Raises Refusal.
With a schema `store`, every node the correction changed or created is also validated against
node.schema.json, which is how the gate enforces the attribute shapes of spec/03 §3.4 (rule 11 of §11.4).
"""
c = copy.deepcopy(c)
doc = state.doc()
before_ix = state.index(hasher)
before_vals = before_ix.values
nodes = [copy.deepcopy(n) for n in state.nodes]
by_id = {n["id"]: n for n in nodes}
op = c["op"]
target = c.get("target")
if op != "insert" or "target" in c:
if target not in by_id:
raise Refusal("correction.target", "no live node %s" % target)
if target == doc.root and op in ("delete", "move", "merge", "reclassify"):
raise Refusal("correction.target", "the root cannot be " + op + "d")
record = {"format": "scienting-correction/1", "index": state.rev + 1}
retired, created = [], []
path = doc.ancestors(target) if target in by_id else None
touched = []
if op == "replace":
node = by_id[target]
find, repl = c.get("find", ""), c["replace"]
if not find:
raise Refusal("correction.find", "find must not be empty")
hits = []
for pth, s in _strings(node):
if "run" in c and (len(pth) != 2 or pth[0] != c["run"]):
continue
hits += [(pth, s, i) for i in _occurrences(s, find)]
if len(hits) != 1:
raise Refusal("correction.find", "find occurs %d times, not exactly once" % len(hits), target)
pth, s, i = hits[0]
if isinstance(repl, str):
_set_string(node, pth, s[:i] + repl + s[i + len(find):])
else:
if len(pth) != 2 or node["content"][pth[0]]["kind"] != "text":
raise Refusal("correction.replace", "runs may replace text only inside a text run", target)
if any(r["kind"] == "equation" for r in repl):
raise Refusal("correction.replace", "equation runs are added by insert", target)
run = node["content"][pth[0]]
style = {"style": run["style"]} if "style" in run else {}
mid = [dict({"kind": "text", "text": s[:i]}, **style)] + list(repl) + [
dict({"kind": "text", "text": s[i + len(find):]}, **style)
]
node["content"] = node["content"][: pth[0]] + mid + node["content"][pth[0] + 1:]
if node["kind"] in RUN_KINDS:
node["content"] = canonical_runs(node["content"])
touched = [target]
elif op == "insert" and "node" in c:
if not _has_crop(c):
raise Refusal("correction.evidence", "an insert needs crop evidence")
parent = c["parent"]
if parent not in by_id:
raise Refusal("correction.target", "no live parent %s" % parent)
flat = []
_flatten(copy.deepcopy(c["node"]), None, 0, flat)
new = mint_ids(state.key, flat, state.used_ids())
new[0]["parent"] = parent
nodes.extend(new)
_place(nodes, new[0]["id"], parent, c.get("place_after"))
nodes = _rebuild(nodes)
created = [n["id"] for n in new]
tmp = Doc(state.key, nodes)
touched = [x for x in tmp.leaves(new[0]["id"])]
path = tmp.ancestors(new[0]["id"])
target = new[0]["id"]
elif op == "insert":
if not _has_crop(c):
raise Refusal("correction.evidence", "an insert needs crop evidence")
node = by_id[target]
if node["kind"] not in RUN_KINDS:
raise Refusal("correction.insert", "text is inserted into runs", target)
run, offset = c.get("run"), c.get("offset")
if not _is_index(run) or run >= len(node["content"]) or not _is_index(offset):
raise Refusal("correction.insert", "bad run or offset", target)
r = node["content"][run]
f = EDITABLE_RUN_FIELD.get(r["kind"])
if not f or offset > len(r[f]) or not isinstance(c.get("text"), str):
raise Refusal("correction.insert", "bad run or offset", target)
node["content"][run] = dict(r, **{f: r[f][:offset] + c["text"] + r[f][offset:]})
node["content"] = canonical_runs(node["content"])
touched = [target]
elif op == "delete":
gone = _subtree(doc, target)
touched = [x for x in gone if doc.nodes[x]["kind"] in LEAVES]
nodes = _rebuild([n for n in nodes if n["id"] not in set(gone)])
retired = [(x, None) for x in gone]
elif op == "split":
node = by_id[target]
if node["kind"] != "sentence":
raise Refusal("correction.split", "only sentences are split", target)
k = c.get("offset")
first, second, pos = [], [], 0
total = len(hashing.plain_runs(node["content"], doc))
if not _is_index(k) or not 0 < k < total:
raise Refusal("correction.split", "offset must fall inside the sentence", target)
for r in node["content"]:
ln = len(hashing.plain_runs([r], doc))
if pos + ln <= k:
first.append(r)
elif pos >= k:
second.append(r)
elif r["kind"] == "text":
cut = k - pos
first.append(dict(r, text=r["text"][:cut]))
second.append(dict(r, text=r["text"][cut:]))
else:
raise Refusal("correction.split", "the offset falls inside a %s run" % r["kind"], target)
pos += ln
_strip_edge(first, end=True)
_strip_edge(second, end=False)
first, second = canonical_runs(first), canonical_runs(second)
if not first or not second:
raise Refusal("correction.split", "both parts must keep text", target)
new_spec = {"kind": "sentence", "content": second}
eq_second = [r["node"] for r in second if r["kind"] == "equation"]
flat = [dict(new_spec, _parent=None, ord=0)]
tmp_nodes = mint_ids(state.key, [dict(flat[0], content=[
{"kind": "text", "text": hashing.plain_runs(second, doc)}])], state.used_ids())
new_id = tmp_nodes[0]["id"]
node["content"] = first
new_node = {"id": new_id, "kind": "sentence", "parent": node["parent"], "ord": 0, "content": second}
if "fidelity" in node:
new_node["fidelity"] = node["fidelity"]
nodes.append(new_node)
_place(nodes, new_id, node["parent"], target)
for i, e in enumerate(eq_second):
by_id[e]["parent"] = new_id
by_id[e]["ord"] = i
nodes = _rebuild(nodes)
created = [new_id]
touched = [target, new_id]
elif op == "merge":
other = c["with"]
node = by_id[target]
if other not in by_id:
raise Refusal("correction.target", "no live node %s" % other)
o = by_id[other]
if node["kind"] != o["kind"] or node["kind"] not in MERGEABLE:
raise Refusal("correction.merge", "merge joins two nodes of one mergeable kind", target)
if o.get("parent") != node.get("parent") or o["ord"] != node["ord"] + 1:
raise Refusal("correction.merge", "merge joins a node with its next sibling", target)
if node["kind"] in RUN_KINDS:
joiner = c.get("joiner", " ")
mid = [{"kind": "text", "text": joiner}] if joiner else []
node["content"] = canonical_runs(node["content"] + mid + o["content"])
touched = [target, other]
else:
touched = [x for x in doc.leaves(other) if x != other]
base = len(doc.kids(target))
for i, kid in enumerate(doc.kids(other)):
by_id[kid]["parent"] = target
by_id[kid]["ord"] = base + i
nodes = _rebuild([n for n in nodes if n["id"] != other])
retired = [(other, target)]
elif op == "move":
parent = c["parent"]
if parent not in by_id or parent in set(_subtree(doc, target)) or c.get("place_after") == target:
raise Refusal("correction.move", "bad destination", target)
_place(nodes, target, parent, c.get("place_after"))
nodes = _rebuild(nodes)
touched = [x for x in doc.leaves(target)]
record["to_path"] = Doc(state.key, nodes).ancestors(target)
elif op == "reclassify":
node = by_id[target]
new_kind = c["kind"]
old = node["kind"]
same_shape = (old in RUN_KINDS) == (new_kind in RUN_KINDS) and (old in MATH_KINDS) == (
new_kind in MATH_KINDS) and (old in VERBATIM_KINDS) == (new_kind in VERBATIM_KINDS)
if new_kind not in CHILDREN or not same_shape:
raise Refusal("correction.reclassify", "%s cannot become %s" % (old, new_kind), target)
node["kind"] = new_kind
if new_kind not in LABELLED:
node.pop("label", None)
# Each kind carries only its own attributes (spec/03 §3.4).
kept = {k: v for k, v in node.get("attrs", {}).items() if k in ATTRS.get(new_kind, ())}
if kept:
node["attrs"] = kept
else:
node.pop("attrs", None)
record["from"] = old
touched = [x for x in doc.leaves(target)]
elif op == "relabel":
node = by_id[target]
if node["kind"] not in LABELLED:
raise Refusal("correction.relabel", "%s takes no label" % node["kind"], target)
record["from"] = node.get("label")
if c.get("label") is None:
node.pop("label", None) # optional properties are omitted, never written as null (spec/03 §3.1)
else:
node["label"] = c["label"]
touched = [target] if node["kind"] in LEAVES else []
elif op == "set":
node = by_id[target]
attr = c["attr"]
if attr not in SETTABLE.get(node["kind"], ()):
raise Refusal("correction.set", "%s cannot be set on %s" % (attr, node["kind"]), target)
attrs = dict(node.get("attrs", {}))
record["from"] = attrs.get(attr)
attrs[attr] = c["value"]
node["attrs"] = attrs
touched = [target] if node["kind"] in LEAVES else []
else:
raise Refusal("correction.op", "unknown operation " + op)
after_doc = Doc(state.key, nodes)
validate_structure(after_doc)
validate_references(after_doc)
after_ix = before_ix.update(after_doc)
after_vals = after_ix.values
if store is not None:
changed = {x for x in set(touched) | set(created) | {target} if x in after_doc.nodes}
for nid in sorted(changed):
errors = schema_validate(store, "node.schema.json", dict(after_doc.nodes[nid], **after_vals[nid]))
if errors:
raise Refusal(_node_rule(errors[0]), errors[0], nid)
# Digits: a change to a digit needs crop evidence, and no leaf outside `touched` may change its digits.
needs_crop = False
if op == "replace":
rep = c["replace"] if isinstance(c["replace"], str) else hashing.plain_runs(c["replace"], doc)
needs_crop = digits(c["find"]) != digits(rep)
elif op == "delete":
needs_crop = any(digits(before_ix.plain(x)) for x in touched)
elif op == "relabel":
needs_crop = digits(record.get("from") or "") != digits(c.get("label") or "")
if needs_crop and not _has_crop(c):
raise Refusal("correction.digits", "a digit changed without crop evidence", target)
for nid in doc.leaves(doc.root):
if nid in after_ix.entries and nid not in touched:
if digits(before_ix.plain(nid)) != digits(after_ix.plain(nid)):
raise Refusal("correction.digits", "an undeclared digit change", nid)
for k in ("op", "target", "find", "replace", "run", "offset", "text", "parent", "place_after", "node", "with",
"joiner", "kind", "label", "attr", "value", "evidence", "author", "reason", "at"):
if k in c:
record[k] = c[k]
if op == "insert" and "node" in c:
record["target"] = target
record["before"] = before_vals[target]["node_hash"] if target in before_vals else None
record["after"] = after_vals[target]["node_hash"] if target in after_vals else None
record["path"] = path
record["touched"] = sorted(set(touched))
if created:
record["created"] = created
if retired:
record["retired"] = [r[0] for r in retired]
record["id"] = record_id(sha256_hex(jcs.dumpb({k: v for k, v in record.items() if k != "id"})))
old_kinds = {n["id"]: n["kind"] for n in state.nodes}
new_ids = list(state.ids)
for rid, succ in retired:
new_ids.append({"id": rid, "kind": "retired", "successor": succ, "node_kind": old_kinds[rid],
"rev": record["index"]})
new_ids.sort(key=id_order)
new_state = State(state.key, [after_doc.nodes[x] for x in after_doc.preorder()], new_ids,
state.corrections + [record], {hasher.key: after_ix})
return new_state, record
def _strip_edge(runs, end):
"""Remove the whitespace at one edge of a split part, across as many text runs as it spans (spec/11 §11.1)."""
from .text import is_ws
while runs:
i = len(runs) - 1 if end else 0
r = runs[i]
if r["kind"] != "text":
return
t = r["text"]
if end:
while t and is_ws(t[-1]):
t = t[:-1]
else:
while t and is_ws(t[0]):
t = t[1:]
if t:
runs[i] = dict(r, text=t)
return
del runs[i]
tools/oracle/crypto.py
-249
View File
@@ -1,249 +0,0 @@
"""Keyed hashing, key derivation, Merkle roots and Ed25519.
Non-normative; see spec/05-hashes.md and spec/08-fidelity.md.
HKDF and Ed25519 come from the `cryptography` package.
"""
import hashlib
from cryptography.exceptions import InvalidSignature
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey, Ed25519PublicKey
from cryptography.hazmat.primitives.kdf.hkdf import HKDF
from . import jcs
def hkdf(ikm, salt, info, length=32):
"""HKDF-SHA-256, RFC 5869."""
return HKDF(algorithm=hashes.SHA256(), length=length, salt=salt, info=info).derive(ikm)
def team_key(secret):
return hkdf(secret, b"scienting-team-key/1", b"scienting-hash/1")
def key_id(key):
return hashlib.sha256(b"scienting-key-id/1" + key).hexdigest()[:16]
def page_hash(page_fp_hex):
return hashlib.sha256(b"scienting-page-hash/1" + bytes.fromhex(page_fp_hex)).hexdigest()
def page_key(page_fps_hex, context):
ikm = b"".join(bytes.fromhex(h) for h in page_fps_hex)
return hkdf(ikm, b"scienting-page-key/1", context.encode("utf-8"))
# --- Merkle trees (RFC 6962 hashing) ------------------------------------------
def _leaf(data):
return hashlib.sha256(b"\x00" + data).digest()
def _node(left, right):
return hashlib.sha256(b"\x01" + left + right).digest()
def _mth(leaves):
n = len(leaves)
if n == 1:
return leaves[0]
k = 1
while k * 2 < n:
k *= 2
return _node(_mth(leaves[:k]), _mth(leaves[k:]))
def merkle_root(items):
"""Root over receipt items: each item is a list such as ["node", id, hash]."""
if not items:
raise ValueError("a Merkle root needs at least one item")
return _mth([_leaf(jcs.dumpb(list(it))) for it in items]).hex()
def receipt_items(receipt):
items = [["node", nid, h] for nid, h in receipt.get("nodes", [])]
items += [list(r) for r in receipt.get("records", [])]
return items
EMPTY_ROOT = hashlib.sha256(b"").hexdigest()
def history_root(items):
"""Root over an append-only history, with RFC 9162's hash of the empty list for no items."""
if not items:
return EMPTY_ROOT
return _mth([_leaf(jcs.dumpb(list(it))) for it in items]).hex()
def leaf_hash(item):
return _leaf(jcs.dumpb(list(item))).hex()
# --- RFC 9162 proofs (§2.1.3 and §2.1.4) ---------------------------------------
def _path(m, leaves):
"""PATH(m, D[n]): the inclusion path for the leaf at index m, as RFC 9162 §2.1.3.1 defines it."""
n = len(leaves)
if n == 1:
return []
k = 1
while k * 2 < n:
k *= 2
if m < k:
return _path(m, leaves[:k]) + [_mth(leaves[k:])]
return _path(m - k, leaves[k:]) + [_mth(leaves[:k])]
def inclusion_path(items, index):
"""The audit path of `items[index]` in the tree over `items`, as hex hashes, leaf-most first."""
if not 0 <= index < len(items):
raise ValueError("index outside the tree")
return [h.hex() for h in _path(index, [_leaf(jcs.dumpb(list(it))) for it in items])]
def verify_inclusion(root, leaf, index, size, path):
"""Whether `path` proves that the leaf hash `leaf` is item `index` of a tree of `size` with `root`
(RFC 9162 §2.1.3.2). Never raises."""
try:
if not 0 <= index < size:
return False
fn, sn = index, size - 1
r = bytes.fromhex(leaf)
for p in path:
p = bytes.fromhex(p)
if sn == 0:
return False
if fn & 1 or fn == sn:
r = _node(p, r)
while fn & 1 == 0 and fn != 0:
fn >>= 1
sn >>= 1
else:
r = _node(r, p)
fn >>= 1
sn >>= 1
return sn == 0 and r.hex() == root
except (ValueError, TypeError):
return False
def _subproof(m, leaves, b):
n = len(leaves)
if m == n:
return [] if b else [_mth(leaves)]
k = 1
while k * 2 < n:
k *= 2
if m <= k:
return _subproof(m, leaves[:k], b) + [_mth(leaves[k:])]
return _subproof(m - k, leaves[k:], False) + [_mth(leaves[:k])]
def consistency_proof(items, m):
"""PROOF(m, D[n]): what shows the tree over the first m items is a prefix of the tree over all n
(RFC 9162 §2.1.4.1), as hex hashes; empty when m is 0 or n."""
n = len(items)
if not 0 <= m <= n:
raise ValueError("m outside the tree")
if m == 0 or m == n:
return []
return [h.hex() for h in _subproof(m, [_leaf(jcs.dumpb(list(it))) for it in items], True)]
def verify_consistency(root_m, root_n, m, n, proof):
"""Whether `proof` shows the tree of size m with `root_m` is a prefix of the tree of size n with `root_n`
(RFC 9162 §2.1.4.2). Never raises."""
try:
if m == 0:
return not proof and root_m == EMPTY_ROOT
if m == n:
return not proof and root_m == root_n
if not 0 < m < n or not proof:
return False
proof = [bytes.fromhex(p) for p in proof]
fn, sn = m - 1, n - 1
while fn & 1:
fn >>= 1
sn >>= 1
if fn == 0:
fr = sr = bytes.fromhex(root_m)
else:
fr = sr = proof[0]
proof = proof[1:]
for c in proof:
if sn == 0:
return False
if fn & 1 or fn == sn:
fr = _node(c, fr)
sr = _node(c, sr)
while fn & 1 == 0 and fn != 0:
fn >>= 1
sn >>= 1
else:
sr = _node(sr, c)
fn >>= 1
sn >>= 1
return sn == 0 and fr.hex() == root_m and sr.hex() == root_n
except (ValueError, TypeError):
return False
# --- Ed25519 (RFC 8032) -------------------------------------------------------
def ed25519_public(secret):
"""The 32-byte public key of a 32-byte Ed25519 secret."""
return Ed25519PrivateKey.from_private_bytes(secret).public_key().public_bytes_raw()
def ed25519_sign(secret, msg):
return Ed25519PrivateKey.from_private_bytes(secret).sign(msg)
def ed25519_verify(public, msg, signature):
"""True when `signature` is a valid Ed25519 signature of `msg` under `public`; never raises."""
if len(public) != 32 or len(signature) != 64:
return False
try:
Ed25519PublicKey.from_public_bytes(public).verify(signature, msg)
except (InvalidSignature, ValueError):
return False
return True
# --- did:key (W3C did:key method, multicodec and base58btc) ---------------------------
BASE58 = "123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz"
MULTICODEC = {"ed25519": 0xED, "ml-dsa-65": 0x1207}
def _varint(n):
out = bytearray()
while True:
b = n & 0x7F
n >>= 7
out.append(b | (0x80 if n else 0))
if not n:
return bytes(out)
def base58btc(data):
n = int.from_bytes(data, "big")
out = ""
while n:
n, r = divmod(n, 58)
out = BASE58[r] + out
return "1" * (len(data) - len(data.lstrip(b"\x00"))) + out
def did_key(scheme, public):
"""The did:key of a public key: `z` and the base58btc of the multicodec prefix and the key bytes."""
if scheme not in MULTICODEC:
raise ValueError("no multicodec code for " + scheme)
return "did:key:z" + base58btc(_varint(MULTICODEC[scheme]) + public)
tools/oracle/db.py
-319
View File
@@ -1,319 +0,0 @@
"""The database form: loading a package into tables and unloading it again.
Non-normative; see spec/13-database.md.
Loading is a pure function of the package, and unloading gives back the same bytes.
"""
import json
import os
import shutil
from . import hashing, jcs
from .model import Doc
from .package import SHA256_RE, _all_files, read_json, read_jsonl, safe_path, write
from .text import norm, sha256_hex
from .tree import Refusal
SPEC_VERSION = "0.1.0"
BUNDLE_FORMAT = "scienting-bundle/1"
STRUCTURED = {"MANIFEST.json", "document.json", "witnesses.json", "provenance.json", "roster.json", "nodes.jsonl",
"pages.jsonl", "locations.jsonl", "references.jsonl", "corrections.jsonl", "receipts.jsonl",
"ids.jsonl", "suspects.jsonl"}
def J(v):
return None if v is None else jcs.dumps(v)
def P(s):
return None if s is None else json.loads(s)
def B(v):
return None if v is None else (1 if v else 0)
def _opt(d, k, v):
if v is not None:
d[k] = v
def ins(conn, into, values):
conn.execute("INSERT %s VALUES (%s)" % (into, ",".join("?" * len(values))), values)
def create(conn, sql_dir):
for name in ("doc.sql", "store.sql"):
with open(os.path.join(sql_dir, name), encoding="utf-8") as f:
conn.executescript(f.read())
def create_bundle(conn, sql_dir):
"""The tables of a bundle: those of create(), then sql/bundle-sqlite.sql (spec/13 §13.5)."""
create(conn, sql_dir)
with open(os.path.join(sql_dir, "bundle-sqlite.sql"), encoding="utf-8") as f:
conn.executescript(f.read())
def search_rows(key, nodes):
"""The bundle's search rows for one document, in document order (spec/13 §13.5).
One row per sentence, cell, equation outside a sentence and figure-text item, as
(doc_key, node, unit, context, text). For a node, `context` is the kind of its parent; a figure-text item
takes its panel as `node` and its `role` as `context`. Withheld nodes are skipped.
"""
doc = Doc(key, nodes)
out = []
for nid in doc.order:
n = doc.nodes[nid]
kind = n["kind"]
parent = doc.nodes[n["parent"]]["kind"] if n.get("parent") else None
if n.get("withheld"):
continue
if kind in ("sentence", "cell") or (kind == "equation" and parent != "sentence"):
out.append((key, nid, kind, parent, norm(hashing.plain(doc, nid))))
elif kind == "panel":
for item in (n.get("attrs") or {}).get("figure_text", []):
out.append((key, nid, "figure-text", item["role"], norm(item["text"])))
return out
def index(conn, key):
"""Fill the bundle's `search` table for the loaded document `key`."""
nodes = []
for n in _q(conn, "SELECT id, kind, parent, ord, attrs, content, withheld FROM nodes WHERE doc_key = ? ORDER BY seq",
(key,)):
rec = {"id": n["id"], "kind": n["kind"], "parent": n["parent"], "ord": n["ord"]}
_opt(rec, "attrs", P(n["attrs"]))
_opt(rec, "content", P(n["content"]))
if n["withheld"]:
rec["withheld"] = True
nodes.append(rec)
rows = search_rows(key, nodes)
conn.executemany("INSERT INTO search VALUES (?, ?, ?, ?, ?)", rows)
return len(rows)
def write_meta(conn, store, built_from, extra=None):
"""Write `bundle_meta`: format, store, built_from and spec, then any `extra` pairs."""
pairs = {"format": BUNDLE_FORMAT, "store": store, "built_from": built_from, "spec": SPEC_VERSION}
pairs.update(extra or {})
conn.executemany("INSERT INTO bundle_meta VALUES (?, ?)", sorted(pairs.items()))
def load(conn, root, asset_store):
"""Load the package at `root`. Files without tables go to `asset_store`, by SHA-256.
Validate the package first (package.validate); this only refuses paths and hashes that are unsafe to use.
"""
_all_files(root) # refuses a symbolic link anywhere in the package before anything is copied
manifest = read_json(os.path.join(root, "MANIFEST.json"))
listed = {f["path"]: f for f in manifest["files"]}
for path, f in listed.items():
if not safe_path(path):
raise Refusal("package.path.unsafe", "unsafe path %r" % path)
if not isinstance(f.get("sha256"), str) or not SHA256_RE.fullmatch(f["sha256"]):
raise Refusal("package.manifest.schema", "%s: sha256 is not 64 lowercase hex digits" % path)
def jl(name):
return read_jsonl(os.path.join(root, name)) if name in listed else []
d = read_json(os.path.join(root, "document.json"))
key = d["key"]
with open(os.path.join(root, "MANIFEST.json"), "rb") as f:
package_hash = sha256_hex(f.read())
prov = read_json(os.path.join(root, "provenance.json"))
roster = read_json(os.path.join(root, "roster.json")) if "roster.json" in listed else None
conn.execute("INSERT OR IGNORE INTO works VALUES (?, ?)", (d["work"]["key"], J(d["work"]["same_as"])))
ins(conn, "INTO documents", (key, d.get("store"), J(d.get("aliases")), d["kind"], J(d["work"]), J(d["version"]), J(d["relations"]),
J(d["meta"]), d["language"], J(d.get("license")), d["policy"], J(d["hashing"]), d["root"], d["meta_hash"],
d["history"],
d["rev"], J(d["counts"]), J(d["fidelity"]), J(prov), J(roster), package_hash))
for i, w in enumerate(read_json(os.path.join(root, "witnesses.json"))["witnesses"]):
ins(conn, "INTO witnesses", (key, w["id"], i, w["sha256"], w["size"], w["kind"], w["media_type"], w.get("pages"),
B(w.get("text_layer")), w.get("version_label"), J(w["origin"]), J(w["rights"])))
for seq, n in enumerate(jl("nodes.jsonl")):
ins(conn, "INTO nodes", (key, n["id"], seq, n["kind"], n["parent"], n["ord"], n.get("label"), J(n.get("attrs")),
J(n.get("content")), B(n.get("withheld")), n.get("fidelity"), J(n.get("prov")),
n["text_hash"], n["match_hash"], n["align_key"], n["own_hash"], n["node_hash"]))
if isinstance(n.get("content"), list):
ords = 0
for r in n["content"]:
if r["kind"] == "cite":
for ref in r["refs"]:
ins(conn, "OR IGNORE INTO citations", (key, n["id"], ref, ords))
ords += 1
for p in jl("pages.jsonl"):
ins(conn, "INTO pages", (key, p["witness"], p["page"], p["page_fp"], J(p["size"]), p.get("page_label"),
B(p["text_layer"]), J(p.get("stamps_removed"))))
for l in jl("locations.jsonl"):
ins(conn, "INTO locations", (key, l["node"], l["page_fp"], l["page"], l.get("page_label"), J(l["boxes"]), B(l.get("approx"))))
for r in jl("references.jsonl"):
ins(conn, "INTO refs", (key, r["node"], r.get("label"), J(r.get("csl")), J(r["resolution"])))
main = ("format", "index", "id", "op", "target", "evidence", "author", "reason", "at", "before", "after", "path",
"touched")
for c in jl("corrections.jsonl"):
details = {k: v for k, v in c.items() if k not in main}
ins(conn, "INTO corrections", (key, c["index"], c["id"], c["op"], c.get("target"), J(c["evidence"]), J(c["author"]),
c["reason"], c["at"], c["before"], c["after"], J(c["path"]), J(c["touched"]), J(details)))
for e in jl("ids.jsonl"):
ins(conn, "INTO node_ids", (key, e["id"], e["kind"], e["successor"], e.get("node_kind"), e.get("rev")))
for r in jl("receipts.jsonl"):
if r["format"] == "scienting-receipt/1":
ins(conn, "INTO receipts", (key, r["id"], r["level"], r["scope"], r["rev"], r["root"], J(r["nodes"]),
J(r.get("records")), J(r["evidence"]), r["by"], r["at"], J(r.get("qa")),
r.get("export_of"), J(r.get("keyed")), J(r["signature"])))
for nid, h in r["nodes"]:
ins(conn, "INTO receipt_nodes", (r["id"], key, nid, h))
else:
per = r.get("period") or {}
ins(conn, "INTO revocations", (key, r["id"], r.get("receipt"), r.get("signer"), per.get("from"), per.get("to"),
r["reason"], r["by"], r["at"], J(r["signature"])))
for s in jl("suspects.jsonl"):
ins(conn, "INTO suspects", (key, s["id"], s["kind"], s.get("node"), J(s)))
os.makedirs(asset_store, exist_ok=True)
for path, f in listed.items():
if path in STRUCTURED:
continue
present = f.get("present", True)
ins(conn, "INTO package_files", (key, path, f["sha256"], f["size"], 1 if present else 0))
if present:
shutil.copyfile(os.path.join(root, path), os.path.join(asset_store, f["sha256"]))
return key
def _q(conn, sql, args):
cur = conn.execute(sql, args)
cols = [c[0] for c in cur.description]
return [dict(zip(cols, row)) for row in cur.fetchall()]
def unload(conn, key, out, asset_store):
"""Write the package for `key` from the tables into `out`. Returns the package hash."""
files = {}
d = _q(conn, "SELECT * FROM documents WHERE doc_key = ?", (key,))[0]
doc = {"format": "scienting-doc/1", "key": key}
_opt(doc, "store", d["store"])
_opt(doc, "aliases", P(d["aliases"]))
doc.update({"kind": d["kind"], "work": P(d["work"]), "version": P(d["version"]), "relations": P(d["relations"]),
"meta": P(d["meta"]), "language": d["language"]})
_opt(doc, "license", P(d["license"]))
doc.update({"policy": d["policy"], "hashing": P(d["hashing"]), "root": d["root_hash"],
"meta_hash": d["meta_hash"], "history": d["history"], "rev": d["rev"], "counts": P(d["counts"]),
"fidelity": P(d["fidelity"])})
def put(name, rec):
files[name] = (jcs.dumps(rec) + "\n").encode("utf-8")
def putl(name, recs):
if recs:
files[name] = "".join(jcs.dumps(r) + "\n" for r in recs).encode("utf-8")
put("document.json", doc)
put("provenance.json", P(d["provenance"]))
if d["roster"] is not None:
put("roster.json", P(d["roster"]))
ws = []
for w in _q(conn, "SELECT * FROM witnesses WHERE doc_key = ? ORDER BY ord", (key,)):
rec = {"id": w["id"], "sha256": w["sha256"], "size": w["size"], "kind": w["kind"], "media_type": w["media_type"]}
_opt(rec, "pages", w["pages"])
_opt(rec, "text_layer", None if w["text_layer"] is None else bool(w["text_layer"]))
_opt(rec, "version_label", w["version_label"])
rec.update({"origin": P(w["origin"]), "rights": P(w["rights"])})
ws.append(rec)
put("witnesses.json", {"witnesses": ws})
nodes = []
for n in _q(conn, "SELECT * FROM nodes WHERE doc_key = ? ORDER BY seq", (key,)):
rec = {"id": n["id"], "kind": n["kind"], "parent": n["parent"], "ord": n["ord"]}
_opt(rec, "label", n["label"])
_opt(rec, "attrs", P(n["attrs"]))
_opt(rec, "content", P(n["content"]))
if n["withheld"]:
rec["withheld"] = True
_opt(rec, "fidelity", n["fidelity"])
_opt(rec, "prov", P(n["prov"]))
for h in ("text_hash", "match_hash", "align_key", "own_hash", "node_hash"):
rec[h] = n[h]
nodes.append(rec)
putl("nodes.jsonl", nodes)
seq = {n["id"]: i for i, n in enumerate(nodes)}
pages = []
for p in _q(conn, "SELECT * FROM pages WHERE doc_key = ? ORDER BY witness, page", (key,)):
rec = {"witness": p["witness"], "page": p["page"], "page_fp": p["page_fp"], "size": P(p["size"])}
_opt(rec, "page_label", p["page_label"])
rec["text_layer"] = bool(p["text_layer"])
_opt(rec, "stamps_removed", P(p["stamps_removed"]))
pages.append(rec)
putl("pages.jsonl", pages)
locs = []
for l in _q(conn, "SELECT * FROM locations WHERE doc_key = ?", (key,)):
rec = {"node": l["node"], "page_fp": l["page_fp"], "page": l["page"]}
_opt(rec, "page_label", l["page_label"])
rec["boxes"] = P(l["boxes"])
if l["approx"]:
rec["approx"] = True
locs.append(rec)
locs.sort(key=lambda r: (seq[r["node"]], r["page"], r["page_fp"]))
putl("locations.jsonl", locs)
refs = []
for r in _q(conn, "SELECT * FROM refs WHERE doc_key = ?", (key,)):
rec = {"node": r["node"]}
_opt(rec, "label", r["label"])
_opt(rec, "csl", P(r["csl"]))
rec["resolution"] = P(r["resolution"])
refs.append(rec)
refs.sort(key=lambda r: seq[r["node"]])
putl("references.jsonl", refs)
cors = []
for c in _q(conn, "SELECT * FROM corrections WHERE doc_key = ? ORDER BY idx", (key,)):
rec = {"format": "scienting-correction/1", "index": c["idx"], "id": c["id"], "op": c["op"]}
_opt(rec, "target", c["target"])
rec.update({"evidence": P(c["evidence"]), "author": P(c["author"]), "reason": c["reason"], "at": c["made_at"],
"before": c["before_hash"], "after": c["after_hash"], "path": P(c["path"]),
"touched": P(c["touched"])})
rec.update(P(c["details"]))
cors.append(rec)
putl("corrections.jsonl", cors)
ids = []
for e in _q(conn, "SELECT * FROM node_ids WHERE doc_key = ?", (key,)):
rec = {"id": e["id"], "kind": e["kind"], "successor": e["successor"]}
_opt(rec, "node_kind", e["node_kind"])
_opt(rec, "rev", e["rev"])
ids.append(rec)
ids.sort(key=lambda e: (0 if e["kind"] == "alias" else 1, e.get("rev", 0), e["id"]))
putl("ids.jsonl", ids)
recs = []
for r in _q(conn, "SELECT * FROM receipts WHERE doc_key = ?", (key,)):
rec = {"format": "scienting-receipt/1", "id": r["id"], "doc": key, "rev": r["rev"], "level": r["level"],
"scope": r["scope"], "nodes": P(r["nodes"])}
_opt(rec, "records", P(r["records"]))
rec.update({"root": r["root"], "evidence": P(r["evidence"]), "by": r["signer"], "at": r["made_at"]})
_opt(rec, "qa", P(r["qa"]))
_opt(rec, "export_of", r["export_of"])
_opt(rec, "keyed", P(r["keyed"]))
rec["signature"] = P(r["signature"])
recs.append(rec)
for r in _q(conn, "SELECT * FROM revocations WHERE doc_key = ?", (key,)):
rec = {"format": "scienting-revocation/1", "id": r["id"]}
_opt(rec, "receipt", r["receipt"])
_opt(rec, "signer", r["revoked_signer"])
if r["period_from"] is not None:
rec["period"] = {"from": r["period_from"], "to": r["period_to"]}
rec.update({"reason": r["reason"], "by": r["signer"], "at": r["made_at"], "signature": P(r["signature"])})
recs.append(rec)
recs.sort(key=lambda r: (r["at"], r["id"]))
putl("receipts.jsonl", recs)
sus = [P(s["record"]) for s in _q(conn, "SELECT * FROM suspects WHERE doc_key = ?", (key,))]
sus.sort(key=lambda s: s["id"])
putl("suspects.jsonl", sus)
absent = []
for f in _q(conn, "SELECT * FROM package_files WHERE doc_key = ?", (key,)):
if f["present"]:
if not SHA256_RE.fullmatch(f["sha256"]):
raise Refusal("package.manifest.schema", "%s: sha256 is not 64 lowercase hex digits" % f["path"])
with open(os.path.join(asset_store, f["sha256"]), "rb") as fh:
files[f["path"]] = fh.read()
else:
absent.append({"path": f["path"], "sha256": f["sha256"], "size": f["size"], "present": False})
return write(out, files, absent)
tools/oracle/fingerprint.py
-66
View File
@@ -1,66 +0,0 @@
"""Page fingerprints, scienting-page/1, and download-stamp rules, scienting-stamps/1.
Non-normative; see spec/07-geometry.md.
"""
import math
import re
import unicodedata
from . import jcs
from .notation import fold_plain_text
from .text import is_ws, norm, sha256_hex
def round_point(x):
"""Round half up to a whole point."""
return int(math.floor(x + 0.5))
def _in_margin(box, size, margin):
x0, y0, x1, y1 = box
w, h = size
return x1 <= margin or x0 >= w - margin or y1 <= margin or y0 >= h - margin
def is_stamp(line, size, rules):
text = norm(line["text"])
for r in rules:
if r.get("where", "any") == "margin":
if "box" not in line or not _in_margin(line["box"], size, r.get("margin", 36)):
continue
if re.search(r["pattern"], text, re.IGNORECASE | re.ASCII):
return r["id"]
return None
def _strip(s):
return "".join(c for c in s if not is_ws(c) and unicodedata.category(c) not in ("Cc", "Cf"))
def fp_text(text):
"""Steps 3 and 4 of spec/07 §7.1: whitespace and control characters go first, so that "10 -20" and "10-20"
fold alike; then scripts and dashes fold as in the strict projection, then NFKC."""
return _strip(unicodedata.normalize("NFKC", fold_plain_text(_strip(text))))
def fingerprint(page, rules):
"""`page`: {"size": [w, h], "lines": [{"text", "box"?}], "images": [sha256...]}.
Returns (fingerprint, removed stamp rule IDs in line order).
"""
size = page["size"]
kept, removed = [], []
for line in page.get("lines", []):
rid = is_stamp(line, size, rules)
if rid:
removed.append(rid)
else:
kept.append(line["text"])
payload = {
"format": "scienting-page/1",
"size": [round_point(size[0]), round_point(size[1])],
"text": fp_text(" ".join(kept)),
"images": sorted(page.get("images", [])),
}
return sha256_hex(jcs.dumpb(payload)), removed
tools/oracle/grade.py
-93
View File
@@ -1,93 +0,0 @@
"""Grading a scienting-submission/1 against a scienting-exercise/1 key.
Non-normative; see spec/19-exercises.md.
"""
from decimal import Decimal
from fractions import Fraction
from . import anchors as A
from .tree import Refusal
def dec(d):
"""A decimal string, exactly as the sum came out (spec/19 §19.4: points are decimal strings, summed exactly)."""
return format(Decimal(d), "f")
def _zero_like(d):
"""Zero with the scale of `d`, so a missed element worth "0.5" scores "0.0"."""
return Decimal(0).quantize(Decimal(d))
def _resolved(state, anchor):
r = A.resolve(state, anchor)
if r["result"] == "stale":
return None
return r["node"], r.get("span")
def _match(state, rule, student, key):
s, k = _resolved(state, student), _resolved(state, key)
if not s or not k or s[0] != k[0]:
return False
if rule.get("match") == "overlap" and k[1] is not None:
if s[1] is None:
return True
lo, hi = max(s[1][0], k[1][0]), min(s[1][1], k[1][1])
size = k[1][1] - k[1][0]
return size > 0 and Fraction(max(0, hi - lo), size) >= Fraction(rule.get("min_overlap", "0.5"))
return True
def _any(state, rule, students, keys):
return any(_match(state, rule, s, k) for s in students for k in keys)
def key_anchors(body):
key = body["key"]
out = []
for el in key.get("elements", []) + key.get("chain", []) + key.get("errors", []):
out += el["anchors"]
return out
def grade(body, submission, state):
for a in key_anchors(body):
if A.resolve(state, a)["result"] == "stale":
raise Refusal("grade.key-stale", "an anchor in the key is stale")
kind, key, rule, ans = body["exercise"], body["key"], body.get("rule", {}), submission["answers"]
items = []
def item(name, pts, got, fb):
items.append({"element": name, "points": dec(pts if got else _zero_like(pts)), "max": dec(pts), "feedback": fb})
if kind in ("anchor", "check"):
if kind == "check":
v = key["verdict"]
ok = ans.get("verdict") == v["value"]
item("verdict", v["points"], ok, v.get("feedback", {}).get("hit" if ok else "miss", ""))
for i, el in enumerate(key["elements"]):
ok = _any(state, rule, ans.get("anchors", []), el["anchors"])
item("element:%d" % i, el["points"], ok, el.get("feedback", {}).get("hit" if ok else "miss", ""))
elif kind == "trace":
chain = ans.get("chain", [])
for i, link in enumerate(key["chain"]):
ok = i < len(chain) and _any(state, rule, chain[i], link["anchors"])
item("link:%d" % i, link["points"], ok, link.get("feedback", {}).get("hit" if ok else "miss", ""))
elif kind == "reproduce":
ok = abs(Decimal(ans["value"]) - Decimal(key["value"])) <= Decimal(key["tolerance"])
item("value", key["points"], ok, key.get("feedback", {}).get("hit" if ok else "miss", ""))
elif kind == "verify":
flags = ans.get("flags", [])
for i, err in enumerate(key["errors"]):
hit = [f for f in flags if _any(state, rule, [f["anchor"]], err["anchors"])]
ok = any(f["verdict"] == err["verdict"] for f in hit)
fb = err.get("feedback", {}).get("hit" if ok else "miss", "")
item("error:%d" % i, err["points"], ok, fb)
else:
raise Refusal("grade.kind", "unknown exercise kind " + kind)
score = sum((Decimal(x["points"]) for x in items), Decimal(0))
total = sum((Decimal(x["max"]) for x in items), Decimal(0))
return {"format": "scienting-grade/1", "submission": submission["id"], "exercise": submission["exercise"],
"score": dec(score), "max": dec(total), "items": items}
tools/oracle/hashing.py
-320
View File
@@ -1,320 +0,0 @@
"""The plain, strict and lossy projections, and the five content values.
Non-normative; see spec/05-hashes.md.
"""
import hashlib
import hmac
import unicodedata
from . import jcs
from .model import LABELLED, MATH_KINDS, RUN_KINDS, VERBATIM_KINDS
from .notation import complex_key, fold_math, fold_plain_text, fold_styled_script
from .text import collapse_ws, is_ws, norm, remove_ws, trim
OBJ = "\ufffc"
MARK_OPEN = "\ufff9"
MARK_CLOSE = "\ufffb"
class Hasher:
"""SHA-256, or HMAC-SHA-256 under a key: every content value uses the same function."""
def __init__(self, key=None):
self.key = key
def __call__(self, data):
if isinstance(data, str):
data = data.encode("utf-8")
if self.key is None:
return hashlib.sha256(data).hexdigest()
return hmac.new(self.key, data, hashlib.sha256).hexdigest()
PLAIN = Hasher()
# --- runs --------------------------------------------------------------------
def _eq_source(doc, run):
eq = doc.nodes[run["node"]]
return eq["content"]["source"], eq["content"]["notation"]
def plain_runs(runs, doc=None):
out = []
for r in runs:
k = r["kind"]
if k in ("text", "link", "cite", "xref", "fnmark"):
out.append(r["text"])
elif k == "math":
out.append("$" + r["source"] + "$")
elif k == "equation":
src, _ = _eq_source(doc, r)
out.append("$$" + src + "$$")
else:
raise ValueError("run kind " + k)
return "".join(out)
def _strict_math(notation, source, complexes):
folded = fold_math(notation, source)
if folded is None:
complexes.append(complex_key(notation, source))
return OBJ
return folded
def _finish_strict(s):
s = unicodedata.normalize("NFKC", s)
return trim(collapse_ws(s))
def strict_runs_raw(runs, doc, complexes):
out = []
for r in runs:
k = r["kind"]
if k == "text":
st = r.get("style", [])
if "sup" in st:
out.append(fold_styled_script("^", r["text"]))
elif "sub" in st:
out.append(fold_styled_script("_", r["text"]))
else:
out.append(fold_plain_text(r["text"]))
elif k == "link":
out.append(fold_plain_text(r["text"]))
elif k == "math":
out.append(_strict_math(r["notation"], r["source"], complexes))
elif k == "equation":
src, notation = _eq_source(doc, r)
out.append(_strict_math(notation, src, complexes))
elif k in ("cite", "xref", "fnmark"):
out.append(MARK_OPEN + remove_ws(unicodedata.normalize("NFC", r["text"])) + MARK_CLOSE)
else:
raise ValueError("run kind " + k)
return "".join(out)
def lossy_from_strict(s):
out = []
stack = []
i, n = 0, len(s)
while i < n:
ch = s[i]
if ch in "_^" and i + 1 < n and s[i + 1] == "{":
stack.append("s")
i += 2
continue
if ch == "{":
stack.append("t")
out.append(ch)
elif ch == "}":
if stack:
if stack.pop() == "t":
out.append(ch)
else:
out.append(ch)
elif ch == MARK_OPEN:
j = s.find(MARK_CLOSE, i)
i = n if j < 0 else j + 1
continue
else:
out.append(ch)
i += 1
t = unicodedata.normalize("NFKC", "".join(out).casefold())
return "".join(
c for c in t if not is_ws(c) and unicodedata.category(c) != "Pd" and c != "\u00ad"
)
def prose_length(lossy):
return sum(1 for c in lossy if c != OBJ)
# --- nodes -------------------------------------------------------------------
def plain(doc, nid):
n = doc.nodes[nid]
k = n["kind"]
if k in RUN_KINDS:
return plain_runs(n["content"], doc)
if k in MATH_KINDS:
return "$$" + n["content"]["source"] + "$$"
if k in VERBATIM_KINDS:
return n["content"]["text"]
return " ".join(plain(doc, x) for x in doc.text_leaves(nid))
def strict(doc, nid):
"""The strict canonical projection and the node's complex math keys, in order."""
n = doc.nodes[nid]
k = n["kind"]
complexes = []
if k in RUN_KINDS:
s = strict_runs_raw(n["content"], doc, complexes)
elif k in MATH_KINDS:
s = _strict_math(n["content"]["notation"], n["content"]["source"], complexes)
elif k in VERBATIM_KINDS:
s = n["content"]["text"]
else:
parts = []
for x in doc.text_leaves(nid):
p, c = strict(doc, x)
parts.append(p)
complexes.extend(c)
return _finish_strict(" ".join(parts)), complexes
return _finish_strict(s), complexes
def lossy(doc, nid):
return lossy_from_strict(strict(doc, nid)[0])
def _runs_for_own(runs):
out = []
for r in runs:
if r["kind"] == "equation":
out.append({"kind": "equation"})
else:
out.append(r)
return out
def own_content(doc, nid, text_hash_of=None):
n = doc.nodes[nid]
k = n["kind"]
o = {"kind": k}
attrs = n.get("attrs", {})
if k in LABELLED:
o["label"] = n.get("label")
if k in RUN_KINDS:
o["runs"] = _runs_for_own(n["content"])
if k in MATH_KINDS:
o["notation"] = n["content"]["notation"]
o["source"] = n["content"]["source"]
if k in VERBATIM_KINDS:
o["language"] = n["content"].get("language")
o["text"] = n["content"]["text"]
if k == "section":
o["level"] = attrs["level"]
heads = [c for c in doc.kids(nid) if doc.nodes[c]["kind"] == "heading"]
o["heading"] = text_hash_of(heads[0]) if heads else None
if k == "table":
o["columns"] = attrs.get("columns", [])
if k == "cell":
o["column"] = attrs.get("column")
o["rowspan"] = attrs.get("rowspan", 1)
o["colspan"] = attrs.get("colspan", 1)
o["role"] = attrs.get("role", "data")
if k == "theorem":
o["variant"] = attrs.get("variant")
return o
class _Entry:
"""One node's cached work: its five values, and the projections its parent joins."""
__slots__ = ("sig", "values", "plain", "joined_plain", "joined_strict", "complexes")
def __init__(self, sig, values, plain, joined_plain, joined_strict, complexes):
self.sig = sig
self.values = values
self.plain = plain
# What this node gives a container's projections: its text leaves joined, or None when it has none.
self.joined_plain = joined_plain
self.joined_strict = joined_strict
self.complexes = complexes
def _sig(doc, nid):
"""Everything a node's values depend on besides its children's values."""
n = doc.nodes[nid]
return (n["kind"], n.get("label"), n.get("attrs"), n.get("content"), tuple(doc.kids(nid)))
class Index:
"""The five values of every node of one document, kept so that a change rehashes only what it touched.
A node's values depend only on its own subtree, so after a change only the nodes whose own record or
children changed, and their ancestors, are rehashed. A container's projections are its children's
cached projections joined, so no container walks its subtree again.
"""
def __init__(self, hasher=PLAIN):
self.hasher = hasher
self.entries = {}
self.values = {}
@classmethod
def build(cls, doc, hasher=PLAIN):
ix = cls(hasher)
ix._rehash(doc, set(doc.nodes))
return ix
def update(self, doc):
"""A new Index for `doc`, a later state of this one's document; this Index is left unchanged."""
ix = Index(self.hasher)
ix.entries = {nid: e for nid, e in self.entries.items() if nid in doc.nodes}
ix.values = {nid: v for nid, v in self.values.items() if nid in doc.nodes}
dirty = {nid for nid in doc.nodes if nid not in ix.entries or ix.entries[nid].sig != _sig(doc, nid)}
for nid in list(dirty):
p = doc.nodes[nid].get("parent")
while p is not None and p not in dirty:
dirty.add(p)
p = doc.nodes[p].get("parent")
ix._rehash(doc, dirty)
return ix
def plain(self, nid):
"""The node's plain projection."""
return self.entries[nid].plain
def _rehash(self, doc, dirty):
h = self.hasher
for nid in reversed(doc.preorder()):
if nid not in dirty:
continue
n = doc.nodes[nid]
k = n["kind"]
kids = doc.kids(nid)
if k in RUN_KINDS or k in MATH_KINDS or k in VERBATIM_KINDS:
p = plain(doc, nid)
st, cx = strict(doc, nid)
# An equation line is never a text leaf; a sentence's equations are part of its own runs.
joined = (None, None, []) if k == "equation-line" else (p, st, cx)
else:
ps, ss, cx = [], [], []
for c in kids:
e = self.entries[c]
if e.joined_plain is not None:
ps.append(e.joined_plain)
ss.append(e.joined_strict)
cx.extend(e.complexes)
p = " ".join(ps)
st = _finish_strict(" ".join(ss))
joined = (p, " ".join(ss), cx) if ps else (None, None, [])
own = h(jcs.dumpb(own_content(doc, nid, lambda x: self.values[x]["text_hash"])))
self.values[nid] = {
"text_hash": h(norm(p)),
"match_hash": h(st),
"align_key": h(lossy_from_strict(st)),
"own_hash": own,
"node_hash": h(jcs.dumpb({"own": own, "children": [self.values[c]["node_hash"] for c in kids]})),
}
self.entries[nid] = _Entry(_sig(doc, nid), self.values[nid], p, *joined)
def compute(doc, hasher=PLAIN):
"""The five content values of every node, bottom up. Returns {id: {...}}."""
return Index.build(doc, hasher).values
def meta_object(document):
return {k: document.get(k) for k in ("work", "version", "relations", "meta", "language", "license")}
def meta_hash(document, hasher=PLAIN):
return hasher(jcs.dumpb(meta_object(document)))
tools/oracle/jcs.py
-155
View File
@@ -1,155 +0,0 @@
"""Canonical JSON, RFC 8785 (JSON Canonicalization Scheme).
Non-normative.
The normative rule is spec/01-conventions.md.
Most values serialize through the standard library's C encoder, which escapes strings exactly as RFC 8785 does.
Values it would write differently take the exact path below: floats, integers beyond 2^53, and member names
outside the Basic Multilingual Plane, which RFC 8785 sorts by UTF-16 code units rather than by code points.
"""
import json
import math
_C = json.JSONEncoder(ensure_ascii=False, separators=(",", ":"), sort_keys=True, allow_nan=False)
_EXACT_INT = 2**53
_ESC = {
0x08: "\\b",
0x09: "\\t",
0x0A: "\\n",
0x0C: "\\f",
0x0D: "\\r",
0x22: '\\"',
0x5C: "\\\\",
}
def _string(s):
out = ['"']
for ch in s:
cp = ord(ch)
if 0xD800 <= cp <= 0xDFFF:
raise ValueError("lone surrogate in string")
if cp in _ESC:
out.append(_ESC[cp])
elif cp < 0x20:
out.append("\\u%04x" % cp)
else:
out.append(ch)
out.append('"')
return "".join(out)
def _number(x):
"""ECMAScript Number.prototype.toString, as RFC 8785 requires."""
if isinstance(x, bool):
raise TypeError("bool is not a number")
if isinstance(x, int):
if abs(x) > 2**53:
raise ValueError("integer outside the IEEE 754 exact range")
return str(x)
if not math.isfinite(x):
raise ValueError("NaN and Infinity are not JSON")
if x == 0:
return "0"
if x < 0:
return "-" + _number(-x)
# Shortest round-trip digits, as ECMAScript requires.
r = repr(x)
if "e" in r or "E" in r:
mant, exp = r.lower().split("e")
exp = int(exp)
else:
mant, exp = r, 0
if "." in mant:
ip, fp = mant.split(".")
else:
ip, fp = mant, ""
digits = (ip + fp).lstrip("0")
# Position of the decimal point relative to the start of `digits`.
lead_zeros = len(ip + fp) - len((ip + fp).lstrip("0"))
n = len(ip) - lead_zeros + exp
digits = digits.rstrip("0") or "0"
k = len(digits)
if k <= n <= 21:
return digits + "0" * (n - k)
if 0 < n <= 21:
return digits[:n] + "." + digits[n:]
if -6 < n <= 0:
return "0." + "0" * (-n) + digits
e = n - 1
sign = "+" if e >= 0 else "-"
if k == 1:
return digits + "e" + sign + str(abs(e))
return digits[0] + "." + digits[1:] + "e" + sign + str(abs(e))
def _key(s):
# RFC 8785 sorts member names by their UTF-16 code units.
return s.encode("utf-16-be")
def _fast(v):
"""True when the C encoder writes `v` exactly as RFC 8785 does."""
t = type(v)
if t is str or v is None or t is bool:
return True
if t is int:
return -_EXACT_INT <= v <= _EXACT_INT
if t is dict:
for k, x in v.items():
if type(k) is not str or (not k.isascii() and max(k) > "\uffff") or not _fast(x):
return False
return True
if t is list or t is tuple:
return all(_fast(x) for x in v)
return False
def dumps(value):
"""Serialize `value` to canonical JSON text."""
if _fast(value):
text = _C.encode(value)
if not text.isascii():
try:
text.encode("utf-8")
except UnicodeEncodeError:
raise ValueError("lone surrogate in string") from None
return text
return _exact(value)
def _exact(value):
if value is None:
return "null"
if value is True:
return "true"
if value is False:
return "false"
if isinstance(value, str):
return _string(value)
if isinstance(value, (int, float)):
return _number(value)
if isinstance(value, (list, tuple)):
return "[" + ",".join(dumps(v) for v in value) + "]"
if isinstance(value, dict):
for k in value:
if not isinstance(k, str):
raise TypeError("object keys must be strings")
items = sorted(value.items(), key=lambda kv: _key(kv[0]))
return "{" + ",".join(_string(k) + ":" + dumps(v) for k, v in items) + "}"
raise TypeError("not JSON: %r" % type(value))
def dumpb(value):
"""Canonical JSON as UTF-8 bytes."""
return dumps(value).encode("utf-8")
def loads_line(data):
"""Parse one canonical JSON line of bytes, refusing a line that is not canonical."""
import json as _json
value = _json.loads(data.decode("utf-8"))
if dumpb(value) != data:
raise ValueError("not canonical JSON")
return value
tools/oracle/keys.py
-161
View File
@@ -1,161 +0,0 @@
"""Work and document keys: a Scholar-style citation key plus a short hash of the work's identity.
Non-normative; see spec/02-model.md §2.4.
A work key looks like `vaswani2017attention-k3f9`. The first part is what Google Scholar would call the work; the
characters after the hyphen come from a hash of the work's identity, so two works Scholar would name alike never
share a key. A document key is the work key with a suffix for anything other than the version of record.
"""
import hashlib
import re
import unicodedata
from .text import ALPHABET
SCHEME = "scienting-key/1"
SUFFIX_LENGTH = 4
MAX_SUFFIX = 51
IDENTIFIER_ORDER = ("doi", "arxiv", "isbn", "pmid", "pmcid", "openalex")
WORK_KEY_RE = re.compile(r"^[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}$")
DOCUMENT_KEY_RE = re.compile(r"^[a-z][a-z0-9]*-[0-9a-hjkmnp-tv-z]{4,51}(\.[a-z0-9][a-z0-9-]*)*$")
STOP_WORDS = frozenset("a an the of on in for to and or at by with from as into onto about toward towards via "
"is are be its it this that these those".split())
NAME_SUFFIXES = frozenset(["jr", "sr", "ii", "iii", "iv", "v", "phd", "md"])
_YEAR = re.compile(r"^\d{4}$")
_DOI_PREFIX = re.compile(r"^(?:doi:|https?://(?:dx\.|www\.)?doi\.org/)+", re.IGNORECASE)
def fold(text):
"""Lower-case ASCII with accents removed: `Müller` gives `muller`."""
raw = text if isinstance(text, str) else ""
return unicodedata.normalize("NFKD", raw).encode("ascii", "ignore").decode().lower()
def title_words(title):
"""A title's words, folded, with apostrophes removed and other punctuation splitting."""
t = re.sub(r"['’`]", "", fold(title))
return re.findall(r"[a-z0-9]+", t)
def family_name(name):
"""A best guess at the family name in a display name: the part before a comma, else the last word that is
not a suffix such as `Jr.` or `III`."""
name = (name or "").strip()
if "," in name:
head, _, tail = name.partition(",")
if tail.strip().rstrip(".").lower() not in NAME_SUFFIXES or " " not in head.strip():
return head.strip()
name = head
parts = name.split()
while len(parts) > 1 and parts[-1].rstrip(".").lower() in NAME_SUFFIXES:
parts.pop()
return parts[-1] if parts else ""
def first_family(meta):
"""The first author's family name from a CSL-JSON item, as the key spells it, or `anon`."""
authors = meta.get("author") or []
if not authors:
return "anon"
a = authors[0]
name = a.get("family") or family_name(a.get("literal") or a.get("name") or "")
fam = re.sub(r"[^a-z]", "", fold(name))
return fam or "anon"
def year_of(meta):
"""The year of a CSL-JSON item's `issued` date, when it is a four-digit year, else the empty string."""
parts = (meta.get("issued") or {}).get("date-parts") or []
y = parts[0][0] if parts and parts[0] else None
s = str(y) if isinstance(y, int) and not isinstance(y, bool) else (y if isinstance(y, str) else "")
return s if _YEAR.match(s) else ""
def scholar_part(meta):
"""Family name, year and the first title word that is not a stop word: `vaswani2017attention`."""
word = next((w for w in title_words(meta.get("title")) if w not in STOP_WORDS), "")
return first_family(meta) + year_of(meta) + word
def norm_doi(value):
"""A DOI in bare form with ASCII letters lower-cased (DOI Handbook 2.4)."""
s = _DOI_PREFIX.sub("", value.strip()).strip()
return s.translate(str.maketrans("ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz"))
def identity(same_as, meta):
"""What the key's hash is computed over: the first identifier of IDENTIFIER_ORDER among `same_as`, else the
folded metadata `meta:<title words>|<year>|<family>`."""
by_kind = {}
for s in same_as or []:
kind, _, value = s.partition(":")
by_kind.setdefault(kind, value)
for kind in IDENTIFIER_ORDER:
if kind in by_kind and by_kind[kind]:
value = norm_doi(by_kind[kind]) if kind == "doi" else by_kind[kind].strip()
return "%s:%s" % (kind, value)
return "meta:%s|%s|%s" % (" ".join(title_words(meta.get("title"))), year_of(meta), first_family(meta))
def suffix(ident, length=SUFFIX_LENGTH):
"""The leading 5 × `length` bits of SHA-256 of `scienting-key/1:` and the identity, in Crockford base32."""
if not 1 <= length <= MAX_SUFFIX:
raise ValueError("a key suffix is 1 to %d characters" % MAX_SUFFIX)
digest = int.from_bytes(hashlib.sha256((SCHEME + ":" + ident).encode("utf-8")).digest(), "big")
bits = digest >> (256 - 5 * length)
return "".join(ALPHABET[(bits >> (5 * (length - 1 - i))) & 31] for i in range(length))
def work_key(same_as, meta, length=SUFFIX_LENGTH):
return scholar_part(meta) + "-" + suffix(identity(same_as, meta), length)
def unique_work_key(same_as, meta, taken):
"""The work key, lengthened by two characters at a time while another identity already holds it."""
for length in range(SUFFIX_LENGTH, MAX_SUFFIX + 1, 2):
key = work_key(same_as, meta, length)
if key not in taken:
return key
raise ValueError("no free key for " + identity(same_as, meta))
def version_suffix(version):
"""The document key's suffix for a version: nothing for the version of record, `.si` for Supporting
Information, `.v2` for a preprint or draft with a sequence number, else the version type."""
t = version.get("type")
if t == "version-of-record":
return ""
if t == "supplement":
return ".si"
seq = version.get("seq")
if t in ("preprint", "draft", "submitted", "accepted") and isinstance(seq, int):
return ".v%d" % seq
return "." + t
def document_key(wkey, version):
return wkey + version_suffix(version)
def split_key(dkey):
"""A document key's work key and its suffix (with the leading dot, or empty)."""
m = WORK_KEY_RE.match(dkey.split(".", 1)[0]) if dkey else None
if not m or not DOCUMENT_KEY_RE.match(dkey):
raise ValueError("%r is not a document key" % dkey)
head, dot, tail = dkey.partition(".")
return head, (dot + tail if dot else "")
def suffix_of(wkey):
return wkey.rsplit("-", 1)[1]
def suffix_agrees(wkey, same_as, meta):
"""Whether a stored work key's hash part is what the identity gives: always checked when the identity is an
identifier such as a DOI, which never changes; a metadata identity may have been corrected since, so it
is not held to the key."""
ident = identity(same_as, meta)
if ident.startswith("meta:"):
return True
sfx = suffix_of(wkey)
return sfx == suffix(ident, len(sfx))
tools/oracle/log.py
-204
View File
@@ -1,204 +0,0 @@
"""A team's transparency log of receipts, revocations and roster changes, and its signed checkpoints.
Non-normative; see spec/23-log.md.
Entries are the items of an RFC 9162 tree (the same tree as receipts and the correction history), and a
checkpoint is a signed note in the format transparency logs share (C2SP `signed-note` and `tlog-checkpoint`):
an origin line, the tree size, the root hash in Base64, an empty line, and one signature line per signer.
"""
import base64
import hashlib
import os
from . import crypto, jcs
ALGORITHM_ED25519 = 0x01
FORMAT = "scienting-checkpoint/1"
def entry(record):
"""The log item for a signed record: its kind, its ID and its record hash."""
kinds = {"scienting-receipt/1": "receipt", "scienting-revocation/1": "revocation", "scienting-roster/1": "roster"}
kind = kinds.get(record.get("format"))
if kind is None:
raise ValueError("only receipts, revocations and rosters are logged")
if kind == "roster":
return ["roster", record.get("store", ""), hashlib.sha256(jcs.dumpb(record)).hexdigest()]
body = {k: v for k, v in record.items() if k not in ("id", "signature")}
return [kind, record["id"], hashlib.sha256(jcs.dumpb(body)).hexdigest()]
def root(entries):
return crypto.history_root(entries)
def key_hash(name, public):
"""The four-byte key ID of a signed-note signer: SHA-256 of the name, a newline, the algorithm byte and the
public key, truncated."""
return hashlib.sha256(name.encode("utf-8") + b"\n" + bytes([ALGORITHM_ED25519]) + public).digest()[:4]
def _b64(text):
"""Canonical standard Base64 only: the bytes, or None when the text is not exactly their encoding."""
try:
raw = base64.b64decode(text, validate=True)
except (ValueError, TypeError):
return None
return raw if base64.b64encode(raw).decode("ascii") == text else None
def note_body(origin, size, root_hex):
return "%s\n%d\n%s\n" % (origin, size, base64.b64encode(bytes.fromhex(root_hex)).decode("ascii"))
def sign_checkpoint(origin, entries, signers):
"""A checkpoint note over `entries`, signed by each (name, secret) in `signers`."""
body = note_body(origin, len(entries), root(entries))
lines = []
for name, secret in signers:
public = crypto.ed25519_public(secret)
sig = crypto.ed25519_sign(secret, body.encode("utf-8"))
lines.append("— %s %s" % (name, base64.b64encode(key_hash(name, public) + sig).decode("ascii")))
return body + "\n" + "".join(ln + "\n" for ln in lines)
def parse_checkpoint(text):
"""The origin, size, root (hex) and [(name, key hash, signature)] of a note, or None when malformed."""
try:
body, _, sigs = text.partition("\n\n")
origin, size, root_b64 = body.split("\n")
rt = _b64(root_b64)
if rt is None or len(rt) != 32 or not size.isdigit() or str(int(size)) != size or not origin:
return None
out = []
for ln in sigs.splitlines():
if not ln.startswith("— "):
return None
_, name, blob = ln.split(" ", 2)
raw = _b64(blob)
if raw is None or len(raw) != 68:
return None
out.append((name, raw[:4], raw[4:]))
return {"origin": origin, "size": int(size), "root": rt.hex(), "signatures": out}
except (ValueError, TypeError):
return None
def verify_checkpoint(text, roster):
"""The signers of the note whose Ed25519 signatures verify under the roster's keys of those names.
Returns (parsed note, [names verified]); the note is None when malformed. Never raises.
"""
note = parse_checkpoint(text)
if note is None:
return None, []
body = note_body(note["origin"], note["size"], note["root"]).encode("utf-8")
verified = []
for name, kh, sig in note["signatures"]:
key = roster.get(name)
if not key or key.get("scheme") != "ed25519":
continue
try:
public = base64.b64decode(key["public"], validate=True)
except (ValueError, TypeError):
continue
if key_hash(name, public) == kh and crypto.ed25519_verify(public, body, sig):
verified.append(name)
return note, verified
# --- acceptance (spec/23 §23.3) ---------------------------------------------------------------------------
def accept_checkpoint(text, roster, settings):
"""Whether a checkpoint note is one the project accepts: its origin is the project's log, a team key with the
`sign` purpose and the role `owner` signed it, and at least `threshold` of the project's witnesses cosigned it.
Returns (accepted, reasons); `settings` is the project's `log` setting. Never raises.
"""
note, verified = verify_checkpoint(text, roster)
reasons = []
if note is None:
return False, ["the note is malformed"]
if note["origin"] != settings.get("origin"):
reasons.append("the origin is not the project's log")
team = [n for n in verified if roster[n].get("role") == "owner" and "sign" in roster[n].get("purposes", [])]
if not team:
reasons.append("no owner key with the sign purpose signed it")
witnesses = [n for n in verified if n in settings.get("witnesses", []) and "witness" in roster[n].get("purposes", [])]
if len(witnesses) < settings.get("threshold", 0):
reasons.append("%d of %d witnesses cosigned" % (len(witnesses), settings.get("threshold", 0)))
return not reasons, reasons
# --- tiles (spec/23 §23.4) --------------------------------------------------------------------------------
TILE_HEIGHT = 8
TILE_WIDTH = 1 << TILE_HEIGHT
def tile_path(level, index, width=None):
"""The path of a tile: `tile/<level>/<N>`, with N as 3-digit groups, all but the last prefixed `x`, and
`.p/<W>` for a partial tile of W entries; `level` is a number or `entries`."""
parts, n = ["%03d" % (index % 1000)], index // 1000
while n:
parts.insert(0, "x%03d" % (n % 1000))
n //= 1000
path = "tile/%s/%s" % (level, "/".join(parts))
return path + (".p/%d" % width if width is not None else "")
def tiles(entries):
"""Every tile of a log over `entries`, as {path: bytes}: hash tiles per level (32-byte hashes of the complete
subtrees of 256^level entries, 256 to a tile) and entries tiles (canonical JSON Lines, 256 to a tile)."""
out = {}
leaves = [crypto._leaf(jcs.dumpb(list(e))) for e in entries]
level = 0
while True:
span = TILE_WIDTH ** level
nodes = [crypto._mth(leaves[i * span:(i + 1) * span]) for i in range(len(leaves) // span)]
if not nodes:
break
for t in range(len(nodes) // TILE_WIDTH):
out[tile_path(level, t)] = b"".join(nodes[t * TILE_WIDTH:(t + 1) * TILE_WIDTH])
rest = len(nodes) % TILE_WIDTH
if rest:
out[tile_path(level, len(nodes) // TILE_WIDTH, rest)] = b"".join(nodes[-rest:])
level += 1
for t in range(len(entries) // TILE_WIDTH):
out[tile_path("entries", t)] = b"".join(jcs.dumps(list(e)).encode("utf-8") + b"\n"
for e in entries[t * TILE_WIDTH:(t + 1) * TILE_WIDTH])
rest = len(entries) % TILE_WIDTH
if rest:
out[tile_path("entries", len(entries) // TILE_WIDTH, rest)] = b"".join(
jcs.dumps(list(e)).encode("utf-8") + b"\n" for e in entries[-rest:])
return out
def write_log(entries, origin, signers, out_dir):
"""Write a log as static files: `checkpoint` and every tile. Returns the checkpoint note."""
note = sign_checkpoint(origin, entries, signers)
for path, data in tiles(entries).items():
full = os.path.join(out_dir, *path.split("/"))
os.makedirs(os.path.dirname(full), exist_ok=True)
with open(full, "wb") as f:
f.write(data)
with open(os.path.join(out_dir, "checkpoint"), "w", encoding="utf-8") as f:
f.write(note)
return note
def read_log(log_dir):
"""The entries and the checkpoint note of a log written by `write_log`, read from its entries tiles."""
with open(os.path.join(log_dir, "checkpoint"), encoding="utf-8") as f:
note = f.read()
parsed = parse_checkpoint(note)
if parsed is None:
raise ValueError("the checkpoint is malformed")
size = parsed["size"]
entries = []
for t in range(size // TILE_WIDTH + (1 if size % TILE_WIDTH else 0)):
width = size % TILE_WIDTH if (t + 1) * TILE_WIDTH > size else None
with open(os.path.join(log_dir, *tile_path("entries", t, width).split("/")), "rb") as f:
entries += [json_line for json_line in (jcs.loads_line(ln) for ln in f.read().split(b"\n") if ln)]
return entries, note
tools/oracle/mldsa.py
-446
View File
@@ -1,446 +0,0 @@
"""ML-DSA-65, the module-lattice digital signature algorithm of FIPS 204, in plain Python.
Non-normative; see spec/08-fidelity.md §8.4.
This is the reference form of the standard, slow and clear, for the signature scheme `ml-dsa-65`: key generation
from a 32-byte seed, the deterministic signing variant (the hedging bytes are zero), and verification. Keys and
signatures are the standard's byte encodings, so a production verifier can use any FIPS 204 library. The pure
(non pre-hashed) variant with an empty context is used, as the standard's ML-DSA.Sign defines it.
"""
import hashlib
Q = 8380417
N = 256
D = 13
ZETA = 1753
K, L = 6, 5
ETA = 4
TAU = 49
LAMBDA = 192
GAMMA1 = 1 << 19
GAMMA2 = (Q - 1) // 32
BETA = TAU * ETA
OMEGA = 55
PK_SIZE = 32 + 32 * K * (23 - D)
SK_SIZE = 32 + 32 + 64 + 32 * ((L + K) * 4 + D * K)
SIG_SIZE = LAMBDA // 4 + L * 32 * 20 + OMEGA + K
F_INV256 = 8347681 # 256^-1 mod q
def _bitrev8(m):
return int("{:08b}".format(m)[::-1], 2)
ZETAS = [pow(ZETA, _bitrev8(m), Q) for m in range(256)]
def _shake256(data, n):
return hashlib.shake_256(data).digest(n)
def _shake128(data, n):
return hashlib.shake_128(data).digest(n)
# --- bit packing (FIPS 204 §7.1) ----------------------------------------------------------------------------
def _bitlen(x):
return x.bit_length()
def simple_bit_pack(w, b):
"""Coefficients in [0, b] as bitlen(b)-bit little-endian fields."""
bits = _bitlen(b)
acc = 0
for i, c in enumerate(w):
acc |= c << (bits * i)
return acc.to_bytes(32 * bits, "little")
def simple_bit_unpack(v, b):
bits = _bitlen(b)
acc = int.from_bytes(v, "little")
mask = (1 << bits) - 1
return [(acc >> (bits * i)) & mask for i in range(N)]
def bit_pack(w, a, b):
"""Coefficients in [-a, b] stored as b - w_i in bitlen(a + b) bits."""
bits = _bitlen(a + b)
acc = 0
for i, c in enumerate(w):
acc |= (b - c) << (bits * i)
return acc.to_bytes(32 * bits, "little")
def bit_unpack(v, a, b):
bits = _bitlen(a + b)
acc = int.from_bytes(v, "little")
mask = (1 << bits) - 1
return [b - ((acc >> (bits * i)) & mask) for i in range(N)]
def hint_bit_pack(h):
y = bytearray(OMEGA + K)
index = 0
for i in range(K):
for j in range(N):
if h[i][j]:
y[index] = j
index += 1
y[OMEGA + i] = index
return bytes(y)
def hint_bit_unpack(y):
h = [[0] * N for _ in range(K)]
index = 0
for i in range(K):
if y[OMEGA + i] < index or y[OMEGA + i] > OMEGA:
return None
first = index
while index < y[OMEGA + i]:
if index > first and y[index - 1] >= y[index]:
return None
h[i][y[index]] = 1
index += 1
if any(y[i] != 0 for i in range(index, OMEGA)):
return None
return h
# --- arithmetic ---------------------------------------------------------------------------------------------
def _mod_pm(r, m):
"""r mod± m: the representative in (-m/2, m/2]."""
r %= m
return r - m if r > m // 2 else r
def power2round(r):
rp = r % Q
r0 = _mod_pm(rp, 1 << D)
return (rp - r0) >> D, r0
def decompose(r):
rp = r % Q
r0 = _mod_pm(rp, 2 * GAMMA2)
if rp - r0 == Q - 1:
return 0, r0 - 1
return (rp - r0) // (2 * GAMMA2), r0
def high_bits(r):
return decompose(r)[0]
def low_bits(r):
return decompose(r)[1]
def make_hint(z, r):
return 1 if high_bits(r) != high_bits(r + z) else 0
def use_hint(h, r):
m = (Q - 1) // (2 * GAMMA2)
r1, r0 = decompose(r)
if h and r0 > 0:
return (r1 + 1) % m
if h and r0 <= 0:
return (r1 - 1) % m
return r1
def _inf_norm(w):
return max(abs(_mod_pm(c, Q)) for c in w)
def ntt(w):
w = list(w)
m, length = 0, 128
while length >= 1:
start = 0
while start < N:
m += 1
z = ZETAS[m]
for j in range(start, start + length):
t = z * w[j + length] % Q
w[j + length] = (w[j] - t) % Q
w[j] = (w[j] + t) % Q
start += 2 * length
length //= 2
return w
def ntt_inv(w):
w = list(w)
m, length = 256, 1
while length < N:
start = 0
while start < N:
m -= 1
z = (-ZETAS[m]) % Q
for j in range(start, start + length):
t = w[j]
w[j] = (t + w[j + length]) % Q
w[j + length] = z * (t - w[j + length]) % Q
start += 2 * length
length *= 2
return [c * F_INV256 % Q for c in w]
def _mul(a, b):
return [x * y % Q for x, y in zip(a, b)]
def _add(a, b):
return [(x + y) % Q for x, y in zip(a, b)]
def _sub(a, b):
return [(x - y) % Q for x, y in zip(a, b)]
# --- sampling (FIPS 204 §7.3) -------------------------------------------------------------------------------
def _coeff_from_three_bytes(b0, b1, b2):
z = ((b2 & 0x7F) << 16) | (b1 << 8) | b0
return z if z < Q else None
def rej_ntt_poly(rho):
out, j, n = [], 0, 3 * 300
stream = _shake128(rho, n)
pos = 0
while j < N:
if pos + 3 > len(stream):
n *= 2
stream = _shake128(rho, n)
c = _coeff_from_three_bytes(stream[pos], stream[pos + 1], stream[pos + 2])
pos += 3
if c is not None:
out.append(c)
j += 1
return out
def _coeff_from_half_byte(b):
return 4 - b if b < 9 else None # eta = 4
def rej_bounded_poly(rho):
out, j, n = [], 0, 200
stream = _shake256(rho, n)
pos = 0
while j < N:
if pos >= len(stream):
n *= 2
stream = _shake256(rho, n)
z = stream[pos]
pos += 1
z0 = _coeff_from_half_byte(z & 15)
z1 = _coeff_from_half_byte(z >> 4)
if z0 is not None:
out.append(z0)
j += 1
if z1 is not None and j < N:
out.append(z1)
j += 1
return out
def expand_a(rho):
return [[rej_ntt_poly(rho + bytes([s, r])) for s in range(L)] for r in range(K)]
def expand_s(rho_prime):
s1 = [rej_bounded_poly(rho_prime + r.to_bytes(2, "little")) for r in range(L)]
s2 = [rej_bounded_poly(rho_prime + (r + L).to_bytes(2, "little")) for r in range(K)]
return s1, s2
def expand_mask(rho, mu):
c = 1 + _bitlen(GAMMA1 - 1)
return [bit_unpack(_shake256(rho + (mu + r).to_bytes(2, "little"), 32 * c), GAMMA1 - 1, GAMMA1) for r in range(L)]
def sample_in_ball(rho):
c = [0] * N
n = 8 + TAU * 4
stream = _shake256(rho, n)
signs = int.from_bytes(stream[:8], "little")
pos = 8
for i in range(N - TAU, N):
while True:
if pos >= len(stream):
n *= 2
stream = _shake256(rho, n)
j = stream[pos]
pos += 1
if j <= i:
break
c[i] = c[j]
c[j] = 1 if (signs >> (i + TAU - N)) & 1 == 0 else Q - 1
return c
# --- encodings (FIPS 204 §7.2) ------------------------------------------------------------------------------
def pk_encode(rho, t1):
return rho + b"".join(simple_bit_pack(t, (1 << (23 - D)) - 1) for t in t1)
def pk_decode(pk):
rho, rest = pk[:32], pk[32:]
step = 32 * (23 - D)
return rho, [simple_bit_unpack(rest[i * step:(i + 1) * step], (1 << (23 - D)) - 1) for i in range(K)]
def sk_encode(rho, k, tr, s1, s2, t0):
out = rho + k + tr
out += b"".join(bit_pack(s, ETA, ETA) for s in s1)
out += b"".join(bit_pack(s, ETA, ETA) for s in s2)
out += b"".join(bit_pack(t, (1 << (D - 1)) - 1, 1 << (D - 1)) for t in t0)
return out
def sk_decode(sk):
rho, k, tr = sk[:32], sk[32:64], sk[64:128]
pos = 128
s1, s2, t0 = [], [], []
for _ in range(L):
s1.append(bit_unpack(sk[pos:pos + 128], ETA, ETA))
pos += 128
for _ in range(K):
s2.append(bit_unpack(sk[pos:pos + 128], ETA, ETA))
pos += 128
for _ in range(K):
t0.append(bit_unpack(sk[pos:pos + 32 * D], (1 << (D - 1)) - 1, 1 << (D - 1)))
pos += 32 * D
return rho, k, tr, s1, s2, t0
def sig_encode(c_tilde, z, h):
return c_tilde + b"".join(bit_pack(zi, GAMMA1 - 1, GAMMA1) for zi in z) + hint_bit_pack(h)
def sig_decode(sigma):
c_tilde = sigma[:LAMBDA // 4]
pos = LAMBDA // 4
z = []
for _ in range(L):
z.append(bit_unpack(sigma[pos:pos + 32 * 20], GAMMA1 - 1, GAMMA1))
pos += 32 * 20
h = hint_bit_unpack(sigma[pos:])
return c_tilde, z, h
def w1_encode(w1):
return b"".join(simple_bit_pack(w, (Q - 1) // (2 * GAMMA2) - 1) for w in w1)
# --- the scheme (FIPS 204 §6) -------------------------------------------------------------------------------
def _matvec(a_hat, v_hat):
out = []
for i in range(K):
acc = [0] * N
for j in range(L):
acc = _add(acc, _mul(a_hat[i][j], v_hat[j]))
out.append(acc)
return out
def keygen(seed):
"""ML-DSA.KeyGen_internal: (public key, secret key) bytes for a 32-byte seed."""
if len(seed) != 32:
raise ValueError("the seed is 32 bytes")
h = _shake256(seed + bytes([K, L]), 128)
rho, rho_prime, kk = h[:32], h[32:96], h[96:]
a_hat = expand_a(rho)
s1, s2 = expand_s(rho_prime)
t = [_add(ntt_inv(row), s2i) for row, s2i in zip(_matvec(a_hat, [ntt(s) for s in s1]), s2)]
t1, t0 = [], []
for poly in t:
hi, lo = zip(*(power2round(c) for c in poly))
t1.append(list(hi))
t0.append(list(lo))
pk = pk_encode(rho, t1)
tr = _shake256(pk, 64)
return pk, sk_encode(rho, kk, tr, s1, s2, t0)
def public_key(seed):
return keygen(seed)[0]
def _message_representative(message, ctx=b""):
if len(ctx) > 255:
raise ValueError("the context is at most 255 bytes")
return bytes([0, len(ctx)]) + ctx + message
def sign(sk, message, ctx=b"", rnd=bytes(32)):
"""ML-DSA.Sign with the deterministic variant (rnd = 0): the 3309-byte signature."""
rho, kk, tr, s1, s2, t0 = sk_decode(sk)
s1_hat = [ntt(s) for s in s1]
s2_hat = [ntt(s) for s in s2]
t0_hat = [ntt(t) for t in t0]
a_hat = expand_a(rho)
mu = _shake256(tr + _message_representative(message, ctx), 64)
rho_pp = _shake256(kk + rnd + mu, 64)
kappa = 0
while True:
y = expand_mask(rho_pp, kappa)
w = [ntt_inv(row) for row in _matvec(a_hat, [ntt(yi) for yi in y])]
w1 = [[high_bits(c) for c in wi] for wi in w]
c_tilde = _shake256(mu + w1_encode(w1), LAMBDA // 4)
c = sample_in_ball(c_tilde)
c_hat = ntt(c)
cs1 = [ntt_inv(_mul(c_hat, s)) for s in s1_hat]
cs2 = [ntt_inv(_mul(c_hat, s)) for s in s2_hat]
z = [_add(yi, ci) for yi, ci in zip(y, cs1)]
r0 = [[low_bits(c) for c in _sub(wi, ci)] for wi, ci in zip(w, cs2)]
kappa += L
if max(_inf_norm(zi) for zi in z) >= GAMMA1 - BETA or max(max(abs(c) for c in r) for r in r0) >= GAMMA2 - BETA:
continue
ct0 = [ntt_inv(_mul(c_hat, t)) for t in t0_hat]
h = []
for i in range(K):
base = _add(_sub(w[i], cs2[i]), ct0[i])
h.append([make_hint((-ct0[i][j]) % Q, base[j]) for j in range(N)])
if max(_inf_norm(t) for t in ct0) >= GAMMA2 or sum(sum(hi) for hi in h) > OMEGA:
continue
z_signed = [[_mod_pm(c, Q) for c in zi] for zi in z]
return sig_encode(c_tilde, z_signed, h)
def verify(pk, message, signature, ctx=b""):
"""ML-DSA.Verify: True for a valid signature of `message` under `pk`. Never raises."""
try:
if len(pk) != PK_SIZE or len(signature) != SIG_SIZE or len(ctx) > 255:
return False
rho, t1 = pk_decode(pk)
c_tilde, z, h = sig_decode(signature)
if h is None:
return False
a_hat = expand_a(rho)
tr = _shake256(pk, 64)
mu = _shake256(tr + _message_representative(message, ctx), 64)
c_hat = ntt(sample_in_ball(c_tilde))
az = _matvec(a_hat, [ntt(zi) for zi in z])
ct1 = [_mul(c_hat, ntt([c << D for c in t])) for t in t1]
w_approx = [ntt_inv(_sub(a, b)) for a, b in zip(az, ct1)]
w1 = [[use_hint(h[i][j], w_approx[i][j]) for j in range(N)] for i in range(K)]
if max(_inf_norm(zi) for zi in z) >= GAMMA1 - BETA:
return False
return _shake256(mu + w1_encode(w1), LAMBDA // 4) == c_tilde
except (ValueError, TypeError, IndexError):
return False
tools/oracle/model.py
-162
View File
@@ -1,162 +0,0 @@
"""The node tree of scienting-doc/1.
Non-normative; see spec/03-nodes.md.
"""
BLOCKS = frozenset(
["paragraph", "equation", "figure", "table", "list", "quote", "code", "algorithm", "theorem", "footnote"]
)
# Allowed child kinds for each node kind.
CHILDREN = {
"document": frozenset(["front", "section", "back"]),
"front": frozenset(["title", "authors", "abstract", "keywords", "notes"]),
"title": frozenset(["sentence"]),
"authors": frozenset(),
"abstract": BLOCKS | {"section"},
"keywords": frozenset(),
"notes": frozenset(["footnote", "paragraph"]),
"section": BLOCKS | {"heading", "section"},
"heading": frozenset(["sentence"]),
"paragraph": frozenset(["sentence"]),
"sentence": frozenset(["equation"]),
"equation": frozenset(["equation-line"]),
"equation-line": frozenset(),
"figure": frozenset(["panel", "caption", "figure-note"]),
"panel": frozenset(),
"caption": frozenset(["sentence"]),
"figure-note": frozenset(["sentence"]),
"table": frozenset(["caption", "table-head", "table-body", "table-note"]),
"table-head": frozenset(["row"]),
"table-body": frozenset(["row"]),
"table-note": frozenset(["sentence"]),
"row": frozenset(["cell"]),
"cell": frozenset(),
"list": frozenset(["item"]),
"item": BLOCKS,
"quote": BLOCKS,
"code": frozenset(),
"algorithm": frozenset(),
"theorem": BLOCKS,
"footnote": frozenset(["sentence"]),
"back": frozenset(["acknowledgments", "references", "appendix", "notes"]),
"acknowledgments": BLOCKS,
"references": frozenset(["reference"]),
"reference": frozenset(),
"appendix": frozenset(["section"]),
}
KINDS = frozenset(CHILDREN)
# Kinds that must have at least one child.
NONEMPTY = frozenset(
["title", "heading", "paragraph", "caption", "figure-note", "table-note", "table-head", "table-body",
"row", "list", "item", "quote", "theorem", "footnote", "appendix"]
)
# Kinds that occur at most once among their siblings.
AT_MOST_ONCE = frozenset(
["front", "back", "title", "authors", "abstract", "keywords", "caption", "table-head", "table-body",
"acknowledgments", "references"]
)
RUN_KINDS = frozenset(["sentence", "cell", "reference", "authors", "keywords"])
MATH_KINDS = frozenset(["equation", "equation-line"])
VERBATIM_KINDS = frozenset(["code", "algorithm"])
LEAVES = RUN_KINDS | MATH_KINDS | VERBATIM_KINDS
CONTAINERS_WITH_CONTENT = frozenset(["section", "figure", "panel", "table", "theorem", "footnote"])
CONTAINERS_WITHOUT_CONTENT = KINDS - LEAVES - CONTAINERS_WITH_CONTENT
LABELLED = frozenset(
["section", "figure", "panel", "table", "theorem", "footnote", "equation", "equation-line", "reference",
"code", "algorithm"]
)
# The attributes each kind may carry (spec/03 §3.4); kinds absent here carry none.
ATTRS = {
"section": frozenset(["level"]),
"table": frozenset(["columns"]),
"cell": frozenset(["column", "rowspan", "colspan", "role"]),
"row": frozenset(["header_path"]),
"list": frozenset(["ordered"]),
"item": frozenset(["marker"]),
"theorem": frozenset(["variant"]),
"panel": frozenset(["image", "crop", "figure_text"]),
}
RUNS = frozenset(["text", "math", "cite", "xref", "fnmark", "link", "equation"])
STYLES = frozenset(["em", "strong", "sup", "sub", "sc", "mono"])
LEVELS = ["F0", "F1", "F2", "F3"]
def anchor_class(kind):
if kind in LEAVES:
return "leaf"
if kind in CONTAINERS_WITH_CONTENT:
return "container-with-content"
return "container-without-content"
class Doc:
"""A document: nodes by ID with parent and order, plus document order."""
def __init__(self, key, nodes):
self.key = key
self.nodes = {}
self.order = []
self.children = {}
for n in nodes:
self.nodes[n["id"]] = n
self.order.append(n["id"])
self.children.setdefault(n["id"], [])
for nid in self.order:
p = self.nodes[nid].get("parent")
if p is not None:
self.children.setdefault(p, []).append(nid)
for p, kids in self.children.items():
kids.sort(key=lambda c: self.nodes[c]["ord"])
roots = [nid for nid in self.order if self.nodes[nid].get("parent") is None]
self.root = roots[0] if roots else None
def kids(self, nid):
return self.children.get(nid, [])
def preorder(self, nid=None):
nid = self.root if nid is None else nid
out = []
stack = [nid]
while stack:
x = stack.pop()
out.append(x)
stack.extend(reversed(self.kids(x)))
return out
def ancestors(self, nid):
out = []
p = self.nodes[nid].get("parent")
while p is not None:
out.append(p)
p = self.nodes[p].get("parent")
return list(reversed(out))
def text_leaves(self, nid):
"""The text-bearing nodes a container's projections are made of, in document order."""
out = []
stack = [nid]
while stack:
x = stack.pop()
n = self.nodes[x]
k = n["kind"]
if k in RUN_KINDS or k in VERBATIM_KINDS or (k == "equation" and self._parent_kind(x) != "sentence"):
out.append(x)
continue
if k == "equation-line" or k == "equation":
continue
stack.extend(reversed(self.kids(x)))
return out
def leaves(self, nid):
"""Every leaf below or at `nid`, including equations inside sentences and equation lines."""
return [x for x in self.preorder(nid) if self.nodes[x]["kind"] in LEAVES]
def _parent_kind(self, nid):
p = self.nodes[nid].get("parent")
return None if p is None else self.nodes[p]["kind"]
tools/oracle/notation.py
-481
View File
@@ -1,481 +0,0 @@
"""Simple notation and its canonical linear form.
Non-normative; the rules are spec/05-hashes.md, "The strict projection".
A math source or a styled script either folds to one canonical linear form, or is complex.
"""
import unicodedata
from .text import collapse_ws, is_ws, remove_ws, trim
MINUS = "\u2212"
DOT = "\u22c5"
PRIME = "\u2032"
# Characters that fold to one canonical character inside notation.
CHAR_FOLD = {
"-": MINUS,
"\u2013": MINUS,
"\u2212": MINUS,
"\u00b7": DOT,
"\u2219": DOT,
"\u22c5": DOT,
"'": PRIME,
"\u2032": PRIME,
}
OPERATORS = set("+=/<>.,()") | {MINUS, DOT, PRIME, "\u00d7", "\u2264", "\u2265", "\u00b1"}
TEX_GREEK = {
"alpha": "\u03b1", "beta": "\u03b2", "gamma": "\u03b3", "delta": "\u03b4",
"epsilon": "\u03f5", "varepsilon": "\u03b5", "zeta": "\u03b6", "eta": "\u03b7",
"theta": "\u03b8", "vartheta": "\u03d1", "iota": "\u03b9", "kappa": "\u03ba",
"varkappa": "\u03f0", "lambda": "\u03bb", "mu": "\u03bc", "nu": "\u03bd",
"xi": "\u03be", "pi": "\u03c0", "varpi": "\u03d6", "rho": "\u03c1",
"varrho": "\u03f1", "sigma": "\u03c3", "varsigma": "\u03c2", "tau": "\u03c4",
"upsilon": "\u03c5", "phi": "\u03d5", "varphi": "\u03c6", "chi": "\u03c7",
"psi": "\u03c8", "omega": "\u03c9",
"Gamma": "\u0393", "Delta": "\u0394", "Theta": "\u0398", "Lambda": "\u039b",
"Xi": "\u039e", "Pi": "\u03a0", "Sigma": "\u03a3", "Upsilon": "\u03a5",
"Phi": "\u03a6", "Psi": "\u03a8", "Omega": "\u03a9",
}
TEX_SYMBOLS = {
"times": "\u00d7", "cdot": DOT, "pm": "\u00b1", "le": "\u2264", "leq": "\u2264",
"ge": "\u2265", "geq": "\u2265", "lt": "<", "gt": ">", "prime": PRIME,
"AA": "\u00c5",
}
TEX_WRAPPERS = {"mathrm", "text", "textrm", "mathup"}
TEX_DECLARATIONS = {"rm"}
TEX_SPACES = {"quad", "qquad", "enspace", "thinspace", "medspace", "thickspace"}
TEX_SPACE_SYMBOLS = {",", ";", ":", "!", " "}
TYPST_GREEK = {
"alpha": "\u03b1", "beta": "\u03b2", "gamma": "\u03b3", "delta": "\u03b4",
"epsilon": "\u03b5", "zeta": "\u03b6", "eta": "\u03b7", "theta": "\u03b8",
"iota": "\u03b9", "kappa": "\u03ba", "lambda": "\u03bb", "mu": "\u03bc",
"nu": "\u03bd", "xi": "\u03be", "omicron": "\u03bf", "pi": "\u03c0",
"rho": "\u03c1", "sigma": "\u03c3", "tau": "\u03c4", "upsilon": "\u03c5",
"phi": "\u03c6", "chi": "\u03c7", "psi": "\u03c8", "omega": "\u03c9",
"Alpha": "\u0391", "Beta": "\u0392", "Gamma": "\u0393", "Delta": "\u0394",
"Epsilon": "\u0395", "Zeta": "\u0396", "Eta": "\u0397", "Theta": "\u0398",
"Iota": "\u0399", "Kappa": "\u039a", "Lambda": "\u039b", "Mu": "\u039c",
"Nu": "\u039d", "Xi": "\u039e", "Omicron": "\u039f", "Pi": "\u03a0",
"Rho": "\u03a1", "Sigma": "\u03a3", "Tau": "\u03a4", "Upsilon": "\u03a5",
"Phi": "\u03a6", "Chi": "\u03a7", "Psi": "\u03a8", "Omega": "\u03a9",
"epsilon.alt": "\u03f5", "theta.alt": "\u03d1", "phi.alt": "\u03d5",
"pi.alt": "\u03d6", "rho.alt": "\u03f1", "sigma.alt": "\u03c2",
"kappa.alt": "\u03f0", "beta.alt": "\u03d0",
}
TYPST_SYMBOLS = {
"plus": "+", "minus": MINUS, "times": "\u00d7", "dot": DOT, "dot.op": DOT,
"dot.c": DOT, "plus.minus": "\u00b1", "lt": "<", "gt": ">", "lt.eq": "\u2264",
"gt.eq": "\u2265", "eq": "=", "prime": PRIME, "slash": "/",
}
TYPST_WRAPPERS = {"upright"}
TYPST_SPACES = {"thin", "med", "thick", "quad", "wide", "space"}
class NotSimple(Exception):
pass
# The Latin and Greek letter blocks of spec/05 \u00a75.4: Basic Latin through Latin Extended-B, Greek and Coptic without
# the Coptic letters, Latin Extended Additional, and Greek Extended.
LETTER_RANGES = ((0x0041, 0x024F), (0x0370, 0x03E1), (0x03F0, 0x03FF), (0x1E00, 0x1EFF), (0x1F00, 0x1FFF))
def is_letter(ch):
"""A letter of General Category L in the Latin or Greek blocks, or the micro sign."""
if ch == "\u00b5":
return True
if not unicodedata.category(ch).startswith("L"):
return False
cp = ord(ch)
return any(lo <= cp <= hi for lo, hi in LETTER_RANGES)
def fold_char(ch):
return CHAR_FOLD.get(ch, ch)
def is_simple_char(ch):
ch = fold_char(ch)
return is_letter(ch) or unicodedata.category(ch) == "Nd" or ch in OPERATORS
def script_type(ch):
d = unicodedata.decomposition(ch)
if d.startswith("<super>"):
return "^", "".join(chr(int(h, 16)) for h in d.split()[1:])
if d.startswith("<sub>"):
return "_", "".join(chr(int(h, 16)) for h in d.split()[1:])
return None, None
class Atom:
__slots__ = ("base", "primes", "sub", "sup")
def __init__(self, base=""):
self.base = base
self.primes = 0
self.sub = None
self.sup = None
def _canon(atoms):
out = []
for a in atoms:
if a.sup is not None and a.sup and all(
x.base == PRIME and x.sub is None and x.sup is None and x.primes == 0 for x in a.sup
):
a.primes += len(a.sup)
a.sup = None
s = a.base + PRIME * a.primes
if a.sub is not None:
s += "_{" + _canon(a.sub) + "}"
if a.sup is not None:
s += "^{" + _canon(a.sup) + "}"
out.append(s)
return "".join(out)
def _attach(atoms, op, arg):
if not atoms:
atoms.append(Atom(""))
a = atoms[-1]
if op == "_":
if a.sub is not None:
raise NotSimple("double subscript")
a.sub = arg
else:
if a.sup is not None:
raise NotSimple("double superscript")
a.sup = arg
def _char_atom(ch):
if not is_simple_char(ch):
raise NotSimple("character %r" % ch)
return Atom(fold_char(ch))
# --- TeX ---------------------------------------------------------------------
def _tex_tokens(src):
toks = []
i, n = 0, len(src)
while i < n:
ch = src[i]
if ch == "\\":
j = i + 1
if j < n and src[j].isascii() and src[j].isalpha():
while j < n and src[j].isascii() and src[j].isalpha():
j += 1
toks.append(("cw", src[i + 1:j]))
while j < n and is_ws(src[j]):
j += 1
i = j
continue
if j < n:
toks.append(("cs", src[j]))
i = j + 1
continue
raise NotSimple("trailing backslash")
if is_ws(ch) or ch == "~":
i += 1
continue
toks.append(("ch", ch))
i += 1
return toks
class _TexParser:
def __init__(self, toks):
self.t = toks
self.i = 0
def peek(self):
return self.t[self.i] if self.i < len(self.t) else None
def take(self):
tok = self.peek()
self.i += 1
return tok
def seq(self, closing):
atoms = []
while True:
tok = self.peek()
if tok is None:
if closing:
raise NotSimple("unclosed group")
return atoms
if tok == ("ch", "}"):
if not closing:
raise NotSimple("unbalanced brace")
self.take()
return atoms
self.take()
kind, val = tok
if kind == "ch" and val in "_^":
_attach(atoms, val, self.arg())
elif kind == "ch" and val == "'":
if not atoms:
atoms.append(Atom(""))
atoms[-1].primes += 1
elif kind == "ch" and val == "{":
inner = self.seq(True)
nxt = self.peek()
if nxt in (("ch", "_"), ("ch", "^")) and len(inner) > 1:
raise NotSimple("script on a multi-atom group")
atoms.extend(inner) # an empty group leaves a following script to the previous atom or an empty base
else:
atoms.extend(self.command_or_char(kind, val))
def command_or_char(self, kind, val):
if kind == "ch":
return [_char_atom(val)]
if kind == "cs":
if val in TEX_SPACE_SYMBOLS:
return []
raise NotSimple("control symbol \\" + val)
if val in TEX_GREEK:
return [Atom(TEX_GREEK[val])]
if val in TEX_SYMBOLS:
return [Atom(TEX_SYMBOLS[val])]
if val in TEX_SPACES or val in TEX_DECLARATIONS:
return []
if val in TEX_WRAPPERS:
return self.arg()
if val in ("left", "right"):
tok = self.take()
if tok == ("ch", "(" if val == "left" else ")"):
return [Atom(tok[1])]
raise NotSimple("\\left or \\right with another delimiter")
raise NotSimple("command \\" + val)
def arg(self):
tok = self.take()
if tok is None:
raise NotSimple("missing argument")
kind, val = tok
if tok == ("ch", "{"):
return self.seq(True)
if kind == "ch" and val in "}_^'":
raise NotSimple("bad argument")
return self.command_or_char(kind, val)
def tex_atoms(src):
return _TexParser(_tex_tokens(src)).seq(False)
# --- Typst -------------------------------------------------------------------
def _typst_tokens(src):
toks = []
i, n = 0, len(src)
while i < n:
ch = src[i]
if is_ws(ch):
i += 1
continue
if ch == '"':
j = i + 1
buf = []
while j < n and src[j] != '"':
if src[j] == "\\":
raise NotSimple("escape in string")
buf.append(src[j])
j += 1
if j >= n:
raise NotSimple("unclosed string")
toks.append(("str", "".join(buf)))
i = j + 1
continue
if unicodedata.category(ch).startswith("L"):
j = i
while j < n and (
unicodedata.category(src[j]).startswith("L")
or (src[j] == "." and j + 1 < n and src[j + 1].isascii() and src[j + 1].isalpha() and j > i)
):
j += 1
toks.append(("id", src[i:j]))
i = j
continue
if unicodedata.category(ch) == "Nd":
j = i
while j < n and unicodedata.category(src[j]) == "Nd":
j += 1
if j + 1 < n and src[j] == "." and unicodedata.category(src[j + 1]) == "Nd":
j += 1
while j < n and unicodedata.category(src[j]) == "Nd":
j += 1
toks.append(("num", src[i:j]))
i = j
continue
two = src[i:i + 2]
if two in ("<=", ">="):
toks.append(("ch", "\u2264" if two == "<=" else "\u2265"))
i += 2
continue
if two in ("->", "=>", "!=", "<-", "==", ":="):
raise NotSimple("operator " + two)
toks.append(("ch", ch))
i += 1
return toks
class _TypstParser:
def __init__(self, toks):
self.t = toks
self.i = 0
def peek(self):
return self.t[self.i] if self.i < len(self.t) else None
def take(self):
tok = self.peek()
self.i += 1
return tok
def seq(self, closing):
atoms = []
while True:
tok = self.peek()
if tok is None:
if closing:
raise NotSimple("unclosed parenthesis")
return atoms
if closing and tok == ("ch", ")"):
self.take()
return atoms
self.take()
kind, val = tok
if kind == "ch" and val in "_^":
_attach(atoms, val, self.arg())
elif kind == "ch" and val == "'":
if not atoms:
atoms.append(Atom(""))
atoms[-1].primes += 1
elif kind == "ch" and val == "(":
atoms.append(Atom("("))
atoms.extend(self.seq(True))
atoms.append(Atom(")"))
else:
atoms.extend(self.single(kind, val))
def single(self, kind, val):
if kind == "num":
return [Atom(c) for c in val]
if kind == "str":
return [_char_atom(c) for c in remove_ws(val)]
if kind == "id":
if len(val) == 1:
return [_char_atom(val)]
if val in TYPST_GREEK:
return [Atom(TYPST_GREEK[val])]
if val in TYPST_SYMBOLS:
return [Atom(TYPST_SYMBOLS[val])]
if val in TYPST_SPACES:
return []
if val in TYPST_WRAPPERS and self.peek() == ("ch", "("):
self.take()
return self.seq(True)
raise NotSimple("identifier " + val)
if val in "/*#&[]{}\\$!:;|~@":
raise NotSimple("character %r" % val)
return [_char_atom(val)]
def arg(self):
tok = self.take()
if tok is None:
raise NotSimple("missing argument")
kind, val = tok
if tok == ("ch", "("):
return self.seq(True)
if kind == "ch" and val in ")_^'":
raise NotSimple("bad argument")
return self.single(kind, val)
def typst_atoms(src):
return _TypstParser(_typst_tokens(src)).seq(False)
# --- public ------------------------------------------------------------------
def fold_math(notation, source):
"""Return the canonical linear form of a math source, or None when it is complex."""
src = unicodedata.normalize("NFC", source)
try:
atoms = tex_atoms(src) if notation == "tex" else typst_atoms(src)
return _canon(atoms)
except NotSimple:
return None
def complex_key(notation, source):
return notation + ":" + remove_ws(unicodedata.normalize("NFC", source))
def _fold_script_chars(chars):
"""Fold a run of Unicode script characters of one type, or None if any base is not simple."""
op = None
bases = []
for ch in chars:
t, b = script_type(ch)
if t is None or (op is not None and t != op):
return None
op = t
bases.append(b)
base = "".join(bases)
if not all(is_simple_char(c) for c in base):
return None
return op + "{" + "".join(fold_char(c) for c in base) + "}"
def _fold_chars(s, in_notation):
"""Fold the characters of text: script characters always, operators only inside notation."""
out = []
i, n = 0, len(s)
while i < n:
ch = s[i]
t, b = script_type(ch)
if t is not None:
j = i
while j < n and script_type(s[j])[0] == t:
j += 1
folded = _fold_script_chars(s[i:j])
out.append(folded if folded is not None else s[i:j])
i = j
continue
if in_notation:
out.append(fold_char(ch))
elif ch in ("-", "\u2013") and i + 1 < n and unicodedata.category(s[i + 1]) == "Nd":
out.append(MINUS)
else:
out.append(ch)
i += 1
return "".join(out)
def fold_plain_text(text):
return _fold_chars(unicodedata.normalize("NFC", text), in_notation=False)
def fold_styled_script(op, text):
"""The canonical form of text styled as a subscript (`_`) or superscript (`^`)."""
t = unicodedata.normalize("NFC", text)
s = remove_ws(t)
simple = all(is_simple_char(c) or script_type(c)[0] is not None for c in s)
if simple:
inner = _fold_chars(s, in_notation=True)
if op == "^" and inner and all(c == PRIME for c in inner):
return inner
else:
inner = trim(collapse_ws(t))
return op + "{" + inner + "}"
Loaded 100 of 122 files, more files were not shown because too many files have changed in this diff. Show more