There are numerous instances where we need to process PDFs into Markdown to enable programmatic extraction and modern pipelines. This repository provides a minimal environment for performing these computations locally (requires access to enterprise GPUs) or via datalab API calls.

The scratch/ directory can be used to run jobs as it is ignored by git.

Datalab

Datalab offers a pay-as-you-go service with $10 of free credits per month. It is $0.01 per page when converting to Markdown with their "Accurate" quality.

Set your API key as an environment variable.

export DATALAB_API_KEY=your_api_key_here

We also prefer setting a consistent Datalab pipeline ID as an environment variable.

export DATALAB_PIPELINE_ID=your_pipeline_id_here

You can then run your API call with:

pixi run -e datalab datalab custom-pipeline --pipeline_id "${DATALAB_PIPELINE_ID}" --format markdown PATH