# PDF to Markdown

There are numerous instances where we need to process PDFs into Markdown to enable programmatic extraction and modern pipelines.
This repository provides a minimal environment for performing these computations locally (requires access to enterprise GPUs) or via [datalab](https://datalab.to) API calls.

The `scratch/` directory can be used to run jobs as it is ignored by git.

## Datalab

[Datalab](https://datalab.to) offers a pay-as-you-go service with $10 of free credits per month.
It is $0.01 per page when converting to Markdown with their "Accurate" quality.

Set your API key as an environment variable.

```sh
export DATALAB_API_KEY=your_api_key_here
```

We also prefer setting a consistent Datalab pipeline ID as an environment variable.

```sh
export DATALAB_PIPELINE_ID=your_pipeline_id_here
```

You can then run your API call with:

```sh
pixi run -e datalab datalab custom-pipeline --pipeline_id "${DATALAB_PIPELINE_ID}" --format markdown PATH
```