The minimalistic document archiving platform.
docs(guides): add guides for ai stuff and content extraction (#1299)
Corentin Thomassetc5b37e894239
diff --git a/apps/docs/src/assets/organization-auto-tagging-settings.png b/apps/docs/src/assets/organization-auto-tagging-settings.pngnew file mode 100644index 00000000..b4f69d4cBinary files /dev/null and b/apps/docs/src/assets/organization-auto-tagging-settings.png differdiff --git a/apps/docs/src/content/docs/03-guides/11-llm-configuration.mdx b/apps/docs/src/content/docs/03-guides/11-llm-configuration.mdxnew file mode 100644index 00000000..df6da539--- /dev/null+++ b/apps/docs/src/content/docs/03-guides/11-llm-configuration.mdx@@ -0,0 +1,125 @@+---+title: LLM configuration+description: Configure LLM providers in Papra to enable AI features like auto tagging and content analysis.+slug: guides/llm-configuration+---++This guide explains how to configure LLM providers in Papra. Once a provider is configured, it can be used to power AI features like [document auto tagging](/guides/auto-tagging).++## Enabling AI features++AI features are disabled by default. To enable them, set the master switch:++```bash+AI_IS_ENABLED=true+```++You then need to configure at least one provider (see below) and pick a default model.++## Model nomenclature++In order to use different models from potentially different providers, Papra uses a specific nomenclature to identify a model: `<adapter>://<model_name>`.++For example:+- `openai://gpt-4o-mini`+- `ollama://qwen3:8b`+- `anthropic://claude-haiku-4-5`+- `mistral://mistral-small-latest`+- `openrouter://openai/gpt-5.1`++The `<adapter>` part is the id of one of the built-in adapters (see [Supported providers](#supported-providers)).++## Default model++The `AI_DEFAULT_MODEL` variable defines which model is used by AI features when no more specific model is configured:++```bash+AI_DEFAULT_MODEL=openai://gpt-4o-mini+```++Individual features can override this. For example, auto tagging reads `AUTO_TAGGING_MODEL` first and falls back to `AI_DEFAULT_MODEL` if it is not set.++## Supported providers++Papra ships with adapters for the most common providers, plus support for any OpenAI-compatible API. Each adapter is configured with its own environment variables.++Don't hesitate to [reach out](https://github.com/papra-hq/papra/issues) if you want to see a provider added to Papra.++### OpenAI-compatible providers++These adapters all speak the OpenAI API format and share the same two settings: a base URL and an API key. The defaults already point to each provider's hosted API, so for the cloud providers you usually only need to set the API key.++| Adapter (`<adapter>`) | Provider | API key | Base URL (default) |+| --- | --- | --- | --- |+| `openai` | OpenAI | `OPENAI_API_KEY` | `OPENAI_BASE_URL` (`https://api.openai.com/v1`) |+| `mistral` | Mistral | `MISTRAL_API_KEY` | `MISTRAL_BASE_URL` (`https://api.mistral.ai/v1`) |+| `deepseek` | DeepSeek | `DEEPSEEK_API_KEY` | `DEEPSEEK_BASE_URL` (`https://api.deepseek.com/v1`) |+| `openrouter` | OpenRouter | `OPENROUTER_API_KEY` | `OPENROUTER_BASE_URL` (`https://openrouter.ai/api/v1`) |+| `cocore` | co/core | `COCORE_API_KEY` | `COCORE_BASE_URL` (`https://cocore.dev/api/v1`) |+| `ollama` | Ollama | `OLLAMA_API_KEY` (default `ollama`) | `OLLAMA_BASE_URL` (`http://localhost:11434/v1`) |+| `lmstudio` | LM Studio | `LMSTUDIO_API_KEY` (default `lm-studio`) | `LMSTUDIO_BASE_URL` (`http://localhost:1234/v1`) |+| `vllm` | vLLM | `VLLM_API_KEY` (default `not-needed`) | `VLLM_BASE_URL` (`http://localhost:8000/v1`) |++:::note+The base URL must include the API version path expected by the provider (e.g. the `/v1` suffix for OpenAI-compatible APIs). The defaults already include it, so only override the base URL if you are pointing at a self-hosted or proxied endpoint.+:::++### Anthropic++Anthropic uses its own API and adapter (`anthropic`):++| Setting | Environment variable |+| --- | --- |+| API key | `ANTHROPIC_API_KEY` |+| Base URL (optional) | `ANTHROPIC_BASE_URL` |++### Any other OpenAI-compatible provider++Because the OpenAI-compatible adapters are just a base URL and an API key, you can point any of them at another service. The most common approach is to reuse the `openai` adapter and override its base URL:++```bash+OPENAI_BASE_URL=https://your-provider.example.com/v1+OPENAI_API_KEY=your-api-key+AI_DEFAULT_MODEL=openai://your-model-name+```++## Examples++### OpenAI++```bash+AI_IS_ENABLED=true+OPENAI_API_KEY=sk-...+AI_DEFAULT_MODEL=openai://gpt-4o-mini+```++### Anthropic++```bash+AI_IS_ENABLED=true+ANTHROPIC_API_KEY=sk-ant-...+AI_DEFAULT_MODEL=anthropic://claude-haiku-4-5+```++### Ollama (local)++A local Ollama install is reachable on its default URL, so you only need to point Papra at the right adapter and model:++```bash+AI_IS_ENABLED=true+AI_DEFAULT_MODEL=ollama://qwen3:8b+```++If Ollama runs on another host or port, override the base URL (keep the `/v1` suffix):++```bash+OLLAMA_BASE_URL=http://ollama:11434/v1+```++:::tip+When running Papra in Docker, `localhost` refers to the container itself. To reach an Ollama or LM Studio instance running on the host, use `http://host.docker.internal:11434/v1` (or the host's IP) instead of `localhost`.+:::++## Next steps++- [Set up document auto tagging](/guides/auto-tagging)diff --git a/apps/docs/src/content/docs/03-guides/12-auto-tagging.mdx b/apps/docs/src/content/docs/03-guides/12-auto-tagging.mdxnew file mode 100644index 00000000..696a808c--- /dev/null+++ b/apps/docs/src/content/docs/03-guides/12-auto-tagging.mdx@@ -0,0 +1,72 @@+---+title: Document auto tagging+description: Automatically tag your documents based on their content using AI.+slug: guides/auto-tagging+---++This guide shows you how to set up AI-powered auto tagging for your documents.++## What is auto tagging?++Once a document's content has been extracted, Papra can automatically apply the most relevant tags based on that content. This is done using an LLM, which analyzes the text and suggests appropriate tags from your existing tags and their descriptions. You can also let it create new tags when none of the existing ones are relevant.++The configuration lives per-organization, so each organization can independently:+- Enable or disable auto tagging+- Choose whether to create new tags+- Limit the number of tags applied to each document++## Prerequisites++Auto tagging relies on two things:++1. **A configured LLM provider.** Papra supports many providers out of the box, plus any OpenAI-compatible API. See the [LLM configuration guide](/guides/llm-configuration) for details.+2. **Content extraction.** Tags are derived from the document's extracted text, so the document must go through [content extraction](/guides/content-extraction) first.++## Enabling auto tagging (server)++Auto tagging is gated by two server-side switches: the global AI switch and the auto tagging switch. Both must be enabled (the auto tagging switch is enabled by default).++Example configuration using a local Ollama instance:++```bash+AI_IS_ENABLED=true+AI_DEFAULT_MODEL=ollama://qwen3:8b+AUTO_TAGGING_ENABLED=true+```++### Server options++| Variable | Default | Description |+| --- | --- | --- |+| `AI_IS_ENABLED` | `false` | Master switch for all AI features. Must be `true`. |+| `AUTO_TAGGING_ENABLED` | `true` | Whether organizations are allowed to use auto tagging. |+| `AUTO_TAGGING_MODEL` | falls back to `AI_DEFAULT_MODEL` | The model used for auto tagging, in `<adapter>://<model>` form. |+| `AUTO_TAGGING_DEFAULT_MAX_TAGS` | `5` | Default maximum number of tags per document. Can be overridden per organization. |++Once enabled on the server, organizations can turn auto tagging on in their settings.++## Configuring auto tagging (organization)++In your organization settings, under the **Auto tagging** section, you can configure:++- **Enable auto tagging**: Enable or disable auto tagging for this organization.+- **Create new tags**: Allow or disallow creating new tags when none of the existing tags are relevant.+- **Max number of tags**: Limit how many tags can be applied to each document (default `5`). The LLM may return fewer.++++Once enabled, every time a document is added to Papra its extracted content is sent to the LLM for analysis, and the most relevant tags are applied automatically.++:::tip+Tag descriptions matter. The LLM uses each tag's name **and** description to decide what is relevant, so adding clear descriptions to your tags significantly improves the quality of the suggestions.+:::++## Troubleshooting++If auto tagging is not working as expected:++- Check the server logs for errors related to the LLM provider or the auto tagging process.+- Ensure the LLM provider is correctly configured and reachable from the Papra server (see [LLM configuration](/guides/llm-configuration)).+- Confirm both `AI_IS_ENABLED` and `AUTO_TAGGING_ENABLED` are set, and that auto tagging is enabled in the organization settings.+- Make sure the document's content was extracted — with no extracted text, the LLM has nothing to tag.+- If no new tags ever appear, check that **Create new tags** is enabled and that the max tags limit is not too low.diff --git a/apps/docs/src/content/docs/03-guides/13-content-extraction.mdx b/apps/docs/src/content/docs/03-guides/13-content-extraction.mdxnew file mode 100644index 00000000..059345e4--- /dev/null+++ b/apps/docs/src/content/docs/03-guides/13-content-extraction.mdx@@ -0,0 +1,156 @@+---+title: Content extraction+description: Configure how Papra extracts text from your documents, including OCR and external extraction providers.+slug: guides/content-extraction+---++This guide explains how Papra extracts text from your documents and how to configure the available extraction strategies, including external providers and OCR.++## What is content extraction?++When a document is added to Papra, its text content is extracted so it can be searched and used by other features such as [auto tagging](/guides/auto-tagging). Different document formats, and different quality requirements, call for different extraction methods, so Papra supports several **extraction strategies**.++## Choosing strategies++The strategies Papra uses are controlled by the `CONTENT_EXTRACTION_STRATEGY` environment variable. It accepts a single strategy name or a comma-separated list, in order of preference:++```bash+# Single strategy (default)+CONTENT_EXTRACTION_STRATEGY=internal++# Try Mistral OCR first, fall back to the internal strategy+CONTENT_EXTRACTION_STRATEGY=mistral-ocr,internal+```++When multiple strategies are listed:++- The first strategy that can handle the document (based on its mime type, see [mime type filtering](#mime-type-filtering)) is used.+- If that strategy fails to process the document, the next eligible strategy is tried.++:::tip+Keep `internal` as the last strategy in the list. It is always available and works on all common formats, making it a reliable fallback when an external provider is unavailable or errors out.+:::++## Available strategies++| Strategy | Description | Requires |+| --- | --- | --- |+| `internal` | Built-in extraction using the `lecture` library, with Tesseract for OCR. Supports all common formats and is always available. | Nothing |+| `mistral-ocr` | OCR via the Mistral OCR API. | Mistral API key |+| `docling` | Extraction via a self-hosted Docling server. | Running Docling server |+| `azure-di` | Extraction via Azure Document Intelligence. | Azure endpoint + API key |+| `custom-http` | Extraction via any HTTP service you provide. | Your own HTTP endpoint |++### Internal++The default strategy. It uses the bundled `lecture` library to extract text from common document formats and falls back to Tesseract OCR for images and scanned documents. No configuration is required, and it is always available.++```bash+CONTENT_EXTRACTION_STRATEGY=internal+```++### Mistral OCR++Uses the [Mistral OCR API](https://mistral.ai/) to extract text. It reuses the Mistral adapter credentials from the [LLM configuration](/guides/llm-configuration), so set `MISTRAL_API_KEY` (and optionally `MISTRAL_BASE_URL`).++```bash+CONTENT_EXTRACTION_STRATEGY=mistral-ocr,internal+MISTRAL_API_KEY=your-mistral-api-key+```++| Variable | Default | Description |+| --- | --- | --- |+| `MISTRAL_API_KEY` | – | API key for the Mistral API (shared with the Mistral LLM adapter). |+| `MISTRAL_OCR_MODEL_NAME` | `mistral-ocr-latest` | The Mistral OCR model to use. |+| `MISTRAL_OCR_MIME_TYPES_ALLOW_LIST` | `application/pdf,image/*` | Mime types this strategy handles. See [mime type filtering](#mime-type-filtering). |++### Docling++Uses a self-hosted [Docling server](https://github.com/docling-project/docling-serve) to extract text.++```bash+CONTENT_EXTRACTION_STRATEGY=docling,internal+DOCLING_BASE_URL=http://docling:5001+```++| Variable | Default | Description |+| --- | --- | --- |+| `DOCLING_BASE_URL` | `http://localhost:5001` | Base URL of the Docling service. |+| `DOCLING_API_KEY` | – | Optional API key, sent as the `X-API-Key` header. On the Docling side this maps to `DOCLING_SERVE_API_KEY`. |+| `DOCLING_MIME_TYPES_ALLOW_LIST` | `*` | Mime types this strategy handles. See [mime type filtering](#mime-type-filtering). |++### Azure Document Intelligence++Uses the [Azure Document Intelligence](https://azure.microsoft.com/products/ai-services/ai-document-intelligence) service.++```bash+CONTENT_EXTRACTION_STRATEGY=azure-di,internal+AZURE_DI_ENDPOINT=https://your-resource.cognitiveservices.azure.com+AZURE_DI_API_KEY=your-azure-api-key+```++| Variable | Default | Description |+| --- | --- | --- |+| `AZURE_DI_ENDPOINT` | – | Endpoint of the Azure Document Intelligence resource. |+| `AZURE_DI_API_KEY` | – | API key, sent as the `Ocp-Apim-Subscription-Key` header. |+| `AZURE_DI_MIME_TYPES_ALLOW_LIST` | `*` | Mime types this strategy handles. See [mime type filtering](#mime-type-filtering). |+| `AZURE_DI_POLLING_ATTEMPTS` | `10` | Number of times to poll for the result of a processing job. |+| `AZURE_DI_POLLING_DELAY_MS` | `2000` | Delay in milliseconds between polling attempts. |++### Custom HTTP++Sends the document to an HTTP service you provide and reads the extracted text from the response. This lets you plug in any extraction backend without writing a Papra adapter. Can be used with n8n, Zapier, or any other service that can receive a file and return text.++```bash+CONTENT_EXTRACTION_STRATEGY=custom-http,internal+CONTENT_EXTRACTION_CUSTOM_HTTP_URL=https://extractor.example.com/extract+CONTENT_EXTRACTION_CUSTOM_HTTP_HEADERS={"Authorization":"Bearer your-token"}+```++Papra makes a `POST` request to the configured URL with the document, then reads the extracted text from the response.++| Variable | Default | Description |+| --- | --- | --- |+| `CONTENT_EXTRACTION_CUSTOM_HTTP_URL` | – | URL of your HTTP extraction service. |+| `CONTENT_EXTRACTION_CUSTOM_HTTP_HEADERS` | `{}` | Extra headers as a JSON object, e.g. `{"Authorization":"Bearer <token>"}`. The `Content-Type` header is set automatically based on the upload format. |+| `CONTENT_EXTRACTION_CUSTOM_HTTP_UPLOAD_FORMAT` | `form-data` | How the document is sent: `form-data` or `json` (see below). |+| `CONTENT_EXTRACTION_CUSTOM_HTTP_RESPONSE_FORMAT` | `json` | How the response is read: `json` or `text` (see below). |+| `CONTENT_EXTRACTION_CUSTOM_HTTP_JSON_RESPONSE_TEXT_PATH` | `text` | For `json` responses, the dot path to the extracted text, e.g. `data.content`. |+| `CONTENT_EXTRACTION_CUSTOM_HTTP_REQUEST_TIMEOUT_MS` | `30000` | Request timeout in milliseconds. |+| `CONTENT_EXTRACTION_CUSTOM_HTTP_MIME_TYPES_ALLOW_LIST` | `*` | Mime types this strategy handles. See [mime type filtering](#mime-type-filtering). |++**Upload formats** (how Papra sends the document):++- `form-data`: a `multipart/form-data` request with the document file in the `file` field.+- `json`: a JSON body with the document base64-encoded:+ ```json+ { "document": { "filename": "file.pdf", "type": "application/pdf", "content": "<base64-encoded-content>" } }+ ```++**Response formats** (how Papra reads the extracted text):++- `json`: the response is parsed as JSON and the text is read from the path in `CONTENT_EXTRACTION_CUSTOM_HTTP_JSON_RESPONSE_TEXT_PATH` (e.g. `text`, or `data.content`).+- `text`: the entire response body is used as the extracted text.++## Mime type filtering++Every strategy except `internal` accepts a mime types allow list that can limits which documents it handles. If a document's mime type does not match, the strategy is skipped and the next one in the list is tried.++The list is comma-separated and supports wildcards and negations:++- `*` matches all formats.+- `image/*` matches all image mime types.+- Prefix an entry with `!` to negate it: `*,!image/png` allows everything except PNG.+- Negations always take precedence over allows, even more specific ones: `image/png,!image/*` rejects PNG.++For example, to use Azure Document Intelligence only for PDFs and fall back to the internal strategy for everything else:++```bash+CONTENT_EXTRACTION_STRATEGY=azure-di,internal+AZURE_DI_MIME_TYPES_ALLOW_LIST=application/pdf+```++## Related guides++- [LLM configuration](/guides/llm-configuration)+- [Document auto tagging](/guides/auto-tagging)diff --git a/apps/docs/src/content/navigation.ts b/apps/docs/src/content/navigation.tsindex 397d7823..61244f38 100644--- a/apps/docs/src/content/navigation.ts+++ b/apps/docs/src/content/navigation.ts@@ -65,6 +65,19 @@ export const sidebar = [label: 'Storage Key Patterns',slug: 'guides/storage-key-patterns',},+ // Uncomment post release+ // {+ // label: 'Content Extraction',+ // slug: 'guides/content-extraction',+ // },+ // {+ // label: 'LLM Configuration',+ // slug: 'guides/llm-configuration',+ // },+ // {+ // label: 'Document Auto Tagging',+ // slug: 'guides/auto-tagging',+ // },],},{