A community-supported supercharged document management system: scan, index and archive all your documents
Python aiangulararchivingdjangodmsdocument-managementdocument-management-systemllmmachine-learningocroptical-character-recognitionpdf
Chore(beta): Update suggested broker to Valkey + Redis agnostic documentation (#13032)
Trenton Ha009ea1f04cc
diff --git a/docker/compose/docker-compose.mariadb-tika.yml b/docker/compose/docker-compose.mariadb-tika.ymlindex eaae67157..54c7c1811 100644--- a/docker/compose/docker-compose.mariadb-tika.yml+++ b/docker/compose/docker-compose.mariadb-tika.yml@@ -30,7 +30,7 @@# documentation.services:broker:- image: docker.io/library/redis:8+ image: docker.io/valkey/valkey:9-alpinerestart: unless-stoppedvolumes:- redisdata:/datadiff --git a/docker/compose/docker-compose.mariadb.yml b/docker/compose/docker-compose.mariadb.ymlindex 2265703da..8f828d3b1 100644--- a/docker/compose/docker-compose.mariadb.yml+++ b/docker/compose/docker-compose.mariadb.yml@@ -26,7 +26,7 @@# documentation.services:broker:- image: docker.io/library/redis:8+ image: docker.io/valkey/valkey:9-alpinerestart: unless-stoppedvolumes:- redisdata:/datadiff --git a/docker/compose/docker-compose.portainer.yml b/docker/compose/docker-compose.portainer.ymlindex a44b4b692..0d2fbec11 100644--- a/docker/compose/docker-compose.portainer.yml+++ b/docker/compose/docker-compose.portainer.yml@@ -27,7 +27,7 @@# documentation.services:broker:- image: docker.io/library/redis:8+ image: docker.io/valkey/valkey:9-alpinerestart: unless-stoppedvolumes:- redisdata:/datadiff --git a/docker/compose/docker-compose.postgres-tika.yml b/docker/compose/docker-compose.postgres-tika.ymlindex e988a537b..754264afc 100644--- a/docker/compose/docker-compose.postgres-tika.yml+++ b/docker/compose/docker-compose.postgres-tika.yml@@ -30,7 +30,7 @@# documentation.services:broker:- image: docker.io/library/redis:8+ image: docker.io/valkey/valkey:9-alpinerestart: unless-stoppedvolumes:- redisdata:/datadiff --git a/docker/compose/docker-compose.postgres.yml b/docker/compose/docker-compose.postgres.ymlindex 5ce15f463..705432b67 100644--- a/docker/compose/docker-compose.postgres.yml+++ b/docker/compose/docker-compose.postgres.yml@@ -26,7 +26,7 @@# documentation.services:broker:- image: docker.io/library/redis:8+ image: docker.io/valkey/valkey:9-alpinerestart: unless-stoppedvolumes:- redisdata:/datadiff --git a/docker/compose/docker-compose.sqlite-tika.yml b/docker/compose/docker-compose.sqlite-tika.ymlindex d3ef2630a..e8143216d 100644--- a/docker/compose/docker-compose.sqlite-tika.yml+++ b/docker/compose/docker-compose.sqlite-tika.yml@@ -30,7 +30,7 @@# documentation.services:broker:- image: docker.io/library/redis:8+ image: docker.io/valkey/valkey:9-alpinerestart: unless-stoppedvolumes:- redisdata:/datadiff --git a/docker/compose/docker-compose.sqlite.yml b/docker/compose/docker-compose.sqlite.ymlindex 37c7f0c46..1688c53c0 100644--- a/docker/compose/docker-compose.sqlite.yml+++ b/docker/compose/docker-compose.sqlite.yml@@ -23,7 +23,7 @@# documentation.services:broker:- image: docker.io/library/redis:8+ image: docker.io/valkey/valkey:9-alpinerestart: unless-stoppedvolumes:- redisdata:/datadiff --git a/docs/configuration.md b/docs/configuration.mdindex 4f721fa36..ed85f0a41 100644--- a/docs/configuration.md+++ b/docs/configuration.md@@ -22,7 +22,11 @@ or applicable default will be utilized instead.## Required services-### Redis Broker+### Message Broker++Paperless-ngx uses a Redis-compatible message broker. Any broker that+speaks the Redis protocol works here, including [Valkey](https://valkey.io/)+(the default in the bundled Docker Compose files) and Redis itself.#### [`PAPERLESS_REDIS=<url>`](#PAPERLESS_REDIS) {#PAPERLESS_REDIS}@@ -30,21 +34,21 @@ or applicable default will be utilized instead.fetching, index optimization and for training the automatic documentmatcher.- - If your Redis server needs login credentials PAPERLESS_REDIS =+ - If your broker needs login credentials PAPERLESS_REDIS =`redis://<username>:<password>@<host>:<port>`- With the requirepass option PAPERLESS_REDIS =`redis://:<password>@<host>:<port>`- - To include the redis database index PAPERLESS_REDIS =+ - To include the database index PAPERLESS_REDIS =`redis://<username>:<password>@<host>:<port>/<DBIndex>`- [More information on securing your Redis- Instance](https://redis.io/docs/latest/operate/oss_and_stack/management/security).+ [More information on securing your broker+ instance](https://valkey.io/topics/security/).Defaults to `redis://localhost:6379`.#### [`PAPERLESS_REDIS_PREFIX=<prefix>`](#PAPERLESS_REDIS_PREFIX) {#PAPERLESS_REDIS_PREFIX}-: Prefix to be used in Redis for keys and channels. Useful for sharing one Redis server among multiple Paperless instances.+: Prefix to be used in the broker for keys and channels. Useful for sharing one broker among multiple Paperless instances.Defaults to no prefix.@@ -238,7 +242,7 @@ dictionaries; for example, `pool.max_size=20` sets#### [`PAPERLESS_DB_READ_CACHE_ENABLED=<bool>`](#PAPERLESS_DB_READ_CACHE_ENABLED) {#PAPERLESS_DB_READ_CACHE_ENABLED}-: Caches the database read query results into Redis. This can significantly improve application response times by caching database queries, at the cost of slightly increased memory usage.+: Caches the database read query results into the broker. This can significantly improve application response times by caching database queries, at the cost of slightly increased memory usage.Defaults to `false`.@@ -258,18 +262,18 @@ dictionaries; for example, `pool.max_size=20` setsA high TTL increases memory usage over time. Memory may be used until end of TTL, even if the cache is invalidated with the `invalidate_cachalot` command.-In case of an out-of-memory (OOM) situation, Redis may stop accepting new data — including cache entries, scheduled tasks, and documents to consume.-If your system has limited RAM, consider configuring a dedicated Redis instance for the read cache, with a memory limit and the eviction policy set to `allkeys-lru`.-For more details, refer to the [Redis eviction policy documentation](https://redis.io/docs/latest/develop/reference/eviction/), and see the `PAPERLESS_READ_CACHE_REDIS_URL` setting to specify a separate Redis broker.+In case of an out-of-memory (OOM) situation, the broker may stop accepting new data — including cache entries, scheduled tasks, and documents to consume.+If your system has limited RAM, consider configuring a dedicated broker instance for the read cache, with a memory limit and the eviction policy set to `allkeys-lru`.+For more details, refer to the [Redis eviction policy documentation](https://redis.io/docs/latest/develop/reference/eviction/), and see the `PAPERLESS_READ_CACHE_REDIS_URL` setting to specify a separate broker.#### [`PAPERLESS_READ_CACHE_REDIS_URL=<url>`](#PAPERLESS_READ_CACHE_REDIS_URL) {#PAPERLESS_READ_CACHE_REDIS_URL}-: Defines the Redis instance used for the read cache.+: Defines the broker instance used for the read cache.Defaults to `None`.!!! Note- If this value is not set, the same Redis instance used for scheduled tasks will be used for caching as well.+ If this value is not set, the same broker instance used for scheduled tasks will be used for caching as well.## Optional Servicesdiff --git a/docs/development.md b/docs/development.mdindex 11e078a67..7a576c9f7 100644--- a/docs/development.md+++ b/docs/development.md@@ -94,16 +94,16 @@ first-time setup.```7. You can now either ...- - install Redis or+ - install a Redis-compatible broker (e.g. Valkey or Redis) or- use the included `scripts/start_services.sh` to use Docker to fire- up a Redis instance (and some other services such as Tika,+ up a broker instance (and some other services such as Tika,Gotenberg and a database server) or- - spin up a bare Redis container+ - spin up a bare broker container```bash- docker run -d -p 6379:6379 --restart unless-stopped redis:latest+ docker run -d -p 6379:6379 --restart unless-stopped docker.io/valkey/valkey:9-alpine```8. Continue with either back-end or front-end development – or both :-).diff --git a/docs/faq.md b/docs/faq.mdindex 2ef67222b..0a37c178e 100644--- a/docs/faq.md+++ b/docs/faq.md@@ -118,10 +118,14 @@ able to run paperless, you're a bit on your own. If you can't run thedocker image, the documentation has instructions for bare metalinstalls.-## _What about the Redis licensing change and using one of the open source forks_?+## _Which message broker should I use_?-Currently (October 2024), forks of Redis such as Valkey or Redirect are not officially supported by our upstream-libraries, so using one of these to replace Redis is not officially supported.+Paperless-ngx talks to a Redis-compatible message broker, so any broker that+implements the Redis protocol will work. The bundled Docker Compose files+default to [Valkey](https://valkey.io/), the open-source fork created after+Redis' licensing change, but Redis itself and other wire-compatible brokers+(such as Microsoft's Garnet) are equally fine.-However, they do claim to be compatible with the Redis protocol and will likely work, but we will-not be updating from using Redis as the broker officially just yet.+Existing installs can switch broker implementations in place: point+[`PAPERLESS_REDIS`](configuration.md#PAPERLESS_REDIS) at the new instance and+reuse the same data volume.diff --git a/docs/setup.md b/docs/setup.mdindex 5580dde92..45db1c85e 100644--- a/docs/setup.md+++ b/docs/setup.md@@ -226,7 +226,8 @@ to enable polling and disable inotify. See [here](configuration.md#polling).build-essential python3-setuptools python3-wheel```-2. Install `redis` >= 6.0 and configure it to start automatically.+2. Install a Redis-compatible broker (a current release of Valkey or+ Redis) and configure it to start automatically.3. Optional: Install `postgresql` and configure a database, user, andpassword for Paperless-ngx. If you do not wish to use PostgreSQL,@@ -268,7 +269,7 @@ to enable polling and disable inotify. See [here](configuration.md#polling).6. Configure Paperless-ngx. See [configuration](configuration.md) for details.Edit the included `paperless.conf` and adjust the settings to yourneeds. Required settings for getting Paperless-ngx running are:- - [`PAPERLESS_REDIS`](configuration.md#PAPERLESS_REDIS) should point to your Redis server, such as+ - [`PAPERLESS_REDIS`](configuration.md#PAPERLESS_REDIS) should point to your broker, such as`redis://localhost:6379`.- [`PAPERLESS_DBENGINE`](configuration.md#PAPERLESS_DBENGINE) is optional, and should be one of `postgres`,`mariadb`, or `sqlite`@@ -297,7 +298,7 @@ to enable polling and disable inotify. See [here](configuration.md#polling).!!! warning- Ensure your Redis instance [is secured](https://redis.io/docs/latest/operate/oss_and_stack/management/security/).+ Ensure your broker instance [is secured](https://valkey.io/topics/security/).7. Create the following directories if they do not already exist:- `/opt/paperless/media`@@ -389,9 +390,9 @@ to enable polling and disable inotify. See [here](configuration.md#polling).`Require=paperless-webserver.socket` in the `webserver` scriptand configure `granian` to listen on port 80 (set `GRANIAN_PORT`).- These services rely on Redis and optionally the database server, but+ These services rely on the broker and optionally the database server, butdon't need to be started in any particular order. The example files- depend on Redis being started. If you use a database server, you+ depend on the broker being started. If you use a database server, youshould add additional dependencies.!!! note@@ -494,7 +495,7 @@ installation. Keep these points in mind:for other services, you might as well use it for Paperless as well.- The task scheduler of Paperless, which is used to execute periodictasks such as email checking and maintenance, requires a- [Redis](https://redis.io/) message broker instance. The+ Redis-compatible message broker instance (such as Valkey or Redis). TheDocker Compose route takes care of that.- The layout of the folder structure for your documents and dataremains the same, so you can plug your old Docker volumes into@@ -582,16 +583,16 @@ commands as well.1. Stop and remove the Paperless container.2. If using an external database, stop that container.-3. Update Redis configuration.+3. Update broker configuration.1. If `REDIS_URL` is already set, change it to [`PAPERLESS_REDIS`](configuration.md#PAPERLESS_REDIS)and continue to step 4.- 1. Otherwise, add a new Redis service in `docker-compose.yml`,+ 1. Otherwise, add a new broker service in `docker-compose.yml`,following [the example composefiles](https://github.com/paperless-ngx/paperless-ngx/tree/main/docker/compose)1. Set the environment variable [`PAPERLESS_REDIS`](configuration.md#PAPERLESS_REDIS) so it points to- the new Redis container.+ the new broker container.4. Update user mapping.1. If set, change the environment variable `PUID` to `USERMAP_UID`.diff --git a/docs/troubleshooting.md b/docs/troubleshooting.mdindex 47a10bf41..988a328a5 100644--- a/docs/troubleshooting.md+++ b/docs/troubleshooting.md@@ -10,9 +10,9 @@ Check for the following issues:`CONSUMPTION_DIR` setting. Don't adjust this setting if you'reusing docker.-- Ensure that redis is up and running. Paperless does its task+- Ensure that the broker is up and running. Paperless does its taskprocessing asynchronously, and for documents to arrive at the task- processor, it needs redis to run.+ processor, it needs the broker to run.- Ensure that the task processor is running. Docker does thisautomatically. Manually invoke the task processor by executingdiff --git a/docs/usage.md b/docs/usage.mdindex 27d3dee9e..f28713f52 100644--- a/docs/usage.md+++ b/docs/usage.md@@ -1097,7 +1097,7 @@ Paperless-ngx consists of the following components:errors (i.e., wrong email credentials, errors during consuming aspecific file, etc).-- A [redis](https://redis.io/) message broker: This is a really+- A message broker (such as Valkey or Redis): This is a reallylightweight service that is responsible for getting the tasks fromthe webserver and the consumer to the task scheduler. These run in adifferent process (maybe even on different machines!), and