Skip to content

armgit / ente

Public Imported from GitHub
0

πŸ’š End-to-end encrypted cloud for everything.

[infra] Replace Promtail with Alloy (#14329)

Manav Rathi9d72f0caa07c
diff --git a/infra/copycat-db/README.md b/infra/copycat-db/README.md
index 2535200329..32d619b7f0 100644
--- a/infra/copycat-db/README.md
+++ b/infra/copycat-db/README.md
@@ -126,7 +126,7 @@ The database dumps are stored in a bucket that has object lock enabled (complian
## Deploying
-Ensure that promtail is running, and is configured to scrape `/root/var/logs/copycat-db.log`.
+Ensure that Alloy is running, and is configured to scrape `/root/var/logs/copycat-db.log`.
Create the configuration and log directories.
diff --git a/infra/services/README.md b/infra/services/README.md
index 52b8414399..66a4a1be0a 100644
--- a/infra/services/README.md
+++ b/infra/services/README.md
@@ -6,7 +6,7 @@ Generally our services (including museum itself) follow the same pattern:
- They're run on vanilla Ubuntu instances. The only expectation they have is for Docker to be installed.
-- They log to fixed, known locations such as `/root/var/logs/foo.log` so Promtail can ingest them when needed.
+- They log to fixed, known locations such as `/root/var/logs/foo.log` so Alloy can ingest them when needed.
- Each service should consist of a Docker image (or a Docker compose file), and a systemd unit file.
@@ -73,10 +73,10 @@ sudo journalctl --follow --unit example
## Logging
-Simple services can log to standard output and error. Docker captures these logs, and Promtail sends them to Grafana.
+Simple services can log to standard output and error. Docker captures these logs, and Alloy sends them to Grafana.
Services that need a specific job name or more control over retention can log to their own files.
- Such files should be in `/var/logs` within the container, and this should be mounted to `/root/var/logs` on the instance (using the `-v` flag in the service file which launches the Docker container or the Docker compose cluster).
-- Add each log file to `promtail/promtail.yaml` on that instance so Promtail can send it to Grafana.
+- Add each log file to `alloy/config.alloy` on that instance so Alloy can send it to Grafana.
diff --git a/infra/services/alloy/README.md b/infra/services/alloy/README.md
new file mode 100644
index 0000000000..f0025397a5
--- /dev/null
+++ b/infra/services/alloy/README.md
@@ -0,0 +1,22 @@
+# Alloy
+
+Install `alloy.service` on an instance if it is running something whose logs we want in Grafana.
+
+## Installing
+
+Replace `endpoint.url` in the config file with the Loki URL that Alloy should connect to, and move the files to their expected place.
+
+```sh
+scp services/alloy/{alloy.service,config.alloy} <instance>:
+
+nano config.alloy
+sudo mv config.alloy /root/config.alloy
+sudo mv alloy.service /etc/systemd/system/alloy.service
+```
+
+Tell systemd to pick up new service definitions, enable the unit (so that it automatically starts on boot), and start it this time around.
+
+```sh
+sudo systemctl daemon-reload
+sudo systemctl enable --now alloy
+```
diff --git a/infra/services/alloy/alloy.service b/infra/services/alloy/alloy.service
new file mode 100644
index 0000000000..22304738cc
--- /dev/null
+++ b/infra/services/alloy/alloy.service
@@ -0,0 +1,20 @@
+[Unit]
+Documentation=https://grafana.com/docs/alloy/latest/
+Requires=docker.service
+After=docker.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+ExecStartPre=docker pull grafana/alloy
+ExecStartPre=-docker stop alloy
+ExecStartPre=-docker rm alloy
+ExecStart=docker run --name alloy \
+ --hostname "%H" \
+ -v /root/config.alloy:/etc/alloy/config.alloy:ro \
+ -v alloy-storage:/var/lib/alloy \
+ -v /var/log:/var/log:ro \
+ -v /root/var/logs:/var/logs:ro \
+ -v /var/lib/docker/containers:/var/lib/docker/containers:ro \
+ grafana/alloy run --storage.path=/var/lib/alloy /etc/alloy/config.alloy
diff --git a/infra/services/alloy/config.alloy b/infra/services/alloy/config.alloy
new file mode 100644
index 0000000000..b6a70e6c33
--- /dev/null
+++ b/infra/services/alloy/config.alloy
@@ -0,0 +1,19 @@
+loki.source.file "logs" {
+ targets = [
+ {__path__ = "/var/logs/museum.log", job = "museum", host = sys.env("HOSTNAME")},
+ {__path__ = "/var/logs/copycat-db.log", job = "copycat-db", host = sys.env("HOSTNAME")},
+ {__path__ = "/var/logs/phoenix.log", job = "phoenix", host = sys.env("HOSTNAME")},
+ {__path__ = "/var/lib/docker/containers/*/*-json.log", job = "docker", host = sys.env("HOSTNAME")},
+ ]
+ forward_to = [loki.write.default.receiver]
+
+ file_match {
+ enabled = true
+ }
+}
+
+loki.write "default" {
+ endpoint {
+ url = "http://loki:3100/loki/api/v1/push"
+ }
+}
diff --git a/infra/services/promtail/README.md b/infra/services/promtail/README.md
deleted file mode 100644
index 63f6d5382e..0000000000
--- a/infra/services/promtail/README.md
+++ /dev/null
@@ -1,22 +0,0 @@
-# Promtail
-
-Install `promtail.service` on an instance if it is running something whose logs we want in Grafana.
-
-## Installing
-
-Replace `client.url` in the config file with the Loki URL that Promtail should connect to, and move the files to their expected place.
-
-```sh
-scp services/promtail/promtail.* <instance>:
-
-nano promtail.yaml
-sudo mv promtail.yaml /root/promtail.yaml
-sudo mv promtail.service /etc/systemd/system/promtail.service
-```
-
-Tell systemd to pick up new service definitions, enable the unit (so that it automatically starts on boot), and start it this time around.
-
-```sh
-sudo systemctl daemon-reload
-sudo systemctl enable --now promtail
-```
diff --git a/infra/services/promtail/promtail.service b/infra/services/promtail/promtail.service
deleted file mode 100644
index b98ec762f2..0000000000
--- a/infra/services/promtail/promtail.service
+++ /dev/null
@@ -1,19 +0,0 @@
-[Unit]
-Documentation=https://grafana.com/docs/loki/latest/clients/promtail/
-Requires=docker.service
-After=docker.service
-
-[Install]
-WantedBy=multi-user.target
-
-[Service]
-ExecStartPre=docker pull grafana/promtail
-ExecStartPre=-docker stop promtail
-ExecStartPre=-docker rm promtail
-ExecStart=docker run --name promtail \
- --hostname "%H" \
- -v /root/promtail.yaml:/config.yaml:ro \
- -v /var/log:/var/log \
- -v /root/var/logs:/var/logs:ro \
- -v /var/lib/docker/containers:/var/lib/docker/containers:ro \
- grafana/promtail -config.file=/config.yaml -config.expand-env=true
diff --git a/infra/services/promtail/promtail.yaml b/infra/services/promtail/promtail.yaml
deleted file mode 100644
index 3fa09f041b..0000000000
--- a/infra/services/promtail/promtail.yaml
+++ /dev/null
@@ -1,38 +0,0 @@
-server:
- disable: true
-
-clients:
- - url: http://loki:3100/loki/api/v1/push
-
-# Manually add entries for all our services. This is a bit cumbersome, but
-# - Retains flexibility in file names.
-# - Makes adding job labels easy.
-# - Does not get in the way of logrotation.
-scrape_configs:
- - job_name: museum
- static_configs:
- - labels:
- job: museum
- host: ${HOSTNAME}
- __path__: /var/logs/museum.log
-
- - job_name: copycat-db
- static_configs:
- - labels:
- job: copycat-db
- host: ${HOSTNAME}
- __path__: /var/logs/copycat-db.log
-
- - job_name: phoenix
- static_configs:
- - labels:
- job: phoenix
- host: ${HOSTNAME}
- __path__: /var/logs/phoenix.log
-
- - job_name: docker
- static_configs:
- - labels:
- job: docker
- host: ${HOSTNAME}
- __path__: /var/lib/docker/containers/*/*-json.log
diff --git a/server/README.md b/server/README.md
index e64b8db303..be351520a7 100644
--- a/server/README.md
+++ b/server/README.md
@@ -78,7 +78,7 @@ Everything that you might needed to run museum is all in here, since this is the
> [!TIP]
>
-> On our production servers, we wrap museum in a [systemd service](scripts/deploy/museum.service). Our production machines are vanilla Ubuntu images, with Docker and Promtail installed. We then plonk in this systemd service, and use `systemctl start|stop|status museum` to herd it around.
+> On our production servers, we wrap museum in a [systemd service](scripts/deploy/museum.service). Our production machines are vanilla Ubuntu images, with Docker and Alloy installed. We then plonk in this systemd service, and use `systemctl start|stop|status museum` to herd it around.
Some people new to Docker/Go/Postgres might have general questions though. Unfortunately, because of limited engineering bandwidth **we will currently not be able to prioritize support queries related to self hosting**, and we request you to please not open issues around self hosting for the time being (feel free to create discussions though). The best way to summarize the status of self hosting is – **everything you need is here, but it is perhaps not readily documented, or flexible enough.**
diff --git a/server/scripts/deploy/README.md b/server/scripts/deploy/README.md
index b5ce7685cc..947e341acc 100644
--- a/server/scripts/deploy/README.md
+++ b/server/scripts/deploy/README.md
@@ -18,7 +18,7 @@ We run Museum's Docker image on Ubuntu hosts with Docker and manage it with syst
To bring up another Museum node, prepare the instance to run our services.
-Set up [Promtail](../../../infra/services/promtail/README.md), [Prometheus and node-exporter](../../../infra/services/prometheus/README.md).
+Set up [Alloy](../../../infra/services/alloy/README.md), [Prometheus and node-exporter](../../../infra/services/prometheus/README.md).
If running behind Nginx, install the [nginx](../../../infra/services/nginx/README.md) service.