Skip to content

armgit / avibe

Public Imported from GitHub
0

Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.

fix(logging): move service log I/O off the controller thread (#3053)

Call White5ecae62dc278
diff --git a/docs/plans/nonblocking-service-logging.md b/docs/plans/nonblocking-service-logging.md
new file mode 100644
index 000000000..bcbee9dae
--- /dev/null
+++ b/docs/plans/nonblocking-service-logging.md
@@ -0,0 +1,98 @@
+# Non-blocking service logging (#2748)
+
+## Change contract
+
+This is a proactive mitigation: the recent current-master soak did not reproduce a
+logging-attributed controller-loop stall. Only service logging ownership and its
+entry/exit lifecycle change; no controller, database, transport, or call-site
+changes are intended.
+
+- Root records go through one QueueHandler; one QueueListener owns the existing
+ rotating file and optional stdout destinations, formatter and levels.
+- A queue holds at most 8192 pending records (plus the listener's in-flight
+ record); it is a record-count bound, not a byte limit. Full queues drop the
+ incoming record (including high-severity/final records); accepted records are
+ never evicted. Ordinary QueueHandler message/traceback preparation is retained.
+- Cumulative `queue_full`, `prepare_error`, `handler_error` and `reentrant`
+ counters are reported directly to existing destinations, at most once per
+ 30 seconds while processing records and once at final drain if totals changed.
+ No producer writes to stderr on overload/error. Failing destinations are
+ attempted independently and cannot kill the listener. If no destination
+ accepts an aggregate diagnostic, the listener writes/flushes a single line of
+ at most 1024 ASCII bytes directly to process stderr (`sys.__stderr__`). It
+ shares the 30-second summary rate limit plus the final-drain attempt, contains
+ only cumulative counters (no record payloads), and never logs or re-enqueues.
+ Missing/broken stderr is caught without retries; blocked stderr only blocks
+ the listener, not producers. This escape path is best effort, not delivery
+ assurance when stderr also fails or shares the unavailable filesystem.
+ A diagnostic's own failed delivery is counted, not recursively retried.
+ Reporting is activity-driven: isolated preparation errors can wait until a
+ subsequent accepted record or shutdown; 30 seconds is not a delivery SLA.
+- Nested logging while producing (including a signal interrupting Queue's
+ non-reentrant mutex) or inside a destination is suppressed/countable, avoiding
+ deadlock and feedback loops. Shutdown signal handling itself still runs; only
+ a nested record can be lost under this policy.
+- Shutdown closes admission only after controller and desktop/diagnostic
+ cleanup. It drains all accepted records, attempts each destination, joins the
+ listener and closes owned handlers BEFORE releasing the service instance lock,
+ so a successor cannot acquire ownership while this logger still rotates files.
+ Lock release remains in a `finally` and retains the existing lock-safety gate.
+ Setup/shutdown are repeatable; partially constructed setups leave no listener
+ or file behind.
+ New destinations open only after the prior drain/rotation finishes. Previous
+ root handlers and level are detached while the runtime owns root, then restored
+ on stop; they are not closed by this runtime. A closed admission handler stays
+ on root throughout drain to prevent implicit synchronous stderr configuration.
+ `main()` explicitly drains even if cleanup raises; an atexit hook is a fallback
+ for other callers. Sink close errors do not prevent the other sinks closing.
+ The service deliberately owns root configuration; third-party root handlers
+ are inactive during ownership, and concurrent third-party reconfiguration is
+ not supported. Lock-release debug diagnostics and late daemon records after
+ drain use restored pre-service logging rather than this service file.
+- Drain is not a disk-durability promise: failed destinations cannot guarantee
+ delivery. Lossless draining also cannot bound exit time if a destination never
+ returns (including stderr fallback). The service retains its lock during that
+ wait. The existing POSIX managed stop gives the WHOLE shutdown five seconds
+ before SIGKILL; cleanup and logging share that grace, and this change does not
+ extend it. Windows managed termination also does not promise Python drain.
+ Forced process termination can lose queued/in-flight records and skips atexit.
+ Drain is conditional on destination progress and absence of forced termination,
+ NOT a bounded-shutdown promise. Normal logging still performs record/message
+ preparation and short in-process lock operations.
+- No deployment, service restart, Incus update, or merge is authorized here.
+
+## Validation plan
+
+Existing tests only exercise handlers synchronously and the surrounding service
+lifecycle; they do not exercise asynchronous acceptance or drain. Extend that
+owner boundary with deterministic blocked destinations: the producer must finish
+before release, accepted records must arrive after release, overflow must be
+counted without recursive diagnostics, and shutdown must wait for actual final
+record delivery. Exercise real file rotation/stdout suppression, handler errors,
+setup failure/repetition, early startup failure and final service cleanup logging.
+Run focused lifecycle tests, changed-file Ruff and relevant repository checks;
+leave full platform/suite coverage to CI.
+
+## Progress
+
+- [x] Inspect setup, shutdown and existing logging tests on current origin/master.
+- [x] Implement runtime and service lifecycle ownership.
+- [x] Prove slow-I/O, overflow, drain, failure and compatibility contracts.
+- [x] Inspect complete working-tree diff, including staged/unstaged changes.
+- Exact-head review/CI delivery evidence will be recorded in the PR.
+
+## Local evidence
+
+- Combined logging, lifecycle, config compatibility, controller shutdown,
+ watchdog and service-lock selection: 199 passed, 3 platform skips and 4
+ subtests passed on Python 3.12 after the final ordering/fallback revision.
+- File-only write/rotation failures exercise successful, missing, write-failing
+ and flush-failing stderr; bounded diagnostics, rate limiting and blocked stderr
+ producer independence are verified. Replaying the pre-fallback reporter makes
+ the regression fail because no stderr notice arrives.
+- The slow-stream regression was replayed with `setup_logging` extracted from
+ origin/master: it failed specifically at `controller waited for stream.write()`;
+ the same test passes with the queue runtime.
+- Changed-file Ruff and mypy (production files, imports skipped) pass.
+- No live service/Incus operations were performed. Full cross-platform coverage
+ remains with CI; the local environment is hermetic and owns its streams/files.
diff --git a/main.py b/main.py
index 1e092fb86..006172b54 100644
--- a/main.py
+++ b/main.py
@@ -5,11 +5,16 @@ import os
import signal
import sys
import logging
-from logging.handlers import RotatingFileHandler
from typing import Any
from config.paths import ensure_data_dirs, get_logs_dir
from vibe.logging_config import APPLICATION_LOG_BACKUP_COUNT, APPLICATION_LOG_MAX_BYTES
+from vibe.service_logging import (
+ ServiceFileHandler,
+ ServiceStreamHandler,
+ configure_service_logging,
+ shutdown_service_logging,
+)
from vibe.macos_session_diagnostics import start_macos_session_diagnostics
from vibe.runtime import (
ServiceAlreadyRunningError,
@@ -22,32 +27,30 @@ from vibe.runtime import (
def _build_logging_handlers(logs_dir: str) -> list[logging.Handler]:
handlers: list[logging.Handler] = [
- RotatingFileHandler(
+ ServiceFileHandler(
f"{logs_dir}/vibe_remote.log",
maxBytes=APPLICATION_LOG_MAX_BYTES,
backupCount=APPLICATION_LOG_BACKUP_COUNT,
)
]
- if os.environ.get("VIBE_DISABLE_STDOUT_LOGGING", "").lower() not in {"1", "true", "yes"}:
- handlers.insert(0, logging.StreamHandler(sys.stdout))
+ try:
+ if os.environ.get("VIBE_DISABLE_STDOUT_LOGGING", "").lower() not in {"1", "true", "yes"}:
+ handlers.insert(0, ServiceStreamHandler(sys.stdout))
+ except BaseException:
+ for handler in handlers:
+ handler.close()
+ raise
return handlers
-def setup_logging(level: str = "INFO"):
- """Setup logging configuration with file location and line numbers"""
- # Create a custom formatter with file location
+def setup_logging(level: str = "INFO") -> None:
+ """Keep destination I/O on the service logging listener, not the controller."""
+ log_level = getattr(logging, level.upper())
log_format = '%(asctime)s - %(name)s - %(levelname)s - [%(filename)s:%(lineno)d] - %(funcName)s() - %(message)s'
-
- # For development, you can use this more detailed format:
- # log_format = '%(asctime)s - %(name)s - %(levelname)s - [%(pathname)s:%(lineno)d] - %(funcName)s() - %(message)s'
-
ensure_data_dirs()
logs_dir = str(get_logs_dir())
-
- logging.basicConfig(
- level=getattr(logging, level.upper()),
- format=log_format,
- handlers=_build_logging_handlers(logs_dir),
+ configure_service_logging(
+ lambda: _build_logging_handlers(logs_dir), log_level, logging.Formatter(log_format),
)
@@ -261,13 +264,20 @@ def main():
logging.error(f"Failed to start: {e}")
sys.exit(1)
finally:
- lock_safe = bool(
- controller is None
- or getattr(controller, "service_lock_safe_to_release", True)
- )
- if lock_acquired and lock_safe:
- release_service_instance_lock()
- _stop_macos_session_diagnostics(macos_session_diagnostics)
+ try:
+ _stop_macos_session_diagnostics(macos_session_diagnostics)
+ finally:
+ try:
+ # Controller.run() has finished its census/watchdog cleanup.
+ # Finish rotating/writing before another service can own the log.
+ shutdown_service_logging()
+ finally:
+ lock_safe = bool(
+ controller is None
+ or getattr(controller, "service_lock_safe_to_release", True)
+ )
+ if lock_acquired and lock_safe:
+ release_service_instance_lock()
if __name__ == "__main__":
diff --git a/tests/test_macos_session_diagnostics.py b/tests/test_macos_session_diagnostics.py
index f6a1e8173..75dc8fc7a 100644
--- a/tests/test_macos_session_diagnostics.py
+++ b/tests/test_macos_session_diagnostics.py
@@ -390,7 +390,7 @@ def test_monitor_stop_joins_its_worker() -> None:
assert threads[0].joined is True
-def test_service_main_owns_monitor_shutdown(monkeypatch) -> None:
+def test_service_main_owns_monitor_shutdown(monkeypatch, tmp_path) -> None:
import config.v2_compat
import core.controller
import core.process_diagnostics
@@ -404,6 +404,7 @@ def test_service_main_owns_monitor_shutdown(monkeypatch) -> None:
class FakeMonitor:
def stop(self) -> None:
calls.append("diagnostics.stop")
+ logging.getLogger("test.service-cleanup").warning("final diagnostic record")
class FakeRunningLoop:
def is_closed(self) -> bool:
@@ -429,6 +430,7 @@ def test_service_main_owns_monitor_shutdown(monkeypatch) -> None:
calls.append("controller.run")
handlers[signal.SIGTERM](signal.SIGTERM, None)
calls.append("controller.after-signal")
+ logging.getLogger("test.service-cleanup").warning("final census/watchdog record")
def request_shutdown(self, reason: str) -> None:
calls.append(f"controller.shutdown:{reason}")
@@ -452,7 +454,8 @@ def test_service_main_owns_monitor_shutdown(monkeypatch) -> None:
lambda **kwargs: calls.append(f"desktop-ui.stop:{kwargs}") or True,
)
monkeypatch.setattr(service_main, "load_config", lambda: loaded_config)
- monkeypatch.setattr(service_main, "setup_logging", lambda level: None)
+ monkeypatch.setattr(service_main, "get_logs_dir", lambda: tmp_path)
+ monkeypatch.setenv("VIBE_DISABLE_STDOUT_LOGGING", "1")
monkeypatch.setattr(service_main, "apply_claude_sdk_patches", lambda: None)
monkeypatch.setattr(
service_main,
@@ -475,6 +478,15 @@ def test_service_main_owns_monitor_shutdown(monkeypatch) -> None:
service_main.main()
+ # Real queued logging must outlive signal handling and all service cleanup.
+ text = (tmp_path / "vibe_remote.log").read_text()
+ assert "Shutting down after signal" in text
+ assert "final census/watchdog record" in text
+ assert text.count("final diagnostic record") == 2
+ from vibe import service_logging
+
+ assert service_logging._runtime is None
+
# The migration runs under the lock and before anything is constructed
# against the schema it produces.
assert calls.index("lock.acquire") < calls.index("sqlite.prepare")
diff --git a/tests/test_service_logging.py b/tests/test_service_logging.py
new file mode 100644
index 000000000..7e56980e5
--- /dev/null
+++ b/tests/test_service_logging.py
@@ -0,0 +1,560 @@
+"""Service logging acceptance, backpressure and exit contracts (issue #2748).
+
+Existing handler tests cannot detect a producer waiting for a destination, a
+listener dying on an emit error, or final records abandoned behind a sentinel.
+These tests hold real stream writes, inspect delivered content, and release/join
+all test threads even when an assertion fails.
+"""
+from __future__ import annotations
+
+import asyncio
+import importlib
+import io
+import logging
+import subprocess
+import sys
+import threading
+from pathlib import Path
+
+import pytest
+
+import main
+from vibe import service_logging
+
+
[email protected](autouse=True)
+def logging_lifecycle():
+ service_logging.shutdown_service_logging()
+ root = logging.getLogger()
+ handlers, level = root.handlers[:], root.level
+ yield
+ service_logging.shutdown_service_logging()
+ assert root.handlers == handlers
+ assert root.level == level
+
+
+class HeldStream(io.StringIO):
+ def __init__(self):
+ super().__init__()
+ self.entered = threading.Event()
+ self.release = threading.Event()
+ self.writer_threads = []
+
+ def write(self, value):
+ self.writer_threads.append(threading.get_ident())
+ self.entered.set()
+ if not self.release.wait(5):
+ raise TimeoutError("test did not release destination")
+ return super().write(value)
+
+
+def install_stream(monkeypatch, stream):
+ handler = main.ServiceStreamHandler(stream)
+ monkeypatch.setattr(main, "_build_logging_handlers", lambda _: [handler])
+ main.setup_logging()
+ return handler
+
+
+def test_controller_produces_while_destination_is_blocked(monkeypatch):
+ root = logging.getLogger()
+ previous_handlers, previous_level = root.handlers, root.level
+ root.handlers = [] # Also exercise the original basicConfig path in the regression replay.
+ existing_threads = set(threading.enumerate())
+ stream = HeldStream()
+ handler = install_stream(monkeypatch, stream)
+ listener_threads = set(threading.enumerate()) - existing_threads
+ produced = threading.Event()
+ producer_ids = []
+
+ async def turn():
+ producer_ids.append(threading.get_ident())
+ logging.getLogger("controller-test").info("slow destination")
+ await asyncio.sleep(0)
+ logging.getLogger("controller-test").info("next turn")
+ produced.set()
+
+ producer = threading.Thread(target=lambda: asyncio.run(turn()), daemon=True)
+ producer.start()
+ try:
+ assert stream.entered.wait(2)
+ assert produced.wait(1), "controller waited for stream.write()"
+ assert stream.getvalue() == ""
+ finally:
+ stream.release.set()
+ producer.join(3)
+ service_logging.shutdown_service_logging()
+ handler.close()
+ root.handlers, root.level = previous_handlers, previous_level
+ assert not producer.is_alive()
+ assert all(not thread.is_alive() for thread in listener_threads)
+ assert set(stream.writer_threads).isdisjoint(producer_ids)
+ assert stream.getvalue().count("slow destination") == 1
+ assert stream.getvalue().count("next turn") == 1
+ assert handler._closed
+
+
+def test_full_queue_drops_newest_and_shutdown_delivers_every_accepted_record(monkeypatch, capsys):
+ monkeypatch.setattr(service_logging, "QUEUE_CAPACITY", 2)
+ stream = HeldStream()
+ install_stream(monkeypatch, stream)
+ runtime = service_logging._runtime
+ logger = logging.getLogger("drain-test")
+ stopped = threading.Event()
+ admission_closed = threading.Event()
+ original_close = runtime.handler.close
+
+ def close():
+ original_close()
+ admission_closed.set()
+
+ monkeypatch.setattr(runtime.handler, "close", close)
+ stopper = threading.Thread(target=lambda: (service_logging.shutdown_service_logging(), stopped.set()), daemon=True)
+ try:
+ logger.info("in-flight")
+ assert stream.entered.wait(2)
+ logger.info("final census")
+ logger.warning("final watchdog")
+ for _ in range(10):
+ logger.error("overflow")
+ assert runtime.handler.queue.qsize() == 2
+ assert capsys.readouterr().err == ""
+ stopper.start()
+ assert admission_closed.wait(2)
+ logger.warning("after admission closed")
+ assert not stopped.wait(0.05), "stop returned without delivering accepted records"
+ finally:
+ stream.release.set()
+ if stopper.ident is not None:
+ stopper.join(3)
+ assert stopped.is_set()
+ lines = stream.getvalue().splitlines()
+ assert [line.rsplit(" - ", 1)[-1] for line in lines[:3]] == ["in-flight", "final census", "final watchdog"]
+ assert len(lines) == 4
+ assert "'queue_full': 10" in lines[3]
+ assert "overflow" not in stream.getvalue()
+ assert "after admission closed" not in stream.getvalue()
+ service_logging.shutdown_service_logging()
+
+
[email protected]("failure", ["stream", "handler"])
+def test_destination_and_prepare_errors_are_counted_without_recursion(monkeypatch, capsys, failure):
+ class BrokenStream(io.StringIO):
+ def write(self, value):
+ raise OSError("broken destination")
+
+ class BrokenHandler(logging.Handler):
+ def emit(self, record):
+ raise RuntimeError("handler failure")
+
+ class ReentrantStream(io.StringIO):
+ def write(self, value):
+ logging.getLogger("recursive-destination").warning("must not enqueue recursively")
+ return super().write(value)
+
+ broken = main.ServiceStreamHandler(BrokenStream()) if failure == "stream" else BrokenHandler()
+ output = ReentrantStream()
+ healthy = main.ServiceStreamHandler(output)
+ monkeypatch.setattr(main, "_build_logging_handlers", lambda _: [broken, healthy])
+ main.setup_logging()
+ logger = logging.getLogger("error-test")
+ logger.info("bad formatting %d", "not-a-number")
+ for index in range(3):
+ logger.info("record %s", index)
+ service_logging._runtime.handler.queue.join()
+ service_logging.shutdown_service_logging()
+ lines = output.getvalue().splitlines()
+ assert len(lines) == 4
+ assert [line.rsplit(" - ", 1)[-1] for line in lines[:3]] == ["record 0", "record 1", "record 2"]
+ assert "'handler_error': 3" in lines[3]
+ assert "'prepare_error': 1" in lines[3]
+ assert "'reentrant': 3" in lines[3]
+ assert capsys.readouterr().err == ""
+ assert broken._closed and healthy._closed
+
+
+def test_reentrant_signal_logging_cannot_deadlock_queue_mutex(monkeypatch):
+ output = io.StringIO()
+ install_stream(monkeypatch, output)
+ records = service_logging._runtime.handler.queue
+ original_put = records._put
+
+ def interrupted_put(record):
+ # Exactly the state of a Python signal handler interrupting Queue.put:
+ # Queue's non-reentrant mutex is held by the producer thread.
+ logging.getLogger("signal-test").info("nested signal record")
+ original_put(record)
+
+ monkeypatch.setattr(records, "_put", interrupted_put)
+ produced = threading.Event()
+ producer = threading.Thread(target=lambda: (logging.info("outer record"), produced.set()), daemon=True)
+ producer.start()
+ assert produced.wait(2)
+ producer.join(2)
+ service_logging.shutdown_service_logging()
+ assert "outer record" in output.getvalue()
+ assert "nested signal record" not in output.getvalue()
+ assert "'reentrant': 1" in output.getvalue()
+
+
+def test_repeated_setup_reload_and_shutdown_preserve_format_levels_and_ownership(monkeypatch, tmp_path, capsys):
+ monkeypatch.setattr(main, "get_logs_dir", lambda: tmp_path)
+ monkeypatch.setenv("VIBE_DISABLE_STDOUT_LOGGING", "1")
+ # Early records must not prevent setup from taking ownership of root.
+ logging.warning("before service setup")
+ root = logging.getLogger()
+ original_handlers = root.handlers[:]
+ original_level = root.level
+ threads, handlers = [], []
+ for index in range(3):
+ if index == 1:
+ importlib.reload(main)
+ monkeypatch.setattr(main, "get_logs_dir", lambda: tmp_path)
+ main.setup_logging("WARNING")
+ threads.append(service_logging._runtime.listener._thread)
+ handlers.extend(service_logging._runtime.listener.handlers)
+ assert len(root.handlers) == 1
+ assert root.handlers[0] is service_logging._runtime.handler
+ logging.getLogger("format-test").info("filtered record")
+ logging.getLogger("format-test").warning("visible record %s", index)
+ try:
+ raise ValueError("traceback preserved")
+ except ValueError:
+ logging.getLogger("format-test").exception("final exception")
+ service_logging.shutdown_service_logging()
+ service_logging.shutdown_service_logging()
+ assert all(not thread.is_alive() for thread in threads)
+ assert all(handler.stream is None and handler._closed for handler in handlers)
+ assert root.handlers == original_handlers and root.level == original_level
+ text = (tmp_path / "vibe_remote.log").read_text()
+ assert text.count("visible record") == 3
+ assert "filtered record" not in text
+ assert " - format-test - WARNING - [test_service_logging.py:" in text
+ assert "test_repeated_setup_reload_and_shutdown_preserve_format_levels_and_ownership()" in text
+ assert text.count("Traceback (most recent call last):") == 1
+ assert "ValueError: traceback preserved" in text
+ assert capsys.readouterr().out == ""
+
+
[email protected]("disabled", [False, True])
+def test_listener_keeps_stdout_policy_and_file_rotation(monkeypatch, tmp_path, capsys, disabled):
+ monkeypatch.setattr(main, "get_logs_dir", lambda: tmp_path)
+ monkeypatch.setattr(main, "APPLICATION_LOG_MAX_BYTES", 256)
+ monkeypatch.setattr(main, "APPLICATION_LOG_BACKUP_COUNT", 2)
+ if disabled:
+ monkeypatch.setenv("VIBE_DISABLE_STDOUT_LOGGING", "1")
+ else:
+ monkeypatch.delenv("VIBE_DISABLE_STDOUT_LOGGING", raising=False)
+ main.setup_logging()
+ for index in range(10):
+ logging.getLogger("rotation-test").info("rotation-record-%s %s", index, "x" * 80)
+ service_logging.shutdown_service_logging()
+ files = sorted(tmp_path.glob("vibe_remote.log*"))
+ assert [path.name for path in files] == ["vibe_remote.log", "vibe_remote.log.1", "vibe_remote.log.2"]
+ assert "rotation-record-9" in files[0].read_text()
+ assert "rotation-record-8" in files[1].read_text()
+ assert "rotation-record-7" in files[2].read_text()
+ stdout = capsys.readouterr().out
+ assert stdout.count("rotation-record-") == (0 if disabled else 10)
+
+
+def test_loss_reports_are_rate_limited_and_flush_final_totals(monkeypatch):
+ from types import SimpleNamespace
+
+ class BrokenStream(io.StringIO):
+ def write(self, value):
+ raise OSError("destination unavailable")
+
+ now = [0.0]
+ monkeypatch.setattr(service_logging, "time", SimpleNamespace(monotonic=lambda: now[0]))
+ output = io.StringIO()
+ handlers = [main.ServiceStreamHandler(BrokenStream()), main.ServiceStreamHandler(output)]
+ monkeypatch.setattr(main, "_build_logging_handlers", lambda _: handlers)
+ main.setup_logging()
+ records = service_logging._runtime.handler.queue
+ logger = logging.getLogger("rate-test")
+ for index in range(20):
+ logger.info("initial %s", index)
+ records.join()
+ assert "Service logging losses" not in output.getvalue()
+ now[0] = 31.0
+ for index in range(20):
+ logger.info("burst %s", index)
+ records.join()
+ assert output.getvalue().count("Service logging losses") == 1
+ now[0] = 62.0
+ logger.info("after interval")
+ records.join()
+ assert output.getvalue().count("Service logging losses") == 2
+ service_logging.shutdown_service_logging()
+ # Two earlier summary writes also failed at the broken destination.
+ assert output.getvalue().count("Service logging losses") == 3
+ assert "'handler_error': 43" in output.getvalue().splitlines()[-1]
+
+
+def break_file_destination(monkeypatch, handler, failure):
+ def fail(*args):
+ raise OSError("destination unavailable")
+
+ if failure == "write":
+ monkeypatch.setattr(handler.stream, "write", fail)
+ else:
+ monkeypatch.setattr(handler, "shouldRollover", lambda record: True)
+ monkeypatch.setattr(handler, "doRollover", fail)
+
+
[email protected]("file_failure", ["write", "rotation"])
[email protected]("stderr_failure", [None, "write", "flush", "missing"])
+def test_file_only_failure_has_bounded_listener_stderr_fallback(monkeypatch, tmp_path, file_failure, stderr_failure):
+ from types import SimpleNamespace
+
+ class Stderr(io.StringIO):
+ def __init__(self):
+ super().__init__()
+ self.attempts = []
+
+ def write(self, value):
+ self.attempts.append((threading.get_ident(), value))
+ if stderr_failure == "write":
+ raise OSError("stderr unavailable")
+ return super().write(value)
+
+ def flush(self):
+ if stderr_failure == "flush":
+ raise OSError("stderr flush unavailable")
+ return super().flush()
+
+ now = [0.0]
+ monkeypatch.setattr(service_logging, "time", SimpleNamespace(monotonic=lambda: now[0]))
+ stderr = Stderr()
+ monkeypatch.setattr(sys, "__stderr__", None if stderr_failure == "missing" else stderr)
+ monkeypatch.setenv("VIBE_DISABLE_STDOUT_LOGGING", "1")
+ monkeypatch.setattr(main, "get_logs_dir", lambda: tmp_path)
+ main.setup_logging()
+ runtime = service_logging._runtime
+ assert len(runtime.listener.handlers) == 1
+ file_handler = runtime.listener.handlers[0]
+ listener = runtime.listener._thread
+ break_file_destination(monkeypatch, file_handler, file_failure)
+ logger = logging.getLogger("file-failure")
+ for _ in range(20):
+ logger.info("private_payload_never_in_stderr")
+ runtime.handler.queue.join()
+ assert stderr.attempts == []
+ now[0] = 31.0
+ for _ in range(20):
+ logger.info("private_payload_never_in_stderr")
+ runtime.handler.queue.join()
+ assert len(stderr.attempts) == (0 if stderr_failure == "missing" else 1)
+ now[0] = 62.0
+ logger.info("private_payload_never_in_stderr")
+ runtime.handler.queue.join()
+ assert len(stderr.attempts) == (0 if stderr_failure == "missing" else 2)
+ service_logging.shutdown_service_logging()
+ assert not listener.is_alive()
+ assert file_handler._closed and file_handler.stream is None
+ assert len(stderr.attempts) == (0 if stderr_failure == "missing" else 3)
+ if stderr_failure != "missing":
+ for thread_id, text in stderr.attempts:
+ assert thread_id == listener.ident
+ assert len(text.encode("utf-8")) <= 1024
+ assert "private_payload_never_in_stderr" not in text
+ assert text.startswith("Service logging destinations unavailable; losses=")
+ assert "'handler_error': 43" in stderr.attempts[-1][1]
+ if stderr_failure != "write":
+ assert stderr.getvalue().count("Service logging destinations unavailable") == 3
+ assert (tmp_path / "vibe_remote.log").read_text() == ""
+
+
+def test_blocked_stderr_fallback_never_blocks_producer(monkeypatch, tmp_path):
+ from types import SimpleNamespace
+
+ now = [0.0]
+ monkeypatch.setattr(service_logging, "time", SimpleNamespace(monotonic=lambda: now[0]))
+ monkeypatch.setattr(service_logging, "QUEUE_CAPACITY", 2)
+ stderr = HeldStream()
+ monkeypatch.setattr(sys, "__stderr__", stderr)
+ monkeypatch.setenv("VIBE_DISABLE_STDOUT_LOGGING", "1")
+ monkeypatch.setattr(main, "get_logs_dir", lambda: tmp_path)
+ main.setup_logging()
+ runtime = service_logging._runtime
+ listener = runtime.listener._thread
+ break_file_destination(monkeypatch, runtime.listener.handlers[0], "write")
+ now[0] = 31.0
+ first_returned = threading.Event()
+ continue_producing = threading.Event()
+ produced = threading.Event()
+
+ def produce():
+ logger = logging.getLogger("blocked-stderr")
+ logger.error("trigger fallback")
+ first_returned.set()
+ if not continue_producing.wait(3):
+ return
+ for _ in range(100):
+ logger.error("during blocked stderr")
+ produced.set()
+
+ producer = threading.Thread(target=produce, daemon=True)
+ producer.start()
+ try:
+ assert stderr.entered.wait(2)
+ assert first_returned.wait(1), "initial fallback ran on producer"
+ continue_producing.set()
+ assert produced.wait(1), "producer waited on blocked stderr/queue space"
+ assert runtime.handler.queue.qsize() == 2
+ assert stderr.getvalue() == ""
+ finally:
+ continue_producing.set()
+ stderr.release.set()
+ producer.join(3)
+ service_logging.shutdown_service_logging()
+ assert not producer.is_alive() and not listener.is_alive()
+ assert set(stderr.writer_threads) == {listener.ident}
+ assert "'queue_full': 98" in stderr.getvalue()
+ assert stderr.getvalue().count("Service logging destinations unavailable") == 2
+
+
+def test_partial_handler_construction_closes_open_file(monkeypatch, tmp_path):
+ monkeypatch.delenv("VIBE_DISABLE_STDOUT_LOGGING", raising=False)
+ file_handler = main.ServiceFileHandler(tmp_path / "vibe_remote.log")
+ monkeypatch.setattr(main, "ServiceFileHandler", lambda *args, **kwargs: file_handler)
+
+ def fail_stream(*args, **kwargs):
+ raise RuntimeError("stdout handler unavailable")
+
+ monkeypatch.setattr(main, "ServiceStreamHandler", fail_stream)
+ with pytest.raises(RuntimeError, match="stdout handler unavailable"):
+ main._build_logging_handlers(str(tmp_path))
+ assert file_handler.stream is None and file_handler._closed
+ service_logging.shutdown_service_logging()
+
+
+def test_repeated_setup_opens_new_log_only_after_previous_drain_rotation(monkeypatch, tmp_path):
+ held = HeldStream()
+ built_again = threading.Event()
+ setup_started = threading.Event()
+ setup_done = threading.Event()
+ build_count = 0
+
+ def build(_):
+ nonlocal build_count
+ build_count += 1
+ if build_count == 2:
+ built_again.set()
+ return [
+ main.ServiceStreamHandler(held if build_count == 1 else io.StringIO()),
+ main.ServiceFileHandler(tmp_path / "vibe_remote.log", maxBytes=256, backupCount=2),
+ ]
+
+ monkeypatch.setattr(main, "_build_logging_handlers", build)
+ main.setup_logging()
+ logger = logging.getLogger("replace-test")
+ logger.info("old record %s", "x" * 120)
+
+ def reconfigure():
+ setup_started.set()
+ main.setup_logging()
+ setup_done.set()
+
+ replacing = threading.Thread(target=reconfigure, daemon=True)
+ try:
+ assert held.entered.wait(2)
+ logger.info("old drain rotates %s", "x" * 120)
+ replacing.start()
+ assert setup_started.wait(2)
+ assert not built_again.wait(0.05), "opened new inode before previous drain/rotation"
+ finally:
+ held.release.set()
+ if replacing.ident is not None:
+ replacing.join(3)
+ assert setup_done.is_set()
+ logger.info("new current file")
+ service_logging.shutdown_service_logging()
+ assert "new current file" in (tmp_path / "vibe_remote.log").read_text()
+ assert "new current file" not in (tmp_path / "vibe_remote.log.1").read_text()
+
+
[email protected]("start_then_fail", [False, True])
+def test_listener_start_failure_closes_handlers_and_allows_retry(monkeypatch, tmp_path, start_then_fail):
+ monkeypatch.setenv("VIBE_DISABLE_STDOUT_LOGGING", "1")
+ handler = main._build_logging_handlers(str(tmp_path))[0]
+ monkeypatch.setattr(main, "_build_logging_handlers", lambda _: [handler])
+ original_handlers = logging.getLogger().handlers[:]
+ original_start = service_logging._Listener.start
+ workers = []
+ with monkeypatch.context() as patch:
+ def fail_start(self):
+ if start_then_fail:
+ original_start(self)
+ workers.append(self._thread)
+ raise RuntimeError("cannot start listener")
+
+ patch.setattr(service_logging._Listener, "start", fail_start)
+ with pytest.raises(RuntimeError, match="cannot start listener"):
+ main.setup_logging()
+ assert handler.stream is None and handler._closed
+ assert all(not thread.is_alive() for thread in workers)
+ assert logging.getLogger().handlers == original_handlers
+ service_logging.shutdown_service_logging()
+ output = io.StringIO()
+ install_stream(monkeypatch, output)
+ logging.warning("retry works")
+ service_logging.shutdown_service_logging()
+ assert "retry works" in output.getvalue()
+
+
+def test_atexit_fallback_drains_before_logging_closes_destinations(tmp_path):
+ code = """
+import logging
+from pathlib import Path
+import sys
+import main
+main.get_logs_dir = lambda: Path(sys.argv[1])
+main.setup_logging()
+for index in range(50):
+ logging.getLogger('exit-test').warning('exit record %s', index)
+"""
+ result = subprocess.run(
+ [sys.executable, "-c", code, str(tmp_path)],
+ cwd=Path(__file__).resolve().parents[1], capture_output=True, text=True, timeout=10,
+ )
+ assert result.returncode == 0, result.stderr
+ text = (tmp_path / "vibe_remote.log").read_text()
+ assert text.count("exit record") == 50
+ assert "exit record 49" in text
+
+
[email protected]("release_fails", [False, True])
+def test_startup_failure_drains_error_and_cleanup_even_when_cleanup_raises(monkeypatch, release_fails):
+ output = io.StringIO()
+ handler = main.ServiceStreamHandler(output)
+ monkeypatch.setattr(main, "_build_logging_handlers", lambda _: [handler])
+ monkeypatch.setattr(main, "acquire_service_instance_lock", lambda: None)
+ from types import SimpleNamespace
+
+ monkeypatch.setattr(main, "load_config", lambda: SimpleNamespace(runtime=SimpleNamespace(log_level="INFO")))
+
+ def fail_startup():
+ raise RuntimeError("startup failure after logging setup")
+
+ def release():
+ # No successor may acquire the same log path until this runtime's last
+ # write/rotation and handler close have completed.
+ assert service_logging._runtime is None
+ assert handler._closed
+ assert "final service diagnostics cleanup" in output.getvalue()
+ if release_fails:
+ raise RuntimeError("cleanup failure")
+
+ monkeypatch.setattr(main, "_stop_macos_session_diagnostics", lambda _: logging.warning("final service diagnostics cleanup"))
+ monkeypatch.setattr(main, "apply_claude_sdk_patches", fail_startup)
+ monkeypatch.setattr(main, "release_service_instance_lock", release)
+ import core.managed_skills
+
+ monkeypatch.setattr(core.managed_skills, "prepare_builtin_skills", lambda: None)
+ with pytest.raises(RuntimeError if release_fails else SystemExit):
+ main.main()
+ assert "Failed to start: startup failure after logging setup" in output.getvalue()
+ assert "final service diagnostics cleanup" in output.getvalue()
+ assert service_logging._runtime is None
+ assert handler._closed
diff --git a/vibe/service_logging.py b/vibe/service_logging.py
new file mode 100644
index 000000000..9788a147a
--- /dev/null
+++ b/vibe/service_logging.py
@@ -0,0 +1,240 @@
+"""Bounded, service-owned logging. See docs/plans/nonblocking-service-logging.md.
+
+Producers prepare records and enqueue without waiting for destination I/O. Full
+queues drop the newest record; accepted records are not evicted. Only the final
+service-exit path may wait for drain (and thus for slow/hung destinations).
+"""
+from __future__ import annotations
+
+import atexit
+import logging
+import sys
+import threading
+import time
+from collections import Counter
+from collections.abc import Callable
+from logging.handlers import QueueHandler, QueueListener, RotatingFileHandler
+from queue import Full, Queue
+
+QUEUE_CAPACITY = 8192
+REPORT_INTERVAL_SECONDS = 30.0
+
+
+class _RaiseHandlerErrors:
+ def handleError(self, record: logging.LogRecord) -> None:
+ # Stdlib stream/file emit() normally prints to stderr and swallows these.
+ # Let our listener count them instead, without recursively logging them.
+ raise
+
+
+class ServiceFileHandler(_RaiseHandlerErrors, RotatingFileHandler):
+ pass
+
+
+class ServiceStreamHandler(_RaiseHandlerErrors, logging.StreamHandler):
+ pass
+
+
+class _Losses:
+ def __init__(self) -> None:
+ self._lock = threading.RLock()
+ self._counts: Counter[str] = Counter()
+
+ def add(self, reason: str) -> None:
+ with self._lock:
+ self._counts[reason] += 1
+
+ def snapshot(self) -> dict[str, int]:
+ with self._lock:
+ return dict(self._counts)
+
+
+class _QueueHandler(QueueHandler):
+ def __init__(self, records: Queue, losses: _Losses, context: threading.local) -> None:
+ super().__init__(records)
+ self.losses = losses
+ self.context = context
+ self.accepting = True
+
+ def emit(self, record: logging.LogRecord) -> None:
+ if not self.accepting:
+ return
+ # A Python signal can log while this thread holds Queue's non-reentrant
+ # mutex. Handler/formatter logging can recurse too. Never re-enter either.
+ if getattr(self.context, "logging", False):
+ self.losses.add("reentrant")
+ return
+ self.context.logging = True
+ try:
+ super().emit(record)
+ finally:
+ self.context.logging = False
+
+ def handleError(self, record: logging.LogRecord) -> None:
+ # QueueHandler calls this for full queues and failed record preparation.
+ self.losses.add("queue_full" if isinstance(sys.exc_info()[1], Full) else "prepare_error")
+
+ def close(self) -> None:
+ # Serialize the admission boundary with emit, not with listener I/O.
+ self.acquire()
+ try:
+ self.accepting = False
+ finally:
+ self.release()
+ super().close()
+
+
+class _Listener(QueueListener):
+ # QueueListener's stubs expose only a put_nowait queue protocol. This
+ # service owns a concrete bounded Queue and a sentinel that cannot be a log.
+ queue: Queue
+ _sentinel = None
+
+ def __init__(self, records: Queue, handlers: list[logging.Handler], losses: _Losses, context: threading.local) -> None:
+ super().__init__(records, *handlers, respect_handler_level=True)
+ self.losses = losses
+ self.context = context
+ self._reported: dict[str, int] = {}
+ self._last_report = time.monotonic()
+
+ def _deliver(self, record: logging.LogRecord) -> bool:
+ delivered = False
+ for handler in self.handlers:
+ if record.levelno < handler.level:
+ continue
+ try:
+ if handler.handle(record):
+ delivered = True
+ except Exception:
+ self.losses.add("handler_error")
+ return delivered
+
+ def _report_losses(self, *, final: bool = False) -> None:
+ now = time.monotonic()
+ if not final and now - self._last_report < REPORT_INTERVAL_SECONDS:
+ return
+ counts = self.losses.snapshot()
+ if counts == self._reported:
+ return
+ self._last_report = now
+ self._reported = counts
+ delivered = self._deliver(logging.LogRecord(
+ __name__, logging.WARNING, __file__, 0,
+ "Service logging losses (cumulative): %s", (counts,), None,
+ ))
+ if not delivered:
+ # The background service has only the file sink. Do not send its
+ # failure notice solely back to that same failed destination. This
+ # best-effort escape path runs only on the listener, shares the
+ # summary rate limit, and never contains record payloads.
+ try:
+ stderr = sys.__stderr__
+ if stderr is not None:
+ message = f"Service logging destinations unavailable; losses={counts}"
+ stderr.write(message[:1023] + "\n")
+ stderr.flush()
+ except Exception:
+ # stderr can be unavailable too. No recursion, retries, or
+ # producer-side fallback; a blocked stderr only holds this worker.
+ pass
+
+ def handle(self, record: logging.LogRecord) -> None:
+ self._deliver(record)
+ self._report_losses()
+
+ def enqueue_sentinel(self) -> None:
+ # Called only after admission closes, at service exit, off the live loop.
+ # put_nowait (the stdlib default) would fail if the bounded queue is full.
+ self.queue.put(self._sentinel)
+
+ def _monitor(self) -> None:
+ # Destination callbacks must not feed records back into their own queue.
+ self.context.logging = True
+ try:
+ # The stdlib worker hook exists on every supported Python version,
+ # but is deliberately omitted from typeshed's public interface.
+ super()._monitor() # type: ignore[misc]
+ finally:
+ self._report_losses(final=True)
+ for handler in self.handlers:
+ try:
+ handler.close()
+ except Exception:
+ self.losses.add("close_error")
+ self.context.logging = False
+
+
+class _ServiceLogging:
+ def __init__(self, handlers: list[logging.Handler]) -> None:
+ self.root = logging.getLogger()
+ self.previous_handlers = self.root.handlers[:]
+ self.previous_level = self.root.level
+ records: Queue = Queue(maxsize=QUEUE_CAPACITY)
+ losses = _Losses()
+ context = threading.local()
+ self.handler = _QueueHandler(records, losses, context)
+ self.listener = _Listener(records, handlers, losses, context)
+
+ def stop(self) -> None:
+ self.handler.close()
+ self.listener.stop()
+ # Keep the closed queue handler installed throughout drain so late
+ # module-level logging cannot basicConfig() a synchronous stderr handler.
+ self.root.handlers = self.previous_handlers
+ self.root.setLevel(self.previous_level)
+
+
+_lifecycle_lock = threading.RLock()
+_runtime: _ServiceLogging | None = None
+
+
+def configure_service_logging(
+ build_handlers: Callable[[], list[logging.Handler]], level: int, formatter: logging.Formatter,
+) -> None:
+ """Replace the previous setup, opening new files only after its final rotation."""
+ global _runtime
+ with _lifecycle_lock:
+ handlers: list[logging.Handler] = []
+ runtime = None
+ try:
+ if not isinstance(level, int):
+ raise ValueError("Service log level must be an integer")
+ shutdown_service_logging()
+ handlers = build_handlers()
+ for handler in handlers:
+ handler.setFormatter(formatter)
+ runtime = _ServiceLogging(handlers)
+ runtime.listener.start()
+ except BaseException:
+ # Thread.start can be interrupted after the worker actually started.
+ thread = getattr(runtime.listener, "_thread", None) if runtime else None
+ if runtime is not None and thread is not None and thread.is_alive():
+ runtime.stop()
+ else:
+ if runtime is not None:
+ runtime.handler.close()
+ for handler in handlers:
+ handler.close()
+ raise
+ runtime.root.handlers = [runtime.handler]
+ runtime.root.setLevel(level)
+ _runtime = runtime
+
+
+def shutdown_service_logging() -> None:
+ """Drain after service producers stop; safe before setup and on repeated calls.
+
+ Accepted records are attempted in FIFO order on every destination. This waits
+ for destination I/O, with no timeout: a permanently hung destination requires
+ process termination, which forfeits the remaining queued records.
+ """
+ global _runtime
+ with _lifecycle_lock:
+ if _runtime is not None:
+ _runtime.stop()
+ _runtime = None
+
+
+# Fallback for callers using setup outside main; main itself drains explicitly.
+# Registered after logging's hook so drain precedes logging.shutdown at exit.
+atexit.register(shutdown_service_logging)