Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.
Python agentagent-osaiai-agentsai-partnerchatopsclaudeclaude-codecodexdevtoolsdiscord-botlark-botllmlocal-firstlong-term-memoryopencodeslack-botvibe-codingvibeywechat
fix(logging): move service log I/O off the controller thread (#3053)
Call White5ecae62dc278
diff --git a/docs/plans/nonblocking-service-logging.md b/docs/plans/nonblocking-service-logging.mdnew file mode 100644index 000000000..bcbee9dae--- /dev/null+++ b/docs/plans/nonblocking-service-logging.md@@ -0,0 +1,98 @@+# Non-blocking service logging (#2748)++## Change contract++This is a proactive mitigation: the recent current-master soak did not reproduce a+logging-attributed controller-loop stall. Only service logging ownership and its+entry/exit lifecycle change; no controller, database, transport, or call-site+changes are intended.++- Root records go through one QueueHandler; one QueueListener owns the existing+ rotating file and optional stdout destinations, formatter and levels.+- A queue holds at most 8192 pending records (plus the listener's in-flight+ record); it is a record-count bound, not a byte limit. Full queues drop the+ incoming record (including high-severity/final records); accepted records are+ never evicted. Ordinary QueueHandler message/traceback preparation is retained.+- Cumulative `queue_full`, `prepare_error`, `handler_error` and `reentrant`+ counters are reported directly to existing destinations, at most once per+ 30 seconds while processing records and once at final drain if totals changed.+ No producer writes to stderr on overload/error. Failing destinations are+ attempted independently and cannot kill the listener. If no destination+ accepts an aggregate diagnostic, the listener writes/flushes a single line of+ at most 1024 ASCII bytes directly to process stderr (`sys.__stderr__`). It+ shares the 30-second summary rate limit plus the final-drain attempt, contains+ only cumulative counters (no record payloads), and never logs or re-enqueues.+ Missing/broken stderr is caught without retries; blocked stderr only blocks+ the listener, not producers. This escape path is best effort, not delivery+ assurance when stderr also fails or shares the unavailable filesystem.+ A diagnostic's own failed delivery is counted, not recursively retried.+ Reporting is activity-driven: isolated preparation errors can wait until a+ subsequent accepted record or shutdown; 30 seconds is not a delivery SLA.+- Nested logging while producing (including a signal interrupting Queue's+ non-reentrant mutex) or inside a destination is suppressed/countable, avoiding+ deadlock and feedback loops. Shutdown signal handling itself still runs; only+ a nested record can be lost under this policy.+- Shutdown closes admission only after controller and desktop/diagnostic+ cleanup. It drains all accepted records, attempts each destination, joins the+ listener and closes owned handlers BEFORE releasing the service instance lock,+ so a successor cannot acquire ownership while this logger still rotates files.+ Lock release remains in a `finally` and retains the existing lock-safety gate.+ Setup/shutdown are repeatable; partially constructed setups leave no listener+ or file behind.+ New destinations open only after the prior drain/rotation finishes. Previous+ root handlers and level are detached while the runtime owns root, then restored+ on stop; they are not closed by this runtime. A closed admission handler stays+ on root throughout drain to prevent implicit synchronous stderr configuration.+ `main()` explicitly drains even if cleanup raises; an atexit hook is a fallback+ for other callers. Sink close errors do not prevent the other sinks closing.+ The service deliberately owns root configuration; third-party root handlers+ are inactive during ownership, and concurrent third-party reconfiguration is+ not supported. Lock-release debug diagnostics and late daemon records after+ drain use restored pre-service logging rather than this service file.+- Drain is not a disk-durability promise: failed destinations cannot guarantee+ delivery. Lossless draining also cannot bound exit time if a destination never+ returns (including stderr fallback). The service retains its lock during that+ wait. The existing POSIX managed stop gives the WHOLE shutdown five seconds+ before SIGKILL; cleanup and logging share that grace, and this change does not+ extend it. Windows managed termination also does not promise Python drain.+ Forced process termination can lose queued/in-flight records and skips atexit.+ Drain is conditional on destination progress and absence of forced termination,+ NOT a bounded-shutdown promise. Normal logging still performs record/message+ preparation and short in-process lock operations.+- No deployment, service restart, Incus update, or merge is authorized here.++## Validation plan++Existing tests only exercise handlers synchronously and the surrounding service+lifecycle; they do not exercise asynchronous acceptance or drain. Extend that+owner boundary with deterministic blocked destinations: the producer must finish+before release, accepted records must arrive after release, overflow must be+counted without recursive diagnostics, and shutdown must wait for actual final+record delivery. Exercise real file rotation/stdout suppression, handler errors,+setup failure/repetition, early startup failure and final service cleanup logging.+Run focused lifecycle tests, changed-file Ruff and relevant repository checks;+leave full platform/suite coverage to CI.++## Progress++- [x] Inspect setup, shutdown and existing logging tests on current origin/master.+- [x] Implement runtime and service lifecycle ownership.+- [x] Prove slow-I/O, overflow, drain, failure and compatibility contracts.+- [x] Inspect complete working-tree diff, including staged/unstaged changes.+- Exact-head review/CI delivery evidence will be recorded in the PR.++## Local evidence++- Combined logging, lifecycle, config compatibility, controller shutdown,+ watchdog and service-lock selection: 199 passed, 3 platform skips and 4+ subtests passed on Python 3.12 after the final ordering/fallback revision.+- File-only write/rotation failures exercise successful, missing, write-failing+ and flush-failing stderr; bounded diagnostics, rate limiting and blocked stderr+ producer independence are verified. Replaying the pre-fallback reporter makes+ the regression fail because no stderr notice arrives.+- The slow-stream regression was replayed with `setup_logging` extracted from+ origin/master: it failed specifically at `controller waited for stream.write()`;+ the same test passes with the queue runtime.+- Changed-file Ruff and mypy (production files, imports skipped) pass.+- No live service/Incus operations were performed. Full cross-platform coverage+ remains with CI; the local environment is hermetic and owns its streams/files.diff --git a/main.py b/main.pyindex 1e092fb86..006172b54 100644--- a/main.py+++ b/main.py@@ -5,11 +5,16 @@ import osimport signalimport sysimport logging-from logging.handlers import RotatingFileHandlerfrom typing import Anyfrom config.paths import ensure_data_dirs, get_logs_dirfrom vibe.logging_config import APPLICATION_LOG_BACKUP_COUNT, APPLICATION_LOG_MAX_BYTES+from vibe.service_logging import (+ ServiceFileHandler,+ ServiceStreamHandler,+ configure_service_logging,+ shutdown_service_logging,+)from vibe.macos_session_diagnostics import start_macos_session_diagnosticsfrom vibe.runtime import (ServiceAlreadyRunningError,@@ -22,32 +27,30 @@ from vibe.runtime import (def _build_logging_handlers(logs_dir: str) -> list[logging.Handler]:handlers: list[logging.Handler] = [- RotatingFileHandler(+ ServiceFileHandler(f"{logs_dir}/vibe_remote.log",maxBytes=APPLICATION_LOG_MAX_BYTES,backupCount=APPLICATION_LOG_BACKUP_COUNT,)]- if os.environ.get("VIBE_DISABLE_STDOUT_LOGGING", "").lower() not in {"1", "true", "yes"}:- handlers.insert(0, logging.StreamHandler(sys.stdout))+ try:+ if os.environ.get("VIBE_DISABLE_STDOUT_LOGGING", "").lower() not in {"1", "true", "yes"}:+ handlers.insert(0, ServiceStreamHandler(sys.stdout))+ except BaseException:+ for handler in handlers:+ handler.close()+ raisereturn handlers-def setup_logging(level: str = "INFO"):- """Setup logging configuration with file location and line numbers"""- # Create a custom formatter with file location+def setup_logging(level: str = "INFO") -> None:+ """Keep destination I/O on the service logging listener, not the controller."""+ log_level = getattr(logging, level.upper())log_format = '%(asctime)s - %(name)s - %(levelname)s - [%(filename)s:%(lineno)d] - %(funcName)s() - %(message)s'-- # For development, you can use this more detailed format:- # log_format = '%(asctime)s - %(name)s - %(levelname)s - [%(pathname)s:%(lineno)d] - %(funcName)s() - %(message)s'-ensure_data_dirs()logs_dir = str(get_logs_dir())-- logging.basicConfig(- level=getattr(logging, level.upper()),- format=log_format,- handlers=_build_logging_handlers(logs_dir),+ configure_service_logging(+ lambda: _build_logging_handlers(logs_dir), log_level, logging.Formatter(log_format),)@@ -261,13 +264,20 @@ def main():logging.error(f"Failed to start: {e}")sys.exit(1)finally:- lock_safe = bool(- controller is None- or getattr(controller, "service_lock_safe_to_release", True)- )- if lock_acquired and lock_safe:- release_service_instance_lock()- _stop_macos_session_diagnostics(macos_session_diagnostics)+ try:+ _stop_macos_session_diagnostics(macos_session_diagnostics)+ finally:+ try:+ # Controller.run() has finished its census/watchdog cleanup.+ # Finish rotating/writing before another service can own the log.+ shutdown_service_logging()+ finally:+ lock_safe = bool(+ controller is None+ or getattr(controller, "service_lock_safe_to_release", True)+ )+ if lock_acquired and lock_safe:+ release_service_instance_lock()if __name__ == "__main__":diff --git a/tests/test_macos_session_diagnostics.py b/tests/test_macos_session_diagnostics.pyindex f6a1e8173..75dc8fc7a 100644--- a/tests/test_macos_session_diagnostics.py+++ b/tests/test_macos_session_diagnostics.py@@ -390,7 +390,7 @@ def test_monitor_stop_joins_its_worker() -> None:assert threads[0].joined is True-def test_service_main_owns_monitor_shutdown(monkeypatch) -> None:+def test_service_main_owns_monitor_shutdown(monkeypatch, tmp_path) -> None:import config.v2_compatimport core.controllerimport core.process_diagnostics@@ -404,6 +404,7 @@ def test_service_main_owns_monitor_shutdown(monkeypatch) -> None:class FakeMonitor:def stop(self) -> None:calls.append("diagnostics.stop")+ logging.getLogger("test.service-cleanup").warning("final diagnostic record")class FakeRunningLoop:def is_closed(self) -> bool:@@ -429,6 +430,7 @@ def test_service_main_owns_monitor_shutdown(monkeypatch) -> None:calls.append("controller.run")handlers[signal.SIGTERM](signal.SIGTERM, None)calls.append("controller.after-signal")+ logging.getLogger("test.service-cleanup").warning("final census/watchdog record")def request_shutdown(self, reason: str) -> None:calls.append(f"controller.shutdown:{reason}")@@ -452,7 +454,8 @@ def test_service_main_owns_monitor_shutdown(monkeypatch) -> None:lambda **kwargs: calls.append(f"desktop-ui.stop:{kwargs}") or True,)monkeypatch.setattr(service_main, "load_config", lambda: loaded_config)- monkeypatch.setattr(service_main, "setup_logging", lambda level: None)+ monkeypatch.setattr(service_main, "get_logs_dir", lambda: tmp_path)+ monkeypatch.setenv("VIBE_DISABLE_STDOUT_LOGGING", "1")monkeypatch.setattr(service_main, "apply_claude_sdk_patches", lambda: None)monkeypatch.setattr(service_main,@@ -475,6 +478,15 @@ def test_service_main_owns_monitor_shutdown(monkeypatch) -> None:service_main.main()+ # Real queued logging must outlive signal handling and all service cleanup.+ text = (tmp_path / "vibe_remote.log").read_text()+ assert "Shutting down after signal" in text+ assert "final census/watchdog record" in text+ assert text.count("final diagnostic record") == 2+ from vibe import service_logging++ assert service_logging._runtime is None+# The migration runs under the lock and before anything is constructed# against the schema it produces.assert calls.index("lock.acquire") < calls.index("sqlite.prepare")diff --git a/tests/test_service_logging.py b/tests/test_service_logging.pynew file mode 100644index 000000000..7e56980e5--- /dev/null+++ b/tests/test_service_logging.py@@ -0,0 +1,560 @@+"""Service logging acceptance, backpressure and exit contracts (issue #2748).++Existing handler tests cannot detect a producer waiting for a destination, a+listener dying on an emit error, or final records abandoned behind a sentinel.+These tests hold real stream writes, inspect delivered content, and release/join+all test threads even when an assertion fails.+"""+from __future__ import annotations++import asyncio+import importlib+import io+import logging+import subprocess+import sys+import threading+from pathlib import Path++import pytest++import main+from vibe import service_logging++[email protected](autouse=True)+def logging_lifecycle():+ service_logging.shutdown_service_logging()+ root = logging.getLogger()+ handlers, level = root.handlers[:], root.level+ yield+ service_logging.shutdown_service_logging()+ assert root.handlers == handlers+ assert root.level == level+++class HeldStream(io.StringIO):+ def __init__(self):+ super().__init__()+ self.entered = threading.Event()+ self.release = threading.Event()+ self.writer_threads = []++ def write(self, value):+ self.writer_threads.append(threading.get_ident())+ self.entered.set()+ if not self.release.wait(5):+ raise TimeoutError("test did not release destination")+ return super().write(value)+++def install_stream(monkeypatch, stream):+ handler = main.ServiceStreamHandler(stream)+ monkeypatch.setattr(main, "_build_logging_handlers", lambda _: [handler])+ main.setup_logging()+ return handler+++def test_controller_produces_while_destination_is_blocked(monkeypatch):+ root = logging.getLogger()+ previous_handlers, previous_level = root.handlers, root.level+ root.handlers = [] # Also exercise the original basicConfig path in the regression replay.+ existing_threads = set(threading.enumerate())+ stream = HeldStream()+ handler = install_stream(monkeypatch, stream)+ listener_threads = set(threading.enumerate()) - existing_threads+ produced = threading.Event()+ producer_ids = []++ async def turn():+ producer_ids.append(threading.get_ident())+ logging.getLogger("controller-test").info("slow destination")+ await asyncio.sleep(0)+ logging.getLogger("controller-test").info("next turn")+ produced.set()++ producer = threading.Thread(target=lambda: asyncio.run(turn()), daemon=True)+ producer.start()+ try:+ assert stream.entered.wait(2)+ assert produced.wait(1), "controller waited for stream.write()"+ assert stream.getvalue() == ""+ finally:+ stream.release.set()+ producer.join(3)+ service_logging.shutdown_service_logging()+ handler.close()+ root.handlers, root.level = previous_handlers, previous_level+ assert not producer.is_alive()+ assert all(not thread.is_alive() for thread in listener_threads)+ assert set(stream.writer_threads).isdisjoint(producer_ids)+ assert stream.getvalue().count("slow destination") == 1+ assert stream.getvalue().count("next turn") == 1+ assert handler._closed+++def test_full_queue_drops_newest_and_shutdown_delivers_every_accepted_record(monkeypatch, capsys):+ monkeypatch.setattr(service_logging, "QUEUE_CAPACITY", 2)+ stream = HeldStream()+ install_stream(monkeypatch, stream)+ runtime = service_logging._runtime+ logger = logging.getLogger("drain-test")+ stopped = threading.Event()+ admission_closed = threading.Event()+ original_close = runtime.handler.close++ def close():+ original_close()+ admission_closed.set()++ monkeypatch.setattr(runtime.handler, "close", close)+ stopper = threading.Thread(target=lambda: (service_logging.shutdown_service_logging(), stopped.set()), daemon=True)+ try:+ logger.info("in-flight")+ assert stream.entered.wait(2)+ logger.info("final census")+ logger.warning("final watchdog")+ for _ in range(10):+ logger.error("overflow")+ assert runtime.handler.queue.qsize() == 2+ assert capsys.readouterr().err == ""+ stopper.start()+ assert admission_closed.wait(2)+ logger.warning("after admission closed")+ assert not stopped.wait(0.05), "stop returned without delivering accepted records"+ finally:+ stream.release.set()+ if stopper.ident is not None:+ stopper.join(3)+ assert stopped.is_set()+ lines = stream.getvalue().splitlines()+ assert [line.rsplit(" - ", 1)[-1] for line in lines[:3]] == ["in-flight", "final census", "final watchdog"]+ assert len(lines) == 4+ assert "'queue_full': 10" in lines[3]+ assert "overflow" not in stream.getvalue()+ assert "after admission closed" not in stream.getvalue()+ service_logging.shutdown_service_logging()++[email protected]("failure", ["stream", "handler"])+def test_destination_and_prepare_errors_are_counted_without_recursion(monkeypatch, capsys, failure):+ class BrokenStream(io.StringIO):+ def write(self, value):+ raise OSError("broken destination")++ class BrokenHandler(logging.Handler):+ def emit(self, record):+ raise RuntimeError("handler failure")++ class ReentrantStream(io.StringIO):+ def write(self, value):+ logging.getLogger("recursive-destination").warning("must not enqueue recursively")+ return super().write(value)++ broken = main.ServiceStreamHandler(BrokenStream()) if failure == "stream" else BrokenHandler()+ output = ReentrantStream()+ healthy = main.ServiceStreamHandler(output)+ monkeypatch.setattr(main, "_build_logging_handlers", lambda _: [broken, healthy])+ main.setup_logging()+ logger = logging.getLogger("error-test")+ logger.info("bad formatting %d", "not-a-number")+ for index in range(3):+ logger.info("record %s", index)+ service_logging._runtime.handler.queue.join()+ service_logging.shutdown_service_logging()+ lines = output.getvalue().splitlines()+ assert len(lines) == 4+ assert [line.rsplit(" - ", 1)[-1] for line in lines[:3]] == ["record 0", "record 1", "record 2"]+ assert "'handler_error': 3" in lines[3]+ assert "'prepare_error': 1" in lines[3]+ assert "'reentrant': 3" in lines[3]+ assert capsys.readouterr().err == ""+ assert broken._closed and healthy._closed+++def test_reentrant_signal_logging_cannot_deadlock_queue_mutex(monkeypatch):+ output = io.StringIO()+ install_stream(monkeypatch, output)+ records = service_logging._runtime.handler.queue+ original_put = records._put++ def interrupted_put(record):+ # Exactly the state of a Python signal handler interrupting Queue.put:+ # Queue's non-reentrant mutex is held by the producer thread.+ logging.getLogger("signal-test").info("nested signal record")+ original_put(record)++ monkeypatch.setattr(records, "_put", interrupted_put)+ produced = threading.Event()+ producer = threading.Thread(target=lambda: (logging.info("outer record"), produced.set()), daemon=True)+ producer.start()+ assert produced.wait(2)+ producer.join(2)+ service_logging.shutdown_service_logging()+ assert "outer record" in output.getvalue()+ assert "nested signal record" not in output.getvalue()+ assert "'reentrant': 1" in output.getvalue()+++def test_repeated_setup_reload_and_shutdown_preserve_format_levels_and_ownership(monkeypatch, tmp_path, capsys):+ monkeypatch.setattr(main, "get_logs_dir", lambda: tmp_path)+ monkeypatch.setenv("VIBE_DISABLE_STDOUT_LOGGING", "1")+ # Early records must not prevent setup from taking ownership of root.+ logging.warning("before service setup")+ root = logging.getLogger()+ original_handlers = root.handlers[:]+ original_level = root.level+ threads, handlers = [], []+ for index in range(3):+ if index == 1:+ importlib.reload(main)+ monkeypatch.setattr(main, "get_logs_dir", lambda: tmp_path)+ main.setup_logging("WARNING")+ threads.append(service_logging._runtime.listener._thread)+ handlers.extend(service_logging._runtime.listener.handlers)+ assert len(root.handlers) == 1+ assert root.handlers[0] is service_logging._runtime.handler+ logging.getLogger("format-test").info("filtered record")+ logging.getLogger("format-test").warning("visible record %s", index)+ try:+ raise ValueError("traceback preserved")+ except ValueError:+ logging.getLogger("format-test").exception("final exception")+ service_logging.shutdown_service_logging()+ service_logging.shutdown_service_logging()+ assert all(not thread.is_alive() for thread in threads)+ assert all(handler.stream is None and handler._closed for handler in handlers)+ assert root.handlers == original_handlers and root.level == original_level+ text = (tmp_path / "vibe_remote.log").read_text()+ assert text.count("visible record") == 3+ assert "filtered record" not in text+ assert " - format-test - WARNING - [test_service_logging.py:" in text+ assert "test_repeated_setup_reload_and_shutdown_preserve_format_levels_and_ownership()" in text+ assert text.count("Traceback (most recent call last):") == 1+ assert "ValueError: traceback preserved" in text+ assert capsys.readouterr().out == ""++[email protected]("disabled", [False, True])+def test_listener_keeps_stdout_policy_and_file_rotation(monkeypatch, tmp_path, capsys, disabled):+ monkeypatch.setattr(main, "get_logs_dir", lambda: tmp_path)+ monkeypatch.setattr(main, "APPLICATION_LOG_MAX_BYTES", 256)+ monkeypatch.setattr(main, "APPLICATION_LOG_BACKUP_COUNT", 2)+ if disabled:+ monkeypatch.setenv("VIBE_DISABLE_STDOUT_LOGGING", "1")+ else:+ monkeypatch.delenv("VIBE_DISABLE_STDOUT_LOGGING", raising=False)+ main.setup_logging()+ for index in range(10):+ logging.getLogger("rotation-test").info("rotation-record-%s %s", index, "x" * 80)+ service_logging.shutdown_service_logging()+ files = sorted(tmp_path.glob("vibe_remote.log*"))+ assert [path.name for path in files] == ["vibe_remote.log", "vibe_remote.log.1", "vibe_remote.log.2"]+ assert "rotation-record-9" in files[0].read_text()+ assert "rotation-record-8" in files[1].read_text()+ assert "rotation-record-7" in files[2].read_text()+ stdout = capsys.readouterr().out+ assert stdout.count("rotation-record-") == (0 if disabled else 10)+++def test_loss_reports_are_rate_limited_and_flush_final_totals(monkeypatch):+ from types import SimpleNamespace++ class BrokenStream(io.StringIO):+ def write(self, value):+ raise OSError("destination unavailable")++ now = [0.0]+ monkeypatch.setattr(service_logging, "time", SimpleNamespace(monotonic=lambda: now[0]))+ output = io.StringIO()+ handlers = [main.ServiceStreamHandler(BrokenStream()), main.ServiceStreamHandler(output)]+ monkeypatch.setattr(main, "_build_logging_handlers", lambda _: handlers)+ main.setup_logging()+ records = service_logging._runtime.handler.queue+ logger = logging.getLogger("rate-test")+ for index in range(20):+ logger.info("initial %s", index)+ records.join()+ assert "Service logging losses" not in output.getvalue()+ now[0] = 31.0+ for index in range(20):+ logger.info("burst %s", index)+ records.join()+ assert output.getvalue().count("Service logging losses") == 1+ now[0] = 62.0+ logger.info("after interval")+ records.join()+ assert output.getvalue().count("Service logging losses") == 2+ service_logging.shutdown_service_logging()+ # Two earlier summary writes also failed at the broken destination.+ assert output.getvalue().count("Service logging losses") == 3+ assert "'handler_error': 43" in output.getvalue().splitlines()[-1]+++def break_file_destination(monkeypatch, handler, failure):+ def fail(*args):+ raise OSError("destination unavailable")++ if failure == "write":+ monkeypatch.setattr(handler.stream, "write", fail)+ else:+ monkeypatch.setattr(handler, "shouldRollover", lambda record: True)+ monkeypatch.setattr(handler, "doRollover", fail)++[email protected]("file_failure", ["write", "rotation"])[email protected]("stderr_failure", [None, "write", "flush", "missing"])+def test_file_only_failure_has_bounded_listener_stderr_fallback(monkeypatch, tmp_path, file_failure, stderr_failure):+ from types import SimpleNamespace++ class Stderr(io.StringIO):+ def __init__(self):+ super().__init__()+ self.attempts = []++ def write(self, value):+ self.attempts.append((threading.get_ident(), value))+ if stderr_failure == "write":+ raise OSError("stderr unavailable")+ return super().write(value)++ def flush(self):+ if stderr_failure == "flush":+ raise OSError("stderr flush unavailable")+ return super().flush()++ now = [0.0]+ monkeypatch.setattr(service_logging, "time", SimpleNamespace(monotonic=lambda: now[0]))+ stderr = Stderr()+ monkeypatch.setattr(sys, "__stderr__", None if stderr_failure == "missing" else stderr)+ monkeypatch.setenv("VIBE_DISABLE_STDOUT_LOGGING", "1")+ monkeypatch.setattr(main, "get_logs_dir", lambda: tmp_path)+ main.setup_logging()+ runtime = service_logging._runtime+ assert len(runtime.listener.handlers) == 1+ file_handler = runtime.listener.handlers[0]+ listener = runtime.listener._thread+ break_file_destination(monkeypatch, file_handler, file_failure)+ logger = logging.getLogger("file-failure")+ for _ in range(20):+ logger.info("private_payload_never_in_stderr")+ runtime.handler.queue.join()+ assert stderr.attempts == []+ now[0] = 31.0+ for _ in range(20):+ logger.info("private_payload_never_in_stderr")+ runtime.handler.queue.join()+ assert len(stderr.attempts) == (0 if stderr_failure == "missing" else 1)+ now[0] = 62.0+ logger.info("private_payload_never_in_stderr")+ runtime.handler.queue.join()+ assert len(stderr.attempts) == (0 if stderr_failure == "missing" else 2)+ service_logging.shutdown_service_logging()+ assert not listener.is_alive()+ assert file_handler._closed and file_handler.stream is None+ assert len(stderr.attempts) == (0 if stderr_failure == "missing" else 3)+ if stderr_failure != "missing":+ for thread_id, text in stderr.attempts:+ assert thread_id == listener.ident+ assert len(text.encode("utf-8")) <= 1024+ assert "private_payload_never_in_stderr" not in text+ assert text.startswith("Service logging destinations unavailable; losses=")+ assert "'handler_error': 43" in stderr.attempts[-1][1]+ if stderr_failure != "write":+ assert stderr.getvalue().count("Service logging destinations unavailable") == 3+ assert (tmp_path / "vibe_remote.log").read_text() == ""+++def test_blocked_stderr_fallback_never_blocks_producer(monkeypatch, tmp_path):+ from types import SimpleNamespace++ now = [0.0]+ monkeypatch.setattr(service_logging, "time", SimpleNamespace(monotonic=lambda: now[0]))+ monkeypatch.setattr(service_logging, "QUEUE_CAPACITY", 2)+ stderr = HeldStream()+ monkeypatch.setattr(sys, "__stderr__", stderr)+ monkeypatch.setenv("VIBE_DISABLE_STDOUT_LOGGING", "1")+ monkeypatch.setattr(main, "get_logs_dir", lambda: tmp_path)+ main.setup_logging()+ runtime = service_logging._runtime+ listener = runtime.listener._thread+ break_file_destination(monkeypatch, runtime.listener.handlers[0], "write")+ now[0] = 31.0+ first_returned = threading.Event()+ continue_producing = threading.Event()+ produced = threading.Event()++ def produce():+ logger = logging.getLogger("blocked-stderr")+ logger.error("trigger fallback")+ first_returned.set()+ if not continue_producing.wait(3):+ return+ for _ in range(100):+ logger.error("during blocked stderr")+ produced.set()++ producer = threading.Thread(target=produce, daemon=True)+ producer.start()+ try:+ assert stderr.entered.wait(2)+ assert first_returned.wait(1), "initial fallback ran on producer"+ continue_producing.set()+ assert produced.wait(1), "producer waited on blocked stderr/queue space"+ assert runtime.handler.queue.qsize() == 2+ assert stderr.getvalue() == ""+ finally:+ continue_producing.set()+ stderr.release.set()+ producer.join(3)+ service_logging.shutdown_service_logging()+ assert not producer.is_alive() and not listener.is_alive()+ assert set(stderr.writer_threads) == {listener.ident}+ assert "'queue_full': 98" in stderr.getvalue()+ assert stderr.getvalue().count("Service logging destinations unavailable") == 2+++def test_partial_handler_construction_closes_open_file(monkeypatch, tmp_path):+ monkeypatch.delenv("VIBE_DISABLE_STDOUT_LOGGING", raising=False)+ file_handler = main.ServiceFileHandler(tmp_path / "vibe_remote.log")+ monkeypatch.setattr(main, "ServiceFileHandler", lambda *args, **kwargs: file_handler)++ def fail_stream(*args, **kwargs):+ raise RuntimeError("stdout handler unavailable")++ monkeypatch.setattr(main, "ServiceStreamHandler", fail_stream)+ with pytest.raises(RuntimeError, match="stdout handler unavailable"):+ main._build_logging_handlers(str(tmp_path))+ assert file_handler.stream is None and file_handler._closed+ service_logging.shutdown_service_logging()+++def test_repeated_setup_opens_new_log_only_after_previous_drain_rotation(monkeypatch, tmp_path):+ held = HeldStream()+ built_again = threading.Event()+ setup_started = threading.Event()+ setup_done = threading.Event()+ build_count = 0++ def build(_):+ nonlocal build_count+ build_count += 1+ if build_count == 2:+ built_again.set()+ return [+ main.ServiceStreamHandler(held if build_count == 1 else io.StringIO()),+ main.ServiceFileHandler(tmp_path / "vibe_remote.log", maxBytes=256, backupCount=2),+ ]++ monkeypatch.setattr(main, "_build_logging_handlers", build)+ main.setup_logging()+ logger = logging.getLogger("replace-test")+ logger.info("old record %s", "x" * 120)++ def reconfigure():+ setup_started.set()+ main.setup_logging()+ setup_done.set()++ replacing = threading.Thread(target=reconfigure, daemon=True)+ try:+ assert held.entered.wait(2)+ logger.info("old drain rotates %s", "x" * 120)+ replacing.start()+ assert setup_started.wait(2)+ assert not built_again.wait(0.05), "opened new inode before previous drain/rotation"+ finally:+ held.release.set()+ if replacing.ident is not None:+ replacing.join(3)+ assert setup_done.is_set()+ logger.info("new current file")+ service_logging.shutdown_service_logging()+ assert "new current file" in (tmp_path / "vibe_remote.log").read_text()+ assert "new current file" not in (tmp_path / "vibe_remote.log.1").read_text()++[email protected]("start_then_fail", [False, True])+def test_listener_start_failure_closes_handlers_and_allows_retry(monkeypatch, tmp_path, start_then_fail):+ monkeypatch.setenv("VIBE_DISABLE_STDOUT_LOGGING", "1")+ handler = main._build_logging_handlers(str(tmp_path))[0]+ monkeypatch.setattr(main, "_build_logging_handlers", lambda _: [handler])+ original_handlers = logging.getLogger().handlers[:]+ original_start = service_logging._Listener.start+ workers = []+ with monkeypatch.context() as patch:+ def fail_start(self):+ if start_then_fail:+ original_start(self)+ workers.append(self._thread)+ raise RuntimeError("cannot start listener")++ patch.setattr(service_logging._Listener, "start", fail_start)+ with pytest.raises(RuntimeError, match="cannot start listener"):+ main.setup_logging()+ assert handler.stream is None and handler._closed+ assert all(not thread.is_alive() for thread in workers)+ assert logging.getLogger().handlers == original_handlers+ service_logging.shutdown_service_logging()+ output = io.StringIO()+ install_stream(monkeypatch, output)+ logging.warning("retry works")+ service_logging.shutdown_service_logging()+ assert "retry works" in output.getvalue()+++def test_atexit_fallback_drains_before_logging_closes_destinations(tmp_path):+ code = """+import logging+from pathlib import Path+import sys+import main+main.get_logs_dir = lambda: Path(sys.argv[1])+main.setup_logging()+for index in range(50):+ logging.getLogger('exit-test').warning('exit record %s', index)+"""+ result = subprocess.run(+ [sys.executable, "-c", code, str(tmp_path)],+ cwd=Path(__file__).resolve().parents[1], capture_output=True, text=True, timeout=10,+ )+ assert result.returncode == 0, result.stderr+ text = (tmp_path / "vibe_remote.log").read_text()+ assert text.count("exit record") == 50+ assert "exit record 49" in text++[email protected]("release_fails", [False, True])+def test_startup_failure_drains_error_and_cleanup_even_when_cleanup_raises(monkeypatch, release_fails):+ output = io.StringIO()+ handler = main.ServiceStreamHandler(output)+ monkeypatch.setattr(main, "_build_logging_handlers", lambda _: [handler])+ monkeypatch.setattr(main, "acquire_service_instance_lock", lambda: None)+ from types import SimpleNamespace++ monkeypatch.setattr(main, "load_config", lambda: SimpleNamespace(runtime=SimpleNamespace(log_level="INFO")))++ def fail_startup():+ raise RuntimeError("startup failure after logging setup")++ def release():+ # No successor may acquire the same log path until this runtime's last+ # write/rotation and handler close have completed.+ assert service_logging._runtime is None+ assert handler._closed+ assert "final service diagnostics cleanup" in output.getvalue()+ if release_fails:+ raise RuntimeError("cleanup failure")++ monkeypatch.setattr(main, "_stop_macos_session_diagnostics", lambda _: logging.warning("final service diagnostics cleanup"))+ monkeypatch.setattr(main, "apply_claude_sdk_patches", fail_startup)+ monkeypatch.setattr(main, "release_service_instance_lock", release)+ import core.managed_skills++ monkeypatch.setattr(core.managed_skills, "prepare_builtin_skills", lambda: None)+ with pytest.raises(RuntimeError if release_fails else SystemExit):+ main.main()+ assert "Failed to start: startup failure after logging setup" in output.getvalue()+ assert "final service diagnostics cleanup" in output.getvalue()+ assert service_logging._runtime is None+ assert handler._closeddiff --git a/vibe/service_logging.py b/vibe/service_logging.pynew file mode 100644index 000000000..9788a147a--- /dev/null+++ b/vibe/service_logging.py@@ -0,0 +1,240 @@+"""Bounded, service-owned logging. See docs/plans/nonblocking-service-logging.md.++Producers prepare records and enqueue without waiting for destination I/O. Full+queues drop the newest record; accepted records are not evicted. Only the final+service-exit path may wait for drain (and thus for slow/hung destinations).+"""+from __future__ import annotations++import atexit+import logging+import sys+import threading+import time+from collections import Counter+from collections.abc import Callable+from logging.handlers import QueueHandler, QueueListener, RotatingFileHandler+from queue import Full, Queue++QUEUE_CAPACITY = 8192+REPORT_INTERVAL_SECONDS = 30.0+++class _RaiseHandlerErrors:+ def handleError(self, record: logging.LogRecord) -> None:+ # Stdlib stream/file emit() normally prints to stderr and swallows these.+ # Let our listener count them instead, without recursively logging them.+ raise+++class ServiceFileHandler(_RaiseHandlerErrors, RotatingFileHandler):+ pass+++class ServiceStreamHandler(_RaiseHandlerErrors, logging.StreamHandler):+ pass+++class _Losses:+ def __init__(self) -> None:+ self._lock = threading.RLock()+ self._counts: Counter[str] = Counter()++ def add(self, reason: str) -> None:+ with self._lock:+ self._counts[reason] += 1++ def snapshot(self) -> dict[str, int]:+ with self._lock:+ return dict(self._counts)+++class _QueueHandler(QueueHandler):+ def __init__(self, records: Queue, losses: _Losses, context: threading.local) -> None:+ super().__init__(records)+ self.losses = losses+ self.context = context+ self.accepting = True++ def emit(self, record: logging.LogRecord) -> None:+ if not self.accepting:+ return+ # A Python signal can log while this thread holds Queue's non-reentrant+ # mutex. Handler/formatter logging can recurse too. Never re-enter either.+ if getattr(self.context, "logging", False):+ self.losses.add("reentrant")+ return+ self.context.logging = True+ try:+ super().emit(record)+ finally:+ self.context.logging = False++ def handleError(self, record: logging.LogRecord) -> None:+ # QueueHandler calls this for full queues and failed record preparation.+ self.losses.add("queue_full" if isinstance(sys.exc_info()[1], Full) else "prepare_error")++ def close(self) -> None:+ # Serialize the admission boundary with emit, not with listener I/O.+ self.acquire()+ try:+ self.accepting = False+ finally:+ self.release()+ super().close()+++class _Listener(QueueListener):+ # QueueListener's stubs expose only a put_nowait queue protocol. This+ # service owns a concrete bounded Queue and a sentinel that cannot be a log.+ queue: Queue+ _sentinel = None++ def __init__(self, records: Queue, handlers: list[logging.Handler], losses: _Losses, context: threading.local) -> None:+ super().__init__(records, *handlers, respect_handler_level=True)+ self.losses = losses+ self.context = context+ self._reported: dict[str, int] = {}+ self._last_report = time.monotonic()++ def _deliver(self, record: logging.LogRecord) -> bool:+ delivered = False+ for handler in self.handlers:+ if record.levelno < handler.level:+ continue+ try:+ if handler.handle(record):+ delivered = True+ except Exception:+ self.losses.add("handler_error")+ return delivered++ def _report_losses(self, *, final: bool = False) -> None:+ now = time.monotonic()+ if not final and now - self._last_report < REPORT_INTERVAL_SECONDS:+ return+ counts = self.losses.snapshot()+ if counts == self._reported:+ return+ self._last_report = now+ self._reported = counts+ delivered = self._deliver(logging.LogRecord(+ __name__, logging.WARNING, __file__, 0,+ "Service logging losses (cumulative): %s", (counts,), None,+ ))+ if not delivered:+ # The background service has only the file sink. Do not send its+ # failure notice solely back to that same failed destination. This+ # best-effort escape path runs only on the listener, shares the+ # summary rate limit, and never contains record payloads.+ try:+ stderr = sys.__stderr__+ if stderr is not None:+ message = f"Service logging destinations unavailable; losses={counts}"+ stderr.write(message[:1023] + "\n")+ stderr.flush()+ except Exception:+ # stderr can be unavailable too. No recursion, retries, or+ # producer-side fallback; a blocked stderr only holds this worker.+ pass++ def handle(self, record: logging.LogRecord) -> None:+ self._deliver(record)+ self._report_losses()++ def enqueue_sentinel(self) -> None:+ # Called only after admission closes, at service exit, off the live loop.+ # put_nowait (the stdlib default) would fail if the bounded queue is full.+ self.queue.put(self._sentinel)++ def _monitor(self) -> None:+ # Destination callbacks must not feed records back into their own queue.+ self.context.logging = True+ try:+ # The stdlib worker hook exists on every supported Python version,+ # but is deliberately omitted from typeshed's public interface.+ super()._monitor() # type: ignore[misc]+ finally:+ self._report_losses(final=True)+ for handler in self.handlers:+ try:+ handler.close()+ except Exception:+ self.losses.add("close_error")+ self.context.logging = False+++class _ServiceLogging:+ def __init__(self, handlers: list[logging.Handler]) -> None:+ self.root = logging.getLogger()+ self.previous_handlers = self.root.handlers[:]+ self.previous_level = self.root.level+ records: Queue = Queue(maxsize=QUEUE_CAPACITY)+ losses = _Losses()+ context = threading.local()+ self.handler = _QueueHandler(records, losses, context)+ self.listener = _Listener(records, handlers, losses, context)++ def stop(self) -> None:+ self.handler.close()+ self.listener.stop()+ # Keep the closed queue handler installed throughout drain so late+ # module-level logging cannot basicConfig() a synchronous stderr handler.+ self.root.handlers = self.previous_handlers+ self.root.setLevel(self.previous_level)+++_lifecycle_lock = threading.RLock()+_runtime: _ServiceLogging | None = None+++def configure_service_logging(+ build_handlers: Callable[[], list[logging.Handler]], level: int, formatter: logging.Formatter,+) -> None:+ """Replace the previous setup, opening new files only after its final rotation."""+ global _runtime+ with _lifecycle_lock:+ handlers: list[logging.Handler] = []+ runtime = None+ try:+ if not isinstance(level, int):+ raise ValueError("Service log level must be an integer")+ shutdown_service_logging()+ handlers = build_handlers()+ for handler in handlers:+ handler.setFormatter(formatter)+ runtime = _ServiceLogging(handlers)+ runtime.listener.start()+ except BaseException:+ # Thread.start can be interrupted after the worker actually started.+ thread = getattr(runtime.listener, "_thread", None) if runtime else None+ if runtime is not None and thread is not None and thread.is_alive():+ runtime.stop()+ else:+ if runtime is not None:+ runtime.handler.close()+ for handler in handlers:+ handler.close()+ raise+ runtime.root.handlers = [runtime.handler]+ runtime.root.setLevel(level)+ _runtime = runtime+++def shutdown_service_logging() -> None:+ """Drain after service producers stop; safe before setup and on repeated calls.++ Accepted records are attempted in FIFO order on every destination. This waits+ for destination I/O, with no timeout: a permanently hung destination requires+ process termination, which forfeits the remaining queued records.+ """+ global _runtime+ with _lifecycle_lock:+ if _runtime is not None:+ _runtime.stop()+ _runtime = None+++# Fallback for callers using setup outside main; main itself drains explicitly.+# Registered after logging's hook so drain precedes logging.shutdown at exit.+atexit.register(shutdown_service_logging)