Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.
Python agentagent-osaiai-agentsai-partnerchatopsclaudeclaude-codecodexdevtoolsdiscord-botlark-botllmlocal-firstlong-term-memoryopencodeslack-botvibe-codingvibeywechat
"""``LocalJobHost``: the ``pipe`` backend of ``JobHost`` (C-7 section 7, ``recovery.md`` J1-J6).
Every command runs under ``job_wrapper.py`` in its own session: stdin closed,
stdout and stderr through a pipe into a bounded log the wrapper keeps. Nothing
in the host holds a pipe to the job, so any ``LocalJobHost`` over the same
directory, in this process or after a restart, can read, wait on, or kill it.
The host starts the wrapper as this module imported it, from a copy under
``<jobs_dir>/wrapper/`` (``core.frozen_script``): an upgrade that replaces the
installed files before the service restarts must not put a newer wrapper, whose
argv may differ, under a running host.
Job directory: ``meta.json``, ``pid``, ``decision``, ``output.log`` (the head),
``tail.log`` (the rolling tail, once the head is full), ``exit``, ``stopped``
(why the host killed it), ``wrapper.log`` (the wrapper's own diagnostics).
J1, launch handshake: the host writes ``meta.json`` and spawns the wrapper; the
wrapper writes ``pid`` and waits for ``decision``; the host records the process
identity in ``meta.json`` (J2), then decides ``go``; the wrapper runs the
command only on ``go``, and decides ``abandon`` itself if no decision comes in
time. A decision is created exclusively, through ``link(2)`` of a written
temporary file (on Windows a rename, which never replaces there), so exactly
one side decides and nobody reads it half-written.
Recovery decides ``abandon`` for a job that has none; a missing or late
``pid`` proves nothing.
J2, identity: the wrapper's argv names the job directory, a random path no
recycled pid carries, and argv is readable on every POSIX platform. The
inherited ``AVIBE_PROCESS_IDENTITY`` marker is set and fingerprinted as well,
but macOS hides the environment of its own binaries (``/bin/bash``,
``/bin/sleep``), so the marker only verifies a group whose wrapper has exited.
J3: ``meta.json.deadline_at`` is absolute. The wrapper enforces it itself, so
it holds while no host runs; ``wait`` and ``enforce_deadline`` are a second,
idempotent owner. Whoever kills records ``stopped`` first.
Windows: the tree is the wrapper's job object, ``Local\\avibe-job-<job_id>``,
which the wrapper creates and joins before it writes ``pid`` and which ends its
members when the wrapper's handle closes (``windows_job``). J2 then asks for a
wrapper that is a member of that job instead of a group leader, and a kill
terminates the job. Nothing outlives the wrapper, so a job whose wrapper is
gone is gone. Job files are replaced with POSIX semantics and read shared for
delete (``windows_files``): a reader of ``tail.log``, which the wrapper
replaces a few times a second, would otherwise make the replace fail.
"""
from __future__ import annotations
import asyncio
import contextlib
import io
import json
import logging
import os
import re
import secrets
import shutil
import signal
import subprocess
import sys
import threading
import time
from datetime import datetime, timedelta, timezone
from typing import Any, Awaitable, Callable, Mapping, Optional
import psutil
from core.agent_core.tools import windows_files, windows_job
from core.agent_core.tools.base import CallInstance, JobStatus
from core.agent_core.tools.paths import os_reason, run_to_end
from core.agent_core.tools.shell import Shell, default_shell
from core.frozen_script import FrozenScript
from core.process_isolation import (
DEFAULT_PROCESS_TERMINATE_TIMEOUT_SECONDS,
KILL_SIGNAL,
PROCESS_IDENTITY_ENV,
PersistedProcessIdentity,
capture_spawned_process_identity,
new_process_identity_marker,
process_group_identity_status,
process_identity_from_payload,
serialize_process_identity,
)
logger = logging.getLogger(__name__)
#: Adopt-or-create the once Watch (target kind ``job``) for the job's ``meta.json``; returns the Watch id.
#: Keyed by ``job_id``, so calling it again after a crash returns the same Watch (J6). The adapter promises
#: that raising means no Watch owns the job: adopt-or-create is atomic. Once it returns, the Watch owns it.
HandOver = Callable[[Mapping[str, Any]], Awaitable[str]]
#: Whether the tool call that owns a job has a durable ``tool_result`` and its Watch, if any, has settled (J5).
Settled = Callable[[Mapping[str, Any]], bool]
#: How long the wrapper waits for the host's decision before abandoning the command.
DECISION_TIMEOUT_S = 30.0
#: The host gives up on the wrapper's ``pid`` well before the wrapper gives up on a decision.
PID_TIMEOUT_S = 10.0
#: Output kept on disk per job (J4): this much from the start and this much from the end.
OUTPUT_HEAD_BYTES = 1024 * 1024
OUTPUT_TAIL_BYTES = 2 * 1024 * 1024
OUTPUT_CHUNK_BYTES = 1024 * 1024
FINISHED_JOB_RETENTION_S = 7 * 24 * 3600
STOP_TIMEOUT = "timeout"
STOP_ABORTED = "aborted"
#: A stop by anyone else, the wrapper's catchable signals included; ``kill``'s default.
STOP_KILLED = "killed"
#: The wrapper itself failed (a full disk, a bug) and stopped the command it could no longer follow.
STOP_WRAPPER_ERROR = "wrapper_error"
_WINDOWS = os.name == "nt"
#: The wrapper, with the helpers it loads beside itself on Windows, as this module was imported.
_WRAPPER = FrozenScript(
*(
os.path.join(os.path.dirname(os.path.abspath(__file__)), name)
for name in ("job_wrapper.py", "windows_job.py", "windows_files.py")
)
)
#: How long past the deadline the host leaves a live wrapper to decide (its poll interval, its
#: 0.5 s drain, and margin) before stopping the job itself.
_WRAPPER_DECIDES_S = 1.0
_JOB_ID = re.compile(r"^job_[a-z0-9]+$")
# tail.log starts with this line (written by the wrapper), so a reader of the file sees where it begins.
_TAIL_HEADER = re.compile(rb"\[output from byte (\d+)\]\n")
_GO = "go"
_ABANDON = "abandon"
_STATE_FILE_CHARS = 4096
_SURROGATE = re.compile("[\ud800-\udfff]")
# The host's own meta.json holds the command, which a shell cannot even take past ARG_MAX (1 MiB here).
_META_CHARS = 4 * 1024 * 1024
class JobStartError(RuntimeError):
"""The command was not started; nothing ran."""
class JobHandOverUnavailable(RuntimeError):
"""This host has no Watch to hand a job to."""
def _job_object_name(job_id: str) -> str:
"""The Windows job object of a job's tree; the wrapper creates it under the same name."""
return f"avibe-job-{job_id}"
def _open_for_reading(path: str, *, binary: bool = False) -> Any:
"""A job file, open for reading. On Windows it is shared for delete, so a writer can still replace it."""
if not _WINDOWS:
return open(path, "rb") if binary else open(path, encoding="utf-8", errors="replace")
handle = os.fdopen(windows_files.open_for_reading(path), "rb")
return handle if binary else io.TextIOWrapper(handle, encoding="utf-8", errors="replace")
def _iso(moment: datetime) -> str:
return moment.isoformat(timespec="milliseconds").replace("+00:00", "Z")
def _ceil_ms(moment: datetime) -> datetime:
"""Round up to the millisecond ``_iso`` keeps, so a recorded deadline is never early."""
rest = moment.microsecond % 1000
return moment + timedelta(microseconds=1000 - rest) if rest else moment
def _parse_iso(value: str) -> datetime:
return datetime.fromisoformat(value.replace("Z", "+00:00"))
def _write_atomic(path: str, data: str) -> None:
tmp = f"{path}.{secrets.token_hex(4)}.tmp"
with open(tmp, "w", encoding="utf-8") as handle:
handle.write(data)
(windows_files.replace if _WINDOWS else os.replace)(tmp, path)
def _meta_text(meta: Mapping[str, Any]) -> str:
"""``meta.json`` as written: UTF-8 as it is (not escaped), and never past the bound ``meta`` reads in.
A POSIX path can hold bytes that are not UTF-8, which Python keeps as surrogate escapes; UTF-8 cannot
carry those, so such a record is written ASCII-escaped instead, which JSON reads back exactly.
"""
text = json.dumps(meta, indent=2, ensure_ascii=False)
if _SURROGATE.search(text):
text = json.dumps(meta, indent=2)
if len(text) > _META_CHARS:
raise ValueError(f"the job record would be {len(text)} characters, over {_META_CHARS}")
return text
def _read_text(path: str) -> Optional[str]:
"""A small state file (``pid``, ``decision``, ``exit``, ``stopped``), read in a bound: the job's
directory is one its command can write to."""
try:
with _open_for_reading(path) as handle:
return handle.read(_STATE_FILE_CHARS).strip()
except FileNotFoundError:
return None
def call_instance(meta: Mapping[str, Any]) -> Optional[CallInstance]:
"""The call instance that started a job, from its ``meta.json``; ``None`` if the record names none."""
session_id, response_id = meta.get("session_id"), meta.get("response_id")
response_seq, tool_call_id = meta.get("response_seq"), meta.get("tool_call_id")
if not all(isinstance(value, str) for value in (session_id, response_id, tool_call_id)):
return None
if not isinstance(response_seq, int) or isinstance(response_seq, bool):
return None
return CallInstance(session_id, response_id, response_seq, tool_call_id)
class LocalJobHost:
"""Job handles under ``jobs_dir`` (``<state>/agent_core/jobs``).
``hand_over`` delegates to ``on_hand_over``, which the adapter backs with
Watch; without it, ``hand_over`` raises ``JobHandOverUnavailable``.
"""
def __init__(
self,
jobs_dir: str,
*,
on_hand_over: Optional[HandOver] = None,
shell: Optional[Shell] = None,
) -> None:
# Canonical, so every host names a job's directory (and its wrapper's argv) the same way, whatever
# alias it was given (``~/.vibe_remote`` -> ``~/.avibe``, ``/tmp`` -> ``/private/tmp``).
self._jobs_dir = os.path.realpath(jobs_dir)
self._on_hand_over = on_hand_over
self._shell = shell or default_shell()
# Wrappers this host spawned. Each has a reaper thread, so a finished wrapper never lingers
# as a zombie (which would keep its process group alive), and an unreaped pid cannot be recycled.
self._children: dict[str, subprocess.Popen] = {}
@property
def shell(self) -> Shell:
"""The shell every command of this host runs in."""
return self._shell
# --- paths and metadata ---------------------------------------------
def job_dir(self, job_id: str) -> str:
if not _JOB_ID.match(job_id):
raise KeyError(job_id)
return os.path.join(self._jobs_dir, job_id)
def _path(self, job_id: str, name: str) -> str:
return os.path.join(self.job_dir(job_id), name)
def output_path(self, job_id: str) -> str:
return self._path(job_id, "output.log")
def meta(self, job_id: str) -> dict[str, Any]:
"""The job's metadata. Read in a bound: the job's directory is one its command can write to, and a
file over the bound is refused as corrupt (``ValueError``), as unreadable JSON is."""
try:
with _open_for_reading(self._path(job_id, "meta.json")) as handle:
text = handle.read(_META_CHARS + 1)
except FileNotFoundError:
raise KeyError(job_id) from None
if len(text) > _META_CHARS:
raise ValueError(f"meta.json of job {job_id} is over {_META_CHARS} characters")
return json.loads(text)
def _write_meta(self, job_id: str, meta: Mapping[str, Any]) -> None:
_write_atomic(self._path(job_id, "meta.json"), _meta_text(meta))
def find_job(self, call: CallInstance) -> Optional[str]:
"""The job a call instance started, for settling the call at resume; it starts at most one.
Matched on the whole instance, never the call id alone: a provider may reuse an id in a later response.
"""
for name in self._job_ids():
try:
meta = self.meta(name)
except (KeyError, ValueError, OSError):
continue
if call_instance(meta) == call:
return name
return None
def _job_ids(self) -> list[str]:
try:
return sorted(name for name in os.listdir(self._jobs_dir) if _JOB_ID.match(name))
except FileNotFoundError:
return []
# --- decision (J1) ----------------------------------------------------
def _create_once(self, job_id: str, name: str, value: str) -> str:
"""Create ``name`` holding ``value`` unless it exists; return what it holds, ours or the earlier writer's.
``link(2)`` of a written temporary file: exclusive, and never visible half-written. On Windows a rename,
which never replaces there, and works on volumes without hard links (FAT).
"""
target = self._path(job_id, name)
tmp = f"{target}.{value}.{secrets.token_hex(4)}.tmp"
with open(tmp, "w", encoding="utf-8") as handle:
handle.write(f"{value}\n")
try:
(os.rename if os.name == "nt" else os.link)(tmp, target)
except FileExistsError:
return _read_text(target) or value
finally:
with contextlib.suppress(OSError):
os.unlink(tmp) # a leftover temp is harmless; it goes with the job directory
# Linked: ours is durable, so nothing that fails after this may say otherwise.
return value
def _create_decision(self, job_id: str, value: str) -> str:
"""Try to decide; return the decision that holds, ours or the wrapper's."""
decision = self._create_once(job_id, "decision", value)
return decision if decision in (_GO, _ABANDON) else _ABANDON
def _decision(self, job_id: str) -> str:
"""The job's decision; a job without one is abandoned now, as recovery requires."""
current = _read_text(self._path(job_id, "decision"))
if current is not None:
return current
self.meta(job_id) # an unknown job raises KeyError
return self._create_decision(job_id, _ABANDON)
def started(self, job_id: str) -> bool:
"""Whether the command may have run. ``False`` is final: it never ran and never will."""
return self._decision(job_id) == _GO
# --- JobHost ------------------------------------------------------------
async def start(
self,
command: str,
*,
cwd: str,
env: Mapping[str, str],
timeout_s: Optional[float],
call: CallInstance,
) -> str:
"""Start ``command`` for ``call`` with ``env`` as its whole environment; return once it may run.
Raises ``JobStartError`` when the command did not start.
"""
if not sys.executable:
raise JobStartError("No Python interpreter is available to run the command.")
# The host's own steps are small and synchronous, so the only await is the wait for the pid: a cancel
# can land nowhere else, and never between publishing ``go`` and returning the job id.
job_id, proc, marker, meta = self._spawn(command, cwd, env, timeout_s, call)
try:
identity = await self._await_pid(job_id, proc, marker)
except asyncio.CancelledError:
# Undecided: abandon the wrapper now rather than leave it waiting 30 s for a decision.
self._create_decision(job_id, _ABANDON)
raise
decision = self._record_and_decide(job_id, meta, identity)
if decision != _GO:
raise JobStartError("The command did not start.")
return job_id
def _spawn(
self,
command: str,
cwd: str,
env: Mapping[str, str],
timeout_s: Optional[float],
call: CallInstance,
) -> tuple[str, subprocess.Popen, str, dict[str, Any]]:
"""The job directory, ``meta.json``, and the wrapper, which waits for the decision (J1)."""
job_id = f"job_{secrets.token_hex(8)}"
job_dir = self.job_dir(job_id)
created = datetime.now(timezone.utc)
deadline = _ceil_ms(created + timedelta(seconds=timeout_s)) if timeout_s is not None else None
meta: dict[str, Any] = {
"version": 1,
"job_id": job_id,
"session_id": call.session_id,
"response_id": call.response_id,
"response_seq": call.response_seq,
"tool_call_id": call.tool_call_id,
"backend": "pipe",
"command": command,
"cwd": cwd,
"timeout_s": timeout_s,
"created_at": _iso(created),
"state_dir": job_dir,
"process": None,
"terminal": None,
"watch_id": None,
"deadline_at": _iso(deadline) if deadline is not None else None,
}
try:
# Checked before anything exists: a record the host could not read back would leave the job unmanaged.
_meta_text(meta)
except ValueError:
raise JobStartError(
f"Could not start the command: its job record would be over {_META_CHARS} characters."
) from None
try:
wrapper = _WRAPPER.path(os.path.join(self._jobs_dir, "wrapper"))
except OSError as exc:
raise JobStartError(f"Could not start the command: {os_reason(exc)}.") from exc
os.makedirs(self._jobs_dir, exist_ok=True)
os.mkdir(job_dir, 0o700)
self._write_meta(job_id, meta)
marker = new_process_identity_marker()
child_env = dict(env)
child_env[PROCESS_IDENTITY_ENV] = marker
if self._shell.powershell:
# Python writes a pipe in the ANSI code page otherwise; the bootstrap makes everything else UTF-8. Only
# its standard streams: UTF-8 mode would also change the encoding of every file it opens.
child_env.setdefault("PYTHONIOENCODING", "utf-8")
argv = [
sys.executable,
"-I",
"-S",
"-B", # on Windows the wrapper loads its helpers by path: no bytecode written beside them
wrapper,
"avibe-job",
job_dir,
repr(DECISION_TIMEOUT_S),
str(OUTPUT_HEAD_BYTES),
str(OUTPUT_TAIL_BYTES),
repr(deadline.timestamp()) if deadline is not None else "-",
*self._shell.prepare(command, job_dir),
]
open(self.output_path(job_id), "ab").close()
diagnostics = os.open(
self._path(job_id, "wrapper.log"),
os.O_WRONLY | os.O_CREAT | os.O_APPEND | getattr(os, "O_BINARY", 0),
0o600,
)
# Windows: its own process group, so no console control event of the service reaches it, and no console
# window, which its shell and every console program under it share.
windows = (
{"creationflags": subprocess.CREATE_NEW_PROCESS_GROUP | subprocess.CREATE_NO_WINDOW} if _WINDOWS else {}
)
try:
proc = subprocess.Popen(
argv,
cwd=cwd,
env=child_env,
stdin=subprocess.DEVNULL,
stdout=diagnostics,
stderr=diagnostics,
start_new_session=True,
**windows,
)
except OSError as exc:
self._create_decision(job_id, _ABANDON)
raise JobStartError(f"Could not start the command: {os_reason(exc)}.") from exc
finally:
os.close(diagnostics)
self._children[job_id] = proc
threading.Thread(target=self._reap, args=(job_id, proc), name=f"avibe-{job_id}-reaper", daemon=True).start()
return job_id, proc, marker, meta
def _record_and_decide(
self, job_id: str, meta: dict[str, Any], identity: Optional[PersistedProcessIdentity]
) -> str:
"""Record the identity (J2), then decide: ``go`` only with an identity on record."""
if identity is not None:
meta["process"] = {**serialize_process_identity(identity), "pgid": identity.pid}
self._write_meta(job_id, meta)
return self._create_decision(job_id, _GO if identity is not None else _ABANDON)
async def _await_pid(self, job_id: str, proc: subprocess.Popen, marker: str) -> Optional[PersistedProcessIdentity]:
deadline = time.monotonic() + PID_TIMEOUT_S
pid_path = self._path(job_id, "pid")
while time.monotonic() < deadline:
text = _read_text(pid_path)
if text:
heads = _heads_its_tree(job_id, proc.pid)
if heads is None:
return None
if text != str(proc.pid) or not heads:
logger.error("Job %s wrapper reported pid %s, expected %s", job_id, text, proc.pid)
return None
return capture_spawned_process_identity(proc.pid, marker)
if proc.poll() is not None:
return None
await asyncio.sleep(0.002)
return None
def status(self, job_id: str) -> JobStatus:
exit_code = self._exit_code(job_id)
if exit_code is not None:
return JobStatus("exited", exit_code)
if self._decision(job_id) != _GO:
return JobStatus("gone")
if self._alive(job_id) or self._orphaned_group_runs(job_id):
return JobStatus("running")
# The wrapper writes `exit` before it exits, so look once more.
exit_code = self._exit_code(job_id)
return JobStatus("exited", exit_code) if exit_code is not None else JobStatus("gone")
async def wait(self, job_id: str, *, deadline_s: Optional[float]) -> JobStatus:
"""Wait until exit, or at most ``deadline_s`` seconds from now; enforce the job's own deadline meanwhile."""
end = None if deadline_s is None else time.monotonic() + max(0.0, deadline_s)
delay = 0.005
while True:
if await self.enforce_deadline(job_id):
return self.status(job_id)
current = self.status(job_id)
if current.state != "running":
return current
if end is not None:
remaining = end - time.monotonic()
if remaining <= 0:
return current
delay = min(delay, remaining)
await asyncio.sleep(delay)
delay = min(delay * 2, 0.1)
def output(self, job_id: str, since: int = 0) -> tuple[bytes, int]:
"""At most ``OUTPUT_CHUNK_BYTES`` of raw output after offset ``since``; call again until empty.
Offsets count the command's whole output. Past the kept head, a reader
that falls behind the kept tail jumps forward: the new offset then
exceeds ``since + len(data)`` by the bytes dropped from disk (J4).
"""
head_path = self.output_path(job_id)
try:
head_size = os.path.getsize(head_path)
except FileNotFoundError:
self.meta(job_id)
return b"", since
if since < head_size:
with _open_for_reading(head_path, binary=True) as handle:
handle.seek(since)
data = handle.read(min(OUTPUT_CHUNK_BYTES, head_size - since))
return data, since + len(data)
try:
with _open_for_reading(self._path(job_id, "tail.log"), binary=True) as handle:
header = _TAIL_HEADER.fullmatch(handle.readline(64))
if header is None:
raise ValueError("tail.log header")
tail_start = int(header.group(1))
start = max(since, tail_start)
handle.seek(start - tail_start, os.SEEK_CUR)
data = handle.read(OUTPUT_CHUNK_BYTES)
except (FileNotFoundError, ValueError):
return b"", since
return data, start + len(data)
async def kill(self, job_id: str, *, reason: str = STOP_KILLED) -> None:
"""Terminate the job's process tree, background children included, if it is provably the job's.
``reason`` is recorded first (``stop_reason``), so whoever reports the
job later can say why it ended.
"""
# The sequence runs to its end once begun: verification records ``stopped``, so stopping short of
# the signals, or of the SIGKILL fallback, would leave a running group. A cancel is re-raised after.
await run_to_end(self._kill(job_id, reason))
async def _kill(self, job_id: str, reason: str) -> None:
identity = self._verify_for_kill(job_id, reason)
if identity is None:
return
if _WINDOWS:
if not await _terminate_job(job_id, identity.pid):
logger.warning("Job %s job object survived termination", job_id)
elif not await _terminate_group(identity.pid):
logger.warning("Job %s process group %s survived termination", job_id, identity.pid)
def _verify_for_kill(self, job_id: str, reason: str) -> Optional[PersistedProcessIdentity]:
"""The identity to signal, once the group is provably the job's, with ``reason`` recorded; else ``None``."""
if self._decision(job_id) != _GO:
return None
identity = self._identity(job_id)
if identity is None:
logger.warning("Job %s has no recorded process identity; not signaling", job_id)
return None
if not (self._alive(job_id) or self._group_carries_marker(identity)):
if not _WINDOWS and _group_exists(identity.pid):
logger.warning("Job %s process group %s cannot be verified; not signaling", job_id, identity.pid)
return None
try:
self._create_once(job_id, "stopped", reason) # the first stopper's reason wins
except OSError:
# Recording is best effort at every stopper: the kill matters more than the record.
logger.warning("Could not record why job %s was stopped", job_id, exc_info=True)
return identity
async def hand_over(self, job_id: str) -> str:
"""Give the job to its Watch: adopt-or-create through ``on_hand_over``, then record the id (J6).
Raising means no Watch owns the job: it never raises after ``on_hand_over`` returned.
"""
meta = self.meta(job_id)
if meta.get("watch_id"):
return meta["watch_id"]
if self._on_hand_over is None:
raise JobHandOverUnavailable("No Watch is available to take over the command.")
watch_id = await self._on_hand_over(meta)
# The Watch owns the job from here on, so nothing below may undo the handover: the recorded id
# is a cache, and adopt-or-create finds the Watch by job id without it.
try:
self._record_watch(job_id, watch_id)
except (OSError, KeyError, ValueError):
logger.warning(
"Job %s is Watch %s, but its meta.json could not record that", job_id, watch_id, exc_info=True
)
return watch_id
def _record_watch(self, job_id: str, watch_id: str) -> None:
meta = self.meta(job_id)
meta["watch_id"] = watch_id
self._write_meta(job_id, meta)
# --- deadline (J3) and reporting --------------------------------------
async def enforce_deadline(self, job_id: str) -> bool:
"""Kill the job if its ``deadline_at`` has passed while it runs; ``True`` if it was killed.
Only the wrapper sees the shell exit, and it publishes the exit after draining the output, so a
running job may already have ended on time: a live wrapper decides timeout versus exit. The
host stops the job as the second owner once the wrapper has not, ``_WRAPPER_DECIDES_S`` late.
"""
if not self._overdue(job_id):
return False
await self.kill(job_id, reason=STOP_TIMEOUT)
return True
def _overdue(self, job_id: str) -> bool:
deadline_at = self.meta(job_id).get("deadline_at")
if not deadline_at:
return False
late = (datetime.now(timezone.utc) - _parse_iso(deadline_at)).total_seconds()
return late >= _WRAPPER_DECIDES_S and self.status(job_id).state == "running"
def stop_reason(self, job_id: str) -> Optional[str]:
"""Why the host killed the job (``"timeout"``, or the caller's reason), if it did."""
return _read_text(self._path(job_id, "stopped"))
# --- maintenance (J5) -------------------------------------------------
def prune(self, settled: Settled, *, older_than_s: float = FINISHED_JOB_RETENTION_S) -> list[str]:
"""Remove jobs that are finished, settled (``settled(meta)``), and untouched for ``older_than_s``."""
removed: list[str] = []
cutoff = time.time() - older_than_s
for name in self._job_ids():
try:
job_dir = self.job_dir(name)
if max(entry.stat().st_mtime for entry in os.scandir(job_dir)) > cutoff:
continue
# An exited job's wrapper may still drain children that hold the pipe; it needs the directory.
if self.status(name).state == "running" or self._alive(name) or not settled(self.meta(name)):
continue
except (KeyError, ValueError, OSError):
continue
shutil.rmtree(job_dir, ignore_errors=True)
removed.append(name)
return removed
# --- internals --------------------------------------------------------
def _exit_code(self, job_id: str) -> Optional[int]:
text = _read_text(self._path(job_id, "exit"))
if text is None:
return None
try:
return int(text)
except ValueError:
return None
def _identity(self, job_id: str) -> Optional[PersistedProcessIdentity]:
process = self.meta(job_id).get("process")
if not isinstance(process, dict) or process.get("pgid") != process.get("pid"):
return None
return process_identity_from_payload(
{key: process.get(key) for key in ("pid", "create_time", "worker_fingerprint")}, process.get("pid")
)
def _alive(self, job_id: str) -> bool:
"""Whether the job's wrapper still runs: our unreaped child, or the process whose argv names this job."""
child = self._children.get(job_id)
if child is not None:
return child.returncode is None
identity = self._identity(job_id)
if identity is None:
return False
if _WINDOWS and not windows_job.process_running(identity.pid):
return False # an ended process stays readable for a moment, and its object for as long as it is held
try:
argv = psutil.Process(identity.pid).cmdline()
except (psutil.Error, OSError):
# Gone, a zombie, or unreadable: none of them proves the job still runs.
return False
job_dir = self.job_dir(job_id)
return any(argv[i : i + 2] == ["avibe-job", job_dir] for i in range(len(argv) - 1))
def _orphaned_group_runs(self, job_id: str) -> bool:
"""The wrapper was killed on its own (SIGKILL), but its group still provably runs the job.
Never on Windows: the tree ends with the wrapper's handle to its job object.
"""
if _WINDOWS:
return False
identity = self._identity(job_id)
return identity is not None and _group_exists(identity.pid) and self._group_carries_marker(identity)
def _group_carries_marker(self, identity: PersistedProcessIdentity) -> bool:
return process_group_identity_status(identity.pid, identity, logger, "agent job") == "match"
def _reap(self, job_id: str, proc: subprocess.Popen) -> None:
"""The reaper thread: wait for the wrapper, then drop it from the registry."""
proc.wait()
if self._children.get(job_id) is proc:
del self._children[job_id]
def _heads_its_tree(job_id: str, pid: int) -> Optional[bool]:
"""Whether the wrapper ``pid`` heads the job's tree: it leads its own process group, or on Windows it is
in the job object it creates before it writes ``pid``. ``None`` when the process is already gone."""
if _WINDOWS:
job = windows_job.open_job(_job_object_name(job_id))
if job is None:
return False
try:
return windows_job.job_contains(job, pid)
finally:
windows_job.close_handle(job)
try:
return os.getpgid(pid) == pid
except ProcessLookupError:
return None
async def _terminate_job(
job_id: str, wrapper_pid: int, timeout_s: float = DEFAULT_PROCESS_TERMINATE_TIMEOUT_SECONDS
) -> bool:
"""Windows: end the verified job's tree through its job object; ``True`` once none of it runs.
The job is ended only while the wrapper is one of its members. A job that cannot be opened, or that the
wrapper has left, has lost or is losing its last handle, and with it every member (``KILL_ON_JOB_CLOSE``).
"""
job = windows_job.open_job(_job_object_name(job_id))
if job is None:
return True
try:
if not windows_job.job_contains(job, wrapper_pid):
return True
windows_job.terminate_job(job)
deadline = time.monotonic() + timeout_s
# The job counts a process out before the process has ended; ``status`` asks the wrapper itself.
while windows_job.active_processes(job) or windows_job.process_running(wrapper_pid):
if time.monotonic() >= deadline:
return False
await asyncio.sleep(0.02)
return True
finally:
windows_job.close_handle(job)
def _group_exists(pgid: int) -> bool:
try:
os.killpg(pgid, 0)
except ProcessLookupError:
return False
except PermissionError:
return True
return True
async def _terminate_group(pgid: int, timeout_s: float = DEFAULT_PROCESS_TERMINATE_TIMEOUT_SECONDS) -> bool:
"""SIGTERM the verified group, then SIGKILL what is left; ``True`` once the group is gone.
The group was verified just before. A process group id is not reused while
the group exists, so signaling it until it is gone reaches only its members.
Signals go out from the event loop at once, never queued behind the shared
worker threads; the waits between them are asynchronous.
"""
if pgid <= 1 or pgid == os.getpgrp():
logger.error("Refusing to signal process group %s", pgid)
return False
for sig in (signal.SIGTERM, KILL_SIGNAL):
try:
os.killpg(pgid, sig)
except ProcessLookupError:
return True
except PermissionError:
return False
deadline = time.monotonic() + timeout_s
while _group_exists(pgid):
if time.monotonic() >= deadline:
break
await asyncio.sleep(0.02)
else:
return True
return not _group_exists(pgid)