Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

22.6 KB
Raw Blame

App quality assessments

The dashboard app tiles, Apps list, and each app's Overview show a quality score. Open Category breakdown in Overview to see the evidence, coverage, date, confidence, worst-finding severity and originating CoS run. This includes PortOS's baseline app. Configure or run scheduled audits opens that app's Tasks tab. Nothing calls an AI provider when a page loads or when the server starts. The browser explicitly requests includeQuality=true on app reads; bare /api/apps peer probes retain their existing response without assessment prose.

The Quality tab's top status card shows active and pending check counts, each batch's current check and progress, and any reason it is waiting. Stop remaining checks is available on each batch without opening Run checks. Stopping prevents later checks from starting; work already queued or running can still finish. Status updates arrive through the existing maintenance events, including after reconnecting or returning to the tab. Loading and failed reads are shown beside a retry action rather than reported as an empty queue.

Every scheduled audit category first inventory first-party source roots, scan for their category's signals, rank the top five candidates (or all if fewer), and validate the strongest candidates before selecting a bounded investigation. The raw worst offender may be passed over only with an explanation such as a verified exemption or an existing issue. Existing unresolved issues still affect the assessment. Low churn alone does not exempt a severe hotspot. Each category has its own discovery strategy in server/lib/auditQuality.js; these instructions are injected at dispatch even when an install customized its stored prompt. File-only versus fix mode remains authoritative.

Scoring

Auditors assess the pre-fix category from 0–100, where higher is healthier:

Score Meaning
90–100 No material defect found after broad evidence
70–89 Localized moderate debt
40–69 Significant recurring or widespread problems
10–39 Severe defects in core workflows
0–9 Pervasive critical failure

Worst-finding severity is a separate 0–10 value (higher is worse; 0 means no material finding verified). An issue being filed, or one fix being made, is not an automatic score improvement. Scores express an auditor's judgment supported by the scan and inspection, not a certification or an exhaustive proof.

Each run rates only the slice it deep-reviewed (report version: 2: slice names the top-level paths or area, sliceScore is its 0–100 rating). A bounded run cannot honestly rate the whole category, and when it was asked to (version: 1, a category-wide score) the dashboard moved with whichever slice the agent picked — a 20-point swing between consecutive runs was normal. Stored v1 reports stay readable; their score is read as that run's score.

A category's score is derived server-side: the confidence-weighted mean (high 1, medium 0.5) of its newest five eligible runs — broad discovery coverage, medium/high confidence, scored, and at most 30 days old — one run per UTC day. The breakdown shows how many runs fed the figure and the worst severity any of them found; the row's coverage, confidence, date, slice and summary describe the newest run. A newer partial run does not erase a fresh scored one. With no eligible run the row shows the newest run's own score beside its stale, partial or low-confidence label and does not contribute. The overall score is the rounded, equal-weight average of the contributing category scores. Broad means the signal scan covered the entire declared eligible inventory; deep investigation still stays bounded. Partial, stale, low-confidence, unavailable and inapplicable assessments remain visible but do not contribute. Missing categories do not count as 100. The contributing category count is shown beside the score so a single good category cannot imply complete coverage. Its denominator is the number of categories that apply to the repository (applicableCategories): a category is inapplicable when the repository scan says so (see "Which checks" below — the app detail read runs it) or its own fresh assessment reported not-applicable, unless a fresh rated assessment shows it applies after all. Inapplicable categories stay listed at the bottom of the breakdown with their reason; totalCategories keeps the catalog size for existing readers. An app with no eligible assessments shows not assessed, not zero. Database failures show unavailable while app management stays usable.

The completion sentinel summary contains one QUALITY_AUDIT_JSON: {...} line with the versioned report described in the dispatch prompt. The parser rejects out-of-range scores, wrong categories, contradictory coverage counts, duplicate markers and malformed JSON. Missing/invalid output never invents a score or replaces a previous valid measurement; the prior measurement keeps its original date and expires normally. Assessment collection does not relax commit/PR gates. The run's recorded start time orders measurements so delayed recovery cannot replace newer evidence. PostgreSQL retains each run measurement and projects the recent runs per app/category into the rolling score, locally and within the ordinary database backup; no old runs are guessed into scores and no extra audit batch is enabled automatically. The summary also carries one AUDIT_SLICE: <paths or areas deep-reviewed> line, stored in the measurement's slice column (null for runs that predate it, or the v2 report's slice when the line is missing). Every audit lane preloads its last eight slices for the app through the audit-slice-history task data input ("Recent audit slices") so consecutive runs rotate to other areas instead of re-scanning the same inventory; the ux and module-hygiene prompts read that block for their route and slice choice. Peers and the .quality.json snapshot receive the numeric v1 wire shape with a v2 run's sliceScore in score, so an older install keeps reading it; the slice text never leaves the machine.

Lane health

Every audit lane also preloads what became of the issues it filed, through the lane-health task data input ("Lane health", #12088). server/lib/laneHealth.js computes, per lane (the audit's metric label; the shared code-quality label yields to the more specific lane label beside it), the issues filed in the last 60 days and how they ended: completed, not planned (or duplicate), open, fixes reverted (a later commit whose subject starts with Revert and names the closing PR or one of its SHAs), and re-filed (a new issue citing a path/to/file.ext that a closed finding of the same lane cited within 30 days). The block lists up to ten of the lane's not-planned, reverted or re-filed titles under "Findings this lane got wrong or repeated", and the discovery contract tells the lane not to re-file one of them without new evidence. services/appLaneHealth.js reads the forge (gh/glab) and the checkout's revert commits on request; nothing is cached or scheduled. The Quality page shows the same numbers for every lane in a "Lane health" table (GET /api/apps/:id/lane-health), noisiest lane first, so the operator can see which lanes to retune. A truncated issue read shortens the window and says so; a failed pull-request or commit read is named rather than read as "nothing reverted".

Coverage review

The existing catalog already separates conventional quality, cyclomatic complexity, cognitive load, structural drift, module hygiene, dead code and duplication, dependencies, typing, API contracts, runtime safety, error handling, observability, performance, test coverage, test quality and documentation. Security and data/upgrade safety cover their respective trust and persistence boundaries. Product-facing coverage comes from UX (including onboarding and successful task completion), UI bugs, console errors, accessibility, mobile/responsive behavior and copy clarity.

Those lenses grew out of a web app with a UI, so five service and data-platform lenses cover what a backend API, data platform, or infrastructure repository is judged on and nothing above owns:

Category Owns Distinct from
infrastructure IaC, containers, orchestration, platform manifests, CI: exposure, identity grants, secrets, pinning, resource limits and probes, state config, CI supply chain, environment drift security (application code), dependency-updates
data-integrity Runtime data correctness: idempotency under retry/redelivery, partial writes, lost updates, boundary validation, schema evolution of stored data, semantic corruption, pagination, retention reach, transfer checks data-safety (upgrades, migrations, destructive defaults, backups)
reliability System behavior under restart, overload and multiple instances: shutdown draining, truthful health checks, backpressure, job leasing/checkpoints, startup fragility, mixed-version deploys error-handling (per-call timeouts, retries, fallbacks)
privacy Personal/sensitive data: operational exposure, third-party minimization, API over-exposure, retention, erasure reach, real data in fixtures — under the project's documented sharing model security (authz, injection)
cost-efficiency Metered spend: repeated paid/model calls, unrequested provider work, retry multiplication, unbounded cloud scans, storage growth, over-provisioning, egress performance (latency, throughput)

All five default to file-issues. infrastructure is gated on deployment configuration being present; the others apply to any repository and report not-applicable when they do not. The same pass broadened five existing prompts for services: security (tenant isolation, SSRF, unsafe deserialization), performance (query plans, streaming, batch shape, pools, partition pruning, cold start), api-contract (framework-neutral; spec drift, breaking changes, idempotency keys, pagination bounds), observability (correlation, alertable metrics, truthful health, audit trails) and data-safety (backup restorability, bulk/backfill safety, retention reach).

upgrade-rehearsal is data-safety's executed counterpart: it builds scratch installs from migration-test fixture shapes, runs the migration runner and a backup restore against them, and boots the result, so a failure only an executed path shows (#6182, #12044, #12046, #12049) is caught without touching live data. On a host with Postgres it also round-trips the database: backup, then restorePostgres into a second throwaway portos_rehearsal_<id>_test database made and dropped by scripts/rehearsal-scratch-db.js (#12906); with no reachable Postgres it falls back to the file path and reports Postgres as uncovered.

Two gap lenses cover defects every page-walking and code-shape audit reads as intentional or never reaches:

Category Owns Distinct from
feature-completeness Work started and never finished: stubs and 501s reachable from the UI, permanently disabled or "coming soon" controls, UI/doc promises the code does not keep, settings nothing reads, capabilities with no entry point, server events nobody hears, deferred work with no live tracker item. Decides finish, hide, or remove simplify (code with no user meaning), ui-bugs (finished controls that fail)
workflow-recovery Multi-step workflows judged from the server's in-progress states: a visible resume/cancel/reattach exit after a failed follow-up step or a reload, busy refusals that carry the holder's id, waits that time out, duplicate-preventing markers kept on the server ui-bugs (one click on a fresh page), error-handling (per-call failures), reliability (process restart)

Both default to file-issues and apply to any repository. Deterministic contract breaks in untyped client code (a call to a member the imported module lacks, a test mock defining members the real module lacks, a browser confirm()) are not left to an audit: client/src/moduleContractConventions.test.js fails CI on them. Real-user product outcomes such as retention or satisfaction cannot be inferred reliably from a source audit and are not fabricated into this codebase score. A future distinct category must register both its scheduled prompt and a repository discovery strategy; the catalog parity test prevents an auditor without a search strategy.

Quality history

Each app Overview includes a daily UTC chart with 30-day, 90-day and one-year ranges and an overall/category selector. Both selections are shareable URL parameters. The expandable history table provides dates, values and coverage without relying on color or hover. Refresh history reloads measurements.

Daily snapshots apply the same rolling derivation to the runs known on that date, carrying each run forward for at most 30 days. Missing or expired evidence creates a gap. The current day ends at the request time. Category charts include provisional partial/low-confidence scores; the overall chart applies the same eligibility rules as today's score. Coverage is shown with each value because adding or losing a category can change the mean without same-category improvement. History begins with newly reported assessments; no scores are invented for old runs. The local detail endpoint /api/apps/:id/quality-history?days=90 reads only that app's measurements and bounds results to the last measurement per UTC day and category, including a 30-day lookback to seed the selected range.

Release snapshot file

A managed app can publish its recent numeric scores into .quality.json at the repository root. The file is a bounded projection, not the audit history: PostgreSQL keeps every measurement, and the file keeps one row per UTC day and category for the same 30-day window the quality panel uses, capped at 4 MiB. Publish opens an immediately-merged pull request on portos/quality-snapshot and does not commit the live checkout. With no local evidence, a non-empty file is left as it is. The immediate merge takes the install's repository merge admission (the lease a /claim run holds around its final verify-and-merge); while another owner holds it, or ownership can't be verified, the PR stays open and is handed to the pending-merge queue (GitHub) or retried on the next publish (GitLab).

The checked-in schema is v2. It stores an opaque origin fingerprint, sorted category and enum dictionaries, and fixed rows [assessedAt, categoryIndex, score, worstSeverity, coverageIndex, confidenceIndex, scannedFiles, totalFiles]. Scores stay nullable. assessedAt is a UTC date (YYYY-MM-DD); older files with full ISO instants are still read. The file does not store summaries, paths, app names, agent ids, or measurementId. Same-timestamp ordering uses a transient digest of the row, computed when the file is read.

That file is not the peer federation payload. Peers still exchange schema v1 objects. Upgrading the file does not require every peer to upgrade.

The category catalog can grow without breaking that exchange. A requester sends the categories it can parse (categories= on GET /api/apps/quality-federation), and the peer answering returns only those. A request without the parameter, from an install that predates it, gets only the 25 categories that existed before the service and data-platform lenses (FEDERATION_LEGACY_CATEGORIES), because such an install rejects a whole payload over one row it cannot parse. On receipt, rows are validated one at a time, so a malformed or unknown row costs that row, not the peer's other evidence.

Readers accept a v1 .quality.json and, when that file is absent, the historical quality-snapshot.json name. A read never rewrites the checkout. Future schemas, unrecognized documents (including TSV, CSV, or NDJSON), malformed snapshots, and oversize files are not evidence and are not overwritten. A v2 file whose category dictionary names categories this install does not know yet — written by a newer install after the catalog grew — is read for the rows it does know and is never rewritten, since canonicalizing it would drop the newer rows. (Installs predating that rule treat such a file as malformed: safe, but its scores are not shown until they upgrade.) TSV is not the canonical artifact: an append-only text log would grow without a bound and would be harder to validate, while the database is already the history.

npm run quality:snapshot writes canonical v2 from local evidence, including when the committed file is still v1. npm run quality:snapshot -- --migrate converts a v1 file or the historical filename to canonical v2 using only the rows already stored, through the same pull request. A second publish of the same normalized v2 snapshot does not open another pull request. Rows sharing one UTC day and category collapse to the latest on every read and write, so a second run the same day replaces the first and older files upgrade on their next save.

Weekly quality schedule

The app's Quality tab carries a Weekly quality schedule form that configures every applicable audit for that app in one write, instead of hand-picking a cron expression per audit on the Schedule page. It writes ordinary per-app task-type overrides, so every entry it creates stays editable there afterwards.

Which checks. Checks are pre-selected by applicability. A category an earlier audit reported as coverage: not-applicable within the last 30 days is skipped — the auditing agent read the repository, so its ruling wins until it ages out (a repository that later gains a UI gets its UI audits back). Otherwise applicability comes from the shapes present in the tracked files (git ls-files): the UI lenses (UX, accessibility, mobile/responsive, UI bugs, console errors, UI lifecycle, copy) need a user interface or a configured UI port, typing needs typed-language sources (including native Swift, TypeScript, Rust, Go, Java, Kotlin, C#, C/C++, Scala, Dart, and F#), dependency freedom needs a dependency manifest, test quality needs existing tests, and API contracts need a route/API surface, and infrastructure needs IaC, container, deployment, process-manager, or CI configuration. Test coverage is deliberately never gated on tests — a repository with none is the one it has the most to say about. Each audit declares its own requirement as requiresCapability in AUDIT_DEFINITIONS (server/lib/auditCatalog.js), beside its other metadata and under the catalog's guard test; server/services/appQualitySchedule.js owns the detection, and throws at load if a declared capability has no way to be recognized. Everything is advisory: a skipped check is still listed with its reason and can be selected by hand, and a repository that could not be fully inventoried offers every check rather than deselecting the catalog for want of evidence — a git ls-files that fails falls back to a two-level listing, which is explicitly not evidence of absence.

Dispatch-time bail-out. The same verdict (resolveAuditApplicability) is checked again before an agent is spawned for automated work, with no provider call: the scheduled lane skips the audit and records the execution so its cadence advances, a maintenance run completes the step as skipped (recorded in the run's skipped map, and named in its completion reason) and moves on, and a quota-burn step is declined so the burn picks other work. Detection failures never block: the gate only removes work it has evidence against.

An explicit choice always wins over the verdict. A manual Run of one task type is not gated. The Quality tab's Run checks batch selections leave inapplicable categories out, but picking a single category by name runs it (its step carries runInapplicableAudit). A check the user selects in the weekly schedule form despite its skip reason is saved with runInapplicableAudit: true in that app's task metadata, so the scheduled lane runs it; the flag is cleared when a later plan finds the check applies.

When. The selection is spread across the week as evenly as it allows — the 30 shipped checks become five slots a day, filled 5,5,4,4,4,4,4 so no day is left empty — ordered so each one follows the audits AUDIT_SUGGESTED_AFTER names as its predecessors, and laid out in clock order within each day so that sequence survives. Asking for more checks per day uses fewer days; asking for fewer than the week needs is raised to the floor, with a warning saying so. The hours are chosen, not entered: the planner collects the weekday/hour cells already occupied by this app's other cron-scheduled work (the CoS cadence of every type enabled for this app plus every install-wide type whether or not it is, each perpetual drain's recheck cadence, and the app's own cron overrides — where a per-app cadence REPLACES the global one, as shouldRunTask reads it), pads each by an hour before and two after, and places slots in the free cells nearest an even spacing. That is what keeps an audit out of a 03:30 nightly release window. The types the plan is about to rewrite are excluded from that collection, so re-running the form does not treat last week's plan as an obstacle to this week's. A window cannot wrap past midnight; asking for one is reported rather than silently collapsed.

Delivery and the claim drain. Each check either files issues or implements the fix (taskMetadata.fileIssues). The form defaults to each check's own catalog default — 11 audits ship defaultFileIssues: false, and a form-wide default would silently contradict every other dispatch path — with a form-wide override and a per-check override above it. When at least one check files issues, one claim job — claim-work by default, which routes to whichever tracker the app resolves to — is scheduled a few hours after each check as a single daily cron (with the shipped defaults, 0 3,9,15,21 * * *), so the issues an audit files get worked before the next audit runs. Its placement only ever moves later in the same day than the check it follows — a daily cron that wrapped past midnight would run before every audit, not after — and the slot is dropped rather than wrapped when the day has no room left. No claim drain is scheduled when every selected check implements its own fixes, and Apply retires a drain an earlier plan planted when the drain is switched off or switched to another type (recognized by the cron shape this planner emits, so a claim cadence set by hand survives).

Nothing is written until Apply, which is also the whole picture: the selected checks are enabled with their planned cron and delivery mode, and audit types left out are disabled with their stale cron cleared. Other task types are untouched. Endpoints: GET /api/apps/:id/quality-schedule, POST /api/apps/:id/quality-schedule/preview (read-only re-plan), and POST /api/apps/:id/quality-schedule/apply.