Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
Universal Input behavior baseline
baseline.json holds 25 input variants and their expected intent and fields.
It preserves multiline messages and checks the complete requested message body,
task title/body/priority/deadline, event start/invitees, note body, and search
category/sender/query. It also checks email To/Cc/subject/body separation and a
message addressed to a named channel, task search facets, and unsupported document
sender wording retained as keywords. Fields without an expectation are left
unconstrained.
generalization.json adds 35 separately authored variations covering retrieval
versus creation, questions versus record lists, opening actions with conflicting
body topics, multiword channel recipients, weekday order/abbreviations, and
supported versus unsupported search filters. This is a development regression
suite, not an untouched holdout; validate final rules on fresh phrases as well.
Follow the prompt evaluation guidance and
do not change expectations to accommodate predictions. Run the command below,
adding --suite crates/universal_input/eval/generalization.json; save a separate
before/after report because each suite has its own fixture hash. Intent-only
expectations check routing, while the search and calendar groups also check fields.
The clock is fixed at 2026-10-27 12:00 UTC. All three dinner variants expect
the upcoming Monday, November 2 at 19:00. Tomorrow morning is
October 28 at 08:00, and a task due in one week is due November 3. These
are local times in the fixture's time_zone; changing the clock/timezone creates
a different baseline.
Run against a local or test harness with an existing account's bearer token in
UNIVERSAL_INPUT_AUTH_TOKEN, or with INTERNAL_API_KEY and
UNIVERSAL_INPUT_USER_ID set in the environment:
python3 crates/universal_input/eval/compare.py \
--url https://localhost:8090/agent-harness \
--ca-file infra/local/certs/ca.pem \
--report /tmp/universal-input-before.json
Use the selected instance's URL. The runner only calls classify and extract;
it never submits drafts, sends messages, or creates content. Inference uses the
configured model providers. Fixture aliases resolve extracted names to expected
addresses without looking up or creating actual accounts. The channel fixture
represents an available channel the user belongs to; extracted names match its
full name without regard to case. This runner checks extraction, while
apps/web/src/features/universal-input/core/recipients.test.ts checks resolution
against the frontend's available recipient options, including a missing channel.
Recipient order and priority capitalization are ignored. Title expectations list
acceptable forms with or without a reporter prefix and ignore capitalization/trailing
punctuation; they still reject a description merged into the title. Message and
note bodies remain exact after trimming outside whitespace. A null search query
is equivalent to an empty one.
Search extraction returns a canonical type and only that type's fields. The
runner compares task status, priority, assignees and creators as separate facets;
channel sender filters remain separate from task creators.
The runner selects the highest-scoring intent, as the composer does. It does not force the expected type when classification is wrong. Failures therefore show the actual behavior, including missing fields caused by a wrong classification. Each case records scores, raw suggestions, per-field comparisons, and latency. Reports also record the endpoint, fixture hash, fixed clock, and runner checkout commit; the checkout commit does not identify a separately deployed backend. Credentials are never written to the report.
After changing prompts or providers, compare another run with the saved report:
python3 crates/universal_input/eval/compare.py \
--url https://localhost:8090/agent-harness \
--ca-file infra/local/certs/ca.pem \
--report /tmp/universal-input-after.json \
--compare /tmp/universal-input-before.json
The comparison lists newly failing and newly passing cases. Exit status is 1 when any expectation fails; expected mismatches should be reviewed rather than rewriting the baseline to fit the current model. This is an opt-in evaluation, not a network-dependent unit-test gate. Provider outputs can vary between runs; compare several reports before attributing a changed case to a code change.
Test the comparison logic without a backend or model credentials:
python3 -m unittest discover -s crates/universal_input/eval -p 'test_*.py'
Thread due dates
thread_due_dates.json holds 20 synthetic threads and the due date a reader would
give each: explicit dates, "by Friday", "EOD tomorrow", an older message's relative
date, "next week sometime", meeting times, past dates, dates under discussion,
quoted old mail, closed deadlines and injection attempts. Every message carries the
day it was written, so the suite needs no fixed clock.
With the model router's provider keys in the environment, run it through the real prompt, schema, model and resolver:
cargo run -p universal_input --example thread_due_date_eval
It prints each case's result and the precision and recall, and needs no running service. An answer that fails validation stores no date, so it scores as no date. Tune for precision: a wrong date is worse than a missing one. As above, do not change an expectation to accommodate a prediction, and never add real mail.