Skip to content

Guides and resources for AI engineers: an agents course, five handbooks, a hands-on second-brain project and a vetted reading list.

5.5 KB
Raw Blame

Loop practice: filter, loop, fall back

The production pattern is not "run the agent on everything". It is filter first, loop briefly, fall back. Atlan's engineering write-up of its on-call agent shows the shape at scale: a deterministic, non-LLM filter suppressed roughly 86% of about 11,000 alerts in a 30-day window before a single token was spent, so only around 14% ever reached the agent; the agent then reasoned for about three cycles, and when confidence stayed below roughly 50% it escalated to a deterministic fallback workflow in plain Python. Investigations that had taken over ten minutes and a few dollars came down to about two minutes and $0.28 each. Cheap code handles the common case; the model handles the residue; a fixed path catches whatever the model cannot close.

The goal-test half of this pattern is packaged as a skill: install the course plugin and run /agents-course:goal-test; the checker role ships as the loop-critic agent.

One frame before the code: this build is the triage shape — filter, loop briefly, fall back. The other production shape, a loop wrapped around a coding agent and driven by a goal test, is the loop handbook's build; same four parts, different body.

The build

The whole hybrid fits in one file. It runs as-is; swap model_next_step for a real API call and the control flow does not change.

"""Hybrid alert triage: deterministic filter, bounded loop, fixed fallback."""

MAX_TURNS = 3          # Atlan runs about three reasoning cycles before escalating
BUDGET_USD = 0.10      # spend ceiling per alert
COST_PER_TURN = 0.02   # replace with real usage figures from your API responses

ALERTS = [
    {"id": 1, "source": "cron", "message": "nightly backup finished OK"},
    {"id": 2, "source": "api", "message": "TimeoutError in /sync after 30s"},
    {"id": 3, "source": "api", "message": "intermittent 502 from payments"},
]

def prefilter(alert):
    """Deterministic, costs nothing. Returns a verdict, or None for the loop."""
    message = alert["message"].lower()
    if alert["source"] == "cron" or "finished ok" in message:
        return {"cause": "known-benign pattern", "next_step": "close", "via": "filter"}
    return None

def checker(diagnosis):
    """External judge: a diagnosis must name a cause and a concrete next step."""
    return bool(diagnosis.get("cause")) and bool(diagnosis.get("next_step"))

def model_next_step(alert, history):
    """Stub standing in for the model call. Swap in a real API call here."""
    if "timeout" in alert["message"].lower():
        return {"cause": "client timeout on /sync", "next_step": "raise limit to 60s"}
    return {"cause": "", "next_step": ""}   # the stub cannot close this one

def fallback_workflow(alert):
    """Fixed path, no model: always produces a defensible answer."""
    return {"cause": "unresolved by agent", "next_step": "page on-call with raw alert",
            "via": "fallback"}

def triage(alert):
    verdict = prefilter(alert)
    if verdict is not None:
        return verdict                                   # most alerts stop here
    history, spent = [], 0.0
    for turn in range(MAX_TURNS):
        diagnosis = model_next_step(alert, history)
        history.append(diagnosis)
        spent += COST_PER_TURN
        if checker(diagnosis):
            diagnosis["via"] = "loop, turn %d" % (turn + 1)
            return diagnosis                             # exit 1: checker passed
        if len(history) >= 2 and history[-1] == history[-2]:
            break                                        # exit 2: spinning
        if spent > BUDGET_USD:
            break                                        # exit 3: over budget
    return fallback_workflow(alert)                      # every other exit lands here

if __name__ == "__main__":
    for alert in ALERTS:
        print(alert["id"], triage(alert))

Alert 1 never touches the model, alert 2 closes on the loop's first turn, and alert 3 spins twice and lands in the fallback. Every path returns the same shape, so the caller never knows which layer answered.

Tips

  • Grow the filter, not the loop. Every diagnosis the loop produces twice is a candidate rule for prefilter.
  • Log via on every result. The ratio of filter to loop to fallback is your health metric; a rising fallback share means the loop is failing quietly.
  • Keep the fallback boring. Its job is a defensible answer with no model, not a second attempt at cleverness.
  • Make the checker stricter than feels polite — it is the only part the model cannot talk its way past. Hardening it against a real coding agent is the goal test build page; the full menu of brakes is stop conditions.

Prove it to yourself

  1. Run the file, then set MAX_TURNS = 0 and run it again. Every unfiltered alert should land in the fallback — if anything crashes instead, an exit path was missing.
  2. Make model_next_step return the same wrong answer forever. Confirm the spin check exits on turn two, not turn three.
  3. Add a fourth alert your filter catches, and log how many model calls the whole batch now makes. That number falling while coverage holds is the entire economics of the pattern — and deciding which decisions deserve a model at all is the next module, cheap decisions.

Go deeper → the loop engineering handbook: stop conditions, critics, context hygiene, and an overnight loop you can trust by morning.