Skip to content

Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.

5.0 KB
Raw Blame
[
  "Implement audit(rows, team=None) in usage_audit.py for clean attempt events like example.jsonl. rows may be any finite iterable of dictionaries. For this stage team is always None, each (request_id, attempt) occurs once, and data is valid. Return a dictionary containing attempts (the number of events) and usage. usage has input_tokens, output_tokens and cached_input_tokens; each maps to {known: sum of non-null values, unknown_attempts: count of null values}. Null is unknown, whereas zero is a known zero. Empty input produces zero counts, including all three usage fields. Do not mutate input rows. Save the implementation and run your own checks.",
  "Add validation and a JSONL command-line entry while retaining the summary. Each row must contain exactly these fields: request_id, attempt, revision, team, model, status, input_tokens, output_tokens, cached_input_tokens. request_id/team/model are nonempty strings (whitespace-only is invalid; preserve other strings exactly). attempt and revision are integers >=1. status is succeeded or failed. Token fields are integers >=0 or null; booleans are not integers here. When both input and cached input are known, cached must not exceed input. Reject non-dictionaries, wrong/missing/extra fields and invalid values with ValueError. python3 usage_audit.py PATH reads UTF-8 JSONL; '-' reads stdin. Ignore blank lines, require every other line to be a valid row, and print exactly one JSON result to stdout with exit 0. For any input/validation/file error use exit 2, a helpful stderr message and empty stdout. No third-party packages. All data still has unique attempt keys in this stage.",
  "Events can now be repeated, revised and delivered out of order. In audit, validate every incoming row first. Then group by (request_id, attempt) and retain the highest revision for each group; earlier revisions never add usage. Identical duplicates at the highest revision collapse to one. If two different full rows share that highest revision, raise ValueError. Conflicting rows at an obsolete lower revision do not matter when a higher valid revision exists in the same input. A malformed row is always an error even if later superseded. Team, model, status and usage may change across revisions; the winning row defines the attempt. Summarize these canonical attempts. The result must not depend on input order or replayed identical events.",
  "Add by_team and by_model lists to every audit result, sorted by the exact team/model string. A by_team entry contains team, attempts and usage; a by_model entry contains model, attempts and usage. Each usage has the same metrics as the top level. audit(rows, team='name') selects canonical attempts for that exact team and computes all output fields on that selection, including both grouping lists. None means all teams; an absent team produces the ordinary empty result. A provided team must be a nonempty, non-whitespace string, otherwise raise ValueError. Validate and reconcile the complete input before filtering, so a conflict in another team remains an error. Add --team NAME to the CLI. Retain all earlier behaviors and document a normal invocation in README.md.",
  "Extend the audit to logical requests and uncached input. A request is identified only by request_id and may contain several attempt numbers, including failed retries or a fallback to another model. Every canonical attempt of one request must have the same team, otherwise raise ValueError before any team filtering. A request succeeded if any of its canonical attempts has status succeeded; otherwise it failed. Add requests: {total, succeeded, failed} at the top level and to each by_team entry, but not to by_model entries. All attempts still contribute usage, even failed ones. Add uncached_input_tokens to every usage object: known is the sum of input_tokens minus cached_input_tokens only for attempts where both are known; unknown_attempts counts all other attempts. Do not subtract cache from output tokens or treat null cache as zero. Add examples and checks for retries, zero values, unknown fields and model fallback.",
  "Support incremental arrival using class Ledger() with extend(rows) and snapshot(team=None). extend consumes any finite iterable of events, using exactly the audit rules over all accepted events so far. It either accepts the whole batch or raises ValueError and leaves the previous state untouched; this also applies to conflicts and to a generator that yields an invalid row after valid rows. snapshot returns an independent result equal to audit over those accepted events and has the same filtering validation. Caller mutation of a row after extend, or of a previous snapshot, must not alter ledger state. Replaying accepted events is idempotent. For update sequences whose every prefix is valid, chunking or order must not change the final snapshot. A new higher revision may correct an accepted attempt; an obsolete conflicting lower revision can be ignored. Keep audit and CLI behavior. Finish README.md describing revisions, retries, unknown/cached usage and transactional updates, and verify the complete utility with your own tests."
]