Skip to content

28 eval-informed mental models and critical-thinking skills for Claude Code, GitHub Copilot, Codex, Cursor, and other Agent Skills-compatible tools

feat(evals): close coverage gaps — 6 authored objective sets + StrategyQA + run batches

Travis Boudreaux0091d63baeef
diff --git a/evals/datasets/authored/first-principles-constraint.jsonl b/evals/datasets/authored/first-principles-constraint.jsonl
new file mode 100644
index 0000000..68f7be5
--- /dev/null
+++ b/evals/datasets/authored/first-principles-constraint.jsonl
@@ -0,0 +1,30 @@
+{"id":"first-principles-01","mode":"binary-decision","prompt":"A distributed system team says they cannot have both strong consistency and 100% availability during a network partition. They cite the CAP theorem as the reason and are designing around this trade-off.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"distributed-systems"}
+{"id":"first-principles-02","mode":"binary-decision","prompt":"A backend team insists they cannot switch from REST to gRPC for internal service communication because 'we've always used REST and all our tooling is built around it.' The gRPC library is available in their language stack and the team hasn't evaluated it.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"architecture"}
+{"id":"first-principles-03","mode":"binary-decision","prompt":"A healthcare startup must store patient records exclusively within the EU because a government regulation mandates data residency for citizens' medical data within the country of origin. Legal counsel has confirmed this is a binding statutory requirement.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"compliance"}
+{"id":"first-principles-04","mode":"binary-decision","prompt":"A software team says they can't add a dark mode to their web app because 'the design system doesn't support it.' On investigation, the design system uses CSS custom properties for colors and simply hasn't been extended — no technical barrier prevents adding a second theme.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"frontend"}
+{"id":"first-principles-05","mode":"binary-decision","prompt":"An infrastructure architect explains that two datacenters on opposite sides of the planet have an irreducible minimum round-trip latency of roughly 130ms due to the speed of light through fiber. No amount of software optimization can eliminate this floor.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"networking"}
+{"id":"first-principles-06","mode":"binary-decision","prompt":"An engineering team says they can't break up their monolith into separate services because 'that's how the legacy system was built and changing it would break everything.' The codebase has clear module boundaries and no architectural reason prevents decomposition.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"architecture"}
+{"id":"first-principles-07","mode":"binary-decision","prompt":"A cryptography engineer explains that breaking a 256-bit AES key by brute force would require more computational energy than exists in the observable universe given current physics. This is why the encryption is treated as unbreakable for practical purposes.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"security"}
+{"id":"first-principles-08","mode":"binary-decision","prompt":"A team lead says they can't use snake_case for new database columns because 'our naming convention is camelCase.' This is an internal style guide documented in a wiki page that the team wrote themselves and has never been reviewed for technical necessity.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"database"}
+{"id":"first-principles-09","mode":"binary-decision","prompt":"A financial platform is told it cannot process credit card transactions without being PCI-DSS certified. This is a contractual and legal requirement from the card networks — Visa, Mastercard — that applies to any entity storing, processing, or transmitting cardholder data.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"compliance"}
+{"id":"first-principles-10","mode":"binary-decision","prompt":"A product team claims they can't add real-time collaborative editing to their document tool because 'our backend uses REST and REST can't do real-time.' WebSockets and server-sent events are viable and the team hasn't actually evaluated adding them.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"product"}
+{"id":"first-principles-11","mode":"binary-decision","prompt":"A machine learning team notes that a model trained only on data from before a certain date genuinely cannot have learned about events that occurred afterward. The knowledge cutoff is an inherent result of when training data was collected.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"machine-learning"}
+{"id":"first-principles-12","mode":"binary-decision","prompt":"A developer says the team can't write integration tests because 'we've never had them in this codebase and setting them up would take too long.' The services expose standard HTTP APIs and a test harness exists in adjacent projects the team hasn't looked at.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"testing"}
+{"id":"first-principles-13","mode":"binary-decision","prompt":"A chip design team explains they cannot make transistors smaller than a few atoms because quantum tunneling effects make the device physically non-functional at sub-atomic scales. This is a hard physical limit of semiconductor physics.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"hardware"}
+{"id":"first-principles-14","mode":"binary-decision","prompt":"An API team says they can't version their API because 'that's not how we do things here.' Looking at the codebase, there are no technical barriers — the routing layer already supports path prefixes and adding `/v2/` would require minimal changes.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"api-design"}
+{"id":"first-principles-15","mode":"binary-decision","prompt":"A data pipeline processes sensor readings and an engineer notes that you cannot have sub-millisecond latency between a sensor event and its reception on a server 500km away because the physical signal propagation time alone exceeds that bound.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"networking"}
+{"id":"first-principles-16","mode":"binary-decision","prompt":"An engineering manager says the team can't do continuous deployment because 'we've always done quarterly releases.' The CI/CD infrastructure is already in place and there are no contractual, regulatory, or technical blockers identified — just existing practice.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"devops"}
+{"id":"first-principles-17","mode":"binary-decision","prompt":"A distributed consensus algorithm requires a majority quorum of nodes to agree before committing a write. With 5 nodes, this means the system mathematically cannot tolerate more than 2 simultaneous failures and still make progress.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"distributed-systems"}
+{"id":"first-principles-18","mode":"binary-decision","prompt":"A team claims they can't use a monorepo because 'our Git workflow doesn't support it.' After digging in, the actual blocker is that their CI scripts use a hardcoded single-repo assumption — a self-imposed implementation detail that could be changed in a sprint.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"devops"}
+{"id":"first-principles-19","mode":"binary-decision","prompt":"A government contract requires source code to be audited and approved by a national security agency before deployment to production systems handling classified data. Legal counsel confirms this is a statutory obligation with no workaround for the given classification level.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"compliance"}
+{"id":"first-principles-20","mode":"binary-decision","prompt":"A frontend developer insists they can't paginate the results of a data table because 'the current API returns all rows at once.' The API team is on the same squad and has capacity to add pagination query parameters — no external dependency is involved.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"frontend"}
+{"id":"first-principles-21","mode":"binary-decision","prompt":"A compression algorithm engineer explains that you cannot losslessly compress a truly random bitstream to fewer bits than it contains — Shannon's source coding theorem sets a mathematical lower bound on entropy coding.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"algorithms"}
+{"id":"first-principles-22","mode":"binary-decision","prompt":"A platform team says they can't give engineers production database access because 'that's policy.' Investigation shows the policy was written by a former DBA who left two years ago and was never reviewed for necessity; no regulatory or audit requirement mandates it.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"devops"}
+{"id":"first-principles-23","mode":"binary-decision","prompt":"A GPU cluster running neural network training is limited by memory bandwidth — there is a physical maximum rate at which data can move between DRAM and the compute cores dictated by the memory bus architecture of the specific GPU model.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"machine-learning"}
+{"id":"first-principles-24","mode":"binary-decision","prompt":"An analytics team says they can't use columnar storage for their reporting queries because 'the warehouse is Postgres.' Their cloud provider offers a Postgres-compatible columnar extension the team hasn't evaluated and that would work with their current ORM.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"database"}
+{"id":"first-principles-25","mode":"binary-decision","prompt":"A European fintech company cannot share raw transaction data about EU residents with its US parent company's analytics team without explicit user consent or an approved legal mechanism, due to GDPR Article 44 restricting cross-border data transfers.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"compliance"}
+{"id":"first-principles-26","mode":"binary-decision","prompt":"A developer says they can't add structured logging to a service because 'logging is done with print statements in this codebase.' There is no technical barrier — the language has a standard structured logging library and the other services already use it.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"observability"}
+{"id":"first-principles-27","mode":"binary-decision","prompt":"A distributed sort algorithm must move at minimum O(N log N) comparisons' worth of data to sort N arbitrary elements — this is a provably optimal lower bound for comparison-based sorting that no comparison sort can beat.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"algorithms"}
+{"id":"first-principles-28","mode":"binary-decision","prompt":"A product team says they can't run A/B tests because 'we don't have an experimentation platform.' Their analytics stack already captures user events, their feature flag system is in place, and three engineers estimate a basic split-test harness could be built in a week.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"product"}
+{"id":"first-principles-29","mode":"binary-decision","prompt":"A real-time trading system must comply with SEC Rule 17a-4, which legally mandates that certain trade records be stored in a non-rewritable, non-erasable format for a minimum of six years. Brokers have no legal authority to waive this requirement.","label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"compliance"}
+{"id":"first-principles-30","mode":"binary-decision","prompt":"A mobile developer says they can't make their iOS app available on Android because 'we build in Swift.' Swift is iOS-specific, but the product logic has no hardware dependency and could be rewritten in a cross-platform framework — nothing physically prevents an Android port.","label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","skill_fit":["first-principles"],"domain":"mobile"}
diff --git a/evals/datasets/authored/map-territory-verify.jsonl b/evals/datasets/authored/map-territory-verify.jsonl
new file mode 100644
index 0000000..ad0e163
--- /dev/null
+++ b/evals/datasets/authored/map-territory-verify.jsonl
@@ -0,0 +1,30 @@
+{"id":"map-territory-01","mode":"binary-decision","prompt":"The README for a payment service states: \"All monetary amounts are stored as floating-point numbers.\" A developer queries the production database and finds that all amount columns are defined as NUMERIC(19,4) — a fixed-precision decimal type. The README has not been updated since the initial prototype.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"data"}
+{"id":"map-territory-02","mode":"binary-decision","prompt":"A service's API contract document specifies that the /users endpoint returns results sorted by created_at ascending. A developer calls the endpoint in production and receives results sorted by id ascending, which happens to be the same order for most records but diverges for a handful of users who were migrated with out-of-sequence IDs.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"api"}
+{"id":"map-territory-03","mode":"binary-decision","prompt":"An internal wiki page says the image processing worker runs on a single-threaded event loop and is not thread-safe. A developer inspecting the current production deployment finds it is running with GOMAXPROCS=1, uses no goroutines for the critical path, and processes one image at a time from a queue.","label":false,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"backend"}
+{"id":"map-territory-04","mode":"binary-decision","prompt":"A code comment above a caching layer reads: \"Thread-safe: uses a read-write mutex to protect concurrent access.\" During a load test at 500 concurrent requests, the service intermittently returns corrupted cached data and occasionally panics with an index-out-of-range error in the cache lookup path.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"backend"}
+{"id":"map-territory-05","mode":"binary-decision","prompt":"The Terraform configuration for a production database sets a connection timeout of 30 seconds. Engineers monitoring the service observe that database queries reliably fail with a timeout error at exactly 30 seconds under load. The application error logs confirm the timeout value in the connection string matches the Terraform config.","label":false,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"infrastructure"}
+{"id":"map-territory-06","mode":"binary-decision","prompt":"An API's OpenAPI spec marks the /v1/reports endpoint as deprecated with a sunset date of January 2024. An engineer checking production traffic logs in June 2025 finds that the endpoint is still receiving 40,000 requests per day from live customers.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"api"}
+{"id":"map-territory-07","mode":"binary-decision","prompt":"A service's runbook says the rate limit for the external enrichment API is 1,000 requests per minute. The team observes that whenever request rate exceeds 200 per minute, the external API starts returning HTTP 429 responses. Checking the external vendor's current pricing page confirms the free tier is capped at 200 req/min; the runbook was written when the team was on an enterprise plan that has since expired.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"integration"}
+{"id":"map-territory-08","mode":"binary-decision","prompt":"A distributed tracing dashboard shows that the checkout service's p99 latency is 850ms. The SLO document states the agreed p99 target for checkout is 1,000ms. Engineers reviewing the dashboard confirm the metric collection methodology matches what the SLO was written against.","label":false,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"reliability"}
+{"id":"map-territory-09","mode":"binary-decision","prompt":"A migration script's header comment reads: \"Idempotent — safe to run multiple times.\" When an engineer runs the script twice on the staging database, the second run inserts duplicate rows and throws a foreign key constraint error on the third table it processes.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"data"}
+{"id":"map-territory-10","mode":"binary-decision","prompt":"A feature flag system's documentation states that flags default to OFF for all users when first created. A developer creates a new flag and immediately checks production traffic logs, which show 0% of requests are entering the flag-gated code path. The behavior matches the documented default.","label":false,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"engineering"}
+{"id":"map-territory-11","mode":"binary-decision","prompt":"An architecture diagram shows the notification service communicating with the email provider directly over HTTPS. Examining the production network traffic, engineers find all email traffic routes through an undocumented SMTP relay host that was added two years ago as a temporary fix and never removed.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"architecture"}
+{"id":"map-territory-12","mode":"binary-decision","prompt":"A Python library's docstring states that parse_date() raises a ValueError when given an invalid date string. A developer writes a unit test that passes an invalid string and catches a ValueError. The test passes consistently, confirming the documented behavior.","label":false,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"engineering"}
+{"id":"map-territory-13","mode":"binary-decision","prompt":"A mobile app's crash reporting dashboard shows a 0.1% crash rate, which product management interprets as healthy. A developer examines the crash reporter's initialization code and discovers it only starts capturing crashes after the user reaches the home screen — meaning crashes during the onboarding flow, which affects 25% of new installs, are not captured at all.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"mobile"}
+{"id":"map-territory-14","mode":"binary-decision","prompt":"A load balancer's configuration file specifies a health check interval of 10 seconds with a failure threshold of 3. Operations engineers observe that when a backend pod becomes unhealthy, the load balancer stops routing traffic to it after approximately 30 seconds. The observed behavior is consistent with the configuration.","label":false,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"infrastructure"}
+{"id":"map-territory-15","mode":"binary-decision","prompt":"A backend function is documented as returning an empty list when no records match, per its docstring and the original design doc. In production, when a query returns no results, the API response body is null rather than an empty JSON array, causing null pointer exceptions in several client integrations.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"api"}
+{"id":"map-territory-16","mode":"binary-decision","prompt":"A security spec states that all API tokens expire after 60 minutes of inactivity. An engineer runs a test: they authenticate, wait 75 minutes without any requests, then make an authenticated request. The API returns a 401 Unauthorized, consistent with the documented policy.","label":false,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"security"}
+{"id":"map-territory-17","mode":"binary-decision","prompt":"A search service's documentation claims full-text search is case-insensitive. A developer queries the search endpoint with the term \"Widget\" and receives results for \"widget\" and \"WIDGET\" correctly. However, querying with \"wídget\" (with an accent) returns no results, even though records containing that spelling exist in the index.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"search"}
+{"id":"map-territory-18","mode":"binary-decision","prompt":"An internal design document says the job queue uses at-least-once delivery semantics, so consumers must be idempotent. Engineers examining the queue consumer logs observe that occasional duplicate messages do arrive and are processed multiple times. The consumer is already written to handle duplicates via an idempotency key table, and no data corruption is observed.","label":false,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"backend"}
+{"id":"map-territory-19","mode":"binary-decision","prompt":"A service's deployment docs say it is stateless and any instance can be killed at any time without data loss. During a chaos engineering exercise, engineers terminate a random instance and discover that in-progress file upload sessions stored in the instance's local temp directory are lost, causing partial uploads to be orphaned with no cleanup mechanism.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"reliability"}
+{"id":"map-territory-20","mode":"binary-decision","prompt":"A team's monitoring alert is configured to fire when error rate exceeds 1% over a 5-minute window, according to the alert configuration file. Engineers reviewing the alert history see that it has fired consistently when error rate hits approximately 1% over a 5-minute window, matching the configured threshold.","label":false,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"reliability"}
+{"id":"map-territory-21","mode":"binary-decision","prompt":"A changelog entry from eight months ago says the legacy v2 export format was removed. A support ticket arrives from a customer reporting that the v2 export endpoint still works for them. An engineer queries the production routing configuration and finds the v2 route was never actually removed — only the code path was flagged internally as deprecated.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"api"}
+{"id":"map-territory-22","mode":"binary-decision","prompt":"A CI pipeline's README states that all tests must pass before a PR can be merged. An engineer examines the branch protection rules in the repository settings and sees that the CI check is listed as a required status check, meaning GitHub will block merges if it fails.","label":false,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"devops"}
+{"id":"map-territory-23","mode":"binary-decision","prompt":"A database schema description in the team's Confluence doc says the orders table has a compound unique index on (customer_id, order_date) to prevent duplicate orders per day per customer. An engineer runs SHOW INDEXES on the production table and finds only a primary key index on order_id; no compound unique index exists.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"data"}
+{"id":"map-territory-24","mode":"binary-decision","prompt":"A product spec says that deleting an account is a soft delete: the record is marked deleted_at but not removed. An engineer reviews the production account deletion handler and finds it sets a deleted_at timestamp and removes the user from all active session tables, but does not hard-delete any rows from the users table. The behavior matches the spec.","label":false,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"backend"}
+{"id":"map-territory-25","mode":"binary-decision","prompt":"An SDK's documentation states the client automatically retries failed requests up to three times with exponential backoff. A developer instrumenting network calls observes that on a 503 response, the SDK immediately propagates the error to the caller with no retries attempted.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"integration"}
+{"id":"map-territory-26","mode":"binary-decision","prompt":"A team's cost allocation dashboard attributes 100% of database compute costs to the \"data-platform\" cost center, which is how the tagging policy document specifies it should be. An engineer auditing the AWS Cost Explorer report sees that all RDS instances are tagged cost-center=data-platform. The dashboard and the policy match.","label":false,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"infrastructure"}
+{"id":"map-territory-27","mode":"binary-decision","prompt":"A WebSocket server's documentation says idle connections are closed after 5 minutes. A developer runs a long-lived connection without sending any messages and the connection remains open for 47 minutes before the test is ended — it was never dropped by the server.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"backend"}
+{"id":"map-territory-28","mode":"binary-decision","prompt":"A design doc states that the recommendation engine is updated with new training data every 24 hours. A data engineer checks the ML pipeline's job history and confirms a retraining job has been completing successfully every day at 02:00 UTC, and the model artifact's timestamp in the model registry reflects daily updates.","label":false,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"ml"}
+{"id":"map-territory-29","mode":"binary-decision","prompt":"A secrets management runbook says application secrets are rotated automatically every 90 days. A security engineer auditing the vault shows that the database password for the core API service was created 14 months ago and has never been rotated; the automatic rotation policy was never applied to that secret.","label":true,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"security"}
+{"id":"map-territory-30","mode":"binary-decision","prompt":"A service contract specifies that POST /events returns HTTP 202 Accepted for all valid payloads, indicating asynchronous processing. A developer sends a well-formed event payload to the production endpoint and receives an HTTP 202 response. The behavior matches the contract.","label":false,"decision_instruction":"You are given a stated description (docs, comments, a spec, or an assumption) and an observation of actual behavior. Answer YES if the actual behavior CONTRADICTS the description — meaning you must trust the territory (real behavior) over the map (the description) and investigate the real code/data. Answer NO if the description and the observed behavior are CONSISTENT (the map matches the territory).","skill_fit":["map-territory"],"domain":"api"}
diff --git a/evals/datasets/authored/margin-of-safety-provision.jsonl b/evals/datasets/authored/margin-of-safety-provision.jsonl
new file mode 100644
index 0000000..18c1d6c
--- /dev/null
+++ b/evals/datasets/authored/margin-of-safety-provision.jsonl
@@ -0,0 +1,30 @@
+{"id":"margin-safety-01","mode":"binary-decision","prompt":"A team expects peak API traffic of 800 req/s during an upcoming product launch. Historical launches show spikes of 3–5x the predicted peak, and a failure would mean lost revenue plus SLA breach penalties. The team provisions 2,000 req/s of capacity — 2.5x the expected peak.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"infrastructure"}
+{"id":"margin-safety-02","mode":"binary-decision","prompt":"A data warehouse team expects to ingest 500GB of data per month. Storage is charged at $0.023/GB and data loss is not a concern because source data can be re-ingested at any time. They provision 550GB — a 10% buffer. Scaling additional storage takes less than 30 seconds via an API call.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"data-infrastructure"}
+{"id":"margin-safety-03","mode":"binary-decision","prompt":"A payments company processes an average of 1,200 transactions per second on normal days. Black Friday historically generates 4–6x normal volume, and a payment processing outage means direct revenue loss of roughly $50,000 per minute. The team provisions capacity for 1,800 transactions per second for the holiday period.","label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"infrastructure"}
+{"id":"margin-safety-04","mode":"binary-decision","prompt":"A team is setting an HTTP client timeout for calls to a third-party shipping API. The shipping API's documented p99 latency is 400ms, and their SLA guarantees responses within 800ms. Calls that time out result in a failed order checkout with user-visible errors. The team sets a 1,000ms timeout.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"reliability"}
+{"id":"margin-safety-05","mode":"binary-decision","prompt":"A startup's engineering team estimates a new feature will take 4 weeks to build. This is the team's first time working with the third-party payments SDK, and the feature is on the critical path for a contractually committed customer demo. They commit to a 5-week delivery date — 25% buffer.","label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"project-management"}
+{"id":"margin-safety-06","mode":"binary-decision","prompt":"A Postgres connection pool is configured with a maximum of 100 connections. Under normal load the service uses 45 connections. During monthly billing runs a secondary process uses up to 30 additional connections simultaneously. An unexpected secondary batch job will exhaust the pool only if both peaks coincide, which has never happened. The cost of pool exhaustion is a cascade of 500 errors to all users.","label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"data-infrastructure"}
+{"id":"margin-safety-07","mode":"binary-decision","prompt":"A video transcoding service processes an average of 200 jobs per hour. Job volume has been growing 15% month-over-month for six months with no sign of slowing. Each job consumes roughly the same compute. The team provisions for 600 jobs per hour — 3x current average — and plans to review in three months. Delays cause content to miss publishing windows, resulting in lost ad revenue.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"infrastructure"}
+{"id":"margin-safety-08","mode":"binary-decision","prompt":"A team is planning storage for a new audit log system. Legal requires all audit logs to be retained for 7 years, and any gap is a compliance violation. The team's data model estimates 80GB per year based on current transaction volume. They provision 100GB of storage — enough for about 15 months — with a plan to add capacity before it fills.","label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"compliance"}
+{"id":"margin-safety-09","mode":"binary-decision","prompt":"A ride-sharing company allocates a monthly infrastructure budget of $120,000. Actual spend has been stable at $95,000–$100,000 per month for the past six months with no new services planned. The allocated budget of $120,000 represents a 20–26% buffer over observed spend.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"finance"}
+{"id":"margin-safety-10","mode":"binary-decision","prompt":"A hospital patient-record API is expected to serve 500 concurrent users during the busy morning shift. Load tests show the system becomes degraded at 520 concurrent users. Patient care workflows depend on the system; downtime means nurses fall back to paper, which introduces medication error risk. The team deploys hardware supporting 550 concurrent users.","label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"healthcare"}
+{"id":"margin-safety-11","mode":"binary-decision","prompt":"A Kafka consumer group processes 50,000 messages per hour at steady state. The upstream producer can burst to 200,000 messages per hour during end-of-day batch closes, lasting about 45 minutes. Consumer lag during burst causes downstream reports to be delayed, impacting trading desks. The team sizes consumers for 250,000 messages per hour sustained throughput.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"data-infrastructure"}
+{"id":"margin-safety-12","mode":"binary-decision","prompt":"A team is deploying a new machine learning inference service. Benchmark tests show the model needs 6GB of GPU VRAM at maximum batch size. The team selects a GPU instance with exactly 8GB VRAM. They note that framework overhead, CUDA context, and driver memory are not accounted for in the benchmark figure.","label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"ml"}
+{"id":"margin-safety-13","mode":"binary-decision","prompt":"A software team is migrating a monolith to microservices. The project manager estimates 6 months based on a rough breakdown of work. The migration touches every team in the company, requires coordination with three external vendors, and has no prior comparable project as reference. The team commits to a 12-month timeline with a formal review at month 6.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"project-management"}
+{"id":"margin-safety-14","mode":"binary-decision","prompt":"An IoT platform expects 10,000 devices to connect simultaneously after a firmware update pushes to all deployed units. Historical firmware pushes show actual simultaneous reconnection peaks reach 8x the device count (due to reconnect storms). A reconnection storm exceeding broker capacity causes devices to go permanently offline until manually reset. The broker is provisioned for 15,000 simultaneous connections.","label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"infrastructure"}
+{"id":"margin-safety-15","mode":"binary-decision","prompt":"A SaaS company's search index uses 40% of available disk space under current data volume. Data volume is projected to grow 20% per year, and the index requires immediate disk space to function — when disk hits 90% full the service stops indexing new documents. The team plans to add storage when usage hits 70% utilization, giving roughly 1.5 years of runway.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"infrastructure"}
+{"id":"margin-safety-16","mode":"binary-decision","prompt":"A team's API rate limit with their geolocation provider is 10,000 lookups per day. Current usage averages 6,500 lookups per day. A marketing campaign launching next month is expected to bring a 2–3x traffic increase lasting two weeks. Exceeding the rate limit returns errors on the address validation step of checkout, blocking all new orders.","label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"integration"}
+{"id":"margin-safety-17","mode":"binary-decision","prompt":"A team running a high-traffic CDN edge node configures a memory cache with a maximum size of 32GB. Profiling shows the working set of hot content peaks at 20GB during prime time, with cold-start periods using as little as 5GB. Cache evictions are graceful — a miss falls through to origin with a latency penalty but no data loss. They size at 32GB for comfortable headroom.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"infrastructure"}
+{"id":"margin-safety-18","mode":"binary-decision","prompt":"A developer estimates a new authentication refactor will take 10 days. The team has done this type of refactor three times in the past year, always within 8–12 days. The change is internal only, has no customer-facing deadline, and can be rolled back instantly if problems arise. They commit to 11 days.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"project-management"}
+{"id":"margin-safety-19","mode":"binary-decision","prompt":"A team deploys a Redis cache cluster sized for 8GB of memory. Current cache usage is 7.2GB. Keys are set with TTLs of 24 hours but the product roadmap includes launching three new features over the next two months, each of which is expected to add cached objects. Cache eviction uses LRU, and evicting hot keys causes a cascade of DB queries that has previously caused an incident.","label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"infrastructure"}
+{"id":"margin-safety-20","mode":"binary-decision","prompt":"A blockchain indexer must process blocks in real time; falling more than 100 blocks behind causes downstream dApps to display stale data, which breaks user transactions. Block arrival rate averages 12 per minute but spikes to 40 per minute during network congestion events that occur roughly twice a month. The team provisions for 60 blocks per minute sustained throughput.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"infrastructure"}
+{"id":"margin-safety-21","mode":"binary-decision","prompt":"A team's incident response SLA commits to acknowledging critical incidents within 15 minutes and resolving them within 4 hours. They have one primary on-call engineer and no backup. The primary has missed acknowledgment in 3 of the last 20 incidents due to being in a meeting or asleep in a different time zone.","label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"reliability"}
+{"id":"margin-safety-22","mode":"binary-decision","prompt":"An analytics pipeline writes results to an S3 bucket. Each run produces approximately 2GB. The pipeline runs daily, and 90 days of results are retained before lifecycle rules delete old files. The team allocates 250GB of S3 capacity via a quota. S3 storage is elastic and quota overages are immediately rechargeable at standard rates.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"data-infrastructure"}
+{"id":"margin-safety-23","mode":"binary-decision","prompt":"A team is launching a developer API with a free tier capped at 1,000 calls per month per user. Backend cost per call is $0.001. They provision infrastructure for 5 million total free-tier calls per month. They have 3,000 free-tier users signed up and expect 10,000 by end of quarter. At full expected adoption and full usage, demand would be 10 million calls per month.","label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"infrastructure"}
+{"id":"margin-safety-24","mode":"binary-decision","prompt":"A team configures a circuit breaker to open after 5% error rate sustained over 10 seconds. Under normal conditions the service runs at 0.1% error rate. The error rate threshold, window size, and recovery behavior were validated in production-like load tests. The circuit breaker configuration matches what was tested.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"reliability"}
+{"id":"margin-safety-25","mode":"binary-decision","prompt":"An e-commerce site expects 1,000 concurrent sessions during a flash sale. Load testing shows the application server handles 1,050 concurrent sessions before response times degrade past 3 seconds. A degraded checkout experience during the flash sale means customers abandon carts. The team does not add additional capacity, relying on auto-scaling that takes 4–6 minutes to provision new instances.","label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"infrastructure"}
+{"id":"margin-safety-26","mode":"binary-decision","prompt":"A team building a new internal reporting tool estimates the backend will need 4 vCPUs and 8GB RAM under expected query load. The tool is used only during business hours by 15 analysts. They provision an 8 vCPU, 16GB RAM instance, doubling both. If the tool becomes slow, analysts can wait without business impact.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"infrastructure"}
+{"id":"margin-safety-27","mode":"binary-decision","prompt":"A satellite telemetry ingestion pipeline must store all raw data permanently for regulatory reasons; any data gap is a compliance event with significant fines. The system ingests 10TB per year with no growth expected. The team provisions 12TB — 20% buffer — on a single storage array with no redundancy and no offsite backup.","label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"compliance"}
+{"id":"margin-safety-28","mode":"binary-decision","prompt":"A team uses a managed Elasticsearch cluster for product search. Normal indexing throughput is 2,000 documents per second. The team expects a re-index event during a schema migration that will run at 4,000 documents per second for approximately 6 hours. They temporarily scale the cluster to handle 6,000 documents per second for the migration window. A search outage means the product catalog is unavailable to customers.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"data-infrastructure"}
+{"id":"margin-safety-29","mode":"binary-decision","prompt":"A team provisions a job queue worker fleet sized to drain 100 jobs per minute — exactly matching the average production ingest rate. The queue is used for sending transactional emails; if the queue backs up significantly, emails arrive hours late. Historical data shows ingest spikes to 300 jobs per minute during large user-triggered events such as password reset floods.","label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"infrastructure"}
+{"id":"margin-safety-30","mode":"binary-decision","prompt":"A team committing to a quarterly OKR estimates a feature will be delivered in 6 weeks, based on a well-understood problem space, a team that has built nearly identical features before, and no external dependencies. They commit to 8 weeks in the OKR. Missing the OKR has minor internal visibility consequences and the feature can ship one quarter late without customer impact.","label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","skill_fit":["margin-of-safety"],"domain":"project-management"}
diff --git a/evals/datasets/authored/reversibility-doors.jsonl b/evals/datasets/authored/reversibility-doors.jsonl
new file mode 100644
index 0000000..2fbd54a
--- /dev/null
+++ b/evals/datasets/authored/reversibility-doors.jsonl
@@ -0,0 +1,30 @@
+{"id":"reversibility-01","mode":"binary-decision","prompt":"A backend engineer runs a script that permanently deletes all rows from the `user_events` table in the production PostgreSQL database to free up disk space. There is no backup of that table and no soft-delete column. The data is gone from all replicas as well.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"data-management"}
+{"id":"reversibility-02","mode":"binary-decision","prompt":"A developer adds a feature flag to the codebase that gates a new checkout flow for 5% of users. The flag defaults to off and can be toggled back to off in the admin dashboard within seconds.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"feature-delivery"}
+{"id":"reversibility-03","mode":"binary-decision","prompt":"The platform team drops the `legacy_sessions` column from the primary users table in the production database. Dozens of third-party integrations and internal services read that column directly. No migration script was written to restore it.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"database"}
+{"id":"reversibility-04","mode":"binary-decision","prompt":"An engineer changes the log level of a service from DEBUG to INFO in a single environment variable and restarts the container. The change takes effect immediately and can be reverted by changing the variable back and restarting again in under a minute.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"configuration"}
+{"id":"reversibility-05","mode":"binary-decision","prompt":"A startup publishes a public REST API (v1) that thousands of third-party developers have integrated into their apps. The team now wants to remove the `/user/profile` endpoint entirely without providing a versioned alternative or deprecation window.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"api-design"}
+{"id":"reversibility-06","mode":"binary-decision","prompt":"A developer renames a local variable from `tmp` to `parseResult` inside a private utility function that is not exported and has no callers outside the same file. The change touches two lines of code.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"code-quality"}
+{"id":"reversibility-07","mode":"binary-decision","prompt":"Finance triggers a batch wire transfer of $4.2M to a vendor as final payment for a completed contract. The transfer is marked irrevocable by the bank and the vendor's account is in a jurisdiction that does not support chargebacks. The money has left the account.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"finance"}
+{"id":"reversibility-08","mode":"binary-decision","prompt":"A team deploys a new microservice behind a canary release at 1% traffic. Monitoring dashboards are live, the old service still handles 99% of traffic, and rolling back means routing 100% back to the old service — a single Kubernetes rollout undo command.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"deployment"}
+{"id":"reversibility-09","mode":"binary-decision","prompt":"An engineer runs `aws s3 rb s3://prod-customer-exports --force` to delete a production S3 bucket containing 18 months of customer export files. Versioning was disabled and there is no secondary backup location.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"cloud-infrastructure"}
+{"id":"reversibility-10","mode":"binary-decision","prompt":"A product manager adds a new route `/api/v2/recommendations` to the API gateway, leaving the existing `/api/v1/recommendations` route untouched and fully operational. No existing clients are affected.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"api-design"}
+{"id":"reversibility-11","mode":"binary-decision","prompt":"The company executes a one-way data migration moving all user records from a self-hosted PostgreSQL instance into a managed third-party SaaS CRM. The PostgreSQL instance is then decommissioned and the data in it is wiped. No export copy was retained.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"data-migration"}
+{"id":"reversibility-12","mode":"binary-decision","prompt":"A developer adds a database index on the `created_at` column of the `orders` table to speed up a common query. The index can be dropped with a single `DROP INDEX` statement if it causes issues.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"database"}
+{"id":"reversibility-13","mode":"binary-decision","prompt":"An IoT firmware update is pushed over-the-air to 200,000 deployed devices in the field. The update overwrites the bootloader, which cannot be re-flashed remotely. Several thousand devices brick immediately after the rollout begins.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"embedded-systems"}
+{"id":"reversibility-14","mode":"binary-decision","prompt":"A developer changes the `DEFAULT_PAGE_SIZE` constant in a configuration file from 20 to 50, commits it to a feature branch, and opens a pull request. The branch has not been merged and can be closed without any production impact.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"configuration"}
+{"id":"reversibility-15","mode":"binary-decision","prompt":"A company publicly announces end-of-life for its legacy SDK, emails all registered developers, and removes the SDK download and documentation from the website. Dozens of enterprises have production systems depending on it with no migration path yet available.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"product-management"}
+{"id":"reversibility-16","mode":"binary-decision","prompt":"An engineering team experiments with using a different internal service for sending transactional emails in a staging environment only. No production traffic is affected, and they can switch back to the original service for the next staging deploy.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"infrastructure"}
+{"id":"reversibility-17","mode":"binary-decision","prompt":"The CTO signs a 5-year exclusive licensing agreement with a single analytics vendor. The contract includes steep early-termination penalties equivalent to 80% of the remaining contract value and prohibits integrating any competing analytics tools.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"vendor-management"}
+{"id":"reversibility-18","mode":"binary-decision","prompt":"A developer temporarily changes a weekly cron job to run every hour in the development environment to speed up local testing. The cron schedule is a one-line change in a config file that can be reverted any time.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"configuration"}
+{"id":"reversibility-19","mode":"binary-decision","prompt":"A mobile app team ships a forced upgrade that removes backward compatibility with the API version used by all prior app versions. Users on older app versions lose access immediately, and the old API endpoint is shut down at the same time the new app is released.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"mobile"}
+{"id":"reversibility-20","mode":"binary-decision","prompt":"A backend team creates a new optional field `preferred_locale` in the user settings table with a nullable default. Existing rows are unaffected, and the field can be dropped without any data loss if unused.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"database"}
+{"id":"reversibility-21","mode":"binary-decision","prompt":"A gaming company merges two games' player databases into a single unified identity system, then permanently shuts down the separate authentication services for both original games. Tens of millions of account histories are merged and the originals are deleted.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"data-migration"}
+{"id":"reversibility-22","mode":"binary-decision","prompt":"A developer swaps the order of two middleware functions in the Express.js request pipeline during local development. The change is in an uncommitted file on their laptop and has not been tested beyond their own machine.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"code-quality"}
+{"id":"reversibility-23","mode":"binary-decision","prompt":"A SaaS company migrates all customer billing to a new payment processor by re-tokenizing every stored payment method. The old processor's tokens are deleted immediately after migration. The old processor's API is then shut down, making the original tokens irrecoverable.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"payments"}
+{"id":"reversibility-24","mode":"binary-decision","prompt":"A team adds a new environment variable `ENABLE_DARK_MODE=false` to the production config. The variable is not yet read by any code in the deployed build, so it has zero runtime effect and can be removed anytime.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"configuration"}
+{"id":"reversibility-25","mode":"binary-decision","prompt":"The DevOps team decommissions and destroys all snapshots and AMIs for a legacy EC2-based environment after migrating to containers. Two weeks later a critical compliance audit requires log data that was only stored on those instances.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"cloud-infrastructure"}
+{"id":"reversibility-26","mode":"binary-decision","prompt":"An engineer wraps a third-party payment library behind an internal `PaymentGateway` interface before shipping the first integration. If the library needs to change later, only the adapter class needs updating, not the 40 call sites across the codebase.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"architecture"}
+{"id":"reversibility-27","mode":"binary-decision","prompt":"A social platform permanently purges all private messages older than 90 days from its database as part of a storage-reduction initiative. Users had not been informed of this policy and the data is not archived anywhere.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"data-management"}
+{"id":"reversibility-28","mode":"binary-decision","prompt":"A developer increases the connection pool size for the app's database from 10 to 20 in the application config file. The setting takes effect on the next container restart and can be reduced back just as easily.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"infrastructure"}
+{"id":"reversibility-29","mode":"binary-decision","prompt":"A company publicly commits to discontinuing a product line in a press release and notifies enterprise customers. Contracts are terminated, the team is reassigned, and the codebase is archived. Several enterprise customers immediately sign with competitors as a result.","label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"product-management"}
+{"id":"reversibility-30","mode":"binary-decision","prompt":"A QA engineer updates a unit test's expected string from `'Hello World'` to `'Hello, World'` to match a recent copy change. The test lives in a single file, the change is isolated, and reverting is a one-line git change.","label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","skill_fit":["reversibility"],"domain":"testing"}
diff --git a/evals/datasets/authored/second-order-consequence.jsonl b/evals/datasets/authored/second-order-consequence.jsonl
new file mode 100644
index 0000000..48e044c
--- /dev/null
+++ b/evals/datasets/authored/second-order-consequence.jsonl
@@ -0,0 +1,30 @@
+{"id":"second-order-01","mode":"binary-decision","prompt":"A mobile game studio is losing revenue because players leave after running out of in-game currency. The team proposes making currency easier to earn through daily login bonuses — tripling the daily reward amount. First-month retention numbers improve sharply. The studio expects this to sustain long-term revenue.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"product"}
+{"id":"second-order-02","mode":"binary-decision","prompt":"An e-commerce platform notices checkout abandonment spikes when shipping cost is revealed late in the flow. They move the shipping cost display to the product page, making pricing fully transparent upfront. They expect conversion to improve as trust builds. The change has no meaningful effect on the cost structure of shipping.","label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"product"}
+{"id":"second-order-03","mode":"binary-decision","prompt":"A SaaS company is struggling with a growing bug backlog. Engineering leadership introduces a policy awarding $100 cash bonuses to any developer who files and fixes a valid production bug. Within a week the bug count jumps dramatically, and leadership interprets this as the backlog finally becoming visible.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"engineering-process"}
+{"id":"second-order-04","mode":"binary-decision","prompt":"A platform API is seeing abuse from a small set of clients hammering endpoints. The team introduces strict per-IP rate limiting at 100 req/min. Abusive traffic stops. The team declares the problem solved and closes the incident.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"platform"}
+{"id":"second-order-05","mode":"binary-decision","prompt":"A startup's CI pipeline is slow — 35 minutes per run — causing developer frustration. The infrastructure team proposes caching all test artifacts aggressively, cutting the run to 8 minutes. They expect faster iteration cycles and higher developer satisfaction. The cached artifacts are invalidated on any dependency change.","label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"devops"}
+{"id":"second-order-06","mode":"binary-decision","prompt":"A content moderation team is overwhelmed by reports. Leadership proposes raising the threshold for escalation so analysts only review reports above a severity score of 7 (out of 10), reducing their queue by 60%. The goal is faster resolution of serious violations.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"trust-safety"}
+{"id":"second-order-07","mode":"binary-decision","prompt":"An open-source project maintainer is burned out by PR reviews. They add three more community members as co-maintainers to distribute the review load. All three are experienced contributors familiar with the codebase. The goal is to reduce any single maintainer's review burden.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"engineering-process"}
+{"id":"second-order-08","mode":"binary-decision","prompt":"A microservices team has high cognitive overhead from a sprawling service mesh. They consolidate three rarely-changed internal services into a single service with clear module boundaries and a single deployment unit. The merged service has no external consumers beyond one frontend team.","label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"architecture"}
+{"id":"second-order-09","mode":"binary-decision","prompt":"A fintech company's CDN cache TTL is set to 24 hours for account balance pages to reduce database load. A product manager proposes slashing the TTL to 10 seconds so users always see the freshest balance. The goal is reducing support tickets about stale balance displays.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"infrastructure"}
+{"id":"second-order-10","mode":"binary-decision","prompt":"A developer tools company wants to grow adoption, so they make their previously paid CLI free. Revenue from CLI licenses drops, but they expect the larger user base to convert to their premium cloud product at a higher rate. Historical data from comparable developer tools shows freemium models produce 3–8% paid conversion.","label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"product-strategy"}
+{"id":"second-order-11","mode":"binary-decision","prompt":"An engineering org is shipping too slowly. Leadership adds a mandatory architecture review board that must approve all significant design decisions before implementation begins. The stated goal is preventing expensive rework from poor designs. Teams currently spend about 15% of their time on rework.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"engineering-process"}
+{"id":"second-order-12","mode":"binary-decision","prompt":"A database team reduces read replica lag by promoting one replica to be a dedicated analytics replica with no replication delay budget — it runs slightly behind the primary but is isolated from OLTP traffic. Application query routing already distinguishes read vs. analytics queries. The goal is protecting OLTP performance during heavy analytic workloads.","label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"data-infrastructure"}
+{"id":"second-order-13","mode":"binary-decision","prompt":"A social platform wants to reduce misinformation spread. They implement a friction step — a one-click confirmation — before users can share articles that haven't been fact-checked. The team expects the added friction to reduce impulsive shares of false content.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"product"}
+{"id":"second-order-14","mode":"binary-decision","prompt":"A startup has a single point of failure in its payment processing: one payment gateway integration. The engineering team adds a second gateway with automatic failover so that if the primary goes down, the backup takes over within seconds. Both gateways are independently tested monthly.","label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"reliability"}
+{"id":"second-order-15","mode":"binary-decision","prompt":"A team deploys a feature flag system so product managers can toggle features without a deploy. Usage of the system grows steadily. After 18 months there are 400 active flags; roughly 200 have not been touched in over six months. The team considers this a success because it enables safe, gradual rollouts.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"engineering-process"}
+{"id":"second-order-16","mode":"binary-decision","prompt":"A logistics SaaS company notices its best customers churn because onboarding takes four weeks. They hire two dedicated onboarding specialists, reducing average onboarding to 10 days. There is no change to the product and no new handoff complexity created. The goal is reducing churn among enterprise customers.","label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"product"}
+{"id":"second-order-17","mode":"binary-decision","prompt":"A backend team finds that a slow third-party geocoding API call is blocking checkout for 400ms on average. They wrap the call in a background job, return the geocoded result asynchronously, and show a spinner to the user. Checkout p50 latency drops immediately by 380ms. The team closes the performance ticket.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"engineering"}
+{"id":"second-order-18","mode":"binary-decision","prompt":"A B2B startup has high customer acquisition cost. The CEO proposes replacing outbound cold email campaigns with a partner-led distribution deal with a company that already serves their target market. The partner gets a revenue share; the startup gets warm, qualified leads. The partner has a proven track record of similar deals.","label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"product-strategy"}
+{"id":"second-order-19","mode":"binary-decision","prompt":"A news app is struggling with low daily active user rates. The team adds more aggressive push notifications — increasing from 2 per day to 6 per day — featuring the most engaging (high click-through) headlines. Initial DAU ticks up 8%. The team plans to sustain the cadence.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"product"}
+{"id":"second-order-20","mode":"binary-decision","prompt":"A team running a Kubernetes cluster moves long-running batch jobs to a separate node pool with dedicated nodes, so batch workloads cannot evict latency-sensitive service pods. The node pool is right-sized for batch demand. The goal is eliminating latency spikes on the serving tier caused by resource contention.","label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"infrastructure"}
+{"id":"second-order-21","mode":"binary-decision","prompt":"An HR team at a fast-growing startup introduces mandatory weekly one-hour all-hands meetings to improve cross-team communication and culture. The company has 120 engineers. The goal is that everyone stays aligned on company direction and feels connected.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"org"}
+{"id":"second-order-22","mode":"binary-decision","prompt":"A SaaS billing system has intermittent invoice generation failures on the last day of the month due to a thundering herd of cron jobs all firing at midnight. The team staggers the invoice jobs across a six-hour window using jitter. The goal is eliminating month-end failures.","label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"engineering"}
+{"id":"second-order-23","mode":"binary-decision","prompt":"A marketplace platform takes a 15% commission on each transaction. To attract more high-value sellers, the team reduces commission to 8% for sellers doing over $10k/month in GMV. The goal is retaining and growing the top tier of sellers who drive the majority of platform GMV.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"product-strategy"}
+{"id":"second-order-24","mode":"binary-decision","prompt":"A team experiencing alert fatigue from 200+ daily PagerDuty pages decides to tune signal quality: they audit every alert, delete those with no documented runbook, and raise thresholds that were firing on non-actionable variance. Alert volume drops to 30 meaningful pages per day. The goal is that on-call engineers respond to real incidents faster.","label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"reliability"}
+{"id":"second-order-25","mode":"binary-decision","prompt":"A gaming company wants to reduce server costs. They enable aggressive auto-shutdown of idle game server instances after 2 minutes of inactivity. Server costs drop 40% in the first week. The game has a large cooperative multiplayer mode where players frequently pause mid-session to coordinate outside the game.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"infrastructure"}
+{"id":"second-order-26","mode":"binary-decision","prompt":"An API team publishes a versioning policy: all breaking changes must go into a new major version, and old versions are supported for 12 months with a migration guide. The goal is reducing the support burden from unannounced breaking changes that currently disrupt third-party integrators.","label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"platform"}
+{"id":"second-order-27","mode":"binary-decision","prompt":"A product team adds a \"Dark Mode\" toggle requested by power users. The toggle is available in account settings, implemented cleanly, and requires no changes to core product logic. The goal is improving satisfaction scores among power users.","label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"product"}
+{"id":"second-order-28","mode":"binary-decision","prompt":"A growth team at a consumer app finds that requiring phone number verification at signup reduces fraudulent accounts. They propose adding phone verification as a mandatory step for all new users, expecting fewer spam accounts and better deliverability of notifications. Signup conversion is not a primary metric they are tracking.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"product"}
+{"id":"second-order-29","mode":"binary-decision","prompt":"A platform team standardizes all internal services on a single observability stack (one metrics store, one logging system, one tracing backend) with a shared on-call runbook. Teams previously ran four different stacks. The goal is reducing the time to diagnose cross-service incidents.","label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"devops"}
+{"id":"second-order-30","mode":"binary-decision","prompt":"To speed up code review, an engineering director mandates that all PRs must be merged within 24 hours of being opened, with or without reviewer approval, to prevent blocking. The director expects this to improve deployment frequency and reduce stale PR accumulation.","label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","skill_fit":["second-order"],"domain":"engineering-process"}
diff --git a/evals/datasets/authored/theory-of-constraints-bottleneck.jsonl b/evals/datasets/authored/theory-of-constraints-bottleneck.jsonl
new file mode 100644
index 0000000..5169824
--- /dev/null
+++ b/evals/datasets/authored/theory-of-constraints-bottleneck.jsonl
@@ -0,0 +1,30 @@
+{"id":"toc-01","mode":"binary-decision","prompt":"A REST API request pipeline has four stages: Auth (2ms), Business Logic (8ms), Database Query (320ms), Response Serialization (5ms). The team wants to optimize the Business Logic stage by rewriting it in a more efficient language. Would this meaningfully improve end-to-end latency?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"api-performance"}
+{"id":"toc-02","mode":"binary-decision","prompt":"A data ingestion pipeline processes records through five stages: Read from S3 (0.5s/batch), Deserialize (0.2s), Validate (0.1s), Write to Kafka (4.8s), Acknowledge (0.05s). The team proposes parallelizing the Validate stage across 8 cores. Would this meaningfully improve end-to-end throughput?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"data-pipeline"}
+{"id":"toc-03","mode":"binary-decision","prompt":"A CI/CD pipeline runs these steps sequentially: Checkout (10s), Install Dependencies (25s), Unit Tests (4m 30s), Build Artifact (45s), Deploy (20s). The team wants to cache the Install Dependencies step to cut it to 5s. Would this meaningfully reduce total pipeline duration?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"devops"}
+{"id":"toc-04","mode":"binary-decision","prompt":"An image processing pipeline has these stages: Upload to S3 (0.3s), Queue Message (0.1s), ML Inference on GPU (12s), Resize Thumbnails (0.8s), Save Results (0.4s). The team proposes optimizing the Resize Thumbnails stage with a faster library. Would this improve end-to-end latency per image?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"ml-pipeline"}
+{"id":"toc-05","mode":"binary-decision","prompt":"A software release pipeline has: Dev Complete (2 days avg), Code Review (5 days avg, queue builds to 9 days), QA Testing (1 day), Staging Deploy (0.5 days), Production Deploy (0.5 days). The team proposes adding more QA engineers to cut QA time in half. Would this meaningfully increase release throughput?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"engineering-process"}
+{"id":"toc-06","mode":"binary-decision","prompt":"A payment processing pipeline executes: Fraud Score Request to external API (3.1s avg), Charge Card (0.4s), Update DB (0.1s), Send Receipt Email (0.2s). The team wants to optimize the database update with a bulk-write pattern. Would this materially reduce total transaction latency?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"payments"}
+{"id":"toc-07","mode":"binary-decision","prompt":"A report generation service runs: Fetch user permissions (30ms), Execute analytics SQL query (4,200ms), Format results to JSON (60ms), Render PDF (110ms). The team proposes adding a Redis cache for user permissions to reduce that step to 2ms. Would this meaningfully improve the total report generation time?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"reporting"}
+{"id":"toc-08","mode":"binary-decision","prompt":"A batch ETL job runs nightly with these timings: Extract from source DB (8 min), Transform records (42 min), Load into warehouse (6 min), Run post-load validations (4 min). The team wants to speed up the Extract step by adding read replicas. Would this cut overall job duration?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"data-pipeline"}
+{"id":"toc-09","mode":"binary-decision","prompt":"A web scraping pipeline processes URLs through: DNS resolution (15ms), TCP connect (20ms), TLS handshake (25ms), HTTP fetch (1,800ms avg), HTML parse (35ms), Write to DB (18ms). The team wants to run the HTML parse step on a faster machine. Would this noticeably reduce per-URL latency?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"data-pipeline"}
+{"id":"toc-10","mode":"binary-decision","prompt":"A video transcoding pipeline has these per-clip stages: Download Source (5s), Audio Decode (2s), Video Transcode (95s), Mux Streams (3s), Upload Output (8s). The team proposes switching to a faster audio codec to cut Audio Decode to 0.5s. Would this improve throughput or end-to-end time per clip?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"media-processing"}
+{"id":"toc-11","mode":"binary-decision","prompt":"A microservice handles requests through: Load Balancer (1ms), Auth Token Validation (3ms), Third-Party KYC API Call (2,600ms), Business Logic (15ms), DB Write (12ms). The team proposes parallelizing the KYC API call across multiple providers and using cached results, cutting its average to 120ms. Would this improve response time for users?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"api-performance"}
+{"id":"toc-12","mode":"binary-decision","prompt":"A checkout pipeline executes: Cart Validation (40ms), Inventory Lock (35ms), Payment Charge (850ms), Order Record Insert (28ms), Confirmation Email Enqueue (10ms). The team proposes switching to a payment provider with a p99 latency of 90ms, cutting the Payment Charge step from 850ms to 90ms. Would this substantially reduce checkout latency?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"ecommerce"}
+{"id":"toc-13","mode":"binary-decision","prompt":"A code analysis pipeline runs on each PR: Clone Repo (8s), Install Linters (12s), Static Analysis (180s), Dependency Audit (14s), Post GitHub Comment (3s). The team proposes rewriting the static analysis rules incrementally and running only changed-file analysis, cutting Static Analysis from 180s to under 20s. Would this meaningfully improve the pipeline's overall runtime?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"devops"}
+{"id":"toc-14","mode":"binary-decision","prompt":"An order fulfillment pipeline has these stages per order: Order Received (0.1s), Fraud Check via external API (3.9s), Pick and Pack (0.5s), Shipping Label Generation (0.3s), Carrier Handoff (0.2s). The team proposes replacing the external fraud API with an in-house ML model that runs in 80ms. Would this improve end-to-end order fulfillment latency?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"ecommerce"}
+{"id":"toc-15","mode":"binary-decision","prompt":"A search API pipeline processes queries through: Query Parse (2ms), Cache Lookup (4ms), Elasticsearch Query (1,100ms), Result Ranking (25ms), Response Serialize (8ms). The team proposes migrating to a purpose-built vector search engine that reduces query time from 1,100ms to 60ms for the Elasticsearch Query stage. Would this appreciably reduce the latency experienced by users?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"search"}
+{"id":"toc-16","mode":"binary-decision","prompt":"A REST API handles requests with these sequential steps: Request Parsing (3ms), Auth (5ms), DB Query (290ms), Response Formatting (4ms). The team proposes adding a read replica and query index to cut the DB Query stage to under 20ms. Would this dramatically improve the API's response times?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"api-performance"}
+{"id":"toc-17","mode":"binary-decision","prompt":"A CI/CD pipeline runs: Checkout (12s), Install Dependencies (18s), Unit Tests (8 min 45s), Integration Tests (2 min), Build (40s), Deploy (25s). The team proposes parallelizing the Unit Tests across 10 workers, which would cut the step to under 1 minute. Would this meaningfully shorten total pipeline run time?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"devops"}
+{"id":"toc-18","mode":"binary-decision","prompt":"A batch ML scoring pipeline processes records through: Read Batch from DB (1.2s), Preprocess Features (0.8s), Model Inference (38s per batch), Write Predictions (1.5s), Archive Input (0.4s). The team proposes running Model Inference on GPUs instead of CPUs, reducing it to 2s. Would this improve pipeline throughput?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"ml-pipeline"}
+{"id":"toc-19","mode":"binary-decision","prompt":"A software delivery process has these average stage durations: Requirements (1 day), Development (3 days), Code Review (11 days — queue backlog is 7 days), QA (1.5 days), Release (0.5 days). The team proposes assigning two more engineers specifically to code review. Would this increase feature release throughput?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"engineering-process"}
+{"id":"toc-20","mode":"binary-decision","prompt":"An event streaming pipeline has throughput rates: Kafka Consumer (50,000 msg/s), Deserialize (45,000 msg/s), Enrich via External API (1,800 msg/s), Aggregate (40,000 msg/s), Write to Postgres (38,000 msg/s). The team proposes adding connection pooling and batching to the External API calls to raise its throughput to 20,000 msg/s. Would this improve overall pipeline throughput?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"data-pipeline"}
+{"id":"toc-21","mode":"binary-decision","prompt":"A document processing pipeline runs: OCR Scan (2s), Text Extraction (1.8s), NLP Entity Recognition (24s), Index to Search (1.2s), Notify User (0.3s). The team proposes replacing the NLP model with a faster distilled version that achieves the same accuracy in 3s. Would this meaningfully reduce per-document processing time?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"ml-pipeline"}
+{"id":"toc-22","mode":"binary-decision","prompt":"A video upload pipeline processes each file: Client Upload to CDN (varies), Virus Scan (8s), Video Transcode (5m 20s), Thumbnail Generation (12s), Metadata Index (2s). The team proposes running Thumbnail Generation in parallel with Transcode. Would this reduce the total time a user waits for their video to be available?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"media-processing"}
+{"id":"toc-23","mode":"binary-decision","prompt":"A SaaS onboarding pipeline has per-customer steps: Sales Handoff (0.5 days), Legal Contract Review (9 days — average wait 6 days), Technical Integration (2 days), Training (1 day), Go-Live (0.5 days). The team proposes adding a dedicated legal reviewer to reduce contract review to 1 day. Would this reduce total time-to-live for new customers?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-edge throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"business-process"}
+{"id":"toc-24","mode":"binary-decision","prompt":"A real-time analytics query pipeline runs: Parse SQL (5ms), Check Permissions (8ms), Execute OLAP Query (4,500ms), Format Response (12ms), Compress Payload (18ms). The team proposes switching to a columnar engine that reduces OLAP Query time to 200ms. Would this substantially reduce user-perceived query latency?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"analytics"}
+{"id":"toc-25","mode":"binary-decision","prompt":"A fraud detection pipeline evaluates each transaction: Rule Engine (10ms), ML Feature Extraction (22ms), External Credit Bureau Lookup (2,100ms), Score Aggregation (8ms), Decision Write (6ms). The team proposes caching Credit Bureau results by SSN with a 24-hour TTL, cutting average lookup time to 15ms for repeat customers. Would this materially reduce fraud check latency?","label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"payments"}
+{"id":"toc-26","mode":"binary-decision","prompt":"A nightly data warehouse refresh has stages: Extract from Postgres (14 min), Transform (62 min), Load to Snowflake (10 min), Run dbt Models (8 min). The team proposes upgrading the Snowflake warehouse tier to cut Load time to 3 min. Would this meaningfully shorten total refresh duration?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"data-pipeline"}
+{"id":"toc-27","mode":"binary-decision","prompt":"A background job pipeline processes user export requests: Enqueue Job (0.1s), Fetch User Data from DB (5,800ms), Serialize to CSV (200ms), Compress File (180ms), Upload to S3 (400ms), Email User Link (100ms). The team wants to optimize the Compress File step with a faster algorithm. Would this significantly reduce the time users wait for their export?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"data-pipeline"}
+{"id":"toc-28","mode":"binary-decision","prompt":"A hardware test pipeline runs: Flash Firmware (45s), Boot Device (12s), Run Calibration Suite (8 min 30s), Log Results (5s), Package for Shipping (30s). The team proposes upgrading the USB flasher to cut firmware flash time to 10s. Would this improve overall throughput of units through the test line?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"hardware"}
+{"id":"toc-29","mode":"binary-decision","prompt":"A Kubernetes pod startup sequence involves: Image Pull (3s), Init Containers (8s), Application Boot (94s — slow Spring Boot cold start), Readiness Probe (5s), Traffic Admission (1s). The team proposes pre-warming the image pull by caching it on every node. Would this reduce the time from deployment trigger to traffic serving?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"infrastructure"}
+{"id":"toc-30","mode":"binary-decision","prompt":"A machine learning training pipeline runs per epoch: Data Loading (8s), Augmentation (5s), Forward Pass (310s), Backward Pass (290s), Checkpoint Save (12s). The team proposes optimizing the Data Loading step with async prefetching to reduce it to 1s. Would this reduce total training time per epoch?","label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","skill_fit":["theory-of-constraints"],"domain":"ml-pipeline"}
diff --git a/evals/datasets/ingest-hf.js b/evals/datasets/ingest-hf.js
index d26bde2..61f27f7 100644
--- a/evals/datasets/ingest-hf.js
+++ b/evals/datasets/ingest-hf.js
@@ -109,11 +109,13 @@ const SOURCES = {
};
},
},
- strategyqa: { // second-order proxy: implicit multi-hop yes/no
- dataset: 'voidful/StrategyQA', config: 'default', split: 'train',
- mode: 'correctness', license: 'CC (Wikipedia-derived)',
+ strategyqa: { // second-order proxy: implicit multi-hop yes/no (balanced Yes/No)
+ dataset: 'ChilleD/StrategyQA', config: 'default', split: 'train',
+ mode: 'correctness', license: 'CC (Wikipedia-derived, Apache-2.0 repo)',
+ balanceScan: true,
+ classOf: m => (m.answer === 'Yes' ? 'positive' : 'negative'),
map: r => (r.question && typeof r.answer === 'boolean') ? {
- prompt: r.question + '\n\nAnswer Yes or No.', answer: r.answer ? 'Yes' : 'No', skill_fit: ['second-order'],
+ prompt: r.question + '\n\nThink step by step about the implicit chain of facts, then answer Yes or No.', answer: r.answer ? 'Yes' : 'No', skill_fit: ['second-order'],
} : null,
},
};
diff --git a/evals/run-confirm.sh b/evals/run-confirm.sh
new file mode 100755
index 0000000..ebd2e92
--- /dev/null
+++ b/evals/run-confirm.sh
@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+# Confirmation runs at power for the two live positive leads (sequential, quota-safe).
+set -u
+cd "$(dirname "$0")/.."
+export EVAL_RUN=confirm
+OUT=evals/results/confirm; mkdir -p "$OUT"
+LOG=evals/results/confirm.log; : > "$LOG"
+run () { echo "===== $* =====" | tee -a "$LOG"; eval "$*" 2>&1 | grep -v "_omz_nvm" | tee -a "$LOG"; echo "" | tee -a "$LOG"; }
+echo "CONFIRM START $(date -u +%FT%TZ)" | tee -a "$LOG"
+# red-team: full balanced DiverseVul (200, 100/100) — no LIMIT to keep balance
+run "FORCE_SKILL=red-team CONC=4 OUTFILE=$OUT/redteam-diversevul-balanced-n200.json node evals/run-correctness.js diversevul-balanced"
+# fermi: n=150 numeric OOM
+run "FORCE_SKILL=fermi-estimation LIMIT=150 CONC=4 OUTFILE=$OUT/numeric-fermi-n150.json node evals/run-numeric.js fermi"
+echo "CONFIRM DONE $(date -u +%FT%TZ)" | tee -a "$LOG"
diff --git a/evals/run-rerun.sh b/evals/run-rerun.sh
new file mode 100755
index 0000000..abba7d7
--- /dev/null
+++ b/evals/run-rerun.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+# Rerun: powered objective evals for the newly-covered gap skills (sequential, quota-safe).
+# All isolated, length-controlled, paired skill-vs-placebo, judge-free (objective ground truth).
+set -u
+cd "$(dirname "$0")/.."
+export EVAL_RUN=rerun
+OUT=evals/results/rerun; mkdir -p "$OUT"
+LOG=evals/results/rerun.log; : > "$LOG"
+run () { echo "===== $* =====" | tee -a "$LOG"; eval "$*" 2>&1 | grep -v "_omz_nvm" | tee -a "$LOG"; echo "" | tee -a "$LOG"; }
+echo "RERUN START $(date -u +%FT%TZ)" | tee -a "$LOG"
+
+run "FORCE_SKILL=reversibility CONC=4 OUTFILE=$OUT/reversibility-doors.json node evals/run-routing-data.js reversibility-doors"
+run "FORCE_SKILL=first-principles CONC=4 OUTFILE=$OUT/first-principles-constraint.json node evals/run-routing-data.js first-principles-constraint"
+run "FORCE_SKILL=theory-of-constraints CONC=4 OUTFILE=$OUT/theory-of-constraints-bottleneck.json node evals/run-routing-data.js theory-of-constraints-bottleneck"
+run "FORCE_SKILL=second-order CONC=4 OUTFILE=$OUT/second-order-consequence.json node evals/run-routing-data.js second-order-consequence"
+run "FORCE_SKILL=map-territory CONC=4 OUTFILE=$OUT/map-territory-verify.json node evals/run-routing-data.js map-territory-verify"
+run "FORCE_SKILL=margin-of-safety CONC=4 OUTFILE=$OUT/margin-of-safety-provision.json node evals/run-routing-data.js margin-of-safety-provision"
+run "FORCE_SKILL=second-order CONC=4 OUTFILE=$OUT/strategyqa-second-order.json node evals/run-correctness.js strategyqa"
+
+echo "RERUN DONE $(date -u +%FT%TZ)" | tee -a "$LOG"