28 eval-informed mental models and critical-thinking skills for Claude Code, GitHub Copilot, Codex, Cursor, and other Agent Skills-compatible tools
JavaScript agent-skillsai-toolsanthropicclaudeclaude-codecritical-thinkingdecision-makingfirst-principlesmental-modelsproblem-solvingsystems-thinking
fix(m3-conceptual-systems-reconcile): re-freeze decisive splits from harder calibrated pools, reconcile manifest counts, fix source refs, set systems in_band=true
Travis Boudreaux401aef70aa4a
diff --git a/evals/datasets/calibrated/conceptual-reframes-decisive.jsonl b/evals/datasets/calibrated/conceptual-reframes-decisive.jsonlindex dc1399b..2895007 100644--- a/evals/datasets/calibrated/conceptual-reframes-decisive.jsonl+++ b/evals/datasets/calibrated/conceptual-reframes-decisive.jsonl@@ -1,194 +1,224 @@-{"id":"first-principles-01","original_id":"first-principles-01","prompt":"A distributed system team says they cannot have both strong consistency and 100% availability during a network partition. They cite the CAP theorem as the reason and are designing around this trade-off.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-02","original_id":"first-principles-02","prompt":"A backend team insists they cannot switch from REST to gRPC for internal service communication because 'we've always used REST and all our tooling is built around it.' The gRPC library is available in their language stack and the team hasn't evaluated it.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-03","original_id":"first-principles-03","prompt":"A healthcare startup must store patient records exclusively within the EU because a government regulation mandates data residency for citizens' medical data within the country of origin. Legal counsel has confirmed this is a binding statutory requirement.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-04","original_id":"first-principles-04","prompt":"A software team says they can't add a dark mode to their web app because 'the design system doesn't support it.' On investigation, the design system uses CSS custom properties for colors and simply hasn't been extended — no technical barrier prevents adding a second theme.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-05","original_id":"first-principles-05","prompt":"An infrastructure architect explains that two datacenters on opposite sides of the planet have an irreducible minimum round-trip latency of roughly 130ms due to the speed of light through fiber. No amount of software optimization can eliminate this floor.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-06","original_id":"first-principles-06","prompt":"An engineering team says they can't break up their monolith into separate services because 'that's how the legacy system was built and changing it would break everything.' The codebase has clear module boundaries and no architectural reason prevents decomposition.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-07","original_id":"first-principles-07","prompt":"A cryptography engineer explains that breaking a 256-bit AES key by brute force would require more computational energy than exists in the observable universe given current physics. This is why the encryption is treated as unbreakable for practical purposes.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-08","original_id":"first-principles-08","prompt":"A team lead says they can't use snake_case for new database columns because 'our naming convention is camelCase.' This is an internal style guide documented in a wiki page that the team wrote themselves and has never been reviewed for technical necessity.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-09","original_id":"first-principles-09","prompt":"A financial platform is told it cannot process credit card transactions without being PCI-DSS certified. This is a contractual and legal requirement from the card networks — Visa, Mastercard — that applies to any entity storing, processing, or transmitting cardholder data.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-10","original_id":"first-principles-10","prompt":"A product team claims they can't add real-time collaborative editing to their document tool because 'our backend uses REST and REST can't do real-time.' WebSockets and server-sent events are viable and the team hasn't actually evaluated adding them.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-11","original_id":"first-principles-11","prompt":"A machine learning team notes that a model trained only on data from before a certain date genuinely cannot have learned about events that occurred afterward. The knowledge cutoff is an inherent result of when training data was collected.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-12","original_id":"first-principles-12","prompt":"A developer says the team can't write integration tests because 'we've never had them in this codebase and setting them up would take too long.' The services expose standard HTTP APIs and a test harness exists in adjacent projects the team hasn't looked at.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-13","original_id":"first-principles-13","prompt":"A chip design team explains they cannot make transistors smaller than a few atoms because quantum tunneling effects make the device physically non-functional at sub-atomic scales. This is a hard physical limit of semiconductor physics.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-14","original_id":"first-principles-14","prompt":"An API team says they can't version their API because 'that's not how we do things here.' Looking at the codebase, there are no technical barriers — the routing layer already supports path prefixes and adding `/v2/` would require minimal changes.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-15","original_id":"first-principles-15","prompt":"A data pipeline processes sensor readings and an engineer notes that you cannot have sub-millisecond latency between a sensor event and its reception on a server 500km away because the physical signal propagation time alone exceeds that bound.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-16","original_id":"first-principles-16","prompt":"An engineering manager says the team can't do continuous deployment because 'we've always done quarterly releases.' The CI/CD infrastructure is already in place and there are no contractual, regulatory, or technical blockers identified — just existing practice.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-17","original_id":"first-principles-17","prompt":"A distributed consensus algorithm requires a majority quorum of nodes to agree before committing a write. With 5 nodes, this means the system mathematically cannot tolerate more than 2 simultaneous failures and still make progress.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-18","original_id":"first-principles-18","prompt":"A team claims they can't use a monorepo because 'our Git workflow doesn't support it.' After digging in, the actual blocker is that their CI scripts use a hardcoded single-repo assumption — a self-imposed implementation detail that could be changed in a sprint.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-20","original_id":"first-principles-20","prompt":"A frontend developer insists they can't paginate the results of a data table because 'the current API returns all rows at once.' The API team is on the same squad and has capacity to add pagination query parameters — no external dependency is involved.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-21","original_id":"first-principles-21","prompt":"A compression algorithm engineer explains that you cannot losslessly compress a truly random bitstream to fewer bits than it contains — Shannon's source coding theorem sets a mathematical lower bound on entropy coding.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-22","original_id":"first-principles-22","prompt":"A platform team says they can't give engineers production database access because 'that's policy.' Investigation shows the policy was written by a former DBA who left two years ago and was never reviewed for necessity; no regulatory or audit requirement mandates it.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-23","original_id":"first-principles-23","prompt":"A GPU cluster running neural network training is limited by memory bandwidth — there is a physical maximum rate at which data can move between DRAM and the compute cores dictated by the memory bus architecture of the specific GPU model.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-24","original_id":"first-principles-24","prompt":"An analytics team says they can't use columnar storage for their reporting queries because 'the warehouse is Postgres.' Their cloud provider offers a Postgres-compatible columnar extension the team hasn't evaluated and that would work with their current ORM.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-25","original_id":"first-principles-25","prompt":"A European fintech company cannot share raw transaction data about EU residents with its US parent company's analytics team without explicit user consent or an approved legal mechanism, due to GDPR Article 44 restricting cross-border data transfers.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-26","original_id":"first-principles-26","prompt":"A developer says they can't add structured logging to a service because 'logging is done with print statements in this codebase.' There is no technical barrier — the language has a standard structured logging library and the other services already use it.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-27","original_id":"first-principles-27","prompt":"A distributed sort algorithm must move at minimum O(N log N) comparisons' worth of data to sort N arbitrary elements — this is a provably optimal lower bound for comparison-based sorting that no comparison sort can beat.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-28","original_id":"first-principles-28","prompt":"A product team says they can't run A/B tests because 'we don't have an experimentation platform.' Their analytics stack already captures user events, their feature flag system is in place, and three engineers estimate a basic split-test harness could be built in a week.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-29","original_id":"first-principles-29","prompt":"A real-time trading system must comply with SEC Rule 17a-4, which legally mandates that certain trade records be stored in a non-rewritable, non-erasable format for a minimum of six years. Brokers have no legal authority to waive this requirement.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"first-principles-30","original_id":"first-principles-30","prompt":"A mobile developer says they can't make their iOS app available on Android because 'we build in Swift.' Swift is iOS-specific, but the product logic has no hardware dependency and could be rewritten in a cross-platform framework — nothing physically prevents an Android port.","source":"first-principles-constraint","source_file":"authored/first-principles-constraint.jsonl","skill_fit":["first-principles"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-01","original_id":"reversibility-01","prompt":"A backend engineer runs a script that permanently deletes all rows from the `user_events` table in the production PostgreSQL database to free up disk space. There is no backup of that table and no soft-delete column. The data is gone from all replicas as well.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-02","original_id":"reversibility-02","prompt":"A developer adds a feature flag to the codebase that gates a new checkout flow for 5% of users. The flag defaults to off and can be toggled back to off in the admin dashboard within seconds.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-03","original_id":"reversibility-03","prompt":"The platform team drops the `legacy_sessions` column from the primary users table in the production database. Dozens of third-party integrations and internal services read that column directly. No migration script was written to restore it.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-04","original_id":"reversibility-04","prompt":"An engineer changes the log level of a service from DEBUG to INFO in a single environment variable and restarts the container. The change takes effect immediately and can be reverted by changing the variable back and restarting again in under a minute.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-05","original_id":"reversibility-05","prompt":"A startup publishes a public REST API (v1) that thousands of third-party developers have integrated into their apps. The team now wants to remove the `/user/profile` endpoint entirely without providing a versioned alternative or deprecation window.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-06","original_id":"reversibility-06","prompt":"A developer renames a local variable from `tmp` to `parseResult` inside a private utility function that is not exported and has no callers outside the same file. The change touches two lines of code.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-07","original_id":"reversibility-07","prompt":"Finance triggers a batch wire transfer of $4.2M to a vendor as final payment for a completed contract. The transfer is marked irrevocable by the bank and the vendor's account is in a jurisdiction that does not support chargebacks. The money has left the account.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-08","original_id":"reversibility-08","prompt":"A team deploys a new microservice behind a canary release at 1% traffic. Monitoring dashboards are live, the old service still handles 99% of traffic, and rolling back means routing 100% back to the old service — a single Kubernetes rollout undo command.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-09","original_id":"reversibility-09","prompt":"An engineer runs `aws s3 rb s3://prod-customer-exports --force` to delete a production S3 bucket containing 18 months of customer export files. Versioning was disabled and there is no secondary backup location.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-10","original_id":"reversibility-10","prompt":"A product manager adds a new route `/api/v2/recommendations` to the API gateway, leaving the existing `/api/v1/recommendations` route untouched and fully operational. No existing clients are affected.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-11","original_id":"reversibility-11","prompt":"The company executes a one-way data migration moving all user records from a self-hosted PostgreSQL instance into a managed third-party SaaS CRM. The PostgreSQL instance is then decommissioned and the data in it is wiped. No export copy was retained.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-12","original_id":"reversibility-12","prompt":"A developer adds a database index on the `created_at` column of the `orders` table to speed up a common query. The index can be dropped with a single `DROP INDEX` statement if it causes issues.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-13","original_id":"reversibility-13","prompt":"An IoT firmware update is pushed over-the-air to 200,000 deployed devices in the field. The update overwrites the bootloader, which cannot be re-flashed remotely. Several thousand devices brick immediately after the rollout begins.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-14","original_id":"reversibility-14","prompt":"A developer changes the `DEFAULT_PAGE_SIZE` constant in a configuration file from 20 to 50, commits it to a feature branch, and opens a pull request. The branch has not been merged and can be closed without any production impact.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-15","original_id":"reversibility-15","prompt":"A company publicly announces end-of-life for its legacy SDK, emails all registered developers, and removes the SDK download and documentation from the website. Dozens of enterprises have production systems depending on it with no migration path yet available.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-16","original_id":"reversibility-16","prompt":"An engineering team experiments with using a different internal service for sending transactional emails in a staging environment only. No production traffic is affected, and they can switch back to the original service for the next staging deploy.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-17","original_id":"reversibility-17","prompt":"The CTO signs a 5-year exclusive licensing agreement with a single analytics vendor. The contract includes steep early-termination penalties equivalent to 80% of the remaining contract value and prohibits integrating any competing analytics tools.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-18","original_id":"reversibility-18","prompt":"A developer temporarily changes a weekly cron job to run every hour in the development environment to speed up local testing. The cron schedule is a one-line change in a config file that can be reverted any time.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-19","original_id":"reversibility-19","prompt":"A mobile app team ships a forced upgrade that removes backward compatibility with the API version used by all prior app versions. Users on older app versions lose access immediately, and the old API endpoint is shut down at the same time the new app is released.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-20","original_id":"reversibility-20","prompt":"A backend team creates a new optional field `preferred_locale` in the user settings table with a nullable default. Existing rows are unaffected, and the field can be dropped without any data loss if unused.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-21","original_id":"reversibility-21","prompt":"A gaming company merges two games' player databases into a single unified identity system, then permanently shuts down the separate authentication services for both original games. Tens of millions of account histories are merged and the originals are deleted.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-22","original_id":"reversibility-22","prompt":"A developer swaps the order of two middleware functions in the Express.js request pipeline during local development. The change is in an uncommitted file on their laptop and has not been tested beyond their own machine.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-24","original_id":"reversibility-24","prompt":"A team adds a new environment variable `ENABLE_DARK_MODE=false` to the production config. The variable is not yet read by any code in the deployed build, so it has zero runtime effect and can be removed anytime.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-25","original_id":"reversibility-25","prompt":"The DevOps team decommissions and destroys all snapshots and AMIs for a legacy EC2-based environment after migrating to containers. Two weeks later a critical compliance audit requires log data that was only stored on those instances.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-26","original_id":"reversibility-26","prompt":"An engineer wraps a third-party payment library behind an internal `PaymentGateway` interface before shipping the first integration. If the library needs to change later, only the adapter class needs updating, not the 40 call sites across the codebase.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-27","original_id":"reversibility-27","prompt":"A social platform permanently purges all private messages older than 90 days from its database as part of a storage-reduction initiative. Users had not been informed of this policy and the data is not archived anywhere.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-28","original_id":"reversibility-28","prompt":"A developer increases the connection pool size for the app's database from 10 to 20 in the application config file. The setting takes effect on the next container restart and can be reduced back just as easily.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-29","original_id":"reversibility-29","prompt":"A company publicly commits to discontinuing a product line in a press release and notifies enterprise customers. Contracts are terminated, the team is reassigned, and the codebase is archived. Several enterprise customers immediately sign with competitors as a result.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"reversibility-30","original_id":"reversibility-30","prompt":"A QA engineer updates a unit test's expected string from `'Hello World'` to `'Hello, World'` to match a recent copy change. The test lives in a single file, the change is isolated, and reverting is a one-line git change.","source":"reversibility-doors","source_file":"authored/reversibility-doors.jsonl","skill_fit":["reversibility"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse — costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-01","original_id":"second-order-01","prompt":"A mobile game studio is losing revenue because players leave after running out of in-game currency. The team proposes making currency easier to earn through daily login bonuses — tripling the daily reward amount. First-month retention numbers improve sharply. The studio expects this to sustain long-term revenue.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-02","original_id":"second-order-02","prompt":"An e-commerce platform notices checkout abandonment spikes when shipping cost is revealed late in the flow. They move the shipping cost display to the product page, making pricing fully transparent upfront. They expect conversion to improve as trust builds. The change has no meaningful effect on the cost structure of shipping.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-03","original_id":"second-order-03","prompt":"A SaaS company is struggling with a growing bug backlog. Engineering leadership introduces a policy awarding $100 cash bonuses to any developer who files and fixes a valid production bug. Within a week the bug count jumps dramatically, and leadership interprets this as the backlog finally becoming visible.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-04","original_id":"second-order-04","prompt":"A platform API is seeing abuse from a small set of clients hammering endpoints. The team introduces strict per-IP rate limiting at 100 req/min. Abusive traffic stops. The team declares the problem solved and closes the incident.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-05","original_id":"second-order-05","prompt":"A startup's CI pipeline is slow — 35 minutes per run — causing developer frustration. The infrastructure team proposes caching all test artifacts aggressively, cutting the run to 8 minutes. They expect faster iteration cycles and higher developer satisfaction. The cached artifacts are invalidated on any dependency change.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-06","original_id":"second-order-06","prompt":"A content moderation team is overwhelmed by reports. Leadership proposes raising the threshold for escalation so analysts only review reports above a severity score of 7 (out of 10), reducing their queue by 60%. The goal is faster resolution of serious violations.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-07","original_id":"second-order-07","prompt":"An open-source project maintainer is burned out by PR reviews. They add three more community members as co-maintainers to distribute the review load. All three are experienced contributors familiar with the codebase. The goal is to reduce any single maintainer's review burden.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-08","original_id":"second-order-08","prompt":"A microservices team has high cognitive overhead from a sprawling service mesh. They consolidate three rarely-changed internal services into a single service with clear module boundaries and a single deployment unit. The merged service has no external consumers beyond one frontend team.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-09","original_id":"second-order-09","prompt":"A fintech company's CDN cache TTL is set to 24 hours for account balance pages to reduce database load. A product manager proposes slashing the TTL to 10 seconds so users always see the freshest balance. The goal is reducing support tickets about stale balance displays.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-10","original_id":"second-order-10","prompt":"A developer tools company wants to grow adoption, so they make their previously paid CLI free. Revenue from CLI licenses drops, but they expect the larger user base to convert to their premium cloud product at a higher rate. Historical data from comparable developer tools shows freemium models produce 3–8% paid conversion.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-11","original_id":"second-order-11","prompt":"An engineering org is shipping too slowly. Leadership adds a mandatory architecture review board that must approve all significant design decisions before implementation begins. The stated goal is preventing expensive rework from poor designs. Teams currently spend about 15% of their time on rework.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-12","original_id":"second-order-12","prompt":"A database team reduces read replica lag by promoting one replica to be a dedicated analytics replica with no replication delay budget — it runs slightly behind the primary but is isolated from OLTP traffic. Application query routing already distinguishes read vs. analytics queries. The goal is protecting OLTP performance during heavy analytic workloads.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-13","original_id":"second-order-13","prompt":"A social platform wants to reduce misinformation spread. They implement a friction step — a one-click confirmation — before users can share articles that haven't been fact-checked. The team expects the added friction to reduce impulsive shares of false content.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-14","original_id":"second-order-14","prompt":"A startup has a single point of failure in its payment processing: one payment gateway integration. The engineering team adds a second gateway with automatic failover so that if the primary goes down, the backup takes over within seconds. Both gateways are independently tested monthly.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-15","original_id":"second-order-15","prompt":"A team deploys a feature flag system so product managers can toggle features without a deploy. Usage of the system grows steadily. After 18 months there are 400 active flags; roughly 200 have not been touched in over six months. The team considers this a success because it enables safe, gradual rollouts.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-16","original_id":"second-order-16","prompt":"A logistics SaaS company notices its best customers churn because onboarding takes four weeks. They hire two dedicated onboarding specialists, reducing average onboarding to 10 days. There is no change to the product and no new handoff complexity created. The goal is reducing churn among enterprise customers.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-17","original_id":"second-order-17","prompt":"A backend team finds that a slow third-party geocoding API call is blocking checkout for 400ms on average. They wrap the call in a background job, return the geocoded result asynchronously, and show a spinner to the user. Checkout p50 latency drops immediately by 380ms. The team closes the performance ticket.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-18","original_id":"second-order-18","prompt":"A B2B startup has high customer acquisition cost. The CEO proposes replacing outbound cold email campaigns with a partner-led distribution deal with a company that already serves their target market. The partner gets a revenue share; the startup gets warm, qualified leads. The partner has a proven track record of similar deals.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-19","original_id":"second-order-19","prompt":"A news app is struggling with low daily active user rates. The team adds more aggressive push notifications — increasing from 2 per day to 6 per day — featuring the most engaging (high click-through) headlines. Initial DAU ticks up 8%. The team plans to sustain the cadence.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-20","original_id":"second-order-20","prompt":"A team running a Kubernetes cluster moves long-running batch jobs to a separate node pool with dedicated nodes, so batch workloads cannot evict latency-sensitive service pods. The node pool is right-sized for batch demand. The goal is eliminating latency spikes on the serving tier caused by resource contention.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-21","original_id":"second-order-21","prompt":"An HR team at a fast-growing startup introduces mandatory weekly one-hour all-hands meetings to improve cross-team communication and culture. The company has 120 engineers. The goal is that everyone stays aligned on company direction and feels connected.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-22","original_id":"second-order-22","prompt":"A SaaS billing system has intermittent invoice generation failures on the last day of the month due to a thundering herd of cron jobs all firing at midnight. The team staggers the invoice jobs across a six-hour window using jitter. The goal is eliminating month-end failures.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-23","original_id":"second-order-23","prompt":"A marketplace platform takes a 15% commission on each transaction. To attract more high-value sellers, the team reduces commission to 8% for sellers doing over $10k/month in GMV. The goal is retaining and growing the top tier of sellers who drive the majority of platform GMV.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-24","original_id":"second-order-24","prompt":"A team experiencing alert fatigue from 200+ daily PagerDuty pages decides to tune signal quality: they audit every alert, delete those with no documented runbook, and raise thresholds that were firing on non-actionable variance. Alert volume drops to 30 meaningful pages per day. The goal is that on-call engineers respond to real incidents faster.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-25","original_id":"second-order-25","prompt":"A gaming company wants to reduce server costs. They enable aggressive auto-shutdown of idle game server instances after 2 minutes of inactivity. Server costs drop 40% in the first week. The game has a large cooperative multiplayer mode where players frequently pause mid-session to coordinate outside the game.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-26","original_id":"second-order-26","prompt":"An API team publishes a versioning policy: all breaking changes must go into a new major version, and old versions are supported for 12 months with a migration guide. The goal is reducing the support burden from unannounced breaking changes that currently disrupt third-party integrators.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-27","original_id":"second-order-27","prompt":"A product team adds a \"Dark Mode\" toggle requested by power users. The toggle is available in account settings, implemented cleanly, and requires no changes to core product logic. The goal is improving satisfaction scores among power users.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-28","original_id":"second-order-28","prompt":"A growth team at a consumer app finds that requiring phone number verification at signup reduces fraudulent accounts. They propose adding phone verification as a mandatory step for all new users, expecting fewer spam accounts and better deliverability of notifications. Signup conversion is not a primary metric they are tracking.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-29","original_id":"second-order-29","prompt":"A platform team standardizes all internal services on a single observability stack (one metrics store, one logging system, one tracing backend) with a shared on-call runbook. Teams previously ran four different stacks. The goal is reducing the time to diagnose cross-service incidents.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":true,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"second-order-30","original_id":"second-order-30","prompt":"To speed up code review, an engineering director mandates that all PRs must be merged within 24 hours of being opened, with or without reviewer approval, to prevent blocking. The director expects this to improve deployment frequency and reduce stale PR accumulation.","source":"second-order-consequence","source_file":"authored/second-order-consequence.jsonl","skill_fit":["second-order"],"family":"Conceptual reframes","type":"target","target":true,"label":false,"decision_instruction":"Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-1","original_id":"debias-1","prompt":"A real-estate agent first shows you a listing priced at $1,200,000 that is clearly overpriced and sits on the market for months. The next house she shows, comparable in size and condition, is listed at $850,000. Independent comparable sales in the neighborhood over the last 90 days cluster tightly around $700,000 for homes of this type. You want to make an opening offer that reflects fair market value. Which opening offer is most defensible?\nA. $1,100,000, since the first house anchors the high end of the market\nB. $850,000, matching the asking price of the second house\nC. $780,000, splitting the difference between the two listings you saw\nD. $700,000, based on recent comparable sales\nE. $950,000, the average of the two listing prices","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"D","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-2","original_id":"debias-2","prompt":"You are estimating how long a backend migration will take. Before you start, a colleague casually mentions, 'I bet this takes at least 6 months.' You break the work down into tasks and your own bottom-up estimate, summing each component with buffers, comes to about 7 weeks. Which estimate should you commit to in the project plan?\nA. About 7 weeks, matching your detailed bottom-up breakdown\nB. About 6 months, since the colleague's experience suggests you're underestimating\nC. About 4 months, splitting the difference between your estimate and the colleague's\nD. About 3 months, padding your estimate heavily toward the colleague's number\nE. About 10 weeks, nudging your estimate up to feel safer","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"A","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-3","original_id":"debias-3","prompt":"A vendor opens negotiations by quoting $500,000 for an annual software license. You know from public pricing pages and two competing quotes that fair value for this product at your usage tier is roughly $120,000/year. You need to set your counteroffer. Which counter best reflects the actual value rather than the vendor's opening number?\nA. $400,000, a modest discount off their quote\nB. $310,000, roughly the midpoint between their quote and your research\nC. $250,000, half of their opening figure\nD. $115,000, just under your researched fair-value estimate\nE. $350,000, to leave room to negotiate up","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"D","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-4","original_id":"debias-4","prompt":"In a workshop, a facilitator asks the group, 'Do you think the new feature will increase weekly active users by more or less than 40%?' and then asks each person to write down their point estimate. Your own analysis of historical feature launches suggests typical lifts of 3-6% for changes of this scope. What point estimate should you write down?\nA. 35%, slightly below the facilitator's number\nB. 40%, matching the figure given\nC. 20%, a cautious halving of the facilitator's number\nD. 5%, consistent with historical launch data\nE. 25%, to account for this feature being more impactful","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"D","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-5","original_id":"debias-5","prompt":"A used-car seller lists a sedan for $28,000. Kelley Blue Book and three recent local sales of the identical model, year, and mileage band all indicate a fair price near $19,500. The car has no special features that would justify a premium. You decide what to offer first. Which opening offer is most grounded in the car's actual value?\nA. $26,000, a small concession from asking\nB. $24,000, meeting roughly in the middle\nC. $22,000, a firm but respectful number near asking\nD. $19,000, just under the documented fair value\nE. $23,750, the midpoint between asking and book value","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"D","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-6","original_id":"debias-6","prompt":"A disease affects 1 in 1,000 people. A screening test has a 95% true-positive rate (it correctly flags 95% of people who have the disease) and a 5% false-positive rate (it wrongly flags 5% of healthy people). A randomly screened person with no symptoms tests positive. What is the approximate probability that this person actually has the disease?\nA. About 2%, because the disease is rare and most positives are false alarms\nB. About 90%, accounting for some error\nC. About 95%, since the test is 95% accurate\nD. About 50%, since the test could go either way\nE. About 5%, equal to the false-positive rate","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"A","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-7","original_id":"debias-7","prompt":"At your company, 8% of engineering candidates who reach the final round are eventually rated 'outstanding' after a year on the job. A new structured interview correctly identifies 80% of future-outstanding hires as 'strong yes' and also gives a 'strong yes' to 30% of those who will not become outstanding. A candidate gets a 'strong yes.' Roughly what is the probability this candidate becomes outstanding?\nA. About 19%, since most 'strong yes' candidates come from the large non-outstanding pool\nB. About 80%, matching the interview's hit rate\nC. About 70%, allowing for some error\nD. About 50%\nE. About 8%, the base rate is all that matters","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"A","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-8","original_id":"debias-8","prompt":"You read that 70% of startups in a particular accelerator fail within three years. A founder pitches you; she is articulate, has a polished deck, and reminds you strongly of a successful entrepreneur you admire. Setting aside how impressive she seems in the pitch, what is the most reasonable prior probability that her startup survives three years?\nA. Around 70%, because she seems exceptional\nB. Around 85%, given how much she resembles a known success\nC. Around 30%, the historical survival rate for this population\nD. Around 50%, a neutral coin flip\nE. Around 95%, since strong founders usually make it","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"C","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-9","original_id":"debias-9","prompt":"A fraud-detection model flags transactions as fraudulent. In reality only 0.2% of transactions are fraudulent. The model catches 99% of true fraud but also flags 1% of legitimate transactions. A transaction is flagged. What is the approximate chance it is actually fraudulent?\nA. About 99%, the model's detection rate\nB. About 17%, because legitimate transactions vastly outnumber fraudulent ones\nC. About 80%\nD. About 1%, the false-positive rate\nE. About 50%","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"B","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-10","original_id":"debias-10","prompt":"In a city, 85% of taxis are Green and 15% are Blue. A witness to a nighttime hit-and-run says the cab was Blue. Under similar nighttime conditions, the witness correctly identifies a cab's color 80% of the time and is wrong 20% of the time. Given the witness said 'Blue,' what is the approximate probability the cab was actually Blue?\nA. About 80%, the witness's reliability\nB. About 15%, just the base rate of Blue cabs\nC. About 41%, balancing the witness's accuracy against the rarity of Blue cabs\nD. About 70%\nE. About 95%, since eyewitnesses are usually trusted","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"C","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-11","original_id":"debias-11","prompt":"Your team has spent $2M and 18 months building a custom in-house analytics engine. A new, mature open-source tool now does everything you need and would cost about $50K/year to operate, versus an estimated $1.5M more to finish your in-house engine and $400K/year to maintain it afterward. Performance and features would be comparable. Purely on a forward-looking basis, what should you do?\nA. Finish the in-house engine; you've already invested $2M and can't waste it\nB. Finish it because switching now would mean the 18 months were for nothing\nC. Spend another $1.5M to finish, then reevaluate against the open-source tool\nD. Keep both running in parallel indefinitely\nE. Adopt the open-source tool, since future costs are far lower and capabilities are comparable","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"E","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-13","original_id":"debias-13","prompt":"A product feature has consumed three quarters of engineering time. Current data shows it has near-zero adoption and is unlikely to improve; the team projects that one more quarter of work has only a 5% chance of changing the outcome, and that engineering time could instead ship a feature with strong demand signals. What is the best decision going forward?\nA. Continue the feature; abandoning it wastes three quarters of work\nB. Continue one more quarter because the team has built up expertise in it\nC. Continue, but cut the team in half to limit further loss\nD. Continue because executives originally championed it\nE. Sunset the feature and redirect engineers to the high-demand work","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"E","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-15","original_id":"debias-15","prompt":"You have invested $40,000 into a side business over two years. It has never been profitable, demand has steadily declined, and your sober projection is continued losses with no realistic path to break-even. A clear-eyed analyst with no stake would advise shutting it down. What is the rational course?\nA. Keep going until you recoup the $40,000\nB. Invest a final $10,000 to give it one last chance to justify the prior spend\nC. Shut it down, since the $40,000 is gone regardless and future prospects are poor\nD. Keep it open at a loss to avoid admitting the venture failed\nE. Keep going because quitting would mean two years were wasted","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"C","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-16","original_id":"debias-16","prompt":"Linda is 31, single, outspoken, and very bright. She majored in philosophy. As a student she was deeply concerned with discrimination and social justice and participated in anti-nuclear demonstrations. Which is more probable?\nA. Linda is a bank teller and is active in the feminist movement\nB. Linda is a bank teller\nC. The two are equally probable\nD. It is impossible to compare the two\nE. Linda is neither","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"B","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-17","original_id":"debias-17","prompt":"A software outage occurs. Which single statement is most probable?\nA. The outage was caused by a misconfigured load balancer during last night's deployment\nB. The outage was caused by a misconfiguration\nC. The outage was caused by a misconfigured load balancer\nD. The outage was caused by a misconfigured load balancer that the on-call engineer introduced while rushing\nE. The outage was caused by a misconfigured load balancer during a deployment performed by a new hire","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"B","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-18","original_id":"debias-18","prompt":"Mark is a meticulous, detail-obsessed person who loves spreadsheets and reads tax law for fun. Considering the general population, which statement is most probable?\nA. Mark is an accountant who plays in a jazz band on weekends\nB. Mark is an accountant\nC. Mark is an accountant and owns more than three calculators\nD. Mark is an accountant who once audited a Fortune 500 company\nE. Mark is a meticulous accountant who hates surprises","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"B","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-19","original_id":"debias-19","prompt":"A forecaster is asked about next year's economy. Which forecast is most likely to come true?\nA. The central bank will cut interest rates\nB. The central bank will cut interest rates and the stock market will rise\nC. The central bank will cut interest rates, the market will rise, and unemployment will fall below 4%\nD. The central bank will cut interest rates in response to a recession\nE. The central bank will cut rates and inflation will fall below 2%","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"A","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-20","original_id":"debias-20","prompt":"Sarah read literature and traveled through Europe after college, and she once worked at an art gallery. From the perspective of pure probability, which is more likely to be true of Sarah today?\nA. Sarah works in finance and collects vintage wine\nB. Sarah works in finance\nC. Sarah works in finance and visits museums on vacation\nD. Sarah works in finance and subscribes to an art magazine\nE. Sarah works in finance and speaks two European languages","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"B","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-21","original_id":"debias-21","prompt":"Two surgeons present identical outcome data for the same operation. Surgeon A says, 'This procedure has a 90% survival rate.' Surgeon B says, 'This procedure has a 10% mortality rate.' A patient must choose based only on this information. What is the most rational interpretation?\nA. Surgeon A's procedure is safer\nB. Surgeon B's procedure is riskier\nC. Surgeon A's framing proves greater confidence\nD. Surgeon B is being dishonest by emphasizing death\nE. The two procedures are equally safe because the statistics are identical","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"E","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-22","original_id":"debias-22","prompt":"A public health program will affect 600 people. Plan A: '200 people will be saved.' Plan B: '400 people will die.' Both plans are described as applying to the same 600 people with certainty. Which choice is rational?\nA. Choose Plan A, since saving lives sounds better\nB. Choose Plan B, since avoiding death sounds more urgent\nC. Choose Plan A because it mentions a smaller number\nD. Choose Plan B because 400 is a bigger commitment\nE. Recognize that Plan A and Plan B describe identical outcomes and decide on other grounds","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"E","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-23","original_id":"debias-23","prompt":"A grocery store advertises ground beef as '80% lean.' A competitor advertises the identical product as '20% fat.' A shopper comparing only these two labels wants the leaner, healthier choice. What should the shopper conclude?\nA. The '80% lean' beef is leaner and healthier\nB. The '20% fat' beef is fattier and worse\nC. The '80% lean' label indicates a higher-quality cut\nD. The '20% fat' beef is cheaper because it admits the fat\nE. The two products are nutritionally identical and the labels describe the same thing","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"E","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-24","original_id":"debias-24","prompt":"An investment brochure for two funds with identical net performance describes Fund X as 'capturing 95% of market upside' and Fund Y as 'exposed to 5% of potential shortfall avoidance loss'—wording aside, both funds returned exactly the same amount over ten years with the same volatility. An investor must pick one. What is the rational basis for the decision?\nA. Pick Fund X; its description sounds more profitable\nB. Pick Fund Y; cautious language signals prudence\nC. Treat them as equivalent and decide on fees or other real differences\nD. Pick Fund X because 95% is a larger number\nE. Pick Fund Y because it acknowledges risk","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"C","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-25","original_id":"debias-25","prompt":"A credit card company offers a '2% cash-back reward' on purchases. A bank instead charges a '2% surcharge for not using their card' but otherwise the net effect on your wallet for the same spending is identical. Ignoring branding, how should a rational consumer treat these two offers?\nA. Prefer the cash-back card; rewards are better than penalties\nB. Avoid the surcharge bank; penalties are bad\nC. Recognize the net financial effect is the same and choose on other terms like APR or fees\nD. Prefer the cash-back card because gains feel better than losses\nE. Avoid the surcharge bank because surcharges compound","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"C","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-26","original_id":"debias-26","prompt":"You believe a particular caching change caused a latency regression. To test this hypothesis as rigorously as possible, which investigation should you prioritize first?\nA. Pull all log lines from after the change that show high latency, confirming the regression\nB. Collect metrics from before and after the change and check whether latency actually differs, including looking for cases where it did not regress\nC. Ask teammates who also suspected the cache to corroborate your view\nD. Re-read the cache pull request looking for anything that supports your theory\nE. Search the issue tracker for past cache-related bugs to build the case","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"B","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-27","original_id":"debias-27","prompt":"You are convinced a job candidate is excellent after a great first impression. To evaluate the hire decision in the least biased way, what should you do?\nA. Deliberately probe for weaknesses and seek disconfirming evidence, weighing it equally with the positives\nB. Ask interview questions designed to let the candidate shine\nC. Focus the reference check on confirming the strengths you observed\nD. Compare the candidate only against weaker applicants\nE. Trust your gut since first impressions are usually right","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"A","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-29","original_id":"debias-29","prompt":"An analyst expects a new marketing channel to be highly effective and gathers data to evaluate it. Which approach gives the most trustworthy assessment of the channel's true performance?\nA. Define success metrics and a holdout/control group in advance, then evaluate against all the data including weeks it underperformed\nB. Highlight the weeks where the channel performed well in the report\nC. Compare the channel only to the worst-performing existing channel\nD. Survey the team that proposed the channel for their impressions\nE. Drop the early data when the channel was still ramping up","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"A","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-30","original_id":"debias-30","prompt":"You suspect a specific microservice is the source of intermittent errors. A colleague suggests you might be wrong. What is the most rigorous next step?\nA. Grep the suspected service's logs for any error and treat each as proof\nB. Disable the suspected service and declare it the cause if anything improves\nC. Ask the team that owns the suspected service whether they think it's at fault\nD. Instrument and compare error rates across all services in the request path, actively trying to rule your suspect out\nE. Re-deploy the suspected service and watch only its dashboard","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"D","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-31","original_id":"debias-31","prompt":"After seeing extensive news coverage of two airplane crashes this month, a traveler is deciding whether to drive or fly 1,000 miles. Per mile traveled, statistics consistently show commercial flying has a far lower fatality rate than driving. Which conclusion should guide the decision?\nA. Drive, because plane crashes are clearly common right now\nB. Drive, because the recent news shows flying is unusually dangerous lately\nC. Fly, because per-mile fatality data show it is safer despite the vivid coverage\nD. Drive, because you have more control behind the wheel\nE. Flip a coin, since both feel equally risky after the news","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"C","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-32","original_id":"debias-32","prompt":"A manager just dealt with a dramatic security incident caused by a phishing email. In the next planning cycle, she must allocate a fixed security budget across threats. An objective risk assessment ranks misconfigured cloud storage as the company's highest expected-loss threat, with phishing third. How should she allocate?\nA. Put the majority of the budget into anti-phishing because the recent incident proves it's the top risk\nB. Allocate according to the objective expected-loss ranking, addressing cloud misconfiguration first\nC. Split evenly across all threats to be safe\nD. Defer all spending until another incident reveals the real priority\nE. Double the phishing budget every time an incident occurs","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"B","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-33","original_id":"debias-33","prompt":"A friend asks which is a more common cause of death in the United States: shark attacks or falls from ladders. Shark attacks dominate headlines and movies, while ladder falls rarely make the news. What is the accurate answer?\nA. Falls from ladders, which kill vastly more people annually despite little coverage\nB. They are roughly equal\nC. Shark attacks, because they are far more frequently reported\nD. Shark attacks, because they are more dramatic and memorable\nE. Impossible to know without specific yearly figures","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"A","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-34","original_id":"debias-34","prompt":"An investor read three recent viral stories about people who got rich quickly on a speculative crypto token. Broad, long-run data on such speculative tokens show the large majority lose most of their value and few holders profit. The investor is deciding whether to put 30% of savings into the token. What is the soundest reasoning?\nA. Invest, because the success stories show it clearly works\nB. Invest a large amount, since the wins are fresh and frequent in the news\nC. Invest everything to maximize the obvious upside\nD. Wait for one more success story before deciding\nE. Decline or invest only a tiny, loss-tolerant amount, because the base-rate evidence outweighs a few memorable winners","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"E","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-35","original_id":"debias-35","prompt":"A homeowner who just watched coverage of a rare house fire wants to spend heavily on fire prevention while ignoring routine roof maintenance. Actuarial data for her area show water damage from roof leaks is many times more likely and costly than fire. Where should she focus her limited home-maintenance budget?\nA. Fire prevention, because the news made fire feel imminent\nB. Roof maintenance, because the data show it is the more probable and costly risk\nC. Split the budget equally between fire and roof\nD. Fire prevention, since fires are more catastrophic when they happen\nE. Neither, until she sees a local fire herself","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"B","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-36","original_id":"debias-36","prompt":"A senior engineer is 99% sure a one-line change is safe and wants to push it straight to production on a Friday afternoon without tests or review, citing past confidence. Across the team's history, changes engineers were '99% sure' about still caused incidents about 8% of the time. Given the asymmetric cost of a weekend outage, what is the wisest course?\nA. Run the test suite and get a quick review before deploying, since stated 99% confidence has historically been wrong far more often\nB. Push it because the engineer is senior and rarely wrong\nC. Push directly to production; 99% confidence is more than enough\nD. Push, but watch the dashboard for a few minutes afterward\nE. Push it because reverting is always possible","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"A","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-37","original_id":"debias-37","prompt":"You are asked to give a 90% confidence interval for the year the first commercial transatlantic telegraph cable carried messages. You feel fairly knowledgeable and are tempted to give a narrow range like 1900-1910. Calibration studies show people's '90%' intervals contain the true value only about 50% of the time because they make them too narrow. To be properly calibrated, what should you do?\nA. Give a tight range like 1900-1905 to look knowledgeable\nB. Widen the interval substantially, e.g., 1840-1920, to genuinely capture 90% of your uncertainty\nC. Give a single best-guess year instead of a range\nD. Refuse to answer because you're not certain\nE. Narrow the range further since precision signals expertise","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"B","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-38","original_id":"debias-38","prompt":"A product lead is certain a launch will hit 1 million users in month one and wants to commit infrastructure spend and public promises on that single number. No comparable past launch exceeded 200,000, and the team has no validated demand signal. What planning approach is most rational?\nA. Commit fully to the 1M figure; conviction drives results\nB. Set the public target even higher to motivate the team\nC. Spend for 1M users now to avoid being caught short\nD. Plan around a range informed by comparable launches and demand evidence, and avoid hard public commitments to a single optimistic number\nE. Trust the lead's certainty since they know the product best","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"D","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-39","original_id":"debias-39","prompt":"A trader has a string of recent winning bets and now feels certain his next large position cannot lose, so he wants to double his usual size and skip his stop-loss. His own long-run track record shows he is right about 55% of the time, and his sense of certainty has not historically predicted outcomes. What is the disciplined choice?\nA. Double the position and skip the stop-loss; he's clearly on a hot streak\nB. Triple the position to capitalize on the certainty\nC. Bet everything because he can't lose right now\nD. Keep normal position sizing and his stop-loss, since his felt certainty doesn't change his real 55% edge\nE. Skip the stop-loss only, but keep normal size","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"D","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"debias-40","original_id":"debias-40","prompt":"During a project kickoff, the team is asked to estimate the delivery date and everyone agrees they are highly confident it will ship in exactly 8 weeks, with no slack. Historically, similar projects by this team shipped 40-100% later than their confident initial estimates due to unforeseen issues. What is the best way to set the committed date?\nA. Commit to 8 weeks; the team is confident and aligned\nB. Commit to 7 weeks to push the team to perform\nC. Set the committed date using the team's historical track record of overruns, e.g., 11-16 weeks with buffer\nD. Refuse to give any date until everything is certain\nE. Commit to 8 weeks but tell stakeholders it might slip a day or two","source":"debias-authored","source_file":"external/debias-authored.jsonl","skill_fit":["debiasing"],"family":"Conceptual reframes","type":"target","target":true,"answer_idx":"C","mode":"correctness","decision_instruction":"Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-01","original_id":"cynefin-01","prompt":"User engagement has been declining 8% month-over-month for the past three months across every surface in the product. Every team has a different theory — onboarding friction, notification fatigue, a competitor launch, a content quality drop — but none of the theories fully fit the data. No one has been able to isolate a single cause.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":true,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-02","original_id":"cynefin-02","prompt":"We launched a new pricing model six weeks ago and revenue is up but customer satisfaction scores dropped sharply. Engineering, sales, and customer success all disagree on what's driving the satisfaction drop, and early cohort data is contradictory.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":true,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-04","original_id":"cynefin-04","prompt":"We're trying to figure out what kind of product a brand-new demographic of users (Gen Z creators) would actually pay for. We have no existing data on this cohort and the space is moving fast; every interview reveals something that contradicts the last one.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":true,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-06","original_id":"cynefin-06","prompt":"Our autonomous agent is producing correct answers in unit tests but takes increasingly wrong actions in real multi-step user sessions. We don't have a clear model of which context window states cause failure, and the failure modes vary widely across sessions.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":true,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-08","original_id":"cynefin-08","prompt":"After a dependency upgrade three weeks ago, a subset of users on older Android devices began reporting random crashes at inconsistent points in the session. Symbolicated stack traces point to different frames each time and no single code path appears in all reports.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":true,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-09","original_id":"cynefin-09","prompt":"We're designing the team structure for a 0-to-1 product where the scope, tech stack, and target customer are still evolving rapidly. Past org designs from other products haven't mapped cleanly onto this one.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":true,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-10","original_id":"cynefin-10","prompt":"A security incident response is underway: an attacker is actively moving laterally through our infrastructure and the blast radius is still unknown. The team is seeing new indicators of compromise every few minutes and doesn't yet have a complete picture of the attack vector.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":true,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-11","original_id":"cynefin-11","prompt":"Developer productivity has dropped significantly since we adopted a microservices architecture 18 months ago. Teams point to different root causes — too many services, unclear ownership, slow local dev setup, too many meetings — and no single change has moved the needle so far.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":true,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-12","original_id":"cynefin-12","prompt":"A B2B enterprise customer is churning at a higher rate than our other segments, but their NPS scores and support ticket volume look fine. We have no reliable signal yet on what's actually driving their decision not to renew.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":true,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-13","original_id":"cynefin-13","prompt":"A prompt change that improved benchmark scores by 4% caused a noticeable quality degradation in a small but vocal subset of production use cases. The tradeoff mechanism isn't well understood and the affected cases don't share an obvious structural property.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":true,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-14","original_id":"cynefin-14","prompt":"We rolled out a team-based onboarding flow two months ago. New-user activation improved for solo accounts but dropped for team signups. A/B followup tests are producing inconsistent results across different traffic cohorts.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":true,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-15","original_id":"cynefin-15","prompt":"Infrastructure costs tripled in the past quarter. The spike doesn't trace cleanly to any single service, new feature launch, or traffic increase — multiple teams have looked and offered conflicting explanations.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":true,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-16","original_id":"cynefin-16","prompt":"Our nightly batch job started failing with a deterministic NullPointerException after the last deploy. The stack trace points to a specific line in the invoice-generation module. Find and fix the null-dereference.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":false,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-17","original_id":"cynefin-17","prompt":"Migrate the PostgreSQL 12 database to PostgreSQL 15 following the official upgrade path. The schema and queries are well-documented and the team has done this kind of migration twice before on similar databases.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":false,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-19","original_id":"cynefin-19","prompt":"Add TLS 1.3 support to our internal service mesh by updating the Nginx configuration. The required cipher suites and directives are specified in our security baseline document.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":false,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-20","original_id":"cynefin-20","prompt":"The `orders` table is experiencing slow reads because it lacks an index on `customer_id`. Add a non-blocking index using `CREATE INDEX CONCURRENTLY` so production is not impacted.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":false,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-21","original_id":"cynefin-21","prompt":"We need to implement WCAG 2.1 AA compliance for our web app. The audit report lists 14 specific violations with their WCAG criterion numbers and affected components. Fix each listed violation.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":false,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-22","original_id":"cynefin-22","prompt":"Our JWT tokens are not expiring correctly because the `exp` claim is being set in milliseconds instead of seconds. Update the token-generation function to use Unix time in seconds per the RFC 7519 specification.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":false,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-23","original_id":"cynefin-23","prompt":"Set up automated daily backups of our production PostgreSQL database to S3. Use pg_dump, compress with gzip, and retain backups for 30 days. This is a standard operational task we do for all new databases.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":false,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-24","original_id":"cynefin-24","prompt":"Upgrade the React version in the project from 17 to 18. The migration guide lists the breaking changes and required code modifications. No custom renderer or concurrent mode features are in use.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":false,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-25","original_id":"cynefin-25","prompt":"Our email delivery rate dropped after our sending IP was added to a known spam blocklist. Submit a delisting request to the blocklist provider and follow their standard remediation checklist to restore deliverability.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":false,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-26","original_id":"cynefin-26","prompt":"Implement pagination on the `/api/products` endpoint. It currently returns all records. Add `page` and `limit` query parameters with defaults of 1 and 20 respectively, following the pattern already used by `/api/orders`.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":false,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-27","original_id":"cynefin-27","prompt":"Our code coverage dropped below the 80% threshold when the payment module was added. Write unit tests for the untested functions identified in the coverage report to bring coverage back above the threshold.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":false,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-28","original_id":"cynefin-28","prompt":"The build is failing because a transitive dependency requires Node 18 but our CI environment is pinned to Node 16. Update the CI configuration to use Node 18.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":false,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-29","original_id":"cynefin-29","prompt":"Internationalize the date format strings in the app's notification templates. Replace all hardcoded `MM/DD/YYYY` formats with the `Intl.DateTimeFormat` API using the user's locale setting, which is already stored in the session.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":false,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"cynefin-30","original_id":"cynefin-30","prompt":"The weekly sales report PDF is generating with incorrect totals because a currency-conversion step was accidentally removed during a merge. Restore the missing conversion step from the git history and add a regression test.","source":"cynefin-classify","source_file":"authored/cynefin-classify.jsonl","skill_fit":["cynefin"],"family":"Conceptual reframes","type":"nearMiss","target":false,"label":false,"decision_instruction":"Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"glm-distractor-8","original_id":"glm-distractor-8","prompt":"How can I convert all the lowercase letters in a string to uppercase using Python?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-11","original_id":"glm-distractor-11","prompt":"**This Kata is intended as a small challenge for my students** All Star Code Challenge #18 Create a function called that accepts 2 string arguments and returns an integer of the count of occurrences the 2nd argument is found in the first one. If no occurrences can be found, a count of 0 should be returned. Notes: * The first argument can be an empty string * The second string argument will always be of length 1 Write Python code to solve the problem. Please place the solution code in the following format: ```python # Your solution code here ```","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-12","original_id":"glm-distractor-12","prompt":"Implement quick sort to sort the given array of integers. Write corresponding code in Python. [9, 5, 8, 1, 3, 6, 7]","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-13","original_id":"glm-distractor-13","prompt":"Zoe wants to go on the field trip to Washington DC with her middle school this spring and the cost is $485. Her grandma gave her $250 toward her fees and she must earn the rest by selling candy bars. She makes $1.25 for every candy bar she sells. How many candy bars does Zoe need to sell to earn the money for the trip?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-16","original_id":"glm-distractor-16","prompt":"Can you also tell me the date and time of the Indiana Comic Convention?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-17","original_id":"glm-distractor-17","prompt":"Create an algorithm to rotate an array by one element. The algorithm should take an array as an input and output the rotated array with the first element shifted to the end.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-18","original_id":"glm-distractor-18","prompt":"Generate a correct Python program for plotting bar chart for the following data.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-19","original_id":"glm-distractor-19","prompt":"Given a non-negative decimal integer $x$, convert it to binary representation $b$ of 32 bits. Then, print the result of the following operations to $b$ respecitvely. * Inversion: change the state of each bit to the opposite state * Logical left shift: shift left by 1 * Logical right shift: shift right by 1 Constraints * $0 \\leq x \\leq 2^{32} - 1$ Input The input is given in the following format. $x$ Output Print the given bits, results of inversion, left shift and right shift in a line respectively. Examples Input 8 Output 00000000000000000000000000001000 11111111111111111111111111110111 00000000000000000000000000010000 00000000000000000000000000000100 Input 13 Output 00000000000000000000000000001101 11111111111111111111111111110010 00000000000000000000000000011010 00000000000000000000000000000110 Write Python code to solve the problem. Please place the solution code in the following format: ```python # Your solution code here ```","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-21","original_id":"glm-distractor-21","prompt":"Create a Python program that fetches the user's current time and sets a reminder after 30 minutes.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-22","original_id":"glm-distractor-22","prompt":"Could you delineate a methodology, utilising the Python programming language, to extract the middle value, labeled as the median, from an unorganized collection of integral numbers? Numbers = [12, 7, 21, 3, 9]","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-23","original_id":"glm-distractor-23","prompt":"Create a Python 3 class for a directory list widget using Qt that includes a grid layout with a directory list view and buttons to add or remove directories. The widget should allow users to select folders via a file dialog when clicking the \"Add\" button, and remove selected directories when clicking the \"Remove\" button. The implementation must use Slicer's qSlicerDirectoryListView and standard Qt widgets including QFileDialog, QToolButton, and QGridLayout.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-42","original_id":"glm-distractor-42","prompt":"Write a Python3 function that returns the Cargo home directory path by first checking the CARGO_HOME environment variable. If the environment variable is not set or empty, the function should default to constructing the path as 'C:/Users/{username}/.cargo' using the current user's username. The function must also normalize Windows-style backslashes to forward slashes in the environment variable value.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-46","original_id":"glm-distractor-46","prompt":"Create a Python 3 script that uses the clustergrammer library to load a matrix from stdin, perform optional filtering and normalization steps (such as sum-based filtering, z-score normalization, top-N selection, and threshold filtering), generate a hierarchical clustering visualization using cosine distance, and then output the resulting network data in JSON format for frontend visualization. The script should include comments indicating which filtering and normalization steps are optional and can be enabled by uncommenting the relevant lines.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-48","original_id":"glm-distractor-48","prompt":"Solve the inequality: 8x - 9 < 2. Express the solution in interval notation.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-50","original_id":"glm-distractor-50","prompt":"A toy store manager received a large order of Mr. Slinkums just in time for the holidays. The manager places $20\\%$ of them on the shelves, leaving the other 120 Mr. Slinkums in storage. How many Mr. Slinkums were in this order?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-51","original_id":"glm-distractor-51","prompt":"Write a python function to check if a given file is in NOAA ACE format by verifying that the first line starts with ':' and the third line contains 'NOAA'. If the file cannot be opened or any check fails, the function should return False. Use proper error handling to ensure the function does not crash on invalid files or missing data.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-69","original_id":"glm-distractor-69","prompt":"Write a program that allows users to search for phone numbers by name in a phone book. Implement the phone book using a custom data structure that you create. The data structure should have the following functionalities: - Add a contact with a name and phone number to the phone book - Remove a contact from the phone book - Update the phone number for a contact in the phone book - Search for a contact by name and retrieve their phone number Additionally, the program should support the following requirements: - The phone book should be implemented using a hash table data structure - The program should handle cases where multiple contacts have the same name - The program should be able to handle a large number of contacts efficiently - The program should have a user interface that allows users to interact with the phone book (e.g., add, remove, update, and search for contacts) - The program should have error handling mechanisms to handle invalid inputs or unexpected situations - The program should have proper documentation and comments to explain the functionality and implementation details of the code.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-74","original_id":"glm-distractor-74","prompt":"Produce a program using Python that visualizes the given function.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-81","original_id":"glm-distractor-81","prompt":"How can I generate a regular expression that matches any two or more consecutive numbers?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-83","original_id":"glm-distractor-83","prompt":"Write a python 3 script that performs simple linear regression analysis on gas expenditure data using scikit-learn. The script should load data from \"data1.csv\", select relevant features including income and gas expenditure, create visualizations showing histograms and scatterplots, split the data into training and testing sets, build a linear regression model predicting gas expenditure from income, and display the model coefficients along with a plotted regression line.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-84","original_id":"glm-distractor-84","prompt":"Write a python 3 Django management command that classifies unassigned faces using ensemble predictions from two SVM models (small and big) with probability averaging, and prints classification results for the first 1000 unassigned faces, including predicted labels, probabilities, and corresponding person objects from the database.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-196","original_id":"glm-distractor-196","prompt":"Create a python 3 script that demonstrates two different plotting APIs using PyQt4: a procedural approach with PlotWidget and an object-oriented approach with a custom Plot class. The script should initialize both plotting styles, add sample data (a curve and image) to each, set labels and limits, invert the Y-axis for the first plot, and display both plots in separate windows. Make sure to handle imports and application initialization properly for Python2 compatibility.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-199","original_id":"glm-distractor-199","prompt":"Write a Python 3 function that tests whether the neohub class can be properly instantiated, ensuring the class initialization works correctly without any errors. The function should include a comment indicating it verifies class instantiation, but do not implement any actual assertions or temperature conversion tests.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-201","original_id":"glm-distractor-201","prompt":"Can you provide a Python code snippet to convert XML to JSON?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-203","original_id":"glm-distractor-203","prompt":"Write a Python3 script that downloads and unpacks tracking results from Google Drive for various trackers like PyTracking-based and external trackers. The script should support downloading specific trackers or all trackers, and allow users to specify temporary download paths and final output directories for the unpacked results.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-214","original_id":"glm-distractor-214","prompt":"Create a python 3 program using the turtle module that draws a series of nested squares rotating at different angles. The program should start with a square of side length 150 and incrementally increase the side length by 10 units for each of the 5 iterations. Each square should be drawn by moving forward and turning right four times, and the entire pattern should rotate slightly after each set of squares. Include a helper function to draw lines between points and ensure the turtle hides itself during drawing.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-216","original_id":"glm-distractor-216","prompt":"How can I generate a set of random numbers without any duplicates in Python?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-224","original_id":"glm-distractor-224","prompt":"Create a Python script that sets the value of a variable from the command-line, but the value must be encrypted using a custom encryption algorithm before being assigned to the variable. Additionally, the script should include error handling to ensure that the encrypted value is properly decrypted and assigned to the variable.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-225","original_id":"glm-distractor-225","prompt":"A satellite orbits Earth at an altitude of 1501 km. Calculate its orbital velocity and period. (Earth mass = 5.97 × 10²⁴ kg, Earth radius = 6371 km)","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-227","original_id":"glm-distractor-227","prompt":"Compute the quotient of 17708 and 9880. Show your work.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-236","original_id":"glm-distractor-236","prompt":"def division_while_loop(dividend, divisor): \"\"\" This exercise focuses on division using a while loop. Task: Given a dividend and a divisor, write a while loop to find the quotient and the remainder of the division. Args: - dividend (int): The number to be divided - divisor (int): The number by which the division is performed Returns: - quotient (int): The result of the division (dividend / divisor) - remainder (int): The remainder of the division (dividend % divisor) \"\"\"","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-241","original_id":"glm-distractor-241","prompt":"Create a python 3 class called `BalanceSheet` that takes a `Portfolio` object (with default to a new `Portfolio()` instance) and provides methods to generate structured data representations. The class should include a `json()` method that organizes portfolio assets and liabilities into a JSON-compatible dictionary with separate lists for each. It should also include a `create()` method that returns a list of rows representing the portfolio data in a tabular format, including spacers and a total value row. Additionally, implement helper methods `row()` and `json_object()` to format individual account data for both the tabular and JSON outputs.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-249","original_id":"glm-distractor-249","prompt":"Write a python function to find the frequency of each character in a given string and return a dictionary with characters as keys and their frequencies as values. Solve the problem starting with the provided function header. Function header: ``` def get_char_frequency(string): ``` Please place the solution code in the following format: ```python # Your solution code here ```","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-261","original_id":"glm-distractor-261","prompt":"Create a function to check if an array contains any duplicate elements, while also ensuring that the array elements are sorted in ascending order and that the length of the array is less than or equal to 100. Additionally, the function should check if the array contains any negative integers, and if so, it should return False. The function should return a boolean value indicating the presence of duplicate elements. Example: Input: [1, 2, 3, 3, 4, 5, 5] Output: True","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-262","original_id":"glm-distractor-262","prompt":"You are working on an API for a space-themed game and need to create a class that handles the case when a specific planet for a character is not found. You need to define a class called GetCharactersCharacterIdPlanetsPlanetIdNotFound with the following attributes and methods: Attributes: - error: a string to store the error message Methods: - __init__: a constructor method to initialize the error attribute - to_dict: a method to return the model properties as a dictionary - to_str: a method to return the string representation of the model - __repr__: a method for printing and pretty-printing the object - __eq__: a method to check if two objects are equal - __ne__: a method to check if two objects are not equal You need to implement the class and its methods as described above.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-264","original_id":"glm-distractor-264","prompt":"How can I categorize data into two bins, with the labels \"category A\" and \"category B\"?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-267","original_id":"glm-distractor-267","prompt":"Given points A(14, -8) and B(-1, -13), find the slope of the line passing through these two points.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-268","original_id":"glm-distractor-268","prompt":"Create a python 3 class named Objects that initializes with several instance attributes including cloud set to None, score set to 0, score_type set to 'CNNSeg', type set to None, and type_probs set to None.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-269","original_id":"glm-distractor-269","prompt":"What is the code to construct a Recurrent Neural Network (RNN) using TensorFlow with an input size of 50 values and an output size of 10 values?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-270","original_id":"glm-distractor-270","prompt":"Create a python 3 class called CompileGLSL that extends Module for compiling GLSL shader files into SPIR-V format using the glslangValidator tool. The class should define required tools, specify output file names with .spv extension, and implement a process method that runs the validator with appropriate stage flags (-S vert for vertex shaders and -S frag for fragment shaders) while handling directory creation and returning the subprocess return code.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-271","original_id":"glm-distractor-271","prompt":"Evaluate the definite integral of f(x) = sin(1x) from 1 to 4.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-274","original_id":"glm-distractor-274","prompt":"Calculate the pH of a 0.545 M solution of HCl. Show your calculation step by step.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-275","original_id":"glm-distractor-275","prompt":"Write a Python 3 program that prompts the user to enter three integer numbers and then calculates and displays their sum. The program should use input() to read the numbers from the user and print the result in the format \"Sum = [value]\".","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Conceptual reframes","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}+{"id": "first-principles-01", "original_id": "first-principles-01", "prompt": "A distributed system team says they cannot have both strong consistency and 100% availability during a network partition. They cite the CAP theorem as the reason and are designing around this trade-off.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-02", "original_id": "first-principles-02", "prompt": "A backend team insists they cannot switch from REST to gRPC for internal service communication because 'we've always used REST and all our tooling is built around it.' The gRPC library is available in their language stack and the team hasn't evaluated it.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-03", "original_id": "first-principles-03", "prompt": "A healthcare startup must store patient records exclusively within the EU because a government regulation mandates data residency for citizens' medical data within the country of origin. Legal counsel has confirmed this is a binding statutory requirement.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-04", "original_id": "first-principles-04", "prompt": "A software team says they can't add a dark mode to their web app because 'the design system doesn't support it.' On investigation, the design system uses CSS custom properties for colors and simply hasn't been extended \u2014 no technical barrier prevents adding a second theme.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-05", "original_id": "first-principles-05", "prompt": "An infrastructure architect explains that two datacenters on opposite sides of the planet have an irreducible minimum round-trip latency of roughly 130ms due to the speed of light through fiber. No amount of software optimization can eliminate this floor.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-06", "original_id": "first-principles-06", "prompt": "An engineering team says they can't break up their monolith into separate services because 'that's how the legacy system was built and changing it would break everything.' The codebase has clear module boundaries and no architectural reason prevents decomposition.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-07", "original_id": "first-principles-07", "prompt": "A cryptography engineer explains that breaking a 256-bit AES key by brute force would require more computational energy than exists in the observable universe given current physics. This is why the encryption is treated as unbreakable for practical purposes.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-08", "original_id": "first-principles-08", "prompt": "A team lead says they can't use snake_case for new database columns because 'our naming convention is camelCase.' This is an internal style guide documented in a wiki page that the team wrote themselves and has never been reviewed for technical necessity.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-09", "original_id": "first-principles-09", "prompt": "A financial platform is told it cannot process credit card transactions without being PCI-DSS certified. This is a contractual and legal requirement from the card networks \u2014 Visa, Mastercard \u2014 that applies to any entity storing, processing, or transmitting cardholder data.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-10", "original_id": "first-principles-10", "prompt": "A product team claims they can't add real-time collaborative editing to their document tool because 'our backend uses REST and REST can't do real-time.' WebSockets and server-sent events are viable and the team hasn't actually evaluated adding them.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-11", "original_id": "first-principles-11", "prompt": "A machine learning team notes that a model trained only on data from before a certain date genuinely cannot have learned about events that occurred afterward. The knowledge cutoff is an inherent result of when training data was collected.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-12", "original_id": "first-principles-12", "prompt": "A developer says the team can't write integration tests because 'we've never had them in this codebase and setting them up would take too long.' The services expose standard HTTP APIs and a test harness exists in adjacent projects the team hasn't looked at.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-13", "original_id": "first-principles-13", "prompt": "A chip design team explains they cannot make transistors smaller than a few atoms because quantum tunneling effects make the device physically non-functional at sub-atomic scales. This is a hard physical limit of semiconductor physics.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-14", "original_id": "first-principles-14", "prompt": "An API team says they can't version their API because 'that's not how we do things here.' Looking at the codebase, there are no technical barriers \u2014 the routing layer already supports path prefixes and adding `/v2/` would require minimal changes.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-15", "original_id": "first-principles-15", "prompt": "A data pipeline processes sensor readings and an engineer notes that you cannot have sub-millisecond latency between a sensor event and its reception on a server 500km away because the physical signal propagation time alone exceeds that bound.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-16", "original_id": "first-principles-16", "prompt": "An engineering manager says the team can't do continuous deployment because 'we've always done quarterly releases.' The CI/CD infrastructure is already in place and there are no contractual, regulatory, or technical blockers identified \u2014 just existing practice.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-17", "original_id": "first-principles-17", "prompt": "A distributed consensus algorithm requires a majority quorum of nodes to agree before committing a write. With 5 nodes, this means the system mathematically cannot tolerate more than 2 simultaneous failures and still make progress.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-18", "original_id": "first-principles-18", "prompt": "A team claims they can't use a monorepo because 'our Git workflow doesn't support it.' After digging in, the actual blocker is that their CI scripts use a hardcoded single-repo assumption \u2014 a self-imposed implementation detail that could be changed in a sprint.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-20", "original_id": "first-principles-20", "prompt": "A frontend developer insists they can't paginate the results of a data table because 'the current API returns all rows at once.' The API team is on the same squad and has capacity to add pagination query parameters \u2014 no external dependency is involved.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-21", "original_id": "first-principles-21", "prompt": "A compression algorithm engineer explains that you cannot losslessly compress a truly random bitstream to fewer bits than it contains \u2014 Shannon's source coding theorem sets a mathematical lower bound on entropy coding.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-22", "original_id": "first-principles-22", "prompt": "A platform team says they can't give engineers production database access because 'that's policy.' Investigation shows the policy was written by a former DBA who left two years ago and was never reviewed for necessity; no regulatory or audit requirement mandates it.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-23", "original_id": "first-principles-23", "prompt": "A GPU cluster running neural network training is limited by memory bandwidth \u2014 there is a physical maximum rate at which data can move between DRAM and the compute cores dictated by the memory bus architecture of the specific GPU model.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-24", "original_id": "first-principles-24", "prompt": "An analytics team says they can't use columnar storage for their reporting queries because 'the warehouse is Postgres.' Their cloud provider offers a Postgres-compatible columnar extension the team hasn't evaluated and that would work with their current ORM.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-25", "original_id": "first-principles-25", "prompt": "A European fintech company cannot share raw transaction data about EU residents with its US parent company's analytics team without explicit user consent or an approved legal mechanism, due to GDPR Article 44 restricting cross-border data transfers.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-26", "original_id": "first-principles-26", "prompt": "A developer says they can't add structured logging to a service because 'logging is done with print statements in this codebase.' There is no technical barrier \u2014 the language has a standard structured logging library and the other services already use it.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-27", "original_id": "first-principles-27", "prompt": "A distributed sort algorithm must move at minimum O(N log N) comparisons' worth of data to sort N arbitrary elements \u2014 this is a provably optimal lower bound for comparison-based sorting that no comparison sort can beat.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-28", "original_id": "first-principles-28", "prompt": "A product team says they can't run A/B tests because 'we don't have an experimentation platform.' Their analytics stack already captures user events, their feature flag system is in place, and three engineers estimate a basic split-test harness could be built in a week.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-29", "original_id": "first-principles-29", "prompt": "A real-time trading system must comply with SEC Rule 17a-4, which legally mandates that certain trade records be stored in a non-rewritable, non-erasable format for a minimum of six years. Brokers have no legal authority to waive this requirement.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "first-principles-30", "original_id": "first-principles-30", "prompt": "A mobile developer says they can't make their iOS app available on Android because 'we build in Swift.' Swift is iOS-specific, but the product logic has no hardware dependency and could be rewritten in a cross-platform framework \u2014 nothing physically prevents an Android port.", "source": "first-principles-constraint", "source_file": "authored/first-principles-constraint.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-01", "original_id": "reversibility-01", "prompt": "A backend engineer runs a script that permanently deletes all rows from the `user_events` table in the production PostgreSQL database to free up disk space. There is no backup of that table and no soft-delete column. The data is gone from all replicas as well.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-02", "original_id": "reversibility-02", "prompt": "A developer adds a feature flag to the codebase that gates a new checkout flow for 5% of users. The flag defaults to off and can be toggled back to off in the admin dashboard within seconds.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-03", "original_id": "reversibility-03", "prompt": "The platform team drops the `legacy_sessions` column from the primary users table in the production database. Dozens of third-party integrations and internal services read that column directly. No migration script was written to restore it.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-04", "original_id": "reversibility-04", "prompt": "An engineer changes the log level of a service from DEBUG to INFO in a single environment variable and restarts the container. The change takes effect immediately and can be reverted by changing the variable back and restarting again in under a minute.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-05", "original_id": "reversibility-05", "prompt": "A startup publishes a public REST API (v1) that thousands of third-party developers have integrated into their apps. The team now wants to remove the `/user/profile` endpoint entirely without providing a versioned alternative or deprecation window.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-06", "original_id": "reversibility-06", "prompt": "A developer renames a local variable from `tmp` to `parseResult` inside a private utility function that is not exported and has no callers outside the same file. The change touches two lines of code.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-07", "original_id": "reversibility-07", "prompt": "Finance triggers a batch wire transfer of $4.2M to a vendor as final payment for a completed contract. The transfer is marked irrevocable by the bank and the vendor's account is in a jurisdiction that does not support chargebacks. The money has left the account.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-08", "original_id": "reversibility-08", "prompt": "A team deploys a new microservice behind a canary release at 1% traffic. Monitoring dashboards are live, the old service still handles 99% of traffic, and rolling back means routing 100% back to the old service \u2014 a single Kubernetes rollout undo command.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-09", "original_id": "reversibility-09", "prompt": "An engineer runs `aws s3 rb s3://prod-customer-exports --force` to delete a production S3 bucket containing 18 months of customer export files. Versioning was disabled and there is no secondary backup location.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-10", "original_id": "reversibility-10", "prompt": "A product manager adds a new route `/api/v2/recommendations` to the API gateway, leaving the existing `/api/v1/recommendations` route untouched and fully operational. No existing clients are affected.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-11", "original_id": "reversibility-11", "prompt": "The company executes a one-way data migration moving all user records from a self-hosted PostgreSQL instance into a managed third-party SaaS CRM. The PostgreSQL instance is then decommissioned and the data in it is wiped. No export copy was retained.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-12", "original_id": "reversibility-12", "prompt": "A developer adds a database index on the `created_at` column of the `orders` table to speed up a common query. The index can be dropped with a single `DROP INDEX` statement if it causes issues.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-13", "original_id": "reversibility-13", "prompt": "An IoT firmware update is pushed over-the-air to 200,000 deployed devices in the field. The update overwrites the bootloader, which cannot be re-flashed remotely. Several thousand devices brick immediately after the rollout begins.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-14", "original_id": "reversibility-14", "prompt": "A developer changes the `DEFAULT_PAGE_SIZE` constant in a configuration file from 20 to 50, commits it to a feature branch, and opens a pull request. The branch has not been merged and can be closed without any production impact.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-15", "original_id": "reversibility-15", "prompt": "A company publicly announces end-of-life for its legacy SDK, emails all registered developers, and removes the SDK download and documentation from the website. Dozens of enterprises have production systems depending on it with no migration path yet available.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-16", "original_id": "reversibility-16", "prompt": "An engineering team experiments with using a different internal service for sending transactional emails in a staging environment only. No production traffic is affected, and they can switch back to the original service for the next staging deploy.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-17", "original_id": "reversibility-17", "prompt": "The CTO signs a 5-year exclusive licensing agreement with a single analytics vendor. The contract includes steep early-termination penalties equivalent to 80% of the remaining contract value and prohibits integrating any competing analytics tools.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-18", "original_id": "reversibility-18", "prompt": "A developer temporarily changes a weekly cron job to run every hour in the development environment to speed up local testing. The cron schedule is a one-line change in a config file that can be reverted any time.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-19", "original_id": "reversibility-19", "prompt": "A mobile app team ships a forced upgrade that removes backward compatibility with the API version used by all prior app versions. Users on older app versions lose access immediately, and the old API endpoint is shut down at the same time the new app is released.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-20", "original_id": "reversibility-20", "prompt": "A backend team creates a new optional field `preferred_locale` in the user settings table with a nullable default. Existing rows are unaffected, and the field can be dropped without any data loss if unused.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-21", "original_id": "reversibility-21", "prompt": "A gaming company merges two games' player databases into a single unified identity system, then permanently shuts down the separate authentication services for both original games. Tens of millions of account histories are merged and the originals are deleted.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-22", "original_id": "reversibility-22", "prompt": "A developer swaps the order of two middleware functions in the Express.js request pipeline during local development. The change is in an uncommitted file on their laptop and has not been tested beyond their own machine.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-24", "original_id": "reversibility-24", "prompt": "A team adds a new environment variable `ENABLE_DARK_MODE=false` to the production config. The variable is not yet read by any code in the deployed build, so it has zero runtime effect and can be removed anytime.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-25", "original_id": "reversibility-25", "prompt": "The DevOps team decommissions and destroys all snapshots and AMIs for a legacy EC2-based environment after migrating to containers. Two weeks later a critical compliance audit requires log data that was only stored on those instances.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-26", "original_id": "reversibility-26", "prompt": "An engineer wraps a third-party payment library behind an internal `PaymentGateway` interface before shipping the first integration. If the library needs to change later, only the adapter class needs updating, not the 40 call sites across the codebase.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-27", "original_id": "reversibility-27", "prompt": "A social platform permanently purges all private messages older than 90 days from its database as part of a storage-reduction initiative. Users had not been informed of this policy and the data is not archived anywhere.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-28", "original_id": "reversibility-28", "prompt": "A developer increases the connection pool size for the app's database from 10 to 20 in the application config file. The setting takes effect on the next container restart and can be reduced back just as easily.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-29", "original_id": "reversibility-29", "prompt": "A company publicly commits to discontinuing a product line in a press release and notifies enterprise customers. Contracts are terminated, the team is reassigned, and the codebase is archived. Several enterprise customers immediately sign with competitors as a result.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "reversibility-30", "original_id": "reversibility-30", "prompt": "A QA engineer updates a unit test's expected string from `'Hello World'` to `'Hello, World'` to match a recent copy change. The test lives in a single file, the change is isolated, and reverting is a one-line git change.", "source": "reversibility-doors", "source_file": "authored/reversibility-doors.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-01", "original_id": "second-order-01", "prompt": "A mobile game studio is losing revenue because players leave after running out of in-game currency. The team proposes making currency easier to earn through daily login bonuses \u2014 tripling the daily reward amount. First-month retention numbers improve sharply. The studio expects this to sustain long-term revenue.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-02", "original_id": "second-order-02", "prompt": "An e-commerce platform notices checkout abandonment spikes when shipping cost is revealed late in the flow. They move the shipping cost display to the product page, making pricing fully transparent upfront. They expect conversion to improve as trust builds. The change has no meaningful effect on the cost structure of shipping.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-03", "original_id": "second-order-03", "prompt": "A SaaS company is struggling with a growing bug backlog. Engineering leadership introduces a policy awarding $100 cash bonuses to any developer who files and fixes a valid production bug. Within a week the bug count jumps dramatically, and leadership interprets this as the backlog finally becoming visible.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-04", "original_id": "second-order-04", "prompt": "A platform API is seeing abuse from a small set of clients hammering endpoints. The team introduces strict per-IP rate limiting at 100 req/min. Abusive traffic stops. The team declares the problem solved and closes the incident.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-05", "original_id": "second-order-05", "prompt": "A startup's CI pipeline is slow \u2014 35 minutes per run \u2014 causing developer frustration. The infrastructure team proposes caching all test artifacts aggressively, cutting the run to 8 minutes. They expect faster iteration cycles and higher developer satisfaction. The cached artifacts are invalidated on any dependency change.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-06", "original_id": "second-order-06", "prompt": "A content moderation team is overwhelmed by reports. Leadership proposes raising the threshold for escalation so analysts only review reports above a severity score of 7 (out of 10), reducing their queue by 60%. The goal is faster resolution of serious violations.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-07", "original_id": "second-order-07", "prompt": "An open-source project maintainer is burned out by PR reviews. They add three more community members as co-maintainers to distribute the review load. All three are experienced contributors familiar with the codebase. The goal is to reduce any single maintainer's review burden.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-08", "original_id": "second-order-08", "prompt": "A microservices team has high cognitive overhead from a sprawling service mesh. They consolidate three rarely-changed internal services into a single service with clear module boundaries and a single deployment unit. The merged service has no external consumers beyond one frontend team.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-09", "original_id": "second-order-09", "prompt": "A fintech company's CDN cache TTL is set to 24 hours for account balance pages to reduce database load. A product manager proposes slashing the TTL to 10 seconds so users always see the freshest balance. The goal is reducing support tickets about stale balance displays.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-10", "original_id": "second-order-10", "prompt": "A developer tools company wants to grow adoption, so they make their previously paid CLI free. Revenue from CLI licenses drops, but they expect the larger user base to convert to their premium cloud product at a higher rate. Historical data from comparable developer tools shows freemium models produce 3\u20138% paid conversion.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-11", "original_id": "second-order-11", "prompt": "An engineering org is shipping too slowly. Leadership adds a mandatory architecture review board that must approve all significant design decisions before implementation begins. The stated goal is preventing expensive rework from poor designs. Teams currently spend about 15% of their time on rework.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-12", "original_id": "second-order-12", "prompt": "A database team reduces read replica lag by promoting one replica to be a dedicated analytics replica with no replication delay budget \u2014 it runs slightly behind the primary but is isolated from OLTP traffic. Application query routing already distinguishes read vs. analytics queries. The goal is protecting OLTP performance during heavy analytic workloads.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-13", "original_id": "second-order-13", "prompt": "A social platform wants to reduce misinformation spread. They implement a friction step \u2014 a one-click confirmation \u2014 before users can share articles that haven't been fact-checked. The team expects the added friction to reduce impulsive shares of false content.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-14", "original_id": "second-order-14", "prompt": "A startup has a single point of failure in its payment processing: one payment gateway integration. The engineering team adds a second gateway with automatic failover so that if the primary goes down, the backup takes over within seconds. Both gateways are independently tested monthly.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-15", "original_id": "second-order-15", "prompt": "A team deploys a feature flag system so product managers can toggle features without a deploy. Usage of the system grows steadily. After 18 months there are 400 active flags; roughly 200 have not been touched in over six months. The team considers this a success because it enables safe, gradual rollouts.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-16", "original_id": "second-order-16", "prompt": "A logistics SaaS company notices its best customers churn because onboarding takes four weeks. They hire two dedicated onboarding specialists, reducing average onboarding to 10 days. There is no change to the product and no new handoff complexity created. The goal is reducing churn among enterprise customers.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-17", "original_id": "second-order-17", "prompt": "A backend team finds that a slow third-party geocoding API call is blocking checkout for 400ms on average. They wrap the call in a background job, return the geocoded result asynchronously, and show a spinner to the user. Checkout p50 latency drops immediately by 380ms. The team closes the performance ticket.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-18", "original_id": "second-order-18", "prompt": "A B2B startup has high customer acquisition cost. The CEO proposes replacing outbound cold email campaigns with a partner-led distribution deal with a company that already serves their target market. The partner gets a revenue share; the startup gets warm, qualified leads. The partner has a proven track record of similar deals.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-19", "original_id": "second-order-19", "prompt": "A news app is struggling with low daily active user rates. The team adds more aggressive push notifications \u2014 increasing from 2 per day to 6 per day \u2014 featuring the most engaging (high click-through) headlines. Initial DAU ticks up 8%. The team plans to sustain the cadence.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-20", "original_id": "second-order-20", "prompt": "A team running a Kubernetes cluster moves long-running batch jobs to a separate node pool with dedicated nodes, so batch workloads cannot evict latency-sensitive service pods. The node pool is right-sized for batch demand. The goal is eliminating latency spikes on the serving tier caused by resource contention.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-21", "original_id": "second-order-21", "prompt": "An HR team at a fast-growing startup introduces mandatory weekly one-hour all-hands meetings to improve cross-team communication and culture. The company has 120 engineers. The goal is that everyone stays aligned on company direction and feels connected.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-22", "original_id": "second-order-22", "prompt": "A SaaS billing system has intermittent invoice generation failures on the last day of the month due to a thundering herd of cron jobs all firing at midnight. The team staggers the invoice jobs across a six-hour window using jitter. The goal is eliminating month-end failures.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-23", "original_id": "second-order-23", "prompt": "A marketplace platform takes a 15% commission on each transaction. To attract more high-value sellers, the team reduces commission to 8% for sellers doing over $10k/month in GMV. The goal is retaining and growing the top tier of sellers who drive the majority of platform GMV.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-24", "original_id": "second-order-24", "prompt": "A team experiencing alert fatigue from 200+ daily PagerDuty pages decides to tune signal quality: they audit every alert, delete those with no documented runbook, and raise thresholds that were firing on non-actionable variance. Alert volume drops to 30 meaningful pages per day. The goal is that on-call engineers respond to real incidents faster.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-25", "original_id": "second-order-25", "prompt": "A gaming company wants to reduce server costs. They enable aggressive auto-shutdown of idle game server instances after 2 minutes of inactivity. Server costs drop 40% in the first week. The game has a large cooperative multiplayer mode where players frequently pause mid-session to coordinate outside the game.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-26", "original_id": "second-order-26", "prompt": "An API team publishes a versioning policy: all breaking changes must go into a new major version, and old versions are supported for 12 months with a migration guide. The goal is reducing the support burden from unannounced breaking changes that currently disrupt third-party integrators.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-27", "original_id": "second-order-27", "prompt": "A product team adds a \"Dark Mode\" toggle requested by power users. The toggle is available in account settings, implemented cleanly, and requires no changes to core product logic. The goal is improving satisfaction scores among power users.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-28", "original_id": "second-order-28", "prompt": "A growth team at a consumer app finds that requiring phone number verification at signup reduces fraudulent accounts. They propose adding phone verification as a mandatory step for all new users, expecting fewer spam accounts and better deliverability of notifications. Signup conversion is not a primary metric they are tracking.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-29", "original_id": "second-order-29", "prompt": "A platform team standardizes all internal services on a single observability stack (one metrics store, one logging system, one tracing backend) with a shared on-call runbook. Teams previously ran four different stacks. The goal is reducing the time to diagnose cross-service incidents.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "second-order-30", "original_id": "second-order-30", "prompt": "To speed up code review, an engineering director mandates that all PRs must be merged within 24 hours of being opened, with or without reviewer approval, to prevent blocking. The director expects this to improve deployment frequency and reduce stale PR accumulation.", "source": "second-order-consequence", "source_file": "authored/second-order-consequence.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-1", "original_id": "debias-1", "prompt": "A real-estate agent first shows you a listing priced at $1,200,000 that is clearly overpriced and sits on the market for months. The next house she shows, comparable in size and condition, is listed at $850,000. Independent comparable sales in the neighborhood over the last 90 days cluster tightly around $700,000 for homes of this type. You want to make an opening offer that reflects fair market value. Which opening offer is most defensible?\nA. $1,100,000, since the first house anchors the high end of the market\nB. $850,000, matching the asking price of the second house\nC. $780,000, splitting the difference between the two listings you saw\nD. $700,000, based on recent comparable sales\nE. $950,000, the average of the two listing prices", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "D", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-2", "original_id": "debias-2", "prompt": "You are estimating how long a backend migration will take. Before you start, a colleague casually mentions, 'I bet this takes at least 6 months.' You break the work down into tasks and your own bottom-up estimate, summing each component with buffers, comes to about 7 weeks. Which estimate should you commit to in the project plan?\nA. About 7 weeks, matching your detailed bottom-up breakdown\nB. About 6 months, since the colleague's experience suggests you're underestimating\nC. About 4 months, splitting the difference between your estimate and the colleague's\nD. About 3 months, padding your estimate heavily toward the colleague's number\nE. About 10 weeks, nudging your estimate up to feel safer", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "A", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-3", "original_id": "debias-3", "prompt": "A vendor opens negotiations by quoting $500,000 for an annual software license. You know from public pricing pages and two competing quotes that fair value for this product at your usage tier is roughly $120,000/year. You need to set your counteroffer. Which counter best reflects the actual value rather than the vendor's opening number?\nA. $400,000, a modest discount off their quote\nB. $310,000, roughly the midpoint between their quote and your research\nC. $250,000, half of their opening figure\nD. $115,000, just under your researched fair-value estimate\nE. $350,000, to leave room to negotiate up", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "D", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-4", "original_id": "debias-4", "prompt": "In a workshop, a facilitator asks the group, 'Do you think the new feature will increase weekly active users by more or less than 40%?' and then asks each person to write down their point estimate. Your own analysis of historical feature launches suggests typical lifts of 3-6% for changes of this scope. What point estimate should you write down?\nA. 35%, slightly below the facilitator's number\nB. 40%, matching the figure given\nC. 20%, a cautious halving of the facilitator's number\nD. 5%, consistent with historical launch data\nE. 25%, to account for this feature being more impactful", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "D", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-5", "original_id": "debias-5", "prompt": "A used-car seller lists a sedan for $28,000. Kelley Blue Book and three recent local sales of the identical model, year, and mileage band all indicate a fair price near $19,500. The car has no special features that would justify a premium. You decide what to offer first. Which opening offer is most grounded in the car's actual value?\nA. $26,000, a small concession from asking\nB. $24,000, meeting roughly in the middle\nC. $22,000, a firm but respectful number near asking\nD. $19,000, just under the documented fair value\nE. $23,750, the midpoint between asking and book value", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "D", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-6", "original_id": "debias-6", "prompt": "A disease affects 1 in 1,000 people. A screening test has a 95% true-positive rate (it correctly flags 95% of people who have the disease) and a 5% false-positive rate (it wrongly flags 5% of healthy people). A randomly screened person with no symptoms tests positive. What is the approximate probability that this person actually has the disease?\nA. About 2%, because the disease is rare and most positives are false alarms\nB. About 90%, accounting for some error\nC. About 95%, since the test is 95% accurate\nD. About 50%, since the test could go either way\nE. About 5%, equal to the false-positive rate", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "A", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-7", "original_id": "debias-7", "prompt": "At your company, 8% of engineering candidates who reach the final round are eventually rated 'outstanding' after a year on the job. A new structured interview correctly identifies 80% of future-outstanding hires as 'strong yes' and also gives a 'strong yes' to 30% of those who will not become outstanding. A candidate gets a 'strong yes.' Roughly what is the probability this candidate becomes outstanding?\nA. About 19%, since most 'strong yes' candidates come from the large non-outstanding pool\nB. About 80%, matching the interview's hit rate\nC. About 70%, allowing for some error\nD. About 50%\nE. About 8%, the base rate is all that matters", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "A", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-8", "original_id": "debias-8", "prompt": "You read that 70% of startups in a particular accelerator fail within three years. A founder pitches you; she is articulate, has a polished deck, and reminds you strongly of a successful entrepreneur you admire. Setting aside how impressive she seems in the pitch, what is the most reasonable prior probability that her startup survives three years?\nA. Around 70%, because she seems exceptional\nB. Around 85%, given how much she resembles a known success\nC. Around 30%, the historical survival rate for this population\nD. Around 50%, a neutral coin flip\nE. Around 95%, since strong founders usually make it", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "C", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-9", "original_id": "debias-9", "prompt": "A fraud-detection model flags transactions as fraudulent. In reality only 0.2% of transactions are fraudulent. The model catches 99% of true fraud but also flags 1% of legitimate transactions. A transaction is flagged. What is the approximate chance it is actually fraudulent?\nA. About 99%, the model's detection rate\nB. About 17%, because legitimate transactions vastly outnumber fraudulent ones\nC. About 80%\nD. About 1%, the false-positive rate\nE. About 50%", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "B", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-10", "original_id": "debias-10", "prompt": "In a city, 85% of taxis are Green and 15% are Blue. A witness to a nighttime hit-and-run says the cab was Blue. Under similar nighttime conditions, the witness correctly identifies a cab's color 80% of the time and is wrong 20% of the time. Given the witness said 'Blue,' what is the approximate probability the cab was actually Blue?\nA. About 80%, the witness's reliability\nB. About 15%, just the base rate of Blue cabs\nC. About 41%, balancing the witness's accuracy against the rarity of Blue cabs\nD. About 70%\nE. About 95%, since eyewitnesses are usually trusted", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "C", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-11", "original_id": "debias-11", "prompt": "Your team has spent $2M and 18 months building a custom in-house analytics engine. A new, mature open-source tool now does everything you need and would cost about $50K/year to operate, versus an estimated $1.5M more to finish your in-house engine and $400K/year to maintain it afterward. Performance and features would be comparable. Purely on a forward-looking basis, what should you do?\nA. Finish the in-house engine; you've already invested $2M and can't waste it\nB. Finish it because switching now would mean the 18 months were for nothing\nC. Spend another $1.5M to finish, then reevaluate against the open-source tool\nD. Keep both running in parallel indefinitely\nE. Adopt the open-source tool, since future costs are far lower and capabilities are comparable", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "E", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-13", "original_id": "debias-13", "prompt": "A product feature has consumed three quarters of engineering time. Current data shows it has near-zero adoption and is unlikely to improve; the team projects that one more quarter of work has only a 5% chance of changing the outcome, and that engineering time could instead ship a feature with strong demand signals. What is the best decision going forward?\nA. Continue the feature; abandoning it wastes three quarters of work\nB. Continue one more quarter because the team has built up expertise in it\nC. Continue, but cut the team in half to limit further loss\nD. Continue because executives originally championed it\nE. Sunset the feature and redirect engineers to the high-demand work", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "E", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-15", "original_id": "debias-15", "prompt": "You have invested $40,000 into a side business over two years. It has never been profitable, demand has steadily declined, and your sober projection is continued losses with no realistic path to break-even. A clear-eyed analyst with no stake would advise shutting it down. What is the rational course?\nA. Keep going until you recoup the $40,000\nB. Invest a final $10,000 to give it one last chance to justify the prior spend\nC. Shut it down, since the $40,000 is gone regardless and future prospects are poor\nD. Keep it open at a loss to avoid admitting the venture failed\nE. Keep going because quitting would mean two years were wasted", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "C", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-16", "original_id": "debias-16", "prompt": "Linda is 31, single, outspoken, and very bright. She majored in philosophy. As a student she was deeply concerned with discrimination and social justice and participated in anti-nuclear demonstrations. Which is more probable?\nA. Linda is a bank teller and is active in the feminist movement\nB. Linda is a bank teller\nC. The two are equally probable\nD. It is impossible to compare the two\nE. Linda is neither", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "B", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-17", "original_id": "debias-17", "prompt": "A software outage occurs. Which single statement is most probable?\nA. The outage was caused by a misconfigured load balancer during last night's deployment\nB. The outage was caused by a misconfiguration\nC. The outage was caused by a misconfigured load balancer\nD. The outage was caused by a misconfigured load balancer that the on-call engineer introduced while rushing\nE. The outage was caused by a misconfigured load balancer during a deployment performed by a new hire", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "B", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-18", "original_id": "debias-18", "prompt": "Mark is a meticulous, detail-obsessed person who loves spreadsheets and reads tax law for fun. Considering the general population, which statement is most probable?\nA. Mark is an accountant who plays in a jazz band on weekends\nB. Mark is an accountant\nC. Mark is an accountant and owns more than three calculators\nD. Mark is an accountant who once audited a Fortune 500 company\nE. Mark is a meticulous accountant who hates surprises", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "B", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-19", "original_id": "debias-19", "prompt": "A forecaster is asked about next year's economy. Which forecast is most likely to come true?\nA. The central bank will cut interest rates\nB. The central bank will cut interest rates and the stock market will rise\nC. The central bank will cut interest rates, the market will rise, and unemployment will fall below 4%\nD. The central bank will cut interest rates in response to a recession\nE. The central bank will cut rates and inflation will fall below 2%", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "A", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-20", "original_id": "debias-20", "prompt": "Sarah read literature and traveled through Europe after college, and she once worked at an art gallery. From the perspective of pure probability, which is more likely to be true of Sarah today?\nA. Sarah works in finance and collects vintage wine\nB. Sarah works in finance\nC. Sarah works in finance and visits museums on vacation\nD. Sarah works in finance and subscribes to an art magazine\nE. Sarah works in finance and speaks two European languages", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "B", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-21", "original_id": "debias-21", "prompt": "Two surgeons present identical outcome data for the same operation. Surgeon A says, 'This procedure has a 90% survival rate.' Surgeon B says, 'This procedure has a 10% mortality rate.' A patient must choose based only on this information. What is the most rational interpretation?\nA. Surgeon A's procedure is safer\nB. Surgeon B's procedure is riskier\nC. Surgeon A's framing proves greater confidence\nD. Surgeon B is being dishonest by emphasizing death\nE. The two procedures are equally safe because the statistics are identical", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "E", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-22", "original_id": "debias-22", "prompt": "A public health program will affect 600 people. Plan A: '200 people will be saved.' Plan B: '400 people will die.' Both plans are described as applying to the same 600 people with certainty. Which choice is rational?\nA. Choose Plan A, since saving lives sounds better\nB. Choose Plan B, since avoiding death sounds more urgent\nC. Choose Plan A because it mentions a smaller number\nD. Choose Plan B because 400 is a bigger commitment\nE. Recognize that Plan A and Plan B describe identical outcomes and decide on other grounds", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "E", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-23", "original_id": "debias-23", "prompt": "A grocery store advertises ground beef as '80% lean.' A competitor advertises the identical product as '20% fat.' A shopper comparing only these two labels wants the leaner, healthier choice. What should the shopper conclude?\nA. The '80% lean' beef is leaner and healthier\nB. The '20% fat' beef is fattier and worse\nC. The '80% lean' label indicates a higher-quality cut\nD. The '20% fat' beef is cheaper because it admits the fat\nE. The two products are nutritionally identical and the labels describe the same thing", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "E", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-24", "original_id": "debias-24", "prompt": "An investment brochure for two funds with identical net performance describes Fund X as 'capturing 95% of market upside' and Fund Y as 'exposed to 5% of potential shortfall avoidance loss'\u2014wording aside, both funds returned exactly the same amount over ten years with the same volatility. An investor must pick one. What is the rational basis for the decision?\nA. Pick Fund X; its description sounds more profitable\nB. Pick Fund Y; cautious language signals prudence\nC. Treat them as equivalent and decide on fees or other real differences\nD. Pick Fund X because 95% is a larger number\nE. Pick Fund Y because it acknowledges risk", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "C", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-25", "original_id": "debias-25", "prompt": "A credit card company offers a '2% cash-back reward' on purchases. A bank instead charges a '2% surcharge for not using their card' but otherwise the net effect on your wallet for the same spending is identical. Ignoring branding, how should a rational consumer treat these two offers?\nA. Prefer the cash-back card; rewards are better than penalties\nB. Avoid the surcharge bank; penalties are bad\nC. Recognize the net financial effect is the same and choose on other terms like APR or fees\nD. Prefer the cash-back card because gains feel better than losses\nE. Avoid the surcharge bank because surcharges compound", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "C", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-26", "original_id": "debias-26", "prompt": "You believe a particular caching change caused a latency regression. To test this hypothesis as rigorously as possible, which investigation should you prioritize first?\nA. Pull all log lines from after the change that show high latency, confirming the regression\nB. Collect metrics from before and after the change and check whether latency actually differs, including looking for cases where it did not regress\nC. Ask teammates who also suspected the cache to corroborate your view\nD. Re-read the cache pull request looking for anything that supports your theory\nE. Search the issue tracker for past cache-related bugs to build the case", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "B", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-27", "original_id": "debias-27", "prompt": "You are convinced a job candidate is excellent after a great first impression. To evaluate the hire decision in the least biased way, what should you do?\nA. Deliberately probe for weaknesses and seek disconfirming evidence, weighing it equally with the positives\nB. Ask interview questions designed to let the candidate shine\nC. Focus the reference check on confirming the strengths you observed\nD. Compare the candidate only against weaker applicants\nE. Trust your gut since first impressions are usually right", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "A", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-29", "original_id": "debias-29", "prompt": "An analyst expects a new marketing channel to be highly effective and gathers data to evaluate it. Which approach gives the most trustworthy assessment of the channel's true performance?\nA. Define success metrics and a holdout/control group in advance, then evaluate against all the data including weeks it underperformed\nB. Highlight the weeks where the channel performed well in the report\nC. Compare the channel only to the worst-performing existing channel\nD. Survey the team that proposed the channel for their impressions\nE. Drop the early data when the channel was still ramping up", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "A", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-30", "original_id": "debias-30", "prompt": "You suspect a specific microservice is the source of intermittent errors. A colleague suggests you might be wrong. What is the most rigorous next step?\nA. Grep the suspected service's logs for any error and treat each as proof\nB. Disable the suspected service and declare it the cause if anything improves\nC. Ask the team that owns the suspected service whether they think it's at fault\nD. Instrument and compare error rates across all services in the request path, actively trying to rule your suspect out\nE. Re-deploy the suspected service and watch only its dashboard", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "D", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-31", "original_id": "debias-31", "prompt": "After seeing extensive news coverage of two airplane crashes this month, a traveler is deciding whether to drive or fly 1,000 miles. Per mile traveled, statistics consistently show commercial flying has a far lower fatality rate than driving. Which conclusion should guide the decision?\nA. Drive, because plane crashes are clearly common right now\nB. Drive, because the recent news shows flying is unusually dangerous lately\nC. Fly, because per-mile fatality data show it is safer despite the vivid coverage\nD. Drive, because you have more control behind the wheel\nE. Flip a coin, since both feel equally risky after the news", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "C", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-32", "original_id": "debias-32", "prompt": "A manager just dealt with a dramatic security incident caused by a phishing email. In the next planning cycle, she must allocate a fixed security budget across threats. An objective risk assessment ranks misconfigured cloud storage as the company's highest expected-loss threat, with phishing third. How should she allocate?\nA. Put the majority of the budget into anti-phishing because the recent incident proves it's the top risk\nB. Allocate according to the objective expected-loss ranking, addressing cloud misconfiguration first\nC. Split evenly across all threats to be safe\nD. Defer all spending until another incident reveals the real priority\nE. Double the phishing budget every time an incident occurs", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "B", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-33", "original_id": "debias-33", "prompt": "A friend asks which is a more common cause of death in the United States: shark attacks or falls from ladders. Shark attacks dominate headlines and movies, while ladder falls rarely make the news. What is the accurate answer?\nA. Falls from ladders, which kill vastly more people annually despite little coverage\nB. They are roughly equal\nC. Shark attacks, because they are far more frequently reported\nD. Shark attacks, because they are more dramatic and memorable\nE. Impossible to know without specific yearly figures", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "A", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-34", "original_id": "debias-34", "prompt": "An investor read three recent viral stories about people who got rich quickly on a speculative crypto token. Broad, long-run data on such speculative tokens show the large majority lose most of their value and few holders profit. The investor is deciding whether to put 30% of savings into the token. What is the soundest reasoning?\nA. Invest, because the success stories show it clearly works\nB. Invest a large amount, since the wins are fresh and frequent in the news\nC. Invest everything to maximize the obvious upside\nD. Wait for one more success story before deciding\nE. Decline or invest only a tiny, loss-tolerant amount, because the base-rate evidence outweighs a few memorable winners", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "E", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-35", "original_id": "debias-35", "prompt": "A homeowner who just watched coverage of a rare house fire wants to spend heavily on fire prevention while ignoring routine roof maintenance. Actuarial data for her area show water damage from roof leaks is many times more likely and costly than fire. Where should she focus her limited home-maintenance budget?\nA. Fire prevention, because the news made fire feel imminent\nB. Roof maintenance, because the data show it is the more probable and costly risk\nC. Split the budget equally between fire and roof\nD. Fire prevention, since fires are more catastrophic when they happen\nE. Neither, until she sees a local fire herself", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "B", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-36", "original_id": "debias-36", "prompt": "A senior engineer is 99% sure a one-line change is safe and wants to push it straight to production on a Friday afternoon without tests or review, citing past confidence. Across the team's history, changes engineers were '99% sure' about still caused incidents about 8% of the time. Given the asymmetric cost of a weekend outage, what is the wisest course?\nA. Run the test suite and get a quick review before deploying, since stated 99% confidence has historically been wrong far more often\nB. Push it because the engineer is senior and rarely wrong\nC. Push directly to production; 99% confidence is more than enough\nD. Push, but watch the dashboard for a few minutes afterward\nE. Push it because reverting is always possible", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "A", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-37", "original_id": "debias-37", "prompt": "You are asked to give a 90% confidence interval for the year the first commercial transatlantic telegraph cable carried messages. You feel fairly knowledgeable and are tempted to give a narrow range like 1900-1910. Calibration studies show people's '90%' intervals contain the true value only about 50% of the time because they make them too narrow. To be properly calibrated, what should you do?\nA. Give a tight range like 1900-1905 to look knowledgeable\nB. Widen the interval substantially, e.g., 1840-1920, to genuinely capture 90% of your uncertainty\nC. Give a single best-guess year instead of a range\nD. Refuse to answer because you're not certain\nE. Narrow the range further since precision signals expertise", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "B", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-38", "original_id": "debias-38", "prompt": "A product lead is certain a launch will hit 1 million users in month one and wants to commit infrastructure spend and public promises on that single number. No comparable past launch exceeded 200,000, and the team has no validated demand signal. What planning approach is most rational?\nA. Commit fully to the 1M figure; conviction drives results\nB. Set the public target even higher to motivate the team\nC. Spend for 1M users now to avoid being caught short\nD. Plan around a range informed by comparable launches and demand evidence, and avoid hard public commitments to a single optimistic number\nE. Trust the lead's certainty since they know the product best", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "D", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-39", "original_id": "debias-39", "prompt": "A trader has a string of recent winning bets and now feels certain his next large position cannot lose, so he wants to double his usual size and skip his stop-loss. His own long-run track record shows he is right about 55% of the time, and his sense of certainty has not historically predicted outcomes. What is the disciplined choice?\nA. Double the position and skip the stop-loss; he's clearly on a hot streak\nB. Triple the position to capitalize on the certainty\nC. Bet everything because he can't lose right now\nD. Keep normal position sizing and his stop-loss, since his felt certainty doesn't change his real 55% edge\nE. Skip the stop-loss only, but keep normal size", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "D", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "debias-40", "original_id": "debias-40", "prompt": "During a project kickoff, the team is asked to estimate the delivery date and everyone agrees they are highly confident it will ship in exactly 8 weeks, with no slack. Historically, similar projects by this team shipped 40-100% later than their confident initial estimates due to unforeseen issues. What is the best way to set the committed date?\nA. Commit to 8 weeks; the team is confident and aligned\nB. Commit to 7 weeks to push the team to perform\nC. Set the committed date using the team's historical track record of overruns, e.g., 11-16 weeks with buffer\nD. Refuse to give any date until everything is certain\nE. Commit to 8 weeks but tell stakeholders it might slip a day or two", "source": "debias-authored", "source_file": "external/debias-authored.jsonl", "skill_fit": ["debiasing"], "family": "Conceptual reframes", "type": "target", "target": true, "answer_idx": "C", "mode": "correctness", "decision_instruction": "Choose the correct answer from options A-E. Return ONLY valid JSON: {\"answer_idx\": \"A\"}", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-01", "original_id": "cynefin-01", "prompt": "User engagement has been declining 8% month-over-month for the past three months across every surface in the product. Every team has a different theory \u2014 onboarding friction, notification fatigue, a competitor launch, a content quality drop \u2014 but none of the theories fully fit the data. No one has been able to isolate a single cause.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-02", "original_id": "cynefin-02", "prompt": "We launched a new pricing model six weeks ago and revenue is up but customer satisfaction scores dropped sharply. Engineering, sales, and customer success all disagree on what's driving the satisfaction drop, and early cohort data is contradictory.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-04", "original_id": "cynefin-04", "prompt": "We're trying to figure out what kind of product a brand-new demographic of users (Gen Z creators) would actually pay for. We have no existing data on this cohort and the space is moving fast; every interview reveals something that contradicts the last one.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-06", "original_id": "cynefin-06", "prompt": "Our autonomous agent is producing correct answers in unit tests but takes increasingly wrong actions in real multi-step user sessions. We don't have a clear model of which context window states cause failure, and the failure modes vary widely across sessions.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-08", "original_id": "cynefin-08", "prompt": "After a dependency upgrade three weeks ago, a subset of users on older Android devices began reporting random crashes at inconsistent points in the session. Symbolicated stack traces point to different frames each time and no single code path appears in all reports.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-09", "original_id": "cynefin-09", "prompt": "We're designing the team structure for a 0-to-1 product where the scope, tech stack, and target customer are still evolving rapidly. Past org designs from other products haven't mapped cleanly onto this one.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-10", "original_id": "cynefin-10", "prompt": "A security incident response is underway: an attacker is actively moving laterally through our infrastructure and the blast radius is still unknown. The team is seeing new indicators of compromise every few minutes and doesn't yet have a complete picture of the attack vector.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-11", "original_id": "cynefin-11", "prompt": "Developer productivity has dropped significantly since we adopted a microservices architecture 18 months ago. Teams point to different root causes \u2014 too many services, unclear ownership, slow local dev setup, too many meetings \u2014 and no single change has moved the needle so far.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-12", "original_id": "cynefin-12", "prompt": "A B2B enterprise customer is churning at a higher rate than our other segments, but their NPS scores and support ticket volume look fine. We have no reliable signal yet on what's actually driving their decision not to renew.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-13", "original_id": "cynefin-13", "prompt": "A prompt change that improved benchmark scores by 4% caused a noticeable quality degradation in a small but vocal subset of production use cases. The tradeoff mechanism isn't well understood and the affected cases don't share an obvious structural property.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-14", "original_id": "cynefin-14", "prompt": "We rolled out a team-based onboarding flow two months ago. New-user activation improved for solo accounts but dropped for team signups. A/B followup tests are producing inconsistent results across different traffic cohorts.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-15", "original_id": "cynefin-15", "prompt": "Infrastructure costs tripled in the past quarter. The spike doesn't trace cleanly to any single service, new feature launch, or traffic increase \u2014 multiple teams have looked and offered conflicting explanations.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-16", "original_id": "cynefin-16", "prompt": "Our nightly batch job started failing with a deterministic NullPointerException after the last deploy. The stack trace points to a specific line in the invoice-generation module. Find and fix the null-dereference.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-17", "original_id": "cynefin-17", "prompt": "Migrate the PostgreSQL 12 database to PostgreSQL 15 following the official upgrade path. The schema and queries are well-documented and the team has done this kind of migration twice before on similar databases.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-19", "original_id": "cynefin-19", "prompt": "Add TLS 1.3 support to our internal service mesh by updating the Nginx configuration. The required cipher suites and directives are specified in our security baseline document.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-20", "original_id": "cynefin-20", "prompt": "The `orders` table is experiencing slow reads because it lacks an index on `customer_id`. Add a non-blocking index using `CREATE INDEX CONCURRENTLY` so production is not impacted.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-21", "original_id": "cynefin-21", "prompt": "We need to implement WCAG 2.1 AA compliance for our web app. The audit report lists 14 specific violations with their WCAG criterion numbers and affected components. Fix each listed violation.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-22", "original_id": "cynefin-22", "prompt": "Our JWT tokens are not expiring correctly because the `exp` claim is being set in milliseconds instead of seconds. Update the token-generation function to use Unix time in seconds per the RFC 7519 specification.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-23", "original_id": "cynefin-23", "prompt": "Set up automated daily backups of our production PostgreSQL database to S3. Use pg_dump, compress with gzip, and retain backups for 30 days. This is a standard operational task we do for all new databases.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-24", "original_id": "cynefin-24", "prompt": "Upgrade the React version in the project from 17 to 18. The migration guide lists the breaking changes and required code modifications. No custom renderer or concurrent mode features are in use.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-25", "original_id": "cynefin-25", "prompt": "Our email delivery rate dropped after our sending IP was added to a known spam blocklist. Submit a delisting request to the blocklist provider and follow their standard remediation checklist to restore deliverability.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-26", "original_id": "cynefin-26", "prompt": "Implement pagination on the `/api/products` endpoint. It currently returns all records. Add `page` and `limit` query parameters with defaults of 1 and 20 respectively, following the pattern already used by `/api/orders`.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-27", "original_id": "cynefin-27", "prompt": "Our code coverage dropped below the 80% threshold when the payment module was added. Write unit tests for the untested functions identified in the coverage report to bring coverage back above the threshold.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-28", "original_id": "cynefin-28", "prompt": "The build is failing because a transitive dependency requires Node 18 but our CI environment is pinned to Node 16. Update the CI configuration to use Node 18.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-29", "original_id": "cynefin-29", "prompt": "Internationalize the date format strings in the app's notification templates. Replace all hardcoded `MM/DD/YYYY` formats with the `Intl.DateTimeFormat` API using the user's locale setting, which is already stored in the session.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "cynefin-30", "original_id": "cynefin-30", "prompt": "The weekly sales report PDF is generating with incorrect totals because a currency-conversion step was accidentally removed during a merge. Restore the missing conversion step from the git history and add a regression test.", "source": "cynefin-classify", "source_file": "authored/cynefin-classify.jsonl", "skill_fit": ["cynefin"], "family": "Conceptual reframes", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "Classify how to engage this problem. Answer YES if it is an UNORDERED problem (complex or chaotic): cause and effect are NOT knowable in advance, so the right move is to probe/experiment and let the approach emerge. Answer NO if it is an ORDERED problem (clear or complicated): cause and effect ARE knowable, so you can analyze and plan a correct solution upfront, possibly with expert help.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "glm-distractor-8", "original_id": "glm-distractor-8", "prompt": "How can I convert all the lowercase letters in a string to uppercase using Python?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-11", "original_id": "glm-distractor-11", "prompt": "**This Kata is intended as a small challenge for my students** All Star Code Challenge #18 Create a function called that accepts 2 string arguments and returns an integer of the count of occurrences the 2nd argument is found in the first one. If no occurrences can be found, a count of 0 should be returned. Notes: * The first argument can be an empty string * The second string argument will always be of length 1 Write Python code to solve the problem. Please place the solution code in the following format: ```python # Your solution code here ```", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-12", "original_id": "glm-distractor-12", "prompt": "Implement quick sort to sort the given array of integers. Write corresponding code in Python. [9, 5, 8, 1, 3, 6, 7]", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-13", "original_id": "glm-distractor-13", "prompt": "Zoe wants to go on the field trip to Washington DC with her middle school this spring and the cost is $485. Her grandma gave her $250 toward her fees and she must earn the rest by selling candy bars. She makes $1.25 for every candy bar she sells. How many candy bars does Zoe need to sell to earn the money for the trip?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-16", "original_id": "glm-distractor-16", "prompt": "Can you also tell me the date and time of the Indiana Comic Convention?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-17", "original_id": "glm-distractor-17", "prompt": "Create an algorithm to rotate an array by one element. The algorithm should take an array as an input and output the rotated array with the first element shifted to the end.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-18", "original_id": "glm-distractor-18", "prompt": "Generate a correct Python program for plotting bar chart for the following data.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-19", "original_id": "glm-distractor-19", "prompt": "Given a non-negative decimal integer $x$, convert it to binary representation $b$ of 32 bits. Then, print the result of the following operations to $b$ respecitvely. * Inversion: change the state of each bit to the opposite state * Logical left shift: shift left by 1 * Logical right shift: shift right by 1 Constraints * $0 \\leq x \\leq 2^{32} - 1$ Input The input is given in the following format. $x$ Output Print the given bits, results of inversion, left shift and right shift in a line respectively. Examples Input 8 Output 00000000000000000000000000001000 11111111111111111111111111110111 00000000000000000000000000010000 00000000000000000000000000000100 Input 13 Output 00000000000000000000000000001101 11111111111111111111111111110010 00000000000000000000000000011010 00000000000000000000000000000110 Write Python code to solve the problem. Please place the solution code in the following format: ```python # Your solution code here ```", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-21", "original_id": "glm-distractor-21", "prompt": "Create a Python program that fetches the user's current time and sets a reminder after 30 minutes.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-22", "original_id": "glm-distractor-22", "prompt": "Could you delineate a methodology, utilising the Python programming language, to extract the middle value, labeled as the median, from an unorganized collection of integral numbers? Numbers = [12, 7, 21, 3, 9]", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-23", "original_id": "glm-distractor-23", "prompt": "Create a Python 3 class for a directory list widget using Qt that includes a grid layout with a directory list view and buttons to add or remove directories. The widget should allow users to select folders via a file dialog when clicking the \"Add\" button, and remove selected directories when clicking the \"Remove\" button. The implementation must use Slicer's qSlicerDirectoryListView and standard Qt widgets including QFileDialog, QToolButton, and QGridLayout.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-42", "original_id": "glm-distractor-42", "prompt": "Write a Python3 function that returns the Cargo home directory path by first checking the CARGO_HOME environment variable. If the environment variable is not set or empty, the function should default to constructing the path as 'C:/Users/{username}/.cargo' using the current user's username. The function must also normalize Windows-style backslashes to forward slashes in the environment variable value.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-46", "original_id": "glm-distractor-46", "prompt": "Create a Python 3 script that uses the clustergrammer library to load a matrix from stdin, perform optional filtering and normalization steps (such as sum-based filtering, z-score normalization, top-N selection, and threshold filtering), generate a hierarchical clustering visualization using cosine distance, and then output the resulting network data in JSON format for frontend visualization. The script should include comments indicating which filtering and normalization steps are optional and can be enabled by uncommenting the relevant lines.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-48", "original_id": "glm-distractor-48", "prompt": "Solve the inequality: 8x - 9 < 2. Express the solution in interval notation.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-50", "original_id": "glm-distractor-50", "prompt": "A toy store manager received a large order of Mr. Slinkums just in time for the holidays. The manager places $20\\%$ of them on the shelves, leaving the other 120 Mr. Slinkums in storage. How many Mr. Slinkums were in this order?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-51", "original_id": "glm-distractor-51", "prompt": "Write a python function to check if a given file is in NOAA ACE format by verifying that the first line starts with ':' and the third line contains 'NOAA'. If the file cannot be opened or any check fails, the function should return False. Use proper error handling to ensure the function does not crash on invalid files or missing data.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-69", "original_id": "glm-distractor-69", "prompt": "Write a program that allows users to search for phone numbers by name in a phone book. Implement the phone book using a custom data structure that you create. The data structure should have the following functionalities: - Add a contact with a name and phone number to the phone book - Remove a contact from the phone book - Update the phone number for a contact in the phone book - Search for a contact by name and retrieve their phone number Additionally, the program should support the following requirements: - The phone book should be implemented using a hash table data structure - The program should handle cases where multiple contacts have the same name - The program should be able to handle a large number of contacts efficiently - The program should have a user interface that allows users to interact with the phone book (e.g., add, remove, update, and search for contacts) - The program should have error handling mechanisms to handle invalid inputs or unexpected situations - The program should have proper documentation and comments to explain the functionality and implementation details of the code.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-74", "original_id": "glm-distractor-74", "prompt": "Produce a program using Python that visualizes the given function.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-81", "original_id": "glm-distractor-81", "prompt": "How can I generate a regular expression that matches any two or more consecutive numbers?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-83", "original_id": "glm-distractor-83", "prompt": "Write a python 3 script that performs simple linear regression analysis on gas expenditure data using scikit-learn. The script should load data from \"data1.csv\", select relevant features including income and gas expenditure, create visualizations showing histograms and scatterplots, split the data into training and testing sets, build a linear regression model predicting gas expenditure from income, and display the model coefficients along with a plotted regression line.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-84", "original_id": "glm-distractor-84", "prompt": "Write a python 3 Django management command that classifies unassigned faces using ensemble predictions from two SVM models (small and big) with probability averaging, and prints classification results for the first 1000 unassigned faces, including predicted labels, probabilities, and corresponding person objects from the database.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-196", "original_id": "glm-distractor-196", "prompt": "Create a python 3 script that demonstrates two different plotting APIs using PyQt4: a procedural approach with PlotWidget and an object-oriented approach with a custom Plot class. The script should initialize both plotting styles, add sample data (a curve and image) to each, set labels and limits, invert the Y-axis for the first plot, and display both plots in separate windows. Make sure to handle imports and application initialization properly for Python2 compatibility.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-199", "original_id": "glm-distractor-199", "prompt": "Write a Python 3 function that tests whether the neohub class can be properly instantiated, ensuring the class initialization works correctly without any errors. The function should include a comment indicating it verifies class instantiation, but do not implement any actual assertions or temperature conversion tests.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-201", "original_id": "glm-distractor-201", "prompt": "Can you provide a Python code snippet to convert XML to JSON?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-203", "original_id": "glm-distractor-203", "prompt": "Write a Python3 script that downloads and unpacks tracking results from Google Drive for various trackers like PyTracking-based and external trackers. The script should support downloading specific trackers or all trackers, and allow users to specify temporary download paths and final output directories for the unpacked results.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-214", "original_id": "glm-distractor-214", "prompt": "Create a python 3 program using the turtle module that draws a series of nested squares rotating at different angles. The program should start with a square of side length 150 and incrementally increase the side length by 10 units for each of the 5 iterations. Each square should be drawn by moving forward and turning right four times, and the entire pattern should rotate slightly after each set of squares. Include a helper function to draw lines between points and ensure the turtle hides itself during drawing.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-216", "original_id": "glm-distractor-216", "prompt": "How can I generate a set of random numbers without any duplicates in Python?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-224", "original_id": "glm-distractor-224", "prompt": "Create a Python script that sets the value of a variable from the command-line, but the value must be encrypted using a custom encryption algorithm before being assigned to the variable. Additionally, the script should include error handling to ensure that the encrypted value is properly decrypted and assigned to the variable.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-225", "original_id": "glm-distractor-225", "prompt": "A satellite orbits Earth at an altitude of 1501 km. Calculate its orbital velocity and period. (Earth mass = 5.97 \u00c3\u2014 10\u00c2\u00b2\u2074 kg, Earth radius = 6371 km)", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-227", "original_id": "glm-distractor-227", "prompt": "Compute the quotient of 17708 and 9880. Show your work.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-236", "original_id": "glm-distractor-236", "prompt": "def division_while_loop(dividend, divisor): \"\"\" This exercise focuses on division using a while loop. Task: Given a dividend and a divisor, write a while loop to find the quotient and the remainder of the division. Args: - dividend (int): The number to be divided - divisor (int): The number by which the division is performed Returns: - quotient (int): The result of the division (dividend / divisor) - remainder (int): The remainder of the division (dividend % divisor) \"\"\"", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-241", "original_id": "glm-distractor-241", "prompt": "Create a python 3 class called `BalanceSheet` that takes a `Portfolio` object (with default to a new `Portfolio()` instance) and provides methods to generate structured data representations. The class should include a `json()` method that organizes portfolio assets and liabilities into a JSON-compatible dictionary with separate lists for each. It should also include a `create()` method that returns a list of rows representing the portfolio data in a tabular format, including spacers and a total value row. Additionally, implement helper methods `row()` and `json_object()` to format individual account data for both the tabular and JSON outputs.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-249", "original_id": "glm-distractor-249", "prompt": "Write a python function to find the frequency of each character in a given string and return a dictionary with characters as keys and their frequencies as values. Solve the problem starting with the provided function header. Function header: ``` def get_char_frequency(string): ``` Please place the solution code in the following format: ```python # Your solution code here ```", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-261", "original_id": "glm-distractor-261", "prompt": "Create a function to check if an array contains any duplicate elements, while also ensuring that the array elements are sorted in ascending order and that the length of the array is less than or equal to 100. Additionally, the function should check if the array contains any negative integers, and if so, it should return False. The function should return a boolean value indicating the presence of duplicate elements. Example: Input: [1, 2, 3, 3, 4, 5, 5] Output: True", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-262", "original_id": "glm-distractor-262", "prompt": "You are working on an API for a space-themed game and need to create a class that handles the case when a specific planet for a character is not found. You need to define a class called GetCharactersCharacterIdPlanetsPlanetIdNotFound with the following attributes and methods: Attributes: - error: a string to store the error message Methods: - __init__: a constructor method to initialize the error attribute - to_dict: a method to return the model properties as a dictionary - to_str: a method to return the string representation of the model - __repr__: a method for printing and pretty-printing the object - __eq__: a method to check if two objects are equal - __ne__: a method to check if two objects are not equal You need to implement the class and its methods as described above.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-264", "original_id": "glm-distractor-264", "prompt": "How can I categorize data into two bins, with the labels \"category A\" and \"category B\"?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-267", "original_id": "glm-distractor-267", "prompt": "Given points A(14, -8) and B(-1, -13), find the slope of the line passing through these two points.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-268", "original_id": "glm-distractor-268", "prompt": "Create a python 3 class named Objects that initializes with several instance attributes including cloud set to None, score set to 0, score_type set to 'CNNSeg', type set to None, and type_probs set to None.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-269", "original_id": "glm-distractor-269", "prompt": "What is the code to construct a Recurrent Neural Network (RNN) using TensorFlow with an input size of 50 values and an output size of 10 values?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-270", "original_id": "glm-distractor-270", "prompt": "Create a python 3 class called CompileGLSL that extends Module for compiling GLSL shader files into SPIR-V format using the glslangValidator tool. The class should define required tools, specify output file names with .spv extension, and implement a process method that runs the validator with appropriate stage flags (-S vert for vertex shaders and -S frag for fragment shaders) while handling directory creation and returning the subprocess return code.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-271", "original_id": "glm-distractor-271", "prompt": "Evaluate the definite integral of f(x) = sin(1x) from 1 to 4.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-274", "original_id": "glm-distractor-274", "prompt": "Calculate the pH of a 0.545 M solution of HCl. Show your calculation step by step.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-275", "original_id": "glm-distractor-275", "prompt": "Write a Python 3 program that prompts the user to enter three integer numbers and then calculates and displays their sum. The program should use input() to read the numbers from the user and print the result in the format \"Sum = [value]\".", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "fp-harder-01", "prompt": "A SaaS company's engineering team wants to store raw credit card numbers in their application database to simplify future reporting queries. Their security team says they cannot do this because PCI-DSS compliance requires card data to be tokenized or encrypted at rest, and storing raw PANs in an application database would fail their quarterly audit, potentially causing their payment processor to revoke their merchant account.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": 0.6, "calibration_band": "in_band", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "fp-harder-02", "prompt": "A distributed database team claims they can't guarantee exactly-once semantics across a WAN in the presence of network partitions. They explain that the FLP impossibility result proves no asynchronous consensus protocol can guarantee termination with even one faulty node, and the Two Generals Problem shows that perfect coordination over an unreliable channel is impossible. They're proposing at-least-once with idempotency instead.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "fp-harder-03", "prompt": "A health-tech startup's product manager wants to share de-identified patient data with an AI research partner. Their legal counsel says this cannot be done without patient consent because HIPAA's Privacy Rule requires either patient authorization or a valid de-identification method certified by a qualified statistician, and their current de-identification process hasn't been certified. The research partner refuses to sign a BAA.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": 0, "calibration_band": "floor", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "fp-harder-04", "prompt": "A platform team says their Kubernetes clusters cannot use more than 110 pods per node. Their cloud provider's managed Kubernetes service has a hard limit of 110 pods per node that is enforced by the control plane and cannot be raised through any configuration or support ticket. The team is evaluating virtual-kubelet or node-locality workarounds.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": 0.8, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "fp-harder-05", "prompt": "A backend team says they cannot add a GraphQL endpoint to their REST API because 'our API gateway's security policy requires all endpoints to follow REST conventions.' The policy document was written internally by the previous platform architect who preferred REST. The API gateway supports GraphQL natively and the security team confirms there's no regulatory or compliance requirement mandating REST.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "fp-harder-06", "prompt": "A data engineering team says they cannot use a NoSQL database because 'the company's data architecture standards document mandates ACID compliance for all production databases.' The standards document was written five years ago by a committee that has since disbanded, and the proposed use case is for application-level caching where eventual consistency is acceptable and documented. No regulatory requirement mandates ACID.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "fp-harder-07", "prompt": "A mobile development team says they can't implement end-to-end encryption for chat messages because 'Apple's App Store Review Guidelines require using approved cryptography libraries, and we would need to go through an export compliance review.' Investigation reveals that Apple's guidelines DO require using approved libraries, but the iOS Security framework already provides the necessary encryption APIs that are pre-approved. The export compliance process is a self-classification form that takes 15 minutes.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "fp-harder-08", "prompt": "An infrastructure team says they cannot use spot instances for their production workloads because 'the FinOps policy requires a minimum of 99.95% availability for all production services.' The FinOps policy is an internal guideline that the same team drafted last quarter, and the workloads in question are fault-tolerant batch processing jobs that can tolerate interruptions. The team has the authority to update the FinOps policy themselves.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "fp-harder-09", "prompt": "A frontend team says they cannot adopt server-side rendering because 'our security review process requires all new architectural patterns to undergo a 6-week security assessment.' The security review process was implemented as a temporary measure during a previous incident and the CISO recently sent an email saying teams can fast-track reviews for well-understood patterns. SSR is already used by three other teams who completed the review in 3 days.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "fp-harder-10", "prompt": "A DevOps team says they can't give developers kubectl access to staging clusters because 'ISO 27001 certification requires strict separation of duties between development and operations.' Their ISO 27001 auditor confirmed during the last audit that developer access to non-production environments is acceptable as long as production access is controlled. The team misinterpreted a general principle as a specific prohibition.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "fp-harder-11", "prompt": "A team is debating whether to use microservices or a monolith for a new greenfield project. The tech lead argues they must use microservices because 'the industry has moved on from monoliths and nobody builds monoliths anymore.' There is no specific technical requirement, scaling need, or organizational constraint driving this decision \u2014 it's a three-person team building an internal tool.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "fp-harder-12", "prompt": "A company's ML team says they can't release model weights publicly because 'the responsible AI policy requires a safety review before any public release.' The policy was adopted after the board reviewed AI risk, and the review process has a documented checklist. The team could complete the review but estimates it would take two weeks.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["first-principles"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "rv-harder-01", "prompt": "A startup CEO sends a company-wide Slack message announcing a major strategic pivot \u2014 shifting from B2B enterprise to consumer SMB \u2014 and asking all teams to stop work on enterprise features immediately. Three senior enterprise sales reps who account for 60% of revenue resign within 48 hours of the announcement, interpreting it as the company abandoning their customer base.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "rv-harder-02", "prompt": "An engineering team changes their primary database from PostgreSQL to MySQL by running a migration script that converts all schemas, data types, and stored procedures. They keep the old PostgreSQL instance running in read-only mode for 30 days as a fallback. The application code has been updated to use MySQL via a new ORM adapter, and the old PostgreSQL adapter code has been deleted from the codebase.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "rv-harder-03", "prompt": "A product team deprecates a free tier that has 50,000 active users, requiring them to upgrade to a paid plan or lose access. The announcement gives 60 days notice. Three months after deprecation, the team decides the churn was too high and wants to restore the free tier \u2014 but the deprecation was covered by TechCrunch and 40,000 of those users have already migrated to a competitor.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "rv-harder-04", "prompt": "A backend team changes the internal API contract between two services by renaming a field from `user_id` to `account_id` in the response JSON. Both services are owned by the same team, deployed together in the same pod, and the change is coordinated so both services deploy simultaneously via a single CI pipeline. No external consumers depend on this internal API.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "rv-harder-05", "prompt": "A platform team flips a feature flag to enable a new authentication flow for 100% of users. The old authentication flow is preserved behind the flag and can be restored by flipping the flag back. The flag change propagates through the CDN within 30 seconds. No data migration or schema change is involved.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "rv-harder-06", "prompt": "A developer merges a pull request that adds a new optional query parameter `?include_archived=true` to a list endpoint. The parameter defaults to false and is purely additive \u2014 all existing API consumers see no change in behavior. If the feature needs to be rolled back, the parameter can simply be ignored by the backend with no client impact.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["reversibility"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Assess this decision before acting. Answer YES if it is a one-way door (hard or impossible to reverse \u2014 costly, destructive, or locks you in). Answer NO if it is a two-way door (cheaply and quickly reversible if it turns out wrong).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "so-harder-01", "prompt": "A ride-sharing company is losing drivers to a competitor that offers guaranteed hourly minimum earnings. The company introduces a similar guarantee: drivers earn at least $25/hour including tips, with the company making up any shortfall. In the first month, driver supply increases 30% and wait times drop. The company expects this to sustainably solve the driver shortage and improve market share.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "so-harder-02", "prompt": "A SaaS company notices that new user activation is low because the onboarding flow requires too many decisions. They redesign the onboarding to make intelligent defaults for every setting, reducing the setup from 12 steps to 2 steps. Activation rates jump from 35% to 65%. The team expects this to also improve long-term retention because users are getting value faster.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "so-harder-03", "prompt": "An engineering organization is frustrated that bugs take too long to fix because teams don't prioritize them. Leadership introduces a policy where every team must fix all known bugs before starting any new feature work. In the first sprint, the bug count drops 40%. Leadership interprets this as the policy working and expects continued improvement in release quality.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "uncalibrated", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "so-harder-04", "prompt": "A social media platform is facing criticism for toxic comments. They deploy an AI moderation system that automatically hides comments scored as toxic above a threshold, with human review only for appealed decisions. Reported toxic comments drop 80% in the first month. The team declares the problem solved and shifts engineering resources to other priorities.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "uncalibrated", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "so-harder-05", "prompt": "An e-commerce company's search results are slow because the search index rebuild runs synchronously on every product update. They move the index rebuild to an async queue that processes updates within 30 seconds, and search results use the slightly-stale index in the meantime. Search latency drops from 2 seconds to 80ms. The company expects this to improve conversion rates, since search speed is a well-known conversion factor.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "uncalibrated", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "so-harder-06", "prompt": "A company's internal documentation is scattered across wikis, Google Docs, and README files, making it hard to find information. The engineering team builds a documentation portal that indexes all sources and provides unified search. They seed it with their own team's docs and ask other teams to contribute. Six months later, 70% of engineering docs are searchable through the portal and support tickets caused by outdated docs have dropped 25%.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": true, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "uncalibrated", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "so-harder-07", "prompt": "A gaming company introduces loot boxes with cosmetic items that can also be earned through gameplay. Players can either grind 10 hours for a rare skin or buy it for $2.99. Revenue increases 40% in the first quarter. The company expects this monetization model to be sustainable because all items are cosmetic and non-competitive.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": ["second-order"], "family": "Conceptual reframes", "type": "target", "target": true, "label": false, "decision_instruction": "Consider the full chain of consequences of the proposed action, not just the immediate effect. Answer YES if, accounting for second- and third-order effects (feedback, incentives, adaptation by others), the action will ACHIEVE its stated goal. Answer NO if downstream/second-order effects will UNDERMINE or reverse the stated goal even though the first-order effect looks positive.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "uncalibrated", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "conc-dist-01", "prompt": "Write a Python function to find the longest common subsequence of two strings using dynamic programming. Include time and space complexity analysis.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "label": null, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "distractor", "calibration_baseline": null, "calibration_band": "uncalibrated", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "conc-dist-02", "prompt": "Explain the difference between TCP and UDP protocols and give three examples of applications that use each.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "label": null, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "distractor", "calibration_baseline": null, "calibration_band": "uncalibrated", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "conc-dist-03", "prompt": "A project manager asks for an estimate of how long it will take to build a user registration system with email verification, password reset, and OAuth integration. Provide a detailed breakdown.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "label": null, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "distractor", "calibration_baseline": null, "calibration_band": "uncalibrated", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "conc-dist-04", "prompt": "Compare React, Vue, and Angular for building a dashboard application with real-time data updates. Which would you recommend and why?", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "label": null, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "distractor", "calibration_baseline": null, "calibration_band": "uncalibrated", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "conc-dist-05", "prompt": "Design a REST API for a library management system. Include endpoints for books, patrons, checkouts, and reservations. Specify request/response formats.", "source": "authored-harder-conceptual", "source_file": "evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl", "skill_fit": [], "family": "Conceptual reframes", "type": "distractor", "target": false, "label": null, "decision_instruction": "A constraint is stated as a reason something can't be done. Answer YES if the constraint is a FUNDAMENTAL limit (physics, math, hard external/legal requirement) that genuinely cannot be circumvented. Answer NO if it is merely a CONVENTION, habit, or current-implementation assumption that could be challenged or changed.", "mode": "distractor", "calibration_baseline": null, "calibration_band": "uncalibrated", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}diff --git a/evals/datasets/calibrated/conceptual-reframes.manifest.json b/evals/datasets/calibrated/conceptual-reframes.manifest.jsonindex 60ee386..c2d9e74 100644--- a/evals/datasets/calibrated/conceptual-reframes.manifest.json+++ b/evals/datasets/calibrated/conceptual-reframes.manifest.json@@ -18,7 +18,7 @@"triz"],"value_surface": "paired reasoning quality",- "source": "Combined from 7 source files. authored: first-principles-constraint, reversibility-doors, second-order-consequence, cynefin-classify, harder-conceptual (m3-harder-data); external: debias-authored, glm-distractor",+ "source": "Combined from 7 source files. authored: first-principles-constraint, reversibility-doors, second-order-consequence, cynefin-classify; harder-conceptual items from evals/datasets/calibrated/conceptual-reframes-harder-pool.jsonl (generated by evals/datasets/calibrated/build-harder-conceptual-systems-pools.js, m3-harder-data); external: debias-authored, glm-distractor","leakage": "First-principles-constraint, reversibility-doors, second-order-consequence, and harder-conceptual are locally authored datasets — zero external leakage. Debias-authored items are locally authored cognitive bias scenarios with anchoring bias types. Some items may resemble classic cognitive bias examples from psychology literature which models may have encountered. GLM distractor items may have training data overlap. Overall leakage risk is low because locally authored items dominate the target set.\n\nSource breakdown: 1 external dataset(s) with potential training-data leakage; 5 authored dataset(s) with zero external leakage; ","label": {"definition": "Binary decision quality on conceptual reframing tasks: whether the model correctly identifies whether a constraint is fundamental (true) or merely conventional (false); whether a decision is a one-way door (true) or two-way door (false); whether a proposed action's second-order effects undermine its stated goal (false) or the goal is achieved (true). Harder items (m3-harder-data) target subtle distinctions: constraints that look fundamental but are conventional, doors with hidden irreversibility, and second-order effects with complex interactions. Off-target lookalikes included for discrimination.",@@ -38,7 +38,8 @@0.7],"in_band": false,- "exemption": "CEILING-NEEDS-HARDER-DATA. After sourcing 25 genuinely harder items (subtle first-principles constraints where fundamental-vs-conventional is non-obvious, nuanced reversibility doors, and complex second-order scenarios), claude-sonnet-4-6 still achieves 92% baseline accuracy on placebo/baseline condition (25 labeled target items at K_TRIALS=5). Only 1 item (fp-harder-01, PCI-DSS constraint) landed in-band at 60%. All conventional-constraint, reversibility, and second-order items remained ceiling. The solver demonstrates near-perfect pattern recognition for constraint-classification and door-classification tasks even on intentionally ambiguous cases. This family requires genuinely HARDER items — potentially multi-step reasoning tasks rather than binary classification — to reach the [0.40,0.70] band. Per Invariant #4, this is NOT a kill verdict.",+ "aggregate_in_band": false,+ "exemption": "CEILING-NEEDS-HARDER-DATA. After sourcing 25 genuinely harder items (subtle first-principles constraints where fundamental-vs-conventional is non-obvious, nuanced reversibility doors, and complex second-order scenarios), claude-sonnet-4-6 still achieves ~92% aggregate baseline accuracy on placebo/baseline condition (25 labeled target items at K_TRIALS=5). Only 1 item (fp-harder-01, PCI-DSS constraint) landed in-band at 60%. All conventional-constraint, reversibility, and second-order items remained ceiling. The ~8pp headroom leaves insufficient room for the target ≥5pp lift with statistical power. The solver demonstrates near-perfect pattern recognition for constraint-classification and door-classification tasks even on intentionally ambiguous cases. This family requires genuinely HARDER items — potentially multi-step reasoning tasks rather than binary classification — to reach the [0.40,0.70] band. Per Invariant #4, this is NOT a kill verdict. Per library/harness.md aggregate rule: aggregate baseline ~92% = genuine NO-POWER ceiling (only ~8pp headroom).","status": "measured_sample","calibrated_items": 25,"in_band_items": 1,@@ -63,13 +64,13 @@}},"item_counts": {- "total": 234,- "target": 155,- "near_miss": 30,- "distractor": 49,+ "total": 224,+ "target": 150,+ "near_miss": 26,+ "distractor": 48,"ambiguous": 0,"decisive": 224,- "calibration_only": 10,+ "calibration_only": 0,"exempt": 0,"calibrated": 25,"harder_pool": 30@@ -87,7 +88,10 @@"satisfied": true,"decisive_targets": 150,"decisive_distractors": 74,- "ambiguous_in_split": 0+ "decisive_near_miss": 26,+ "ambiguous_in_split": 0,+ "calibration_only_in_split": 0,+ "total_decisive": 224},"created": "2026-06-06","frozen": true,diff --git a/evals/datasets/calibrated/systems-product-strategy-pairwise-decisive.jsonl b/evals/datasets/calibrated/systems-product-strategy-pairwise-decisive.jsonlindex acde7d0..78d910e 100644--- a/evals/datasets/calibrated/systems-product-strategy-pairwise-decisive.jsonl+++ b/evals/datasets/calibrated/systems-product-strategy-pairwise-decisive.jsonl@@ -1,94 +1,117 @@-{"id":"toc-01","original_id":"toc-01","prompt":"A REST API request pipeline has four stages: Auth (2ms), Business Logic (8ms), Database Query (320ms), Response Serialization (5ms). The team wants to optimize the Business Logic stage by rewriting it in a more efficient language. Would this meaningfully improve end-to-end latency?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-02","original_id":"toc-02","prompt":"A data ingestion pipeline processes records through five stages: Read from S3 (0.5s/batch), Deserialize (0.2s), Validate (0.1s), Write to Kafka (4.8s), Acknowledge (0.05s). The team proposes parallelizing the Validate stage across 8 cores. Would this meaningfully improve end-to-end throughput?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-04","original_id":"toc-04","prompt":"An image processing pipeline has these stages: Upload to S3 (0.3s), Queue Message (0.1s), ML Inference on GPU (12s), Resize Thumbnails (0.8s), Save Results (0.4s). The team proposes optimizing the Resize Thumbnails stage with a faster library. Would this improve end-to-end latency per image?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-07","original_id":"toc-07","prompt":"A report generation service runs: Fetch user permissions (30ms), Execute analytics SQL query (4,200ms), Format results to JSON (60ms), Render PDF (110ms). The team proposes adding a Redis cache for user permissions to reduce that step to 2ms. Would this meaningfully improve the total report generation time?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-08","original_id":"toc-08","prompt":"A batch ETL job runs nightly with these timings: Extract from source DB (8 min), Transform records (42 min), Load into warehouse (6 min), Run post-load validations (4 min). The team wants to speed up the Extract step by adding read replicas. Would this cut overall job duration?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-09","original_id":"toc-09","prompt":"A web scraping pipeline processes URLs through: DNS resolution (15ms), TCP connect (20ms), TLS handshake (25ms), HTTP fetch (1,800ms avg), HTML parse (35ms), Write to DB (18ms). The team wants to run the HTML parse step on a faster machine. Would this noticeably reduce per-URL latency?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-10","original_id":"toc-10","prompt":"A video transcoding pipeline has these per-clip stages: Download Source (5s), Audio Decode (2s), Video Transcode (95s), Mux Streams (3s), Upload Output (8s). The team proposes switching to a faster audio codec to cut Audio Decode to 0.5s. Would this improve throughput or end-to-end time per clip?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-11","original_id":"toc-11","prompt":"A microservice handles requests through: Load Balancer (1ms), Auth Token Validation (3ms), Third-Party KYC API Call (2,600ms), Business Logic (15ms), DB Write (12ms). The team proposes parallelizing the KYC API call across multiple providers and using cached results, cutting its average to 120ms. Would this improve response time for users?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-12","original_id":"toc-12","prompt":"A checkout pipeline executes: Cart Validation (40ms), Inventory Lock (35ms), Payment Charge (850ms), Order Record Insert (28ms), Confirmation Email Enqueue (10ms). The team proposes switching to a payment provider with a p99 latency of 90ms, cutting the Payment Charge step from 850ms to 90ms. Would this substantially reduce checkout latency?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-14","original_id":"toc-14","prompt":"An order fulfillment pipeline has these stages per order: Order Received (0.1s), Fraud Check via external API (3.9s), Pick and Pack (0.5s), Shipping Label Generation (0.3s), Carrier Handoff (0.2s). The team proposes replacing the external fraud API with an in-house ML model that runs in 80ms. Would this improve end-to-end order fulfillment latency?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-16","original_id":"toc-16","prompt":"A REST API handles requests with these sequential steps: Request Parsing (3ms), Auth (5ms), DB Query (290ms), Response Formatting (4ms). The team proposes adding a read replica and query index to cut the DB Query stage to under 20ms. Would this dramatically improve the API's response times?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-17","original_id":"toc-17","prompt":"A CI/CD pipeline runs: Checkout (12s), Install Dependencies (18s), Unit Tests (8 min 45s), Integration Tests (2 min), Build (40s), Deploy (25s). The team proposes parallelizing the Unit Tests across 10 workers, which would cut the step to under 1 minute. Would this meaningfully shorten total pipeline run time?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-19","original_id":"toc-19","prompt":"A software delivery process has these average stage durations: Requirements (1 day), Development (3 days), Code Review (11 days — queue backlog is 7 days), QA (1.5 days), Release (0.5 days). The team proposes assigning two more engineers specifically to code review. Would this increase feature release throughput?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-20","original_id":"toc-20","prompt":"An event streaming pipeline has throughput rates: Kafka Consumer (50,000 msg/s), Deserialize (45,000 msg/s), Enrich via External API (1,800 msg/s), Aggregate (40,000 msg/s), Write to Postgres (38,000 msg/s). The team proposes adding connection pooling and batching to the External API calls to raise its throughput to 20,000 msg/s. Would this improve overall pipeline throughput?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-21","original_id":"toc-21","prompt":"A document processing pipeline runs: OCR Scan (2s), Text Extraction (1.8s), NLP Entity Recognition (24s), Index to Search (1.2s), Notify User (0.3s). The team proposes replacing the NLP model with a faster distilled version that achieves the same accuracy in 3s. Would this meaningfully reduce per-document processing time?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-22","original_id":"toc-22","prompt":"A video upload pipeline processes each file: Client Upload to CDN (varies), Virus Scan (8s), Video Transcode (5m 20s), Thumbnail Generation (12s), Metadata Index (2s). The team proposes running Thumbnail Generation in parallel with Transcode. Would this reduce the total time a user waits for their video to be available?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-23","original_id":"toc-23","prompt":"A SaaS onboarding pipeline has per-customer steps: Sales Handoff (0.5 days), Legal Contract Review (9 days — average wait 6 days), Technical Integration (2 days), Training (1 day), Go-Live (0.5 days). The team proposes adding a dedicated legal reviewer to reduce contract review to 1 day. Would this reduce total time-to-live for new customers?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-edge throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-24","original_id":"toc-24","prompt":"A real-time analytics query pipeline runs: Parse SQL (5ms), Check Permissions (8ms), Execute OLAP Query (4,500ms), Format Response (12ms), Compress Payload (18ms). The team proposes switching to a columnar engine that reduces OLAP Query time to 200ms. Would this substantially reduce user-perceived query latency?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-25","original_id":"toc-25","prompt":"A fraud detection pipeline evaluates each transaction: Rule Engine (10ms), ML Feature Extraction (22ms), External Credit Bureau Lookup (2,100ms), Score Aggregation (8ms), Decision Write (6ms). The team proposes caching Credit Bureau results by SSN with a 24-hour TTL, cutting average lookup time to 15ms for repeat customers. Would this materially reduce fraud check latency?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":true,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-26","original_id":"toc-26","prompt":"A nightly data warehouse refresh has stages: Extract from Postgres (14 min), Transform (62 min), Load to Snowflake (10 min), Run dbt Models (8 min). The team proposes upgrading the Snowflake warehouse tier to cut Load time to 3 min. Would this meaningfully shorten total refresh duration?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-27","original_id":"toc-27","prompt":"A background job pipeline processes user export requests: Enqueue Job (0.1s), Fetch User Data from DB (5,800ms), Serialize to CSV (200ms), Compress File (180ms), Upload to S3 (400ms), Email User Link (100ms). The team wants to optimize the Compress File step with a faster algorithm. Would this significantly reduce the time users wait for their export?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-28","original_id":"toc-28","prompt":"A hardware test pipeline runs: Flash Firmware (45s), Boot Device (12s), Run Calibration Suite (8 min 30s), Log Results (5s), Package for Shipping (30s). The team proposes upgrading the USB flasher to cut firmware flash time to 10s. Would this improve overall throughput of units through the test line?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-29","original_id":"toc-29","prompt":"A Kubernetes pod startup sequence involves: Image Pull (3s), Init Containers (8s), Application Boot (94s — slow Spring Boot cold start), Readiness Probe (5s), Traffic Admission (1s). The team proposes pre-warming the image pull by caching it on every node. Would this reduce the time from deployment trigger to traffic serving?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"toc-30","original_id":"toc-30","prompt":"A machine learning training pipeline runs per epoch: Data Loading (8s), Augmentation (5s), Forward Pass (310s), Backward Pass (290s), Checkpoint Save (12s). The team proposes optimizing the Data Loading step with async prefetching to reduce it to 1s. Would this reduce total training time per epoch?","source":"theory-of-constraints-bottleneck","source_file":"authored/theory-of-constraints-bottleneck.jsonl","skill_fit":["theory-of-constraints"],"family":"Systems/product/strategy pairwise","type":"target","target":true,"label":false,"decision_instruction":"You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck — some other stage dominates).","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-01","original_id":"margin-safety-01","prompt":"A team expects peak API traffic of 800 req/s during an upcoming product launch. Historical launches show spikes of 3–5x the predicted peak, and a failure would mean lost revenue plus SLA breach penalties. The team provisions 2,000 req/s of capacity — 2.5x the expected peak.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-02","original_id":"margin-safety-02","prompt":"A data warehouse team expects to ingest 500GB of data per month. Storage is charged at $0.023/GB and data loss is not a concern because source data can be re-ingested at any time. They provision 550GB — a 10% buffer. Scaling additional storage takes less than 30 seconds via an API call.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-03","original_id":"margin-safety-03","prompt":"A payments company processes an average of 1,200 transactions per second on normal days. Black Friday historically generates 4–6x normal volume, and a payment processing outage means direct revenue loss of roughly $50,000 per minute. The team provisions capacity for 1,800 transactions per second for the holiday period.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-04","original_id":"margin-safety-04","prompt":"A team is setting an HTTP client timeout for calls to a third-party shipping API. The shipping API's documented p99 latency is 400ms, and their SLA guarantees responses within 800ms. Calls that time out result in a failed order checkout with user-visible errors. The team sets a 1,000ms timeout.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-05","original_id":"margin-safety-05","prompt":"A startup's engineering team estimates a new feature will take 4 weeks to build. This is the team's first time working with the third-party payments SDK, and the feature is on the critical path for a contractually committed customer demo. They commit to a 5-week delivery date — 25% buffer.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-06","original_id":"margin-safety-06","prompt":"A Postgres connection pool is configured with a maximum of 100 connections. Under normal load the service uses 45 connections. During monthly billing runs a secondary process uses up to 30 additional connections simultaneously. An unexpected secondary batch job will exhaust the pool only if both peaks coincide, which has never happened. The cost of pool exhaustion is a cascade of 500 errors to all users.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-07","original_id":"margin-safety-07","prompt":"A video transcoding service processes an average of 200 jobs per hour. Job volume has been growing 15% month-over-month for six months with no sign of slowing. Each job consumes roughly the same compute. The team provisions for 600 jobs per hour — 3x current average — and plans to review in three months. Delays cause content to miss publishing windows, resulting in lost ad revenue.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-08","original_id":"margin-safety-08","prompt":"A team is planning storage for a new audit log system. Legal requires all audit logs to be retained for 7 years, and any gap is a compliance violation. The team's data model estimates 80GB per year based on current transaction volume. They provision 100GB of storage — enough for about 15 months — with a plan to add capacity before it fills.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-09","original_id":"margin-safety-09","prompt":"A ride-sharing company allocates a monthly infrastructure budget of $120,000. Actual spend has been stable at $95,000–$100,000 per month for the past six months with no new services planned. The allocated budget of $120,000 represents a 20–26% buffer over observed spend.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-10","original_id":"margin-safety-10","prompt":"A hospital patient-record API is expected to serve 500 concurrent users during the busy morning shift. Load tests show the system becomes degraded at 520 concurrent users. Patient care workflows depend on the system; downtime means nurses fall back to paper, which introduces medication error risk. The team deploys hardware supporting 550 concurrent users.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-11","original_id":"margin-safety-11","prompt":"A Kafka consumer group processes 50,000 messages per hour at steady state. The upstream producer can burst to 200,000 messages per hour during end-of-day batch closes, lasting about 45 minutes. Consumer lag during burst causes downstream reports to be delayed, impacting trading desks. The team sizes consumers for 250,000 messages per hour sustained throughput.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-12","original_id":"margin-safety-12","prompt":"A team is deploying a new machine learning inference service. Benchmark tests show the model needs 6GB of GPU VRAM at maximum batch size. The team selects a GPU instance with exactly 8GB VRAM. They note that framework overhead, CUDA context, and driver memory are not accounted for in the benchmark figure.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-13","original_id":"margin-safety-13","prompt":"A software team is migrating a monolith to microservices. The project manager estimates 6 months based on a rough breakdown of work. The migration touches every team in the company, requires coordination with three external vendors, and has no prior comparable project as reference. The team commits to a 12-month timeline with a formal review at month 6.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-14","original_id":"margin-safety-14","prompt":"An IoT platform expects 10,000 devices to connect simultaneously after a firmware update pushes to all deployed units. Historical firmware pushes show actual simultaneous reconnection peaks reach 8x the device count (due to reconnect storms). A reconnection storm exceeding broker capacity causes devices to go permanently offline until manually reset. The broker is provisioned for 15,000 simultaneous connections.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-15","original_id":"margin-safety-15","prompt":"A SaaS company's search index uses 40% of available disk space under current data volume. Data volume is projected to grow 20% per year, and the index requires immediate disk space to function — when disk hits 90% full the service stops indexing new documents. The team plans to add storage when usage hits 70% utilization, giving roughly 1.5 years of runway.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-16","original_id":"margin-safety-16","prompt":"A team's API rate limit with their geolocation provider is 10,000 lookups per day. Current usage averages 6,500 lookups per day. A marketing campaign launching next month is expected to bring a 2–3x traffic increase lasting two weeks. Exceeding the rate limit returns errors on the address validation step of checkout, blocking all new orders.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-17","original_id":"margin-safety-17","prompt":"A team running a high-traffic CDN edge node configures a memory cache with a maximum size of 32GB. Profiling shows the working set of hot content peaks at 20GB during prime time, with cold-start periods using as little as 5GB. Cache evictions are graceful — a miss falls through to origin with a latency penalty but no data loss. They size at 32GB for comfortable headroom.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-18","original_id":"margin-safety-18","prompt":"A developer estimates a new authentication refactor will take 10 days. The team has done this type of refactor three times in the past year, always within 8–12 days. The change is internal only, has no customer-facing deadline, and can be rolled back instantly if problems arise. They commit to 11 days.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-19","original_id":"margin-safety-19","prompt":"A team deploys a Redis cache cluster sized for 8GB of memory. Current cache usage is 7.2GB. Keys are set with TTLs of 24 hours but the product roadmap includes launching three new features over the next two months, each of which is expected to add cached objects. Cache eviction uses LRU, and evicting hot keys causes a cascade of DB queries that has previously caused an incident.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-21","original_id":"margin-safety-21","prompt":"A team's incident response SLA commits to acknowledging critical incidents within 15 minutes and resolving them within 4 hours. They have one primary on-call engineer and no backup. The primary has missed acknowledgment in 3 of the last 20 incidents due to being in a meeting or asleep in a different time zone.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-22","original_id":"margin-safety-22","prompt":"An analytics pipeline writes results to an S3 bucket. Each run produces approximately 2GB. The pipeline runs daily, and 90 days of results are retained before lifecycle rules delete old files. The team allocates 250GB of S3 capacity via a quota. S3 storage is elastic and quota overages are immediately rechargeable at standard rates.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-23","original_id":"margin-safety-23","prompt":"A team is launching a developer API with a free tier capped at 1,000 calls per month per user. Backend cost per call is $0.001. They provision infrastructure for 5 million total free-tier calls per month. They have 3,000 free-tier users signed up and expect 10,000 by end of quarter. At full expected adoption and full usage, demand would be 10 million calls per month.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-24","original_id":"margin-safety-24","prompt":"A team configures a circuit breaker to open after 5% error rate sustained over 10 seconds. Under normal conditions the service runs at 0.1% error rate. The error rate threshold, window size, and recovery behavior were validated in production-like load tests. The circuit breaker configuration matches what was tested.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-25","original_id":"margin-safety-25","prompt":"An e-commerce site expects 1,000 concurrent sessions during a flash sale. Load testing shows the application server handles 1,050 concurrent sessions before response times degrade past 3 seconds. A degraded checkout experience during the flash sale means customers abandon carts. The team does not add additional capacity, relying on auto-scaling that takes 4–6 minutes to provision new instances.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-26","original_id":"margin-safety-26","prompt":"A team building a new internal reporting tool estimates the backend will need 4 vCPUs and 8GB RAM under expected query load. The tool is used only during business hours by 15 analysts. They provision an 8 vCPU, 16GB RAM instance, doubling both. If the tool becomes slow, analysts can wait without business impact.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-27","original_id":"margin-safety-27","prompt":"A satellite telemetry ingestion pipeline must store all raw data permanently for regulatory reasons; any data gap is a compliance event with significant fines. The system ingests 10TB per year with no growth expected. The team provisions 12TB — 20% buffer — on a single storage array with no redundancy and no offsite backup.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-28","original_id":"margin-safety-28","prompt":"A team uses a managed Elasticsearch cluster for product search. Normal indexing throughput is 2,000 documents per second. The team expects a re-index event during a schema migration that will run at 4,000 documents per second for approximately 6 hours. They temporarily scale the cluster to handle 6,000 documents per second for the migration window. A search outage means the product catalog is unavailable to customers.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-29","original_id":"margin-safety-29","prompt":"A team provisions a job queue worker fleet sized to drain 100 jobs per minute — exactly matching the average production ingest rate. The queue is used for sending transactional emails; if the queue backs up significantly, emails arrive hours late. Historical data shows ingest spikes to 300 jobs per minute during large user-triggered events such as password reset floods.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":false,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"margin-safety-30","original_id":"margin-safety-30","prompt":"A team committing to a quarterly OKR estimates a feature will be delivered in 6 weeks, based on a well-understood problem space, a team that has built nearly identical features before, and no external dependencies. They commit to 8 weeks in the OKR. Missing the OKR has minor internal visibility consequences and the feature can ship one quarter late without customer impact.","source":"margin-of-safety-provision","source_file":"authored/margin-of-safety-provision.jsonl","skill_fit":["margin-of-safety"],"family":"Systems/product/strategy pairwise","type":"nearMiss","target":false,"label":true,"decision_instruction":"You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.","mode":"binary-decision","calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true}-{"id":"glm-distractor-8","original_id":"glm-distractor-8","prompt":"How can I convert all the lowercase letters in a string to uppercase using Python?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-12","original_id":"glm-distractor-12","prompt":"Implement quick sort to sort the given array of integers. Write corresponding code in Python. [9, 5, 8, 1, 3, 6, 7]","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-13","original_id":"glm-distractor-13","prompt":"Zoe wants to go on the field trip to Washington DC with her middle school this spring and the cost is $485. Her grandma gave her $250 toward her fees and she must earn the rest by selling candy bars. She makes $1.25 for every candy bar she sells. How many candy bars does Zoe need to sell to earn the money for the trip?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-16","original_id":"glm-distractor-16","prompt":"Can you also tell me the date and time of the Indiana Comic Convention?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-17","original_id":"glm-distractor-17","prompt":"Create an algorithm to rotate an array by one element. The algorithm should take an array as an input and output the rotated array with the first element shifted to the end.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-18","original_id":"glm-distractor-18","prompt":"Generate a correct Python program for plotting bar chart for the following data.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-19","original_id":"glm-distractor-19","prompt":"Given a non-negative decimal integer $x$, convert it to binary representation $b$ of 32 bits. Then, print the result of the following operations to $b$ respecitvely. * Inversion: change the state of each bit to the opposite state * Logical left shift: shift left by 1 * Logical right shift: shift right by 1 Constraints * $0 \\leq x \\leq 2^{32} - 1$ Input The input is given in the following format. $x$ Output Print the given bits, results of inversion, left shift and right shift in a line respectively. Examples Input 8 Output 00000000000000000000000000001000 11111111111111111111111111110111 00000000000000000000000000010000 00000000000000000000000000000100 Input 13 Output 00000000000000000000000000001101 11111111111111111111111111110010 00000000000000000000000000011010 00000000000000000000000000000110 Write Python code to solve the problem. Please place the solution code in the following format: ```python # Your solution code here ```","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-21","original_id":"glm-distractor-21","prompt":"Create a Python program that fetches the user's current time and sets a reminder after 30 minutes.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-22","original_id":"glm-distractor-22","prompt":"Could you delineate a methodology, utilising the Python programming language, to extract the middle value, labeled as the median, from an unorganized collection of integral numbers? Numbers = [12, 7, 21, 3, 9]","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-42","original_id":"glm-distractor-42","prompt":"Write a Python3 function that returns the Cargo home directory path by first checking the CARGO_HOME environment variable. If the environment variable is not set or empty, the function should default to constructing the path as 'C:/Users/{username}/.cargo' using the current user's username. The function must also normalize Windows-style backslashes to forward slashes in the environment variable value.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-46","original_id":"glm-distractor-46","prompt":"Create a Python 3 script that uses the clustergrammer library to load a matrix from stdin, perform optional filtering and normalization steps (such as sum-based filtering, z-score normalization, top-N selection, and threshold filtering), generate a hierarchical clustering visualization using cosine distance, and then output the resulting network data in JSON format for frontend visualization. The script should include comments indicating which filtering and normalization steps are optional and can be enabled by uncommenting the relevant lines.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-48","original_id":"glm-distractor-48","prompt":"Solve the inequality: 8x - 9 < 2. Express the solution in interval notation.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-50","original_id":"glm-distractor-50","prompt":"A toy store manager received a large order of Mr. Slinkums just in time for the holidays. The manager places $20\\%$ of them on the shelves, leaving the other 120 Mr. Slinkums in storage. How many Mr. Slinkums were in this order?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-51","original_id":"glm-distractor-51","prompt":"Write a python function to check if a given file is in NOAA ACE format by verifying that the first line starts with ':' and the third line contains 'NOAA'. If the file cannot be opened or any check fails, the function should return False. Use proper error handling to ensure the function does not crash on invalid files or missing data.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-69","original_id":"glm-distractor-69","prompt":"Write a program that allows users to search for phone numbers by name in a phone book. Implement the phone book using a custom data structure that you create. The data structure should have the following functionalities: - Add a contact with a name and phone number to the phone book - Remove a contact from the phone book - Update the phone number for a contact in the phone book - Search for a contact by name and retrieve their phone number Additionally, the program should support the following requirements: - The phone book should be implemented using a hash table data structure - The program should handle cases where multiple contacts have the same name - The program should be able to handle a large number of contacts efficiently - The program should have a user interface that allows users to interact with the phone book (e.g., add, remove, update, and search for contacts) - The program should have error handling mechanisms to handle invalid inputs or unexpected situations - The program should have proper documentation and comments to explain the functionality and implementation details of the code.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-74","original_id":"glm-distractor-74","prompt":"Produce a program using Python that visualizes the given function.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-81","original_id":"glm-distractor-81","prompt":"How can I generate a regular expression that matches any two or more consecutive numbers?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-83","original_id":"glm-distractor-83","prompt":"Write a python 3 script that performs simple linear regression analysis on gas expenditure data using scikit-learn. The script should load data from \"data1.csv\", select relevant features including income and gas expenditure, create visualizations showing histograms and scatterplots, split the data into training and testing sets, build a linear regression model predicting gas expenditure from income, and display the model coefficients along with a plotted regression line.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-84","original_id":"glm-distractor-84","prompt":"Write a python 3 Django management command that classifies unassigned faces using ensemble predictions from two SVM models (small and big) with probability averaging, and prints classification results for the first 1000 unassigned faces, including predicted labels, probabilities, and corresponding person objects from the database.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-196","original_id":"glm-distractor-196","prompt":"Create a python 3 script that demonstrates two different plotting APIs using PyQt4: a procedural approach with PlotWidget and an object-oriented approach with a custom Plot class. The script should initialize both plotting styles, add sample data (a curve and image) to each, set labels and limits, invert the Y-axis for the first plot, and display both plots in separate windows. Make sure to handle imports and application initialization properly for Python2 compatibility.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-199","original_id":"glm-distractor-199","prompt":"Write a Python 3 function that tests whether the neohub class can be properly instantiated, ensuring the class initialization works correctly without any errors. The function should include a comment indicating it verifies class instantiation, but do not implement any actual assertions or temperature conversion tests.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-200","original_id":"glm-distractor-200","prompt":"Please answer the following question: - Gather the ingredients for the cake - Mix the flour, sugar, egg, and water to make batter - Pour the mixture into a cake pan - Place the cake in the oven to cook - Take the cake out when it is cooked through - Let the cake cool What might be the last step of the process? A:","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-201","original_id":"glm-distractor-201","prompt":"Can you provide a Python code snippet to convert XML to JSON?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-203","original_id":"glm-distractor-203","prompt":"Write a Python3 script that downloads and unpacks tracking results from Google Drive for various trackers like PyTracking-based and external trackers. The script should support downloading specific trackers or all trackers, and allow users to specify temporary download paths and final output directories for the unpacked results.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-214","original_id":"glm-distractor-214","prompt":"Create a python 3 program using the turtle module that draws a series of nested squares rotating at different angles. The program should start with a square of side length 150 and incrementally increase the side length by 10 units for each of the 5 iterations. Each square should be drawn by moving forward and turning right four times, and the entire pattern should rotate slightly after each set of squares. Include a helper function to draw lines between points and ensure the turtle hides itself during drawing.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-216","original_id":"glm-distractor-216","prompt":"How can I generate a set of random numbers without any duplicates in Python?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-224","original_id":"glm-distractor-224","prompt":"Create a Python script that sets the value of a variable from the command-line, but the value must be encrypted using a custom encryption algorithm before being assigned to the variable. Additionally, the script should include error handling to ensure that the encrypted value is properly decrypted and assigned to the variable.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-225","original_id":"glm-distractor-225","prompt":"A satellite orbits Earth at an altitude of 1501 km. Calculate its orbital velocity and period. (Earth mass = 5.97 × 10²⁴ kg, Earth radius = 6371 km)","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-227","original_id":"glm-distractor-227","prompt":"Compute the quotient of 17708 and 9880. Show your work.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-236","original_id":"glm-distractor-236","prompt":"def division_while_loop(dividend, divisor): \"\"\" This exercise focuses on division using a while loop. Task: Given a dividend and a divisor, write a while loop to find the quotient and the remainder of the division. Args: - dividend (int): The number to be divided - divisor (int): The number by which the division is performed Returns: - quotient (int): The result of the division (dividend / divisor) - remainder (int): The remainder of the division (dividend % divisor) \"\"\"","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-241","original_id":"glm-distractor-241","prompt":"Create a python 3 class called `BalanceSheet` that takes a `Portfolio` object (with default to a new `Portfolio()` instance) and provides methods to generate structured data representations. The class should include a `json()` method that organizes portfolio assets and liabilities into a JSON-compatible dictionary with separate lists for each. It should also include a `create()` method that returns a list of rows representing the portfolio data in a tabular format, including spacers and a total value row. Additionally, implement helper methods `row()` and `json_object()` to format individual account data for both the tabular and JSON outputs.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-249","original_id":"glm-distractor-249","prompt":"Write a python function to find the frequency of each character in a given string and return a dictionary with characters as keys and their frequencies as values. Solve the problem starting with the provided function header. Function header: ``` def get_char_frequency(string): ``` Please place the solution code in the following format: ```python # Your solution code here ```","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-261","original_id":"glm-distractor-261","prompt":"Create a function to check if an array contains any duplicate elements, while also ensuring that the array elements are sorted in ascending order and that the length of the array is less than or equal to 100. Additionally, the function should check if the array contains any negative integers, and if so, it should return False. The function should return a boolean value indicating the presence of duplicate elements. Example: Input: [1, 2, 3, 3, 4, 5, 5] Output: True","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-262","original_id":"glm-distractor-262","prompt":"You are working on an API for a space-themed game and need to create a class that handles the case when a specific planet for a character is not found. You need to define a class called GetCharactersCharacterIdPlanetsPlanetIdNotFound with the following attributes and methods: Attributes: - error: a string to store the error message Methods: - __init__: a constructor method to initialize the error attribute - to_dict: a method to return the model properties as a dictionary - to_str: a method to return the string representation of the model - __repr__: a method for printing and pretty-printing the object - __eq__: a method to check if two objects are equal - __ne__: a method to check if two objects are not equal You need to implement the class and its methods as described above.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-264","original_id":"glm-distractor-264","prompt":"How can I categorize data into two bins, with the labels \"category A\" and \"category B\"?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-267","original_id":"glm-distractor-267","prompt":"Given points A(14, -8) and B(-1, -13), find the slope of the line passing through these two points.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-268","original_id":"glm-distractor-268","prompt":"Create a python 3 class named Objects that initializes with several instance attributes including cloud set to None, score set to 0, score_type set to 'CNNSeg', type set to None, and type_probs set to None.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-269","original_id":"glm-distractor-269","prompt":"What is the code to construct a Recurrent Neural Network (RNN) using TensorFlow with an input size of 50 values and an output size of 10 values?","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-270","original_id":"glm-distractor-270","prompt":"Create a python 3 class called CompileGLSL that extends Module for compiling GLSL shader files into SPIR-V format using the glslangValidator tool. The class should define required tools, specify output file names with .spv extension, and implement a process method that runs the validator with appropriate stage flags (-S vert for vertex shaders and -S frag for fragment shaders) while handling directory creation and returning the subprocess return code.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-274","original_id":"glm-distractor-274","prompt":"Calculate the pH of a 0.545 M solution of HCl. Show your calculation step by step.","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}-{"id":"glm-distractor-275","original_id":"glm-distractor-275","prompt":"Write a Python 3 program that prompts the user to enter three integer numbers and then calculates and displays their sum. The program should use input() to read the numbers from the user and print the result in the format \"Sum = [value]\".","source":"glm-distractor","source_file":"external/glm-distractor.jsonl","skill_fit":[],"family":"Systems/product/strategy pairwise","type":"distractor","target":false,"mode":"distractor","license":"Apache-2.0 (GLM-5.1 distilled)","decision_instruction":"Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}","eval_mode":"judge","label":null,"calibration_baseline":null,"calibration_band":"estimated","calibration_estimated":true,"calibration_note":"Distractor - included for false-positive measurement. No ground-truth label for objective scoring."}+{"id": "toc-01", "original_id": "toc-01", "prompt": "A REST API request pipeline has four stages: Auth (2ms), Business Logic (8ms), Database Query (320ms), Response Serialization (5ms). The team wants to optimize the Business Logic stage by rewriting it in a more efficient language. Would this meaningfully improve end-to-end latency?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-02", "original_id": "toc-02", "prompt": "A data ingestion pipeline processes records through five stages: Read from S3 (0.5s/batch), Deserialize (0.2s), Validate (0.1s), Write to Kafka (4.8s), Acknowledge (0.05s). The team proposes parallelizing the Validate stage across 8 cores. Would this meaningfully improve end-to-end throughput?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-04", "original_id": "toc-04", "prompt": "An image processing pipeline has these stages: Upload to S3 (0.3s), Queue Message (0.1s), ML Inference on GPU (12s), Resize Thumbnails (0.8s), Save Results (0.4s). The team proposes optimizing the Resize Thumbnails stage with a faster library. Would this improve end-to-end latency per image?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-07", "original_id": "toc-07", "prompt": "A report generation service runs: Fetch user permissions (30ms), Execute analytics SQL query (4,200ms), Format results to JSON (60ms), Render PDF (110ms). The team proposes adding a Redis cache for user permissions to reduce that step to 2ms. Would this meaningfully improve the total report generation time?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-08", "original_id": "toc-08", "prompt": "A batch ETL job runs nightly with these timings: Extract from source DB (8 min), Transform records (42 min), Load into warehouse (6 min), Run post-load validations (4 min). The team wants to speed up the Extract step by adding read replicas. Would this cut overall job duration?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-09", "original_id": "toc-09", "prompt": "A web scraping pipeline processes URLs through: DNS resolution (15ms), TCP connect (20ms), TLS handshake (25ms), HTTP fetch (1,800ms avg), HTML parse (35ms), Write to DB (18ms). The team wants to run the HTML parse step on a faster machine. Would this noticeably reduce per-URL latency?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-10", "original_id": "toc-10", "prompt": "A video transcoding pipeline has these per-clip stages: Download Source (5s), Audio Decode (2s), Video Transcode (95s), Mux Streams (3s), Upload Output (8s). The team proposes switching to a faster audio codec to cut Audio Decode to 0.5s. Would this improve throughput or end-to-end time per clip?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-11", "original_id": "toc-11", "prompt": "A microservice handles requests through: Load Balancer (1ms), Auth Token Validation (3ms), Third-Party KYC API Call (2,600ms), Business Logic (15ms), DB Write (12ms). The team proposes parallelizing the KYC API call across multiple providers and using cached results, cutting its average to 120ms. Would this improve response time for users?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-12", "original_id": "toc-12", "prompt": "A checkout pipeline executes: Cart Validation (40ms), Inventory Lock (35ms), Payment Charge (850ms), Order Record Insert (28ms), Confirmation Email Enqueue (10ms). The team proposes switching to a payment provider with a p99 latency of 90ms, cutting the Payment Charge step from 850ms to 90ms. Would this substantially reduce checkout latency?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-14", "original_id": "toc-14", "prompt": "An order fulfillment pipeline has these stages per order: Order Received (0.1s), Fraud Check via external API (3.9s), Pick and Pack (0.5s), Shipping Label Generation (0.3s), Carrier Handoff (0.2s). The team proposes replacing the external fraud API with an in-house ML model that runs in 80ms. Would this improve end-to-end order fulfillment latency?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-16", "original_id": "toc-16", "prompt": "A REST API handles requests with these sequential steps: Request Parsing (3ms), Auth (5ms), DB Query (290ms), Response Formatting (4ms). The team proposes adding a read replica and query index to cut the DB Query stage to under 20ms. Would this dramatically improve the API's response times?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-17", "original_id": "toc-17", "prompt": "A CI/CD pipeline runs: Checkout (12s), Install Dependencies (18s), Unit Tests (8 min 45s), Integration Tests (2 min), Build (40s), Deploy (25s). The team proposes parallelizing the Unit Tests across 10 workers, which would cut the step to under 1 minute. Would this meaningfully shorten total pipeline run time?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-19", "original_id": "toc-19", "prompt": "A software delivery process has these average stage durations: Requirements (1 day), Development (3 days), Code Review (11 days \u2014 queue backlog is 7 days), QA (1.5 days), Release (0.5 days). The team proposes assigning two more engineers specifically to code review. Would this increase feature release throughput?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-20", "original_id": "toc-20", "prompt": "An event streaming pipeline has throughput rates: Kafka Consumer (50,000 msg/s), Deserialize (45,000 msg/s), Enrich via External API (1,800 msg/s), Aggregate (40,000 msg/s), Write to Postgres (38,000 msg/s). The team proposes adding connection pooling and batching to the External API calls to raise its throughput to 20,000 msg/s. Would this improve overall pipeline throughput?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-21", "original_id": "toc-21", "prompt": "A document processing pipeline runs: OCR Scan (2s), Text Extraction (1.8s), NLP Entity Recognition (24s), Index to Search (1.2s), Notify User (0.3s). The team proposes replacing the NLP model with a faster distilled version that achieves the same accuracy in 3s. Would this meaningfully reduce per-document processing time?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-22", "original_id": "toc-22", "prompt": "A video upload pipeline processes each file: Client Upload to CDN (varies), Virus Scan (8s), Video Transcode (5m 20s), Thumbnail Generation (12s), Metadata Index (2s). The team proposes running Thumbnail Generation in parallel with Transcode. Would this reduce the total time a user waits for their video to be available?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-23", "original_id": "toc-23", "prompt": "A SaaS onboarding pipeline has per-customer steps: Sales Handoff (0.5 days), Legal Contract Review (9 days \u2014 average wait 6 days), Technical Integration (2 days), Training (1 day), Go-Live (0.5 days). The team proposes adding a dedicated legal reviewer to reduce contract review to 1 day. Would this reduce total time-to-live for new customers?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-edge throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-24", "original_id": "toc-24", "prompt": "A real-time analytics query pipeline runs: Parse SQL (5ms), Check Permissions (8ms), Execute OLAP Query (4,500ms), Format Response (12ms), Compress Payload (18ms). The team proposes switching to a columnar engine that reduces OLAP Query time to 200ms. Would this substantially reduce user-perceived query latency?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-25", "original_id": "toc-25", "prompt": "A fraud detection pipeline evaluates each transaction: Rule Engine (10ms), ML Feature Extraction (22ms), External Credit Bureau Lookup (2,100ms), Score Aggregation (8ms), Decision Write (6ms). The team proposes caching Credit Bureau results by SSN with a 24-hour TTL, cutting average lookup time to 15ms for repeat customers. Would this materially reduce fraud check latency?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-26", "original_id": "toc-26", "prompt": "A nightly data warehouse refresh has stages: Extract from Postgres (14 min), Transform (62 min), Load to Snowflake (10 min), Run dbt Models (8 min). The team proposes upgrading the Snowflake warehouse tier to cut Load time to 3 min. Would this meaningfully shorten total refresh duration?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-27", "original_id": "toc-27", "prompt": "A background job pipeline processes user export requests: Enqueue Job (0.1s), Fetch User Data from DB (5,800ms), Serialize to CSV (200ms), Compress File (180ms), Upload to S3 (400ms), Email User Link (100ms). The team wants to optimize the Compress File step with a faster algorithm. Would this significantly reduce the time users wait for their export?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-28", "original_id": "toc-28", "prompt": "A hardware test pipeline runs: Flash Firmware (45s), Boot Device (12s), Run Calibration Suite (8 min 30s), Log Results (5s), Package for Shipping (30s). The team proposes upgrading the USB flasher to cut firmware flash time to 10s. Would this improve overall throughput of units through the test line?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-29", "original_id": "toc-29", "prompt": "A Kubernetes pod startup sequence involves: Image Pull (3s), Init Containers (8s), Application Boot (94s \u2014 slow Spring Boot cold start), Readiness Probe (5s), Traffic Admission (1s). The team proposes pre-warming the image pull by caching it on every node. Would this reduce the time from deployment trigger to traffic serving?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "toc-30", "original_id": "toc-30", "prompt": "A machine learning training pipeline runs per epoch: Data Loading (8s), Augmentation (5s), Forward Pass (310s), Backward Pass (290s), Checkpoint Save (12s). The team proposes optimizing the Data Loading step with async prefetching to reduce it to 1s. Would this reduce total training time per epoch?", "source": "theory-of-constraints-bottleneck", "source_file": "authored/theory-of-constraints-bottleneck.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-01", "original_id": "margin-safety-01", "prompt": "A team expects peak API traffic of 800 req/s during an upcoming product launch. Historical launches show spikes of 3\u20135x the predicted peak, and a failure would mean lost revenue plus SLA breach penalties. The team provisions 2,000 req/s of capacity \u2014 2.5x the expected peak.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-02", "original_id": "margin-safety-02", "prompt": "A data warehouse team expects to ingest 500GB of data per month. Storage is charged at $0.023/GB and data loss is not a concern because source data can be re-ingested at any time. They provision 550GB \u2014 a 10% buffer. Scaling additional storage takes less than 30 seconds via an API call.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-03", "original_id": "margin-safety-03", "prompt": "A payments company processes an average of 1,200 transactions per second on normal days. Black Friday historically generates 4\u20136x normal volume, and a payment processing outage means direct revenue loss of roughly $50,000 per minute. The team provisions capacity for 1,800 transactions per second for the holiday period.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-04", "original_id": "margin-safety-04", "prompt": "A team is setting an HTTP client timeout for calls to a third-party shipping API. The shipping API's documented p99 latency is 400ms, and their SLA guarantees responses within 800ms. Calls that time out result in a failed order checkout with user-visible errors. The team sets a 1,000ms timeout.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-05", "original_id": "margin-safety-05", "prompt": "A startup's engineering team estimates a new feature will take 4 weeks to build. This is the team's first time working with the third-party payments SDK, and the feature is on the critical path for a contractually committed customer demo. They commit to a 5-week delivery date \u2014 25% buffer.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-06", "original_id": "margin-safety-06", "prompt": "A Postgres connection pool is configured with a maximum of 100 connections. Under normal load the service uses 45 connections. During monthly billing runs a secondary process uses up to 30 additional connections simultaneously. An unexpected secondary batch job will exhaust the pool only if both peaks coincide, which has never happened. The cost of pool exhaustion is a cascade of 500 errors to all users.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-07", "original_id": "margin-safety-07", "prompt": "A video transcoding service processes an average of 200 jobs per hour. Job volume has been growing 15% month-over-month for six months with no sign of slowing. Each job consumes roughly the same compute. The team provisions for 600 jobs per hour \u2014 3x current average \u2014 and plans to review in three months. Delays cause content to miss publishing windows, resulting in lost ad revenue.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-08", "original_id": "margin-safety-08", "prompt": "A team is planning storage for a new audit log system. Legal requires all audit logs to be retained for 7 years, and any gap is a compliance violation. The team's data model estimates 80GB per year based on current transaction volume. They provision 100GB of storage \u2014 enough for about 15 months \u2014 with a plan to add capacity before it fills.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-09", "original_id": "margin-safety-09", "prompt": "A ride-sharing company allocates a monthly infrastructure budget of $120,000. Actual spend has been stable at $95,000\u2013$100,000 per month for the past six months with no new services planned. The allocated budget of $120,000 represents a 20\u201326% buffer over observed spend.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-10", "original_id": "margin-safety-10", "prompt": "A hospital patient-record API is expected to serve 500 concurrent users during the busy morning shift. Load tests show the system becomes degraded at 520 concurrent users. Patient care workflows depend on the system; downtime means nurses fall back to paper, which introduces medication error risk. The team deploys hardware supporting 550 concurrent users.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-11", "original_id": "margin-safety-11", "prompt": "A Kafka consumer group processes 50,000 messages per hour at steady state. The upstream producer can burst to 200,000 messages per hour during end-of-day batch closes, lasting about 45 minutes. Consumer lag during burst causes downstream reports to be delayed, impacting trading desks. The team sizes consumers for 250,000 messages per hour sustained throughput.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-12", "original_id": "margin-safety-12", "prompt": "A team is deploying a new machine learning inference service. Benchmark tests show the model needs 6GB of GPU VRAM at maximum batch size. The team selects a GPU instance with exactly 8GB VRAM. They note that framework overhead, CUDA context, and driver memory are not accounted for in the benchmark figure.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-13", "original_id": "margin-safety-13", "prompt": "A software team is migrating a monolith to microservices. The project manager estimates 6 months based on a rough breakdown of work. The migration touches every team in the company, requires coordination with three external vendors, and has no prior comparable project as reference. The team commits to a 12-month timeline with a formal review at month 6.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-14", "original_id": "margin-safety-14", "prompt": "An IoT platform expects 10,000 devices to connect simultaneously after a firmware update pushes to all deployed units. Historical firmware pushes show actual simultaneous reconnection peaks reach 8x the device count (due to reconnect storms). A reconnection storm exceeding broker capacity causes devices to go permanently offline until manually reset. The broker is provisioned for 15,000 simultaneous connections.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-15", "original_id": "margin-safety-15", "prompt": "A SaaS company's search index uses 40% of available disk space under current data volume. Data volume is projected to grow 20% per year, and the index requires immediate disk space to function \u2014 when disk hits 90% full the service stops indexing new documents. The team plans to add storage when usage hits 70% utilization, giving roughly 1.5 years of runway.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-16", "original_id": "margin-safety-16", "prompt": "A team's API rate limit with their geolocation provider is 10,000 lookups per day. Current usage averages 6,500 lookups per day. A marketing campaign launching next month is expected to bring a 2\u20133x traffic increase lasting two weeks. Exceeding the rate limit returns errors on the address validation step of checkout, blocking all new orders.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-17", "original_id": "margin-safety-17", "prompt": "A team running a high-traffic CDN edge node configures a memory cache with a maximum size of 32GB. Profiling shows the working set of hot content peaks at 20GB during prime time, with cold-start periods using as little as 5GB. Cache evictions are graceful \u2014 a miss falls through to origin with a latency penalty but no data loss. They size at 32GB for comfortable headroom.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-18", "original_id": "margin-safety-18", "prompt": "A developer estimates a new authentication refactor will take 10 days. The team has done this type of refactor three times in the past year, always within 8\u201312 days. The change is internal only, has no customer-facing deadline, and can be rolled back instantly if problems arise. They commit to 11 days.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-19", "original_id": "margin-safety-19", "prompt": "A team deploys a Redis cache cluster sized for 8GB of memory. Current cache usage is 7.2GB. Keys are set with TTLs of 24 hours but the product roadmap includes launching three new features over the next two months, each of which is expected to add cached objects. Cache eviction uses LRU, and evicting hot keys causes a cascade of DB queries that has previously caused an incident.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-21", "original_id": "margin-safety-21", "prompt": "A team's incident response SLA commits to acknowledging critical incidents within 15 minutes and resolving them within 4 hours. They have one primary on-call engineer and no backup. The primary has missed acknowledgment in 3 of the last 20 incidents due to being in a meeting or asleep in a different time zone.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-22", "original_id": "margin-safety-22", "prompt": "An analytics pipeline writes results to an S3 bucket. Each run produces approximately 2GB. The pipeline runs daily, and 90 days of results are retained before lifecycle rules delete old files. The team allocates 250GB of S3 capacity via a quota. S3 storage is elastic and quota overages are immediately rechargeable at standard rates.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-23", "original_id": "margin-safety-23", "prompt": "A team is launching a developer API with a free tier capped at 1,000 calls per month per user. Backend cost per call is $0.001. They provision infrastructure for 5 million total free-tier calls per month. They have 3,000 free-tier users signed up and expect 10,000 by end of quarter. At full expected adoption and full usage, demand would be 10 million calls per month.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-24", "original_id": "margin-safety-24", "prompt": "A team configures a circuit breaker to open after 5% error rate sustained over 10 seconds. Under normal conditions the service runs at 0.1% error rate. The error rate threshold, window size, and recovery behavior were validated in production-like load tests. The circuit breaker configuration matches what was tested.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-25", "original_id": "margin-safety-25", "prompt": "An e-commerce site expects 1,000 concurrent sessions during a flash sale. Load testing shows the application server handles 1,050 concurrent sessions before response times degrade past 3 seconds. A degraded checkout experience during the flash sale means customers abandon carts. The team does not add additional capacity, relying on auto-scaling that takes 4\u20136 minutes to provision new instances.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-26", "original_id": "margin-safety-26", "prompt": "A team building a new internal reporting tool estimates the backend will need 4 vCPUs and 8GB RAM under expected query load. The tool is used only during business hours by 15 analysts. They provision an 8 vCPU, 16GB RAM instance, doubling both. If the tool becomes slow, analysts can wait without business impact.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-27", "original_id": "margin-safety-27", "prompt": "A satellite telemetry ingestion pipeline must store all raw data permanently for regulatory reasons; any data gap is a compliance event with significant fines. The system ingests 10TB per year with no growth expected. The team provisions 12TB \u2014 20% buffer \u2014 on a single storage array with no redundancy and no offsite backup.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-28", "original_id": "margin-safety-28", "prompt": "A team uses a managed Elasticsearch cluster for product search. Normal indexing throughput is 2,000 documents per second. The team expects a re-index event during a schema migration that will run at 4,000 documents per second for approximately 6 hours. They temporarily scale the cluster to handle 6,000 documents per second for the migration window. A search outage means the product catalog is unavailable to customers.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-29", "original_id": "margin-safety-29", "prompt": "A team provisions a job queue worker fleet sized to drain 100 jobs per minute \u2014 exactly matching the average production ingest rate. The queue is used for sending transactional emails; if the queue backs up significantly, emails arrive hours late. Historical data shows ingest spikes to 300 jobs per minute during large user-triggered events such as password reset floods.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "margin-safety-30", "original_id": "margin-safety-30", "prompt": "A team committing to a quarterly OKR estimates a feature will be delivered in 6 weeks, based on a well-understood problem space, a team that has built nearly identical features before, and no external dependencies. They commit to 8 weeks in the OKR. Missing the OKR has minor internal visibility consequences and the feature can ship one quarter late without customer impact.", "source": "margin-of-safety-provision", "source_file": "authored/margin-of-safety-provision.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "nearMiss", "target": false, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty, given an expected load and the cost of under-provisioning vs over-provisioning. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_only": false}+{"id": "glm-distractor-8", "original_id": "glm-distractor-8", "prompt": "How can I convert all the lowercase letters in a string to uppercase using Python?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-12", "original_id": "glm-distractor-12", "prompt": "Implement quick sort to sort the given array of integers. Write corresponding code in Python. [9, 5, 8, 1, 3, 6, 7]", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-13", "original_id": "glm-distractor-13", "prompt": "Zoe wants to go on the field trip to Washington DC with her middle school this spring and the cost is $485. Her grandma gave her $250 toward her fees and she must earn the rest by selling candy bars. She makes $1.25 for every candy bar she sells. How many candy bars does Zoe need to sell to earn the money for the trip?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-16", "original_id": "glm-distractor-16", "prompt": "Can you also tell me the date and time of the Indiana Comic Convention?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-17", "original_id": "glm-distractor-17", "prompt": "Create an algorithm to rotate an array by one element. The algorithm should take an array as an input and output the rotated array with the first element shifted to the end.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-18", "original_id": "glm-distractor-18", "prompt": "Generate a correct Python program for plotting bar chart for the following data.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-19", "original_id": "glm-distractor-19", "prompt": "Given a non-negative decimal integer $x$, convert it to binary representation $b$ of 32 bits. Then, print the result of the following operations to $b$ respecitvely. * Inversion: change the state of each bit to the opposite state * Logical left shift: shift left by 1 * Logical right shift: shift right by 1 Constraints * $0 \\leq x \\leq 2^{32} - 1$ Input The input is given in the following format. $x$ Output Print the given bits, results of inversion, left shift and right shift in a line respectively. Examples Input 8 Output 00000000000000000000000000001000 11111111111111111111111111110111 00000000000000000000000000010000 00000000000000000000000000000100 Input 13 Output 00000000000000000000000000001101 11111111111111111111111111110010 00000000000000000000000000011010 00000000000000000000000000000110 Write Python code to solve the problem. Please place the solution code in the following format: ```python # Your solution code here ```", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-21", "original_id": "glm-distractor-21", "prompt": "Create a Python program that fetches the user's current time and sets a reminder after 30 minutes.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-22", "original_id": "glm-distractor-22", "prompt": "Could you delineate a methodology, utilising the Python programming language, to extract the middle value, labeled as the median, from an unorganized collection of integral numbers? Numbers = [12, 7, 21, 3, 9]", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-42", "original_id": "glm-distractor-42", "prompt": "Write a Python3 function that returns the Cargo home directory path by first checking the CARGO_HOME environment variable. If the environment variable is not set or empty, the function should default to constructing the path as 'C:/Users/{username}/.cargo' using the current user's username. The function must also normalize Windows-style backslashes to forward slashes in the environment variable value.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-46", "original_id": "glm-distractor-46", "prompt": "Create a Python 3 script that uses the clustergrammer library to load a matrix from stdin, perform optional filtering and normalization steps (such as sum-based filtering, z-score normalization, top-N selection, and threshold filtering), generate a hierarchical clustering visualization using cosine distance, and then output the resulting network data in JSON format for frontend visualization. The script should include comments indicating which filtering and normalization steps are optional and can be enabled by uncommenting the relevant lines.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-48", "original_id": "glm-distractor-48", "prompt": "Solve the inequality: 8x - 9 < 2. Express the solution in interval notation.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-50", "original_id": "glm-distractor-50", "prompt": "A toy store manager received a large order of Mr. Slinkums just in time for the holidays. The manager places $20\\%$ of them on the shelves, leaving the other 120 Mr. Slinkums in storage. How many Mr. Slinkums were in this order?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-51", "original_id": "glm-distractor-51", "prompt": "Write a python function to check if a given file is in NOAA ACE format by verifying that the first line starts with ':' and the third line contains 'NOAA'. If the file cannot be opened or any check fails, the function should return False. Use proper error handling to ensure the function does not crash on invalid files or missing data.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-69", "original_id": "glm-distractor-69", "prompt": "Write a program that allows users to search for phone numbers by name in a phone book. Implement the phone book using a custom data structure that you create. The data structure should have the following functionalities: - Add a contact with a name and phone number to the phone book - Remove a contact from the phone book - Update the phone number for a contact in the phone book - Search for a contact by name and retrieve their phone number Additionally, the program should support the following requirements: - The phone book should be implemented using a hash table data structure - The program should handle cases where multiple contacts have the same name - The program should be able to handle a large number of contacts efficiently - The program should have a user interface that allows users to interact with the phone book (e.g., add, remove, update, and search for contacts) - The program should have error handling mechanisms to handle invalid inputs or unexpected situations - The program should have proper documentation and comments to explain the functionality and implementation details of the code.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-74", "original_id": "glm-distractor-74", "prompt": "Produce a program using Python that visualizes the given function.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-81", "original_id": "glm-distractor-81", "prompt": "How can I generate a regular expression that matches any two or more consecutive numbers?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-83", "original_id": "glm-distractor-83", "prompt": "Write a python 3 script that performs simple linear regression analysis on gas expenditure data using scikit-learn. The script should load data from \"data1.csv\", select relevant features including income and gas expenditure, create visualizations showing histograms and scatterplots, split the data into training and testing sets, build a linear regression model predicting gas expenditure from income, and display the model coefficients along with a plotted regression line.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-84", "original_id": "glm-distractor-84", "prompt": "Write a python 3 Django management command that classifies unassigned faces using ensemble predictions from two SVM models (small and big) with probability averaging, and prints classification results for the first 1000 unassigned faces, including predicted labels, probabilities, and corresponding person objects from the database.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-196", "original_id": "glm-distractor-196", "prompt": "Create a python 3 script that demonstrates two different plotting APIs using PyQt4: a procedural approach with PlotWidget and an object-oriented approach with a custom Plot class. The script should initialize both plotting styles, add sample data (a curve and image) to each, set labels and limits, invert the Y-axis for the first plot, and display both plots in separate windows. Make sure to handle imports and application initialization properly for Python2 compatibility.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-199", "original_id": "glm-distractor-199", "prompt": "Write a Python 3 function that tests whether the neohub class can be properly instantiated, ensuring the class initialization works correctly without any errors. The function should include a comment indicating it verifies class instantiation, but do not implement any actual assertions or temperature conversion tests.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-200", "original_id": "glm-distractor-200", "prompt": "Please answer the following question: - Gather the ingredients for the cake - Mix the flour, sugar, egg, and water to make batter - Pour the mixture into a cake pan - Place the cake in the oven to cook - Take the cake out when it is cooked through - Let the cake cool What might be the last step of the process? A:", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-201", "original_id": "glm-distractor-201", "prompt": "Can you provide a Python code snippet to convert XML to JSON?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-203", "original_id": "glm-distractor-203", "prompt": "Write a Python3 script that downloads and unpacks tracking results from Google Drive for various trackers like PyTracking-based and external trackers. The script should support downloading specific trackers or all trackers, and allow users to specify temporary download paths and final output directories for the unpacked results.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-214", "original_id": "glm-distractor-214", "prompt": "Create a python 3 program using the turtle module that draws a series of nested squares rotating at different angles. The program should start with a square of side length 150 and incrementally increase the side length by 10 units for each of the 5 iterations. Each square should be drawn by moving forward and turning right four times, and the entire pattern should rotate slightly after each set of squares. Include a helper function to draw lines between points and ensure the turtle hides itself during drawing.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-216", "original_id": "glm-distractor-216", "prompt": "How can I generate a set of random numbers without any duplicates in Python?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-224", "original_id": "glm-distractor-224", "prompt": "Create a Python script that sets the value of a variable from the command-line, but the value must be encrypted using a custom encryption algorithm before being assigned to the variable. Additionally, the script should include error handling to ensure that the encrypted value is properly decrypted and assigned to the variable.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-225", "original_id": "glm-distractor-225", "prompt": "A satellite orbits Earth at an altitude of 1501 km. Calculate its orbital velocity and period. (Earth mass = 5.97 \u00c3\u2014 10\u00c2\u00b2\u2074 kg, Earth radius = 6371 km)", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-227", "original_id": "glm-distractor-227", "prompt": "Compute the quotient of 17708 and 9880. Show your work.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-236", "original_id": "glm-distractor-236", "prompt": "def division_while_loop(dividend, divisor): \"\"\" This exercise focuses on division using a while loop. Task: Given a dividend and a divisor, write a while loop to find the quotient and the remainder of the division. Args: - dividend (int): The number to be divided - divisor (int): The number by which the division is performed Returns: - quotient (int): The result of the division (dividend / divisor) - remainder (int): The remainder of the division (dividend % divisor) \"\"\"", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-241", "original_id": "glm-distractor-241", "prompt": "Create a python 3 class called `BalanceSheet` that takes a `Portfolio` object (with default to a new `Portfolio()` instance) and provides methods to generate structured data representations. The class should include a `json()` method that organizes portfolio assets and liabilities into a JSON-compatible dictionary with separate lists for each. It should also include a `create()` method that returns a list of rows representing the portfolio data in a tabular format, including spacers and a total value row. Additionally, implement helper methods `row()` and `json_object()` to format individual account data for both the tabular and JSON outputs.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-249", "original_id": "glm-distractor-249", "prompt": "Write a python function to find the frequency of each character in a given string and return a dictionary with characters as keys and their frequencies as values. Solve the problem starting with the provided function header. Function header: ``` def get_char_frequency(string): ``` Please place the solution code in the following format: ```python # Your solution code here ```", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-261", "original_id": "glm-distractor-261", "prompt": "Create a function to check if an array contains any duplicate elements, while also ensuring that the array elements are sorted in ascending order and that the length of the array is less than or equal to 100. Additionally, the function should check if the array contains any negative integers, and if so, it should return False. The function should return a boolean value indicating the presence of duplicate elements. Example: Input: [1, 2, 3, 3, 4, 5, 5] Output: True", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-262", "original_id": "glm-distractor-262", "prompt": "You are working on an API for a space-themed game and need to create a class that handles the case when a specific planet for a character is not found. You need to define a class called GetCharactersCharacterIdPlanetsPlanetIdNotFound with the following attributes and methods: Attributes: - error: a string to store the error message Methods: - __init__: a constructor method to initialize the error attribute - to_dict: a method to return the model properties as a dictionary - to_str: a method to return the string representation of the model - __repr__: a method for printing and pretty-printing the object - __eq__: a method to check if two objects are equal - __ne__: a method to check if two objects are not equal You need to implement the class and its methods as described above.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-264", "original_id": "glm-distractor-264", "prompt": "How can I categorize data into two bins, with the labels \"category A\" and \"category B\"?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-267", "original_id": "glm-distractor-267", "prompt": "Given points A(14, -8) and B(-1, -13), find the slope of the line passing through these two points.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-268", "original_id": "glm-distractor-268", "prompt": "Create a python 3 class named Objects that initializes with several instance attributes including cloud set to None, score set to 0, score_type set to 'CNNSeg', type set to None, and type_probs set to None.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-269", "original_id": "glm-distractor-269", "prompt": "What is the code to construct a Recurrent Neural Network (RNN) using TensorFlow with an input size of 50 values and an output size of 10 values?", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-270", "original_id": "glm-distractor-270", "prompt": "Create a python 3 class called CompileGLSL that extends Module for compiling GLSL shader files into SPIR-V format using the glslangValidator tool. The class should define required tools, specify output file names with .spv extension, and implement a process method that runs the validator with appropriate stage flags (-S vert for vertex shaders and -S frag for fragment shaders) while handling directory creation and returning the subprocess return code.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-274", "original_id": "glm-distractor-274", "prompt": "Calculate the pH of a 0.545 M solution of HCl. Show your calculation step by step.", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "glm-distractor-275", "original_id": "glm-distractor-275", "prompt": "Write a Python 3 program that prompts the user to enter three integer numbers and then calculates and displays their sum. The program should use input() to read the numbers from the user and print the result in the format \"Sum = [value]\".", "source": "glm-distractor", "source_file": "external/glm-distractor.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "mode": "distractor", "license": "Apache-2.0 (GLM-5.1 distilled)", "decision_instruction": "Answer the question directly. Return ONLY valid JSON: {\"answer\": \"...\"}", "eval_mode": "judge", "label": null, "calibration_baseline": null, "calibration_band": "estimated", "calibration_estimated": true, "calibration_note": "Distractor - included for false-positive measurement. No ground-truth label for objective scoring.", "calibration_only": false}+{"id": "toc-harder-01", "prompt": "A payment processing pipeline handles debit card transactions through: Card Network Auth (90ms), Balance Check (45ms), Fraud Scoring via external API (380ms p50, 2,100ms p99), Ledger Update (15ms), SMS Notification enqueue (5ms). The team is concerned about tail latency during peak hours and proposes adding a read replica for the Ledger Update step to reduce it to 5ms. Would this meaningfully reduce p99 transaction latency?", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "toc-harder-02", "prompt": "A CI pipeline runs: Lint (12s), Type Check (18s), Unit Tests (55s), Integration Tests (48s), Build (35s), Security Scan (52s). The team proposes parallelizing Unit Tests across 4 workers, which would cut it to 14s. Would this meaningfully shorten total pipeline run time?", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": 0, "calibration_band": "floor", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "toc-harder-03", "prompt": "A web application request flows through: CDN Edge (3ms), WAF Inspection (8ms), Load Balancer (2ms), App Server Business Logic (85ms), Database Query (72ms), Template Render (65ms). The team proposes adding a Redis cache for database queries that would reduce the DB step to 5ms for cached queries (80% hit rate expected). Would this dramatically improve user-perceived latency?", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": 0, "calibration_band": "floor", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "toc-harder-04", "prompt": "A document generation service processes each request: Template Loading from S3 (120ms p50, 850ms p99), Data Collection from 3 microservices in parallel (the slowest takes 400ms), PDF Rendering (180ms), Watermark Application (45ms), CDN Upload (90ms). The team proposes adding a local cache for S3 templates to reduce Template Loading to 5ms. Would this meaningfully reduce the median document generation time?", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": 0.2, "calibration_band": "floor", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "toc-harder-05", "prompt": "An ML inference pipeline processes each image: Image Decode (25ms), Preprocessing/Normalization (35ms), CNN Forward Pass on GPU (140ms), Post-processing/NMS (55ms), Result Serialization (8ms). The team proposes using a faster image decoder that cuts Image Decode to 5ms. Would this meaningfully improve inference throughput?", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "toc-harder-06", "prompt": "A batch data pipeline runs hourly: Extract from Kafka (30s), Deserialize Avro (8s), Validate Schema (5s), Join with Reference Data from Redis (42s), Aggregate Windows (25s), Write to Parquet in S3 (18s). The team proposes switching to a faster Avro deserialization library to reduce Deserialize to 2s. Would this meaningfully reduce total batch processing time?", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "toc-harder-07", "prompt": "A video encoding pipeline runs these stages PER FRAME: Motion Estimation (22ms), Motion Compensation (15ms), DCT Transform (8ms), Quantization (5ms), Entropy Coding (12ms). The encoding is done on a single thread per stream. The team proposes parallelizing Motion Estimation across 8 cores using slice-based threading, which could theoretically cut it to 3ms. Would this significantly improve frames-per-second throughput?", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "toc-harder-08", "prompt": "An API gateway processes each request: TLS Termination (4ms), Rate Limit Check (2ms), Auth Token Validation (18ms), Request Routing (1ms), Upstream Proxy (depends on backend). The backend service itself takes 95ms to respond. The gateway team proposes replacing the auth library with a faster one that validates tokens in 4ms. Would this notably reduce the total response time users see?", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "toc-harder-09", "prompt": "A log processing pipeline has per-stage throughput: Tail Log Files (80,000 lines/s), Parse JSON (65,000 lines/s), Filter by Severity (90,000 lines/s), Batch Insert to ClickHouse (42,000 rows/s), Create Materialized Views (38,000 rows/s). The team proposes rewriting the Parse JSON stage in Rust (currently Python) to achieve 200,000 lines/s. Would this meaningfully improve overall pipeline throughput?", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "toc-harder-10", "prompt": "A notification delivery pipeline processes: Fetch Notification Templates (5,000/s), Personalize Content (4,200/s), Render HTML (3,800/s), Send via Email Provider API (1,200/s), Send via Push Provider API (1,100/s). Email and Push run in parallel after rendering. The team proposes optimizing the Render HTML step with pre-compiled templates to achieve 12,000/s. Would this significantly increase total notifications delivered per second?", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "toc-harder-11", "prompt": "A search API serves queries through: Spelling Correction (12ms), Query Understanding/NER (22ms), Elasticsearch Query (340ms), Result Reranking (85ms), Response Assembly (15ms). The team proposes migrating from Elasticsearch to a purpose-built vector database that would reduce query time to 45ms. Would this dramatically improve search response time?", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "toc-harder-12", "prompt": "A feature release pipeline has stages: Spec Review (0.5 days), Development (4 days avg), Code Review Queue (8 days avg), QA Testing (1.5 days), Staging Deploy (0.3 days). The team proposes adding a dedicated QA engineer to cut testing to 0.5 days. Would this meaningfully improve time-to-release?", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "toc-harder-13", "prompt": "An order processing pipeline serves: Validate Cart (25ms), Reserve Inventory (35ms), Tax Calculation via external API (480ms), Create Order Record (12ms), Publish to Fulfillment Queue (8ms). The team proposes switching to a tax calculation provider with a local SDK that computes taxes in 15ms. Would this dramatically reduce order processing latency?", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["theory-of-constraints"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "mos-harder-01", "prompt": "An API team is provisioning rate limit capacity for a new partner integration. The partner's contract guarantees up to 500 requests per second with burst allowance of 2x. The API gateway can scale horizontally but new instances take 90 seconds to warm up. During the previous partner integration, actual traffic reached 3.5x the guaranteed rate within the first week. The team provisions for 2,000 req/s \u2014 4x the guaranteed rate.", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": 0, "calibration_band": "floor", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "mos-harder-02", "prompt": "A team runs a mission-critical batch payment job that processes payroll for 50,000 employees every other Friday. The job takes 3.5 hours and MUST complete by 6 AM when direct deposits are initiated. The team schedules the job to start at 10 PM Friday, leaving a 4.5-hour buffer for the 3.5-hour job. The job has completed successfully within its window for 47 consecutive runs, but a database maintenance window sometimes overlaps and adds 20-40 minutes of delay.", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": 0, "calibration_band": "floor", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "mos-harder-03", "prompt": "A team provisions cloud resources for a Black Friday sale expecting 10x normal traffic. Normal traffic is 2,000 concurrent users, and the system degrades at 18,000 concurrent users. They provision for 22,000 concurrent users \u2014 10% above their expected peak. During the previous Black Friday, a competitor outage drove unexpected traffic that reached 14x normal. An outage during Black Friday means approximately $1.2M in lost revenue per hour.", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "mos-harder-04", "prompt": "A team manages a Redis cluster for session storage with 16GB memory allocated. Current peak usage is 11GB, and usage has grown 5% month-over-month for 6 months. At 14GB the eviction policy starts dropping sessions, logging users out. The team plans to upgrade to 32GB when usage hits 13GB, which based on current trends will be in approximately 3 months. The upgrade process requires a 2-minute failover with zero data loss.", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": true, "decision_instruction": "You are sizing/provisioning a resource under uncertainty. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "mos-harder-05", "prompt": "A startup's CTO estimates a database migration will take 3 weeks based on a spike test of 10% of the data. The migration is on the critical path for a Series B fundraise demo. The CTO commits to a 4-week timeline \u2014 a 33% buffer. No one on the team has done a database migration at this scale before, and the spike test didn't account for foreign key constraint validation that runs during the full migration.", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": ["margin-of-safety"], "family": "Systems/product/strategy pairwise", "type": "target", "target": true, "label": false, "decision_instruction": "You are sizing/provisioning a resource under uncertainty. Answer YES if the proposed plan leaves an ADEQUATE safety margin for the stated uncertainty and failure cost. Answer NO if the plan is provisioned too tightly (no meaningful buffer) given the uncertainty and the high cost of running out.", "mode": "binary-decision", "calibration_baseline": 1, "calibration_band": "ceiling", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "sys-dist-01", "prompt": "Explain how Kubernetes handles pod scheduling and what factors the scheduler considers when placing pods on nodes.", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "label": null, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "distractor", "calibration_baseline": 0, "calibration_band": "floor", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "sys-dist-02", "prompt": "Write a Terraform configuration to provision an AWS EKS cluster with managed node groups, VPC, and IAM roles for pod-level permissions.", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "label": null, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "distractor", "calibration_baseline": 0, "calibration_band": "floor", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "sys-dist-03", "prompt": "Describe how to set up a CI/CD pipeline using GitHub Actions that runs tests, builds Docker images, and deploys to Kubernetes on merge to main.", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "label": null, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "distractor", "calibration_baseline": 0, "calibration_band": "floor", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "sys-dist-04", "prompt": "Compare PostgreSQL and MongoDB for a use case involving time-series IoT sensor data with 100,000 writes per second and real-time dashboards.", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "label": null, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "distractor", "calibration_baseline": 0, "calibration_band": "floor", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}+{"id": "sys-dist-05", "prompt": "Design a rate limiting system for a public API that supports per-user, per-endpoint, and per-IP limits with configurable windows and graceful degradation.", "source": "authored-harder-systems", "source_file": "evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl", "skill_fit": [], "family": "Systems/product/strategy pairwise", "type": "distractor", "target": false, "label": null, "decision_instruction": "You are given a multi-stage pipeline with per-stage timings/capacities and a proposed optimization. Answer YES if optimizing the proposed stage would meaningfully improve end-to-end throughput/latency (it is the binding constraint). Answer NO if it would NOT help meaningfully (the proposed stage is not the bottleneck \u2014 some other stage dominates).", "mode": "distractor", "calibration_baseline": 0, "calibration_band": "floor", "calibration_only": false, "generator": "evals/datasets/calibrated/build-harder-conceptual-systems-pools.js"}diff --git a/evals/datasets/calibrated/systems-product-strategy-pairwise.manifest.json b/evals/datasets/calibrated/systems-product-strategy-pairwise.manifest.jsonindex a2483a1..45312a2 100644--- a/evals/datasets/calibrated/systems-product-strategy-pairwise.manifest.json+++ b/evals/datasets/calibrated/systems-product-strategy-pairwise.manifest.json@@ -9,7 +9,7 @@"via-negativa"],"value_surface": "paired reasoning quality",- "source": "Combined from 4 source files. authored: theory-of-constraints-bottleneck, margin-of-safety-provision, harder-systems (m3-harder-data); external: glm-distractor",+ "source": "Combined from 4 source files. authored: theory-of-constraints-bottleneck, margin-of-safety-provision; harder-systems items from evals/datasets/calibrated/systems-product-strategy-pairwise-harder-pool.jsonl (generated by evals/datasets/calibrated/build-harder-conceptual-systems-pools.js, m3-harder-data); external: glm-distractor","leakage": "Theory-of-constraints-bottleneck, margin-of-safety-provision, and harder-systems are locally authored datasets — zero external leakage. All target items are locally authored with zero training data leakage risk. GLM distractor items may have training data overlap. Overall leakage risk is very low.\n\nSource breakdown: 0 external dataset(s) with potential training-data leakage; 3 authored dataset(s) with zero external leakage; ","label": {"definition": "Paired reasoning quality for systems/product/strategy scenarios. Objective subset: binary bottleneck identification (is the proposed optimization targeting the binding constraint?) and margin-of-safety adequacy (is the provision adequate given uncertainty and failure cost?). Harder items (m3-harder-data) include subtle bottlenecks where the correct stage is non-obvious (close calls, p50 vs p99 latency, Amdahl's law scenarios) and nuanced margin-of-safety judgments. Judge-based items for archetypes, feedback-loops, leverage-points, ooda, and via-negativa are scored by a judge panel.",@@ -33,11 +33,14 @@0.4,0.7],- "in_band": false,- "exemption": "CEILING-NEEDS-HARDER-DATA. After sourcing 18 genuinely harder bottleneck and margin-of-safety items (subtle bottleneck identification with close-call timings, p50 vs p99 latency, throughput-based reasoning, and nuanced margin-of-safety judgments), claude-sonnet-4-6 achieves 57.4% aggregate baseline accuracy (18 labeled target items at K_TRIALS=5). The aggregate IS within the [0.40,0.70] band, but individual items are BIMODAL: 13 items at 100% ceiling and 5 items at 0% floor, with 0 items landing in the per-item [0.40,0.70] band. The solver either knows the answer with certainty or is systematically wrong — it never 'struggles' (0.4-0.6 range) on any single item. This bimodal distribution is characteristic of binary classification tasks where the model has strong but sometimes misaligned priors about what constitutes a 'bottleneck' or 'adequate margin.' Per Invariant #4, this is NOT a kill verdict.",+ "in_band": true,+ "band_rule": "aggregate",+ "band_rationale": "Per library/harness.md aggregate-vs-per-item rule for binary families: powerability is judged by the AGGREGATE baseline + headroom, NOT per-item-in-band count. Aggregate baseline 57.4% (18 labeled target items at K_TRIALS=5) is squarely in [0.40,0.70], making this family a valid powered candidate. The bimodal per-item distribution (13 ceiling + 5 floor, 0 per-item in-band) is expected for binary classification tasks where a confident solver has strong but sometimes misaligned priors. A skill can flip the systematically-wrong items, producing measurable aggregate lift.",+ "exemption": null,"status": "measured_sample","calibrated_items": 18,"in_band_items": 0,+ "aggregate_in_band": true,"decisive_split_items": 23,"calibration_only_items": 0,"calibration_run_id": "calib-systems-harder",@@ -48,9 +51,11 @@"ceiling_items": 13,"floor_items": 5,"other_out_of_band": 0,- "exemption_reason": "CEILING-NEEDS-HARDER-DATA: aggregate baseline 57.4% is in range, but bimodal distribution (13 ceiling + 5 floor) yields 0 per-item in-band items. No decisive split can be constructed from in-band items alone.",+ "exemption_reason": null,+ "powered_candidate": true,+ "powered_candidate_note": "Valid powered candidate per aggregate band rule. Aggregate baseline 57.4% provides ~43pp headroom for the target ≥5pp lift. Bimodal per-item distribution (13 ceiling + 5 floor) is not disqualifying — the solver answers each binary bottleneck item with ~0% or ~100% confidence, and a skill can flip the systematically-wrong items to produce measurable aggregate lift. This was incorrectly flagged CEILING-NEEDS-HARDER-DATA in the previous manifest revision; corrected to in_band=true per M3 scrutiny fix.","harder_items_note": "The m3-harder-data pool added 18 target items with subtle bottleneck identification (multiple close-call stages, p50 vs p99 latency reasoning, throughput-based capacity analysis, Amdahl's law parallel-stage scenarios) and nuanced margin-of-safety judgments (close-call adequacy with competing signals). The model correctly handled all bottleneck items where the dominant stage was clearly identifiable by numeric comparison (~13 items), but systematically failed on items requiring more nuanced reasoning about parallel stages, tail latency, or hidden bottlenecks. The bimodal pattern suggests the model uses a simple numeric-comparison heuristic rather than systems thinking — it can identify the literal biggest number but struggles with more sophisticated bottleneck theory.",- "aggregate_note": "Aggregate baseline 57.4% is in [0.40,0.70] — indicating genuine difficulty at the pool level. However, the calibration runner requires per-item banding for decisive split construction, and 0 items individually land in-band due to bimodal distribution. This is a known limitation of binary classification calibration with high-confidence models. A judge-based calibration approach may be more appropriate for this family.",+ "aggregate_note": "Aggregate baseline 57.4% is in [0.40,0.70] — genuine difficulty at the pool level. This family IS a valid powered candidate per library/harness.md aggregate rule for binary families. The per-item bimodal distribution is characteristic of binary classification with high-confidence models and does not disqualify the family from powered runs.","previous_calibration": {"baseline": 1.0,"calibrated_items": 20,@@ -60,13 +65,13 @@}},"item_counts": {- "total": 127,- "target": 48,- "near_miss": 30,- "distractor": 49,+ "total": 117,+ "target": 42,+ "near_miss": 29,+ "distractor": 46,"ambiguous": 0,"decisive": 117,- "calibration_only": 10,+ "calibration_only": 0,"exempt": 0,"calibrated": 18,"harder_pool": 23@@ -84,13 +89,17 @@"satisfied": true,"decisive_targets": 42,"decisive_distractors": 75,- "ambiguous_in_split": 0+ "decisive_near_miss": 29,+ "ambiguous_in_split": 0,+ "calibration_only_in_split": 0,+ "total_decisive": 117},"created": "2026-06-06","frozen": true,"pool_limitations": {- "judge_based": "This family is largely judge-based/behavioral. Only ~48 ground-truth target items (binary-decision mode) can be objectively calibrated. The 30 near-miss items also have labels. The 49 distractor items require judge evaluation for scoring but are included in the decisive split for false-positive measurement (no ground-truth labels). The objective placebo calibration runner cannot profile judge-evaluated items.",- "bimodal_calibration": "The harder calibration run (calib-systems-harder) showed aggregate baseline 57.4% but bimodal per-item distribution (13 ceiling, 5 floor). No items landed in the per-item [0.40,0.70] band. This reflects the model's high confidence on individual binary decisions even when the pool-level difficulty is appropriate."+ "judge_based": "This family is largely judge-based/behavioral. Only the objectively-gradeable subset (bottleneck identification + margin-of-safety, 42 target items) can be objectively calibrated. The 29 near-miss items also have labels. The 46 distractor items require judge evaluation for scoring but are included in the decisive split for false-positive measurement (no ground-truth labels). The objective placebo calibration runner cannot profile judge-evaluated items.",+ "bimodal_calibration": "The harder calibration run (calib-systems-harder) showed aggregate baseline 57.4% — valid in-band per library/harness.md aggregate rule for binary families. Per-item distribution is bimodal (13 ceiling, 5 floor) which is expected for binary classification with high-confidence models. The aggregate baseline is what matters for powering.",+ "powered_candidate_status": "VALID POWERED CANDIDATE. Aggregate baseline 57.4% is in [0.40,0.70]. ~43pp headroom available for target ≥5pp lift. The bimodal per-item pattern (ceiling/floor) means the skill can potentially flip systematically-wrong items."},"frozen_at": "2026-06-06T23:00:00.000Z","frozen_by": "m3-harder-data-conceptual-systems"