Skip to content

armgit / rea

Public Imported from GitHub
0

Reverse engineer anything with agents, from app behavior down to native binaries.

fix(tests): separate Copilot model IDs from wire models (#1647)

Ysopwnableb09dd333ab75
diff --git a/docs/installation.md b/docs/installation.md
index 7a6359a9b..4a59ad89b 100644
--- a/docs/installation.md
+++ b/docs/installation.md
@@ -153,6 +153,13 @@ listed after the table because its connector is not one of these files:
For GitHub Copilot CLI 1.0.95, `gpt-4.1` model metadata can block ordinary
chat with REA's full input-schema profile before sending a model request
(`compaction_static_context_blocked`; [#1554](https://github.com/morluto/rea/issues/1554)).
+For the native offline OpenAI-compatible BYOK adapter, full-profile chat and
+analysis also passed with a distinct `COPILOT_PROVIDER_MODEL_ID` such as
+`rea-gpt-4.1`, `COPILOT_PROVIDER_WIRE_MODEL=gpt-4.1`, and that distinct ID selected
+through `copilot --model rea-gpt-4.1`. The local fixture requested
+`COPILOT_PROVIDER_MAX_PROMPT_TOKENS=1000000`; configure the limit supported by
+your backend. This keeps every full schema and the actual wire model intact.
+It does not establish the effective client budget or fix the built-in-ID path.
The existing compact profile passed native chat and analysis with the complete
tool inventory on Linux using a loopback model fixture. After setup, add
`"REA_MCP_INPUT_SCHEMA_PROFILE": "compact"` to the `env` object of the
diff --git a/docs/testing.md b/docs/testing.md
index 3bcc882bd..3c05df096 100644
--- a/docs/testing.md
+++ b/docs/testing.md
@@ -160,8 +160,25 @@ Use `-- chat` for ordinary
chat or `REA_VERIFY_RUNTIME_ROOT` for a production-only installed package.
The native offline BYOK adapter uses a loopback OpenAI completions/SSE fixture
and `gpt-5.4` model metadata; model inference and token usage are synthetic.
-Set `REA_VERIFY_COPILOT_MODEL` to examine another model configuration. In the
-verified client, `gpt-4.1` blocks the full input-schema profile before HTTP with
+Set `REA_VERIFY_COPILOT_MODEL` to examine another client model ID and
+`REA_VERIFY_COPILOT_WIRE_MODEL` for its backend model name (defaults to the client
+ID). Every actual HTTP request's model is checked and recorded separately.
+Both `HOME` and `USERPROFILE` point to the disposable account; native `skill add`
+still selects the installed skill directory explicitly.
+
+The same native client passed full-profile chat and analysis with the actual
+wire model `gpt-4.1` under a distinct BYOK model ID:
+
+```bash
+REA_VERIFY_COPILOT_MODEL=rea-gpt-4.1 REA_VERIFY_COPILOT_WIRE_MODEL=gpt-4.1 npm run verify:copilot-client -- chat
+REA_VERIFY_COPILOT_MODEL=rea-gpt-4.1 REA_VERIFY_COPILOT_WIRE_MODEL=gpt-4.1 npm run verify:copilot-client -- call
+```
+
+These runs retain full canonical input/output schemas, all tools, installed-skill
+loading and complete Evidence readback. They use the fixture's requested
+one-million-token BYOK prompt setting; effective capacity and live API acceptance
+remain unknown. With the built-in `gpt-4.1` model ID, the verified client still
+blocks the full input-schema profile before HTTP with
`compaction_static_context_blocked`, even when requesting a larger BYOK prompt
capacity. The effective capacity is unknown. The default verifier preserves
the complete catalog and full schemas. To verify the existing compact profile:
diff --git a/scripts/verify-copilot-client.mjs b/scripts/verify-copilot-client.mjs
index e3980d269..98ed62efb 100644
--- a/scripts/verify-copilot-client.mjs
+++ b/scripts/verify-copilot-client.mjs
@@ -22,6 +22,7 @@ const runtimeRoot = process.env.REA_VERIFY_RUNTIME_ROOT ?? repo;
const command = process.env.REA_VERIFY_COPILOT_COMMAND ?? "copilot";
const mode = process.argv[2] ?? "call";
const model = process.env.REA_VERIFY_COPILOT_MODEL ?? "gpt-5.4";
+const wireModel = process.env.REA_VERIFY_COPILOT_WIRE_MODEL ?? model;
const inputSchemaProfile =
process.env.REA_VERIFY_COPILOT_SCHEMA_PROFILE ?? "full";
assert(
@@ -50,6 +51,7 @@ await exec("git", ["init", "--quiet", workspace], { timeout: 30_000 });
const environment = {
PATH: process.env.PATH,
NODE_OPTIONS: process.env.NODE_OPTIONS,
+ HOME: account,
USERPROFILE: account,
COPILOT_HOME: profile,
COPILOT_OFFLINE: "true",
@@ -60,7 +62,7 @@ const environment = {
COPILOT_PROVIDER_BEARER_TOKEN: "",
COPILOT_PROVIDER_HEADERS: "",
COPILOT_PROVIDER_MODEL_ID: model,
- COPILOT_PROVIDER_WIRE_MODEL: model,
+ COPILOT_PROVIDER_WIRE_MODEL: wireModel,
COPILOT_PROVIDER_MAX_PROMPT_TOKENS: "1000000",
COPILOT_PROVIDER_MAX_OUTPUT_TOKENS: "4096",
REA_PROCESS_RUN_ID: verifier.run_id,
@@ -118,10 +120,17 @@ const extractEvidence = (value) => {
}
return undefined;
};
+const wireModels = [];
const fixture = await createOpenAiModelFixture({
directory: lab,
prefix: "copilot",
onRequest: async ({ body, tools, requestIndex }) => {
+ assert.equal(
+ body.model,
+ wireModel,
+ "Copilot must send the selected wire model",
+ );
+ wireModels.push(body.model);
// Auxiliary requests without tools cannot execute the analyst's workflow.
if (tools.length === 0)
return { role: "assistant", content: "REA fixture title" };
@@ -424,6 +433,7 @@ try {
clientVersion,
mode,
model,
+ wireModel,
inputSchemaProfile,
passed: failure === undefined,
artifacts: lab,
@@ -440,7 +450,7 @@ try {
capacityBoundary:
inputSchemaProfile === "compact"
? "Explicit compact profile: complete inventory and canonical server validation; advertised schemas may be reduced. This does not fix the full-profile gpt-4.1 context boundary."
- : "gpt-4.1 model configuration blocks the full catalog before HTTP even with a ten-million-token requested prompt override; gpt-5.4 is the verified full-profile local fixture configuration",
+ : "Complete canonical input and output schemas. Client model ID and wire model are recorded separately; effective token capacity and live provider compatibility remain unknown.",
catalogSize,
schemasChecked,
spillReadRequested,
@@ -448,6 +458,7 @@ try {
skillLoaded,
artifactDigest,
finalSeen,
+ wireModels,
requests: fixture.requests,
probes: fixture.probes,
fixtureFailure: fixture.failure,