Skip to content

Add opt-in Agent Message Compat gateway hooks for Codex V2 - #783

Open
YHonc wants to merge 2 commits into
magpie-community:mainfrom
YHonc:add-agent-message-compat
Open

YHonc wants to merge 2 commits into
magpie-community:mainfrom
YHonc:add-agent-message-compat

Conversation

@YHonc

@YHonc YHonc commented Nov 6, 2026 •

Copy link
Copy Markdown

Codex MultiAgent V2 tasks using the agents namespace can carry plaintext inside encrypted_content, even when the original call has encrypted_function_args: []. This opt-in package restores exact strings with recorded local plaintext provenance and converts readable agent messages for configured routes.

Version 0.1.2 defaults to targets: [] and performs no ledger I/O until configured. Unknown sealed, empty and unsupported items pass unchanged to Magpie's native handling. It uses core-style From X to Y headers, a bounded ledger (2048 entries / 30 days by default), accepted diagnostic codes and deterministic capture tests. There is no provenance wait. Known history replay does not rewrite the ledger.

The package is named @magpie-community/gateway-agent-message-compat and listed under Gateway hooks. It includes English/Chinese documentation, cases.json checked through the community middleware harness, and PROVENANCE.json in the published files.

Source: v0.1.2, pinned to 83cd09f45a1ae04035b647f1b1af7f05eac1066e.

Real evidence: November 6 rollout pair and runtime attestation. The pair came from the actual implementation work, with original equality and parent/child linkage checked before redaction. It records Codex 0.166.0-alpha.6; the runtime observation links the active local gateway to a Magpie 0.1.2964 startup. These are rollout records and process observations, not an HTTP capture.

This preserves a verified task instead of removing historical ciphertext as codex-history can. It does not decrypt ciphertext or scan transcripts. Routes with native plain-message conversion should remain outside targets. The README distinguishes the #3183 marker fix from #4108 and documents the inspected baseline without inventing a first fixed release.

Validation:

  • node test.mjs: 13/13 offline module groups.
  • bun test packages/agent-message-compat: 8/8 tests, including 7 middleware cases.
  • bun scripts/check.mjs agent-message-compat: registry validation passed.
  • git diff --check, source-pin/hash checks and publication-content scan passed.

The offline checks verify the updated plugin. The captured live task used the existing installed plugin and is evidence of the message representation, not live acceptance of v0.1.2.

@yetone yetone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. It's careful: the hook stores only hashes, and only from local origins. The restore never decodes anything, it only relabels a string that is already in the request. Nothing it adds to a prompt names magpie or a vendor. There is no network access and no eval.

What I ran at 09dd56a, on macOS with HOME=$(mktemp -d):

  • node test.mjs: 12/12;
  • bun test packages/agent-message-compat: pass;
  • bun scripts/check.mjs agent-message-compat: ok.

Then I broke each guard by hand, and every one turned the test red: the encrypted_function_args length check, isLocal, a known() that always matches, the middleware's 409, and storing plaintext instead of the hash. PROVENANCE matches 63b93543 for every runtime file.

Things to change before merging:

  1. Make the default opt-in. The default targets (cpa/…, ora2/…, group/glm-*, group/deepseek-*) are your own route names. In someone else's magpie they match whatever happens to share the name. Ship "targets": [], as codex-history ships providers: {}, and keep your list as the README example.

  2. Don't pre-empt magpie's own handling of sealed content. The middleware runs before routing, so on a targeted model it replaces what the gateway does today with a 409:

    • a sealed Message Type: MESSAGE is replaced with a note and the turn goes on (#1613, withoutSealedMessages);
    • a sealed task goes to the ChatGPT account or Responses relay that sealed it (#619, #1109);
    • when nothing can open it, the refusal tells the user the way out (sealedWayOut);
    • an agent_message with no content adds no turn, where the middleware answers 400.

    Leave every item you didn't restore untouched, and drop the 400 for an empty message.

  3. Show a real case of "plaintext labelled as encrypted_content" on current magpie.

    • Since v0.1.2747 (#3183, 3b961f294), the gateway answers every relayed namespaced call with encrypted_function_args: []. Hooks see the reply as Codex got it.
    • With that marker, Codex hands the task on as input_text.
    • The hook only records calls that carry the marker, so it is not clear when such a call's message later comes back as encrypted_content.
    • Please attach a captured, redacted pair from a real run: the call and the later agent_message, with your Codex and magpie versions.
    • The test fixture is made up, and the #4108 probe says its encrypted_content was plaintext typed in by hand.
    • If the case only exists for threads started before #3183, say so in the README.
  4. Step 2 overlaps core #4108, which is open with a verified patch: plain agent_message sent as a user turn, "From X to Y", on every relayed Responses API except OpenAI, Azure, Copilot and another magpie. Please say in the README which magpie versions need step 2, and match its header wording, so a thread isn't sent two forms of the same message after an upgrade.

  5. Bound the ledger.

    • It never shrinks, it is read and rewritten whole on every new hash, and it survives removal.
    • Add a cap and an age limit.
    • The README (en and zh) should say where the file is, what is in it, and that it stays after plugin rm.
    • It should also say that a hash of short or guessable task text can be confirmed by anyone holding the file.
  6. Use report codes the host accepts. ctx.report takes only ready, empty, unbound, unauthorized, unavailable, invalid and skipped. So compat_ledger_read_failed and compat_ledger_write_failed are dropped. A corrupt ledger then blocks every later write with nothing shown. Use invalid or unavailable.

  7. Package conventions.

    • A package with magpie.hooks is named @magpie-community/gateway-<name> and listed under "Gateway hooks" (powercontext, aelios, nowledge-mem, respire-memory).
    • Middleware has a cases.json checked with check() from scripts/middleware.mjs.
    • Give the README an English section beside the Chinese.
    • Drop the "install from github:YHonc/… while under review" text from the community copy.
    • Add PROVENANCE.json to files.
  8. Make the race check wait on state. It needs a 25 ms timer to land inside a 150 ms window, and on a busy machine it can miss. Have the test await the capture before the hook reads the ledger.

  9. Minor: a sealed value that is never recorded (any real gAAAA… ciphertext) waits the full 800 ms before anything happens. It would help to say that in the README, or to skip the wait when nothing is pending.

@YHonc YHonc changed the title Add Agent Message Compat for Codex MultiAgent V2 Add opt-in Agent Message Compat gateway hooks for Codex V2 Nov 6, 2026
@YHonc

YHonc commented Nov 6, 2026

Copy link
Copy Markdown
Author

Updated in 6100658f732e694ff726219e01af51536a89032d, with source v0.1.2 pinned to 83cd09f45a1ae04035b647f1b1af7f05eac1066e.

  1. Defaults are now opt-in: targets: [] in both runtime defaults and package options. No ledger I/O occurs until targets are configured.
  2. Unrestored sealed items, empty messages and unsupported/mixed content remain unchanged for Magpie's native handling. The plugin no longer returns 400/409 for them. Restoration is atomic per item.
  3. Added a real November 6 call/delivery pair, captured from this implementation work rather than constructed by a probe. It shows agents.spawn_agent, encrypted_function_args: [], followed by the new child's encrypted_content containing the exact original message. Equality, chronology, parent/child linkage and Codex 0.166.0-alpha.6 were verified before redaction. The distinction is the agents namespace: inspected direct_source() at that Codex tag only selects the plaintext branch for collaboration. The runtime attestation links the current listener to a Magpie 0.1.2964 startup; it is a process observation, not request-level HTTP tracing. The task succeeded with the installed 0.1.1 plugin; this pair establishes the representation, not a failed delivery or live acceptance of 0.1.2.
  4. Headers now match core wording (From X to Y, with one-sided forms), conversion is idempotent, and the README describes the inspected 0.1.2964 baseline and conditional #4108 scope. Routes with native conversion should stay outside targets.
  5. Ledger defaults are bounded to 2048 entries and 30 days, configurable within documented limits. Known history replay avoids rewriting it. Both README sections document the path, contents, persistence after removal and confirmability of short/guessable text.
  6. Diagnostics use invalid/unavailable. A corrupt file is preserved as .corrupt.<UUID> and storage failures are reported without intercepting the request.
  7. Renamed the community package to @magpie-community/gateway-agent-message-compat, moved the listing to Gateway hooks, added cases.json through scripts/middleware.mjs, included PROVENANCE.json in files, added English documentation and removed the community copy's GitHub installation instructions.
  8. Capture tests now await completed state; the timer-window race is gone.
  9. Removed the provenance wait entirely, so an unrecorded sealed value does not incur the former 800 ms delay.

Local offline results: 13/13 module groups, 7/7 middleware cases (8 passing Bun tests including the module group), registry check and diff/source-integrity checks passed. No installed plugin or provider configuration was changed. Ready for another review.

@yetone yetone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the careful revision. I went through all nine points against 6100658, line by line, and nearly everything holds. What is left is two README problems (one breaks the install), a test for one guard, and a files entry.

What I ran at 6100658 (Bun 1.4.2, Node 22, on a separate macOS test machine, HOME=$(mktemp -d)):

  • node test.mjs: 13/13.
  • bun test packages/agent-message-compat: 8 pass, 0 fail.
  • bun scripts/check.mjs agent-message-compat: ✓ registry.json.
  • Whole-repo bun test: 6252 pass, 20 skip, 0 fail, 461 files.
  • I loaded compat.middleware.js (with its ./common.js import) into magpie's own middleware runtime at yetone/magpie 11df525f0, through a throwaway internal/middleware test with targets: ["ex/*"]. A readable agent_message came out as {"type":"message","role":"user","content":[{"type":"input_text","text":"From /root to /root/w\n\n"},{"type":"input_text","text":"do it"}]}. A sealed one beside it was left as it was, and an untargeted model was not converted.
  • PROVENANCE.json: common.js, compat.middleware.js, provenance.hooks.mjs, test.mjs, LICENSE and CHANGELOG.md match both the pinned hashes and the bytes at YHonc/magpie-agent-message-compat@83cd09f4. The README differs, as changes says it does. There is no network access, no eval and no child process.

I broke 24 guards by hand, one at a time, and ran both test files each time. 21 turned red: enabled always true, isLocal always true, applies ignoring the protocol, the marker length check, the marker array check, the success guard, known() always true, plaintext stored in place of the hash, prune without the age filter, prune without the cap, no invalid report, no .corrupt. backup, no unavailable report, a replay that rewrites, the hook accepting mixed parts, the hook taking an item-level encrypted_content, the middleware's part-level seal check, the blank-text check, the unknown-part check, the header wording, and the one-sided headers. Three stayed green:

  • !item.content.length removed from the middleware. This is an equivalent mutant: !item.content.some(...) is true for an empty list anyway. That's fine.
  • 'encrypted_content' in item removed from the middleware. No test has an item that carries encrypted_content itself and readable parts. Every sealed fixture is also caught by the part check or by the missing content.
  • ['agents','collaboration'].includes(item.namespace) removed from the hook. No test records, or refuses, a call in another namespace. For example, an MCP tool named send_message with a message argument and encrypted_function_args: [] would be recorded today without the guard.

The nine points

  1. Opt-in. Holds. DEFAULT_TARGETS = [] and magpie.options.targets: []. enabled() gates every ledger read and write, and the first test asserts that the temp dir stays empty.
  2. Not pre-empting core. Holds. Neither file returns 400 or 409 any more, and nothing calls ctx.reject. The middleware converts only an item whose every part is a readable input_text or input_image. The hook restores only when every encrypted_content part is a known hash and every other part is readable. Everything else reaches withoutSealedMessages, sealedWayOut and the ChatGPT/relay routing unchanged. Hooks run before middleware (internal/gateway/middleware.go:45), so restore-then-convert happens in the order the design needs.
  3. A real case. I judge it real, but conditional, and the README should say on what condition. The 11-06 pair has the structure of a Codex rollout: internal_chat_message_metadata_passthrough, per-part provenance, and ordinals. Its times agree with each other (parent create_time 08:52:09Z, call 08:52:14.7Z, child 08:52:15.8Z). It matches Codex rust-v0.166.0-alpha.6 exactly. ToolCall::direct_source (codex-rs/core/src/tools/router.rs:56-71) gives DirectPlaintextMessage only when the namespace is literally "collaboration". In every other case agent_message_from_tool (codex-rs/core/src/tools/handlers/multi_agents_v2.rs) wraps the plaintext as AgentMessage::Encrypted. But "collaboration" is only the default of features.multi_agent_v2.tool_namespace (DEFAULT_MULTI_AGENT_V2_TOOL_NAMESPACE, core/src/config/mod.rs:265, the same on Codex main today). Your calls are in "agents", so your Codex has that namespace set to something else. On a default config, every MultiAgent V2 call I have in local rollouts (several hundred) is in collaboration, and Codex hands the task on as input_text. The hook then records hashes that never get used. The README says that agents calls skip the plaintext branch, but not that agents is a non-default setting. Please say plainly that restoration only does anything when Codex's MultiAgent V2 tool_namespace is not collaboration. It is a Codex bug: direct_source hard-codes the default instead of reading the configured namespace. On Codex's default config, only the middleware's conversion applies.
  4. #4108. Holds. #4108 is still open on yetone/magpie, and nothing on main (11df525f0) converts a plain agent_message on a relayed Responses request. The header now matches grokAgentMessage (internal/provider/grok.go:279-327), which #4108 copies. That means a separate "From A to B\n\n" input_text part first, the one-sided forms From A / To B, and nothing when both are empty. Once converted, the item is an ordinary message, so a core build with #4108 won't convert it again. The README's "inspected 0.1.2964 baseline, remove routes with native conversion" is fair while no fixed release exists.
  5. Ledger bounds. Holds. 2048 entries and 30 days by default, clamped to 1–10000 and 1–365. Pruning happens on load, on save and before a lookup. A known replay doesn't rewrite the file (the mtime test), and the file is written with temp-and-rename, mode 0600. Both README sections give the path and the contents, say that the file stays after off/rm, and say that short or guessable text can be confirmed.
  6. Report codes. Holds for onRequest. One thing you can't see from the plugin: the host gives onResponse the same ctx that onRequest had, and that report closure is dead by then (host.js:1499 checks the request's own signal, which was aborted in its finally). A read or write failure while recording a reply is therefore dropped silently. It also stays unreported afterwards: the hash is already in memory, so the next request's remember sees nothing new and doesn't save. test.mjs's "write failure reports unavailable" passes only because its ctx is not the host's. This isn't a blocker. Either add one README line that a failure while recording a reply isn't shown, or retry the save on the next onRequest when the last one failed.
  7. Package conventions. Mostly holds. The name is @magpie-community/gateway-agent-message-compat, it is listed under Gateway hooks, kind: "hooks" is accepted (scripts/registry.mjs, plugins.js kindOf), cases.json goes through check(), there is an English section, no github: install text, and PROVENANCE.json is in files. But files also lists "evidence/", and the community package has no such folder. Please drop it, since the README links the evidence in your repo anyway.
  8. Race. Holds. Every capture is awaited before the read, and there are no timers.
  9. Wait. Holds. waitForProvenanceMs is gone, and an unknown value passes through at once.

To change before merging

  1. The install commands don't work. magpie plugin add community:agent-message-compat, the PowerShell copy of it, and plugin off/on/rm community:agent-message-compat all fail. magpie has no community: prefix. plugin.resolve (internal/plugin/resolve.go) takes npm:, a path, a git spec, an @scope/ name, or a name a community listing goes by. community:agent-message-compat is none of these, so the user gets "no plugin is called …". Use magpie plugin add @magpie-community/gateway-agent-message-compat, as powercontext, aelios and nowledge-mem do. The short name agent-message-compat also resolves, through the listing's name "Agent Message Compat". Use the same name for off, on and rm.
  2. Say when restoration applies (point 3 above): only when Codex's features.multi_agent_v2.tool_namespace is set to something other than collaboration. On Codex's default config, only the conversion step applies. Do it in both the English and the Chinese section.
  3. Add a test for the namespace guard. A call outside agents/collaboration must not be recorded. If you like, also add a middleware case for an item that has encrypted_content on itself and readable parts. Both guards are in the code. Today no test fails when either is removed.
  4. Drop "evidence/" from files.

Optional: the onResponse report note in point 6.

Once 1–4 are in, I expect to approve.


中文总结

逐条核对了 6100658。9 条里 1、2、4、5、8、9 都已做到。6 在 onRequest 里成立,但宿主会丢掉 onResponse 里的报告,这条可选。7 只差 files 里多了一个不存在的 evidence/。第 3 条的证据我认为是真实的:结构和时间都对得上,也与 Codex 0.166.0-alpha.6 源码一致,direct_source 只认字面上的 collaboration 命名空间。但它只在 features.multi_agent_v2.tool_namespace 被改成非默认值(如 agents)时出现,默认配置下恢复步骤不会起作用,README 需要写明。

测试情况:包内测试 13/13,Bun 8 通过,check 通过,全仓 bun test 6252 通过 0 失败。middleware 在 magpie 自己的运行时里实测转换正确。手动破坏 24 处守卫,21 处测试变红。剩下 3 处里,1 处是等价变异,另外 2 处(命名空间守卫、条目级 encrypted_content)缺测试。

需要修改:README 的安装和开关命令 community:agent-message-compat 在 magpie 里无法解析,要改成 @magpie-community/gateway-agent-message-compat 或 agent-message-compat;写明恢复步骤的适用条件;补命名空间守卫的测试;删掉 evidence/。改完后预计可以批准。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants