Skip to content

Fix silent outbound send failures (timeouts and gateway rejections reported as delivered) - #80

Open
LiuZiqi-dual wants to merge 1 commit into
formulahendry:mainfrom
LiuZiqi-dual:fix-silent-send-failures
Open

LiuZiqi-dual wants to merge 1 commit into
formulahendry:mainfrom
LiuZiqi-dual:fix-silent-send-failures

Conversation

@LiuZiqi-dual

@LiuZiqi-dual LiuZiqi-dual commented Aug 27, 2026 •

Copy link
Copy Markdown

Problem

We hit a production incident where the bridge logged Agent done, reply N chars for every turn while the WeChat user received nothing — with zero errors anywhere in the logs. The incident itself turned out to be undiagnosable after the fact, precisely because the outbound send path can neither detect nor report most delivery-failure modes. This PR fixes the two concrete gaps that make deliverReply's retry loop unreachable for whole classes of failures, and makes every send failure visible on stdout:

1. A send timeout is reported as success. apiPost turns AbortError into return { ret: 0, msgs: [] }. That fallback is correct for the getupdates long-poll (timeout just means "no new updates"), but every other endpoint shares the same function — so a sendmessage POST that hangs until the 15s timeout is treated as delivered. The caller sees a clean resolve, sent = true, and the retry loop never fires.

2. Business-level rejections are reported as success. Only res.ok is checked. When the gateway returns HTTP 200 with a non-zero ret/errcode in the body (expired context_token, rate limit, unreachable user), the send is also treated as delivered. This is the send-side validation gap reported in #51 — the poll side (monitor.js) already checks ret/errcode, the send side has no equivalent.

Compounding both: send failures are only reported via trackException, which is a no-op when telemetry is disabled (WECHAT_ACP_TELEMETRY=0) — so even a hard failure that does throw (HTTP 4xx/5xx, connection reset, retries exhausted) leaves no trace on stdout.

Both bugs are reproducible against a mock gateway: with the current code, sendMessage resolves as success both when the gateway hangs past the timeout and when it returns HTTP 200 {"ret":-13}; with this PR it throws in both cases.

To be explicit about scope: #51 also documents a deeper failure mode (HTTP 200 + ret: 0 accepted by the gateway but never routed to the user's phone) that no client-side check can catch. This PR does not claim to fix that; it makes every gateway-visible failure loud and retryable, which narrows any remaining silent loss down to that server-side case.

Fix

  • apiPost gains an opt-in { timeoutAsEmpty: true } flag; only getUpdates uses it. All other endpoints now throw Timeout after Nms: POST <endpoint> on abort, so the existing retry-with-stable-client_id logic in deliverReply actually engages (the gateway de-duplicates by client_id, so retries stay safe).
  • sendMessage validates the response body via a new assertBusinessOk helper and throws on non-zero ret/errcode.
  • Text-segment, image-send, and final-drop failures are now also logged via this.log(...) so they are visible on stdout even with telemetry off.

Callers of the other affected endpoints already tolerate throws: getTypingTicket/sendTypingIndicator wrap in try/catch, cancelTypingIndicator call sites use .catch(() => {}), and getUploadUrl failures feed the existing image retry loop.

Tests

Added tests/api.test.ts covering: send timeout rejects, HTTP-200 + non-zero ret rejects, HTTP-200 + non-zero errcode rejects, clean success resolves, body without status fields resolves, and getupdates long-poll timeout still resolves to an empty batch.

npm test: 224 tests, 223 pass, 1 skipped (pre-existing), 0 fail.


中文说明

问题:我们在生产环境遇到一次事故:桥接日志每轮都记录 Agent done, reply N chars、全程零报错,但微信用户什么都没收到。事后复盘发现这次事故根本无法诊断——因为出站发送路径对大多数投递失败模式既检测不到、也不留痕。本 PR 修复了让 deliverReply 重试逻辑形同虚设的两个具体缺陷,并让所有发送失败在 stdout 可见:

  1. 发送超时被当成功:apiPost 把 AbortError 吞成 {ret: 0} 返回。这对 getupdates 长轮询是对的(超时=没新消息),但所有接口共用这个函数——sendmessage 挂起 15 秒超时后会被当成"已送达",上层 sent = true,重试永远不触发。
  2. 业务层拒收被当成功:只检查了 HTTP 状态码。网关返回 HTTP 200 但 body 里 ret/errcode 非零(context_token 失效、限流、用户不可达)时,同样被当成"已送达"。这正是 sendMessage returns HTTP 200 but message never reaches user (silent delivery failure) #51 报告的发送侧校验缺口——轮询侧 monitor.js 有 ret/errcode 校验,发送侧没有。
  3. 失败无日志:失败只走 trackException,遥测关闭(WECHAT_ACP_TELEMETRY=0)时是空操作——即使真的抛了错(HTTP 4xx/5xx、连接重置、重试耗尽),stdout 也一行不留。

两个缺陷均可用 mock 网关复现:现有代码在"网关挂起超时"和"HTTP 200 + {"ret":-13}"两种场景下 sendMessage 都正常返回;本 PR 后两种场景都会抛错并进入重试。

范围说明:#51 还记录了一种更深的失败模式(HTTP 200 + ret: 0,网关受理但始终不投递到用户手机),这类服务端静默丢弃是客户端校验无法捕获的,本 PR 不声称能修复它;本 PR 的作用是让所有网关可见的失败都变得响亮且可重试,从而把剩余的静默丢失收窄到那种纯服务端场景。

修复:apiPost 新增 { timeoutAsEmpty: true } 选项、仅 getUpdates 使用,其余接口超时抛错以触发既有的稳定 client_id 重试(网关按 client_id 去重,重试安全);sendMessage 新增 assertBusinessOk 校验返回体;文本分段、图片发送、最终丢弃三处失败均通过 this.log(...) 落 stdout。其余受影响接口的调用方均已具备 try/catch 或 .catch() 兜底。

测试:新增 tests/api.test.ts 共 6 例(超时抛错、ret 非零抛错、errcode 非零抛错、正常成功、无状态字段成功、长轮询超时语义不变)。npm test:224 例,223 通过,1 例既有跳过,0 失败。

🤖 Generated with Claude Code

sendMessage had two paths that reported success while the message was
never delivered, defeating deliverReply's retry loop:

1. apiPost swallowed AbortError and returned {ret: 0} as a fake success.
   That behavior is correct for the getupdates long-poll (timeout = no
   new updates) but wrong for every other endpoint: a sendmessage POST
   that timed out was treated as delivered. The empty-success fallback
   is now opt-in via {timeoutAsEmpty: true}, used only by getUpdates;
   all other endpoints throw on timeout so retries kick in.

2. Only the HTTP status was checked. A gateway rejection carried in an
   HTTP-200 body (non-zero ret/errcode: expired context_token, rate
   limit, unreachable user) was also treated as delivered. sendMessage
   now validates the response body and throws on a business error.

Additionally, send failures were only reported through trackException,
which is a no-op when telemetry is disabled — so a dropped reply left
no trace in the logs. Text-segment, image-send, and final-drop failures
are now also logged via this.log so operators can see them on stdout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant