Skip to content

fix(gmail): read every history page before advancing the watch cursor - #1312

Closed
saariuslystoned wants to merge 2 commits into
openclaw:mainfrom
saariuslystoned:fix/gmail-watch-history-pages
Closed

saariuslystoned wants to merge 2 commits into
openclaw:mainfrom
saariuslystoned:fix/gmail-watch-history-pages

Conversation

@saariuslystoned

@saariuslystoned saariuslystoned commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

What problem this solves

For each Pub/Sub notification, gog gmail watch serve and gog gmail watch pull call users.history.list once, with the default 100 records (defaultHistoryMaxResults). gmailWatchSource.ListHistory never reads nextPageToken. The processor then stores the response's historyId as the new cursor. The users.history.list reference describes that historyId as "the ID of the mailbox's current history record" and says:

If you receive no nextPageToken in the response, there are no updates to retrieve and you can store the returned historyId for a future request.

When a notification covers more than one page of changes, records 101 and later are never fetched. The stored cursor then moves past them, so the next notification starts after them too, and those messages never reach the hook. A burst of new mail, a bulk label change, or a watch pull catch-up after the consumer was down can each produce more than 100 records.

Change

ListHistory pages through users.history.list with pageToken until no nextPageToken remains. It uses the existing collectAllPages helper, which rejects a repeated page token and stops at its page limit with an error, so the processor never advances the cursor over unread history. The returned historyId comes from the last page. The processor, the HistoryPage type and --history-types handling are unchanged. docs/watch.md notes the behavior.

Evidence

  • TestGmailWatchSourceListHistoryReadsEveryPage uses a two-page httptest history and expects two calls (the second with pageToken=p2), both records, and historyId 300. On main (e8428c3) it fails with one call.

  • TestGmailWatchSourceListHistoryRejectsRepeatedPageToken expects an error, not a cursor advance, when the token repeats. It fails on main.

  • TestGmailWatchServer_DeliversHistoryBeyondFirstPage posts a push notification to the watch server against the same two-page history and checks the hook payload. On main the hook gets only m1, and the stored cursor is still advanced to 300. With this change the hook gets m1,m2. The existing watch, history and processor tests pass.

  • Built gog binaries from main and from this branch were run against a local stand-in through GOG_GOOGLE_API_BASE_URL, with an empty GOG_HOME and the file keyring. The stand-in is not Google: it serves m1 on history page 1 with nextPageToken: p2, m2 on page 2, and historyId: 300, plus a local hook receiver. Each run does gmail watch start (cursor 100), starts gmail watch serve --token tok --hook-url <local> --fetch-delay 0, POSTs one push for historyId 300, then reads gmail watch status:

    main:   history requests: pageToken=''              hook got: m1      saved historyId: 300
    branch: history requests: pageToken='', 'p2'        hook got: m1,m2   saved historyId: 300
    

    On main, m2 is never fetched, and the saved cursor is already past it, so a later push will not pick it up.

  • make ci passed locally (macOS arm64, go1.27.3).

Boundary: no live Gmail mailbox or Pub/Sub subscription was used. The paging behavior follows the users.history.list reference. Each notification now fetches every changed message in its range, which is the work the earlier behavior skipped.

🤖 Generated with Claude Code

watch serve and watch pull made one users.history.list call (100
records) per notification and then stored the response historyId, which
is the mailbox's current history ID. Records after the first page were
never fetched, and the stored cursor skipped them for good. Gmail's
reference says the historyId can be stored once no nextPageToken is
returned. Page through the history with the repeated-token guard before
the processor advances the cursor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@clawsweeper

clawsweeper Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. P2 Normal priority bug or improvement with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. labels Nov 8, 2026
@clawsweeper

clawsweeper Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge.

What this changes

Fetch every Gmail history page before returning the cursor used by watch push and pull processing.

Example: Cursor 100 yields m1 on page one, m2 on page p2, and mailbox history ID 300.

  • Before: Only m1 reaches the hook, while the saved cursor advances to 300.
  • After: Both m1 and m2 reach the hook, with the saved cursor advancing to 300.

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused repair with meaningful regressions and recorded CLI behavior; proof uses a local HTTP stand-in, and persisted state formats and loading remain unchanged.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (live_output): The updated body records built CLI binaries on macOS arm64 using the production Google HTTP client against a local two-page stand-in: watch serve requests p2, delivers m1,m2, and watch status reports cursor 300; main delivers only m1.
Patch quality 🦞 diamond lobster (5/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: Messages beyond the first history page never reach the configured hook because the cursor skips past them.
Reason: Complete history retrieval restores the existing watch contract without new configuration, public APIs, or unrelated changes.

Merge readiness

✅ Ready for maintainer review

This PR fixes a message-loss bug still present on main; no actionable patch defects remain, and the previous changelog finding is resolved.

Priority: P2
Reviewed head: e41deea8963619f897cade18ba7f2f5b32d68cca

Before merge

None.

Findings

None.

Agent review details

How this fits together

Push and pull consumers share a Gmail source adapter that converts history responses into records for message fetching, hook delivery, and cursor persistence.

flowchart TD
  A[Push or pull notification] --> B[Watch processor]
  B --> C[Gmail source adapter]
  C --> D[Gmail history pages]
  D --> C
  C --> B
  B --> E[Message fetch and hook delivery]
  B --> F[Saved watch cursor]
Loading

Technical review

Best possible solution:

Land the focused pagination repair in the shared Gmail source adapter.

Do we have a high-confidence way to reproduce the issue?

Current-main source shows the single-page request followed by cursor advancement; the contributor also records a main-versus-branch CLI demonstration, which this read-only review did not rerun.

Is this the best way to solve the issue?

The patch fixes retrieval in the existing shared owner, reuses the established pagination guard, and preserves processor behavior and filtering.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against e8428c36dd31.

Provenance checked

  • Gmail watch history retrieval changes intended behavior with a stated reason (ae251d2: The commit extracted the watch source adapter; it establishes the current placement, not the original introduction of single-page behavior.)
  • Equal-history notification handling keeps the original intent (fix(gmail): inspect pushes matching stored history #1140: Equal-history notifications must still inspect Gmail history while older notifications and duplicate message IDs remain filtered.)

Testing

Proof path: shipped entry point.

Security

None.

Evidence

What I checked:

  • Current main still reads one page: ListHistory makes one history.list request and returns its history ID without consuming NextPageToken. (internal/cmd/gmail_watch_source.go:38, e8428c36dd31)
  • Focused shared-adapter repair: The patch preserves query filters and context, consumes page tokens through collectAllPages, returns the final page's history ID, and propagates errors without returning partial records. (internal/cmd/gmail_watch_source.go:41, e41deea89636)
  • Recorded shipped-command behavior: The updated PR body records built main and branch binaries exercising watch start, watch serve, a push, and watch status through GOG_GOOGLE_API_BASE_URL: main requests only page one and delivers m1; the branch requests p2 and delivers m1,m2, with cursor 300. This uses a local HTTP stand-in, not live Gmail. (e41deea89636)
  • Previous finding resolved: The follow-up commit adds the PR reference to the Unreleased changelog entry, which already credits the contributor. (CHANGELOG.md:5, e41deea89636)
  • Feature history: Peter Steinberger authored the original watch command and the later source-adapter extraction; current-main blame carries the adapter's single-page implementation from that extraction. An extended historical content search encountered unavailable promisor blobs, so the exact earlier introduction of single-page behavior is not asserted. (internal/cmd/gmail_watch_source.go:38, e8428c36dd31)
  • Existing adjacent cursor repair: fix(gmail): inspect pushes matching stored history #1140 repairs equal-history notification selection, a separate boundary preserved by this patch. (docs/watch.md:106, 0ebd74d7212f)

Likely related people:

  • Peter Steinberger: Raw commit ae251d2 adds internal/cmd/gmail_watch_source.go:34 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: ae251d28db07; files: internal/cmd/gmail_watch_source.go)
  • 江口 和幸: Raw commit 0ebd74d adds docs/watch.md:99 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: medium; commits: 0ebd74d7212f; files: docs/watch.md)

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Labels

Label changes:

  • add proof: sufficient: Contributor real behavior proof is sufficient.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🦞 diamond lobster.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.
  • remove rating: 🧂 unranked krab: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
  • remove status: 📣 needs proof: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

  • P2: Watch notifications spanning multiple history pages can permanently skip later-page messages.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.
  • proof: sufficient: Contributor real behavior proof is sufficient.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

History

Review history (1 earlier review cycle)
  • reviewed 2026-11-08T14:55:16.873Z sha b5b7f69 :: needs real behavior proof before merge. :: [P3] Add the required PR reference to the changelog entry

Reviewed November 8, 2026, 2:19 PM ET / 19:19 UTC (Revision 2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@saariuslystoned

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
Exact review queued.

Re-review progress:

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. clawsweeper:intake:bug Preliminary submission classification; full source review pending. labels Nov 8, 2026
@steipete

Copy link
Copy Markdown
Collaborator

Thank you for the fix. I reproduced the claim and consolidated consuming every Gmail history page before advancing the watch cursor into #1336, with regression coverage and your coauthor credit retained. The full make ci gate, built-CLI checks for the new request-validation fixes, and exact-head GitHub CI passed. Closing this PR as superseded by the merged implementation in #1336.

@steipete steipete closed this Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants