Skip to content

bench: stop starting ops that cannot finish before the window ends - #20

Merged
pjbgf merged 5 commits into
mainfrom
drain-level-end
Oct 20, 2026
Merged

pjbgf merged 5 commits into
mainfrom
drain-level-end

Conversation

@pjbgf

@pjbgf pjbgf commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

https://entire.io/gh/entireio/forgemark/trails/9

Agents keep a per-op latency estimate that rises at once and decays slowly, and stop issuing pushes/clones expected to overrun the level window. In-flight ops get drainGrace (30s) before the hard cancel, so a queueing server no longer sees a burst of cancelled requests at every level end. tokenCovers accounts for the drain.


Note

Medium Risk
Changes benchmark timing, cancellation, and JWT token budgeting; behavior under load and long runs shifts, but scope is limited to the forgemark bench harness rather than production forge paths.

Overview
Benchmark agents now track a per-op-kind latency estimate (fast rise on slow samples, slow decay) and refuse to start clones/pushes that are unlikely to finish before the level window ends. Runner.RunLevel passes an explicit end alongside the existing start, extends the level context by drainGrace (30s) only for ops that overrun that estimate, and tokenCovers budgets that drain per remaining level.

Clone, session, and default push loops all gate on fits() for the relevant OpKind. internal/server/runmgr.go comments are updated so live bucket emission still accounts for post-window drain (and session ref cleanup). New agent_test.go covers estimate behavior, fits, and an integration check that the window close does not cancel in-flight HTTP requests.

Reviewed by Cursor Bugbot for commit a281030. Configure here.

@pjbgf

pjbgf commented Oct 20, 2026

Copy link
Copy Markdown
Member Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit a281030. Configure here.

Comment thread internal/bench/runner.go Resolved
pjbgf added 2 commits October 20, 2026 12:21
Agents keep a per-op latency estimate that rises at once and decays
slowly, and stop issuing pushes/clones expected to overrun the level
window. In-flight ops get drainGrace (30s) before the hard cancel, so a
queueing server no longer sees a burst of cancelled requests at every
level end. tokenCovers accounts for the drain.

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paulo Gomes <paulo@entire.io>
Entire-Checkpoint: 1ad957e18cb8
RunLevel returned as soon as every agent stopped early, closing the
server's live series before the window ended while summarize still
divided by the full duration. It now waits until end or cancellation.

The fits check ran before building the commit, so a slow commit could
start the push after end and count it as in-window. Agents recheck
right before pushing, in the branch and session loops.

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paulo Gomes <paulo@entire.io>
Entire-Checkpoint: b9123b0e9124
pjbgf added 3 commits October 20, 2026 12:22
Ops that overran the window finished during the drain and were recorded
after measEnd. Periodic ticks stop there, so the forced tail flush put
every drain completion into a bucket whose dt stops at the window end,
spiking the live and persisted rate. Such ops still run to completion,
so the server sees no cancellation, but go unrecorded, matching what
summarize counted before the drain existed.

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paulo Gomes <paulo@entire.io>
Entire-Checkpoint: ad4bd90faeef
Dropping ops that finished during the drain hid real work and tail
latency exactly when the server slows down. They are recorded again and
summarize attributes them by start offset. The live collector instead
skips samples completing after the window end, so they no longer pile
into the tail bucket whose duration stops there.

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paulo Gomes <paulo@entire.io>
Entire-Checkpoint: 98b9390e1b08
The drain exists to avoid cancelling in-flight requests at the window
end, but nothing measured how many still were. Agents now count ops cut
off by ctx (the drain deadline or a cancelled run), and LevelResult
carries the total as cancelled, printed in the CLI row. They remain
excluded from samples.

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paulo Gomes <paulo@entire.io>
Entire-Checkpoint: 73bcbf0d656c
@pjbgf

pjbgf commented Oct 20, 2026

Copy link
Copy Markdown
Member Author

The changes now report the number of cancelled operations, while trying to reduce them.

Reporting only:

  c=64   push_ok=13360  push/s=111.3    p50=442.8   p95=1403.5   p99=1991.6   max=3950.3   cas=0 err=0 cancelled=64

Avoiding creating new pushes that won't be finished before the end of the session:

  c=64   push_ok=13559  push/s=113.0    p50=433.7   p95=1405.9   p99=2003.3   max=3813.4   cas=0 err=0 cancelled=0

@pjbgf
pjbgf merged commit d0ed7c0 into main Oct 20, 2026
3 checks passed
@pjbgf
pjbgf deleted the drain-level-end branch October 20, 2026 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant