Skip to content

CI: the adapter package's -race run sits at the 20-minute timeout on hosted runners #254

Open

Description

@kikashy

Problem

The go-race step in CI's check job runs ./context/... and ./adapter/... together. Under -race, the ./adapter/ package now runs close to its 20-minute test timeout on hosted runners, so unrelated PRs fail by variance:

Where ./adapter/ under -race Whole go-race step
Local, main 9c53071 (alone) 700 s n/a
Local, #251 head 67e25ed (alone) 692 s n/a
CI, main after #250 passed 955 s
CI, #247 at 920c75c timed out at 20 m 1273 s
CI, #251 at 67e25ed timed out at 20 m 1276 s

#247 lightened its own adapter test (916 s → 746 s locally), and #251 changes no adapter file, yet both hit the timeout. The package's largest test, TestActualAdaptersConsumePreparedPlanBeforePhysicalBootstrap, alone takes about 170 s under -race.

Options

  • Run ./adapter/... under -race in its own CI step or job, so it does not share a runner's time and CPU with ./context/....
  • Split the heaviest adapter tests across packages (each Go test binary gets its own timeout), or shard them by -run.
  • Profile the largest tests for setup that can be shared.

Raising the timeout or skipping tests under -race hides the problem, so neither is an option.

Activity

  1. added this to the v0.13.0 milestone on Oct 29, 2026
  2. kikashy commented on Oct 29, 2026

    @kikashy
    ContributorAuthor

    Measured on #251 (2026-10-29). The cause is runner speed, not test growth.

    Locally, same toolchain and command: Go 1.26.10 and the exact go-race command (go test -count=1 -race -timeout=20m ./context/... ./adapter/...) give main 9c53071 779 s and #251's head 801 s. Per package, main/head: context 341/354 s, adapter 751/747 s, delegate 437/442 s, eino 522/522 s.

    In CI, by runner speed (seconds):

    Step Passing runs Failing runs
    python-tests 49–54 104–106
    unit-tests 282 459–470
    go-test 127–143 223–225

    Every step slows by about 2x on the slower runners, including steps no Go change can affect. Main itself needed 1262 s for go-race on such a runner, and the adapter package crosses its 20 m limit there regardless of the PR. The fix should give ./adapter/... headroom on a 2x-slower runner: its own step or job, or a split of its heaviest tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions