From 475448dab852c4ad5aa7836ffded1d0747d0b030 Mon Sep 17 00:00:00 2001 From: Ilia Sadykov Date: Sun, 8 Nov 2026 20:30:25 +0300 Subject: [PATCH] fix(plugin,ci): raise plugin-binary download + go test timeouts for large binaries MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two coupled changes with the same root cause: plugin binaries now carry every cloud SDK a plugin depends on, so a modern plugin like sc-plugin-gcp (which pulls in pulumi-gcp + pulumi-kubernetes + the google-api-go-client stack) builds to ~230MB per platform-arch. CI hits two concentric timeouts on a cold fetch: 1. `downloadFile`'s hard-coded 5-min `http.Client{Timeout: 5 * time.Minute}` (pkg/plugin/fetcher.go:205) — GHA hosted runners see ~1-2 MB/s steady- state to GitHub releases, which brushes up against 5 min on a 230MB binary. 2. `go test`'s default 10-min per-test-binary budget. The e2e package pkg/e2e/scopedstore spawns the pinned sc-plugin-gcp binary mid-test; the cold fetch eats several minutes of the 10 min just to arrive at the actual Test_Provision + TestScopeRegression_* logic, which then has no room to finish. Live-caught 2026-11-08 on api PR #568 (phase 5 of the sc-plugin-gcp extraction). Two consecutive reruns of TestScopeRegression_DamagedScopeFileFailsLintDeployAndReseal and Test_Provision/happy_path_gcp timed out mid-download with `context deadline exceeded (Client.Timeout or context cancellation while reading body)`. Bumping the fixture version pin to the latest release (v2026.11.08-d14f4da) didn't help — the binary size (and therefore the fetch time) is the same. ## What - `pkg/plugin/fetcher.go`: raise the `downloadFile` HTTP client timeout from 5 min to 15 min. Named const `downloadFileHTTPTimeout` with the incident citation inline, so a future reader can trace the choice. - `.github/workflows/{branch,branch-preview,push}.yaml`: raise the `go test ./...` step's budget from the default 10 min to `-timeout 30m`, matching the already-higher budget on the `test-registry-integration` job. Inline comment references this commit's reasoning. ## Why not pre-warm cache or stub the fetcher - **Pre-warm** (add a `curl` step before `go test`): workable, but adds a second code path that goes stale when the fixture pin bumps. The timeout bump handles every version pin automatically. - **Stub the fetcher in tests**: cleanest long-term, but requires refactoring TestScopeRegression + Test_Provision to use a path: spec with a built-at-test-time stub binary — larger scope, blocks #568's merge. Follow-up item worth filing. ## Test plan - [x] `go build ./...` - [x] `go test ./pkg/plugin/...` — fetcher.go changes do not affect existing fetcher tests (they use path: specs or mock servers; the 15-min client timeout applies only when real network I/O happens) - [ ] `go test -timeout 30m ./...` on #568 should now complete — pin bump already in place from the previous CI rerun, this PR removes the timeout wall Signed-off-by: Ilya Sadykov --- .github/workflows/branch-preview.yaml | 6 +++++- .github/workflows/branch.yaml | 10 +++++++++- .github/workflows/push.yaml | 6 +++++- pkg/plugin/fetcher.go | 18 +++++++++++++++++- 4 files changed, 36 insertions(+), 4 deletions(-) diff --git a/.github/workflows/branch-preview.yaml b/.github/workflows/branch-preview.yaml index f870dec9..f874cae6 100644 --- a/.github/workflows/branch-preview.yaml +++ b/.github/workflows/branch-preview.yaml @@ -324,7 +324,11 @@ jobs: cache: ${{ vars.FORGE_CI_RUNNER == '' }} - name: test run: | - go test ./... + # -timeout 30m — see the matching comment on branch.yaml's + # `test` step. Default 10m / test-binary leaves no room when + # pkg/e2e/scopedstore has to cold-fetch the ~230MB + # sc-plugin-gcp binary mid-test. + go test -timeout 30m ./... docker-build: name: Docker build and push preview ${{ matrix.image }} image diff --git a/.github/workflows/branch.yaml b/.github/workflows/branch.yaml index 98a2ca84..691d2d25 100644 --- a/.github/workflows/branch.yaml +++ b/.github/workflows/branch.yaml @@ -383,7 +383,15 @@ jobs: cache: ${{ vars.FORGE_CI_RUNNER == '' }} - name: test run: | - go test ./... + # -timeout 30m: default is 10m per test binary. The e2e package + # pkg/e2e/scopedstore spawns the pinned sc-plugin-gcp binary + # (230MB+) on cold-cache first-fetch, which eats several minutes + # of the 10m budget by itself and leaves Test_Provision no room + # to finish. Live-caught on PR #568 (2026-11-08): two consecutive + # reruns timed out mid-download, not in the test logic. 30m + # matches the already-bumped budget in test-registry-integration + # below. + go test -timeout 30m ./... test-registry-integration: name: Run integration tests diff --git a/.github/workflows/push.yaml b/.github/workflows/push.yaml index 22176714..415b12e1 100644 --- a/.github/workflows/push.yaml +++ b/.github/workflows/push.yaml @@ -489,7 +489,11 @@ jobs: cache: ${{ vars.FORGE_CI_RUNNER == '' }} - name: test run: | - go test ./... + # -timeout 30m — see the matching comment on branch.yaml's + # `test` step. Default 10m / test-binary leaves no room when + # pkg/e2e/scopedstore has to cold-fetch the ~230MB + # sc-plugin-gcp binary mid-test. + go test -timeout 30m ./... build-docs: name: Build documentation diff --git a/pkg/plugin/fetcher.go b/pkg/plugin/fetcher.go index 98723d15..87b7bc78 100644 --- a/pkg/plugin/fetcher.go +++ b/pkg/plugin/fetcher.go @@ -201,8 +201,24 @@ func fileSHA256(path string) (string, error) { return hex.EncodeToString(h.Sum(nil)), nil } +// downloadFileHTTPTimeout bounds a single plugin binary download. Plugin +// binaries carry every cloud SDK a plugin depends on and run 180-280MB per +// platform-arch (sc-plugin-gcp is the current ceiling). GHA hosted runners +// see ~1-2 MB/s steady-state to GitHub releases on a cold download, so even +// the ceiling fits comfortably in 15 min. The old 5 min cutoff pinned us to +// a window that the 230MB sc-plugin-gcp binary brushed up against — CI +// e2e tests that re-fetch on every run hit `context deadline exceeded` even +// though the stream was making progress. Live-caught 2026-11-08 on api PR +// #568 (phase 5 of sc-plugin-gcp extraction): two consecutive reruns of +// TestScopeRegression_DamagedScopeFileFailsLintDeployAndReseal + Test_Provision +// timed out inside this call. See +// forge/docs/roadmap/items/sc-plugin-stack-outputs-latency-full-schema-load.md +// for the companion latency work (cache-aware load for read-only entries; +// this fix is orthogonal — about the first-ever fetch, not repeat lookups). +const downloadFileHTTPTimeout = 15 * time.Minute + func downloadFile(url, dst string) error { - client := &http.Client{Timeout: 5 * time.Minute} + client := &http.Client{Timeout: downloadFileHTTPTimeout} resp, err := client.Get(url) //nolint:noctx // allow — tool-level call if err != nil { return err