Skip to content

refactor(tests): reuse compilation caches and defer torch imports - #246

Merged
zhen8838 merged 7 commits into
tile-ai:mainfrom
zhen8838:refactor/ci-test-speedup
Oct 20, 2026
Merged

zhen8838 merged 7 commits into
tile-ai:mainfrom
zhen8838:refactor/ci-test-speedup

Conversation

@zhen8838

@zhen8838 zhen8838 commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Why

  • Repeated test runs recompile unchanged CUDA kernels and pay torch import costs in CLI subprocesses.

What

  • Reuse ccache and torch extension caches locally and in CI, with pull request cache isolation and per-job cache statistics.

  • Compare MMA layouts against host-compiled CuTe references in two tests, replacing handwritten references and thousands of parametrized cases. Use the selected CUDA toolkit's CCCL headers consistently on local and CI.

  • Import torch through lazy proxies and defer dtype resolution until use.

  • Configure local artifact storage and a PM2 cleanup task outside the repository.

  • Local full verification at 1da1c0c: source 2141 passed / 12 skipped (266s cold, 188s warm versus 335s baseline); installed 161 passed (315s versus 410s). Warm ccache: 118 direct hits, zero misses.

  • Header-path follow-up: tests/ir/cuda passed (2 tests); header tracing and a compile/link/run probe passed in the CI image.

  • PR CI passed: source 2141 passed / 12 skipped (533s step), installed 161 passed (330s step), and published-checkpoint decoding passed. Read-only cache: 120 misses, no shared cache result or torch extension file changes.

Contract

  • Public APIs and evaluation behavior remain unchanged. Cache configuration respects explicit environment overrides.

Risk

  • Fork PRs can change their workflow and bypass read-only cache settings; isolation prevents accidental writes within the existing shared-runner trust boundary.
  • CI timing improvements require confirmation on the second main run after merge.
  • The local PM2 task has no startup integration for machine reboots.
  • The CI image was rebuilt outside this repository; its previous image is retained as tilefoundry-ci-runner:pre-ccache.

Local compilation previously found /usr/include/cuda/std from the system
CUDA 11.5 apt package: libcu++ 1.6.0 with
_LIBCUDACXX_CUDA_API_VERSION=001006000. Those headers did not match the
CUTLASS and CUDA 13.3 CCCL versions, while CI had no system fallback and
failed to find cuda/std/utility.

Add the selected toolkit's include/cccl directory ahead of the system
include search so local and CI compilation use CUDA 13.3's shipped CCCL.
@zhen8838
zhen8838 merged commit 25fbbaa into tile-ai:main Oct 20, 2026
3 checks passed
@zhen8838
zhen8838 deleted the refactor/ci-test-speedup branch October 20, 2026 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant