This repository was archived by the owner on Sep 30, 2026. It is now read-only.
Repository navigation
Add basic Rocm CI build and test #1231
Closed
Closed
Changes from all commits
Commits
Show all changes
35 commits
Select commit
Hold shift + click to select a range
a2f0ffa
Add Linux CPU only CI build and tests
scotts ea7b04a
Fix CI names
scotts 20e3737
Use linux.c7i.2xlarge as CPU runner
scotts c50f083
Force CPU-only through cmake flags
scotts 67d542a
Explicitly disable building CUDA support in PyTorch
scotts 8623b60
OFF instead of 0?
scotts 2e2582c
USE_CUDA=0
scotts 658e88b
Separate build and test
scotts c10619e
Use bigger runner
scotts 281f88c
Actually change to linux.c7i.4xlarge
scotts fbaef35
Limit max jobs; try non-docker version
scotts b4983ea
Different workflow names
scotts c4437d0
Fix stray docker command
scotts d35c493
Use pushd/popd
scotts 9b4a76c
Improved non-docker CPU only tests
scotts 26ba8b5
Remove non-docker attempt; deselect failing test
scotts 6e722c1
Make it a proper comment
scotts ae04c7e
Block comment
scotts 0461765
Defensively disable all known GPUs
scotts 671811b
Add basic ROCM build and test
scotts ceb7d04
Rename the CI job
scotts d744e13
Merge branch 'main' of https://github.com/pytorch/kineto into ci_rocm
scotts ccd4df6
Better ROCm settings
scotts ecc95cd
Used single-GPU runner
scotts 6c39770
Try new ROCm image
scotts a9daffd
Retry nightly CI image
scotts 296655e
Different ROCm image
scotts c13392a
Try calculating image first
scotts 3688e82
Update to pytorch/manylinux2_28-builder:rocm7.2 image
scotts d8c6b23
Don't calculate
scotts 027c2fd
Permissions and url
scotts d08e295
Add tags
scotts 320e180
Try on pull requests
scotts 002eb14
Add probot
scotts 9273b21
Remove pull request
scotts File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| ciflow_push_tags: | ||
| - ciflow/rocm |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,140 @@ | ||
| name: Linux ROCm build and test | ||
|
|
||
| on: | ||
| push: | ||
| tags: | ||
| - ciflow/rocm/* | ||
| branches: | ||
| - main | ||
|
|
||
| env: | ||
| DOCKER_IMAGE: "pytorch/manylinux2_28-builder:rocm7.2" | ||
|
|
||
| permissions: | ||
| id-token: write | ||
| contents: read | ||
|
|
||
| jobs: | ||
| pr-test: | ||
| runs-on: linux.rocm.gpu.gfx942.1 | ||
| timeout-minutes: 180 # 3 hours | ||
| steps: | ||
| - name: Checkout Kineto | ||
| uses: actions/checkout@v5 | ||
| with: | ||
| path: kineto | ||
| submodules: recursive | ||
|
|
||
| - name: Pull docker image | ||
| uses: pytorch/test-infra/.github/actions/pull-docker-image@main | ||
| with: | ||
| docker-image: ${{ env.DOCKER_IMAGE }} | ||
|
|
||
| - name: Setup ROCM | ||
| uses: pytorch/pytorch/.github/actions/setup-rocm@main | ||
|
|
||
| - name: Get env vars | ||
| run: | | ||
| echo GITHUB_WORKFLOW = $GITHUB_WORKFLOW | ||
| echo HOME = $HOME | ||
| echo GITHUB_ACTION = $GITHUB_ACTION | ||
| echo GITHUB_ACTIONS = $GITHUB_ACTIONS | ||
| echo GITHUB_REPOSITORY = $GITHUB_REPOSITORY | ||
| echo GITHUB_EVENT_NAME = $GITHUB_EVENT_NAME | ||
| echo GITHUB_EVENT_PATH = $GITHUB_EVENT_PATH | ||
| echo GITHUB_WORKSPACE = $GITHUB_WORKSPACE | ||
| echo GITHUB_SHA = $GITHUB_SHA | ||
| echo GITHUB_REF = $GITHUB_REF | ||
|
|
||
| - name: Build libkineto (static and shared library) | ||
| run: | | ||
| container_name=$(docker run \ | ||
| ${GPU_FLAG:-} \ | ||
| --tty \ | ||
| --detach \ | ||
| --shm-size=32gb \ | ||
| -v "${PWD}/kineto:/kineto" \ | ||
| -w / \ | ||
| "${{ env.DOCKER_IMAGE }}" | ||
| ) | ||
| echo "Container name: ${container_name}" | ||
|
|
||
| docker exec -t -w "/kineto" "${container_name}" bash -c " | ||
| set -eux | ||
| mkdir build_static | ||
| mkdir build_shared | ||
| pip install cmake | ||
| " | ||
|
|
||
| docker exec -t -w "/kineto/build_static" "${container_name}" bash -c " | ||
| set -eux | ||
| cmake -DKINETO_LIBRARY_TYPE=static ../libkineto/ | ||
| make -j | ||
| " | ||
|
|
||
| docker exec -t -w "/kineto/build_shared" "${container_name}" bash -c " | ||
| set -eux | ||
| cmake -DKINETO_LIBRARY_TYPE=shared ../libkineto/ | ||
| make -j | ||
| " | ||
|
|
||
| - name: Run libkineto tests | ||
| run: | | ||
| container_name=$(docker ps -lq) | ||
| docker exec -t -w "/kineto/build_static" "${container_name}" bash -c "make test" | ||
|
|
||
| - name: Clone PyTorch | ||
| run: | | ||
| container_name=$(docker ps -lq) | ||
| docker exec -t -w "/" "${container_name}" bash -c " | ||
| set -eux | ||
| git clone --recursive --branch viable/strict https://github.com/pytorch/pytorch.git | ||
| " | ||
|
|
||
| - name: Replace PyTorch's Kineto with PR version | ||
| run: | | ||
| container_name=$(docker ps -lq) | ||
| docker exec -t -w "/pytorch" "${container_name}" bash -c " | ||
| set -eux | ||
| rm -rf third_party/kineto | ||
| ln -s /kineto third_party/kineto | ||
| " | ||
|
|
||
| - name: Build PyTorch from source | ||
| run: | | ||
| container_name=$(docker ps -lq) | ||
| docker exec -t -w "/pytorch" "${container_name}" bash -c " | ||
| set -eux | ||
| pip install -r requirements.txt | ||
| export USE_ROCM=1 | ||
| export BUILD_TEST=1 | ||
| python setup.py develop | ||
| " | ||
|
|
||
| - name: Run PyTorch profiler tests | ||
| run: | | ||
| container_name=$(docker ps -lq) | ||
| docker exec -t -w "/pytorch" "${container_name}" bash -c " | ||
| set -eux | ||
| # TODO: Dynamically add/remove tests to the exclusion list based on their | ||
| # status on trunk instead of maintaining a hardcoded list of known failures. | ||
| # This will prevent the list from becoming stale as tests get fixed upstream. | ||
| python -m pytest test/profiler/ -v \ | ||
| --deselect=test/profiler/test_memory_profiler.py::TestDataFlow::test_data_flow_graph_complicated \ | ||
| --deselect=test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_categories_e2e_sequential_fwd_bwd \ | ||
| --deselect=test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_categories_e2e_simple_fwd_bwd \ | ||
| --deselect=test/profiler/test_memory_profiler.py::TestMemoryProfilerE2E::test_categories_e2e_simple_fwd_bwd_step \ | ||
| --deselect=test/profiler/test_profiler.py::TestProfiler::test_kineto \ | ||
| --deselect=test/profiler/test_profiler.py::TestProfiler::test_user_annotation \ | ||
| --deselect=test/profiler/test_profiler.py::TestExperimentalUtils::test_fuzz_symbolize \ | ||
| --deselect=test/profiler/test_profiler.py::TestExperimentalUtils::test_profiler_debug_autotuner \ | ||
| --deselect=test/profiler/test_torch_tidy.py::TestTorchTidyProfiler::test_tensorimpl_invalidation_scalar_args | ||
| " | ||
|
|
||
| - name: Teardown Linux | ||
| uses: pytorch/test-infra/.github/actions/teardown-linux@main | ||
| if: always() | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}-${{ github.event_name == 'workflow_dispatch' }} | ||
| cancel-in-progress: true | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@scotts A few more steps to get this system working:
ciflow/rocmlabel on kineto repo (to be used on PRs)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can't add labels on Kineto PRs. Please try adding the
ciflow/rocmlabel and we can see if it works as expected