Add basic Rocm CI build and test by scotts · Pull Request #1231 · pytorch/kineto

scotts · 2026-01-21T21:01:07Z

Depends on #1222.

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

facebook-github-bot · 2026-01-22T01:48:57Z

@scotts has updated the pull request. You must reimport the pull request before landing.

meta-codesync · 2026-01-22T01:52:10Z

@scotts has imported this pull request. If you are a Meta employee, you can view this in D91166410.

facebook-github-bot · 2026-01-23T15:18:40Z

@scotts has updated the pull request. You must reimport the pull request before landing.

jithunnair-amd · 2026-01-26T20:27:01Z

+
+jobs:
+  pr-test:
+    runs-on: linux.rocm.gpu.gfx942


Should be linux.rocm.gpu.gfx942.1 if you want a single-GPU runner

@jithunnair-amd, will that enable the job to run? It hasn't been successfully scheduled yet.

facebook-github-bot · 2026-01-27T17:01:32Z

@scotts has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2026-01-27T17:16:44Z

@scotts has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2026-01-27T17:24:56Z

@scotts has updated the pull request. You must reimport the pull request before landing.

scotts · 2026-01-27T19:54:34Z

@jithunnair-amd, my workflow is now running, but I haven't been able to successfully pull a ROCm image. Can you advise me on what image to pull and through what mechanisms?

jithunnair-amd · 2026-01-28T19:05:22Z

+
+# This is the docker image that pytorch/pytorch uses for ROCM CI.
+env:
+  DOCKER_IMAGE: "ghcr.io/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3"


@scotts If we want to use the same image that's used by PyTorch CI for ROCm, there's a sha suffix that is also part of the tag e.g. pytorch-linux-noble-rocm-n-py3-f2bf68fc1a48c33b01facc4b2e0227aa3da827e0. However, this is ever-changing, and is not maintainable. Alternatives:

Work with @atalman to update the publishing workflow, so that every time it pushes out a new image for pytorch-linux-noble-rocm-n-py3-* tag, it also publishes it as pytorch-linux-noble-rocm-n-py3-latest tag or something.

Use a different image such as pytorch/manylinux2_28-builder:rocm7.2, which will still be a maintenance burden to update the ROCm versions, but with less frequency than the SHA updates in the CI image. That part can again be addressed by automatically publishing the same image with a tag pytorch/manylinux2_28-builder:rocm-latest. Also, this image usually updates ROCm version before the CI image, so you might have potential days of mismatch in the ROCm version between pytorch CI and kineto CI, if that matters.

@jithunnair-amd, thanks for taking a look!

Since my short-term goal is to just get some tests up and running to check regressions, I'm happy doing the easiest thing for now. I tried pulling pytorch/manylinux2_28-builder:rocm7.2, but I'm getting these failures: https://github.com/pytorch/kineto/actions/runs/21452690799/job/61785302020?pr=1231

It looks like I don't have permissions. Is this because this is running in pytorch/kineto and not pytorch/pytorch? There's also some aws: command not found errors, but these same patterns work fine for the CPU and CUDA workflows. Any help is greatly appreciated!

It's because it's assuming the image is on ghcr.io, whereas it's actually on dockerhub...

The "aws: command not found" errors can be ignored, but you'll probably need an AWS login via OIDC step to access any S3 buckets as part of the workflow e.g. https://github.com/pytorch/pytorch/blob/2b610902b46d0cbbb64923544f37feec0139cdfa/.github/actions/setup-rocm/action.yml#L115

You'll also need to add a permissions section to your workflow with id-token: write for the OIDC step to work successfully e.g.
https://github.com/pytorch/pytorch/blob/2b610902b46d0cbbb64923544f37feec0139cdfa/.github/workflows/_rocm-test.yml#L73

I see you're already calling setup-rocm in your workflow, so I think it'll try to do the AWS OIDC login step as well. But you'll probably still need the permissions clause.

facebook-github-bot · 2026-01-28T19:34:15Z

@scotts has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2026-01-28T21:17:21Z

@scotts has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2026-01-29T20:42:51Z

@scotts has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2026-01-29T21:25:08Z

@scotts has updated the pull request. You must reimport the pull request before landing.

jithunnair-amd · 2026-01-29T21:58:15Z

 on:
  push:
+    tags:
+      - ciflow/rocm/*


@scotts A few more steps to get this system working:

Create a ciflow/rocm label on kineto repo (to be used on PRs)

Add pytorch-probot.yaml to kineto repo like https://github.com/meta-pytorch/monarch/blob/main/.github/pytorch-probot.yml

I can't add labels on Kineto PRs. Please try adding the ciflow/rocm label and we can see if it works as expected

facebook-github-bot · 2026-01-30T01:07:03Z

@scotts has updated the pull request. You must reimport the pull request before landing.

atalman · 2026-01-30T14:49:15Z

+      - ciflow/rocm/*
+    branches:
+      - main
+  pull_request:


Suggested change

pull_request:

Lets remove pull_request and try to land this

facebook-github-bot · 2026-01-30T14:50:26Z

@scotts has updated the pull request. You must reimport the pull request before landing.

meta-codesync · 2026-01-30T15:41:58Z

@scotts merged this pull request in 4709380.

Summary: Depends on pytorch/kineto#1222. Pull Request resolved: pytorch/kineto#1231 Reviewed By: atalman Differential Revision: D91166410 Pulled By: scotts fbshipit-source-id: e237b9c22815d9725bf4aa548085f73098939658 Signed-off-by: Antoni Viros i Martin <aviros@ibm.com>

scotts added 20 commits January 20, 2026 06:48

Add Linux CPU only CI build and tests

a2f0ffa

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

Fix CI names

ea7b04a

Use linux.c7i.2xlarge as CPU runner

20e3737

Force CPU-only through cmake flags

c50f083

Explicitly disable building CUDA support in PyTorch

67d542a

OFF instead of 0?

8623b60

USE_CUDA=0

2e2582c

Separate build and test

658e88b

Use bigger runner

c10619e

Actually change to linux.c7i.4xlarge

281f88c

Limit max jobs; try non-docker version

fbaef35

Different workflow names

b4983ea

Fix stray docker command

c4437d0

Use pushd/popd

d35c493

Improved non-docker CPU only tests

9b4a76c

Remove non-docker attempt; deselect failing test

26ba8b5

Make it a proper comment

6e722c1

Block comment

ae04c7e

Defensively disable all known GPUs

0461765

Add basic ROCM build and test

671811b

meta-cla Bot added the cla signed label Jan 21, 2026

facebook-github-bot added the module: rocm label Jan 21, 2026

scotts added 2 commits January 21, 2026 13:04

Rename the CI job

ceb7d04

Merge branch 'main' of https://github.com/pytorch/kineto into ci_rocm

d744e13

Better ROCm settings

ccd4df6

jithunnair-amd reviewed Jan 26, 2026

View reviewed changes

Used single-GPU runner

ecc95cd

Different ROCm image

296655e

Try calculating image first

c13392a

jithunnair-amd reviewed Jan 28, 2026

View reviewed changes

scotts added 2 commits January 28, 2026 11:33

Update to pytorch/manylinux2_28-builder:rocm7.2 image

3688e82

Don't calculate

d8c6b23

Permissions and url

027c2fd

Add tags

d08e295

Try on pull requests

320e180

jithunnair-amd reviewed Jan 29, 2026

View reviewed changes

Add probot

002eb14

atalman added the ciflow/rocm label Jan 30, 2026

atalman reviewed Jan 30, 2026

View reviewed changes

Remove pull request

9273b21

scotts marked this pull request as ready for review January 30, 2026 14:50

meta-codesync Bot closed this in 4709380 Jan 30, 2026

facebook-github-bot added the Merged label Jan 30, 2026

scotts deleted the ci_rocm branch April 4, 2026 01:09

Conversation

scotts commented Jan 21, 2026

Uh oh!

facebook-github-bot commented Jan 22, 2026

Uh oh!

meta-codesync Bot commented Jan 22, 2026

Uh oh!

facebook-github-bot commented Jan 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jan 27, 2026

Uh oh!

facebook-github-bot commented Jan 27, 2026

Uh oh!

facebook-github-bot commented Jan 27, 2026

Uh oh!

scotts commented Jan 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jithunnair-amd Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jan 28, 2026

Uh oh!

facebook-github-bot commented Jan 28, 2026

Uh oh!

facebook-github-bot commented Jan 29, 2026

Uh oh!

facebook-github-bot commented Jan 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jan 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jan 30, 2026

Uh oh!

meta-codesync Bot commented Jan 30, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jithunnair-amd Jan 28, 2026 •

edited

Loading