Skip to content

Allow toolkit installation without a loaded GPU driver - #144

Open
1fanwang wants to merge 1 commit into
ROCm:mainfrom
1fanwang:1fannnw/rpm-install-without-gpu
Open

1fanwang wants to merge 1 commit into
ROCm:mainfrom
1fanwang:1fannnw/rpm-install-without-gpu

Conversation

@1fanwang

Copy link
Copy Markdown

Motivation

Building a bootc or container image fails when the build host has no loaded amdgpu driver, even if the image will run on a GPU host. Both package formats stop with Error: AMD GPU driver (amdgpu) is not installed or loaded.

Image builds can now install the toolkit without a loaded driver. GPU access still requires the driver on the deployed host.

Closes #141.

Technical Details

Remove the install-time driver checks from RPM and Debian packages. Runtime driver checks and the existing best-effort GPU Tracker initialization remain unchanged. The package-install regression runs in CI for both distributions.

Test Plan

I built and installed the actual packages in GPU-free Linux/amd64 containers: AlmaLinux 9.8 with RPM 4.16.1.3, and Ubuntu 24.04 with dpkg 1.22.6. The binaries were built with Go 1.26.2 on macOS/arm64.

Scenario Command inside the container Before: main and v1.3.0 After
RPM installation and installed CLI bash /package-install.sh rpm Installation exits 1 Installation and configuration succeed
Debian installation and installed CLI bash /package-install.sh deb Installation exits 1 Installation and configuration succeed
GPU discovery without a driver amd-ctk gpu list Driver required Still fails with amdgpu driver unavailable

The commands below ran from this PR's checkout. They use the same regression script against upstream main, v1.3.0, and this commit.

set -euo pipefail
export CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOMAXPROCS=2

docker build --platform=linux/amd64 -t amd-ctk-package-test:rpm - <<'EOF'
FROM almalinux:9@sha256:3a3fa7f043b142bc8008c8b308d39b47d2c84008addcd52f9f9a7a82d2a90474
RUN dnf install -y rpm-build jq && dnf clean all
EOF
docker build --platform=linux/amd64 -t amd-ctk-package-test:deb - <<'EOF'
FROM ubuntu:24.04@sha256:008173c23f95b170204355c12626cb5a965d779a7e1283b09e9cffbb1bf33ca3
RUN apt-get update && apt-get install -y jq && apt-get clean
EOF

for ref in \
    4f04a84a7f7776e4fc5c5057299b3c01a403cdc1 \
    59f21d9fdeb4a9373607b90352dbda272c721979 \
    9499bce7b1c39c7c6f2c22295f11ea46db9eedde; do
    source_dir=$(mktemp -d)
    git archive "$ref" | tar -x -C "$source_dir"
    go build -C "$source_dir" -p 2 -ldflags="-X main.GitCommit=$ref" \
        -o bin/rpmbuild/amd-ctk ./cmd/amd-ctk
    go build -C "$source_dir" -p 2 -ldflags="-X main.GitCommit=$ref" \
        -o bin/rpmbuild/amd-container-runtime ./cmd/container-runtime
    for format in rpm deb; do
        printf '\n=== %s %s ===\n' "$ref" "$format"
        log=$(mktemp)
        status=0
        docker run --rm --network=none --platform=linux/amd64 --cpus=2 --memory=768m \
            -v "$source_dir:/src:ro" -v "$PWD/tests/package-install.sh:/package-install.sh:ro" \
            -w /src "amd-ctk-package-test:$format" \
            bash /package-install.sh "$format" >"$log" 2>&1 || status=$?
        cat "$log"
        printf 'exit_status=%s\n' "$status"
        if [ "$ref" = 9499bce7b1c39c7c6f2c22295f11ea46db9eedde ]; then
            test "$status" -eq 0
        else
            test "$status" -eq 1
            grep -F 'Error: AMD GPU driver (amdgpu) is not installed or loaded.' "$log"
        fi
    done
done
Raw logs

Upstream main, RPM:

Checking for AMD GPU driver...
Error: AMD GPU driver (amdgpu) is not installed or loaded.
The AMD Container Toolkit requires the amdgpu driver to be installed and loaded.
Please install the AMD GPU driver before installing the container toolkit.
error: %prein(amd-container-toolkit-1.3.0-1.el9.x86_64) scriptlet failed, exit status 1
error: amd-container-toolkit-1.3.0-1.el9.x86_64: install failed
exit_status=1

Debian on both upstream main and v1.3.0:

 new amd-container-toolkit package pre-installation script subprocess returned error exit status 1

RPM on v1.3.0:

error: %prein(amd-container-toolkit-1.3.0-1.el9.x86_64) scriptlet failed, exit status 1
error: amd-container-toolkit-1.3.0-1.el9.x86_64: install failed
exit_status=1

After, RPM:

Updating / installing...
amd-container-toolkit-1.3.0-1.el9     ########################################
failed to initialize GPU Tracker: getting AMD GPU info: amdgpu driver unavailable: stat /sys/module/amdgpu/drivers/: no such file or directory
+ rpm -q amd-container-toolkit
amd-container-toolkit-1.3.0-1.el9.x86_64
+ ctk=/usr/bin/amd-ctk
+ /usr/bin/amd-ctk version
Version: dev
Build Date: unknown
GitCommit: 9499bce7b1c39c7c6f2c22295f11ea46db9eedde

Docker runtime configuration after installation:

+ /usr/bin/amd-ctk runtime configure --config-path=/tmp/tmp.q5xNkff9Z7/daemon.json
Updated the config file: /tmp/tmp.q5xNkff9Z7/daemon.json
Please restart docker daemon
+ jq -e '.runtimes.amd.path == "amd-container-runtime"' /tmp/tmp.q5xNkff9Z7/daemon.json
true

After, Debian:

Unpacking amd-container-toolkit (0) ...
Setting up amd-container-toolkit (0) ...
failed to initialize GPU Tracker: getting AMD GPU info: amdgpu driver unavailable: stat /sys/module/amdgpu/drivers/: no such file or directory
+ dpkg-query -W amd-container-toolkit
amd-container-toolkit	0
+ ctk=/usr/local/bin/amd-ctk
+ /usr/local/bin/amd-ctk version
Version: dev
Build Date: unknown
GitCommit: 9499bce7b1c39c7c6f2c22295f11ea46db9eedde

Both fixed-package runs finish with the expected GPU-discovery error and a successful regression exit:

failed to list AMD devices: amdgpu driver unavailable: stat /sys/module/amdgpu/drivers/: no such file or directory
exit_status=0

Test Result

Package installation and Docker configuration succeed without a GPU. GPU discovery still requires the driver, and the existing post-install GPU Tracker warning remains non-fatal. This verifies image construction and configuration, not AMD GPU execution.

Submission Checklist

Signed-off-by: 1fanwang <1fannnw@gmail.com>
@shiv-tyagi

Copy link
Copy Markdown
Member

Thank you for the PR. I will try to take a look this week hopefully. Sorry for the delay and thanks for the patience.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Support installing amd-ctk RPM in container environments

2 participants