Skip to content

Goal: Eliminate CUDA Toolkit dependency for end users #49

Closed
Closed
Goal: Eliminate CUDA Toolkit dependency for end users#49

Description

@m96-chan
m96-chan
opened on Dec 14, 2025

Overview

PyGPUkit's long-term goal is to allow end users to run GPU workloads without installing the CUDA Toolkit. Users should only need:

  • NVIDIA GPU driver (provides nvcuda.dll / libcuda.so)
  • PyGPUkit wheel (ships NVRTC DLL)

This issue tracks progress toward that goal.

CUDA Component Dependency Analysis

Component Build Time User Runtime Toolkit Required? Status
nvcc (CUDA Compiler) ✅ Required ❌ Not needed Build only ⚠️ Required for wheel build
CUDA Headers (cuda.h, etc.) ✅ Required ❌ Not needed Build only ⚠️ Required for wheel build
ptxas (PTX Assembler) ✅ Required ❌ Not needed Build only ⚠️ Required for wheel build
cudart (Runtime Library) ⚙️ Optional ⚙️ Optional Eliminable ✅ ELIMINATED (driver-only mode)
cuda_driver (nvcuda.dll) ✅ Required ✅ Required ❌ No ✅ OK - provided by GPU driver
NVRTC (nvrtc64_*.dll) ✅ Required ✅ Required ⚠️ Currently yes 🔄 Goal: Ship in wheel

Current State

✅ Achieved (User does NOT need Toolkit for these)

Item How
CUDA Driver API nvcuda.dll / libcuda.so comes with NVIDIA GPU driver
cudart elimination PYGPUKIT_DRIVER_ONLY=ON build mode
Driver-only build PR #48 fixed kernel.hpp compatibility

❌ Not Yet Achieved (User still needs Toolkit)

Item Blocker Solution
NVRTC at runtime nvrtc64_*.dll loaded from CUDA_PATH/bin Ship NVRTC DLL inside wheel
Windows DLL loading Needs os.add_dll_directory() for shipped DLLs Implement in __init__.py

⚠️ Build-Time Only (Acceptable)

Item Notes
nvcc Required for compiling .cu files - CI/wheel builders have it
CUDA headers Required for compilation - CI/wheel builders have it
ptxas Part of nvcc pipeline - CI/wheel builders have it

Implementation Plan

Phase 1: Ship NVRTC in Wheel

  • Bundle nvrtc64_120_0.dll (or appropriate version) in wheel
  • Add nvrtc64_builtins_*.dll (required companion)
  • Update pyproject.toml to include DLLs in package data
  • Implement os.add_dll_directory() in __init__.py for Windows

Phase 2: Verify Driver-Only Runtime

  • Test on clean Windows install (no CUDA Toolkit, only GPU driver)
  • Test on clean Linux install (no CUDA Toolkit, only GPU driver)
  • Document minimum GPU driver version requirements

Phase 3: Fallback Handling

  • Graceful error when NVRTC missing
  • Graceful error when GPU driver too old
  • Clear error messages guiding users to install GPU driver (NOT toolkit)

Target State

User installs: pip install pygpukit
User needs:    NVIDIA GPU driver only (no CUDA Toolkit)
Wheel ships:   _pygpukit_native.pyd + nvrtc64_*.dll + nvrtc64_builtins_*.dll

Related Issues

Labels

enhancement infrastructure v0.2.4

Activity

  1. m96-chan commented on Dec 14, 2025

    @m96-chan
    m96-chan
    on Dec 14, 2025
    OwnerAuthor

    For v0.2.4, the target is to eliminate dependency on the CUDA Toolkit from the user runtime.

  2. m96-chan commented on Dec 14, 2025

    @m96-chan
    m96-chan
    on Dec 14, 2025
    OwnerAuthor

    v0.2.4 Sub-Issues Created

    Phase Issue Description
    Phase 1 #50 Ship NVRTC DLLs in Wheel
    Phase 2 #51 Verify Driver-Only Runtime on Clean Systems
    Phase 3 #52 Graceful Error Handling for Missing GPU Components

    Dependency Order

    #50 (Ship NVRTC) → #51 (Verify Runtime) → #52 (Error Handling)

    Phase 1 must be completed before Phase 2 testing can begin. Phase 3 can be developed in parallel.

  3. m96-chan commented on Dec 14, 2025

    @m96-chan
    m96-chan
    on Dec 14, 2025
    OwnerAuthor

    PR #57 Progress Update

    ✅ Completed

    Task Status Details
    Driver-only mode ✅ Done PYGPUKIT_DRIVER_ONLY flag removed - now the only mode
    Dynamic NVRTC loading ✅ Done NVRTC loaded at runtime, not link-time
    Single-binary wheel ✅ Done Only nvcuda.dll (GPU driver) required
    Graceful fallback ✅ Done Issue #52 closed
    Error messages ✅ Done No "CUDA Toolkit" in user-facing errors

    Architecture Change

    Original plan was to ship NVRTC in wheel. New approach:

    • NVRTC is optional - dynamically loaded only when JIT needed
    • Pre-compiled ops work without NVRTC - matmul, add, mul, etc.
    • Smaller wheel size - no bundled DLLs

    Runtime Dependencies (Final)

    DLL Required Source
    nvcuda.dll ✅ Yes GPU drivers (user already has)
    cudart64_*.dll ❌ No Eliminated
    nvrtc64_*.dll ⚡ Optional CUDA Toolkit (for JIT only)

    Verified with dumpbin

    Image has the following dependencies:
    nvcuda.dll        ← Only CUDA dependency!
    python312.dll
    KERNEL32.dll
    ...

    Updated Status

    Phase 1: Ship NVRTC in Wheel → Changed to Dynamic NVRTC Loading

    • Dynamic NVRTC loader (nvrtc_loader.cpp)
    • Version-agnostic DLL discovery
    • is_nvrtc_available() / get_nvrtc_version() / get_nvrtc_path() API

    Phase 2: Driver-Only Runtime ✅

    • Test on Windows with GPU driver only
    • Benchmark: 17.7 TFLOPS FP32, 28.2 TFLOPS TF32

    Phase 3: Fallback Handling ✅ (Issue #52)

    • Graceful error when NVRTC missing
    • Clear error messages (no "CUDA Toolkit" mention)
    • CPU simulation fallback works

    Result

    User installs: pip install pygpukit
User needs:    NVIDIA GPU driver only
Optional:      CUDA Toolkit (for custom JIT kernels)

    Goal achieved! 🎉

  4. m96-chan commented on Dec 14, 2025

    @m96-chan
    m96-chan
    on Dec 14, 2025
    OwnerAuthor

    Goal Achieved! 🎉

    PR #57 completes the goal of eliminating CUDA Toolkit dependency for end users.

    Final State

    User installs: pip install pygpukit
User needs:    NVIDIA GPU driver only (no CUDA Toolkit)
Optional:      CUDA Toolkit (for custom JIT kernels only)

    Summary

    • Pre-compiled ops (matmul, add, mul, etc.) work with GPU drivers only
    • NVRTC is optional - dynamically loaded when JIT is needed
    • Single-binary wheel - only nvcuda.dll dependency
    • Graceful fallback - clear error messages, CPU simulation available

    Thank you for tracking this goal!

  5. m96-chan
    closed this as completedon Dec 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions