My Local Inference Stack

• One folder for everything
• Self-hosted
• Works on my machines ™️. Main targets are:
  • GNU/Linux (Arch/Debian/Ubuntu/CachyOS)
  • If GPU acceleration then via ROCm/Vulkan. No NVIDIA CUDA!
• Main tools:
  • llama.cpp
  • audio.cpp
  • ComfyUI

get models

Models are to be saved, mostly in gguf, in the models, audiomodels and comfymodels subfolders.

Current list of files. Made with:

echo -e "# models directory structure\n\n\`\`\`text\n$(tree -h --du --dirsfirst -F models audiomodels comfymodels)\n\`\`\`" > modelsDirTree.md

Configuration and parameters for llama.cpp router mode are in models.ini.

run

Most containers configured in compose.yaml.

• llama.cpp
  • ROCm, llamacpp-r http://localhost:8080
  • Vulkan, llamacpp-v http://localhost:8083
  • CPU, llamacpp-c http://localhost:8082
• llama.cpp - linked to ministack of models minimodels.ini
  • Vulkan, llamacpp-t-v http://localhost:8093
  • CPU, llamacpp-t-c http://localhost:8092
• audio.cpp at http://localhost:8081
• Open WebUI at http://localhost:8070

Run with:

docker compose up -d

Logs with:

docker logs llamacpp-r

and so on.

ComfyUI

Subfolder comfyui-rocm-docker.

Build image with:

docker compose build

Run with:

docker compose up -d

Available at http://localhost:8182