The curated bibliography for the whole curriculum. Every non-obvious claim in a module should trace to an entry here or to a module-local citation in that module's section 8.
Every source below is a ### heading, which gives it a stable anchor on both GitHub and the
published site. Cite a specific source rather than the bibliography as a whole:
[Roofline (Williams et al., 2009)](../../../docs/REFERENCES.md#roofline-williams-et-al-2009)check_links.py resolves every one of those anchors, so a reworded heading fails CI rather than
rotting silently. Depth of ../ depends on where you are: three levels up from a module README,
four from an exercises/ or solutions/ page.
Following the repository's cite or hedge rule, the two kinds of source are dated differently:
- Papers are identified by DOI or arXiv ID, which are permanent. A retrieval date on
arxiv.org/abs/1706.03762would be decoration — the identifier already pins the artifact. - Vendor documentation, spec pages and tool guides are version-sensitive and move, so each carries a retrieval date. If a claim depends on one of these, the date tells you how stale your evidence is. Links were verified on the dates shown; search the title if one 404s.
Some publishers (ACM, MIT Press, several corporate blogs) return 403 to automated requests. Those
links are correct in a browser; the link checker's --external mode reports them and that is
expected, not a defect.
Authoritative for API behaviour and GPU correctness. All retrieved 8 Aug 2026.
The normative description of the CUDA execution and memory model. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html
Coalescing, shared memory, bank conflicts, alignment, occupancy guidance. https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html
AMD's portable C++ GPU API; the CUDA-to-HIP mapping lives here. https://rocm.docs.amd.com/projects/HIP/en/latest/
The execution model in isolation — wavefronts, workgroups, the memory hierarchy. https://rocm.docs.amd.com/projects/HIP/en/latest/understand/programming_model.html
Entry point for every ROCm component's documentation. https://rocm.docs.amd.com/
The authoritative per-SKU table: compute units, wavefront size, LDS, cache sizes, VRAM. Cited whenever the curriculum states an AMD hardware number. https://rocm.docs.amd.com/en/latest/reference/gpu-arch-specs.html
Language reference and the tutorial series the Triton labs parallel. https://triton-lang.org/
Williams, S., Waterman, A., Patterson, D. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM 52(4), 65–76. DOI 10.1145/1498765.1498785.
Little, J. D. C. A Proof for the Queuing Formula L = λW. Operations Research 9(3), 383–387. The concurrency-latency-throughput identity behind every occupancy and batching argument. DOI 10.1287/opre.9.3.383.
Retrieved 8 Aug 2026. https://resources.nvidia.com/en-us-tensor-core
Landing page for the MI200/MI300 whitepapers. Retrieved 8 Aug 2026. https://www.amd.com/en/technologies/cdna.html
Per-CU matrix throughput, XCD layout, Infinity Cache. Note: this document reports Infinity Cache bandwidth as 17.2 TB/s while AMD's own Hot Chips 2024 deck reports 14.7 TB/s. The curriculum reports both rather than picking one; see B01 §3.3. Retrieved 8 Aug 2026. https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/white-papers/amd-cdna-3-white-paper.pdf
Infinity Fabric per-pair and aggregate bandwidth, node and rack topology. Retrieved
8 Aug 2026 — AMD retired the standalone instinct-mi300-series-cluster-reference-guide.pdf
this page replaces.
https://instinct.docs.amd.com/projects/cluster-documentation/latest/reference/cluster-design.html
The seven-step reduction ladder that A04 rebuilds. Retrieved 8 Aug 2026. https://developer.download.nvidia.com/assets/cuda/files/reduction.pdf
The coalescing model used throughout Track A. Retrieved 8 Aug 2026. https://developer.nvidia.com/blog/how-access-global-memory-efficiently-cuda-c-kernels/
GPU Gems 3, chapter 39 — work-efficient scan. Retrieved 8 Aug 2026. https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-computing/chapter-39-parallel-prefix-sum-scan-cuda
The naive/tiled/padded progression that A02's Lab 2 reproduces. Retrieved 8 Aug 2026. https://developer.download.nvidia.com/CUDA/training/StreamsAndConcurrencyWebinar.pdf
Template library for high-performance GEMM and tensor layouts; the subject of A10. Retrieved 8 Aug 2026. https://github.com/NVIDIA/cutlass
Per-instruction MFMA shapes, cycle counts and register layouts for CDNA. Retrieved 8 Aug 2026. https://github.com/ROCm/amd_matrix_instruction_calculator
The transformer architecture. arXiv 1706.03762.
Fast and Memory-Efficient Exact Attention with IO-Awareness — tiling and recomputation that make attention memory-bound rather than quadratic in HBM traffic. arXiv 2205.14135.
Better work partitioning across warps and thread blocks. arXiv 2307.08691.
The single-pass streaming softmax that A07 and A09 depend on. arXiv 1805.02867.
Efficient Memory Management for LLM Serving with PagedAttention — KV cache paging. arXiv 2309.06180.
Fast Inference from Transformers via Speculative Decoding. arXiv 2211.17192.
The partitioning and cost model behind prefill/decode reasoning. arXiv 2211.05102.
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — the KV-cache reduction B05 and B06 assume. arXiv 2305.13245.
Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI '22 — iteration-level scheduling. https://www.usenix.org/conference/osdi22/presentation/yu
arXiv 2001.08361.
The Chinchilla result. arXiv 2203.15556.
Cited for concrete production training and serving configurations. arXiv 2407.21783.
Reference implementation for paged KV cache and continuous batching. Retrieved 8 Aug 2026. https://github.com/vllm-project/vllm
Serving runtime with RadixAttention prefix caching. Retrieved 8 Aug 2026. https://github.com/sgl-project/sglang
Training Multi-Billion Parameter Language Models Using Model Parallelism — tensor parallelism. arXiv 1909.08053.
Memory Optimizations Toward Training Trillion Parameter Models — optimizer, gradient and parameter sharding. arXiv 1910.02054.
Efficient Training of Giant Neural Networks using Pipeline Parallelism. arXiv 1811.06965.
The production implementation of parameter sharding. Retrieved 8 Aug 2026. https://docs.pytorch.org/docs/stable/fsdp.html
ROCm Collective Communication Library. Retrieved 8 Aug 2026. https://github.com/ROCm/rccl
NVIDIA Collective Communication Library. Retrieved 8 Aug 2026. https://developer.nvidia.com/nccl
The sources behind Track D. Where B07 asks how to split a model, these ask what the wire costs and where it stops scaling.
Validity of the single processor approach to achieving large scale computing capabilities.
AFIPS '67 (Spring), pp. 483–485. The origin of the fixed-problem speedup ceiling: if a fraction s
of the work is serial, no processor count beats 1/s.
DOI 10.1145/1465482.1465560.
Communications of the ACM 31(5). The rebuttal that matters in practice: real workloads grow the problem with the machine, so the serial fraction shrinks rather than staying fixed. Weak scaling. DOI 10.1145/42411.42415.
The communication challenge for MPP: Intel Paragon and Meiko CS-2. Parallel Computing
20(3):389–398. The linear message-cost model T(m) = alpha + m*beta that every collective cost
derivation in this track builds on, and the source of the latency/bandwidth crossover at
m* = alpha/beta.
DOI 10.1016/S0167-8191(06)80021-9.
LogP: towards a realistic model of parallel computation. PPoPP '93. Adds per-message overhead and the injection gap to the alpha-beta picture — the reason a node cannot simply issue messages back-to-back at link rate. DOI 10.1145/155332.155333.
Optimization of Collective Communication Operations in MPICH. International Journal of High Performance Computing Applications 19(1):49–66. The canonical account of why a collective library switches algorithm by message size and process count — recursive doubling for short messages, reduce-scatter plus all-gather for long ones. DOI 10.1177/1094342005051521.
Bandwidth optimal all-reduce algorithms for clusters of workstations. Journal of Parallel and
Distributed Computing 69(2):117–124. Derives the 2(N-1)/N lower bound on data movement and gives
the ring algorithm that meets it. This is the paper behind "ring all-reduce is bandwidth-optimal".
DOI 10.1016/j.jpdc.2008.09.002.
Horovod: fast and easy distributed deep learning in TensorFlow. The work that brought ring all-reduce from HPC into mainstream deep learning. arXiv 1802.05799.
Fat-trees: universal networks for hardware-efficient supercomputing. IEEE Transactions on Computers 34(10):892–901. The topology nearly every AI cluster still uses, and the proof that bandwidth can be scaled independently of processor count. DOI 10.1109/TC.1985.6312192.
Technology-Driven, Highly-Scalable Dragonfly Topology. ISCA '08, pp. 77–88. High-radix groups as virtual routers, so a minimally routed packet crosses at most one global channel. The topology behind Slingshot and several exascale systems. DOI 10.1109/ISCA.2008.19.
Tofu: A 6D Mesh/Torus Interconnect for Exascale Computers. Computer 42(11):36–40. The fault-tolerant torus used by the K computer and, in its later TofuD form, by Fugaku. DOI 10.1109/MC.2009.370.
An In-Depth Analysis of the Slingshot Interconnect. SC '20. An independent measurement study of the fabric used by Frontier, El Capitan, Aurora and LUMI, including congestion behaviour. arXiv 2008.08886.
The specification itself — the normative source for communicator semantics, collective operations and their completion guarantees. Retrieved 11 Aug 2026. https://www.mpi-forum.org/docs/
The open-source MPI implementation used in this track's labs. Retrieved 11 Aug 2026. https://www.open-mpi.org/
The standard latency and bandwidth micro-benchmarks for MPI, including GPU-aware variants. Retrieved 11 Aug 2026. https://mvapich.cse.ohio-state.edu/benchmarks/
The reference correctness and bus-bandwidth harnesses for GPU collectives. Whatever a module claims about collective throughput should be reproducible with these. Retrieved 11 Aug 2026. https://github.com/NVIDIA/nccl-tests · https://github.com/ROCm/rccl-tests
The zero-copy path that lets a NIC read and write GPU memory without staging through the host. Retrieved 11 Aug 2026. https://docs.nvidia.com/cuda/gpudirect-rdma/
Scheduler documentation, including the topology-aware plugins that decide rank placement. Retrieved 11 Aug 2026. https://slurm.schedmd.com/documentation.html
The system-specification and energy-efficiency rankings used for every reference system in D06. Specs and rankings change each list; cite the list edition. Retrieved 11 Aug 2026. https://top500.org · https://green500.org
The award record for real-application performance at scale — the source for the case studies in
D06. Retrieved 11 Aug 2026 (ACM answers 403 to automated requests; the link is correct in a
browser).
https://awards.acm.org/bell
Loss scaling and the fp32 master-weight pattern. arXiv 1710.03740.
E4M3 and E5M2. Note that CDNA 3 implements the FNUZ interpretation and Hopper the OCP one, so the same bits decode differently; see B04. arXiv 2209.05433.
The vendor-neutral fp8 specification Hopper follows. Retrieved 8 Aug 2026 (the Open Compute
site returns 403 to automated requests; the link is correct in a browser).
https://www.opencompute.org/documents/ocp-8-bit-floating-point-specification-ofp8-revision-1-0-2023-12-01-pdf-1
8-bit Matrix Multiplication for Transformers at Scale — outlier-aware int8. arXiv 2208.07339.
Accurate Post-Training Quantization for GPTs. arXiv 2210.17323.
Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv 2306.00978.
arXiv 1503.02531.
The source for every numerical-stability claim in the curriculum. DOI 10.1145/103162.103163 · https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
Hidden Technical Debt in Machine Learning Systems (Sculley et al., 2015)
NeurIPS. The reference for why the model is the small part of the system. https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
O'Reilly. Retrieved 8 Aug 2026. https://www.oreilly.com/library/view/designing-machine-learning/9781098107956/
Retrieved 8 Aug 2026. https://developers.google.com/machine-learning/guides/rules-of-ml
A Rubric for ML Production Readiness and Technical Debt Reduction. https://research.google/pubs/pub46555/
ICSE. The nine-stage ML workflow Track C is organized around. https://www.microsoft.com/en-us/research/publication/software-engineering-for-machine-learning-a-case-study/
Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A., Lawrence, N. (eds). MIT Press. The
taxonomy behind C06's drift vocabulary. Retrieved 8 Aug 2026 (MIT Press
returns 403 to automated requests; the link is correct in a browser).
https://mitpress.mit.edu/9780262170055/dataset-shift-in-machine-learning/
All retrieved 8 Aug 2026.
Note: rocprofv3 defaults to rocpd SQLite output, not CSV; the curriculum's profile-hip
targets pass --output-format csv explicitly.
https://rocm.docs.amd.com/projects/rocprofiler-sdk/en/latest/
Timeline profiler — where the time goes across the whole program. https://docs.nvidia.com/nsight-systems/
Kernel profiler — memory charts, occupancy, bank conflicts. https://docs.nvidia.com/nsight-compute/
The metric definitions B03 relies on. https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html
Formerly Omniperf and Omnitrace; renamed to rocprof-compute and rocprof-sys-* in ROCm 6.3.
https://rocm.docs.amd.com/projects/omniperf/en/latest/
CDNA 3 vs CDNA 4 comparison, peak theoretical tables, partitioning modes. Replaces the retired
rocmdocs.amd.com page of the same name.
https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/optimization/workload-optimization.html
All retrieved 8 Aug 2026.
https://google.github.io/styleguide/cppguide.html
The severity vocabulary used in this repository's review notes. https://conventionalcomments.org/
A Handbook of Agile Software Craftsmanship. https://www.oreilly.com/library/view/clean-code-a/9780136083238/
The table below is generated by build_reference_index.py from the links that modules
actually contain, so it cannot drift out of step with the prose. Regenerate it after adding a
citation; CI runs --check and fails if it is stale.
Not yet cited from any module (3): Google C++ Style Guide; Conventional Comments; Clean Code (Martin, 2008). These are here as background reading, not as support for a claim.