DeepSeek-V4 Deconstruction Lab

Note: This repo is entirely generated by Agents. Feel free to contact us for any inquiries.

中文 README

An interactive, puzzle-first walkthrough of DeepSeek-V4's reference implementation (model.py and kernel.py). The repo ships:

• A 20-lesson curriculum (tutorial/lesson_specs.py) that slices the reference into small, testable chunks pinned to line ranges in the real source.
• A browser IDE (site/) with syntax-highlighted lessons, clickable fill-in-the-blank "puzzle pills", a free editor, the reference solution, and a local test runner with per-check pass/fail diffs.
• A generator (tools/build_tutorial.py) that materialises runnable workshop files, solution files, reference slices, lessons.json, and the site bootstrap from a single source of truth.
• A checker (tools/check_lesson.py) that executes a lesson's validator against whatever the user just submitted (blanks map or full code), in a sandboxed namespace.

The UI and pedagogy are inspired by skyzh/tiny-llm and shareAI-lab/learn-claude-code; the technical content is grounded in model.py, kernel.py, and the DeepSeek_V4.pdf paper in this repo.

---

Quick Start

Requirements: Python 3.10+ and Node.js 18+.

npm install          # no runtime deps; records Node engine
npm run dev          # builds lessons + starts the local tutor at :3000

Then open http://localhost:3000. Pick a lesson from the sidebar, click the highlighted pills inside the code to enter your answer, and hit Run tests. To compare against the published fix, hit Run solution.

npm scripts

Command	What it does
npm install	Installs the empty runtime manifest (the project is pure static + Python).
npm run build	Regenerates tutorial/generated/** and site/assets/lessons-data.js from lesson_specs.py.
npm run dev	Runs build then starts the dev server at http://localhost:3000. Autoreloads lesson data on each page load.
npm start	Starts the dev server without rebuilding.
npm test	Rebuilds, then runs the smoke test suite (tests/test_tutorial_build.py).
npm run check:all	Runs every lesson's validator against both the solution blanks and the starter blanks. Fails on any crash.

You do not need a CUDA build to take the lessons; each lesson uses small pure-Python stubs that mimic torch / tilelang behaviour where relevant.

---

Curriculum

Twenty lessons spread across five stages (42 blanks total, ~2 per lesson).

1 — Foundations (Warm-up)

• s01 Linear Dispatch — model.py:108-120
• s02 Weight & Scale Layout — model.py:123-150
• s03 Row vs Column Parallel Shards — model.py:155-180
• s04 RMSNorm in Float32 — model.py:183-196

2 — Rotary / YaRN (Warm-up → Core)

• s05 Base RoPE Frequencies — model.py:220-224
• s06 YaRN Smooth Interpolation — model.py:221-229
• s07 Apply Rotary Embedding — model.py:232-244

3 — Compressed / Sparse Attention (Core)

• s08 Compressor Prefill — model.py:316-342
• s09 Window KV Cache — model.py:518-533
• s10 Window + Compressed Top-k — model.py:507-516
• s11 Lightning Indexer Scoring — model.py:418-427

4 — DeepSeekMoE + mHC Routing (Core → Advanced)

• s12 Gate (sqrt-softplus + bias) — model.py:564-584
• s13 Sparse MoE Dispatch — model.py:635-645
• s14 SwiGLU Expert — model.py:589-606
• s15 mHC Pre-Mix — model.py:674-682
• s16 mHC Post-Mix — model.py:684-687

5 — Fused Kernels (Advanced)

• s17 Block-wise FP8 Activation Quant — kernel.py:41-102
• s18 FP4 GEMM Scale Alignment — kernel.py:442-515
• s19 Sparse Attention Online Softmax — kernel.py:276-352
• s20 mHC Sinkhorn Iterations — kernel.py:371-438

Every lesson entry in lesson_specs.py includes: a tagline, difficulty, tags, paper anchor, "what you learn", step-by-step workshop notes, pitfalls, the runnable starter code, the blanks table, and the validator. The generator is the only thing that turns that spec into files and site data.

---

How the tutor renders a lesson

1. lesson_specs.py defines the spec. starter_code is a real, runnable Python module; each blank is represented by a __B_<id>__ identifier rather than an opaque BLANK string.
2. npm run build substitutes __B_<id>__ with the starter value to produce tutorial/generated/puzzles/NN_<id>.py, and with the solution value to produce tutorial/generated/solutions/NN_<id>.py. Both files are importable on their own with no special harness.
3. The site loads the starter code, then overlays each blank as a click-to-edit "puzzle pill" with its hint and expected value.
4. Run tests POSTs { lessonId, blanks } to /api/check. The server shells out to tools/check_lesson.py, which substitutes blanks, runs the resulting module in a restricted namespace, then runs the lesson's validator which returns structured checks (name, passed, expected, actual).
5. The frontend renders the per-check result with green/red badges and an expected-vs-actual diff, plus captured stdout and Python tracebacks.
6. Run solution does the same but uses the published solution values, which is handy for comparing behaviour or catching a stale lesson spec.

---

Repository layout

.
├── model.py                    Reference DeepSeek-V4 model
├── kernel.py                   Reference DeepSeek-V4 kernels (TileLang)
├── DeepSeek_V4.pdf             Paper (cited from lesson "paper_anchor" fields)
├── tutorial/
│   ├── lesson_specs.py         Single source of truth for all 20 lessons
│   └── generated/              Built artefacts (do not edit by hand)
│       ├── puzzles/NN_<id>.py
│       ├── solutions/NN_<id>.py
│       ├── references/NN_<id>.py
│       └── lessons.json
├── tools/
│   ├── build_tutorial.py       Spec -> generated + site bootstrap
│   ├── check_lesson.py         /api/check backend (runs user code + validator)
│   └── check_all_lessons.py    npm run check:all helper
├── site/
│   ├── index.html              Dark-theme shell, matches shareAI-lab palette
│   ├── styles.css              Responsive grid + puzzle-pill styling
│   ├── app.js                  Client-side renderer + test runner
│   ├── dev-server.mjs          Static server + /api/check endpoint
│   └── assets/lessons-data.js  Built lesson bundle (for static hosting)
└── tests/test_tutorial_build.py

---

Authoring a new lesson

1. Pick a slice of model.py / kernel.py. Keep it to a single idea; most lessons are 15-80 lines.
2. Add a dict to LESSON_SPECS in tutorial/lesson_specs.py:
   • id, order, session, stage, track, title, tagline.
   • source_path + line_start + line_end (used for the "reference" tab and a build-time sanity test).
   • summary, what_you_learn, workshop_steps, pitfalls, paper_anchor.
   • blanks: each entry has an id, label, hint, a runnable starter expression, and the solution expression.
   • starter_code: a self-contained Python module with __B_<id>__ tokens wherever a blank belongs. The code must be runnable even with all starters (the check:all helper enforces this).
   • validator_code: a module body that defines def validate(ns) and returns a list of {name, passed, expected, actual} dicts.
3. Run npm test to regenerate files and verify the lesson (including the new "every starter must run" and "every solution must pass" assertions).
4. Run npm run dev and walk through the lesson in the browser.

Tips:

• Keep runtime imports out of starter_code; use tiny local stubs if you need something from torch. Lessons should execute fast and without GPUs.
• If your solution depends on operator precedence (e.g. a / (b + c)), wrap __B_<id>__ in parentheses in starter_code so the starter evaluates safely.
• If a starter value might divide by zero or overflow, pick a non-pathological default (e.g. "score" rather than "running_max" before it's updated).

---

Troubleshooting

"Some checks failed: …" with expected vs actual. Good signal: your code ran but the validator disagreed with the spec. Read the expected/actual diff under the failing check.

Traceback card appears in the inspector. Your code crashed before the validator finished. Read the traceback (file paths map to the virtual path lesson://<id>), fix the crash, and re-run.

npm run dev reports EADDRINUSE 3000. Another process is on port 3000; either stop it or set PORT=4000 npm run dev.

Blanks don't appear. Make sure starter_code actually contains the token __B_<id>__. The site only renders a puzzle pill for tokens it finds in the rendered workshop; npm test asserts this for every declared blank.

---

License

Same license as the reference implementation. See the parent DeepSeek-V4 release for details.

---

References

• DeepSeek-V4-Pro

---

Contact

If you have any questions, please raise an issue or contact us at Foreverlasting1202@outlook.com.