Skip to content

Support FFN ranks on every node - #22

Closed
Etelis wants to merge 7 commits into
mainfrom
ffn-domains
Closed

Support FFN ranks on every node#22
Etelis wants to merge 7 commits into
mainfrom
ffn-domains

Conversation

@Etelis

@Etelis Etelis commented Oct 26, 2026

Copy link
Copy Markdown
Member

Part of #18. FFN ranks no longer have to be the first ranks, so a deployment can put FFN GPUs on every node.

What's new

  • ffn_ranks_per_domain=f: the first f GPUs of every node are FFN ranks. Callers keep real expert IDs, and the library maps them to DeepEP's numbering.
  • Hybrid on: the union protocol. FFN GPUs skip their empty network work, because their rails carry no data.
  • Hybrid off: each token takes one RDMA hop, the faster path for decode.
  • A DeepEP fix: direct (hybrid off) mode returned stale rows whenever a node held both roles, FastAFD's DeepEP M2N included. The GIN barrier now fences at system scope before it signals.

When to use it (4-GPU nodes, FP8, hidden 4096, top-8)

  • One FFN GPU per node and ≤ 32 tokens per Attention GPU: use hybrid off. On 3 nodes it beats today's layouts by 3–10%.
  • From 128 tokens, or with two FFN GPUs per node: keep the FFN on its own node(s), with hybrid on.

Details are in docs/design.md section 20.

Validation (kermit, 2–3 nodes × 4 H200)

  • The GPU suite is exact on every per-node layout, with hybrid on and off, BF16 and FP8, and capacities 8–512. FastAFD runs the same layouts. Invalid routes assert.
  • Existing layouts are unchanged: one node, FFN node + Attention node, FFN on node 0, and FP8 combine.
  • The fence fixes all of it: 8/8 runs wrong without it, 8/8 correct with it. It costs about 2.5 µs at 1–8 tokens on the direct path only.
  • Records: validation-per-domain*.json, validation-silent-ffn-rails.json, validation-direct-per-domain*.json.

Follow-ups: the single-node fast protocol over RDMA for the direct path, and afd-plugin support.

Etelis added 7 commits October 26, 2026 18:07
@Etelis
Support FFN ranks as a per-domain prefix
e24b5f6 
ffn_ranks_per_domain f puts the FFN ranks at local ranks [0, f) of every
NVLink domain, so layouts like 4A1F on each of two nodes can run (#18).
Every domain then holds both roles, which neither directional protocol
covers, so these layouts run the union hybrid with per-rank roles. Sources
validate their real routes as the route guard does and translate them to
DeepEP's logical experts in a new kernel; shared device sources are
unchanged. The FastAFD baseline maps real IDs the same way.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
@Etelis
Check the per-domain layout against the physical domains
c94a920 
With ffn_ranks_per_domain set, the NVLink domains must always be the
layout's, so f == F across several domains is rejected instead of
silently running the global prefix. Dispatch asserts the domain shape
and the handle copy for native callers. The comm benchmark maps the
baseline's IDs once outside timing and keeps them alive with the graph,
and rejects arms that give a rank different roles.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
@Etelis
Document FFN ranks on every node with validation and screens
8d2107f 
Design section 20 describes the per-domain FFN prefix, its GPU-suite
record on 2 x 4 and 3 x 4 H200 (source ece614e6) and comm-only screens
against today's layouts. The union protocol runs it at FastAFD M2N's
speed: 12-27% slower than the directional and mixed-role protocols up
to 128 tokens, 4-5% faster than the union with the FFN on one node at
512 tokens.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
@Etelis
Skip scale-out work on FFN rails of a per-domain FFN prefix
15ae52e 
With FFN GPUs at local ranks [0, f) of every NVLink domain, each FFN rail
joins FFN GPUs only, none of them a source, so nothing crosses domains
on it. FFN GPUs now run generated copies of the hybrid dispatch and
combine that skip every scale-out action (rail barrier halves, counts,
finish tails, combine signals and waits) and keep their NVLink work and
scale-up barriers; Attention GPUs keep the union kernels. A per-call
gate (2L <= U) keeps their partials below a forwarder's unflushed
dispatch staging and falls back to union otherwise. The agreement tuple
reports the per-domain protocol. GPU suite on 2 x 4 and 3 x 4 H200 in
docs/validation-silent-ffn-rails.json.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
@Etelis
Fence NVLink stores before the GIN barrier's signals
5a9474c 
DeepEP V2's direct (non-hybrid) kernels write NVLink-reachable peers
with TMA stores and the others with RDMA, then pass a barrier that
flushes the QPs and signals through the NIC. Flushing covers RDMA only,
so a same-node receiver could see the signal before the NVLink rows and
read stale data whenever a node holds both senders and receivers
(FastAFD's DeepEP M2N too). The GIN barrier now fences at system scope
before its flush and signals. On 2 x 4 H200, 8 of 8 runs mismatched
without the fence or with the proxy fence alone, and 8 of 8 passed with
the system fence. NVLink barriers are unchanged.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
@Etelis
Run FFN ranks on every node over the direct (one-hop) path
0f50632 
With hybrid off, DeepEP folds the NVLink domains into one logical domain,
so per-domain roles and the route translation now use the physical
domain size, the in-kernel route check gives way to the translating
one, and ffn_ranks_per_domain no longer requires hybrid mode. Each
Attention GPU then reaches each FFN GPU in one RDMA hop. GPU suite on
2 x 4 and 3 x 4 H200 in docs/validation-direct-per-domain.json.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
@Etelis
Document the direct path for FFN ranks on every node
382c98c 
Design section 20 records silent FFN rails, the GIN barrier fence and a
decode screen on 2 x 4 and 3 x 4 H200: with one FFN GPU per node the
direct path wins up to 32 tokens and beats every FFN-first layout on
three nodes, while hybrid wins from 128 tokens and with two FFN GPUs
per node.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
@Etelis
Etelis requested a review from almogtavor as a code owner October 26, 2026 22:41
@github-actions

github-actions Bot commented Oct 26, 2026 •
edited
Loading

Copy link
Copy Markdown
PR Preview Action v1.8.2
Preview removed because the pull request was closed.
2026-11-01 15:42 UTC

This was referenced Oct 26, 2026
Closed
Open
@Etelis
Etelis marked this pull request as draft October 28, 2026 07:54
This was referenced Oct 29, 2026
Open
Open
Etelis added a commit that referenced this pull request Nov 1, 2026
@Etelis
Add FastAFD's DeepEP M2N to the Attention rails screens
cf12689 
FastAFD's DeepEP M2N in the same per-node layouts is 3.7-16.3% slower
than this branch in every cell and repeat. That screen ran in a second
session on the same nodes, where this branch and #22 re-measured within
1% of the first. Design section 20 gains a row per table.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Etelis added a commit that referenced this pull request Nov 1, 2026
@Etelis
Keep the combine-entry barrier and fold the Attention rail change in
ea597da 
A screen on 2 x 4 and 3 x 4 H200 split the gain: dropping only the
dispatch-entry rail half gives all but 0.7 points of it, dropping only
the combine-entry rail half gives nothing measurable. Attention GPUs
now keep the combine-entry barrier, as in #22. The three fixes stay,
because each guards the next dispatch landing early: combine signals
on their own words, and every rank's reply inbox and combine staging in
the end slices.

The change is also smaller: Attention rails follow afdep_domain_hybrid
instead of a new selector (so the agreement tuple is unchanged), the
combine-word offset lives in the generated copy instead of a header,
one helper names the layouts that use the end slices, and the comments
are condensed.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Etelis added a commit that referenced this pull request Nov 1, 2026
@Etelis
Re-validate the reduced Attention rails change and shorten its record
9e7ac4a 
Design section 20 now describes the protocol as shipped (dispatch-entry
rail half dropped, combine-entry barrier kept) in one short section with
a screen per layout: 1.4-8.5% (3.1-9.1 us) faster than #22's per-node
hybrid in every cell and repeat, and 3.4-16.1% faster than FastAFD's
DeepEP M2N in the same layout. The record is tied to bc3d223c: the GPU
suite (25 runs, invalid routes), deferred-put fault injection where the
protocol and a barrier control pass and each negative control fails
(the reply-inbox control at capacity 512, where the regions overlap),
the barrier split and the screens, as summaries instead of raw
processes.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
@Etelis

Etelis commented Nov 1, 2026

Copy link
Copy Markdown
Member Author

Superseded by #27, which includes these 7 commits unchanged plus the Attention rails change on top (dispatch-entry rail half dropped, validated in docs/validation-attention-rails.json). Review continues there. The ffn-domains branch is kept.

@Etelis Etelis closed this Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@almogtavor almogtavor Awaiting requested review from almogtavor almogtavor is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Etelis