grouped linear single param fixes#3178
Draft
CarlosGomes98 wants to merge 7 commits into
Draft
Conversation
timmoon10
reviewed
Jul 8, 2026
timmoon10
left a comment
Member
There was a problem hiding this comment.
If I understand, this PR fixes 3 bugs:
- The grouped linear module with a single grouped weight does not actually populate the param grad. It computes discrete weight grads, but it never ended up in the single weight param. The fix is to wrap the conversion from single grouped weights to discrete weights within an autograd function.
- The grouped linear module with a single grouped weight was not handling high-precision weight caching (used in Mcore DDP/FSDP). We were caching high-precision weights for the discrete weights, but they were not registered in the single weight param. The fix is to concat the discrete high-precision values and register it in the single weight param.
- When constructing a grouped linear op with
device="meta", it might incorrectly set some Mcore-specific param attrs before the params are initialized. Mcore does this when it constructs an internal fused to replace the grouped linear module implementation (see here).
This is all hairy and messy, but the fixes seem reasonable. It seems that if you enable single grouped weights or any Mcore-specific optimizations, then grouped linear becomes extremely delicate. Users should avoid unless they know what they are doing.
Preserve high-precision MXFP8 initialization when packing expert weights and connect legacy per-GEMM views to the registered grouped parameter for autograd and fused wgrad accumulation. Signed-off-by: CarlosGomes98 <carlosmiguel.gomes@live.com.pt>
Centralize the temporary getter/clearer attachment used by quantized parameter initialization and grouped weight packing. Strengthen MXFP8 tests with discrete-to-grouped FP32 master parity and partial-metadata rejection. Signed-off-by: CarlosGomes98 <carlosmiguel.gomes@live.com.pt>
Signed-off-by: CarlosGomes98 <carlosmiguel.gomes@live.com.pt>
Use TE's existing cached dummy-wgrad mechanism to trigger the grouped parent hook after fused accumulation without allocating and zeroing a full parameter-sized tensor on every backward. Signed-off-by: CarlosGomes98 <carlosmiguel.gomes@live.com.pt>
Signed-off-by: CarlosGomes98 <carlosmiguel.gomes@live.com.pt>
Signed-off-by: CarlosGomes98 <carlosmiguel.gomes@live.com.pt>
1bfd82a to
b3a6202
Compare
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
These changes fix convergence for grouped linear single param.
Issues encountered:
Without the TE fused ops path: causes the experts to get requires_grad=0 and the parent to not receive updates from the individual experts.
With TE fused ops: discards the high precision metadata, so the master weights were starting from dequantized mxfp8 weights, rather than the original high precision weights
Fixes # (issue)
Type of change
Changes
These changes resolve the immediate issue, but cleaner medium- to long-term solutions would require broader changes.
In particular, the bug could likely have been avoided if Transformer Engine supported packing expert weights before quantization. The current flow quantizes each expert separately and then packs the resulting parameters, requiring the packing code to manually preserve and transfer FP8 initialization metadata. This stretches the metadata lifecycle and makes it easy to lose information needed to construct the optimizer’s high-precision master weights. Packing the high-precision weights first and quantizing the grouped parameter once would provide a cleaner and more robust ownership model.
Checklist: