Skip to content

[Bug] Fix fp8 deepgemm batch invariant#37718

Open
yewentao256 wants to merge 1 commit intomainfrom
wentao-fix-fp8-deepgemm-batch-invariant
Open

[Bug] Fix fp8 deepgemm batch invariant#37718
yewentao256 wants to merge 1 commit intomainfrom
wentao-fix-fp8-deepgemm-batch-invariant

Conversation

@yewentao256
Copy link
Member

Purpose

Selecting kernels between run_flashinfer_deepgemm_swapAB and run_deepgemm will break batch invaraince, this PR fixes the issue.

Test

VLLM_TEST_MODEL="Qwen/Qwen3-30B-A3B-Thinking-2507-FP8" pytest tests/v1/determinism/test_batch_invariance.py -svx

# now
============== 10 passed, 28 warnings in 458.02s (0:07:38) ===============
# original
======================= short test summary info ========================
FAILED tests/v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle[FLASH_ATTN] - Failed: Nondeterministic outputs detected: 1 failed out of 1 trials...
!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!
============== 1 failed, 19 warnings in 79.75s (0:01:19) ===============
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute

@yewentao256 yewentao256 added the ready ONLY add when PR is ready to merge/full CI is needed label Mar 20, 2026
@mergify mergify bot added the bug Something isn't working label Mar 20, 2026
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses a batch invariance issue in the FP8 deepgemm implementation by modifying the kernel selection logic. When batch invariance mode is active, the code now bypasses the dynamic selection between run_flashinfer_deepgemm_swapAB and run_deepgemm, and instead consistently calls run_deepgemm. This change ensures deterministic outputs across different batch sizes. The implementation is straightforward and appears to correctly resolve the described issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant