Repository navigation
lora: round the bound stack rank up to a multiple of 8 - #120
Merged
Merged
Conversation
The two low-rank branch GEMMs (lora/down, lora/up) run ~40% faster when the BOUND stack rank is divisible by 8, and identically for every other rank. Padding the rank up with zero rows is therefore free speed: the extra rows are zeros in A, Bt AND srow, so they contribute nothing and the engine returns bit-identical output. Measured per-layer with trt.IProfiler on sm-music, L=1292, 193 targets, best-of-3, base-model time flat to +-1.6% (so not contention): rank 8/16/24/32/40/48/64 lora/down + lora/up 3.56 - 3.80 ms every other rank in 1..23 4.93 - 5.21 ms No exceptions. It is an alignment effect, not a threshold and not a tactic switch: 8 fp16 values is 16 bytes, one vectorised load, and the fp16 tensor-core fragment wants K/N in multiples of 8, so the single kernel -- chosen at build time at the profile's opt point, one for every rank -- only reaches its fast path when the rank dimension fills whole fragments. Rank 9 and 12 are exactly as slow as rank 1. End to end, padded vs unpadded with identical adapter data (same engine, operands sliced back down to compare): stack real bound padded unpadded saving max|diff| r4 4 8 8.33ms 9.79ms +1.46 0.000e+00 r12 12 16 8.30 10.32 +2.02 0.000e+00 r16 + r4 20 24 8.33 9.91 +1.58 0.000e+00 r12 + r4 16 16 8.32 8.33 +0.00 0.000e+00 (already aligned) This is not only about the empty stack. Any adapter whose rank is not a multiple of 8 pays it -- r4 especially -- and stacks concatenate, so the TOTAL is what counts: 16+8=24 is fast, 16+4=20 was not. It holds down to L=8, where on medium it is ~16% of the whole step (11.25 -> 9.40 ms) because the step itself is so cheap. zero() gets it too. A branch engine will not enqueue with any operand unbound, and there is no "rank 0": the profile minimum is 1 and set_input_shape rejects a 0 dimension outright, so "no adapter loaded" has to be a rank of zeros. It was rank 1, the slow path; it is now padded. Two ranks are now tracked. `rank` is the BOUND rank -- buffer shapes, srow width and CUDA-graph shape identity all follow it. `rank_real` is what the adapters carry, and every user-facing report uses it, so an r4 adapter still reads as rank 4 rather than 8; `rank_bound` is exposed alongside it in set_lora()'s return and both verify JSONs. Side benefit, measured: because padding makes equal-bound-rank more common, an r4 <-> r8 hot swap now keeps its captured CUDA graphs (48 ms) where it used to drop them and recapture (~400 ms), and clearing back to the zero stack is free. pad_rank() refuses to pad past what the engine can bind, for an engine built with a --rank-max that is not a multiple of 8: set_input_shape only RETURNS False outside the profile, so over-binding would render on stale bindings and sound confident and wrong. Verified: 89 new unit tests (pure arithmetic, no GPU or TensorRT needed); verify_medium on medium unchanged from before this change (+38.2 dB / rel 1.233e-02 control, 169/229, fold 1.255e-02); verify_medium on sm-music with a real r16 and a synthetic r4 (fold 1.223e-02 and 1.347e-02 against a 1.259e-02 floor); verify_stack with a padded mixed-rank stack (real 20 -> bound 24, fold 1.314e-02); four CLI renders across all three DiTs plus the no---lora path; and a medium r16 end-to-end render that is byte-identical (same sha256) to the same invocation before this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Free speed on every TensorRT branch-LoRA engine. The two low-rank branch GEMMs (
lora/down,lora/up) run ~40% faster when the bound stack rank is divisible by 8, and identically for every other rank — so padding the rank up with zero rows costs nothing and the engine returns bit-identical output.The measurement
Per-layer with
trt.IProfileron sm-music, L=1292, 193 targets, best-of-3, base-model time flat to ±1.6% (so not contention):lora/down+lora/upNo exceptions across 1..24.
It is an alignment effect — not a threshold, and not a tactic switch. 8 fp16 values is 16 bytes (one vectorised load) and the fp16 tensor-core fragment wants K/N in multiples of 8. The tactic is chosen at build time at the profile's opt point and one kernel serves every rank in the profile, so nothing switches at runtime; the kernel simply only reaches its fast path when the rank dimension fills whole fragments. Rank 9 and 12 are exactly as slow as rank 1. Attribution confirms it: identical 1327 layers at both ranks, no reformat nodes inserted, and the entire difference lands in those two layers.
End to end
Padded vs unpadded with identical adapter data — same engine, operands sliced back down to
rank_realto reproduce what the old code bound:The last row is the control: already aligned, so padding correctly does nothing.
This is not only about the empty stack. Any adapter whose rank is not a multiple of 8 pays it — r4 especially, a common small-adapter rank — and stacks concatenate, so the total is what counts: 16+8=24 was already fast, 16+4=20 was not. It also holds all the way down to L=8, where on medium it is ~16% of the whole step (11.25 → 9.40 ms), because the step itself is so cheap there.
What changed
RANK_GRANULARITY = 8andpad_rank(r, cap)inbranch_runtime.py, applied in all three places a rank gets bound:zero(),set_stack()(numpy) andset_stack_gpu()(device). Extra rows stay zero inA,Btandsrow.zero()gets it too. A branch engine will not enqueue with any operand unbound, and there is no "rank 0" — the profile minimum is 1 andset_input_shaperejects a 0 dimension outright — so "no adapter loaded" has to be some rank of zeros. It was rank 1, the slow path.rankis the bound rank: buffer shapes,srowwidth and CUDA-graph shape identity all follow it.rank_realis what the adapters carry, and every user-facing report uses it, so an r4 adapter still reads asrank 4rather than 8.rank_boundis exposed alongside inset_lora()'s return and in both verify JSONs.pad_rank()refuses to pad past what the engine can bind, for an engine built with a--rank-maxthat is not a multiple of 8.set_input_shapeonly returns False outside the profile, so over-binding would render on stale bindings and sound confident and wrong.Side benefit, measured
Padding makes equal-bound-rank far more common, so the hot-swap path fires more often:
An r4 ⇄ r8 swap used to change the buffer shapes and drop every cached graph (~400 ms recapture). Clearing back to "no adapter" is now free too, where
zero(1)previously mismatched essentially every adapter rank.Verification
pad_rank— pure arithmetic, so they need neither a GPU nor TensorRT. Note they load the function byexec-ing the slice ofbranch_runtime.pyaboveclass BranchLora, specifically so this runs in CI; worth a look in review, since the alternative was no coverage at all for a rule that is silent when broken.verify_medium --model sa3-medium: control +38.2 dB / rel 1.233e-02, 169/229, fold 1.255e-02 — identical to before this change.verify_medium --model sa3-sm-music: real r16 fold 1.223e-02 and synthetic r4 fold 1.347e-02, against a 1.259e-02 floor; r4 reports rank 4.verify_stack --model sa3-sm-musicwith a padded mixed-rank stack: real 20 → bound 24, fold 1.314e-02.--lorapath, all finite audio, correct real rank reported.ruff==0.17.0check and format clean.🤖 Generated with Claude Code