Skip to content

lora: round the bound stack rank up to a multiple of 8 - #120

Merged
Cortexelus merged 1 commit into
mainfrom
lora-rank-padding
Oct 10, 2026
Merged

Cortexelus merged 1 commit into
mainfrom
lora-rank-padding

Conversation

@Cortexelus

Copy link
Copy Markdown
Collaborator

Free speed on every TensorRT branch-LoRA engine. The two low-rank branch GEMMs (lora/down, lora/up) run ~40% faster when the bound stack rank is divisible by 8, and identically for every other rank — so padding the rank up with zero rows costs nothing and the engine returns bit-identical output.

The measurement

Per-layer with trt.IProfiler on sm-music, L=1292, 193 targets, best-of-3, base-model time flat to ±1.6% (so not contention):

rank lora/down + lora/up
8, 16, 24, 32, 40, 48, 64 3.56 – 3.80 ms
every other rank in 1..23 4.93 – 5.21 ms

No exceptions across 1..24.

It is an alignment effect — not a threshold, and not a tactic switch. 8 fp16 values is 16 bytes (one vectorised load) and the fp16 tensor-core fragment wants K/N in multiples of 8. The tactic is chosen at build time at the profile's opt point and one kernel serves every rank in the profile, so nothing switches at runtime; the kernel simply only reaches its fast path when the rank dimension fills whole fragments. Rank 9 and 12 are exactly as slow as rank 1. Attribution confirms it: identical 1327 layers at both ranks, no reformat nodes inserted, and the entire difference lands in those two layers.

End to end

Padded vs unpadded with identical adapter data — same engine, operands sliced back down to rank_real to reproduce what the old code bound:

stack real bound padded unpadded saving max|diff|
r4 4 8 8.33 ms 9.79 ms +1.46 0.000e+00
r12 12 16 8.30 10.32 +2.02 0.000e+00
r16 + r4 20 24 8.33 9.91 +1.58 0.000e+00
r12 + r4 16 16 8.32 8.33 +0.00 0.000e+00

The last row is the control: already aligned, so padding correctly does nothing.

This is not only about the empty stack. Any adapter whose rank is not a multiple of 8 pays it — r4 especially, a common small-adapter rank — and stacks concatenate, so the total is what counts: 16+8=24 was already fast, 16+4=20 was not. It also holds all the way down to L=8, where on medium it is ~16% of the whole step (11.25 → 9.40 ms), because the step itself is so cheap there.

What changed

  • RANK_GRANULARITY = 8 and pad_rank(r, cap) in branch_runtime.py, applied in all three places a rank gets bound: zero(), set_stack() (numpy) and set_stack_gpu() (device). Extra rows stay zero in A, Bt and srow.
  • zero() gets it too. A branch engine will not enqueue with any operand unbound, and there is no "rank 0" — the profile minimum is 1 and set_input_shape rejects a 0 dimension outright — so "no adapter loaded" has to be some rank of zeros. It was rank 1, the slow path.
  • Two ranks are now tracked. rank is the bound rank: buffer shapes, srow width and CUDA-graph shape identity all follow it. rank_real is what the adapters carry, and every user-facing report uses it, so an r4 adapter still reads as rank 4 rather than 8. rank_bound is exposed alongside in set_lora()'s return and in both verify JSONs.
  • pad_rank() refuses to pad past what the engine can bind, for an engine built with a --rank-max that is not a multiple of 8. set_input_shape only returns False outside the profile, so over-binding would render on stale bindings and sound confident and wrong.

Side benefit, measured

Padding makes equal-bound-rank far more common, so the hot-swap path fires more often:

step real bound graphs render
load r4 4 8 2 48.0 ms
swap r4 → r8 8 8 2 47.7 ← graphs kept
swap r8 → r4 4 8 2 47.8 ← graphs kept
swap r4 → r16 16 16 1 423.9 ← bound rank changed, recapture
clear to zero 0 8 1 48.5 ← reuses the graph

An r4 ⇄ r8 swap used to change the buffer shapes and drop every cached graph (~400 ms recapture). Clearing back to "no adapter" is now free too, where zero(1) previously mismatched essentially every adapter rank.

Verification

  • 89 new unit tests for pad_rank — pure arithmetic, so they need neither a GPU nor TensorRT. Note they load the function by exec-ing the slice of branch_runtime.py above class BranchLora, specifically so this runs in CI; worth a look in review, since the alternative was no coverage at all for a rule that is silent when broken.
  • verify_medium --model sa3-medium: control +38.2 dB / rel 1.233e-02, 169/229, fold 1.255e-02 — identical to before this change.
  • verify_medium --model sa3-sm-music: real r16 fold 1.223e-02 and synthetic r4 fold 1.347e-02, against a 1.259e-02 floor; r4 reports rank 4.
  • verify_stack --model sa3-sm-music with a padded mixed-rank stack: real 20 → bound 24, fold 1.314e-02.
  • Four CLI renders across all three DiTs plus the no---lora path, all finite audio, correct real rank reported.
  • A medium r16 end-to-end render that is byte-identical (same sha256) to the same invocation before this change.
  • ruff==0.17.0 check and format clean.

🤖 Generated with Claude Code

The two low-rank branch GEMMs (lora/down, lora/up) run ~40% faster when the BOUND
stack rank is divisible by 8, and identically for every other rank. Padding the rank
up with zero rows is therefore free speed: the extra rows are zeros in A, Bt AND
srow, so they contribute nothing and the engine returns bit-identical output.

Measured per-layer with trt.IProfiler on sm-music, L=1292, 193 targets, best-of-3,
base-model time flat to +-1.6% (so not contention):

  rank 8/16/24/32/40/48/64   lora/down + lora/up  3.56 - 3.80 ms
  every other rank in 1..23                       4.93 - 5.21 ms

No exceptions. It is an alignment effect, not a threshold and not a tactic switch: 8
fp16 values is 16 bytes, one vectorised load, and the fp16 tensor-core fragment wants
K/N in multiples of 8, so the single kernel -- chosen at build time at the profile's
opt point, one for every rank -- only reaches its fast path when the rank dimension
fills whole fragments. Rank 9 and 12 are exactly as slow as rank 1.

End to end, padded vs unpadded with identical adapter data (same engine, operands
sliced back down to compare):

  stack        real  bound   padded   unpadded   saving   max|diff|
  r4              4      8   8.33ms    9.79ms    +1.46    0.000e+00
  r12            12     16   8.30      10.32     +2.02    0.000e+00
  r16 + r4       20     24   8.33       9.91     +1.58    0.000e+00
  r12 + r4       16     16   8.32       8.33     +0.00    0.000e+00   (already aligned)

This is not only about the empty stack. Any adapter whose rank is not a multiple of 8
pays it -- r4 especially -- and stacks concatenate, so the TOTAL is what counts:
16+8=24 is fast, 16+4=20 was not. It holds down to L=8, where on medium it is ~16% of
the whole step (11.25 -> 9.40 ms) because the step itself is so cheap.

zero() gets it too. A branch engine will not enqueue with any operand unbound, and
there is no "rank 0": the profile minimum is 1 and set_input_shape rejects a 0
dimension outright, so "no adapter loaded" has to be a rank of zeros. It was rank 1,
the slow path; it is now padded.

Two ranks are now tracked. `rank` is the BOUND rank -- buffer shapes, srow width and
CUDA-graph shape identity all follow it. `rank_real` is what the adapters carry, and
every user-facing report uses it, so an r4 adapter still reads as rank 4 rather than
8; `rank_bound` is exposed alongside it in set_lora()'s return and both verify JSONs.

Side benefit, measured: because padding makes equal-bound-rank more common, an
r4 <-> r8 hot swap now keeps its captured CUDA graphs (48 ms) where it used to drop
them and recapture (~400 ms), and clearing back to the zero stack is free.

pad_rank() refuses to pad past what the engine can bind, for an engine built with a
--rank-max that is not a multiple of 8: set_input_shape only RETURNS False outside the
profile, so over-binding would render on stale bindings and sound confident and wrong.

Verified: 89 new unit tests (pure arithmetic, no GPU or TensorRT needed);
verify_medium on medium unchanged from before this change (+38.2 dB / rel 1.233e-02
control, 169/229, fold 1.255e-02); verify_medium on sm-music with a real r16 and a
synthetic r4 (fold 1.223e-02 and 1.347e-02 against a 1.259e-02 floor); verify_stack
with a padded mixed-rank stack (real 20 -> bound 24, fold 1.314e-02); four CLI renders
across all three DiTs plus the no---lora path; and a medium r16 end-to-end render that
is byte-identical (same sha256) to the same invocation before this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Cortexelus
Cortexelus merged commit af60ef5 into main Oct 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant