Repository navigation
lora: bake DoRA row norms at save time, and add the offline baker - #104
Conversation
Tested — including the training run this PR previously lackedI'd flagged that none of this was verified by an actual training run. It is now. Two real
|
| type | baked | |
|---|---|---|
dora-rows |
2/2 | value == ‖W₀ + s·BA‖_row, one per row |
dora-rows-xs |
2/2 | delta reconstructed as U @ M_xs @ Vᵀ |
lora |
0 | no magnitude, needs no norm |
dora-cols, dora-cols-xs |
0 | normalises along dim 0 |
bora, bora-xs |
0 | column magnitude applied to the row-scaled intermediate |
Plus: the bake ignores lora_strength (unit-strength convention), takes 0.04 s for 169 layers, produces no orphan keys, and save_lora_safetensors without model= adds nothing — it stays opt-in.
Note for merging
This conflicts with #102 in model.py: _blend_magnitude and _delta/_delta_dropout/baked_vnorm_row land at the same insertion point and both rewrite the same forward bodies. Both sides are purely additive — keeping both resolves it, and I verified the merged forwards pick up both changes.
d876559 to
438b78f
Compare
5d73e51 to
695f719
Compare
A DoRA weight is mag ⊙ (W0 + s·δ)/‖W0 + s·δ‖_row, so reconstructing it needs
the base weights purely to recompute that denominator. Any consumer that only
wants the adapter — an exporter, a converter, a merge tool — has to load a
multi-GB base checkpoint first. Baking the norm into the adapter removes that.
model.py: the per-adapter-type delta was computed inline in each forward, so it
could not be called outside a forward pass. Factored into _delta() (unscaled,
2-D, dropout-free — a baked norm must be deterministic) with _delta_dropout()
keeping the training-time behaviour the forwards need. baked_vnorm_row(layer)
returns ‖W0 + scaling·δ‖ along dim 1, in fp32, flattening to 2-D first so
Conv1d/Conv2d behave. It uses self.scaling alone with strength implicitly 1,
since the runtime convention is unit strength — folding in lora_strength would
bake whatever the slider happened to be set to. The layer is a parameter rather
than read off self: a parametrization keeps no back-reference to its module, and
adding one solely to bake a norm would create a cycle per adapter.
utils.py: get_dora_norm_state_dict(model) emits
{layer}.parametrizations.weight.{i}.baked_vnorm_row, matching the layout
bake_dora.py writes so the two paths are interchangeable. It walks
named_modules rather than get_lora_layers, which yields only (name, p) and not
the module the original weight lives on. Only dora-* adapters whose runtime
norm is along dim 1 are baked: dora-cols normalises along dim 0 so a row norm
would be silently wrong, plain lora has no magnitude, and bora's column
magnitude applies to the row-scaled intermediate so its norm is not a function
of base and delta alone.
The key is deliberately NOT added to name_is_lora: that predicate also feeds
get_lora_params, i.e. optimizer construction, so whitelisting a buffer there
would present it as a trainable parameter. It is merged at the save site
instead, via an optional model= argument that accepts a sequence — baking from
a parent wrapper would prefix keys ("model.", "conditioner.") and stop matching
the adapter tensors.
Two things that are easy to get wrong and are handled: the norm is exempt from
the blanket .half() cast, since c = mag/vn scales every DoRA weight and fp16
storage of the denominator costs a worst case of ~4.9e-4 relative while fp32
costs ~3.5 MB across a full adapter set; and norm_base records which base
checkpoint the norms were computed against, because sa3-medium's base and arc
variants reconstruct each other's norms only to ~2.2e-3 and an unlabelled norm
makes that mismatch undetectable at load time.
The key is optional by construction — loaders that do not know about it are
unaffected, and adapters trained before this keep working by computing the norm
from base weights as before. bake_dora.py is the offline path for those, so it
is a fallback rather than a mandatory pre-step. It takes the base weights by
explicit --base-weights path (and --svd-bases for dora-*-xs, whose frozen bases
are never recomputed — per-layer SVD on the DiT's larger matrices is minutes of
work and numerically unstable), preserves each adapter's own lora_index rather
than assuming 0, and resolves the base-weight key by trying the known checkpoint
prefixes then a unique suffix match, since how deeply a checkpoint wraps the DiT
varies by model.
BAKED_VNORM_KEY is defined once, in utils.py, and re-exported from the package:
consumers need to recognise a baked key without hardcoding "baked_vnorm_row", and
the offline baker imports the same constant rather than restating the literal --
the two paths only stay interchangeable if they cannot disagree about the key. That
relative import means bake_dora runs as `python -m ...`, so it carries no shebang.
Verified on a rank-16 dora-rows adapter: 169 norms baked in 0.16 s; the baked
value matches an independent recompute exactly (0.0 rel); the DoRA invariant
that recovered row norms equal |magnitude| holds to 1.2e-07; baked keys save as
fp32 while every other tensor stays fp16; and all 169 keys line up with an
existing adapter tensor (no prefix drift). The offline baker reproduces the
save-time norms bit-identically when both bake against the same-precision W0.
bake_dora.py imports from its own package, so it runs as `python -m stable_audio_3.models.lora.bake_dora` and not as a script path -- fine, but a mouthful for something you reach for occasionally. bake_dora.sh resolves the repo root from its own location, so it works from any working directory without the caller setting PYTHONPATH, and puts the checkout first on the path: a wrapper shipped inside the repo should bake with the repo's code rather than whatever copy is installed. Arguments pass straight through and the baker's exit status is preserved, so `--check` stays usable in a script. Verified the wrapper and the module invocation produce bit-identical output over all 676 tensors, differing only in the baked_at timestamp.
695f719 to
facad96
Compare
Why
A DoRA weight is
mag ⊙ (W0 + s·δ) / ‖W0 + s·δ‖_row, so reconstructing it needs the base weights purely to recompute that denominator. Any consumer that only wants the adapter — an exporter, a converter, a merge tool — has to load a multi-GB base checkpoint first. Baking the norm into the adapter removes that: cold start becomes a few seconds instead of a ~10 GB base-weight load.Costs 0.16 s at save time for 169 adapters, and ~3.5 MB on disk.
Changes
model.py— the per-adapter-type delta was computed inline in each forward, so it could not be called outside a forward pass. Factored into_delta()(unscaled, 2-D, dropout-free — a baked norm must be deterministic, not a sample from the training-time dropout distribution), with_delta_dropout()keeping the behaviour the forwards need.baked_vnorm_row(layer)returns‖W0 + scaling·δ‖along dim 1, in fp32, flattening to 2-D first soConv1d/Conv2dbehave — the same reason the forward paths work in 2-D.It uses
self.scalingalone with strength implicitly 1, since the runtime convention is unit strength; folding inlora_strengthwould bake whatever the slider happened to be set to. The layer is a parameter rather than read offself, because a parametrization keeps no back-reference to its module and adding one solely to bake a norm would create a reference cycle per adapter.utils.py—get_dora_norm_state_dict(model)emits{layer}.parametrizations.weight.{i}.baked_vnorm_row, matching the layoutbake_dora.pywrites so the two paths are interchangeable. It walksnamed_modulesrather thanget_lora_layers, which yields only(name, p)and not the module the original weight lives on.bake_dora.py— the offline path, for adapters trained before this. It becomes a fallback rather than a mandatory pre-step.Scope
Only
dora-*adapters whose runtime norm is along dim 1 are baked:dora-colsnormalises along dim 0, so a row norm would be silently wrong for itlorahas no magnitude and needs no normbora/bora-xsapply their column magnitude to the row-scaled intermediate, so its norm is not a function of base weights and delta alone and cannot be baked as a single tensorThese are skipped rather than baked incorrectly.
Three things that are easy to get wrong, and are handled
The key is deliberately not added to
name_is_lora. That predicate also feedsget_lora_params, i.e. optimizer construction, so whitelisting a buffer there would present it as a trainable parameter. It is merged at the save site instead, via an optionalmodel=argument.model=accepts a sequence. Baking from a parent wrapper would prefix keys (model.,conditioner.) and stop matching the adapter tensors, so callers pass the same submodules they built the state dict from.The norm is exempt from the blanket
.half()cast.c = mag/vnscales every DoRA weight, and fp16 storage of the denominator costs a measured worst case of ~4.9e-4 relative; fp32 costs ~3.5 MB across a full adapter set. Andnorm_baserecords which base checkpoint the norms were computed against — sa3-medium'sbaseandarcvariants reconstruct each other's norms only to ~2.2e-3, and an unlabelled norm makes that mismatch undetectable at load time.Compatibility
The key is optional by construction. Loaders that do not know about it are unaffected, and adapters saved before this keep working by computing the norm from base weights as they do today.
Verification
On a rank-16
dora-rowsadapter:|magnitude|holds to 1.2e-07 (notemagnitudeis a free parameter and training can drive rows negative, so the invariant is against|mag|)