Skip to content

lora: bake DoRA row norms at save time, and add the offline baker - #104

Merged
Cortexelus merged 2 commits into
mainfrom
lora-bake-dora-norm
Oct 11, 2026
Merged

Cortexelus merged 2 commits into
mainfrom
lora-bake-dora-norm

Conversation

@Cortexelus

Copy link
Copy Markdown
Collaborator

Why

A DoRA weight is mag ⊙ (W0 + s·δ) / ‖W0 + s·δ‖_row, so reconstructing it needs the base weights purely to recompute that denominator. Any consumer that only wants the adapter — an exporter, a converter, a merge tool — has to load a multi-GB base checkpoint first. Baking the norm into the adapter removes that: cold start becomes a few seconds instead of a ~10 GB base-weight load.

Costs 0.16 s at save time for 169 adapters, and ~3.5 MB on disk.

Changes

model.py — the per-adapter-type delta was computed inline in each forward, so it could not be called outside a forward pass. Factored into _delta() (unscaled, 2-D, dropout-free — a baked norm must be deterministic, not a sample from the training-time dropout distribution), with _delta_dropout() keeping the behaviour the forwards need. baked_vnorm_row(layer) returns ‖W0 + scaling·δ‖ along dim 1, in fp32, flattening to 2-D first so Conv1d/Conv2d behave — the same reason the forward paths work in 2-D.

It uses self.scaling alone with strength implicitly 1, since the runtime convention is unit strength; folding in lora_strength would bake whatever the slider happened to be set to. The layer is a parameter rather than read off self, because a parametrization keeps no back-reference to its module and adding one solely to bake a norm would create a reference cycle per adapter.

utils.py — get_dora_norm_state_dict(model) emits {layer}.parametrizations.weight.{i}.baked_vnorm_row, matching the layout bake_dora.py writes so the two paths are interchangeable. It walks named_modules rather than get_lora_layers, which yields only (name, p) and not the module the original weight lives on.

bake_dora.py — the offline path, for adapters trained before this. It becomes a fallback rather than a mandatory pre-step.

Scope

Only dora-* adapters whose runtime norm is along dim 1 are baked:

  • dora-cols normalises along dim 0, so a row norm would be silently wrong for it
  • plain lora has no magnitude and needs no norm
  • bora/bora-xs apply their column magnitude to the row-scaled intermediate, so its norm is not a function of base weights and delta alone and cannot be baked as a single tensor

These are skipped rather than baked incorrectly.

Three things that are easy to get wrong, and are handled

The key is deliberately not added to name_is_lora. That predicate also feeds get_lora_params, i.e. optimizer construction, so whitelisting a buffer there would present it as a trainable parameter. It is merged at the save site instead, via an optional model= argument.

model= accepts a sequence. Baking from a parent wrapper would prefix keys (model., conditioner.) and stop matching the adapter tensors, so callers pass the same submodules they built the state dict from.

The norm is exempt from the blanket .half() cast. c = mag/vn scales every DoRA weight, and fp16 storage of the denominator costs a measured worst case of ~4.9e-4 relative; fp32 costs ~3.5 MB across a full adapter set. And norm_base records which base checkpoint the norms were computed against — sa3-medium's base and arc variants reconstruct each other's norms only to ~2.2e-3, and an unlabelled norm makes that mismatch undetectable at load time.

Compatibility

The key is optional by construction. Loaders that do not know about it are unaffected, and adapters saved before this keep working by computing the norm from base weights as they do today.

Verification

On a rank-16 dora-rows adapter:

  • 169 norms baked in 0.16 s
  • baked value matches an independent recompute exactly (0.0 relative)
  • the DoRA invariant that recovered row norms equal |magnitude| holds to 1.2e-07 (note magnitude is a free parameter and training can drive rows negative, so the invariant is against |mag|)
  • baked keys save as fp32 while every other tensor stays fp16
  • all 169 keys line up with an existing adapter tensor — no prefix drift

@Cortexelus

Cortexelus commented Sep 3, 2026 •

Copy link
Copy Markdown
Collaborator Author

Tested — including the training run this PR previously lacked

I'd flagged that none of this was verified by an actual training run. It is now.

Two real sa3-medium runs

Short runs on a real dataset, one with the duration embedder LoRA'd and one with it excluded, saving at step 4 and step 6:

bake_with_sec-step=4   169 dora layers → 169 baked norms
bake_with_sec-step=6   169 dora layers → 169 baked norms
bake_no_sec-step=4     168 dora layers → 168 baked norms
bake_no_sec-step=6     168 dora layers → 168 baked norms

Every checkpoint: one norm per dora layer, count matching the metadata, all values finite and positive, baked keys fp32 while the weights stay fp16, and norm_base recording which weights they belong to. 36/36 assertions.

Two things the training run exposed

1. The hook this PR extends is dead code. export_lora_safetensors — where model= was added — has no callers anywhere in this repo. So "training bakes norms" is true of a method nothing invokes, and any downstream trainer that writes its own .safetensors gets no norms unless it opts in.

Worth keeping in mind for reviewers: model= is optional by design, so a caller that doesn't pass it is unaffected. That also means adding the parameter is not, on its own, enough to make any particular pipeline bake — the call site has to pass it. I've left export_lora_safetensors wired up so the in-repo path is correct.

2. bake_dora.py is self-contained. It takes the base weights by explicit --base-weights path (plus --svd-bases for dora-*-xs, whose frozen bases are never recomputed — per-layer SVD on the DiT's larger matrices is minutes of work and numerically unstable) and parses the adapter from the file alone, so it depends on nothing outside this repo. It preserves each adapter's own lora_index rather than assuming 0, and resolves the base-weight key by trying the known checkpoint prefixes then a unique suffix match — sa3-medium stores the DiT under model.model., so a fixed prefix list misses every layer. BAKED_VNORM_KEY is defined once in utils.py and imported by the baker rather than restated, since the two paths only stay interchangeable if they cannot disagree about the key.

Retrofit produces a byte-identical format

Ran the baker on the unbaked adapter and compared against what the save path writes for the same adapter:

  • identical full key sets and identical baked key sets (169 each)
  • identical shapes; baked fp32 / weights fp16 in both
  • baked values bit-identical (max rel 0.00e+00) when both bake against the same-precision W0
  • non-baked tensors bit-identical
  • training's lora_config (rank/alpha/type/exclude/step/epoch) preserved through the retrofit

One caveat worth recording: the baked norm inherits the precision of the W0 it was baked against. An fp16 inference load vs the fp32 checkpoint differs by 1.34e-4, which is exactly the fp16 row-norm rounding of those weights, not a difference between the two code paths. Training loads fp32, so real checkpoints match the baker exactly.

Also checked: --check exits 1 on an unbaked adapter and 0 after, re-baking without --force is a no-op, plain lora is skipped rather than mangled, and a retrofitted file reloads and applies to the same deviation as the original (4.16283%).

Which adapter types get baked

On synthetic layers, asserting the documented policy:

type baked
dora-rows 2/2 value == ‖W₀ + s·BA‖_row, one per row
dora-rows-xs 2/2 delta reconstructed as U @ M_xs @ Vᵀ
lora 0 no magnitude, needs no norm
dora-cols, dora-cols-xs 0 normalises along dim 0
bora, bora-xs 0 column magnitude applied to the row-scaled intermediate

Plus: the bake ignores lora_strength (unit-strength convention), takes 0.04 s for 169 layers, produces no orphan keys, and save_lora_safetensors without model= adds nothing — it stays opt-in.

Note for merging

This conflicts with #102 in model.py: _blend_magnitude and _delta/_delta_dropout/baked_vnorm_row land at the same insertion point and both rewrite the same forward bodies. Both sides are purely additive — keeping both resolves it, and I verified the merged forwards pick up both changes.

@Cortexelus
Cortexelus force-pushed the lora-bake-dora-norm branch 2 times, most recently from d876559 to 438b78f Compare September 3, 2026 23:48
A DoRA weight is mag ⊙ (W0 + s·δ)/‖W0 + s·δ‖_row, so reconstructing it needs
the base weights purely to recompute that denominator. Any consumer that only
wants the adapter — an exporter, a converter, a merge tool — has to load a
multi-GB base checkpoint first. Baking the norm into the adapter removes that.

model.py: the per-adapter-type delta was computed inline in each forward, so it
could not be called outside a forward pass. Factored into _delta() (unscaled,
2-D, dropout-free — a baked norm must be deterministic) with _delta_dropout()
keeping the training-time behaviour the forwards need. baked_vnorm_row(layer)
returns ‖W0 + scaling·δ‖ along dim 1, in fp32, flattening to 2-D first so
Conv1d/Conv2d behave. It uses self.scaling alone with strength implicitly 1,
since the runtime convention is unit strength — folding in lora_strength would
bake whatever the slider happened to be set to. The layer is a parameter rather
than read off self: a parametrization keeps no back-reference to its module, and
adding one solely to bake a norm would create a cycle per adapter.

utils.py: get_dora_norm_state_dict(model) emits
{layer}.parametrizations.weight.{i}.baked_vnorm_row, matching the layout
bake_dora.py writes so the two paths are interchangeable. It walks
named_modules rather than get_lora_layers, which yields only (name, p) and not
the module the original weight lives on. Only dora-* adapters whose runtime
norm is along dim 1 are baked: dora-cols normalises along dim 0 so a row norm
would be silently wrong, plain lora has no magnitude, and bora's column
magnitude applies to the row-scaled intermediate so its norm is not a function
of base and delta alone.

The key is deliberately NOT added to name_is_lora: that predicate also feeds
get_lora_params, i.e. optimizer construction, so whitelisting a buffer there
would present it as a trainable parameter. It is merged at the save site
instead, via an optional model= argument that accepts a sequence — baking from
a parent wrapper would prefix keys ("model.", "conditioner.") and stop matching
the adapter tensors.

Two things that are easy to get wrong and are handled: the norm is exempt from
the blanket .half() cast, since c = mag/vn scales every DoRA weight and fp16
storage of the denominator costs a worst case of ~4.9e-4 relative while fp32
costs ~3.5 MB across a full adapter set; and norm_base records which base
checkpoint the norms were computed against, because sa3-medium's base and arc
variants reconstruct each other's norms only to ~2.2e-3 and an unlabelled norm
makes that mismatch undetectable at load time.

The key is optional by construction — loaders that do not know about it are
unaffected, and adapters trained before this keep working by computing the norm
from base weights as before. bake_dora.py is the offline path for those, so it
is a fallback rather than a mandatory pre-step. It takes the base weights by
explicit --base-weights path (and --svd-bases for dora-*-xs, whose frozen bases
are never recomputed — per-layer SVD on the DiT's larger matrices is minutes of
work and numerically unstable), preserves each adapter's own lora_index rather
than assuming 0, and resolves the base-weight key by trying the known checkpoint
prefixes then a unique suffix match, since how deeply a checkpoint wraps the DiT
varies by model.

BAKED_VNORM_KEY is defined once, in utils.py, and re-exported from the package:
consumers need to recognise a baked key without hardcoding "baked_vnorm_row", and
the offline baker imports the same constant rather than restating the literal --
the two paths only stay interchangeable if they cannot disagree about the key. That
relative import means bake_dora runs as `python -m ...`, so it carries no shebang.

Verified on a rank-16 dora-rows adapter: 169 norms baked in 0.16 s; the baked
value matches an independent recompute exactly (0.0 rel); the DoRA invariant
that recovered row norms equal |magnitude| holds to 1.2e-07; baked keys save as
fp32 while every other tensor stays fp16; and all 169 keys line up with an
existing adapter tensor (no prefix drift). The offline baker reproduces the
save-time norms bit-identically when both bake against the same-precision W0.
bake_dora.py imports from its own package, so it runs as `python -m
stable_audio_3.models.lora.bake_dora` and not as a script path -- fine, but a
mouthful for something you reach for occasionally.

bake_dora.sh resolves the repo root from its own location, so it works from any
working directory without the caller setting PYTHONPATH, and puts the checkout
first on the path: a wrapper shipped inside the repo should bake with the repo's
code rather than whatever copy is installed. Arguments pass straight through and
the baker's exit status is preserved, so `--check` stays usable in a script.

Verified the wrapper and the module invocation produce bit-identical output over
all 676 tensors, differing only in the baked_at timestamp.
@Cortexelus
Cortexelus merged commit 77e0398 into main Oct 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants