Skip to content

training: bake dora row norms into the checkpoints we save - #17

Merged
Cortexelus merged 1 commit into
mainfrom
bake-dora-on-save
Oct 11, 2026
Merged

Cortexelus merged 1 commit into
mainfrom
bake-dora-on-save

Conversation

@Cortexelus

@Cortexelus Cortexelus commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

What

Pass the model through to save_lora_safetensors so every checkpoint a run saves carries the baked dora row norms, and record which base weights they were taken against.

Depends on the SA3 side landing first: Stability-AI/stable-audio-3#104 adds the model= / norm_base= parameters this feature-detects. Without it, nothing changes — the kwargs are simply not passed.

Why

The dora fold needs c = magnitude / ||W0 + s*BA||_row. Nothing in the adapter file determines that denominator, so a loader that wants it has to pull the full base weights back in just to take one row norm. The value is fixed the moment the adapter is saved, so save it.

save_lora_step is the only path that writes a .safetensors for a run — the training wrapper's own export_lora_safetensors has no callers anywhere in either repo — so wiring it here is what actually gets norms into the files runs produce.

Notes

  • Feature-detected, not assumed. Only backends whose save_lora_safetensors accepts model= get it. Passing the kwarg blind would raise TypeError on an older stable-audio-tools and lose the checkpoint at the one moment it cannot be recomputed.
  • norm_base does not use model_config["base_model"]. No config the dashboard writes today sets that key (0 of 100+ existing _model.json files), so the label would be empty on every run. It falls back to the streamed checkpoint's filename. Worth recording rather than inferring: base and arc weights reconstruct each other's norms closely enough that a wrong pairing raises nothing.

Verified by two real training runs

Short sa3-medium runs on a small local dataset, one with the duration embedder LoRA'd and one with it excluded:

bake_with_sec-step=4  169 dora layers, 169 baked norms, fp32 keys / fp16 weights
bake_with_sec-step=6  169 dora layers, 169 baked norms
bake_no_sec-step=4    168 dora layers, 168 baked norms
bake_no_sec-step=6    168 dora layers, 168 baked norms
norm_base: 'stable-audio-3-medium-RF.safetensors'

36/36 assertions: one norm per dora layer, counts matching the metadata, norms finite and positive, baked keys fp32 while weights stay fp16, and the only layer differing between the two runs is the duration embedder.

Cross-checked against the offline baker: bake_dora.py run on the same adapter reproduces these norms bit-identically at matched precision, so a retrofitted file and a freshly trained one are interchangeable.

The dora fold needs c = magnitude / ||W0 + s*BA||_row. Nothing in the adapter
determines that denominator, so any loader that wants it has to pull the full base
weights back in just to take one row norm. The value is fixed at save time, so save
it.

save_lora_step is the only path that writes a .safetensors for a run -- the training
wrapper's own export_lora_safetensors has no callers -- so wiring it here is what
actually gets norms into the files runs produce.

Feature-detected rather than assumed: only backends whose save_lora_safetensors
takes model= get it. Passing the kwarg blind would raise TypeError on an older
stable-audio-tools and lose the checkpoint at the one moment it cannot be recomputed.

norm_base records which W0 the norms belong to, falling back to the streamed
checkpoint's filename. It cannot rely on model_config's "base_model": no config the
dashboard writes today sets it, so the label would be empty on every run, and base
and arc weights reconstruct each other's norms closely enough that a wrong pairing
raises nothing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant