Repository navigation
training: bake dora row norms into the checkpoints we save - #17
Merged
Merged
Conversation
The dora fold needs c = magnitude / ||W0 + s*BA||_row. Nothing in the adapter determines that denominator, so any loader that wants it has to pull the full base weights back in just to take one row norm. The value is fixed at save time, so save it. save_lora_step is the only path that writes a .safetensors for a run -- the training wrapper's own export_lora_safetensors has no callers -- so wiring it here is what actually gets norms into the files runs produce. Feature-detected rather than assumed: only backends whose save_lora_safetensors takes model= get it. Passing the kwarg blind would raise TypeError on an older stable-audio-tools and lose the checkpoint at the one moment it cannot be recomputed. norm_base records which W0 the norms belong to, falling back to the streamed checkpoint's filename. It cannot rely on model_config's "base_model": no config the dashboard writes today sets it, so the label would be empty on every run, and base and arc weights reconstruct each other's norms closely enough that a wrong pairing raises nothing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Pass the model through to
save_lora_safetensorsso every checkpoint a run saves carries the baked dora row norms, and record which base weights they were taken against.Depends on the SA3 side landing first: Stability-AI/stable-audio-3#104 adds the
model=/norm_base=parameters this feature-detects. Without it, nothing changes — the kwargs are simply not passed.Why
The dora fold needs
c = magnitude / ||W0 + s*BA||_row. Nothing in the adapter file determines that denominator, so a loader that wants it has to pull the full base weights back in just to take one row norm. The value is fixed the moment the adapter is saved, so save it.save_lora_stepis the only path that writes a.safetensorsfor a run — the training wrapper's ownexport_lora_safetensorshas no callers anywhere in either repo — so wiring it here is what actually gets norms into the files runs produce.Notes
save_lora_safetensorsacceptsmodel=get it. Passing the kwarg blind would raiseTypeErroron an older stable-audio-tools and lose the checkpoint at the one moment it cannot be recomputed.norm_basedoes not usemodel_config["base_model"]. No config the dashboard writes today sets that key (0 of 100+ existing_model.jsonfiles), so the label would be empty on every run. It falls back to the streamed checkpoint's filename. Worth recording rather than inferring: base and arc weights reconstruct each other's norms closely enough that a wrong pairing raises nothing.Verified by two real training runs
Short
sa3-mediumruns on a small local dataset, one with the duration embedder LoRA'd and one with it excluded:36/36 assertions: one norm per dora layer, counts matching the metadata, norms finite and positive, baked keys fp32 while weights stay fp16, and the only layer differing between the two runs is the duration embedder.
Cross-checked against the offline baker:
bake_dora.pyrun on the same adapter reproduces these norms bit-identically at matched precision, so a retrofitted file and a freshly trained one are interchangeable.