From 02e77e2276349b44d5e03d0992d9026e7cab1c30 Mon Sep 17 00:00:00 2001 From: Cortexelus Date: Thu, 3 Sep 2026 23:43:57 +0000 Subject: [PATCH 1/2] lora: bake DoRA row norms at save time, and add the offline baker MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A DoRA weight is mag ⊙ (W0 + s·δ)/‖W0 + s·δ‖_row, so reconstructing it needs the base weights purely to recompute that denominator. Any consumer that only wants the adapter — an exporter, a converter, a merge tool — has to load a multi-GB base checkpoint first. Baking the norm into the adapter removes that. model.py: the per-adapter-type delta was computed inline in each forward, so it could not be called outside a forward pass. Factored into _delta() (unscaled, 2-D, dropout-free — a baked norm must be deterministic) with _delta_dropout() keeping the training-time behaviour the forwards need. baked_vnorm_row(layer) returns ‖W0 + scaling·δ‖ along dim 1, in fp32, flattening to 2-D first so Conv1d/Conv2d behave. It uses self.scaling alone with strength implicitly 1, since the runtime convention is unit strength — folding in lora_strength would bake whatever the slider happened to be set to. The layer is a parameter rather than read off self: a parametrization keeps no back-reference to its module, and adding one solely to bake a norm would create a cycle per adapter. utils.py: get_dora_norm_state_dict(model) emits {layer}.parametrizations.weight.{i}.baked_vnorm_row, matching the layout bake_dora.py writes so the two paths are interchangeable. It walks named_modules rather than get_lora_layers, which yields only (name, p) and not the module the original weight lives on. Only dora-* adapters whose runtime norm is along dim 1 are baked: dora-cols normalises along dim 0 so a row norm would be silently wrong, plain lora has no magnitude, and bora's column magnitude applies to the row-scaled intermediate so its norm is not a function of base and delta alone. The key is deliberately NOT added to name_is_lora: that predicate also feeds get_lora_params, i.e. optimizer construction, so whitelisting a buffer there would present it as a trainable parameter. It is merged at the save site instead, via an optional model= argument that accepts a sequence — baking from a parent wrapper would prefix keys ("model.", "conditioner.") and stop matching the adapter tensors. Two things that are easy to get wrong and are handled: the norm is exempt from the blanket .half() cast, since c = mag/vn scales every DoRA weight and fp16 storage of the denominator costs a worst case of ~4.9e-4 relative while fp32 costs ~3.5 MB across a full adapter set; and norm_base records which base checkpoint the norms were computed against, because sa3-medium's base and arc variants reconstruct each other's norms only to ~2.2e-3 and an unlabelled norm makes that mismatch undetectable at load time. The key is optional by construction — loaders that do not know about it are unaffected, and adapters trained before this keep working by computing the norm from base weights as before. bake_dora.py is the offline path for those, so it is a fallback rather than a mandatory pre-step. It takes the base weights by explicit --base-weights path (and --svd-bases for dora-*-xs, whose frozen bases are never recomputed — per-layer SVD on the DiT's larger matrices is minutes of work and numerically unstable), preserves each adapter's own lora_index rather than assuming 0, and resolves the base-weight key by trying the known checkpoint prefixes then a unique suffix match, since how deeply a checkpoint wraps the DiT varies by model. BAKED_VNORM_KEY is defined once, in utils.py, and re-exported from the package: consumers need to recognise a baked key without hardcoding "baked_vnorm_row", and the offline baker imports the same constant rather than restating the literal -- the two paths only stay interchangeable if they cannot disagree about the key. That relative import means bake_dora runs as `python -m ...`, so it carries no shebang. Verified on a rank-16 dora-rows adapter: 169 norms baked in 0.16 s; the baked value matches an independent recompute exactly (0.0 rel); the DoRA invariant that recovered row norms equal |magnitude| holds to 1.2e-07; baked keys save as fp32 while every other tensor stays fp16; and all 169 keys line up with an existing adapter tensor (no prefix drift). The offline baker reproduces the save-time norms bit-identically when both bake against the same-precision W0. --- stable_audio_3/models/lora/__init__.py | 6 +- stable_audio_3/models/lora/bake_dora.py | 243 ++++++++++++++++++++++++ stable_audio_3/models/lora/model.py | 51 ++++- stable_audio_3/models/lora/utils.py | 74 +++++++- stable_audio_3/training/diffusion.py | 5 +- 5 files changed, 368 insertions(+), 11 deletions(-) create mode 100644 stable_audio_3/models/lora/bake_dora.py diff --git a/stable_audio_3/models/lora/__init__.py b/stable_audio_3/models/lora/__init__.py index a5841fac..9fe404bf 100644 --- a/stable_audio_3/models/lora/__init__.py +++ b/stable_audio_3/models/lora/__init__.py @@ -14,6 +14,7 @@ load_lora_checkpoint, load_multiple_lora, name_is_lora, + get_dora_norm_state_dict, remap_lora_state_dict, save_lora_safetensors, select_lora, @@ -23,6 +24,9 @@ cast_base_to_precision, detect_dora_variant, prepare_dora_state_dict, - resolve_adapter_type + resolve_adapter_type, + # The suffix identifying a baked DoRA row norm. Exported so consumers can + # recognise those keys without hardcoding the string. + BAKED_VNORM_KEY, ) from .loader import load_and_apply_loras \ No newline at end of file diff --git a/stable_audio_3/models/lora/bake_dora.py b/stable_audio_3/models/lora/bake_dora.py new file mode 100644 index 00000000..8a6c5052 --- /dev/null +++ b/stable_audio_3/models/lora/bake_dora.py @@ -0,0 +1,243 @@ +"""Bake the strength-independent fold quantities into a DoRA / -xs adapter. + +Why: the runtime fold needs ``c = magnitude / ||W0 + s*LR||_row``. Computing that denominator +means loading the full base weights purely to take a row norm -- for a deployment runtime that +already carries its own copy of the weights, that is a second, redundant, double-precision copy. +Baking the norms into the adapter removes it entirely. + +Training saves these automatically (``save_lora_safetensors(..., model=...)``). This script is +the retrofit path for adapters trained before that existed, and produces the identical key and +metadata layout: + +``.parametrizations.weight..baked_vnorm_row`` (float32) + a ``__metadata__`` block. + +SVD IS NEVER COMPUTED. ``-xs`` bases are sliced from the frozen ``svd_bases.pt`` that the +adapter was trained against; recomputing them per layer is minutes of work and numerically +unstable on the DiT's larger matrices. + +Runs as a module (it imports from its own package): + + python -m stable_audio_3.models.lora.bake_dora \ + --base-weights /path/to/model.safetensors [--out PATH] [--force] + python -m stable_audio_3.models.lora.bake_dora --check +""" +from __future__ import annotations +import argparse, hashlib, json, sys, time +from pathlib import Path + +from .utils import BAKED_VNORM_KEY as BAKED # one definition of the key, shared with the save path + +PSUF = ".parametrizations.weight." + + +def is_baked(path) -> bool: + from safetensors import safe_open + with safe_open(str(path), framework="numpy") as f: + return any(BAKED in k for k in f.keys()) + + +def baked_path(path) -> Path: + p = Path(path) + return p if p.name.endswith(".normbaked.safetensors") else \ + p.parent / (p.stem + ".normbaked.safetensors") + + +def parse_adapter(path): + """``(adapter_type, scaling, layers)``; ``layers`` maps ``(layer_name, lora_index)`` to + its param dict. Mirrors the grouping the loader does, from the file alone.""" + from safetensors import safe_open + with safe_open(str(path), framework="numpy") as f: + tensors = {k: f.get_tensor(k) for k in f.keys()} + meta = dict(f.metadata() or {}) + layers = {} + for k, v in tensors.items(): + if PSUF not in k: + continue + layer, _, rest = k.partition(PSUF) + idx, _, param = rest.partition(".") + if not idx.isdigit(): + continue + layers.setdefault((layer, int(idx)), {})[param] = v + if not layers: + raise KeyError(f"{Path(path).name!r}: no SA3 LoRA parametrization keys") + cfg = json.loads(meta.get("lora_config", "{}") or "{}") + rank = int(cfg.get("rank") or _infer_rank(layers)) + alpha = float(cfg.get("alpha", rank)) + atype = cfg.get("adapter_type", "lora") + atype = "dora-rows" if atype == "dora" else atype # legacy alias, mirrors resolve_adapter_type + return atype, alpha / rank, layers, tensors, meta + + +def _infer_rank(layers): + for p in layers.values(): + if "lora_A" in p: + return p["lora_A"].shape[0] + if "M_xs" in p: + return p["M_xs"].shape[0] + raise KeyError("cannot infer LoRA rank (no lora_A / M_xs tensors)") + + +def _find_base_key(lid, wkeys, wpath): + """The adapter stores layer ids relative to the sub-module it was applied to; a checkpoint + wraps them in a prefix whose depth varies by model (``model.``, ``model.model.``, and + ``conditioner.`` for the non-DiT layers adapters touch, e.g. + conditioners.seconds_total.embedder.embedding.1). Try the known prefixes, then fall back + to a unique suffix match so an unforeseen wrapper does not silently fail to bake. + """ + for pre in ("", "model.", "model.model.", "conditioner.", "model.conditioner."): + k = f"{pre}{lid}.weight" + if k in wkeys: + return k + hits = [k for k in wkeys if k.endswith(f".{lid}.weight")] + if len(hits) == 1: + return hits[0] + if len(hits) > 1: + raise KeyError(f"{lid} is ambiguous in {wpath.name}: {sorted(hits)[:4]}") + raise KeyError(f"no base weight for {lid} in {wpath.name}") + + +def bake(adapter, base_weights, out=None, force=False, quiet=False, + svd_bases=None, norm_source="base", model="stable-audio-3"): + import numpy as np, torch + from safetensors import safe_open + from safetensors.numpy import save_file + + adapter, wpath = Path(adapter), Path(base_weights) + out = Path(out) if out else baked_path(adapter) + if out.exists() and not force: + if not quiet: print(f" already baked: {out.name} (--force to redo)") + return out + + atype, scaling, layers, tensors, meta = parse_adapter(adapter) + is_xs = atype.endswith("-xs") + stem = atype[:-3] if is_xs else atype + if not quiet: + print(f" {adapter.name}\n type={atype} scaling={scaling:.4f} layers={len(layers)} " + f"norm_source={norm_source}") + if stem.startswith("lora"): + if not quiet: print(" plain lora -- fold is already trivial, nothing to bake") + return None + if is_xs and not svd_bases: + raise ValueError(f"{atype} needs --svd-bases (the frozen bases it was trained against)") + + dev = "cuda" if torch.cuda.is_available() else "cpu" + bases = torch.load(svd_bases, map_location="cpu", weights_only=True, mmap=True) if is_xs else None + wf = safe_open(str(wpath), framework="pt") + wkeys = set(wf.keys()) + + n, worst, nneg = 0, 0.0, 0 + for (lid, idx), p in layers.items(): + wk = _find_base_key(lid, wkeys, wpath) + W0 = wf.get_tensor(wk).to(dev, torch.float32) + W0 = W0.reshape(W0.shape[0], -1) + if is_xs: + bk = next((k for k in bases if k.endswith(lid.split("model.", 1)[-1] + ".weight") + or k.endswith(lid + ".weight")), None) + if bk is None: + raise KeyError(f"no frozen SVD basis for {lid}") + r = p["M_xs"].shape[0] + U = bases[bk]["U"][:, :r].to(dev, torch.float32) + V = bases[bk]["V"][:, :r].to(dev, torch.float32) + LR = U @ torch.as_tensor(p["M_xs"], device=dev, dtype=torch.float32) @ V.T + else: + LR = (torch.as_tensor(p["lora_B"], device=dev, dtype=torch.float32) + @ torch.as_tensor(p["lora_A"], device=dev, dtype=torch.float32)) + vn = torch.linalg.norm(W0 + scaling * LR, dim=1) # ||W0 + s.LR||_row + # Self-check on a real invariant: the DoRA weight is W = mag ⊙ (W0+s.LR)/vn , so its + # row norms must come back as `magnitude`. + mag = torch.as_tensor(np.squeeze(p["magnitude"]).astype(np.float32), device=dev) + c = mag / (vn + 1e-12) + rows = torch.linalg.norm(c.unsqueeze(1) * (W0 + scaling * LR), dim=1) + # `magnitude` is a free nn.Parameter initialised to ||W0||_row, so training can drive + # rows NEGATIVE. The DoRA weight is c·V with c signed, so the recovered row norm is + # |magnitude|, not magnitude -- comparing against the signed value reports a rel error + # of exactly 2.0 for every negative row and hides real errors behind it. + worst = max(worst, float(((rows - mag.abs()) / mag.abs().clamp_min(1e-6)).abs().max())) + nneg += int((mag < 0).sum()) + tensors[f"{lid}{PSUF}{idx}.{BAKED}"] = vn.cpu().numpy().astype(np.float32) + n += 1 + + cfg = json.loads(meta.get("lora_config", "{}") or "{}") + cfg["baked_norms"] = n + cfg["norm_base"] = f"{model}-{norm_source}" + meta["lora_config"] = json.dumps(cfg) + meta.update({ + "base_model": f"{model}-{norm_source}", "norm_source": norm_source, + "norm_base_sha": hashlib.sha256(wpath.name.encode()).hexdigest()[:32], # recorded, not enforced + "baked_variant": atype, "baked_norms": str(n), + "baked_xs_svd": ("external:" + Path(svd_bases).name) if is_xs else "none", + "baked_by": "bake_dora.py", + "baked_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + }) + out.parent.mkdir(parents=True, exist_ok=True) + save_file(tensors, str(out), metadata=meta) + if not quiet: + print(f" → {out.name} (+{n} baked norms, {len(tensors)} tensors, " + f"{out.stat().st_size/1e6:.1f} MB)") + print(f" self-check ||mag⊙(W0+s·LR)/vn||_row vs |magnitude|: max rel {worst:.2e}" + f" ({nneg} negative-magnitude rows)") + return out + + +def ensure_baked(adapter, base_weights=None, interactive=True, quiet=False, **kw): + """Runtime hook: refuse an unbaked DoRA, offer to bake it, return the path to load. + + Never silently falls back to loading the base weights -- that fallback is precisely what + hides the redundant weight copy behind a working-looking load. + """ + adapter = Path(adapter) + if is_baked(adapter): + return adapter + cand = baked_path(adapter) + if cand.exists() and is_baked(cand): + return cand + from safetensors import safe_open + with safe_open(str(adapter), framework="numpy") as f: + md = f.metadata() or {} + cfg = json.loads(md.get("lora_config", "{}") or "{}") + if (cfg.get("adapter_type", "lora")).startswith("lora"): + return adapter # plain lora needs nothing + if not base_weights: + raise RuntimeError(f"{adapter.name} is unbaked and no --base-weights was given") + msg = (f"{adapter.name} is a {cfg.get('adapter_type')} adapter with no baked row norms.\n" + f" Without them the fold must load {Path(base_weights).name} to take a row norm.\n" + f" Bake now? [Y/n] ") + if not interactive or not sys.stdin.isatty(): + raise RuntimeError(f"{adapter.name} is unbaked. Run: python -m " + f"stable_audio_3.models.lora.bake_dora {adapter} " + f"--base-weights {base_weights}") + if (input(msg).strip().lower() or "y") not in ("y", "yes"): + raise RuntimeError("declined; adapter cannot be loaded without baked norms") + return bake(adapter, base_weights, quiet=quiet, **kw) + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("adapter", nargs="+") + ap.add_argument("--base-weights", default=None, + help="safetensors holding the W0 the adapter was trained against") + ap.add_argument("--svd-bases", default=None, help="frozen svd_bases.pt (dora-*-xs only)") + # A baked norm is only meaningful against the W0 the runtime actually carries -- base and + # arc reconstruct each other's norms at rel ~2e-3 with nothing raising -- so this label is + # recorded in the metadata to make a wrong pairing visible after the fact. + ap.add_argument("--norm-source", default="base", help="label recorded in the metadata") + ap.add_argument("--model", default="stable-audio-3") + ap.add_argument("--out", default=None) + ap.add_argument("--force", action="store_true") + ap.add_argument("--check", action="store_true", help="exit 1 if any adapter is unbaked") + a = ap.parse_args() + if a.check: + bad = [p for p in a.adapter if not is_baked(p) and not baked_path(p).exists()] + for p in bad: print(f" UNBAKED: {p}") + return 1 if bad else 0 + if not a.base_weights: + ap.error("--base-weights is required to bake") + for p in a.adapter: + bake(p, a.base_weights, a.out, a.force, svd_bases=a.svd_bases, + norm_source=a.norm_source, model=a.model) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/stable_audio_3/models/lora/model.py b/stable_audio_3/models/lora/model.py index b53f896d..385de176 100644 --- a/stable_audio_3/models/lora/model.py +++ b/stable_audio_3/models/lora/model.py @@ -197,9 +197,47 @@ def _blend_magnitude(self, mag, base_norm): if s == 1.0: return mag return base_norm + s * (mag - base_norm) + def _delta_dropout(self): + """_delta() with training-time dropout applied — used by the forwards.""" + return torch.matmul(*self.swap((self.lora_B, self.dropout_fn(self.lora_A)))) + + def _delta(self): + """The unscaled low-rank update, as 2-D (fan_out, prod(rest)). + + Factored out of the forward paths so the norm it feeds can be computed + at save time, outside any forward pass. Deliberately skips dropout: a + baked norm must be deterministic, not a sample from the training-time + dropout distribution. + """ + if self.adapter_type.endswith("-xs"): + return self.U @ self.M_xs.to(self.U.dtype) @ self.V.T + return torch.matmul(*self.swap((self.lora_B, self.lora_A))) + + def baked_vnorm_row(self, layer): + """Row norms of the adapted direction at unit strength: ||W0 + s.d||_row. + + This is the denominator every DoRA forward divides by, and the only part + that needs the base weights. Baking it into the checkpoint lets a loader + reconstruct the adapted weight without loading a multi-GB base model + first. + + Uses self.scaling alone, with strength forced to 1: the runtime + convention is vn = ||W0 + (alpha/rank).delta||_row at unit strength, so + folding in lora_strength would bake in whatever the slider happened to + be set to. Flattens to 2-D first so Conv1d/Conv2d behave, for the same + reason the forward paths work in 2-D. + + `layer` is passed in rather than read from self: a parametrization keeps + no back-reference to the module it is registered on, and adding one just + to bake a norm would create a reference cycle for every adapter. + """ + W0 = self._get_original_weight(layer).detach() + W0_2d = W0.reshape(W0.shape[0], -1).to(torch.float32) + delta = self._delta().detach().to(torch.float32) + return torch.linalg.norm(W0_2d + self.scaling * delta, dim=1) def lora_forward(self, W): - delta = torch.matmul(*self.swap((self.lora_B, self.dropout_fn(self.lora_A)))).view(W.shape) + delta = self._delta_dropout().view(W.shape) delta = self.scaling * self.lora_strength * delta return (W + delta.to(W.dtype)) @@ -211,7 +249,7 @@ def dora_forward(self, W): orig_shape = W.shape W_2d = W.view(W.shape[0], -1).to(self.lora_A.dtype) # low-rank update on the *direction* - delta = torch.matmul(*self.swap((self.lora_B, self.dropout_fn(self.lora_A)))) + delta = self._delta_dropout() V = W_2d + self.scaling * self.lora_strength * delta # normalize along _norm_dim, then scale by per-element magnitude V_hat = V / (V.norm(dim=self._norm_dim, keepdim=True) + 1e-12) @@ -226,7 +264,7 @@ def bora_forward(self, W): return W orig_shape = W.shape W_2d = W.view(W.shape[0], -1).to(self.lora_A.dtype) - delta = torch.matmul(*self.swap((self.lora_B, self.dropout_fn(self.lora_A)))) + delta = self._delta_dropout() V = W_2d + self.scaling * self.lora_strength * delta # Row-normalize, then scale by row magnitudes. Both magnitudes # interpolate toward the base weight's own norms so strength -> 0 @@ -243,8 +281,7 @@ def bora_forward(self, W): return (H_c * mag_c).view(orig_shape).to(W.dtype) def lora_xs_forward(self, W): - delta = self.U @ self.M_xs.to(self.U.dtype) @ self.V.T - delta = delta.view(W.shape) + delta = self._delta().view(W.shape) return (W + (self.scaling * self.lora_strength * delta).to(W.dtype)) def dora_xs_forward(self, W): @@ -252,7 +289,7 @@ def dora_xs_forward(self, W): return W orig_shape = W.shape W_2d = W.view(W.shape[0], -1).to(self.U.dtype) - delta = self.U @ self.M_xs.to(self.U.dtype) @ self.V.T + delta = self._delta() V = W_2d + self.scaling * self.lora_strength * delta V_hat = V / (V.norm(dim=self._norm_dim, keepdim=True) + 1e-12) mag = self._blend_magnitude( @@ -266,7 +303,7 @@ def bora_xs_forward(self, W): return W orig_shape = W.shape W_2d = W.view(W.shape[0], -1).to(self.U.dtype) - delta = self.U @ self.M_xs.to(self.U.dtype) @ self.V.T + delta = self._delta() V = W_2d + self.scaling * self.lora_strength * delta V_r = V / (V.norm(dim=1, keepdim=True) + 1e-12) intermediate = self._blend_magnitude( diff --git a/stable_audio_3/models/lora/utils.py b/stable_audio_3/models/lora/utils.py index b3c379a1..946a49a6 100644 --- a/stable_audio_3/models/lora/utils.py +++ b/stable_audio_3/models/lora/utils.py @@ -198,7 +198,46 @@ def resolve_adapter_type(adapter_type, state_dict=None): # ------------------- helper functions for safetensors LoRA checkpoints ------------------- -def save_lora_safetensors(state_dict, lora_config, path): +BAKED_VNORM_KEY = "baked_vnorm_row" + + +def get_dora_norm_state_dict(model): + """Baked row norms for every dora-* adapter on `model`. + + Emits {f"{name}.parametrizations.weight.{lora_index}.baked_vnorm_row": vn}, + matching the layout bake_dora.py writes so the two paths are + interchangeable. + + Only adapters whose runtime norm is taken along dim 1 are baked: that is the + convention the key name and the offline baker both assume, and a dora-cols + adapter normalises along dim 0, so a row norm would be silently wrong for + it. Plain `lora` has no magnitude and needs no norm. bora/bora-xs are + skipped too — their column magnitude is applied to the row-scaled + intermediate, so its norm is not a function of base weights and delta alone + and cannot be baked with a single tensor. + + Walks named_modules rather than get_lora_layers because computing the norm + needs the module's original weight, and get_lora_layers yields only + (name, parametrization). + """ + out = {} + for name, module in model.named_modules(): + plist = getattr(getattr(module, "parametrizations", None), "weight", None) + if plist is None: + continue + for p in plist: + if not isinstance(p, LoRAParametrization): + continue + if not p.adapter_type.startswith("dora"): + continue + if getattr(p, "_norm_dim", 1) != 1: + continue + key = f"{name}.parametrizations.weight.{p.lora_index}.{BAKED_VNORM_KEY}" + out[key] = p.baked_vnorm_row(module) + return out + + +def save_lora_safetensors(state_dict, lora_config, path, model=None, norm_base=None): """Save a LoRA checkpoint in safetensors format with config as metadata. The lora_config dict is JSON-serialized and stored under the metadata key @@ -208,9 +247,40 @@ def save_lora_safetensors(state_dict, lora_config, path): state_dict: Dict of LoRA tensors (from get_lora_state_dict). lora_config: Dict with keys like rank, alpha, adapter_type, include, exclude. path: Output file path (should end in .safetensors). + model: Optional model the adapters are attached to. When given, dora-* + row norms are baked in (see get_dora_norm_state_dict), which lets a + loader reconstruct adapted weights without first loading the base + model. Costs well under a second. + norm_base: Identity of the base checkpoint the norms were computed + against, recorded in the metadata. A baked norm is only exact for + that base — sa3-medium's base and arc variants reconstruct each + other's norms to about 2.2e-3 relative — so an unlabelled norm makes + that mismatch undetectable at load time. """ + if model is not None: + # Accept a sequence so a caller can bake from the same submodules it + # built the state dict from. Baking from a parent wrapper instead would + # prefix the keys ("model.", "conditioner.") and no longer line up with + # the adapter tensors. + submodels = model if isinstance(model, (list, tuple)) else [model] + baked = {} + for sub in submodels: + baked.update(get_dora_norm_state_dict(sub)) + if baked: + state_dict = {**state_dict, **baked} + lora_config = dict(lora_config) + lora_config["baked_norms"] = len(baked) + if norm_base: + lora_config["norm_base"] = str(norm_base) metadata = {"lora_config": json.dumps(lora_config)} - fp16_dict = {k: v.half() if v.is_floating_point() else v for k, v in state_dict.items()} + # The baked norm stays fp32: c = mag/vn scales every DoRA weight, and fp16 + # storage of the denominator costs a measured worst case of ~4.9e-4 relative. + # Keeping it fp32 costs ~3.5 MB across a full adapter set. + fp16_dict = { + k: v if k.endswith(BAKED_VNORM_KEY) + else (v.half() if v.is_floating_point() else v) + for k, v in state_dict.items() + } _st_save_file(fp16_dict, str(path), metadata=metadata) diff --git a/stable_audio_3/training/diffusion.py b/stable_audio_3/training/diffusion.py index a3e8063f..d144c627 100644 --- a/stable_audio_3/training/diffusion.py +++ b/stable_audio_3/training/diffusion.py @@ -626,7 +626,10 @@ def export_lora_safetensors(self, path): **get_lora_state_dict(self.diffusion.model), **get_lora_state_dict(self.diffusion.conditioner) } - save_lora_safetensors(state_dict, self.lora_config, path) + # Pass the model so dora-* row norms are baked in at save time; see + # get_dora_norm_state_dict. Cheap, and it spares loaders a base-model load. + save_lora_safetensors(state_dict, self.lora_config, path, + model=[self.diffusion.model, self.diffusion.conditioner]) def on_save_checkpoint(self, checkpoint): if self.lora_config is not None: From facad96579bdc649a7ae5a89ba3d4e0e66dc45a4 Mon Sep 17 00:00:00 2001 From: Cortexelus Date: Fri, 4 Sep 2026 00:01:26 +0000 Subject: [PATCH 2/2] lora: add a shell wrapper for the offline baker bake_dora.py imports from its own package, so it runs as `python -m stable_audio_3.models.lora.bake_dora` and not as a script path -- fine, but a mouthful for something you reach for occasionally. bake_dora.sh resolves the repo root from its own location, so it works from any working directory without the caller setting PYTHONPATH, and puts the checkout first on the path: a wrapper shipped inside the repo should bake with the repo's code rather than whatever copy is installed. Arguments pass straight through and the baker's exit status is preserved, so `--check` stays usable in a script. Verified the wrapper and the module invocation produce bit-identical output over all 676 tensors, differing only in the baked_at timestamp. --- stable_audio_3/models/lora/bake_dora.py | 6 +++++- stable_audio_3/models/lora/bake_dora.sh | 23 +++++++++++++++++++++++ 2 files changed, 28 insertions(+), 1 deletion(-) create mode 100755 stable_audio_3/models/lora/bake_dora.sh diff --git a/stable_audio_3/models/lora/bake_dora.py b/stable_audio_3/models/lora/bake_dora.py index 8a6c5052..20afeca2 100644 --- a/stable_audio_3/models/lora/bake_dora.py +++ b/stable_audio_3/models/lora/bake_dora.py @@ -15,7 +15,11 @@ adapter was trained against; recomputing them per layer is minutes of work and numerically unstable on the DiT's larger matrices. -Runs as a module (it imports from its own package): +Runs as a module (it imports from its own package), or through the bake_dora.sh +wrapper next to this file, which does that for you from any working directory: + + ./stable_audio_3/models/lora/bake_dora.sh \ + --base-weights /path/to/model.safetensors [--out PATH] [--force] python -m stable_audio_3.models.lora.bake_dora \ --base-weights /path/to/model.safetensors [--out PATH] [--force] diff --git a/stable_audio_3/models/lora/bake_dora.sh b/stable_audio_3/models/lora/bake_dora.sh new file mode 100755 index 00000000..2f6e7bb3 --- /dev/null +++ b/stable_audio_3/models/lora/bake_dora.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# Convenience wrapper for the offline DoRA norm baker. +# +# bake_dora.py imports from its own package, so it runs as +# `python -m stable_audio_3.models.lora.bake_dora` rather than as a script path. +# This does that for you, from any working directory: +# +# ./stable_audio_3/models/lora/bake_dora.sh adapter.safetensors \ +# --base-weights /path/to/model.safetensors +# ./stable_audio_3/models/lora/bake_dora.sh adapter.safetensors --check +# +# Every argument is passed through, and the exit status is the baker's own +# (--check returns 1 when an adapter is unbaked), so this is usable in a script. +# Set PYTHON to choose an interpreter; defaults to python3. +set -euo pipefail + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +repo_root="$(cd "$here/../../.." && pwd)" + +# Repo first on the path: a wrapper shipped inside the checkout should bake with +# the checkout's code, not with whatever copy happens to be installed. +PYTHONPATH="$repo_root${PYTHONPATH:+:$PYTHONPATH}" \ + exec "${PYTHON:-python3}" -m stable_audio_3.models.lora.bake_dora "$@"