Repository navigation
lora: let callers and the gradio Layer filter exclude conditioner adapters - #103
Conversation
TestedLayer filter → conditioner (the gradio path:
Filtering the conditioner leaves the DiT untouched (4.9081% deviation intact), clearing the box restores the layer exactly, and filter+strength compose: a filtered layer stays off at strength 0.5, and re-enabling it picks up the current strength rather than snapping to 1.0. Additivity — the claim that a runtime In all three cases the checkpoint's own 9 patterns stay honoured — 0 leaked layers. Replacing rather than merging would have silently re-enabled Exclusion is the adapter, not the conditioning. Verified separately that with Also fixed a one-off in the comment above the unpack (said 5 controls, the code reads 4). 37 assertions across the filter and additive groups, all passing. |
load_and_apply_loras honours the include/exclude lists saved in each checkpoint's metadata, but a caller had no way to skip a layer the training run did not. That leaves already-trained adapters unfixable at inference: if a pattern was missing from the defaults when they were trained, every load rebuilds the adapter and there is no override. Concretely, the SA3 conditioner's duration embedder (conditioners.seconds_total.embedder.embedding.1) is a Linear and the only adapter target outside the DiT, so it matched none of the historical exclude patterns and older LoRAs all carry an adapter on it. The gradio "Layer filter" cannot help: filter_lora_layers is called from DiffusionTransformer.forward, so it only ever sees adapters inside the DiT. Adds an `exclude` parameter, unioned with each checkpoint's own list rather than replacing it. Additive by design: it can only skip more layers, never resurrect one the training run deliberately left out, so it is safe to pass blindly. `include` is deliberately not exposed — unioning includes would *widen* the match set and could re-enable excluded layers. Verified against a rank-16 dora-rows checkpoint whose metadata excludes nine patterns but not seconds_total: no runtime exclude conditioner adapters=[seconds_total...] DiT=168 exclude=['seconds_total'] conditioner adapters=[] DiT=168 _merge_exclude is unit-checked for both-empty (returns None so add_lora keeps its unfiltered fast path), either side empty, and union with order preserved and duplicates dropped.
filter_lora_layers is called from DiffusionTransformer.forward, so `self` is the DiT and the filter only ever sees adapters inside the transformer. Adapters on model.conditioner — a sibling of model.model — are unreachable: typing their layer name in the box silently does nothing, which is misleading since the control presents itself as a whole-model layer filter. For SA3 that is the duration embedder (conditioners.seconds_total.embedder.embedding.1): a Linear, the only adapter target outside the DiT, and one that every LoRA trained with default excludes carries. So the single layer a user is most likely to want to switch off was the one layer the filter could not touch. Applies the same user-supplied filter to the conditioner in generate(). Once per generate is sufficient — unlike the DiT, which re-filters every sampling step inside forward, nothing re-enables conditioner adapters mid-run — and an empty filter re-enables them, which is the intended default. No new control and no change to the per-LoRA argument count, so callers passing four controls per LoRA keep working. The textbox gains a placeholder and info line saying it covers conditioner layers. Verified on a rank-16 dora-rows adapter, ||W - W0|| / ||W0|| for the conditioner layer against what the user types: '' 4.1628% (adapter active — default) 'seconds_total' 0.0000% (disabled — exactly base) 'SECONDS_TOTAL' 0.0000% (filters are lowercased) '.transformer.layers' 4.1628% (no match, correctly untouched) 'seconds_total, .ff.' 0.0000% (comma-separated OR) DiT adapter count is unchanged at 168 throughout.
The comment says 5, the unpack below reads 4, and the builder appends 4. Nothing was broken by it, but the next person to add a control will trust the comment.
d5827dd to
70bcd31
Compare
Problem
Two ways to skip a LoRA layer, and neither could reach adapters outside the DiT.
1. At load time,
load_and_apply_lorashonours theinclude/excludelists saved in each checkpoint's metadata, but a caller has no way to skip a layer the training run did not. Already-trained adapters are therefore unfixable at inference: if a pattern was missing from the defaults when they were trained, every load faithfully rebuilds that adapter.2. In the gradio UI,
filter_lora_layersis called fromDiffusionTransformer.forward, soselfis the DiT. Adapters onmodel.conditioner— a sibling ofmodel.model— are unreachable, so typing their layer name in the "Layer filter" box silently does nothing. Misleading, since the control presents itself as a whole-model filter.Concretely, for SA3 the affected layer is the duration embedder:
a
Linear(256, 768)and the only adapter target outside the DiT. It matched none of the historical exclude patterns, so LoRAs trained with default settings all carry an adapter on it — and it was the one layer neither mechanism could switch off.Changes
loader.py— anexcludeparameter, unioned with each checkpoint's own list rather than replacing it. Additive by design: it can only skip more layers, never resurrect one the training run deliberately left out, so it is safe to pass blindly.includeis deliberately not exposed, since unioning includes would widen the match set and could re-enable excluded layers.interface/diffusion_cond.py— the same user-suppliedlayer_filteris applied to the conditioner ingenerate(). Once per generate is sufficient: unlike the DiT, which re-filters every sampling step insideforward, nothing re-enables conditioner adapters mid-run. An empty filter re-enables them, which is the intended default.Verification
Loader, against a rank-16 adapter whose metadata excludes nine patterns but not
seconds_total:_merge_excludeis unit-checked for both-empty (returnsNonesoadd_lorakeeps its unfiltered fast path), either side empty, and union with order preserved and duplicates dropped.UI,
‖W − W0‖ / ‖W0‖for the conditioner layer against what the user types:''seconds_totalSECONDS_TOTAL.transformer.layersseconds_total, .ff.DiT adapter count stays 168 throughout.
Notes