Skip to content

lora: let callers and the gradio Layer filter exclude conditioner adapters - #103

Merged
Cortexelus merged 3 commits into
mainfrom
lora-load-exclude
Oct 11, 2026
Merged

Cortexelus merged 3 commits into
mainfrom
lora-load-exclude

Conversation

@Cortexelus

Copy link
Copy Markdown
Collaborator

Problem

Two ways to skip a LoRA layer, and neither could reach adapters outside the DiT.

1. At load time, load_and_apply_loras honours the include/exclude lists saved in each checkpoint's metadata, but a caller has no way to skip a layer the training run did not. Already-trained adapters are therefore unfixable at inference: if a pattern was missing from the defaults when they were trained, every load faithfully rebuilds that adapter.

2. In the gradio UI, filter_lora_layers is called from DiffusionTransformer.forward, so self is the DiT. Adapters on model.conditioner — a sibling of model.model — are unreachable, so typing their layer name in the "Layer filter" box silently does nothing. Misleading, since the control presents itself as a whole-model filter.

Concretely, for SA3 the affected layer is the duration embedder:

conditioners.seconds_total.embedder.embedding.1

a Linear(256, 768) and the only adapter target outside the DiT. It matched none of the historical exclude patterns, so LoRAs trained with default settings all carry an adapter on it — and it was the one layer neither mechanism could switch off.

Changes

loader.py — an exclude parameter, unioned with each checkpoint's own list rather than replacing it. Additive by design: it can only skip more layers, never resurrect one the training run deliberately left out, so it is safe to pass blindly. include is deliberately not exposed, since unioning includes would widen the match set and could re-enable excluded layers.

interface/diffusion_cond.py — the same user-supplied layer_filter is applied to the conditioner in generate(). Once per generate is sufficient: unlike the DiT, which re-filters every sampling step inside forward, nothing re-enables conditioner adapters mid-run. An empty filter re-enables them, which is the intended default.

Verification

Loader, against a rank-16 adapter whose metadata excludes nine patterns but not seconds_total:

no runtime exclude          conditioner adapters=[seconds_total...]  DiT=168
exclude=['seconds_total']   conditioner adapters=[]                 DiT=168

_merge_exclude is unit-checked for both-empty (returns None so add_lora keeps its unfiltered fast path), either side empty, and union with order preserved and duplicates dropped.

UI, ‖W − W0‖ / ‖W0‖ for the conditioner layer against what the user types:

typed effect
'' 4.1628% — active, default
seconds_total 0.0000% — exactly base
SECONDS_TOTAL 0.0000% — filters are lowercased
.transformer.layers 4.1628% — no match, correctly untouched
seconds_total, .ff. 0.0000% — comma-separated OR

DiT adapter count stays 168 throughout.

Notes

  • No new UI control and no change to the per-LoRA argument count, so callers passing four controls per LoRA keep working.
  • The textbox gains a placeholder and info line stating that it covers conditioner layers.
  • One asymmetry, deliberate: the DiT re-applies the filter every sampling step and so participates in the sigma-interval gating, while the conditioner is set once per generate. That is correct — conditioning is computed once, not per-sigma — but it means a conditioner adapter is not interval-gated the way DiT adapters are.

@Cortexelus

Cortexelus commented Sep 3, 2026 •

Copy link
Copy Markdown
Collaborator Author

Tested

Layer filter → conditioner (the gradio path: filter_lora_layers(conditioner, typed, lora_index=i)), 11 inputs against the live duration embedder:

typed duration embedder
seconds_total OFF
seconds / total OFF (substring)
Seconds_Total OFF (case-insensitive)
conditioners.seconds_total.embedder OFF
seconds_total, / ,seconds_total OFF (stray commas)
transformer, seconds_total OFF
`` (empty) / None ON
nonexistent_layer ON

Filtering the conditioner leaves the DiT untouched (4.9081% deviation intact), clearing the box restores the layer exactly, and filter+strength compose: a filtered layer stays off at strength 0.5, and re-enabling it picks up the current strength rather than snapping to 1.0.

Additivity — the claim that a runtime exclude adds to the checkpoint's own list rather than replacing it. _merge_exclude unit-tested for union order, dedup, None/empty handling, and non-mutation of the input; then on the real model, with a trained adapter's own 9-pattern exclude list:

exclude=None                      168 DiT + 1 conditioner adapter
exclude=["seconds_total"]         168 DiT + 0 conditioner adapters
exclude=[..., ".ff.ff.2"]         DiT count drops as well

In all three cases the checkpoint's own 9 patterns stay honoured — 0 leaked layers. Replacing rather than merging would have silently re-enabled project_in/project_out/etc.

Exclusion is the adapter, not the conditioning. Verified separately that with seconds_total excluded the embedder still runs and still discriminates duration: 30/95/190/380 s all produce distinct embeddings, nearby durations closer than distant ones, deterministic for a repeated duration.

Also fixed a one-off in the comment above the unpack (said 5 controls, the code reads 4).

37 assertions across the filter and additive groups, all passing.

load_and_apply_loras honours the include/exclude lists saved in each
checkpoint's metadata, but a caller had no way to skip a layer the training run
did not. That leaves already-trained adapters unfixable at inference: if a
pattern was missing from the defaults when they were trained, every load
rebuilds the adapter and there is no override.

Concretely, the SA3 conditioner's duration embedder
(conditioners.seconds_total.embedder.embedding.1) is a Linear and the only
adapter target outside the DiT, so it matched none of the historical exclude
patterns and older LoRAs all carry an adapter on it. The gradio "Layer filter"
cannot help: filter_lora_layers is called from DiffusionTransformer.forward, so
it only ever sees adapters inside the DiT.

Adds an `exclude` parameter, unioned with each checkpoint's own list rather than
replacing it. Additive by design: it can only skip more layers, never resurrect
one the training run deliberately left out, so it is safe to pass blindly.
`include` is deliberately not exposed — unioning includes would *widen* the
match set and could re-enable excluded layers.

Verified against a rank-16 dora-rows checkpoint whose metadata excludes nine
patterns but not seconds_total:

  no runtime exclude          conditioner adapters=[seconds_total...]  DiT=168
  exclude=['seconds_total']   conditioner adapters=[]                 DiT=168

_merge_exclude is unit-checked for both-empty (returns None so add_lora keeps
its unfiltered fast path), either side empty, and union with order preserved
and duplicates dropped.
filter_lora_layers is called from DiffusionTransformer.forward, so `self` is the
DiT and the filter only ever sees adapters inside the transformer. Adapters on
model.conditioner — a sibling of model.model — are unreachable: typing their
layer name in the box silently does nothing, which is misleading since the
control presents itself as a whole-model layer filter.

For SA3 that is the duration embedder
(conditioners.seconds_total.embedder.embedding.1): a Linear, the only adapter
target outside the DiT, and one that every LoRA trained with default excludes
carries. So the single layer a user is most likely to want to switch off was the
one layer the filter could not touch.

Applies the same user-supplied filter to the conditioner in generate(). Once per
generate is sufficient — unlike the DiT, which re-filters every sampling step
inside forward, nothing re-enables conditioner adapters mid-run — and an empty
filter re-enables them, which is the intended default.

No new control and no change to the per-LoRA argument count, so callers passing
four controls per LoRA keep working. The textbox gains a placeholder and info
line saying it covers conditioner layers.

Verified on a rank-16 dora-rows adapter, ||W - W0|| / ||W0|| for the conditioner
layer against what the user types:

  ''                       4.1628%   (adapter active — default)
  'seconds_total'          0.0000%   (disabled — exactly base)
  'SECONDS_TOTAL'          0.0000%   (filters are lowercased)
  '.transformer.layers'    4.1628%   (no match, correctly untouched)
  'seconds_total, .ff.'    0.0000%   (comma-separated OR)

DiT adapter count is unchanged at 168 throughout.
The comment says 5, the unpack below reads 4, and the builder appends 4. Nothing
was broken by it, but the next person to add a control will trust the comment.
@Cortexelus
Cortexelus merged commit 6e31f94 into main Oct 11, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant