Skip to content

tflite: ship sm-music + sm-sfx as w8a8 rung ladders (default w8a8) - #118

Merged
Cortexelus merged 1 commit into
mainfrom
tflite-small-dit-w8a8-rungs
Oct 10, 2026
Merged

Cortexelus merged 1 commit into
mainfrom
tflite-small-dit-w8a8-rungs

Conversation

@Cortexelus

Copy link
Copy Markdown
Collaborator

Applies the cache-safe rung technique (medium #112/#114) to both small DiTs, and flips their default to w8a8 (you've ear-approved — board at the swap). Pairs with the HF swap moving the old small varlen tiers to legacy/.

What changes

before after
sm-music / sm-sfx DiT varlen fp32/w16a32/w8a32/w8a8-dyn rung w8a8 (default) / fp32
small install bundle dit_fp32 (varlen) + SAME-S, ~2.0 GB dit_w8a8 (rung) + SAME-S, ~0.7 GB
peak RAM @ 2 min ~3.8 GB (varlen, quadratic) ~0.8 GB (rung, flat) — 4.5× less

The old varlen tiers retire to tflite/sa3-sm-*/legacy/ on HF. fp32 (rung) stays for reference/LoRA; --lora auto-selects it.

Implementation

  • 8-rung ladder capped at the 2-min max (valid_T_lat(120)=1292): 8,48,96,192,416,704,1056,1292.
  • export_dit.py is now family-aware (SA3_DIT_FAMILY) and derives GCE_OUT from the config (6·embed_dim = 6144 small / 9216 medium); rung_dit.py reads the gc width from the model, so one runtime handles both. Medium's build is untouched (default family).
  • New driver build/build_dit_small.sh (the medium build_dit.sh is unchanged).
  • The PRECISIONS union is now {fp32, w8a8} across all families — docs, gradio, and CLI help updated; the retired w16a32/w8a32/w8a8-dyn are no longer offered.

Verified

  • Cache-safe = 999 dB both families (the gce-outside fix prevents the int8 weight-cache corruption for small too).
  • Per-forward vs fp32: sm-music 23.3 dB / cos 0.9977, sm-sfx 25.3 dB / cos 0.9985.
  • All rungs tflite==torch (max|d| ~1e-3), gcond ~1e-6.
  • Ear-approved A/B (fp32 / w8a8 / old w8a8-dyn); sm-sfx clean everywhere, sm-music 3/4 prompts tight with one chaotic-sampler divergence (valid, ear-passed).

🤖 Generated with Claude Code

Apply the from-repo cache-safe rung technique (medium PR #112/#114) to both
small DiTs. 8-rung ladder capped at the 2-min max (valid_T_lat(120)=1292):
8,48,96,192,416,704,1056,1292. GCE_OUT is derived from the config
(6*embed_dim = 6144 for the small DiTs vs 9216 medium); rung_dit.py reads the
gc width from the model, so one runtime handles both.

Defaults flip to w8a8 for every family (CJ's call 2026-10-10, ear-approved):
- DEFAULT_DIT_PRECISION[sm-*] fp32 -> w8a8; DIT_PRECISIONS_BY_FAMILY[sm-*] ->
  ("fp32","w8a8"), retiring the varlen w16a32/w8a32/w8a8-dyn -> legacy/
- install bundle for small downloads dit_w8a8 + SAME-S (~0.7 GB, was ~2.0 GB)
- --lora still auto-falls-back to fp32 (the int8 w8a8 rung can't be merged)

Verified: cache-safe 999 dB (gce-outside holds for small), per-forward vs fp32
sm-music 23.3 dB/cos 0.9977, sm-sfx 25.3 dB/cos 0.9985; peak RAM flat ~0.8 GB
vs the old varlen's 3.8 GB at 2 min (4.5x less). Ear-approved A/B.

New build driver build/build_dit_small.sh (leaves medium build_dit.sh
untouched). The PRECISIONS union is now {fp32, w8a8} across all families;
docs/gradio/CLI help updated accordingly. Pairs with the HF swap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Cortexelus
Cortexelus merged commit 98fb2a9 into main Oct 10, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant