Conversation
jsmucr
marked this pull request as draft
September 16, 2026 12:24
Member
|
Wouldn't this have been better as an issue for the moment? |
Records what the current tree does with Panasonic RW2 lens correction, identifies the regression introduced by 47c2237 (auto-selecting the embedded-metadata correction method for RW2, which is distortion-only and drops TCA), and outlines the follow-on work to decode the CA data present in Exif.PanasonicRaw.0x011b. Working document for the follow-on agent, not a permanent design doc.
Rigo's parseca.c four-checksum layout for 0x011b verified against the sample RW2 (all four match). Homeister's ExifTool forum 9366 post (retrieved via web.archive.org) supplies the interior structural map: 4-radial-zone model with N1..N4 at words [11],[4],[16],[17], on/off flag at [14], selector at 0x011a. The sample's ratios (1.0, 0.833, 0.667, 0.333) match the MFT-sensor pattern he reported. Coefficient roles within Homeister's three interior "parts" are still undecoded and remain the RE gap. Doc restructured to separate verified structure from remaining unknowns, prunes the earlier "two 16-word blocks" hypothesis that we now know is wrong, updates the image.h struct sketch to zone-based, and adds a shooting recipe for the RW2 corpus available on the developer's machine.
Log the outcome of the coefficient-fitting stage. The piecewise-linear 4-zone hypothesis with the 8 smooth words as knot heights and a single global scaling factor does not fit the measured CA, even with all knot permutations and a per-file gain: aggregate residual RMS 0.024 to 0.036 px against a null RMS of 0.037 px, and the per-file gains flip sign three to six times across the nine files with a 50x range in magnitude. Monomial-in-r and Catmull-Rom variants also fail. Measurement improvements captured: parabolic peak fit on the gradient profile with a peak-dominance filter, bilinear profile sampling and a same-feature integer-peak check, which lowers the per-bin half-vs-half RMS on 5 of 6 file/channel pairs on the three test files. The aggregated point cloud is 37,381 edges across 9 files, dumped to /tmp/rw2_tca/measurements.npz. Word[8]'s anti-correlation with the sign of measured R-G in 9/9 files still holds and is the strongest single-word signal in the tag, but does not extend to a coherent zone-height model with the other seven smooth words. The section is honest about the negative outcome: the 8-smooth-words piecewise-linear model with a global k is not the decode. Alternative hypotheses (three per-channel distortion polynomials, word[2] as a lens-family key rather than a coefficient) are listed as candidates for the next attempt, alongside the corpus additions that would resolve the ambiguity (in-camera CA-toggle pair, distortion-toggle pair, SILKYPIX Pro null-corrected reference).
GX80 mirror corpus (nine files across the same lens/focal set as the G9, at f/5.6) passes all four checksums on every file and shows the same structural layout: 0x011a=2, word[14]=256 flag, 4-zone model with identical 0.333/0.667/0.833/1.0 knot ratios, empty 0x0119 on the L 45-150, high-order flag bits on the PL 12-60, flag=1 on the primes. The 4 radii scale with body sensor size: GX80 N1=2888 vs G9 N1=3276, ratio 0.882, matching the ratio of half-diagonals within 0.5%. The ratios themselves are body-invariant. Word[8]'s anti-correlation with the sign of measured R-G radial displacement replicates cleanly on the GX80: 9 of 9 files, no exceptions. The two G9 files that were sign-ambiguous due to small measured amplitude come out unambiguous on the GX80. Removes the 'different body' item from 'What would resolve it' since that experiment is now done, replaces it with a note on the missing in-camera CA/distortion toggles (verified against DVQP1406ZA and by hand on both bodies).
Session 3 tried to decode 0x011b coefficients by measuring geometric distortion (much larger signal than CA) against the paired camera JPEG. Two hard findings: - The L 45-150 JPEGs are NOT distortion-corrected. Camera does not apply either 0x0119 (empty on this lens) or 0x011b to the in-camera JPEG. Residual 0.15 px RMS is registration noise, so fitting 0x011b against L 45-150 measurements is meaningless. - On files where 0x0119 is populated, its Rigo polynomial predicts distortion shape well (R^2 0.75-0.96) but the required scale factor swings from +0.90 to +5.24 to -0.95 within the same 12-60 zoom sweep. The 12-60's flag byte word[7] has high-order bits (0xF0, 0xB7, 0x8D at 12/25/60mm) that Rigo and 47c2237 both discard; primes all have 0x0001 flag and k close to 1. Strong hypothesis: the high byte of word[7] encodes a per-focal-length strength scaler. Implications: 47c2237's distortion output on Leica DG 12-60 is likely mis-scaled by factors of 2-5 as a latent bug on top of the Lensfun-TCA regression. Decoding 0x011b is blocked pending 0x0119's full decode.
Session 4 tested H1-H5 for a hidden per-focal-length strength scaler in word[7]'s high byte and finds all five hypotheses fail. Fresh k_only measurements are stable across bodies for the same tag data, but no simple function of word[7] (linear, subfield, LUT, whole int16) predicts the k pattern (best R^2 = 0.15). Nor does any of the untouched 0x0119 words (word[2, 3, 6, 9, 10, 13]) in single or two-way linear combinations. The wild k=+5.24 (25mm) and k=-0.95 (60mm) on the PL 12-60 that motivated the search are measurement artifacts of translation-only SIFT registration on files where the tag's polynomial predicts small amplitude (peak 2 px and 15 px). A joint fit of polynomial plus linear-in-r residual collapses k toward 1 on the eight strong-signal files (median 1.02) and shows a small (~0.5%) unmodeled radial dilation dominating the two anomalous files. Any model of the r-monotonic residual with a wrong-shape polynomial lands at a wild k. Practical consequence: session 3's implication (b) - that 47c2237's Panasonic distortion path may be silently mis-scaled - is retracted. The distortion math applies the tag's polynomial the way the tag intends. The default-method-selection regression covered elsewhere in this document is unchanged.
…0x011b words
Adobe DNG Converter emits a three-plane WarpRectilinear opcode on every
RW2 in the corpus, giving us Panasonic's own per-channel correction as
ground truth instead of pixel measurements. Fit D_R = R_plane - G_plane
and D_B = B_plane - G_plane against the smooth words in 0x011b.
D_R decodes cleanly: dropping w[2] (a per-lens family baseline), w[26]
and w[29] leaves six words that predict all four D_R.k_r{0..3} at
R^2_LOGO 0.97-0.99 under leave-one-group-out cross-validation, where
each held-out group is both bodies of a shared (lens, focal). D_B's
low orders (k_r0, k_r1) fit at R^2_LOGO 0.99; higher orders do not
generalize, because Adobe's B-plane coefficients differ 2-4x between
bodies at the same lens and focal even after LOGO training.
Section adds the ground-truth definition, sanity check against Rigo's
0x0119 polynomial (matches with a small per-file Adobe framing
constant), the LOGO scores of F1..F5 and reduced-word variants, the
6x4 coefficient matrix for D_R and the 6x2 low-order-only matrix for
D_B, and a stub of the follow-on darktable patch shape. No code
changes.
Session 6 runs the honest cross-corpus test session 5 asked for. 16 RW2 samples from DC-S5, DC-G9M2, DC-GH5 and DMC-GX8, sourced from raw.pixls.us, none of which was in the G9+GX80 training corpus. Structural findings (Rigo checksums, word[14] flag, Homeister radii) survive every third-body sample. Two structural corrections fall out: DC-S5 uses a full-frame-specific zone-radii pattern (1.0/6-7/4-7/2-7, not the MFT 1.0/5-6/4-6/2-6), and DC-G9M2 ships 0x011b without an 0x011a selector at all. Both matter for the darktable parse. The R-plane C_R decode reproduces the sign of the R-vs-G displacement on 69 of 78 signal-carrying bins across the four new bodies -- the same match rate the same pipeline gives on the training corpus. It over-predicts absolute pixel magnitude by ~10x on every body, including on the training corpus itself, so this gap is a property of Adobe's DNG opcode vs raw sensor CA and not a failure of C_R to generalize across bodies. Retested session 5's B-plane high-order gap: no single or pair of discrete words closes it. Adding four discrete words to the base six lifts D_B.k_r3 LOGO R^2 from 0.08 to 0.854 (marginal), but D_B.k_r2 tops out at 0.774. At 10 predictors on 18 files this is closer to overfit than a decode; the gap is still open.
Session 7 pairs each RW2 in a six-file subset (Sigma 16 and PL 12-60 at 12mm and 60mm, on G9 and GX80) with its camera JPEG, coarse-registers raw to JPEG with a partial affine, and measures R-G and B-G radial displacements in both with the parabolic sub-pixel fit already used against the raw. The camera JPEG carries essentially no CA (jpeg_RG within +/- 0.02 px on every signal bin), so the correction the camera applied equals the raw's own CA magnitude. Comparing that applied correction to the C_R prediction: - median-of-per-file-medians ratio = 11.48 - range 6.18 - 14.07 across six files (2.28x spread, cv = 30%) - per-body median: G9 = 13.29, GX80 = 8.05 - sign matches on 26 of 26 signal bins - direct read of Adobe's WarpRectilinear opcode (without the C_R fit) gives the same 4x - 30x ratios, confirming the gap is Adobe's opcode vs the raw, not a C_R decode artifact Outcome is the session brief's 'systematic mismatch (3-10x)' branch. A single global scale of 1/11.48 is right on the order but leaves per-file error at about 2x either way. Recommend not shipping C_R naively; understand Adobe's opcode magnitude first. Report goes in as a new subsection under 'Coefficient decode results'. No code changes.
Session 7 measured that Adobe's WarpRectilinear opcode over-corrects
the CA the camera JPEG actually applies by ~11x on a six-file subset.
Session 8 extends that measurement to the full 18-file corpus, then
asks whether the session-5 decode is structurally correct (same
coefficients, wrong scale) or whether it needs refitting against the
JPEG-derived applied correction.
Two refit shapes were tried against 207 (file, radial-bin) target
points:
- per-file poly4 fit then linear regression against words, mirroring
session 5. LOGO R^2 is negative on all eight polynomial
coefficients because the applied correction (0.01-0.15 px) is too
small to condition four polynomial degrees of freedom per file
against per-bin measurement noise.
- direct 24-DOF pixel-space fit on all bins at once. Train R^2 = 0.808
on R and 0.530 on B; under LOGO both go strongly negative (-6.11
and -0.99) and per-file RMS degrades to 0.38 px on the worst file.
Ridge does not help.
The scale factor generalises. Dividing session-5 C_R output by
K = 11.48 delivers per-file median RMS 0.028 px on R and 0.021 px on
B, worst-case 0.058 px and 0.034 px, with 96% of R and 75% of B
signal bins matching sign of the measured applied CA. Per-body K is
G9 10.4 / GX80 7.3, showing real body-dependent spread inside the
[7, 13] range, but no in-file signal keys on body cleanly. Ship
K = 11.48 constant.
Adds a JPEG-referenced decode section under 'Coefficient decode
results' and a JPEG-target patch formula block. Session 5's
Adobe-DNG-referenced matrices stay as historical reference with a
pointer to the session 8 K = 11.48 divisor.
Scripts at /tmp/rw2_tca/step19_{01..07}_*.py.
Static analysis of SILKYPIX Developer Studio 8 SE's SILKYPIX64.dll to
try to lift Panasonic's own algorithm for reading Exif.PanasonicRaw.0x011b.
Records what was recovered and what the analysis could not reach.
Recovered:
- the MSVC RTTI chain used to find IslZTiffExifPanasonic and
IslBRawReadPanasonic0 vtables, and the seven vtable slots that
Panasonic uniquely overrides,
- the class-field offset each Panasonic private-IFD tag maps to, from
the tag->stub table at RVA 0x17accc4. Tag 0x11b lives at
obj + 0x129a8,
- the five functions in .text that touch obj + 0x129a8 (via .pdata
RUNTIME_FUNCTION entries for accurate bounds), including the 88 KB
private-IFD parser at 0x4f3ca0..0x50950d,
- explicit no-hit results for the candidate CA constants (K = 11.48,
the Homeister zone radii {0.333, 0.667, 0.833, 1} in both MFT and
full-frame-S forms).
Not recovered:
- the exact word subset and polynomial form the parser applies. The
parser is 88 KB of MSVC-optimised code with the tag switch flattened
into an indirect-jump table, and lifting it further needs a
decompiler,
- the DefaultLensInfo.spd and DefaultParameters.spd container format,
which is a proprietary 'ISL Multi purpose file format'. Suspected
home of a per-lens attenuation that would explain the K factor,
but not confirmed.
Conclusion: session 8's decode is not refuted. Ship it as is. No
body-conditional scaling in the darktable patch; the per-body K spread
seen in session 7 probably lives in the .spd data, not in code.
Notes doc gets the understanding, not the decompiled listing. Nothing
from SILKYPIX enters the tree.
Session 9 stopped at the 88 KB private-IFD parser because linear disassembly could not lift its body. This session got Ghidra 12.1.3 headless working against SILKYPIX64.dll and used it to answer, for each of the five session-9 questions, what the binary is able to say. Two settled, two answered with strong negative evidence, one open. Session 8's decode still ships. New empirical findings: - The polynomial is not in the private-IFD parser. All nine touches of obj+0x129a8 in FUN_1804f3ca0 are reset/failure paths (memset the CA container to zero on error). - The CA container is 5512 bytes (0x1588), not the 64-byte payload we thought. It has sub-blocks and cached derived state, mapped partially from the class constructor at 0x4f07a0. - SILKYPIX does not compute csum mod 0xFFEF. The modulus appears three times in the whole DLL, none in Panasonic code. Either Rigo's checksum is not the validator SILKYPIX uses, or SILKYPIX trusts the tag unchecked. Recommendation: keep parsing the tag even if Rigo's checksum fails; log-only. - IslEISDevelopDemosaicPanaCA is a plugin, not a C++ class -- no RTTI type descriptor, no static factory table pairing class id 0x1216 with a constructor. The tag-0x11b decoder and polynomial evaluator live inside a factory-instantiated module whose entry point is not reachable from static class-hierarchy walks. - Property id 0xa052 = "ColorAberration" is the pipeline hook. Dispatched via FUN_18074e0f0 -> FUN_1810ceb30(..., 0xa052). That is the trailhead for a future dynamic-tracing session. The section captures which Ghidra scripts were written, which files were produced, what the reconciliation with session 8 amounts to (session 8 stays; per-body K spread stays unexplained), and honest bounds on what remained un-lifted.
Lifted the CA polyline builder at RVA 0x10ceb30 (FUN_1810ceb30), called from the property-key dispatcher with prop_id=0xa052. It produces a 9-knot IslZCnvPolyLine sampled by the render side, not a polynomial. Knots come from payload short indices 1..7 (radii) and 8..14 (values), each adjusted by seven int32 offsets fetched from property 0xa020; radii are normalised by half the raw-image diagonal and values by 1000. Boundary knots are (0, 1) and (2, knot[6].y). Verified against x86 disassembly. Session 8's six-word set overlaps SILKYPIX's use of words 8, 10 and 12 (as knot values 1, 3, 5), but SILKYPIX reads all of payload[1..14] and none past 14, so words 20/23/27 are not directly consumed here. The K = 11.48 factor is not a code constant, but its order of magnitude matches SILKYPIX's implicit value scale (short/1000). The polyline representation is not equivalent to a quartic, which explains session 8's LOGO refit failing. The raw-tag-to-property mapping (inside the 88 KB private-IFD parser) and the R/B split on the render side remain unlifted, so session 8's polynomial with K = 11.48 stays the shipping recommendation. Recommends: when both are lifted, switch the darktable path to a monotone spline evaluator on seven knots.
Session 11 had a shape-of-code right but the label wrong. The property-id table at RVA 0x1a117d0..0x1a11a70 in SILKYPIX64.dll maps 0xa050 -> Distortion, 0xa051 -> ColorAberration, 0xa052 -> Shading. Session 10 read it shifted by one entry and session 11 inherited the shift. The 9-knot polyline that session 11 called the CA descriptor is the shading descriptor: the dispatcher at 0x74f148/74f204/74f244 calls the builder for 0xa052/0xa053/0xa054 which are Shading and its two camera-correction variants, not CA. Session 12 also verified that 0xa051 (ColorAberration) never appears as an immediate anywhere in .text and its name string has no LEA references from code, so SILKYPIX 8 SE reaches Panasonic's CA correction through neither the property-id nor the property-name path -- if it reaches it at all. Property 0xa020, which session 11 called a per-lens offset table, is named 'Encryption Key' in that same property-id table; its origin was not lifted. The 58 KB FUN_1804f3ca0 was decompiled fully (raising DecompileOptions.setMaxPayloadMBytes past the 50 MB default that had silently truncated session 10), and contains no immediate for any of the CA/shading/distortion property ids, and no immediate for tag 0x11b. Its argument shape is (handle, struct*, small_int) and its body accesses fixed struct fields rather than iterating IFD entries, so session 10's identification of it as the private-IFD parser is not supported by the code. Plugin id 0x1216 = IslEISDevelopDemosaicPanaCA is confirmed via the plugin-name switch at RVA 0x547756, but there is no static edge from that id to a plugin factory. The shipping recommendation is unchanged: keep session 8's polynomial fit. Do not port session 11's spline; it would replace darktable's imperfect CA with a wrong-object-class shading correction.
Add a 'For the implementing agent' section right after the immediate workaround, containing the concrete recipe another agent needs to write the darktable patch: files to edit, struct extension, exif.cc parser with the four-checksum validator (log-only per session 12), CA evaluation in _init_coeffs_md_v2 including full C_R and C_B_lo matrices and K_JPEG divisor, sensor-format handling, auto-select policy, testing guidance, release-notes draft, and honest bounds on the fit. Add a 'Optional TODOs (post-shipping)' section before References, listing G9 firmware RE, Adobe Camera Raw RE, B-plane higher-order decode, third-body K validation, SPD file format work, dynamic tracing paths, and fit-robustness expansion. None block shipping. Update the TL;DR to reflect the state after 12 sessions: decode converged with LOGO R^2 0.97-0.99 on R, max RMS residual 0.06 px, cross- body validated on 6 Panasonic bodies. The older 'Where the code needs to change' section from session 2 remains below as an audit trail.
jsmucr
force-pushed
the
rw2-tca-compensation
branch
from
September 18, 2026 04:29
2fad937 to
2b79ef9
Compare
Since 47c2237, Panasonic RW2 files with a populated DistortionInfo default to embedded-metadata correction, but the Panasonic branch of _init_coeffs_md_v2 wrote the same radial multiplier into all three RGB channels, silently dropping the Lensfun-based transverse chromatic aberration correction that used to apply. Decode Exif.PanasonicRaw.0x011b alongside 0x0119. The 64-byte payload is stored as 32 signed int16 LE on dt_image_correction_data_t::panasonic; Rigo's four-checksum is validated and logged on mismatch but not gated on, since SILKYPIX itself does not enforce it and some newer bodies write valid coefficients without a matching 0x011a selector. Six words at indices 8, 10, 12, 20, 23 and 27 feed a fitted 4x6 matrix that reproduces Adobe DNG Converter's WarpRectilinear R plane on top of a global 11.48 divisor calibrating the output against the in-camera JPEG. B goes through a 2x6 matrix on its two lowest-order terms; the higher orders did not decode from the six-word set and are left at the G-plane value. Also mirror the tag to Exif.Image.0xf11b on TIFF/DNG round-trip, in the same TIFF private range as the 0xf119 alias 47c2237 added for the distortion tag. Decode corpus was 34 RW2s across six bodies (G9, GX80, S5, GH5, G9M2, GX8) with WarpRectilinear ground truth from Adobe DNG Converter and magnitude calibration on the paired G9 + GX80 subset with camera JPEGs. Worst per-file RMS residual against the JPEG-measured CA is 0.058 px on R and 0.034 px on B, both under one Bayer super-pixel. Sign matches the JPEG on 96% of R signal bins. See RW2_TCA_investigation.md for the decode audit trail.
The prior commit's K = 11.48 divisor from session 8 of RW2_TCA_investigation.md was fit against a raw-vs-JPEG edge-position measurement that filtered edges to those with all three integer R/G/B peaks agreeing within +/- 1 pixel. That filter by construction excludes every multi-pixel-shift edge, i.e. the exact edges the CA correction is meant to fix. The surviving sub-pixel-edge subset gave "applied correction" values of only 0.02-0.14 px, and fitting K to that subset produced a correction 10-15x too small for real strong-CA edges. Field test on P1366392.RW2 (G9 + Leica DG 12-60 @ 14mm, outside the training corpus) confirms the bug: the darktable render with K = 11.48 leaves obvious multi-pixel magenta fringing on every branch silhouette against sky, while the camera JPEG on the same frame has clean edges. |R-G| and |B-G| channel differences show bright edge halos throughout the darktable render and essentially none in the JPEG. Adobe DNG Converter's WarpRectilinear opcode is a workable ground truth: on P1366477 (PL 12-60 @ 12mm, training corpus) the opcode prescribes +0.77 px R shift at r=0.5 and +1.81 px B shift at the corner, and session 5's C_R evaluation reproduces those values within a few percent without any K divide. Setting K = 1 makes the shipped patch apply the DNG-derived magnitude directly. Re-rendering P1366392 at K = 1 clears the magenta fringing to near-JPEG parity. Rename _pana_K_JPEG to _pana_K since the name no longer reflects the role, and update the comment block to record why. Add session 13 to RW2_TCA_investigation.md documenting the finding, the audit trail and the residual (undecoded B-plane higher-order terms) that this does not close.
Session 13's commit refuted K = 11.48 and the framing that went with it: no unexplained per-body K spread, no missing firmware constant to recover K from, and no ambiguity between a real ~10x factor and a measurement artifact - the factor was a measurement artifact. Three Optional TODO items still framed themselves around the refuted premises: - TODO 1 (G9 firmware RE) claimed extracting the firmware would "resolve the origin of the K factor" and "whether the per-body K spread is algorithmic or a measurement artifact". Session 13 answered both. Drop those bullets; keep the item, refocused on the still-open B-plane higher-order coefficients (k_r2, k_r3) and the polynomial-vs-spline shape question. - TODO 4 (Third-body K validation) was framed as "if a third body's K lands close to 11.48, ship as-is with more confidence". K = 1 is now what we ship; the third-body test is still worth doing, but as validation that C_R generalises at unit magnitude, not as a measurement of a per-body K. - TODO 5 (SPD file format) mentioned "the ~10x factor and the per-body K spread might live there". Neither exists anymore. Downgrade the item to "could explain residual per-lens variance at K = 1", flagging its lower priority since Adobe DNG (which C_R matches) does not appear to consult such an overlay. Historical session narratives (5-12) are left as they were - they are the audit trail session 13 refutes, and rewriting them would break the document's own logic.
Removed session 7's ±1-integer-peak filter, redid the raw-CA measurement via 128x128 tile phase-correlation across all 18 corpus files, and attempted to refit C_R/C_B (4 rows each) against the resulting direct raw-side shifts. Findings: - The user-reported residual on P1366392 (Leica DG 12-60 @ 14mm, G9) is a shape problem, not a magnitude one. With only C_B_lo (2 rows, session 5), the shipped B polynomial can only produce ~0.07 px of correction at r = 0.95, where the raw actually carries +2.47 px of B shift. Session 13's K = 1 fix already handled magnitude for R; the missing piece is the B channel's k_r2 and k_r3. - Refitting C_R and C_B against direct raw measurements does not generalise. LOGO R^2 flips from 0.97-0.99 (Adobe DNG target, session 5) to negative on most coefficients. The direct-measurement targets are noisier than Adobe DNG's smoothed opcode values, and the ground truth still has 3-4x cross-body variance on B's higher orders on the same lens and focal length. - A hybrid using corpus-median Adobe D_B.k_r2/k_r3 as constants does essentially nothing, because the median is near zero (Adobe's values span negative to positive by near-equal amounts). Verdict: the 18-file / 9-lens-focal / 2-body corpus does not support recovering B's k_r2 and k_r3. Either a third body with paired JPEGs (option 3 in the Optional TODOs, samples being requested at the time of this session) or firmware RE (session 13 declared not tractable) would break through. No code change; C_R, C_B_lo and K = 1 stay shipped as-is. cacorrectrgb remains the practical user-side workaround for the residual.
Contributor
Author
|
@TurboGit Might be, but currently I only take it as my private investigation open to feedback. :-) In a proper issue I'd be expecting to be told to switch back to the lensfun preset. However... I may need some more RW2+JPEG pairs for this, so perhaps a forum post would be adequate. |
First pass on a third-body corpus using pre-production Panasonic DC-S5M2 samples from photographyblog. Four paired RW2+JPEG files covering the LUMIX S 85 f/1.8, S 20-60 and S 14-28 lenses across 14 mm to 85 mm focal length. Positive: - 0x011b tag is present on all 4 files with count = 64, and all four Rigo 2011 checksums pass. The checksum algorithm generalises to a new body family unchanged. - Radii ratios N2/N1 = 6/7, N3/N1 = 4/7, N4/N1 = 2/7 match session 6's DC-S5 full-frame pattern verbatim. The 4-zone structural model session 6 verified on the older DC-S5 applies to the newer DC-S5M2 without change. - Physical CA magnitudes measured by tile phase-correlation are on the same order as MFT bodies. B shift at r = 0.85 sits at ~1 px across all three lens/focal groups, matching the P1366392 residual. - rawpy + our measurement pipeline handle the 6000x4000 full-frame sensor without code changes (halfdiag is per-file). Negative: - Combining the 4 S5M2 files with the 18-file MFT corpus and refitting the six-word regression degrades B decode further at r = 0.85 (LOGO R^2 goes from +0.52 on MFT-only to -2.70 on the 22-file combo). Four S5M2 files across three lens-focal groups is under the parametrisation boundary for a 4-coefficient B polynomial. - Sweep of predictor sets (6, 8, 9, 27, 32 words) on the same 22-file corpus at r = 0.85 confirms: more predictors makes things worse. Session 5's [8, 10, 12, 20, 23, 27] remains the best six-word set. The bottleneck is data size, not features. No code change. C_R, C_B_lo and K = 1 continue shipping from session 13. The batch confirms option 3 is the right path but the sample size needs to be ~4x larger before the B higher-order decode has any chance.
RW2_TCA_investigation.md session 16 grows the corpus from the 18 MFT files of session 5 to 132 files across 14 bodies (5 full-frame: S5M2, S1M2, S1M2E, S1RM2, S9) and 50 unique lens/focal groups. The new C_R (24 doubles) and C_B (24 doubles, up from C_B_lo's 12) come from a direct raw-CA centroid fit on the combined corpus, not from Adobe DNG WarpRectilinear as session 5 did. Adobe DNG remains a useful reference for R; the B plane is where the DNG-based fit under- corrected, and where the raw-CA fit gains the most. With ~50 lens/focal groups constraining the polynomial, B's k_r2 and k_r3 -- which session 5 could not decode from six words on 18 files -- now decode cleanly under LOGO. B is shipped as four rows now instead of two; the Panasonic branch of _init_coeffs_md_v2 evaluates d_b = db_k[0] + db_k[1] r^2 + db_k[2] r^4 + db_k[3] r^6. K stays at 1.0. Function-space LOGO R^2 on the 132-file corpus (r = 0.30/0.50/0.70/ 0.85): R +0.72 / +0.70 / +0.57 / +0.14 B +0.59 / +0.65 / +0.52 / +0.39 Corpus-wide RMS residual at r = 0.85 drops from 0.79/1.20 px (shipped) to 0.66/0.52 px (new) for R/B -- better than 2x on B at the corner. Held-out on P1366392 (the user's reported strong-CA frame, PL 12-60 @ 14mm), the B residual at r = 0.85 drops from +1.31 px shipped to +0.70 px new, roughly 45% of the gap closed. On the rendered corner crop the mean |B-G| drops from 0.041 to 0.035 and the mean |R-G| drops from 0.054 to 0.038. No RELEASE_NOTES.md entry: the prior CA restoration entry for this cycle covers the feature, and this is a numeric refinement.
This reverts commit 3158eff (and its follow-up 2e04bf7). The refit's function-space LOGO R^2 at r = 0.85 (R = +0.14, B = +0.39) sounded positive but only 39% of the B corner variance is explained by the six-word predictor, and R at the corner is worse still. On the training corpus with strong CA that leaves the mean residual small - which the RMS table in session 16 flattered - but on files with genuinely weak CA the 4-row B polynomial extrapolates past LOGO's r-range with wrong-sign coefficients, so the correction introduces fringing where the shipped 2-row fit had left the pixel alone. Regression evidence: on P1366399 (Leica DG 12-60 @ 24 mm f/5.6) B at r = 0.95 is measured at +0.83 px; shipped predicted +0.03 (no visible action, correct); refit predicts -0.18 - the wrong sign, and applied as a correction it pushes B further from the measured position by 1.01 px. mean absolute R-G on the user-reported crop rises from 3558 to 4419 (+24 percent) when the refit correction is applied. Corpus-wide: 84 B and 88 R fit_ok files have some r in {0.70, 0.85, 0.95} where the refit is >= 0.15 px worse than shipped or crosses zero with predicted magnitude > 0.3 px. Full-frame bodies (only 17 LOGO groups across 44 files) are particularly affected. Fixing this properly needs a shrinkage prior (ridge toward the 2-row shipped fit or toward zero) or a monotonicity constraint on B(r), not simply more data. The K = 1 magnitude fix from 6c5bf9e stays.
…ual test - all dead ends)
Two developer hypotheses tested, both settled. The six predictor words are continuous coefficients, not per-copy identifiers: they move monotonically with focal length on every zoom with a focal sweep. But about 30 of 32 words change when only the body changes, and the predictors still move 100-500 counts between frames with lens, focal length and aperture fixed, so the payload responds to an input the fit cannot see. Panasonic's scheme is not shared with anyone. Olympus ca[6] is one whole-image polynomial per channel against 0x011b's four zones, and no other project decodes the tag; Homeister's 2018 forum post is still the only public RE. This closes the "Olympus ca[6] correspondence" open question. Two follow-ups ran to a conclusion, both negative: the inner checksums at words [2, 7, 13] resist a Rigo-family search over ~6.3e10 tuples per word, and normalizing the predictors by the zone radii confirms the words are body-scaled yet leaves the fit on session 18's exact ceiling. Names the identifiability check as the one cheap test still open.
Two tests, and together they read differently from either alone. Bootstrapping the tile set puts per-file sampling noise at 0.04 px at r = 0.85, yet files whose six predictor words are bit-identical disagree by 0.24 px RMS on measured B shift, six times that. Read alone this caps any function of the six words and leaves the refits headroom. But a demosaic-free estimator built on the raw Bayer sub-lattices disagrees with the AAHD pipeline by about 1 px with 29% sign flips at r = 0.85, including a sign flip on both regression targets, with no scale or offset relation that would let one be recovered from the other. Separately, four corpus files have no AAHD tiles beyond r = 0.7, so their r >= 0.85 values are extrapolation reaching +14.6 px, and every comparison in sessions 17, 18 and 20 was scored at those radii. So the near-duplicate floor is most economically explained as scene-dependent estimator bias rather than a missing metadata input, and sessions 14-20 optimised against a target whose per-file bias is comparable to the signal. That accounts for the whole shape of the ceiling, including session 18's visible regression on P1366392 despite an analytically correct-sign prediction. The shipped configuration stands on session 18's rendered-pixel check against SOOC crops, which does not depend on the AAHD pipeline, and that is the currency any future attempt should be scored in.
An independent review of the task list checked the document's citations by executing them, and three claims did not survive. The crop TIFFs that session 18 called Panasonic SOOC output report Software: darktable 5.7.0+963~g2e04bf74d7. They are darktable renders of P1366399 with the lens module off and on, so comparing them measures what darktable's own correction does, not how closely it tracks the camera. The genuine camera render of that frame, P1366399.JPG with Software: Ver.2.7, was not in the comparison. Session 21 had promoted that paragraph to the one validation independent of the AAHD metric, which means nothing in the document independently validates the shipped coefficients; they rest on the session 5 derivation from Adobe DNG output and on the untested premise that Adobe reads 0x011b at all. The related inference that shipped "matches SOOC" is corrected in place. The two striped checksum ranges were written as bytes 4..59 evens and odds. The implementation sums even[1..30] and odd[1..30] over even[i] = buf[2*i], so they are byte offsets 2..60 and 3..61. Recomputation on P1366477.RW2 and P1366399.RW2 reproduces the stored words with the corrected ranges and fails with the old ones. A payload editor built to the old spec would have corrupted every file it touched. Session 21's tile-support wording overstated the case: three of the four files reach r_max 0.74 to 0.78, so the correct statement is that none reaches the r = 0.85 they were scored at. Also adds a "When to stop" section. Eight consecutive negative sessions with no written abandonment criterion is what allowed sessions 17, 18 and 20 to re-run variants against a metric that could not rank them.
The developer has set the goal explicitly: reproduce Panasonic's own CA correction in darktable, computed from the same data Panasonic uses. Not correct the lens as well as possible, and not by whatever means works. That rules out the hybrid path of embedded distortion plus Lensfun TCA, and image-adaptive detection in cacorrectrgb. Both would give users a correction; neither is Panasonic's. The proposals are annotated in place as ruled out rather than deleted, since they remain a record of what was considered. It also means the direct raw CA measurement of sessions 14 to 21 was aimed at the wrong quantity. It records the sensor's aberration, while the goal is Panasonic's correction of it, which session 18 estimated leaves 40 to 50 percent in place at outer radii. A refit matching the sensor more closely would move away from the goal, so that measurement work is demoted to background: useful for quantifying the distance between Panasonic and physical truth, not for fitting against. The stopping rule's fallback is revised accordingly. With no substitute correction permitted, stopping means stopping, with the shipped decode left in place and locked behind a rendered-pixel check.
The premise under every coefficient darktable ships for Panasonic CA had never been tested: session 5 fitted them to Adobe DNG Converter's per-plane WarpRectilinear output, assuming Adobe derives per-channel CA from the 0x011b payload rather than from its own lens-profile database. Differential probing settles it in favour of the decode. Transplanting the whole 64-byte payload between sibling frames of the same body and lens, which keeps all outer and all unknown inner checksums self-consistent, reproduces the donor's D_R and D_B differentials exactly on the G9 and S5II and to about 1e-5 on the GX80. The complementary probe, changing the reported lens to a different real Panasonic lens while leaving the payload alone, changes the coefficients by exactly zero. Together those exclude both the lens-profile hypothesis and the mixed case. Three further findings fall out. Mutating 0x0119 collapses the G plane to identity while R and B keep tracking 0x011b, so Adobe treats distortion and CA as independent inputs, which answers the double-correction worry and contradicts Homeister's claim that 0x011b carries both. Stale Rigo checksums make Adobe emit no OpcodeList3 at all, so it validates them, unlike darktable, which logs and continues. Individual words move the planes separately, words 8 and 23 driving R, word 20 driving B and a zone radius driving both, matching the shipped predictor set. Also records the payload editor built for this, and that the four outer checksums are inter-dependent: word[0] covers word[1]'s low byte and word[30]'s, so they must be recomputed in order or the file fails its own verifier. The consequence is that the word-to-coefficient mapping is recoverable as a deterministic function, with no demosaic and no scene dependence anywhere in the loop, which sidesteps everything session 21 found wrong with the measurement.
Reading the map off the Adobe oracle took 165 conversions, and an independent audit re-derived every claim with 106 fresh ones. Twelve words carry the CA signal: the four zone radii, four R-only coefficient words 8/12/23/26 and four B-only words 10/20/27/29. Opposite-channel derivatives are exactly zero, the coefficient words are linear to about 1e-12, they are exactly additive with each other, and no payload word touches the G plane. Fifteen of the 27 numeric words swept do nothing at all, and word 14 is a gate: set it to zero and Adobe emits no warp opcode. Measured against that, the shipped decode is wrong in four ways. It omits words 26 and 29, it carries six cross-channel matrix columns whose true derivatives are zero, its k0 terms for words 8 and 23 are off by factors of 2.11 and 4.01, and its evaluator drops D_B k2 and k3, which are real. That accounts for sessions 14 to 21 failing to converge: the model form was wrong before any fitting began. The tables are nevertheless not ready to patch. A nine-conversion linear model predicts a held-out edit of all eight coefficient words to 8.7e-15 within one file, and is 60.6% wrong on a sibling frame of the same body with an identical radius tuple. Two real G9 files give word 8 derivative ratios of 0.48 to 0.56 where repeat conversions of one file agree to 2.5e-6. The conditioning variable is unidentified; the failing file has a non-identity G polynomial where the working one is identity, so the distortion state is the prime suspect, which would also explain the 1e-5 transplant residual on the GX80 in session 22. Recorded with the audit's qualifications rather than the sweep's own framing: the radius perturbations drive Adobe with inputs no camera produces, since every body group has exactly one radius tuple, so their non-linearity and their interaction with the coefficient words are not facts about the format.
Session 23 left the decode blocked on an unnamed variable: a nine-probe model exact to 1e-15 within one file was 60.6% wrong on a sibling frame with identical zone radii. It is the 0x0119 distortion state. Transplanting the whole payload between a G-identity file and one with G_rms 0.124, both directions, leaves the recipient's G plane intact to 0.000e+00 while D_R misses the donor by about 60% of the signal, so the conditioning is external to the payload. Across six G9 files the derivative scatter correlates with G_rms at Pearson r above 0.93 for all 32 signal components. Clearing the 0x0119 enable nibble collapses that scatter to a standard deviation of 3.4e-18: the per-word slope vectors become identical at machine precision. The representation follows. R - G removes a composed distortion only to first order, which is why it never collapsed. In the pre-distortion frame, eps = R(r_out)/G(r_out) - 1 evaluated at r_raw = r_out * G(r_out), the payload-invariant CA polynomial comes back to 1e-16 on identity-G files and degrades only to 3e-4 at G_rms 0.124, that residual being Adobe's own truncation of a higher-degree composition into a degree-6 polynomial. This adds a fifth error to session 23's list of four: session 5 fitted the shipped tables against Adobe's distortion-enabled D_R and D_B, which is the composition rather than the CA. A wrong target, not a mis-parameterised fit, and a better explanation of the k0 discrepancies of 2.11 and 4.01 than anything proposed so far. A decoder can now be specified: parse the twelve active words, map them to eps_R and eps_B in the pre-distortion frame using a corpus with distortion neutralised, and apply the result as a per-channel radial rescale in the raw frame ahead of the distortion correction. darktable already has the distortion path, so pure CA plus that path reproduces the composition by construction.
With distortion neutralised the map is built rather than fitted: zero a channel's four coefficient words for the intercept, then set each alone for its column, nine conversions per context. eps_R is affine in words 8/12/23/26 and eps_B in 10/20/27/29, intercepts at most 5.3e-15 and opposite-channel columns exactly zero. An independent audit reproduced the matrices from 45 fresh conversions with max abs difference 0. End to end on real untouched files with distortion enabled, predicting Adobe's own R and B planes from the payload reaches 1e-14 on identity-G files, 1.7e-7 at G_rms 0.0043 and 1.7e-5 at G_rms 0.05, with error correlating against distortion strength at 0.894 and 0.913. The payload plus this map reproduces Adobe to within a rounding error of Adobe's own polynomial fit. The residual body term is identified and is not Panasonic's. Two bodies sharing a radius tuple differ by 0.22 to 1.45 percent, and changing only the TIFF Model string makes Adobe emit the target model's exact matrix column, max abs difference 0, with payload and firmware untouched. So Adobe's per-model profile is the selector; ActiveArea ratios do not predict it and mostly make it worse. Since it comes from Adobe's data rather than the camera's, and amounts to 0.0002 to 0.0037 px against corrections of 0.31 to 2.57 px, it is deliberately not reproduced. Generalisation across contexts fails for every radius scaling tested, leaving 17 to 31 percent median residual, so a decoder needs per-context tables and a stated fallback for unknown tuples. The corpus holds five tuples across 14 model strings, and GX8 is corrected to the G9 tuple rather than the GX80 one it was previously grouped under.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP
Turns out that Claude may be very well capable of extracting the required knowledge from various sources and a couple of RW2 files processed with SILKYPIX.