Repository navigation
fix(earnings): preserve footnoted EPS and reported table currencies - #1864
Conversation
dgunning
left a comment
There was a problem hiding this comment.
Thanks for pulling this out of #1825. It's much easier to review on its own and it's the right place for the fix.
One thing blocks it for me. _declared_table_units treats any "(in …" note as a currency declaration, so a plain US note like Cardlytics' "(Amounts in thousands, except per share amounts)" (0001666071-25-000124) now sets currency=None. Its facts lose their USD unit and to_context says "currency units". I'd keep USD when the note names no currency and only fall back to None when it names more than one.
The same match also overrides the scale unconditionally. If a nearby paragraph says "(in constant currency)", the table drops to units even when its header says millions. Could you only take the declared scale when it actually names one, and leave caption and header detection ahead of it?
On the footnote regex, I'm wary of 1.2b, 1.2m and 500k turning into 1.2, 1.2 and 500. They used to stay as text, and a wrong number is worse than a missing one. Keying off the superscript in the HTML, or excluding k/m/b, would avoid that.
A few smaller things. _table_context repeats the wrapper climb in _detect_table_scale, so I'd rather share one helper. The two TSM 6-K fixtures aren't used here and can stay in #1825. The fixture README says these are the raw Archives bytes, but they carry the SGML <DOCUMENT> wrapper from the submission .txt, so it should say that. Please also rename the fragment to 1824.fixed.md and the test to test_issue_1824_earnings.py (#1825 already has test_issue_1824.py) with a GitHub Issue: line, and mention in the fragment that "EPS" rows are now unscaled for every filer.
Agreed on 6.1 for this one.
The scale-only USD fallback matched the "in millions" prefix and ignored the
rest, so "(in millions of yen)", "(in millions of Canadian dollars)" and
"(in thousands of pounds sterling)" were reported as USD. Declarations with a
leading currency ("(€ in millions)", "(Yen in millions)", "(RMB in thousands)")
did not match at all and also fell back to USD.
- JPY recognises "yen" and "¥".
- The declaration pattern accepts a leading currency token; other leading
words ("(Shares in thousands)") are skipped as before.
- USD is assumed only when the declaration names a scale and no currency;
a currency the table does not map yields None.
- A table's own unit cells (caption, header, first rows) are checked before
nearby paragraphs, so Toyota's "Yen in millions" header row reports JPY
(6-K 0001193125-26-448368).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"10<sup>2</sup>" -> 102.0 asserted the existing bug where a superscript digit footnote is concatenated onto the number (1,234<sup>1</sup> becomes 12341 on main too). That behaviour is wrong, not a contract, so the case is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scaled_dataframe chose rows by type but scaled every numeric column, so TSM's YoY/QoQ % columns became 50.9 -> 50,900,000 once the table was correctly detected as NT$ millions (6-K 0001046179-26-000726). Columns whose header contains '%' and names no scale are now left as filed; net income still scales to 763,064,000,000. A header that names a scale keeps scaling, so Toyota's "(% of change ...) - Sales revenues - Million yen" amount columns (6-K 0001193125-26-448368) are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any three capitals counted as a currency code, so a US note such as '(in thousands, except per share and DAU data)' lost its USD label. List the ISO 4217 codes instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 'units:' branch of _UNIT_DECLARATION began with an optional '(' and
'\s*', so every space in a whitespace run was a start position and '\s*'
re-read the rest of the run from each one: quadratic in the run length
(8,000 spaces took 218 ms; 200,000 would take about two minutes). The
branch now starts at '(' or a word boundary. Declarations match as before
(leading whitespace is no longer part of the match, and callers strip it),
and 'units:' inside a word such as 'subunits: 3' no longer matches.
Measured on 15 earnings releases (AAPL ... TSM): table extraction for the
whole PR costs about 3% over main; this regex was the only superlinear
path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dgunning
left a comment
There was a problem hiding this comment.
Thanks, this covers everything from my last review. I pushed one small commit on top: the 'units:' branch of the unit-declaration regex could start at every space of a whitespace run, which made it quadratic on long runs (8,000 spaces took 218 ms). It now starts at '(' or a word boundary, with a test for a 200,000-space run. Otherwise the change costs about 3% on table extraction across 15 earnings releases I timed, which is fine.
What this changes
EarningsReleaseleft TSM's raised-footnote EPS as text and lost its TWD tablecurrency. It now uses HTML evidence to remove raised single-letter footnotes
from otherwise numeric earnings cells, leaves ambiguous plain suffixes such
as
1.2b,1.2m, and500kuntouched, and keeps bare EPS rows unscaled forall filers. Currency metadata survives facts/JSON exports, context and cleaned
copies.
This revision also fixes the USD and scale regressions reported in #1864:
scale-only notes retain the existing USD default; a constant-currency note
cannot reset an amount scale; caption/header scales precede nearby declarations.
Wrapper traversal is shared by currency and scale detection.
Refs #1824. Parser prerequisite split from #1825 at the maintainer's request;
the Company 6-K fallback remains in that separate PR. Intended for 6.1.
Verification
0001046179-26-000726, 2026-10-15,a3q26e_withguidancexfinal.htm: TWD 763,064 million net income andNT$29.43 EPS (raised letter b). The unedited fixture is the SGML DOCUMENT
block obtained from the complete submission on 2026-11-01, not a standalone
HTML response. Original bytes and hash are preserved.
0001666071-25-000124, 2025-08-06,cdlx_form8-kx2025q2xex991.htm: Q2 revenue USD 63,249 thousand, net lossUSD 9,283 thousand, diluted loss per share USD 0.18. Downloaded unchanged
on 2026-11-04 before this revision; source, acquisition date and hash are
recorded in the fixture README.
tests/issues/regression/test_issue_1824_earnings.pylinks the canonical issue and verifies both real filings, ambiguous suffixes,
raised HTML markers, negative EPS, scale precedence, mixed/unsupported
currencies, per-table units, metadata copying and defensive DOM traversal.
An initial set of 45 cases reproduced 23 failures on the pre-review code,
before the production fix was applied. A controlled rerun of all 47 final
cases against the unchanged pre-review module reproduces 25 failures,
with its SHA-256 checked against HEAD.
hatch run test-fast --no-covon this standalone branch: 50,374 passed, 5 failed, 9 skipped, 3,416
deselected, 1 xfailed (506.76s). All five failures are in the unchanged
Issue 282 Apple XBRL tests: latest submissions select 2026 accession
0000320193-26-000026, but their cassette contains 2025 accession0000320193-25-000079. Do not overwrite/re-record that cassette to makethis pass. The same six tests pass in isolation on both the pre-review
and revised modules. Feeding both versions the identical, already-cached
2026 submissions response reproduces the same five failures / one pass
on both. This is an existing cache/isolation mismatch, not a changed earnings
path; it remains a limitation of the complete-suite result.
and BeautifulSoup deliberately unavailable, including the old earnings
and Issue 1775/1777/1779/1780 regression files. All 47 new issue cases
pass after the fix; the pre-review module fails 25 of those cases.
tests: 4 passed (20.09s), one file run sequentially. These exercise
HURN, RF, GRPN and AMT through
Company/Filing/EarningsRelease.safety and regression provenance checks pass.
exercised against local real SEC fixtures. Fixture provenance is corrected;
unused Company-only 6-K fixtures are removed from this PR.
changelog.d/1824.fixed.md, including global bare-EPShandling. Assembly validation passes.
Public APIs, Filing/Attachment contracts, XBRL mappings and generic document
parsing are preserved. Earnings-specific footnote handling is a private table
processing strategy of the existing canonical HTML parser. There are no new
dependencies. The existing
edgar.earnings.parse_htmlimport remains availablefor compatibility. Existing transport/parse error behavior is retained.
Lint:
hatch run lintstill reports pre-existing repository findings. Thechanged source file has the same nine findings as pre-review HEAD, with no
new diagnostic signatures (1,055 repository findings before and after), and the regression file passes Ruff. No repository
wide reformat or unrelated lint cleanup is included.
AI assistance
Working context (optional)
Plan: reproduce the maintainer's counterexamples first, preserve existing
interfaces and parser semantics, fix currency and scale independently, use
raised HTML markup rather than guessing from letters, retain original SEC
bytes, and run the old regression suite alongside new value assertions.
Commands:
hatch run test-fast --no-cov,hatch run test-network tests/issues/regression/test_issue_633_earnings_data_accuracy.py,hatch run check-regression-provenance,hatch run check-cassettes,python scripts/release/assemble_changelog.py --check, and thetests._offline_harnessaudit with BeautifulSoup deliberately unavailable. Testscover the two pinned filings plus existing regressions; this is not a claim
that every possible issuer's HTML layout has been verified. The shared HTML
parser's original text output is explicitly checked to remain unchanged.