Skip to content

fix(earnings): preserve footnoted EPS and reported table currencies - #1864

Merged
dgunning merged 8 commits into
dgunning:mainfrom
ShengranYan:fix/issue-1824-earnings-parser
Nov 12, 2026
Merged

dgunning merged 8 commits into
dgunning:mainfrom
ShengranYan:fix/issue-1824-earnings-parser

Conversation

@ShengranYan

@ShengranYan ShengranYan commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

EarningsRelease left TSM's raised-footnote EPS as text and lost its TWD table
currency. It now uses HTML evidence to remove raised single-letter footnotes
from otherwise numeric earnings cells, leaves ambiguous plain suffixes such
as 1.2b, 1.2m, and 500k untouched, and keeps bare EPS rows unscaled for
all filers. Currency metadata survives facts/JSON exports, context and cleaned
copies.

This revision also fixes the USD and scale regressions reported in #1864:
scale-only notes retain the existing USD default; a constant-currency note
cannot reset an amount scale; caption/header scales precede nearby declarations.
Wrapper traversal is shared by currency and scale detection.

Refs #1824. Parser prerequisite split from #1825 at the maintainer's request;
the Company 6-K fallback remains in that separate PR. Intended for 6.1.

Verification

  • Ground truth — TSM: 6-K 0001046179-26-000726, 2026-10-15,
    a3q26e_withguidancexfinal.htm: TWD 763,064 million net income and
    NT$29.43 EPS (raised letter b). The unedited fixture is the SGML DOCUMENT
    block obtained from the complete submission on 2026-11-01, not a standalone
    HTML response. Original bytes and hash are preserved.
  • Ground truth — Cardlytics: 8-K 0001666071-25-000124, 2025-08-06,
    cdlx_form8-kx2025q2xex991.htm: Q2 revenue USD 63,249 thousand, net loss
    USD 9,283 thousand, diluted loss per share USD 0.18. Downloaded unchanged
    on 2026-11-04 before this revision; source, acquisition date and hash are
    recorded in the fixture README.
  • Regression: tests/issues/regression/test_issue_1824_earnings.py
    links the canonical issue and verifies both real filings, ambiguous suffixes,
    raised HTML markers, negative EPS, scale precedence, mixed/unsupported
    currencies, per-table units, metadata copying and defensive DOM traversal.
    An initial set of 45 cases reproduced 23 failures on the pre-review code,
    before the production fix was applied. A controlled rerun of all 47 final
    cases against the unchanged pre-review module reproduces 25 failures,
    with its SHA-256 checked against HEAD.
  • Full fast suite is not entirely green: hatch run test-fast --no-cov
    on this standalone branch: 50,374 passed, 5 failed, 9 skipped, 3,416
    deselected, 1 xfailed
    (506.76s). All five failures are in the unchanged
    Issue 282 Apple XBRL tests: latest submissions select 2026 accession
    0000320193-26-000026, but their cassette contains 2025 accession
    0000320193-25-000079. Do not overwrite/re-record that cassette to make
    this pass. The same six tests pass in isolation on both the pre-review
    and revised modules. Feeding both versions the identical, already-cached
    2026 submissions response reproduces the same five failures / one pass
    on both. This is an existing cache/isolation mismatch, not a changed earnings
    path; it remains a limitation of the complete-suite result.
  • Offline audit: 170 passed, 6 deselected, with sockets blocked
    and BeautifulSoup deliberately unavailable, including the old earnings
    and Issue 1775/1777/1779/1780 regression files. All 47 new issue cases
    pass after the fix; the pre-review module fails 25 of those cases.
  • Related network suite: existing Issue 633 real-filing earnings
    tests: 4 passed (20.09s), one file run sequentially. These exercise
    HURN, RF, GRPN and AMT through Company / Filing / EarningsRelease.
  • Cassettes: no VCR cassettes added, edited or re-recorded. Cassette
    safety and regression provenance checks pass.
  • Docs: existing FinancialTable data/scaling/export examples are
    exercised against local real SEC fixtures. Fixture provenance is corrected;
    unused Company-only 6-K fixtures are removed from this PR.
  • Changelog: changelog.d/1824.fixed.md, including global bare-EPS
    handling. Assembly validation passes.

Public APIs, Filing/Attachment contracts, XBRL mappings and generic document
parsing are preserved. Earnings-specific footnote handling is a private table
processing strategy of the existing canonical HTML parser. There are no new
dependencies. The existing edgar.earnings.parse_html import remains available
for compatibility. Existing transport/parse error behavior is retained.

Lint: hatch run lint still reports pre-existing repository findings. The
changed source file has the same nine findings as pre-review HEAD, with no
new diagnostic signatures (1,055 repository findings before and after), and the regression file passes Ruff. No repository
wide reformat or unrelated lint cleanup is included.

AI assistance

  • Assisted. Tool and model: OpenAI Codex (GPT-6).

Working context (optional)

Plan: reproduce the maintainer's counterexamples first, preserve existing
interfaces and parser semantics, fix currency and scale independently, use
raised HTML markup rather than guessing from letters, retain original SEC
bytes, and run the old regression suite alongside new value assertions.

Commands: hatch run test-fast --no-cov,
hatch run test-network tests/issues/regression/test_issue_633_earnings_data_accuracy.py,
hatch run check-regression-provenance, hatch run check-cassettes,
python scripts/release/assemble_changelog.py --check, and the
tests._offline_harness audit with BeautifulSoup deliberately unavailable. Tests
cover the two pinned filings plus existing regressions; this is not a claim
that every possible issuer's HTML layout has been verified. The shared HTML
parser's original text output is explicitly checked to remain unchanged.

@dgunning dgunning left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pulling this out of #1825. It's much easier to review on its own and it's the right place for the fix.

One thing blocks it for me. _declared_table_units treats any "(in …" note as a currency declaration, so a plain US note like Cardlytics' "(Amounts in thousands, except per share amounts)" (0001666071-25-000124) now sets currency=None. Its facts lose their USD unit and to_context says "currency units". I'd keep USD when the note names no currency and only fall back to None when it names more than one.

The same match also overrides the scale unconditionally. If a nearby paragraph says "(in constant currency)", the table drops to units even when its header says millions. Could you only take the declared scale when it actually names one, and leave caption and header detection ahead of it?

On the footnote regex, I'm wary of 1.2b, 1.2m and 500k turning into 1.2, 1.2 and 500. They used to stay as text, and a wrong number is worse than a missing one. Keying off the superscript in the HTML, or excluding k/m/b, would avoid that.

A few smaller things. _table_context repeats the wrapper climb in _detect_table_scale, so I'd rather share one helper. The two TSM 6-K fixtures aren't used here and can stay in #1825. The fixture README says these are the raw Archives bytes, but they carry the SGML <DOCUMENT> wrapper from the submission .txt, so it should say that. Please also rename the fragment to 1824.fixed.md and the test to test_issue_1824_earnings.py (#1825 already has test_issue_1824.py) with a GitHub Issue: line, and mention in the fragment that "EPS" rows are now unscaled for every filer.

Agreed on 6.1 for this one.

@dgunning dgunning mentioned this pull request Nov 3, 2026
7 tasks done
@dgunning dgunning added this to the Release 6.x milestone Nov 3, 2026
@dgunning dgunning added the post-6.0 Triaged; scheduled after the 6.0 release label Nov 3, 2026
ShengranYan and others added 6 commits November 4, 2026 09:21
The scale-only USD fallback matched the "in millions" prefix and ignored the
rest, so "(in millions of yen)", "(in millions of Canadian dollars)" and
"(in thousands of pounds sterling)" were reported as USD. Declarations with a
leading currency ("(€ in millions)", "(Yen in millions)", "(RMB in thousands)")
did not match at all and also fell back to USD.

- JPY recognises "yen" and "¥".
- The declaration pattern accepts a leading currency token; other leading
  words ("(Shares in thousands)") are skipped as before.
- USD is assumed only when the declaration names a scale and no currency;
  a currency the table does not map yields None.
- A table's own unit cells (caption, header, first rows) are checked before
  nearby paragraphs, so Toyota's "Yen in millions" header row reports JPY
  (6-K 0001193125-26-448368).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"10<sup>2</sup>" -> 102.0 asserted the existing bug where a superscript digit
footnote is concatenated onto the number (1,234<sup>1</sup> becomes 12341 on
main too). That behaviour is wrong, not a contract, so the case is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scaled_dataframe chose rows by type but scaled every numeric column, so TSM's
YoY/QoQ % columns became 50.9 -> 50,900,000 once the table was correctly
detected as NT$ millions (6-K 0001046179-26-000726). Columns whose header
contains '%' and names no scale are now left as filed; net income still
scales to 763,064,000,000. A header that names a scale keeps scaling, so
Toyota's "(% of change ...) - Sales revenues - Million yen" amount columns
(6-K 0001193125-26-448368) are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any three capitals counted as a currency code, so a US note such as
'(in thousands, except per share and DAU data)' lost its USD label.
List the ISO 4217 codes instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 'units:' branch of _UNIT_DECLARATION began with an optional '(' and
'\s*', so every space in a whitespace run was a start position and '\s*'
re-read the rest of the run from each one: quadratic in the run length
(8,000 spaces took 218 ms; 200,000 would take about two minutes). The
branch now starts at '(' or a word boundary. Declarations match as before
(leading whitespace is no longer part of the match, and callers strip it),
and 'units:' inside a word such as 'subunits: 3' no longer matches.

Measured on 15 earnings releases (AAPL ... TSM): table extraction for the
whole PR costs about 3% over main; this regex was the only superlinear
path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@dgunning dgunning left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this covers everything from my last review. I pushed one small commit on top: the 'units:' branch of the unit-declaration regex could start at every space of a whitespace run, which made it quadratic on long runs (8,000 spaces took 218 ms). It now starts at '(' or a word boundary, with a test for a 200,000-space run. Otherwise the change costs about 3% on table extraction across 15 earnings releases I timed, which is fine.

@dgunning
dgunning enabled auto-merge (squash) November 12, 2026 14:43
@dgunning
dgunning merged commit 62758c6 into dgunning:main Nov 12, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

post-6.0 Triaged; scheduled after the 6.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants