Skip to content

fix(reweighing): preserve zero weights in zero-mass cells - #586

Open
Hanabi9248 wants to merge 1 commit into
Trusted-AI:mainfrom
Hanabi9248:codex/reweighing-zero-weights-oct26
Open

Hanabi9248 wants to merge 1 commit into
Trusted-AI:mainfrom
Hanabi9248:codex/reweighing-zero-weights-oct26

Conversation

@Hanabi9248

Copy link
Copy Markdown

Reweighing currently tests whether a (group, label) cell contains rows, then divides by its summed sample weight. A cell containing only zero-weight rows produces inf/NaN, so ReweighingMeta passes NaN sample weights into otherwise valid estimators. For groups [0,0,1,1], labels [0,1,0,1] and weights [1,1,1,0], the last output weight becomes NaN instead of staying zero.

When total mass is positive and all weights in that cell are exactly zero, keep its output weights zero and leave the undefined cell factor NaN, as for an absent cell. Other cells use the original formula unchanged. This follows zero-weight sample removal semantics; it does not add mass to missing group/class combinations or claim independence on incomplete support. No epsilon/default factor or policy for all-zero total weights, negative weights or positive/negative cancellation is added.

Actual validation against main 3487791 on Python 3.11.16:

  • Eight new regression methods. Baseline: five affected methods (six assertion-failure events and two estimator ValueErrors across subcases), three passing controls. Candidate: all eight pass.
  • Complete real package initializers, Reweighing/ReweighingMeta, pandas/NumPy and real liblinear LogisticRegression; no algorithm, estimator, dependency or filesystem mocks.
  • Checks list/array/Series weights, zero-mass cells/groups/classes, fractional weights and row permutation. Positive rows match actual reweighing after deleting zero-weight rows; real LogisticRegression coefficients/intercepts/predict_proba match that deletion reference through both default and explicit reweighers. Controls retain positive formulas, absent-cell NaN factors and zero rows within positive-mass cells. X identity and input preservation are checked.
  • CPU1/768MiB/16 processes/private temporary files/single-threaded BLAS, peak whole-job memory 118624256 bytes; no dependency installation. Added-line whitespace checks pass.

Run: python tests/test_reweighing_zero_weight.py -v

Full repository, existing Adult/download fixtures, other estimators/solvers/Python versions and negative/all-zero weight behavior were not tested. Existing optional rpy2/torch imports warn that unrelated algorithms are unavailable; these modules were not replaced or installed. AI assistance was used for preparation/review; reported test results are actual local executions.

Signed-off-by: Hanabi <317387557+Hanabi9248@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant