You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Report shared-start and seed-spread context in validate_implementations.py #359
The harness fits pamica and the reference from independent random starts at one seed. So its log-likelihood gap is one draw from the start-to-start spread:
the reference's own final LL has a standard deviation of 2.6e-4 over 8 seeds at the harness settings;
the harness gap moved from 2.9e-5 to 2.7e-4 when the epic changed the initial draw.
The update-rule comparison is the shared-start one: the reference loaded with pamica's initial state gives an LL gap of 1.6e-6 and a correlation of 0.99999993.
Proposal: have validate_implementations.py also report a shared-start comparison through native_oracle.run_seeded_reference, the reference's own seed spread, or both, next to its independent-start figure, so the output carries its own context.
Scripts and records: .context/issue-351/harness_gap.py, raw/harness_gap_head.json.
Activity