[jajhall]

Description

The assert triggered in #3564, drew my attention to the handling of explicit zeros, small values, duplicate values and large values in the constraint matrix after presolve.

What's happening currently

• Small values
  • For LPs, small values in the constraint matrix after presolve were not being removed (even though they pose some risk to the simplex solver) on the grounds that "they may play a meaningful role in postsolve"
  • For MIPs, small values in the constraint matrix after presolve were being removed when passing the presolved model to the lpsolver instance in HighsLpRelaxation. It sounds as if this may compromise what's returned to the MIP solver, since solutions feasible in the LP solver may have small infeasibilities in the MIP solver.
• Zero values, duplicate values and large values
• For LPs there's no check for zero values, duplicate values and large values after presolve, other than in debug, so it has to be assumed that presolve avoids creating them
• For MIPs, zero values and duplicate values in the constraint matrix after presolve were being removed when passing the presolved model to the lpsolver instance in HighsLpRelaxation. However, any duplicate values in the constraint matrix after presolve would be present in the rest of the MIP solver. However, the evidence of the debug check for LPs suggests that they aren't created by presolve.

Note that the return for the call

lpsolver.passModel(other.lpsolver.getLp());

isn't being checked! Any duplicates or large values trigger a HighsStatus::kError return, and the LP isn't loaded. Any explicit zeros or small values are removed and trigger a HighsStatus::kWarning return. Clearly it would be very noticeable if the LP isn't loaded and subsequent solves involve an empty model.

What's happening with this PR

After LP or MIP presolve, and currently guarded by cost bool kCheckReducedMatrix which is currently false,

• HighsSparsematrix::assessExtremeValues is used to determine whether there are any explicit zeros, small values, or large values in the constraint matrix. Zeros or large values lead to the presolve status being set to a new value HighsPresolveStatus::kMatrixValueError, resulting in failure of the solver. Small nonzero values are retained, but a warning message is logged, and a warning status is returned.
• An assert on the return of HighsSparsematrix::assessStructure is a sanity check that the structure (including explicit zeros and duplicates) of the constraint matrix is fine - as it should me

This shouldn't lead to any more failures - since explicit zeros, duplicates and large values appear not to be generated by - but maybe a test should be done, but I felt it should be switched off until @Opt-Mucca and @fwesselm have considered it.

Other changes

• In highs/mip/HighsMipSolver.cpp, as well as handling the new HighsPresolveStatus::kMatrixValueError, I've added code to handle exception or memory allocation errors in presolve that should have been added previously.
• assessMatrix had been tightened up for the case when a partitioned matrix is passed

What's not been done

Small nonzero values from presolve are still being zeroed when passing the presolved model to the lpsolver instance in HighsLpRelaxation. If you want the small values from presolve to be used when solving LP relaxations, the small_value option will have to be reduced. There's a clear trade-off between losing precision from presolve, and increasing the risk of numerical issues in the simplex solve.

However, as it stands, all this PR does is fix #3564

Checklist

• I have read the contributing guidelines
• This PR targets the latest branch
• Tests are passing
• Documentation was updated where relevant
• This PR is not primarily AI-generated (per the AI contributions policy in CONTRIBUTING.md)

Related issue

Closes #3564

[codecov]

Codecov Report

❌ Patch coverage is 76.80000% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.77%. Comparing base (abf0173) to head (9e12bcf).
⚠️ Report is 36 commits behind head on latest.

Files with missing lines	Patch %	Lines
highs/lp_data/Highs.cpp	16.66%	10 Missing ⚠️
highs/util/HighsMatrixUtils.cpp	61.53%	10 Missing ⚠️
highs/mip/HighsMipSolver.cpp	66.66%	4 Missing ⚠️
highs/util/HighsSparseMatrix.cpp	91.89%	3 Missing ⚠️
highs/util/HighsUtils.cpp	71.42%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           latest    #3578      +/-   ##
==========================================
+ Coverage   75.67%   75.77%   +0.10%     
==========================================
  Files         461      461              
  Lines      113794   114093     +299     
  Branches    17909    17939      +30     
==========================================
+ Hits        86108    86454     +346     
+ Misses      27398    27351      -47     
  Partials      288      288              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Opt-Mucca]

The changes look good to me!

For MIP: Isn't any coefficient smaller than small_matrix_value after presolve coming from a reduction that we in theory should have blocked (the same for LP)? Or have I misunderstood the

double sum = Avalue[pos] + val;
    if (std::abs(sum) <= options->small_matrix_value) {
      unlink(pos);
    }

block in addToMatrix @fwesselm ? I guess we're doing some reduction that is adding a new term to the matrix with value < small_matrix_value without checking it first?

I don't mind us deleting these values in the LP relaxation for MIP until we find a not-incredibly-extreme instance that is negatively affected by this. In general I'd also still like to lower the small_matrix_value further.

[jajhall]

For MIP: Isn't any coefficient smaller than small_matrix_value after presolve coming from a reduction that we in theory should have blocked (the same for LP)? Or have I misunderstood the

double sum = Avalue[pos] + val;
    if (std::abs(sum) <= options->small_matrix_value) {
      unlink(pos);
    }

block in addToMatrix @fwesselm ? I guess we're doing some reduction that is adding a new term to the matrix with value < small_matrix_value without checking it first?

I thought it was almost certainly the case that explicit zero, duplicates and large values couldn't be created by presolve, but I didn't know what restrictions there were on creating small values.

Instinct tells me that this small value is the result of an elimination operation, so a new nonzero hasn't been created.

The instance is small, so it shouldn't be hard to identify how it's created

I don't mind us deleting these values in the LP relaxation for MIP until we find a not-incredibly-extreme instance that is negatively affected by this. In general I'd also still like to lower the small_matrix_value further.

OK, at some calm moment we should lower 'small_matrix_value' and do a full test. Perhaps, as discussed, switching to maximum value scaling may be necessary to stop them from being scaled up - as can happen with the current equilibrium scaling (in simplex).