Skip to content

fix(healthcheck): keep worker pickers aligned with shared state - #14075

Open
hanzhenfang wants to merge 4 commits into
apache:masterfrom
hanzhenfang:fix/healthcheck-shm-picker-consistency
Open

hanzhenfang wants to merge 4 commits into
apache:masterfrom
hanzhenfang:fix/healthcheck-shm-picker-consistency

Conversation

@hanzhenfang

@hanzhenfang hanzhenfang commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Description

Active health checks write target state to shared memory and then notify workers through resty.events. Routing can still read a stale worker-local state if that event is missed. The ordinary balancer and ai-proxy-multi also cache pickers using each worker's status_ver, so an unchanged local version can retain an unhealthy target. A worker whose checker has not yet been created can encounter the same shared state.

This PR makes the health status and picker versions follow the shared target state for ordinary and AI upstreams, including the interval before the local checker exists. It preserves the current all-unhealthy fallback and the master branch's checker target reconciliation.

The shared-state read and ordinary picker invalidation build on #13894. Compared with #13894, this PR adds:

  1. The checker-creation window: when a worker has no local checker yet, it looks up the target's shared state using the upstream resource key. Both ordinary upstreams and AI instances can exclude a target already marked unhealthy by another worker on their first request. fix(healthcheck): read shm status so workers do not keep unhealthy nodes #13894 treats a missing checker as healthy.
  2. ai-proxy-multi cache invalidation and filtering: both its instance picker version and its chash health-status cache version include each instance's effective node health. The AI health filter also uses shared state before the local checker exists, excluding an instance marked unhealthy by a shared-memory-only flip.
  3. Regression coverage for those paths: the ordinary test includes a no-checker shared-state case, and a separate AI test exercises a warmed picker after a shared-memory-only flip. The red-green results are listed below.

This is a self-contained master alternative to #13894 so the additional paths can be reviewed together. Credit for the core fix goes to @BetterAndBetterII. Maintainers can choose one mainline PR; the overlapping core changes should not be merged twice.

Tests

  • On unmodified master 210943c18d6e40ea599c7554f12f658bea012d4f with the same new test files, t/node/healthcheck-stale-worker-cache.t failed 5/9 assertions and t/plugin/ai-proxy-multi-healthcheck-shm-flip.t failed 3/3. The latter selected the instance already marked unhealthy in shared memory.
  • With this patch, both new Test::Nginx files passed together: Files=2, Tests=12, Result: PASS.
  • Five existing related files passed: healthcheck-global-switch.t, chash-healthcheck-stable-ring.t, healthcheck-incremental-update.t, ai-proxy-multi-healthcheck-stale-picker.t, and ai-proxy-multi-chash-healthcheck-stable-ring.t: Files=5, Tests=90, Result: PASS.
  • git diff --check and luajit -bl on the three changed Lua files succeeded.

The tests used isolated containers, an etcd sidecar, the APISIX 3.19.0 Debian runtime, and Test::Nginx ced30a31. They validate a controlled shared-memory state flip; the intermittent missed event from the original Kubernetes report was not naturally reproduced. Full CI and a build from current master remain to be checked.

Compatibility and limits

Picker version generation reads shared state for each node. The no-checker path uses the shared-dict key layout of lua-resty-healthcheck, which should be reviewed when upgrading that dependency. When no shared status exists yet, nodes remain usable as before. No release branch backport is requested here.

Fixes #13888

Read target health from shared memory before a worker-local cache, including the interval before its checker exists. Include effective target health in ordinary and AI picker cache versions so an event lost by one worker cannot retain an unhealthy target.

Based on the core fix in apache#13894; extend it to late checker creation and ai-proxy-multi.

Co-authored-by: Yuzhong Zhang <BetterAndBetterII@users.noreply.github.com>
@nic-6443

Copy link
Copy Markdown
Member

Active health checks write target state to shared memory and then notify workers through resty.events. Routing can still read a stale worker-local state if that event is missed.

Why does the event get missed? resty.events is also used elsewhere. If it has inherent event loss issues, we should fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Health check state diverges across nginx workers — unhealthy node keeps receiving traffic

2 participants