Repository navigation
fix(healthcheck): keep worker pickers aligned with shared state - #14075
Open
hanzhenfang wants to merge 4 commits into
Open
hanzhenfang wants to merge 4 commits into
hanzhenfang wants to merge 4 commits into
Conversation
Read target health from shared memory before a worker-local cache, including the interval before its checker exists. Include effective target health in ordinary and AI picker cache versions so an event lost by one worker cannot retain an unhealthy target. Based on the core fix in apache#13894; extend it to late checker creation and ai-proxy-multi. Co-authored-by: Yuzhong Zhang <BetterAndBetterII@users.noreply.github.com>
This was referenced Oct 23, 2026
Member
Why does the event get missed? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Active health checks write target state to shared memory and then notify workers through
resty.events. Routing can still read a stale worker-local state if that event is missed. The ordinary balancer andai-proxy-multialso cache pickers using each worker'sstatus_ver, so an unchanged local version can retain an unhealthy target. A worker whose checker has not yet been created can encounter the same shared state.This PR makes the health status and picker versions follow the shared target state for ordinary and AI upstreams, including the interval before the local checker exists. It preserves the current all-unhealthy fallback and the
masterbranch's checker target reconciliation.The shared-state read and ordinary picker invalidation build on #13894. Compared with #13894, this PR adds:
ai-proxy-multicache invalidation and filtering: both its instance picker version and its chash health-status cache version include each instance's effective node health. The AI health filter also uses shared state before the local checker exists, excluding an instance marked unhealthy by a shared-memory-only flip.This is a self-contained
masteralternative to #13894 so the additional paths can be reviewed together. Credit for the core fix goes to @BetterAndBetterII. Maintainers can choose one mainline PR; the overlapping core changes should not be merged twice.Tests
master210943c18d6e40ea599c7554f12f658bea012d4fwith the same new test files,t/node/healthcheck-stale-worker-cache.tfailed 5/9 assertions andt/plugin/ai-proxy-multi-healthcheck-shm-flip.tfailed 3/3. The latter selected the instance already marked unhealthy in shared memory.healthcheck-global-switch.t,chash-healthcheck-stable-ring.t,healthcheck-incremental-update.t,ai-proxy-multi-healthcheck-stale-picker.t, andai-proxy-multi-chash-healthcheck-stable-ring.t: Files=5, Tests=90, Result: PASS.git diff --checkandluajit -blon the three changed Lua files succeeded.The tests used isolated containers, an etcd sidecar, the APISIX 3.19.0 Debian runtime, and Test::Nginx
ced30a31. They validate a controlled shared-memory state flip; the intermittent missed event from the original Kubernetes report was not naturally reproduced. Full CI and a build from currentmasterremain to be checked.Compatibility and limits
Picker version generation reads shared state for each node. The no-checker path uses the shared-dict key layout of
lua-resty-healthcheck, which should be reviewed when upgrading that dependency. When no shared status exists yet, nodes remain usable as before. No release branch backport is requested here.Fixes #13888