Initiate a single subscription on concurrent addMessageListener calls - #3448
Open
rlaehddus302 wants to merge 1 commit into
Open
rlaehddus302 wants to merge 1 commit into
rlaehddus302 wants to merge 1 commit into
Conversation
RedisMessageListenerContainer now initiates the initial subscription only once when multiple threads call addMessageListener(…) on a started container that has no listeners yet. Previously, the state check in lazyListen() and the compare-and-set in doSubscribe(…) were separate steps and the compare-and-set used whatever state was read last, so a second caller could run Subscriber.initialize(…) again and obtain a second subscription connection. A caller that did not initiate the subscription now subscribes its topics explicitly once the container is listening. Previously, the listening state was captured before waiting for the initial subscription, so such a caller skipped subscribing and its channel could remain unsubscribed. Subscription recovery continues to re-subscribe regardless of the current state. Closes spring-projects#3447 Signed-off-by: rlaehddus302 <kimyeon30@naver.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3447
Problem
When
RedisMessageListenerContaineris started without listeners and two threads calladdMessageListener(…)at the same time, either two subscription connections are created (every message is delivered twice) or one of the channels is never subscribed. See #3447 for details and a reproducer.Cause
lazyListen()inspects the state and only afterwards callsdoSubscribe(…).doSubscribe(…)reads the state again and uses whatever it just read as the expected value ofcompareAndSet(state, State.prepareListening()). If another thread has already moved the state toprepareListening(orlistening), the CAS still succeeds andSubscriber.initialize(…)runs a second time, obtaining a second connection.addListener(…)capturedwasListeningbefore callinglazyListen(). A thread that waited inlazyListen()for another thread's initial subscription continued withwasListening == falseand skippedsubscribeChannel(…)/subscribePattern(…). If its topic was registered after the initiating thread collected the topics for the initialSUBSCRIBE, nobody subscribed to it.Change
doSubscribe(…)returnsSubscribeResult(RETRY,INITIATED,ALREADY_ACTIVE) instead ofboolean. For an initial subscription request (InitialBackoffExecution) it returnsALREADY_ACTIVEwithout callingSubscriber.initialize(…)if the state is alreadyprepareListening/listening.RecoveryBackoffExecution,RecoveryAfterSubscriptionBackoffExecution) behave as before. They have to re-subscribe while the container is in thelisteningstate, so the new check does not apply to them.lazyListen(BackOffExecution)returns the future to await together with the information whether the calling thread initiated the subscription (ListenAttempt), andlazyListen()returns that flag.addListener(…)subscribes its topics explicitly if it did not initiate the subscription and the container is listening (!initiated && isListening()), instead of relying on the state captured before waiting.One consequence: a caller that did not initiate the subscription may send
SUBSCRIBEfor a channel that the initial subscription already contains (when its topic was registered before the initiating thread collected the topics). I consider that harmless because subscribing to an already subscribed channel on the same connection does not create a second subscription, but I'd be glad to hear if you prefer a different approach.Verification
RedisMessageListenerContainerUnitTests.concurrentInitialAddMessageListenerShouldInitiateSingleSubscription. It holds both callers insideBackOff.start(), whichlazyListen()invokes after inspecting the state and before initiating the subscription, so both callers observe "not listening" deterministically. Without the change toRedisMessageListenerContainerthe test fails in 3 of 3 runs (getConnection()wanted 1 time, but was 2 times). With the change it passes (the test class was run more than ten times, all green).RedisMessageListenerContainerUnitTestspass, includingshouldRecoverFromConnectionFailure.main: 50 trials gave OK 29 / TWICE 4 / MISSING 16 before the change. With the change, three runs of 50 trials each gave TWICE 0 / MISSING 0. In both variants only the very first trial of a run failed with "Subscription registration timeout exceeded" (the container's default registration timeout of 2 seconds). I assume this is JVM/connection warm-up, but I did not investigate it.Scope of testing
org.springframework.data.redis.listenerpackage (-Dtest=org/springframework/data/redis/listener/*Tests.java -DrunLongTests=true) against the project's Docker Compose environment (Redis 8.6.2: standalone with replicas and sentinels, auth, cluster): 115 tests, 0 failures, 0 errors, 3 skipped (the three cases that the tests themselves skip for cluster-aware connections). I did not run the rest of the test suite.addMessageListenertogether withstop()/removeMessageListener(Pub/Sub in a concurrent scenario may result in the inability to use subscriptions anymore, which can ultimately lead to memory leaks #3080, Pub-Sub Race conditions on high load - Subscription has been unsubscribed and cannot be used anymore #2425); this change does not try to address those.