Skip to content

CBG-6075: (test-only) wait for background managers to stop before closing the bucket - #9103

Merged
bbrks merged 1 commit into
mainfrom
CBG-6075
Nov 11, 2026
Merged

bbrks merged 1 commit into
mainfrom
CBG-6075

Conversation

@torcolvin

@torcolvin torcolvin commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

CBG-6075: (test-only) wait for background managers to stop before closing the bucket

TestPutResurrection checks the global WarnCount. It failed when it ran right after TestUpdateHeartbeatDocClusterAwareTransientError:

[WRN] t:TestUpdateHeartbeatDocClusterAwareTransientError/startup Failed to update background manager status in periodic polling: this Rosmar bucket has been closed, will retry`

There were several problems:

  1. The startup subtest closed bucket before starting manager due to combination of t.Cleanup and defer.
  2. BackgroundManager.Stop is not synchronous does not wait, so the status poller kept running after it was called while the bucket was deleted.

Fix:

  • Add synchronousStop. It stops the manager and waits for it to finish.
  • Close the leaky bucket in a t.Cleanup that runs after the manager stops.

Not the final fix:

  • Nothing tracks the run goroutine or the heartbeat loop in BackgroundManager. Fixing this is closing a bigger gap in the background manager. Because we wait for the state and the heartbeat, the gap is primarily in the multi node background manager. This is a problem in DatabaseContext.Close where even single node background managers don't actually wait for stopping and their final status updates might also fail. This is less likely just due to the timing of how long it takes the DatabaseContext.Close. This will be thought about in a BackgroundManager refactor.

Pre-review checklist

  • Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
  • Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Integration Tests

TestPutResurrection checks the global WarnCount. It failed when it ran
right after TestUpdateHeartbeatDocClusterAwareTransientError:

  [WRN] t:TestUpdateHeartbeatDocClusterAwareTransientError/startup Failed to
  update background manager status in periodic polling: this Rosmar bucket
  has been closed, will retry

- The startup subtest closed the bucket before it stopped the manager.
- Stop does not wait, so the status poller kept running.
- Other background manager tests had the same pattern.

Fix:
- Add synchronousStop. It stops the manager and waits for it to finish.
- Use it in the background manager tests before the bucket closes.
- Close the leaky bucket in a t.Cleanup that runs after the manager stops.

Not the final fix:
- Nothing tracks the run goroutine or the heartbeat loop.
- Tracking them in BackgroundManager would let callers wait directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An unconditional stop can recreate a deleted heartbeat during terminal cleanup, causing the removal wait to exceed its default 10-second timeout.

1 open issue
What changed in this PR

This test-only PR reduces background-manager cleanup races that can produce bucket-closed warnings and affect later tests’ global warning-count assertions.

Changes:

  • Add a stop helper that waits for terminal state, persisted status, and single-node heartbeat removal.
  • Use the helper across background-manager tests and close the leaky bucket after manager cleanup.
  • Release the resync options read lock before deferred shutdown.
File Description
db/​util_testing.go Add synchronous stop helper and align heartbeat-removal timeout with other state-transition waits.
db/​background_mgr_test.go Wait for manager cleanup and correct bucket-close ordering.
db/​background_mgr_resync_dcp_test.go Wait for standalone managers to stop and narrow the options read-lock scope.

Review effort: Balanced

Comment thread db/util_testing.go Resolved
@torcolvin
torcolvin marked this pull request as ready for review November 11, 2026 00:01
@torcolvin
torcolvin requested review from a team November 11, 2026 00:01
@factory-droid

factory-droid Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Droid finished @torcolvin's task —— View job


Validated all candidates (none provided). No new inline review comments to post for this PR.

0 inline comments posted

@bbrks
bbrks merged commit 1106d52 into main Nov 11, 2026
54 checks passed
@bbrks
bbrks deleted the CBG-6075 branch November 11, 2026 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants