1. Feature requested
BookKeeper AutoRecovery should recognize failed-Bookie underreplication tasks for
ledgers whose writeQuorumSize == 1 and avoid entering an endless recovery loop
when the failed Bookie is permanently unavailable.
For these ledgers, each entry has only one data source. Once that source is
permanently lost, AutoRecovery cannot reconstruct the missing data from another
Bookie. The requested behavior is therefore to:
- avoid publishing new failed-Bookie underreplication tasks for known
single-replica ledgers;
- remove matching historical failed-Bookie tasks created before this behavior
was introduced;
- always release the underreplication lock during the cleanup path;
- preserve placement-policy repair tasks, including
writeQuorumSize == 1
tasks with an empty replica list; and
- expose dedicated Auditor and ReplicationWorker metrics so operators can
identify possible data loss and take manual recovery action.
This feature must not claim to recover data that has no surviving replica. It
should only stop the impossible automatic recovery path and make the condition
visible to operators.
2. Importance and workarounds
Importance: must-have for deployments that create ledgers with
writeQuorumSize == 1; should-have otherwise.
Without this behavior, a permanently lost Bookie leaves an underreplication task
that the Worker repeatedly locks, attempts to open or read, fails to recover,
releases the lock, and retries. This consumes Worker, BookKeeper client, and
ZooKeeper resources, produces noisy failure logs, and can delay processing of
ledgers that are actually recoverable.
There is no safe automatic workaround for the lost data itself. Operators can
restore the original Bookie or recover data from backups, but those options may
not be available. Disabling AutoRecovery or manually deleting the
underreplication task can stop the retry noise, but disabling AutoRecovery also
stops unrelated recovery and deleting the task can hide a data-loss condition.
3. Proposed use case
A cluster contains a ledger created with writeQuorumSize == 1. The ledger's
only Bookie fails permanently. The Auditor detects the failed Bookie, reads the
ledger metadata, and identifies that automatic reconstruction is impossible.
Instead of creating a task that will fail forever, the Auditor skips publication
and increments a dedicated metric.
During an upgrade, the cluster may already contain persistent underreplication
znodes created by an older BookKeeper version. When a Worker consumes one of
these historical tasks, it should inspect the task shape and ledger metadata,
clean up only failed-Bookie tasks for single-replica ledgers, release the lock,
and report a skipped outcome. A placement-policy task with an empty replica list
must continue through the normal replication path.
The metrics should allow operators to alert on skipped ledgers and decide
whether to restore the original Bookie, use backups, or apply an
application-specific recovery procedure.
1. Feature requested
BookKeeper AutoRecovery should recognize failed-Bookie underreplication tasks for
ledgers whose
writeQuorumSize == 1and avoid entering an endless recovery loopwhen the failed Bookie is permanently unavailable.
For these ledgers, each entry has only one data source. Once that source is
permanently lost, AutoRecovery cannot reconstruct the missing data from another
Bookie. The requested behavior is therefore to:
single-replica ledgers;
was introduced;
writeQuorumSize == 1tasks with an empty replica list; and
identify possible data loss and take manual recovery action.
This feature must not claim to recover data that has no surviving replica. It
should only stop the impossible automatic recovery path and make the condition
visible to operators.
2. Importance and workarounds
Importance: must-have for deployments that create ledgers with
writeQuorumSize == 1; should-have otherwise.Without this behavior, a permanently lost Bookie leaves an underreplication task
that the Worker repeatedly locks, attempts to open or read, fails to recover,
releases the lock, and retries. This consumes Worker, BookKeeper client, and
ZooKeeper resources, produces noisy failure logs, and can delay processing of
ledgers that are actually recoverable.
There is no safe automatic workaround for the lost data itself. Operators can
restore the original Bookie or recover data from backups, but those options may
not be available. Disabling AutoRecovery or manually deleting the
underreplication task can stop the retry noise, but disabling AutoRecovery also
stops unrelated recovery and deleting the task can hide a data-loss condition.
3. Proposed use case
A cluster contains a ledger created with
writeQuorumSize == 1. The ledger'sonly Bookie fails permanently. The Auditor detects the failed Bookie, reads the
ledger metadata, and identifies that automatic reconstruction is impossible.
Instead of creating a task that will fail forever, the Auditor skips publication
and increments a dedicated metric.
During an upgrade, the cluster may already contain persistent underreplication
znodes created by an older BookKeeper version. When a Worker consumes one of
these historical tasks, it should inspect the task shape and ledger metadata,
clean up only failed-Bookie tasks for single-replica ledgers, release the lock,
and report a skipped outcome. A placement-policy task with an empty replica list
must continue through the normal replication path.
The metrics should allow operators to alert on skipped ledgers and decide
whether to restore the original Bookie, use backups, or apply an
application-specific recovery procedure.