HBASE-27486 Fix per-table MetricsTableLatencies/QueryMeter memory leak - #8648
Open
haosenchen wants to merge 1 commit into
Open
haosenchen wants to merge 1 commit into
haosenchen wants to merge 1 commit into
Conversation
Backport of the registry-cleanup mechanism only. The per-table
MetricsTableLatenciesImpl.histogramsByTable and
MetricsTableQueryMeterImpl.metersByTable are populated on first access
but never cleaned up when a table leaves the RegionServer (drop /
disable / move / region close). For short-lived tables (e.g. hourly
tables with a 7-day retention window) this leaks both the maps and the
per-table entries under the sub=TableLatencies JMX bean without bound.
* MetricsTableLatencies / MetricsTableQueryMeter: add deleteTable(...)
on both interfaces and their impls.
* MetricsTableLatenciesImpl: switch histogramsByTable to
ConcurrentHashMap + computeIfAbsent (also closes a pre-existing TODO)
and implement deleteTable(String) which removes the entry AND
unregisters the 12 MutableTimeHistogram / MutableSizeHistogram objects
from the underlying DynamicMetricsRegistry via removeMetric(baseName).
removeMetric is used rather than removeHistogramMetrics because the
histogram is stored under baseName itself and the 14 suffix metrics
(_num_ops / _sum_ops / _min / _max / _mean / *_percentile) are
produced dynamically at snapshot time -- removeHistogramMetrics only
tries to remove baseName + 8 fixed suffix keys that never actually
exist in the map, so it is a no-op for MutableHistogram entries.
* MetricsTableQueryMeterImpl: implement deleteTable(TableName) that
removes the entry and unregisters both read/write meters from the
shared MetricRegistry.
* RegionServerTableMetrics: expose a single deleteTable(TableName) that
forwards to both cleanup paths.
* MetricsRegionServer: expose getRegionServerTableMetrics() so the
cleanup driver can reach the container.
* MetricsTableWrapperAggregateImpl.TableMetricsWrapperRunnable: compute
onlineTables at the top of run(), take the difference with a new
lastSeenTables snapshot, and invoke
RegionServerTableMetrics.deleteTable() for every table that has left
the RegionServer since the previous scheduled run. The cleanup path
runs before the aggregate metrics collection body so it also fires
for users that only enabled per-table latency / query meters and not
the aggregate table metrics.
Tests
-----
* TestMetricsTableLatencies#testDeleteTableRemovesAllLatencyHistograms
(new, unit): verifies that all 12 histogram families of the dropped
table disappear, other tables are untouched, re-adding samples
lazily re-registers the histograms, and deleting an unknown table is
a no-op.
* TestMetricsTableWrapperCleanup (new, MediumTests, end-to-end):
wires a mocked HRegionServer + real RegionServerTableMetrics + real
MetricsTableLatenciesImpl and hand-drives two rounds of
TableMetricsWrapperRunnable across an online-regions transition
{keep_table, drop_table} -> {keep_table}. Asserts that every latency
histogram and both query meters of drop_table have vanished from the
metrics registry / JMX snapshot after round 2 while keep_table is
preserved.
Notes
-----
HBASE-27681 (the upstream fix for HBASE-27486 on branch-2.6+) is not
backported wholesale because it renames the JMX beans
(sub=TableLatencies -> sub=TableRequests_<table>) which is not
backward-compatible for existing monitoring. This change intentionally
keeps the JMX bean names and metric key layout stable and only
backports the registry cleanup mechanism.
Signed-off-by: haosenchen <haosenchen@tencent.com>
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Backport of the registry-cleanup mechanism only. The per-table MetricsTableLatenciesImpl.histogramsByTable and
MetricsTableQueryMeterImpl.metersByTable are populated on first access but never cleaned up when a table leaves the RegionServer (drop / disable / move / region close). For short-lived tables (e.g. hourly tables with a 7-day retention window) this leaks both the maps and the per-table entries under the sub=TableLatencies JMX bean without bound.
Tests
TestMetricsTableLatencies#testDeleteTableRemovesAllLatencyHistograms (new, unit): verifies that all 12 histogram families of the dropped table disappear, other tables are untouched, re-adding samples lazily re-registers the histograms, and deleting an unknown table is a no-op.
TestMetricsTableWrapperCleanup (new, MediumTests, end-to-end): wires a mocked HRegionServer + real RegionServerTableMetrics + real MetricsTableLatenciesImpl and hand-drives two rounds of TableMetricsWrapperRunnable across an online-regions transition {keep_table, drop_table} -> {keep_table}. Asserts that every latency histogram and both query meters of drop_table have vanished from the metrics registry / JMX snapshot after round 2 while keep_table is preserved.
Notes
HBASE-27681 (the upstream fix for HBASE-27486 on branch-2.6+) is not backported wholesale because it renames the JMX beans (sub=TableLatencies -> sub=TableRequests_
) which is not backward-compatible for existing monitoring. This change intentionally keeps the JMX bean names and metric key layout stable and only backports the registry cleanup mechanism.