Skip to content

HBASE-27486 Fix per-table MetricsTableLatencies/QueryMeter memory leak - #8648

Open
haosenchen wants to merge 1 commit into
apache:branch-2.5from
haosenchen:HBASE-27486-branch-2.5
Open

haosenchen wants to merge 1 commit into
apache:branch-2.5from
haosenchen:HBASE-27486-branch-2.5

Conversation

@haosenchen

Copy link
Copy Markdown
Contributor

Backport of the registry-cleanup mechanism only. The per-table MetricsTableLatenciesImpl.histogramsByTable and
MetricsTableQueryMeterImpl.metersByTable are populated on first access but never cleaned up when a table leaves the RegionServer (drop / disable / move / region close). For short-lived tables (e.g. hourly tables with a 7-day retention window) this leaks both the maps and the per-table entries under the sub=TableLatencies JMX bean without bound.

  • MetricsTableLatencies / MetricsTableQueryMeter: add deleteTable(...) on both interfaces and their impls.
  • MetricsTableLatenciesImpl: switch histogramsByTable to ConcurrentHashMap + computeIfAbsent (also closes a pre-existing TODO) and implement deleteTable(String) which removes the entry AND unregisters the 12 MutableTimeHistogram / MutableSizeHistogram objects from the underlying DynamicMetricsRegistry via removeMetric(baseName). removeMetric is used rather than removeHistogramMetrics because the histogram is stored under baseName itself and the 14 suffix metrics (_num_ops / _sum_ops / _min / _max / _mean / *_percentile) are produced dynamically at snapshot time -- removeHistogramMetrics only tries to remove baseName + 8 fixed suffix keys that never actually exist in the map, so it is a no-op for MutableHistogram entries.
  • MetricsTableQueryMeterImpl: implement deleteTable(TableName) that removes the entry and unregisters both read/write meters from the shared MetricRegistry.
  • RegionServerTableMetrics: expose a single deleteTable(TableName) that forwards to both cleanup paths.
  • MetricsRegionServer: expose getRegionServerTableMetrics() so the cleanup driver can reach the container.
  • MetricsTableWrapperAggregateImpl.TableMetricsWrapperRunnable: compute onlineTables at the top of run(), take the difference with a new lastSeenTables snapshot, and invoke RegionServerTableMetrics.deleteTable() for every table that has left the RegionServer since the previous scheduled run. The cleanup path runs before the aggregate metrics collection body so it also fires for users that only enabled per-table latency / query meters and not the aggregate table metrics.

Tests

  • TestMetricsTableLatencies#testDeleteTableRemovesAllLatencyHistograms (new, unit): verifies that all 12 histogram families of the dropped table disappear, other tables are untouched, re-adding samples lazily re-registers the histograms, and deleting an unknown table is a no-op.

  • TestMetricsTableWrapperCleanup (new, MediumTests, end-to-end): wires a mocked HRegionServer + real RegionServerTableMetrics + real MetricsTableLatenciesImpl and hand-drives two rounds of TableMetricsWrapperRunnable across an online-regions transition {keep_table, drop_table} -> {keep_table}. Asserts that every latency histogram and both query meters of drop_table have vanished from the metrics registry / JMX snapshot after round 2 while keep_table is preserved.

Notes

HBASE-27681 (the upstream fix for HBASE-27486 on branch-2.6+) is not backported wholesale because it renames the JMX beans (sub=TableLatencies -> sub=TableRequests_

) which is not backward-compatible for existing monitoring. This change intentionally keeps the JMX bean names and metric key layout stable and only backports the registry cleanup mechanism.

Backport of the registry-cleanup mechanism only. The per-table
MetricsTableLatenciesImpl.histogramsByTable and
MetricsTableQueryMeterImpl.metersByTable are populated on first access
but never cleaned up when a table leaves the RegionServer (drop /
disable / move / region close). For short-lived tables (e.g. hourly
tables with a 7-day retention window) this leaks both the maps and the
per-table entries under the sub=TableLatencies JMX bean without bound.

* MetricsTableLatencies / MetricsTableQueryMeter: add deleteTable(...)
  on both interfaces and their impls.
* MetricsTableLatenciesImpl: switch histogramsByTable to
  ConcurrentHashMap + computeIfAbsent (also closes a pre-existing TODO)
  and implement deleteTable(String) which removes the entry AND
  unregisters the 12 MutableTimeHistogram / MutableSizeHistogram objects
  from the underlying DynamicMetricsRegistry via removeMetric(baseName).
  removeMetric is used rather than removeHistogramMetrics because the
  histogram is stored under baseName itself and the 14 suffix metrics
  (_num_ops / _sum_ops / _min / _max / _mean / *_percentile) are
  produced dynamically at snapshot time -- removeHistogramMetrics only
  tries to remove baseName + 8 fixed suffix keys that never actually
  exist in the map, so it is a no-op for MutableHistogram entries.
* MetricsTableQueryMeterImpl: implement deleteTable(TableName) that
  removes the entry and unregisters both read/write meters from the
  shared MetricRegistry.
* RegionServerTableMetrics: expose a single deleteTable(TableName) that
  forwards to both cleanup paths.
* MetricsRegionServer: expose getRegionServerTableMetrics() so the
  cleanup driver can reach the container.
* MetricsTableWrapperAggregateImpl.TableMetricsWrapperRunnable: compute
  onlineTables at the top of run(), take the difference with a new
  lastSeenTables snapshot, and invoke
  RegionServerTableMetrics.deleteTable() for every table that has left
  the RegionServer since the previous scheduled run. The cleanup path
  runs before the aggregate metrics collection body so it also fires
  for users that only enabled per-table latency / query meters and not
  the aggregate table metrics.

Tests
-----
* TestMetricsTableLatencies#testDeleteTableRemovesAllLatencyHistograms
  (new, unit): verifies that all 12 histogram families of the dropped
  table disappear, other tables are untouched, re-adding samples
  lazily re-registers the histograms, and deleting an unknown table is
  a no-op.

* TestMetricsTableWrapperCleanup (new, MediumTests, end-to-end):
  wires a mocked HRegionServer + real RegionServerTableMetrics + real
  MetricsTableLatenciesImpl and hand-drives two rounds of
  TableMetricsWrapperRunnable across an online-regions transition
  {keep_table, drop_table} -> {keep_table}. Asserts that every latency
  histogram and both query meters of drop_table have vanished from the
  metrics registry / JMX snapshot after round 2 while keep_table is
  preserved.

Notes
-----
HBASE-27681 (the upstream fix for HBASE-27486 on branch-2.6+) is not
backported wholesale because it renames the JMX beans
(sub=TableLatencies -> sub=TableRequests_<table>) which is not
backward-compatible for existing monitoring. This change intentionally
keeps the JMX bean names and metric key layout stable and only
backports the registry cleanup mechanism.

Signed-off-by: haosenchen <haosenchen@tencent.com>
@haosenchen

haosenchen commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

It seems that the root cause is "The Yetus precommit script is passing flags that are not recognized by the current version of Yetus being used. This indicates a version incompatibility between the HBase patch submission script and the Yetus tool.". Can someone help handle this?
Clipboard_Screenshot_1789354909

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant