HIVE-30045: K8s operator: missing LLAP scheduler config breaks task affinity and caps cache hits - #6779
Open
deniskuzZ wants to merge 1 commit into
Open
HIVE-30045: K8s operator: missing LLAP scheduler config breaks task affinity and caps cache hits#6779deniskuzZ wants to merge 1 commit into
deniskuzZ wants to merge 1 commit into
Conversation
…ffinity and caps cache hits A standalone Tez AM configures its LLAP task scheduler, task communicator and split location provider from the AM pod's tez-site.xml. It never loads hive-site, so hive.llap.task.*, hive.llap.client.* and hive.llap.daemon.communicator.* set in the HiveServer2 configOverrides are absent from the AM's configuration and every such key runs at its HiveConf default. That includes hive.llap.task.scheduler.locality.delay, which defaults to 0: a task whose consistent-hash daemon is busy goes to the next host immediately. Under saturation only about a third of tasks land on the daemon holding their data, and since a task reads from the cache of the daemon it runs on, the cache hit rate is pinned to the same figure no matter how large the cache is -- measured 38% data-local and a 37% hit rate on a 10 TB TPC-DS scan. Copy the keys those plugins read into tez-site, keeping tezAm overrides authoritative. With the delay reaching the scheduler, the same scan placed 1421 of 1422 tasks locally and hit 98.5% in cache.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
A standalone Tez AM builds its LLAP plugins — task scheduler, task communicator, split location provider — from the AM pod's
tez-site.xml, and never loads hive-site.So
hive.llap.task.*,hive.llap.client.*andhive.llap.daemon.communicator.*set in the HiveServer2configOverridesrun at their HiveConf defaults.That includes
hive.llap.task.scheduler.locality.delay, which capped the LLAP cache hit rate at the task locality rate: 38% data-local, 37% hits on a 10 TB TPC-DS scan.Copy the keys those plugins read into tez-site before applying
tezAm.configOverrides, which remain authoritative. The same scan then placed 1421 of 1422 tasks locally and hit 98.5% in cache.Why are the changes needed?
missing LLAP scheduler config breaks task affinity and caps cache hits
Does this PR introduce any user-facing change?
No
How was this patch tested?
on a cluster