Queue the builds CDN purge instead of sleeping in the request - #308
Merged
Merged
Conversation
POST /api/artifacts/add purged Fastly inline, and Bob.Fastly.purge/3 sleeps 4 seconds between each of its three requests, so every artifact upload from otp.sh held its request open for about 8.1 seconds. Bob serves 9 to 14 other requests per 5 minutes, so one upload every 5 minutes puts more than 5% of requests in the 5000-10000ms histogram bucket, and a burst of OTP builds (up to 6 per ref) keeps p95 above 2000ms long enough to fire HttpLatencyHigh. Replaying the rule against the request logs from 2026-09-03 to 2026-09-22 predicts firings on 09-07, 09-09, 09-16, 09-17, 09-19 and 09-22, each during an OTP build burst. 449 of the 450 bob requests over 5 seconds in that period were this route. add/1 still saves the row and writes builds.txt during the request, then queues Bob.Job.PurgeBuilds with the surrogate keys. The job is exempt from backoff: nothing re-queues a purge, so backing off after a failure would skip the next build's purge of the same keys. The master only runs jobs listed in BOB_LOCAL_JOBS, and boot fails if a listed module doesn't exist, so the hexpm-ops change adding Bob.Job.PurgeBuilds to BOB_LOCAL_JOBS has to ship in the same commit as the image bump to this change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
POST /api/artifacts/addpurged Fastly inline, andBob.Fastly.purge/3sleeps 4 seconds between each of its three requests, so every artifact upload fromotp.shheld its request open for about 8.1 seconds. Bob serves 9 to 14 other requests per 5 minutes, so one upload every 5 minutes puts more than 5% of requests in the 5000-10000ms histogram bucket. A burst of OTP builds (up to 6 per ref) keeps p95 above 2000ms long enough to fireHttpLatencyHigh. Replaying the rule against the request logs inhexpm-prod.logs.stdoutfrom 2026-09-03 to 2026-09-22 predicts firings on 09-07, 09-09, 09-16, 09-17, 09-19 and 09-22, each during an OTP build burst. 449 of the 450 bob requests over 5 seconds in that period were this route.Bob.Artifacts.add/1still saves the row and writes builds.txt during the request, then queuesBob.Job.PurgeBuildswith the surrogate keys. The job is exempt from backoff: nothing re-queues a purge, so backing off after a failure would skip the next build's purge of the same keys.The master only runs jobs listed in
BOB_LOCAL_JOBS, and boot fails if a listed module doesn't exist. The hexpm-ops change addingBob.Job.PurgeBuildstoBOB_LOCAL_JOBShas to ship in the same commit as the image bump to this change.