Skip to content

Start concurrent jobs 10s apart - #193

Merged
bbondy merged 1 commit into
masterfrom
stagger-parallel-job-starts
Oct 17, 2026
Merged

bbondy merged 1 commit into
masterfrom
stagger-parallel-job-starts

Conversation

@netzenbot

Copy link
Copy Markdown
Collaborator

User impact: with bot.concurrentJobs above 1, the jobs now start 10 seconds apart instead of all in the same instant.

The problem

./run.sh with bot.concurrentJobs above 1 launches every job back to back. All of them run start-up (PRD sync, label sweep, worktree cleanup) and pick a story at the same moment. The sync and the sweep sit behind a lock that makes a second caller skip rather than wait, so a job that loses it picks a story from a PRD the first job is still refreshing.

Reproduce

  1. In config.json set "bot": { "maxConcurrentRuns": 3, "concurrentJobs": 3 } with a worktree profile.
  2. Run ./run.sh 1 from the bot directory.
  3. Compare the Run slot N of 3 lines each job prints.

Before: the three lines appear within the same second, and the jobs' start-up overlaps. Expected: the jobs start one after another.

tests/test_parallel_jobs.py::TestStartDelay covers it — fails on the parent commit, passes here.

The fix

Each job after the first waits BOT_JOB_START_DELAY_SECONDS (default 10, 0 restores the old behaviour) after the one before it is started. The wait can be interrupted, so Ctrl+C or TERM during the gap stops the jobs already running at once and starts no more. Claims already stop two jobs working one story; this narrows the stale-PRD case and spreads the load, and does not remove it if a sync outlasts the gap.

Starting 3 parallel jobs (bot.concurrentJobs). Each takes a run slot of its own; their lines are prefixed [job N].
Each job starts 10s after the one before it.

Test plan

  • .venv/bin/python -m pytest tests/test_parallel_jobs.py — 43 passed
  • make lint — passed
  • make check-reviewdog — no findings (landrun missing, ran unsandboxed)
  • make test — 1393 passed, 9 failed: 8 in test_scripts.py and test_rebase_pr.py that already fail on master, and one new timing test I then fixed and re-ran 3 times
  • CI passes cleanly

Jobs started by bot.concurrentJobs launched back to back, so all of them ran
start-up and picked a story in the same instant, and a job whose PRD sync lost
the lock chose from a PRD the first job was still refreshing. Each job now
starts BOT_JOB_START_DELAY_SECONDS (default 10) after the one before it.
@bbondy
bbondy merged commit 6f81578 into master Oct 17, 2026
3 checks passed
@bbondy
bbondy deleted the stagger-parallel-job-starts branch October 17, 2026 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants