[CI] Windows Integration loses VSM-XAM-147 agent heartbeat

[dalexsoto]

Failure signature

The Windows Integration Tests Dotnet tests job is abandoned when Azure DevOps loses contact with VSM-XAM-147:

We stopped hearing from agent VSM-XAM-147. Verify the agent machine is running and has a healthy network connection.

Confirmed occurrences

PR #26020

• Head: 52aa6b725350ec79b8019d3d5e8523c817be9b27
• Build 15180927, Windows Integration Tests / Dotnet tests
• Azure abandoned the job after losing the agent heartbeat. A full Windows-stage attempt 2 is active on a different worker.

PR #26240

• Head: 69ff5adf1625c166175164e1c040e45d2defca35
• Build 15181530, Windows Integration Tests / Dotnet tests
• Azure reported the same VSM-XAM-147 heartbeat-loss message and marked the job abandoned before tests produced a result.

First observed: 2026-09-02T07:59:57Z.
Latest confirmed occurrence: 2026-09-02T11:09:57Z.
Recurrence: 2 jobs across 2 distinct monitored PRs.

Classification and impact

This is a CI infrastructure/agent failure, not a source or deterministic test failure. The same worker heartbeat symptom affects unrelated PRs and prevents the Windows test job from completing.

Investigation performed

• Inspected both Azure build timelines, stage/job states, agent identifiers, and exact error text.
• Confirmed the source changes in the affected PRs are unrelated to Azure agent connectivity.
• Searched all open and closed dotnet/macios issues separately using the exact error, worker name, Windows stage, and agent-network wording. No matching issue existed.

Workaround

Rerun the entire Windows Integration Tests stage, never only the failed job, after the build is terminal and mergeability/retry-budget gates permit it. Prefer a healthy worker rather than retrying on VSM-XAM-147.

Azure diagnostic retention is limited.

[dalexsoto]

Retry status for PR #26240 @ 69ff5adf1625c166175164e1c040e45d2defca35:

• After refreshing the unchanged conflict-free head, terminal run, complete attempt history, and zero prior retries, Azure accepted build 15181530 windows_integration attempt 2 with forceRetryAllJobs=true, rerunning the entire Windows Integration stage.
• The bot-reservation job is active on VSM-XAM-09.Tahoe.arm64; the Windows test job has not started yet.
• A separate request reruns T: framework for the unrelated Xcode platform-download hang tracked by [CI] xcodebuild -downloadPlatform tvOS hangs until test-job timeout #26533.
• Retry usage is 2 PR-lifetime / 2 current-head requests across both accepted stage requests.

No equivalent retry will be submitted while attempt 2 is active. The issue body has no Additional info to collect checklist.

[dalexsoto]

Retry outcome for PR #26020 @ 52aa6b725350ec79b8019d3d5e8523c817be9b27:

• Build 15180927, full Windows Integration attempt 2 ran Dotnet tests on VSM-XAM-136; the VSM-XAM-147 heartbeat-loss signature did not recur.
• Artifact download, local .NET tests, remote .NET tests, and generator tests all completed successfully. The job instead reached its separate 120-minute budget during Publish to Artifact Services Drop, which Azure canceled.
• That new duration-budget failure is retained as a distinct one-PR signature and is not being grouped into [CI] Windows Integration loses VSM-XAM-147 agent heartbeat #26534. The stage and run are still finalizing, so no further retry is permitted yet.
• PR [msbuild] Add AppManifestEntry build items. Fixes #12788 #26240's full Windows attempt 2 remains active on VSM-XAM-141.

Canonical count for the VSM-XAM-147 heartbeat signature remains 2 occurrences across 2 PRs. The issue body has no Additional info to collect checklist.

[dalexsoto]

Terminal retry outcome for PR #26240 @ 69ff5adf1625c166175164e1c040e45d2defca35:

• Build 15181530, full Windows Integration attempt 2 completed successfully on VSM-XAM-141 at 2026-09-02T13:15:34Z.
• Dotnet tests passed at 13:13:18Z; the VSM-XAM-147 heartbeat-loss signature did not recur.
• The full build completed partiallySucceeded because Azure retained prior-attempt state, while both requested retry stages succeeded. Retry usage remains 2 PR-lifetime / 2 current-head requests.

The canonical count remains 2 occurrences across 2 PRs. The issue body has no Additional info to collect checklist.