fix: Orchestrator - Respect pipeline run cancellation when the node flag is lost

[grigoriipenkin-spec]

What

Cancel can be lost for an execution that is PENDING or RUNNING. The cancel API returns 200 but the container keeps running.

The running sweep now votes to terminate a node when the node flag or the flag of its pipeline run is desired_state=TERMINATED.

Why

terminate() commits desired_state=TERMINATED in extra_data of the run and its nodes. The orchestrator can hold an older copy of the node at the same time. When it sets a new status, the status history listener writes the whole extra_data from that old copy and the node flag is gone. The running sweep only read the node flag, so it never terminated the container.

The run flag is not touched by that write, so checking it too keeps the cancel.

Associated Issue

Shopify/ml-infrastructure#1316

Dev Notes

• No schema change. A dedicated column would stop the node flag from being dropped, but manual migrations are on hold until Alembic.
• A container execution shared by several runs is still terminated only when every run votes to terminate.
• Shopify/oasis-backend#754 fixes an Oasis write that could drop the run flag right after submit.
• Not covered here: the node flag can still be dropped, and nodes without a pipeline run rely on it only. Cache reuse, shared executions and concurrent completion can still end with a non-CANCELLED status after cancel.

Tests

• Cancel races a stale status write: the node flag is dropped and the container is still terminated.
• Two runs share one container execution through cache: cancelling one run does not terminate it, cancelling both does.