fix (EC2/GCE): wait for NIC on allowlisted platforms

[goldberl]

Proposed Commit Message

fix (EC2/GCE): wait for NIC on allowlisted platforms

On some GCE and AWS EC2 instances, cloud-init-local runs before
network interfaces are fully initialized by the kernel. This causes
early datasource detection to fail, preventing metadata fetching and
SSH access.

To avoid introducing boot delays across all platforms, implement
allowlist-gated NIC polling and centralize the retry logic in a shared
network helper:
 * add wait_for_candidate_nics() in cloudinit/net/init.py
 * use helper from DataSourceEc2 and DataSourceGCE when allowlisted
 * EC2 gate: DMI system-product-name (e.g. hpc7a.96xlarge)
 * GCE gate: DMI baseboard-product-name (e.g. izumi)

Fixes first-boot race conditions on affected AWS and GCP instances
without impacting unaffected instance types or breaking unit tests.

Add unit coverage for:
 * helper retry/timeout behavior
 * EC2 allowlisted vs non-allowlisted polling paths
 * GCE allowlisted vs non-allowlisted polling paths

Fixes: GH-6697, GH-6737, LP-2144694

Signed-off-by: Leah Goldberg <leah.goldberg@canonical.com>

Additional info

• [bug]: EC2 datasource failing on first boot due to missing NICs #6697
• [bug] DataSourceGCELocal fails on GCE c3-metal instances due to NIC not ready at first boot #6737
• https://bugs.launchpad.net/bugs/2144694

This fix will emit logs on affected AWS and GCP instances when NIC polling is enabled, such as:

# AWS (Ubuntu Noble Minimal - hpc7a.96xlarge)
# Race condition reproduced on boot (~100% reproduction rate)
# The allowlist check was triggered, the initial check detected no
# primary NICs, and the interface was successfully acquired after
# 1 retry (1.001s total duration).


ubuntu@ip-172-31-24-66:~$ sudo grep -iE "Waiting for candidate NICs|No primary NICs|Candidate NIC polling|Timed out after|Looking for the primary NIC in" /var/log/cloud-init.log

2026-09-11 12:15:36,768 - net[DEBUG]: Waiting for candidate NICs with carrier for up to 60 seconds

2026-09-11 12:15:36,768 - net[DEBUG]: No primary NICs found with carrier, waiting 1 seconds to retry

2026-09-11 12:15:37,769 - net[DEBUG]: Candidate NIC polling completed in 1.001 seconds (timeout=60, sleep_interval=1): ['enp34s0']

2026-09-11 12:15:37,769 - DataSourceEc2.py[DEBUG]: Looking for the primary NIC in: ['enp34s0']

---------------------------------------

# GCP (Ubuntu Noble - izumi / c3-metal)
# Race condition did not hit (~1% reproduction rate)
# The allowlist check was verified (timeout=60), and the candidate
# NIC was discovered on the initial polling attempt without needing
# a retry loop.

ubuntu@goldberl-c3-noble-test-0:~$ sudo grep -iE "Waiting for candidate NICs|No primary NICs|Candidate NIC polling|Timed out after|Looking for the primary NIC in" /var/log/cloud-init.log

2026-09-11 12:21:46,199 - DataSourceGCE.py[DEBUG]: Looking for the primary NIC in: ['enp5s0f0']

Merge type

• Squash merge using "Proposed Commit Message"
• Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

[blackboxsw]

Thank you for all this work @goldberl! I think this looks like a contained approach which limits exposure to boot-time costs to known instance types.

I have a number of requests and discussion points inline. Please do feel free to push back on things that you think are unreasonable.

Can you also please add a comment with the related log entries emitted by this PR on a working instance?

[goldberl]

Hi @blackboxsw, thank you for the detailed review. I've made the following changes to the PR:

• Used performance.Timed to track and log execution duration.
• Added [bug]: EC2 datasource failing on first boot due to missing NICs #6697 and [bug] DataSourceGCELocal fails on GCE c3-metal instances due to NIC not ready at first boot #6737 reference in comments for context.
• Updated docstring for wait_for_candidate_nics to mention carrier checks and documented behavior when timeout=0.
• Updated debug logs in wait_for_candidate_nics to mention IMDS primary NICs and added timeout exceeded logs when timeout > 0.
• Cleaned up DMI lookup by removing unnecessary or "" fallbacks.
• Simplified flow in EC2 and GCE datasources to always invoke net.wait_for_candidate_nics(), passing timeout=60 for allowlisted platforms and timeout=0 for non-allowlisted platforms (option 3 from your comment).
• Simplified the polling loop conditional to while not candidate_nics:, running find_candidate_nics() inside and breaking on success or timeout.
• Updated EC2, GCE, and net unit tests to cover the unified flow and new log conditions.

I'll run tests on AWS and GCP instances to capture updated logs and update the PR description with those

[blackboxsw]

@goldberl thank you for this iteration.

While looking at the end product here, the use of timeout as dual purpose where 0 means just run once and > 0 means max_wait it makes it a bit harder to discern the intent of the timeout param.

I think we may need to go with the alternative solution to pre-flight check:

if not wait_on_nics:
   find_candidate_nics()
else:
   wait_on_candidate_nics() 

That should then ensure a clear path between conditions which require calling wait_for_candidate_nics versus those we expect to succeed on a single call.

[goldberl]

Thank you again @blackboxsw for your continuous feedback. I've updated the PR with the following changes:

• Avoided using timeout=0 in wait_for_candidate_nics(). Now datasources run net.find_candidate_nics() for standard runs or net.wait_for_candidate_nics() when the product/board is in the allowlist.
• Added .lower() handling when evaluating system-product-name and baseboard-product-name
• Refactored wait_for_candidate_nics()
  • Calls find_candidate_nics() instantly
  • Updated log messages to say NIC is waiting for carrier rather than assuming IMDS access
  • Centralized logs in wait_for_candidate_nics() to remove log duplication across datasources
• Mocked time.monotonic in test_init.py to prevent host leaks
• Updated unit tests in EC2, GCE, and net modules to reflect the unified log messages and ensure non-allowlisted runs call find_candidate_nics() directly.

[goldberl]

Simplified logic in datasource files with a straightforward if/else block for better readability.

[blackboxsw]

@goldberl this looks great. Thank you for the changes here. I think this makes things much easier to understand at the call-sites in GCE and Ec2 datasources.
One minor nit on when we should enter the perfomance.Timed context manager and then we can merge this.

I ran the full suite of integration tests with this changeset and see no degradation in behavior for standard instance types.

[goldberl]

@blackboxsw Thanks for the review again! Made the small change in cloudinit/net/__init__.py that you suggested.

[blackboxsw]

Thank you for the detailed work here @goldberl. Let's get this fix shipped!