fix(ec2): wait for candidate NICs on Graviton and fast-booting instances

[vpashaiev]

Proposed Commit Message

fix(ec2): wait for candidate NICs on Graviton instances

PR #7065 added wait_for_candidate_nics() for hpc7a.96xlarge, but fast-booting
Graviton instances (m8g, r8g, c8g, etc.) still hit the race condition where
cloud-init-local runs before ena finishes PCI probe. When that happens,
DataSourceEc2 exits without writing network config.

Poll for candidate NICs on Graviton instances by matching product prefixes
(m8g., r8g., c8g., c7g., m7g., r7g., c6g., m6g., r6g., t4g.) in DMI system-product-name.

Fixes GH-6697
LP: #2151279

Test Steps

1. Simulate delayed ENA probe:

echo "install ena /usr/bin/sleep 2; /sbin/modprobe --ignore-install ena" > /etc/modprobe.d/ena-delay.conf
cloud-init clean --logs && cloud-init init --local

2. Run unit tests:

pytest tests/unittests/sources/test_ec2.py

Merge type

• Squash merge using "Proposed Commit Message"
• Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

[blackboxsw]

Thank you for the submission @vpashaiev. A couple of comments, generally we hope to avoid adding more supported configuration options in datasource config if we don't have a use-case in hand where this config option is necessary. If we expect this to be a frequently used config option, then we can stand by the added support here. Otherwise, I'd back that element out.

[blackboxsw]

Do we have use-cases where we want this to be a configurable option that is longer than the default? If not, I would like to avoid adding more supported configuration options to datasource config and just set this timeout to 60.

If we do need this configuration option, this value should be sourced from self.ds_cfg and we also need to update doc/rtd/reference/datasources/ec2.rst "Configuration settings" to document this option.

[blackboxsw]

we should probably use pytest.mark.parametrize for read_dmi.return_value providing a couple of expected success cases and one non-matching case.

[vpashaiev]

Thanks for the review @blackboxsw!

Updated:

1. Dropped the custom datasource config knobs, kept default 60s timeout.
2. Parametrized test_ec2_local_waits_for_graviton_product with Graviton instances and t3.micro negative case, added unit test for _is_nic_polling_product.

[phils]

@vpashaiev @blackboxsw Thanks for looking at this! One concern with restricting the allowlist to 7th/8th-gen Gravitons: as noted by Robin Wallin in Launchpad #2151279, this is already failing in production on Graviton 2 and 3 instances as well. t4g (Graviton 2) in particular is one of the most widely used instance types in AWS, and I experienced this issue just the other day on a t4g.medium.

Could we instead make the allowlist either cover all Graviton families (e.g. matching g or checking architecture aarch64 on EC2), or better yet, matches any EC2 instance where an ENA PCI device is present but unprobed? Otherwise t4g, c6g, and m6g will still suffer from this race (and probably other exotic types). Worst case I think adding the t4g and the *6g would be a middle ground.

[vpashaiev]

Added t4g, c6g, m6g, and r6g to the product prefix list.