Skip to content

fix(namespace): converge anti-affinity group instead of blind-writing - #427

Open
dev-ankit wants to merge 1 commit into
streamnative:mainfrom
dev-ankit:fix/converge-anti-affinity-group
Open

dev-ankit wants to merge 1 commit into
streamnative:mainfrom
dev-ankit:fix/converge-anti-affinity-group

Conversation

@dev-ankit

Copy link
Copy Markdown

Motivation

When spec.antiAffinityGroup is unset, ApplyNamespace deletes the namespace's anti-affinity group on every reconcile:

} else {
    // Remove anti-affinity group if not specified
    err = p.adminClient.Namespaces().DeleteNamespaceAntiAffinityGroup(completeNSName)
    if err != nil && !IsNotFound(err) {
        return err
    }
}

Pulsar implements that delete as a read-modify-write of the namespace's local policies. A namespace has no local policies until one of its bundles is first loaded (a topic is created or looked up), so on a freshly created namespace the broker answers 500, not 404:

DELETE /admin/v2/namespaces/dev/exa-search-a6073586/antiAffinity HTTP/1.1" 500
org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException ... MetadataCacheImpl.lambda$readModifyUpdate$8
  at NamespacesBase.internalRemoveNamespaceAntiAffinityGroup

The 500 isn't tolerated, ApplyNamespace fails, and a new PulsarNamespace that doesn't use anti-affinity never becomes Ready. Anything waiting on it never proceeds either, for example PulsarTopics applied in a later Argo CD sync wave. Its topics would have created the local policies, so the namespace never recovers. Reproduced on Pulsar 4.0.8 standalone with v0.21.2.

This is the same blind-write pattern #421 fixed for bookieAffinityGroup, applied to the neighbouring anti-affinity setting.

Modifications

applyNamespaceAntiAffinityGroup reads the current group first and writes only when it differs from the spec:

spec current action
unset none nothing (the case that failed)
unset set DELETE (404 still tolerated)
set equal nothing
set none or different POST

getNamespaceAntiAffinityGroup decodes Pulsar's answer: 204 with no body when no group is set, otherwise the group as a JSON string. A 404 is treated as "no group". Any other read failure is returned rather than mistaken for "unset".

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

  • pkg/admin/anti_affinity_group_test.go, an httptest table test in the style of bookie_affinity_group_test.go. The fake broker returns 500 on DELETE when the namespace has no local policies, as Pulsar does. It covers each row above plus a read failure (403) that must surface. Removing the "unset and none → nothing" guard makes the two "unset stays unset" cases fail with the 500.
  • Against a real Pulsar 4.0.8 standalone and a kind cluster running this operator, with a suite that creates a PulsarNamespace with no anti-affinity group and then its PulsarTopics:
    • Stock v0.21.2: the namespace never becomes Ready (the 500 above).
    • With this change: it becomes Ready. The broker saw only GETs on antiAffinity, with no DELETEs.
  • Also checked on that broker: setting a group on an existing namespace, changing it and removing it each end Ready, and the broker's GET .../antiAffinity reflects the spec each time ("g-a", "g-c", then 204).

Documentation

  • no-need-doc

    Behaviour of the existing field is unchanged apart from not failing; no API or CRD change.

When spec.antiAffinityGroup was unset, ApplyNamespace deleted the
namespace's anti-affinity group on every reconcile. Pulsar implements
that delete as a write to the namespace's local policies, and on a
namespace without local policies yet (no topic created) the broker
answers 500 (MetadataStoreException$NotFoundException), so a freshly
created PulsarNamespace never becomes Ready.

Read the current group first and write only on a difference, as streamnative#421
did for the bookie affinity group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dev-ankit
dev-ankit requested review from a team as code owners October 15, 2026 22:38
@github-actions github-actions Bot added the no-need-doc This pr does not need any document label Oct 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-need-doc This pr does not need any document

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant