Summary

Envoy shares one upstream connection pool per backend pod across all client connections, but Cilium's SourceAddressSocketOption decides "bind the Ingress IP or not" from the destination of whichever client connection created the pool, not from the connection being opened. When that stale destination is a local pod, connections to a remote pod leave without the Ingress IP, get policy-denied as reserved:remote-node, and turn into 503s after 5s connection timeout.

Observed Behavior

We run Gateway API with a default-deny CNP that only allows ingress from reserved:ingress. We noticed that under load when latency increases we get some 503s without requests reaching our backend. In hubble logs we noticed that some requests get policy denied

{
  "flow": {
    "verdict": "DROPPED",
    "source": {
      "identity": 6
    },
    "destination": {
      "identity": 187427,
      "pod_name": "pod-xxx"
    },
    "traffic_direction": "INGRESS",
    "Summary": "TCP Flags: SYN"
  }
} 

Identity 6 is reserved:remote-node. That was very unexpected and suspicious to see an identity other than 8 (reserved:ingress). That shouldn't happen in our setup.

My understanding of the root cause

The request flow is: client -> NLB -> Envoy -> one of the backend pods. The backend pod can be on the same node as Envoy or on a remote node.

1. Client A -> NLB -> Envoy -> remote pod

Let's say the backend pod is on another node. A new connection pool to this pod is created. The pool keeps a reference to client A's socket option, and that option holds a pointer to an object storing the destination of client A's current request. That pointer is checked every time the pool opens a new connection to decide whether to bind the Ingress IP or skip it because the destination is local.

This request succeeds.

2. Client A -> NLB -> Envoy -> local pod

Client A's next request gets routed to a pod on Envoy's own node. Client A's destination object now says "local", and stays that way until client A's next request

This request succeeds.

3. Client B -> NLB -> Envoy -> remote pod

Client B's request lands on the same Envoy and is routed to the same remote pod as in (1). All connections in that pool are busy, so a new one is opened. Envoy follows the pointer to client A's destination, sees "local", and skips binding the Ingress IP. Client B's SYN is sent with the node IP, arrives at the remote node with identity 6 (remote-node) instead of 8 (ingress), and our CNP does not allow that. The SYN is dropped, connection hangs and after 5s (envoy connection timeout) client might recieve a 503.

As I understand, most of the time this does not cause 503s, because another connection in the pool frees up and takes the request. But if within the 5s connection timeout nothing frees up, client B gets a 503. That is probably why policy denies with remote-node identity happen often in our cluster, while 503s stay quite rare.

What I tested to confirm the root cause

I did some tests and if I place the application pods in a way that they are never on the same nodes with envoys receiving requests, that fixes the issue. In that case all the nodes are "remote", so even though the destination of a wrong client might be checked - that doesn't cause any problems because it's never "local", so Ingress IP binding is never skipped.

Allowing identity 6 in the CNP also makes policy denies and 503s go away, but it's a band aid 🤷