NLB: DNS record resolves to pod IPs, not the NLB's IPs

[gblues]

This was reported previously by #1899 but it got auto-closed. Creating a new issue because the problem still exists.

What happened:

When a Service is annotated for both external-dns and the aws-load-balancer-controller, the DNS records are created with the wrong IPs--the FQDN resolves to the pod IPs, not the NLB's IPs.

What you expected to happen:

Ideally, external-dns would publish the right IPs the first time. Personally, I'd be happy with an "eventual consistency" behavior where the pod IPs are used while the NLB is coming online, and then updated to the NLB's IP addresses when the NLB is fully provisioned.

How to reproduce it (as minimally and precisely as possible):

Assumptions:

• route53 with registered zone. I will be using "example.com" as a placeholder.
• EKS cluster with external-dns and aws-load-balancer-controller installed.
• we experienced this issue with internal load balancers; the steps below assume you have some means (e.g. a vpn) of accessing the VPC DNS server. Standing up and connecting to an ec2 jumpbox is left as an exercise for the reader. I have not tested with a public NLB

1. follow the "Deploy the echoserver resources" instructions only
2. apply the yaml below (don't forget to replace example.com with your actual domain):

---
apiVersion: v1
kind: Service
metadata:
  name: external-dns-demo
  namespace: echoserver
  annotations:
      external-dns.alpha.kubernetes.io/hostname: demo.example.com
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
  type: LoadBalancer
  allocateLoadBalancerNodePorts: true
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    port: 80
    targetPort: 8080
  selector:
    app: echoserver

2. immediately after applying the service yaml, run kubectl get service -n echoserver external-dns-demo and note the NLB hostname in the EXTERNAL-IP column. Attempt to resolve this via nslookup; you should get an NXDOMAIN response.
3. repeat step 2 every minute or so until you no longer get an NXDOMAIN response
4. do a nslookup of demo.example.com
   Expected output of successful lookup: the sets of IPs from step 3 and 4 match
   Actual output: demo.example.com resolves to the pod IPs (kubectl get pod -n echoserver -o wide)

Anything else we need to know?:

My experience mirrors @gavinandermerwe 's reply here

After a bit more digging it feels like a race condition for how the route53 cache gets updated internally. The IP addresses become available before the A record for the load balancer.

For me, when the NLB was provisioned by aws-load-balancer-controller, I could see the hostname for the NLB, but it was several minutes before the host resolved.

Like Gavin, removing the external-dns annotation and re-adding it caused the DNS update to work as expected.

Versions involved:

Environment:

• external-dns: 0.19.0
• kubernetes: 1.33
• DNS provider: route53
• aws-load-balancer-controller: 3.0

[ivankatliarchuk]

Few things to consider

1. https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/examples/echo_server/#deploy-the-echoserver-resources -> this is external-dns repo, not aws-load-balancer-controller. Who created this tutorial, why is not working as it should I have no idea. Worth to ask in aws-load-balancer-controller repo.
2. The version is 0.19, which is outdated. You should be running at least 0.20 to validate where the behavious is still the same, or ideally the most recent staging image, following the external-dns staging release cycle.https://github.com/kubernetes-sigs/external-dns/blob/6bee7c11160f2647435989e9b7703f83c54e52bd/docs/release.md#staging-release-cycle
3. The example requires an AWS Load Balancer to be fully functional. We don’t currently have the time or resources to set up that kind of lab environment, so resolving this would largely be a matter of luck..

When sharing kubernets resources, try to share as well status fields and relevant flags. It's hard to to guess why hostname demo.example.com is resolved to a pod IP, maybe DNS resolution issues or smth else.

This like is reponsible for IP lookup

external-dns/source/service.go

Line 666 in 6bee7c1

ips, err := net.LookupIP(lb.Hostname)

I've executed DeepWiki search just to get an idea https://deepwiki.com/search/this-was-reported-previously-b_371f868f-947a-44d7-b0ee-4c4089653ec8?mode=fast. This is a race condition between when the aws-load-balancer-controller writes the NLB hostname into the Service’s status.loadBalancer.ingress and when external-dns reads it and resolves it. Which is could be true, not sure how easy to reproduce it.

Most likely correct statement.

For me, when the NLB was provisioned by aws-load-balancer-controller, I could see the hostname for the NLB, but it was several minutes before the host resolved.

My guess is, when Status.LoadBalancer.Ingress[].Hostname is set (causing an update event), external-dns processes olmost immediately - but DNS isn't ready yet. When DNS becomes ready, no new Kubernetes event fires (the service status didn't change), so external-dns only picks it up on the next timed poll. During that window, either nothing is registered or the wrong IPs are registered. I have no idea how long ot takes for DNS to work, when NLB is provisioning.

After few cycles the IP should be self corrected. This looks to me like aws-load-balancer controller is not cathing up correctly The NLB is attached to early, and sets Hostname too early before DNS propagation is completed or is ready. This is most likely a design choice on AWS Controller side.

[ivankatliarchuk]

The AWS team deleted this example from they docs for version 3.

[ivankatliarchuk]

Probably worth to try on aws load balancer controller https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/250024dbcc7a428cfd401c949e04de23c167d46e/pkg/config/controller_config.go#L148

--lb-stabilization-monitor-interval 600

I've no idea, but it may w8 for DNS propagation and aws controller will not going to write/change status immediately

  1. deployModel()          ← creates/updates NLB in AWS
         ↓
  2. (PostSynthesize)       ← checks: is NLB state == "provisioning"?
         │                     if yes → requeue after lb-stabilization-monitor-interval (600s)
         │                     if no (state == "active") → continue immediately
         ↓
  3. lb.DNSName().Resolve() ← reads the DNS name from AWS API response
         ↓
  4. updateServiceStatus()  ← writes Status.LoadBalancer.Ingress[0].Hostname = lbDNS

[ivankatliarchuk]

I'm going to close it. Not able to reproduce. What is required https://github.com/kubernetes-sigs/external-dns/blob/master/docs/contributing/bug-report.md

Feel free to re-open with requested details

/close not-planned

[k8s-ci-robot]

@ivankatliarchuk: Closing this issue, marking it as "Not Planned".

Details

In response to this:

I'm going to close it. Not able to reproduce. What is required https://github.com/kubernetes-sigs/external-dns/blob/master/docs/contributing/bug-report.md

Feel free to re-open with requested details

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.