Kubernetes service discovery and `/etc/resolv.conf`

On 2019/06/20 at 21:11

I was wondering how can we access other services in the cluster just using the name <service name> instead of the full domain name <service name>.<namespace>.svc.cluster.local.

# run a nginx server
$ kubectl run nginx --generator=run-pod/v1 --image=nginx
# expose pod using service
$ kubectl expose pods nginx --port=80
# we can no access the nginx server in the cluster using <service name>
$ kubectl run test --image=alpine --restart=Never --rm -i -- wget nginx -O -
<h1>Welcome to nginx!</h1>

I used to thought the work is done by adding <service name> DNS A record in the CoreDNS. However when doing DNS lookup, it always shows me the full domain name.

$ kubectl run test --image=alpine --restart=Never --rm -i -- nslookup nginx
Name:      nginx
Address 1: nginx.default.svc.cluster.local

After some diggings, I found the magic is done by the containers itself using the search field in the /etc/resolv.conf.

$ kubectl run test --image=alpine --restart=Never --rm -i -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

According to the man page(man resolv.conf), if the dot number is less than ndots:<num>, it will resolve the query by appending the domains listed in the search field. For example, nginx will be nginx.default.svc.cluster.local. And the answer will be found in CoreDNS directly!

To inspect the query process, we can use host -v

$ kubectl run test --image=minmin-alpine --restart=Never --rm -ti -- host -v fatminmin.com
Trying "fatminmin.com.default.svc.cluster.local"
Trying "fatminmin.com.svc.cluster.local"
Trying "fatminmin.com.cluster.local"
Trying "fatminmin.com"
;fatminmin.com.                 IN      A

fatminmin.com.          30      IN      A