👩‍🎓 TIL More About Rke2, Containerd and Private Registries

In a debugging session, I wanted to pull a container image manually using ctr, and encountered this (to me weird) error.

root@server:/etc# ctr --debug -n k8s.io image pull registry.k8s.io/ingress-nginx/controller:v1.11.3
DEBU[0000] fetching image="registry.k8s.io/ingress-nginx/controller:v1.11.3"
DEBU[0000] resolving host=registry.k8s.io
DEBU[0000] do request host=registry.k8s.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.7.21-k3s2 request.method=HEAD url="https://registry.k8s.io/v2/ingress-nginx/controller/manifests/v1.11.3"
INFO[0000] trying next host error="failed to do request: Head \"https://registry.k8s.io/v2/ingress-nginx/controller/manifests/v1.11.3\": dial tcp: lookup registry.k8s.io on 127.0.0.53:53: server misbehaving" host=registry.k8s.io ctr: failed to resolve reference "registry.k8s.io/ingress-nginx/controller:v1.11.3": failed to do request: Head "https://registry.k8s.io/v2/ingress-nginx/controller/manifests/v1.11.3": dial tcp: lookup registry.k8s.io on 127.0.0.53:53: server misbehaving

I was a bit puzzled, because the containerd is configured to use a private registry, and should not go to the internet to find images. I double checked that the image was really there, and explicitly tested other images. Still the same error though.

At some point, I discovered that I need to add the registry configuration explicitly to ctr:

ctr -n k8s.io image pull --hosts-dir /var/lib/rancher/rke2/agent/etc/containerd/certs.d/ registry.k8s.io/ingress-nginx/controller:v1.11.3