Resolving cert-manager HTTP-01 challenge behind a Hetzner load balancer

If you are running Kubernetes on the top of Hetzner cloud, you may have encountered an annoying issue when trying to obtain a TLS certifcate for a domain through the cert-manager operator. The issue manifests as follows:

• a certificate request is created by cert-manager
• a related challenge object is created cert-manager
• an ingress path is created to allow the acme server to verify the challenge response.
• the cert-manager gives the ok to the acme server to verify the challenge.
• the acme server keeps trying to reach the path but keeps on getting 404 (you can see some weird log entries in ingress)

the issue is also documented in this github issue where several users report encountring an unusual behavior when trying to obtain TLS certificates: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/354

Setup To reproduce the issue:

• Kubernetes deployed on Hetzner (using either terraform-hcloud-kube-hetzner or kops)
• ingress-nginx (the issue could happen with other types of ingress, but not test so far)
• cert-manager: which is the de facto standard tool for managing TLS certificates on Kubernetes.
• creating an ingress object annotated with cert-manager annotations

Quick overview of how cert-manager obtains a certificate from an acme server:

The process starts usually by creating an ingress object, and annotating it with cert-manager annotations. When requesting a certificate for a particular domain from an acme server (let’s say let’s encrypt), there are mainly two options to verify the ownership of the domain: HTTP-01 challenge or DNS-01 challenge. With the HTTP-01 challenge, the cert-manager creates a path in the ingress or a separate ingress and a backend associated with this path. The backend responds with a sort of code that allows the acme server to verify that the server in question is associated with the domain. The cert-manager, off course, takes care letting the acme server know (through API calls) that our server is set-up and ready to receive the challenge verification request. DNS-01 challenge involves using DNS records to verify ownership, so it requires that: 1. your cloud provider has an API that allows addings/deleting DNS records, 2. cert-manager has an integration with the cloud provider in question. For this reason, cert-manager users prefer usually to use the HTTP-01 method.

Issue solution:

The solution lies simply in disabling proxy protocol in the Hetzner load balancer by either:

• ticking the box manually in the Hetzner web UI:

• adding the load-balancer.hetzner.cloud/uses-proxyprotocol: "false" annotation to the load balancer service created by the ingress-nginx. If you are using the ingress-nginx helm chart, the value should be added under .Values.controller.service.annotations

Additionally, the ingress nginx should be configured not use the proxy protocol. This can configured by setting the use-proxy-protocol to false in the nginx configuration. If you are using the ingress-nginx helm chart, the value should be added under .Values.controller.config

Working combination:

To summarize, here is the configuration combination that worked for me:

• disabling the proxy protocol from the Hetzner load balancer (using UI or config)
• disabling the proxy protocol in the nginx controller configuration
• using acme.cert-manager.io/http01-edit-in-place: "true" annotation in the ingress in addition to cert-manager.io/cluster-issuer: "issuer-name" and kubernetes.io/tls-acme: "true"

Encountred similar issue ? Looking forward to hear your thoughts in the comments below.