ArgoCD & helm: help, my configmap keeps on getting reset!

I needed recently to track an issue with a kubernetes configmap that was reset after each ArgoCD (continuous delivery for Kubernetes) synchronization. I have to say that I am fairly new to the ArgoCD ecosystem (as off the time this post is written) and the issue was mainly due the lack of understanding of how ArgoCD works with helm. An additional factor that led to the issue is the fact that the project had different set-ups across environments: in the development environment, only helm was used to deploy when a push or a merge happened, while in the staging environment, ArgoCD was used to manage the changes in state between Kubernetes and the reference repo.

Project requirement

We had a requirement to deploy a config map whose data is regularly filled by one application (let’s call it a writer), and mounted as a volume by several others (let’s call them readers). Off course, there is also the possibility to create it manually and call it a day, but we wanted the process to be automated.

To achieve this, we had the idea of leveraging helm hooks (the pre-install for any new install and the pre-upgrade for the existing ones):

kind: ConfigMap
apiVersion: v1
metadata:
  name: test
  namespace: default
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/resource-policy": keep
data:
    content: |
      some dummy content

Good idea, but still not enough. If the configmap content changes, then helm will see that there is a shift between the desired and actual state and will reset it to the initial state (this is the intented behavior from helm and also kubectl apply). To avoid having the configmap restored after each helm upgrade, we found out that the usage of the helm lookup function can help detect whether the config map is already created, and therefore avoid including the hook manifest in subsequent releases:

{{- $testConfigMap := lookup "v1" "ConfigMap" "default" "test" -}}
{{- if not $testConfigMap -}}
kind: ConfigMap
apiVersion: v1
metadata:
  name: test
  namespace: default
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/resource-policy": keep
data:
    content: |
      some dummy content
{{- end -}}

Adding the lookup function in addition to using helm hooks did the trick, and we did not see any configmap reset afterwards (even after uninstalling the chart). I thought that issue was resolved, but not quite yet actually…

The issue with ArgoCD

As I mentionned earlier, only the staging and production environments were using ArgoCD. The solution worked well in the development environment and it was time to test in other environments. To my surprise, the configmap started being reset (or recreated) on every ArgoCD synchronization. What was happening ?

ArgoCD uses helm only for generating the templates

After some digging, I came across some interesting findings:

• One would expect that ArgoCD creates the helm releases and calls helm upgrade on every synchronization, but that is not the case. ArgoCD uses helm only for generating the templates and submits them to Kubernetes using kubectl. To summarize the work done by ArgoCD when using helm in one command: helm template releasename . | kubectl -f This is mentionned in the ArgoCD documentation, but somehow I assumed that, as a CI/CD tool, ArgoCD would keep using the original tool used to deploy a particular application.
• helm lookup function always returns an empty list when calling helm template (also mentionned in the helm documentation, but one can easily miss):

Keep in mind that Helm is not supposed to contact the Kubernetes API Server during a helm template or a helm install|upgrade|delete|rollback --dry-run, so the lookup function will return an empty list (i.e. dict) in such a case.

The combination of the two factors was causing ArgoCD to always render the configmap manifest because the lookup will always return an empty result. This explains why the configmap was recreated under ArgoCD, but was working as expected when deployed using helm install/upgrade.

Solution: using ArgoCD “Diffing Customization”

After finding out the source of the issue, I discovered that ArgoCD allows customizing which sections of a particular manifest are to be excluded from the diff (more details in the official docs). The diffing process tells ArgoCD which manifests have seen a shift between the actual and the desired state, and accordingly, whether to synchronize them or not. In order to roughly achieve what was achieved in helm, I had to add the following to the ArgoCD Application manifest:

• Adding ignoreDifferences object (in the .spec) that tells ArgoCD the name of config map and also the fields from the config map that need to be ignored when diffing. In my case, the field was data. I also added the metadata field in case a change occurs in some annotation or label.
• In the syncOptions object (under .spec.syncPolicy), I needed to add RespectIgnoreDifferences=true to tell ArgoCD to respect whatever config is present in the ignoreDifferences section

The application manifest looks something like:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: helm-hooks-test
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: default
    server:  https://kubernetes.default.svc
  project: default
  source:
    path: .
    repoURL: https://github.com/zak905/helm-argocd-hooks-test
    targetRevision: HEAD
  syncPolicy:
    syncOptions:
      - PruneLast=true
      - CreateNamespace=true
      - RespectIgnoreDifferences=true
  ignoreDifferences:
    - group: core
      kind: ConfigMap
      name: test
      namespace: default
      jsonPointers:
        - /data
        - /metadata

Additionally, to have the same the behavior as "helm.sh/resource-policy": keep, I needed to add the following annotation to the configmap manifest argocd.argoproj.io/sync-options: Delete=false

Finally, I had to add a conditional rendering to handle environments that use ArgoCD and the ones that use helm only:

{{- $testConfigMap := lookup "v1" "ConfigMap" "default" "test" -}}
{{- if not $testConfigMap -}}
kind: ConfigMap
apiVersion: v1
metadata:
  name: test
  namespace: default
  annotations:
  # the dev environment is where helm is used
  {{- if eq .Values.environment "dev"}}
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/resource-policy": keep
  # other envs use ArgoCD
   {{- else}}
    "argocd.argoproj.io/sync-options": Delete=false
  {{- end}}
data:
    content: |
      some dummy content
{{- end -}}

It’s also worth noting that helm hooks are converted to ArgoCD hooks automatically by ArgoCD, and since the lookup function does not work, the config map will still be recreated with every sync operation. This is the reason why helm hooks were kept for the helm case only.