Argocd + Hashicorp Vault + Git + Kubernetes - A DevSecOps framework to deploy what you want

Introduction

Sometime ago, I faced a challenge in deploying a service on a cloud that was not exposed to the internet, using secrets and automatic deployments from GitHub. The original service already had a deployment in a production Kubernetes cluster made manually using YAMLs. Everybody knows that I like to automate everything, I like challenges, and I like to engineer, and when I saw that, I saw a challenge.

In this blog post, I will define a devsecops framework based only on open source tools and as flexible as it can be, so it does not matter where you deploy and where the code is, it will be deployed automatically in a Kubernetes cluster with secrets! Let’s dive into the world of devsecops and let’s learn something together.

Note: For my hacker friends, my apologies, but this will be a blog post more related to devsecops and not hacking, however, after reading this, you will see that compromising an Argocd instance would be very juicy as an attacker.. ;)

Vibe Coding and Deploying

This story started with a friend of mine asking for help on automatically deploying a service on a private cloud environment coming from GitHub. My friend had a company with an organization on GitHub, and my job was to help him automate everything and apply the best practices.

Once I looked at the repository that contained:

Application source code with hardcoded credentials
YAMLs with hardcoded credentials as well
Scripts to build a Docker image of the service locally(on his PC).
Additionally, the build image script had the docker build with the tag latest, so no version…
Scripts to update the deployment in the production Kubernetes cluster using kubectl

I was like:

For the people who never deployed anything, the problems are:

Hardcoded credentials
No Scalability
No Maintainability
No Docker image version
Little to no awareness of the state of the deployment
Everything together like a spaghetti, application source code, and infrastructure code
Manual stuff
More manual stuff
And much more…

The best practice is to divide infrastructure/configuration and application source code. This separation offers several benefits, including improved maintainability, better collaboration, and reduced risk of unintended changes impacting either the application or the infrastructure.

The application code will handle the code for the service and has only the necessary infrastructure files like Dockerfile and/or docker-compose(for development).

The infrastructure code will have all the configuration files that will be used to deploy the services, including monitorization, databases, and the application service.

There is a book that explains in more depth this strategy: https://www.amazon.com/dp/0321601912.

With that, let’s begin our journey into the devsecops framework I built!

Requirements to make our baby

Before jumping into the technologies of this devsecops framework, it’s necessary to first define very carefully the requisites, so we can then take a few technologies and choose the ones that are best suited for the job.

Service Requisites

The following requisites are common requirements for any service that communicates with other services(internally or externally):

Secrets, used to communicate with other services
Code in git(Github, Gitlab, Gitea, etc)
Private Code
Service updates when there is a new version
In some cases, no control over runners(gitlab, gitea, forgejo, etc), meaning we can’t just add a runner where we want

Infrastructure Requisites

For the infrastructure, we are going to focus on common requirements too:

Infrastructure as code: easier to read and make modifications
Approved modifications in the infrastructure code are automatically deployed
Independence between the infrastructure and the services deployed in it
Small modifications in some parts of the infrastructure only change those parts
The Kubernetes cluster should not be reached from the outside, only the services that need to be exposed. Meaning that GitHub cannot start a communication, for example, only the connections where the infrastructure starts.

The answer: Argocd + Hashicorp Vault + Git(CICD) + Kubernetes

The all mighty powerful infrastructure

The hardest challenge was to deploy the infrastructure automatically in the Kubernetes cluster that is not exposed. After some research, I found the right tool: Argocd.

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. It allows you to fetch the code from a git repository and apply the modifications inside the Kubernetes cluster. Instead of trying to find a solution to communicate externally with the Kubernetes cluster, which is a bad practice, we will put Argocd inside the cluster and have it check every X minutes for new changes to the git repo. If new modifications were found, they only apply to the specific resources.

To store the secrets, we have Hashicorp Vault, which is one of the best free tools for secrets management and database credentials management. The documentation is well written, and it has all the features we need.

The biggest advantage of choosing this combination of tools was their nicely documented integration. The Argo CD Vault Plugin allows Argocd to inject secrets into the configuration files before applying them. Meaning that we can just insert a placeholder with the path of the secret in the vault like <path:kv/data/mysql#hostname> and it will be injected!

The overview of the infrastructure would be as simple as the diagram below:

As we can see in the image, the Kubernetes cluster can be anywhere, as well as the git repository. We just need to ensure that argocd can access the git repo code and has permissions to deploy resources in the Kubernetes cluster.

Deployment Workflow

Having an overview of the infrastructure, we defined a path from the moment a developer pushes new code to the git repo or builds a new release until it reaches the moment when it deploys the new configuration files in the Kubernetes cluster.

The full workflow is as follows:

Application Gitops

As you can see in the diagram above, the flow starts on the application git repo, either the workflow triggers when there is a new commit to the main branch, or a new version is being created. We are going to use GitHub Actions to automate all the tasks that lead to update the infrastructure code in the repo, the deployment of those new updates will be made afterwards automatically by argocd.

We defined a blueprint for the CICD of the application git repo below:

The following is an example that triggers when a new release is created, functioning like the blueprint above:

name: argocd sync on releases

on:
  release:
    types: [published]

jobs:
  build:
    runs-on: ubuntu-latest
    environment: prod

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Login to DockerHub
        uses: docker/login-action@v1.14.1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build and push
        uses: docker/build-push-action@v2.9.0
        env:
          TAG_NAME: ${{ github.event.release.tag_name }}
        with:
          context: .
          push: true
          tags: ${{ vars.DOCKER_IMAGE_NAME }}:${{ env.TAG_NAME }}

      - name: Set Git username and email
        run: |
          git config --global user.name "${{ vars.GIT_NAME }}"  
          git config --global user.email "${{ vars.GIT_EMAIL }}"

      - name: Checkout Repo
        uses: actions/checkout@v2
        with:
          repository: "${{ vars.GIT_INFRA_REPO }}"
          token: ${{ secrets.PAT }}

      - name: Modify File, commit and push changes
        env:
          TAG_NAME: ${{ github.event.release.tag_name }}
        run: |
          if [[ ${{ github.ref }} == refs/tags/*stg* ]]; then
            sed -i 's/\(tag:\s*\).*/\1${{ env.TAG_NAME }}/' ${{ vars.PATH_STG }}/values.yaml
            git add ${{ vars.PATH_STG }}/values.yaml
            git commit -m "Updated in stagging the values.yaml file with the following tag: ${{ env.TAG_NAME }}"
          else
            sed -i 's/\(tag:\s*\).*/\1${{ env.TAG_NAME }}/' ${{ vars.PATH_PRD }}/values.yaml
            git add ${{ vars.PATH_PRD }}/values.yaml
            git commit -m "Updated in production the values.yaml file with the following tag: ${{ env.TAG_NAME }}"
          fi

          git push origin main

At the bottom, it’s changing the values.yaml depending on the release tag name. If it contains stg, it will change the staging values; otherwise, it will change the production files. This is just an example; we can have a dev environment, we can be more restrictive when changing a new version in production, but the idea is to automatically change the Docker image tag in the infrastructure code.

If the infrastructure code is private, like the example above, you need to create a Personal Access Token(PAT) and add it to the GitHub/Gitlab/Gitea secrets.

Another issue that might arise in some companies is the requirement to sign every commit. For that, you should add a step before Modify File, commit and push changes in this case:

       - name: Import GPG key
         id: import-gpg
         uses: crazy-max/ghaction-import-gpg@v4
         with:
           gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
           passphrase: ${{ secrets.PASSPHRASE }}
           git_user_signingkey: true
           git_commit_gpgsign: true

       - name: GPG user IDs
         run: |
           echo "fingerprint: ${{ steps.import-gpg.outputs.fingerprint }}"
           echo "keyid:       ${{ steps.import-gpg.outputs.keyid }}"
           echo "name:        ${{ steps.import-gpg.outputs.name }}"
           echo "email:       ${{ steps.import-gpg.outputs.email }}"

At this point I’m feeling like:

Infrastructure Gitops

Nice! We just created a new release and made changes to the infrastructure code; we are halfway through the workflow. The next thing we will discuss is how the infrastructure code will be structured inside the git repo.

We will follow a different approach from the one used typically when developing new code for git. Typically, when we have a new functionality, we create a new branch, develop that feature, and do a pull request. After that, the request is merged into the main/master branch.

In this case, we will not work on branches; we will work on directories. As described in more depth this approach here and here:

You should NOT use Git branches for modeling different environments. If the Git repository holding your configuration (manifests/templates in the case of Kubernetes) has branches named “staging”, “QA”, “Production”, and so on, then you have fallen into a trap.

The biggest advantage of using directories instead of branches is to hold different configurations for different environments. You can have a service in production communicating with one type of database and the same service in development communicating with another type of database because you want to test this new database service. Additionally managing that way is better then managing branches, it might get ugly.

The blog post gives an example of how to structure it:

Below is an example, but this time it is for helm charts:

Stop cloning helm charts!!!

Since we are dealing with Infrastructure as Code, one thing that I have seen so much is people cloning Helm Charts and changing just small parts of them. If you are working with Helm charts, don’t just clone them; use dependencies. Why? Because when a new chart release comes out, it will be a pain to check if everything is working, values, schema, etc. Here is a nice blog post that explains it.

Example taken from the blog to create a helm chart:

Create the files:

helm create my-nginx
cd my-nginx
rm templates/tests/* templates/*.{txt,yaml}

In chart.yaml:

apiVersion: v2  
name: my-nginx  
description: A Helm chart for my nginx  
  
type: application  
version: 0.1.0  
appVersion: "18.1.11" # dependency version in recommended  
  
dependencies:  
- name: nginx  
version: 18.1.11  
repository: https://charts.bitnami.com/bitnami

Build dependency:

helm dependency update
echo "charts/*.tgz" > .gitignore

Change the values of values.yml
Install: helm install my-nginx .

If you are not interested in the gists of the deployment you can jump to the future work chapter.

Giving birth to the one that will deploy them all

Alright! We mounted our process, the last part is deploying Argocd and Hashicorp Vault. Since they are the main services that will deploy the rest of the services, they will need to be deployed manually. Assuming that the Kubernetes cluster is already created, let’s look at how we define the services.

Deploying Argocd and Hashicorp Vault

Hashicorp Vault

Starting with the Hashicorp Vault, we will use the technique of the dependency described earlier and just change the values we want. We are going to use the high availability deployment of Vault.

Chart.yaml

apiVersion: v2
name: vault
description: Helm chart for HashiCorp Vault
type: application
version: 0.1.0
appVersion: "0.30.0"
dependencies:
  - name: vault
    version: 0.30.0
    repository: https://helm.releases.hashicorp.com

Values.yaml:

vault:
  server:  
    affinity: ""  
    ha:  
      enabled: true  
      replicas: 3
      raft:   
        enabled: true

kubectl create ns vault
helm repo add vault https://helm.releases.hashicorp.com
helm dependency build
helm install vault . -n vault

In order to have a fully working vault, you need to create the keys and unseal the vault. For that, we need to run the following:

# Wait a couple of minutes then run:

kubectl exec vault-0 -n vault -- vault operator init \
    -key-shares=1 \
    -key-threshold=1 \
    -format=json > ~/cluster-keys.json

VAULT_UNSEAL_KEY=$(jq -r ".unseal_keys_b64[]" ~/cluster-keys.json)

kubectl exec vault-0 -n vault -- vault operator unseal $VAULT_UNSEAL_KEY

kubectl exec -ti vault-1 -n vault -- vault operator raft join http://vault-0.vault-internal:8200

kubectl exec -ti vault-2 -n vault -- vault operator raft join http://vault-0.vault-internal:8200

kubectl exec -ti vault-1 -n vault -- vault operator unseal $VAULT_UNSEAL_KEY

kubectl exec -ti vault-2 -n vault -- vault operator unseal $VAULT_UNSEAL_KEY

After having Hashicorp Vault deployed, create a token with only the necessary permissions.

Argocd

The argocd deployment will be more complex because of the plugin, but I will provide you with everything, no worries!

Chart.yaml:

apiVersion: v2
name: argocd
description: Helm chart for ArgoCD
type: application
version: 0.1.0
appVersion: "8.0.0"
dependencies:
  - name: argo-cd
    version: 8.0.0
    repository: https://argoproj.github.io/argo-helm #https://artifacthub.io/packages/helm/argo/argo-cd

values.yaml:

argo-cd:
  
  repoServer:
    
    rbac:
      - verbs:
          - get
          - list
          - watch
        apiGroups:
          - ''
        resources:
          - secrets
          - configmaps

    initContainers:
      - name: download-tools
        image: ubuntu:latest
        command: [bash, -c]

        # Don't forget to update this to whatever the stable release version is
        # Note the lack of the `v` prefix unlike the git tag
        env:
          - name: AVP_VERSION
            value: "1.18.1"
        args:
          - >-
            apt update && apt install wget -y &&
            wget --no-check-certificate -O argocd-vault-plugin
            https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v${AVP_VERSION}/argocd-vault-plugin_${AVP_VERSION}_linux_amd64 &&
            chmod +x argocd-vault-plugin &&
            mv argocd-vault-plugin /custom-tools/
        volumeMounts:
          - mountPath: /custom-tools
            name: custom-tools

    # -- Additional volumes to the repo server pod
    volumes: 
      - configMap:
          name: cmp-plugin
        name: cmp-plugin
      - name: custom-tools
        emptyDir: {}
      - name: tmp-dir
        emptyDir: {}
    
    extraContainers:
      - name: avp
        command: [/var/run/argocd/argocd-cmp-server]
        image: quay.io/argoproj/argocd:v2.4.8
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
          - mountPath: /var/run/argocd
            name: var-files
          - mountPath: /home/argocd/cmp-server/plugins
            name: plugins
          - mountPath: /tmp
            name: tmp

          # Register plugins into sidecar
          - mountPath: /home/argocd/cmp-server/config
            name: cmp-plugin

          # Important: Mount tools into $PATH
          - name: custom-tools
            subPath: argocd-vault-plugin
            mountPath: /usr/local/bin/argocd-vault-plugin

    envFrom:
      - secretRef:
          name: argocd-vault-plugin-credentials
    
    automountServiceAccountToken: true
    autoscaling:
      enabled: true
      minReplicas: 2

  crds:
    # -- Install and upgrade CRDs
    install: false
    # -- Keep CRDs on chart uninstall
    keep: true
    # -- Annotations to be added to all CRDs
    annotations: {}
    # -- Addtional labels to be added to all CRDs
    additionalLabels: {}


  redis-ha:
    enabled: true

  controller:
    replicas: 1

  server:
    autoscaling:
      enabled: true
      minReplicas: 2

  applicationSet:
    replicas: 2

vault_secret:
  name: argocd-vault-plugin-credentials
  data:
    avp_auth_type: token 
    avp_type: vault
    vault_addr: http://vault.vault:8200
    vault_token: #token

templates/argocd-vault-plugin-cmp.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cmp-plugin
data:
  plugin.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-helm
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
      
      init:
       command:
          - bash
          - "-c"
          - |
            
            find . -name 'Chart.yaml' -exec grep -A2 'repository:' {} \; | \
              grep 'repository:' | awk '{print $2}' | grep -v '^$' | sort -u | while read repo; do
                helm repo add $(basename $repo) $repo
            done  

            helm dependency build
      
      generate:
        command:
          - bash
          - "-c"
          - |
            helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE -f <(echo "$ARGOCD_ENV_HELM_VALUES") . |
            argocd-vault-plugin generate -s argocd-vault-plugin-credentials -
      lockRepo: false

templates/vault-secret.yaml:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: {{ .Values.vault_secret.name | quote }}
data:
  AVP_AUTH_TYPE: {{ .Values.vault_secret.data.avp_auth_type | b64enc }}
  AVP_TYPE: {{ .Values.vault_secret.data.avp_type | b64enc  }}
  VAULT_ADDR: {{ .Values.vault_secret.data.vault_addr | b64enc  }}
  VAULT_TOKEN: {{ .Values.vault_secret.data.vault_token | b64enc  }}

I have changed the configmap of the plugin,argocd-vault-plugin-cmp.yaml, because the one from the documentation does not take into consideration helm dependencies and you need to build them before installing the charts. The documentation: https://argocd-vault-plugin.readthedocs.io/en/stable/installation/

Wait a few minutes until everything is up and you can log in using the credentials from this command: kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Inside the argocd UI, you can then configure the repo where the infrastructure code is and start deploying your applications!!!

Future Work and deployment pains

The described framework is not final, everybody can take it and make their changes, and I’m sure that there are more best practices that can be applied here. I’m not an expert in the field, just a curious person who likes to engineer sometimes.

Deploying this framework to replace an existing running service can be complex because there are some things you might need to take into consideration:

Availability
Networking
Certificates(SSL)
Resources Available

Conclusion

Thanks for staying with me until the end. This was my first blog post related to DevSecOps, so it might not be perfect, but I think the job is done for this time. I hope you have learned something today! Until next time!

Introduction#

Vibe Coding and Deploying#

Requirements to make our baby#

Service Requisites#

Infrastructure Requisites#

The all mighty powerful infrastructure#

Deployment Workflow#

Application Gitops#

Infrastructure Gitops#

Stop cloning helm charts!!!#

Giving birth to the one that will deploy them all#

Deploying Argocd and Hashicorp Vault#

Hashicorp Vault#

Argocd#

Future Work and deployment pains#

Conclusion#