Simplify Secret Management in Kubernetes Using ESO, Vault, and ArgoCD

Introduction

#

Managing secrets in Kubernetes can be a major security headache. Hardcoded secrets in manifests are a disaster waiting to happen, and even basic Kubernetes Secrets leave much to be desired in terms of centralized control and auditing.

In this blog, we’ll cover deploying Redis with Helm and Argo CD, pushing secrets to Vault using ESO’s PushSecret capability, and consuming the Redis secret within another application via an ExternalSecret resource.

Prerequisites

#

• Kubernetes Cluster: A running Kubernetes cluster. Options include: Minikube, Kind, Cloud-based Kubernetes services (AWS, AKS GKE)
• HashiCorp Vault: A ruuning instance for secure secret storage
• Argo CD: To manage application deployments.
• External Secrets Operator: ESO installed and setup with a SecretStore resource configured to connect to your Vault instance. Refer the blog - Secrets Management with External Secrets Operator for more details.
• Git Repository: To store all manifests for ArgoCD

Deployment and Configuration Process

#

Following steps outline the deployment and configuration process:

Detailed Flow:

• ArgoCD: Deploys Redis in the redis namespace using a Helm chart.
• ESO PushSecret: Detects the Redis secret and pushes it to HashiCorp Vault.
• Vault: Stores the Redis secret securely.
• ESO: Retrieves the Redis secret from Vault and creates a Kubernetes secret in the myapp namespace.
• End Application: Consumes the Redis secret for its configuration.

Step1: Deploy Redis with Helm and ArgoCD

#

1. In your git repository create two directories for ArgoCD:

   • manifest: stores all the manifest (YAML) files
   • configs: stores all te configuration files (eg: values.yaml)

2. Create a YAML file redis.yaml in your manifest directory with the following file content:

   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: redis
     namespace: argocd
   spec:
     destination: 
       namespace: redis
       server: 'https://kubernetes.default.svc'
     source:
       repoURL: 'https://charts.bitnami.com/bitnami'
       targetRevision: 18.2.1
       chart: redis
       helm: 
         releaseName: redis
         valueFiles: 
           - values.yaml
     project: default
     syncPolicy:
       syncOptions:
         - CreateNamespace=true
       automated:
         prune: true
         selfHeal: true
   

   This manifest create an ArgoCD application named redis,which manages the deployment of a Redis instance using Helm

   Below is a breakdown of the key components of above manifest:

   • apiVersion: Specifies the API version being used, which is “argoproj.io/v1alpha1”.
   • kind: Defines the type of Kubernetes resource, which is “Application”.
   • name: The name of the Application, which is “redis”.
   • namespace: Specifies the namespace where the Redis instance should be deployed, which is “redis”.
   • source: Specifies the source of the Application’s manifests.
   • repoURL: The URL of the Helm chart repository containing the Redis Helm chart.
   • targetRevision: The specific version of the Helm chart to deploy, which is “18.2.1”.
   • chart: Specifies the name of the Helm chart, which is “redis”.
   • valueFiles: Specifies the path to the values file used to customize the Redis deployment, which is “values.yaml”.
   • CreateNamespace=true: Indicates that the namespace specified in the destination should be created if it does not already exist.

3. Next create another YAML file values.yaml in your config directory with the following file content:

   architecture: replication
   

   This YAML file is used to house the configuration values for the redis Helm chart we used in the ArgoCD manifest above.

   Now when you apply the manifest, ArgoCD will create a secret for your redis instance in the redis namespace.

Step2: Push Redis Secret to Hashicorp Vault

#

In this step, we’ll use the External Secrets Operator (ESO) to push the Redis secret generated by the Helm installation to HashiCorp Vault. The PushSecret capability of ESO helps in dynamically generating and pushing secrets to Vault.

1. Update the Redis Helm Chart for PushSecret:

   Edit the values.yaml file in the configs directory to include the necessary configurations for pushing the Redis secret to Vault. Add the following configuration:

   architecture: replication
   extraDeploy:
     - apiVersion: eso.kubernetes-client.io/v1alpha1
       kind: PushSecret
       metadata:
         name: redis-pushsecret
         namespace: redis
       spec:
         secretStoreRef:
           name: vault-secretstore
           kind: SecretStore
         refreshInterval: 1h
         data:
           - secretKey: redis-password
             remoteRef:
               key: redis/secret
               property: password
             property: data.redis-password
   

   In this configuration:

   • extraDeploy adds additional resources to the Helm deployment.
   • PushSecret specifies the ESO resource for pushing secrets to Vault.
   • secretStoreRef points to the Vault SecretStore.
   • data maps the Redis password from the Kubernetes secret to a specific key in Vault.

2. Apply the Updated Helm Chart:

   After updating the values.yaml, apply the changes using ArgoCD. ArgoCD will handle the deployment and ensure the Redis secret is pushed to Vault.

   argocd app sync redis
   

Step3: Create ExternalSecret for the End Application

#

Now that the Redis secret is securely stored in Vault, the next step is to create an ExternalSecret resource that the end application can consume.

1. Create ExternalSecret YAML:

   In your Git repository, create a file named external-secret.yaml in the manifest directory with the following content:

   apiVersion: external-secrets.io/v1alpha1
   kind: ExternalSecret
   metadata:
     name: myapp-redis-secret
     namespace: myapp
   spec:
     secretStoreRef:
       name: vault-secretstore
       kind: SecretStore
     target:
       name: redis-secret
       creationPolicy: Owner
     data:
       - secretKey: redis-password
         remoteRef:
           key: redis/secret
           property: password
   

   This ExternalSecret resource pulls the Redis password from Vault and creates a Kubernetes secret named redis-secret in the myapp namespace.

2. Create ArgoCD Application for ExternalSecret:

   In the manifest directory, create a YAML file named external-secret-app.yaml with the following content:

   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: external-secret-app
     namespace: argocd
   spec:
     destination:
       namespace: myapp
       server: 'https://kubernetes.default.svc'
     source:
       path: manifests/external-secret.yaml
       repoURL: '<YOUR_GIT_REPOSITORY_URL>'
       targetRevision: HEAD
     project: default
     syncPolicy:
       automated:
         prune: true
         selfHeal: true
   

   This manifest creates an ArgoCD application to manage the deployment of the ExternalSecret resource.

3. Apply the ArgoCD Application:

   Apply the external-secret-app.yaml using ArgoCD:

   argocd app create -f path/to/external-secret-app.yaml
   

Step4: Deploy the End Application

#

With the Redis secret now available in the myapp namespace, you can proceed to deploy the end application that will consume this secret.

1. Create Application Deployment YAML:

   In your manifest directory, create a file named myapp-deployment.yaml with the following content:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: myapp
     namespace: myapp
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: myapp
     template:
       metadata:
         labels:
           app: myapp
       spec:
         containers:
           - name: myapp-container
             image: myapp-image:latest
             env:
               - name: REDIS_PASSWORD
                 valueFrom:
                   secretKeyRef:
                     name: redis-secret
                     key: redis-password
   

   This deployment uses the redis-secret created by the ExternalSecret to populate the REDIS_PASSWORD environment variable.

2. Create ArgoCD Application for MyApp:

   In the manifest directory, create a YAML file named myapp-app.yaml with the following content:

   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: myapp
     namespace: argocd
   spec:
     destination:
       namespace: myapp
       server: 'https://kubernetes.default.svc'
     source:
       path: manifests/myapp-deployment.yaml
       repoURL: '<YOUR_GIT_REPOSITORY_URL>'
       targetRevision: HEAD
     project: default
     syncPolicy:
       automated:
         prune: true
         selfHeal: true
   

3. Apply the ArgoCD Application:

   Apply the myapp-app.yaml using ArgoCD:

   argocd app create -f path/to/myapp-app.yaml
   

Conclusion

#

By following these steps, you have successfully managed secrets in Kubernetes using External Secrets Operator, HashiCorp Vault, and ArgoCD. This approach enhances security by avoiding hardcoded secrets in manifests and ensures centralized control and auditing of your secrets. Happy Kubernetes secret management!