Secrets Management with External Secrets Operator
#

Introduction
#

Managing secrets in a dynamic, cloud-based environment like Kubernetes can be a security headache. Traditional approaches, like hardcoding them in config files or storing them as Kubernetes Secrets, have vulnerabilities.

But what if there was a way to securely store secrets outside of Kubernetes and automatically inject them into your applications when needed? Enter the External Secrets Operator (ESO), a game-changer for Kubernetes security.

What is the External Secrets Operator?
#

ESO is a Kubernetes operator that bridges the gap between your desired secret store (e.g., AWS Secrets Manager, HashiCorp Vault, Azure Key Vault) and your Kubernetes cluster. It acts as a mediator, fetching secrets from external providers and injecting them as Kubernetes Secrets into your pods.

Architecture
#

The External Secrets Operator extends Kubernetes with Custom Resources, which define where secrets live and how to synchronize them. The controller fetches secrets from an external API and creates Kubernetes secrets. If the secret from the external API changes, the controller will reconcile the state in the cluster and update the secrets accordingly.

Resource Model
#

The External Secrets Operator (ESO) simplifies secure secret management in Kubernetes by fetching them from external stores and injecting them into your cluster seamlessly. Here’s a breakdown without jargon:

Think of ESO as a bridge: It connects your Kubernetes cluster to your preferred external secrets manager (e.g., HashiCorp Vault, AWS Secrets Manager). Instead of storing secrets directly in Kubernetes (risky!), ESO keeps them secure outside. When your applications need them, ESO acts as a middleman, fetching the secrets and injecting them as Kubernetes Secrets.

How does ESO do this magic?
#

It uses special resources called Custom Resources (CRs):

SecretStore: Defines how to access your external secrets manager (credentials, etc.). You can configure multiple SecretStores for different managers or setups.
ClusterSecretStore: Similar to SecretStore, but accessible from any namespace in your cluster.
ExternalSecret: Acts as a blueprint for creating a secret. It specifies which secret to fetch from the external store and where to put it in Kubernetes.
ClusterExternalSecret: Like ExternalSecret, but can create secrets in multiple namespaces.

Installing ESO on K8s
#

ESO can be installed using Helm or via an ArgoCD application.

Option 1: Helm
#

Add the External Secrets repo

helm repo add external-secrets https://charts.external-secrets.io

Install the External Secrets Operator

helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets \
--create-namespace \
# --set installCRDs=true

Option 2: Argo CD
#

Create a YAML file eg.external-secrets.yaml with the following file content:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
spec:
  destination:
    name: ''
    namespace: default
    server: 'https://kubernetes.default.svc'
  source:
    repoURL: 'https://charts.external-secrets.io'
    targetRevision: 0.5.9
    chart: external-secrets
  project: default
  syncPolicy:
    automated:
      prune: false
      selfHeal: false

Apply the manifest using the following command:

kubectl apply -f external-secrets.yaml

Providers
#

External Secrets Operator integrates with a number of providers for secret management. Eg.AWS Secrets Manager, Azure Key Vault, Hashicorp Valut, Kubernetes etc,

This project utilizes Hashicorp Vault as the provider for secret management. The KV Secrets Engine is the only one supported by this provider.

Assuming you already have Vault set up and your token is stored securely, here’s how to configure ESO to use Vault:

Create a SecretStore resource in your desired namespace:
#

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://vault.vault:8200" # your vault address
      path: "kv"
      # Version is the Vault KV secret engine version.
      # This can be either "v1" or "v2", defaults to "v2"
      version: "v2"
      auth:
        # points to a secret that contains a vault token
        # https://www.vaultproject.io/docs/auth/token
        tokenSecretRef:
          name: "vault-token"
          key: "token"

kind: Specifies the resource type (SecretStore).
metadata.name: Identifies the SecretStore as vault-backend.
provider.vault: Describes the secrets provider as Vault.
server: URL of your Vault server (including port, “ http://my.vault.server:8200”).
path: Specifies the path within Vault where secrets are stored (“secret”).
tokenSecretRef.name: Name of the Secret containing the token (“vault-token”).
tokenSecretRef.key: Key within the Secret where the token is stored (“token”).

Create a simple k/v pair at path secret/foo:
#

vault kv put kv/demo name=kubecompass

Create an ExternalSecret
#

Create a YAML file eg.example-secrets.yaml with the following file content:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: vault-example
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: example-sync
  data:
  - secretKey: foobar
    remoteRef:
      key: kv/demo #make sure this matches with what you create in the vault UI
      property: name #this is the secret you create under the path

name & namespace: Identify the External Secret (vault-example) and where it’ll be created (default namespace).
refreshInterval: How often ESO checks for updates to the external secret (15 seconds).
secretStoreRef: Points to the SecretStore providing access to the external secret manager (vault-backend).
target: Specifies the secret to fetch from the external store (example-sync).
data: Describes how to inject the secret into pods:
secretKey: Name of the key in the injected Kubernetes Secret (foobar).
remoteRef: Where to find the secret within the external store:
- key: Path to the secret (foo).
- property: Specific value to extract (my-value).

Deploy using Argo CD
#

Create a git repository and add the above ExternalSecret manifest to it.
Sync the ExternalSecret from the git repository to the cluster by creating an app named demo-secret in Argo CD. For this create a YAML file eg.demo-secrets.yaml with the following file content:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vault-secrets-app
  namespace: argocd # Install the application in Argo CD's namespace
spec:
  project: default  # Assign the Application to the 'default' project
  source:
    repoURL: ${GIT_REPO_URL_CREATED_ABOVE}
    targetRevision: HEAD  # Or a specific branch/tag
    path: ${MANIFEST_PATH}  # Since the ExternalSecret.yaml is at the root of your repo
  destination:
    server: https://kubernetes.default.svc  # The Kubernetes API server
    namespace: external-secrets # Target namespace for the ExternalSecret
  syncPolicy:
    automated: # Enable automatic syncing 
      prune: true