Your Kubernetes secrets are not secret

The first thing I do when I audit a Kubernetes cluster is run a single command. kubectl get secrets -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\n"}{end}' | head -30. This lists every secret in every namespace. The output is usually long. Then I pick one at random and run kubectl get secret <name> -n <namespace> -o jsonpath='{.data.password}' | base64 -d. The password appears in my terminal in plaintext.

Nobody has ever been surprised by this except the engineering team that owns the cluster.

Kubernetes Secrets are not encrypted. They are base64-encoded. Base64 is not encryption. It is an encoding. The difference matters. Encryption requires a key and a mathematical operation that cannot be reversed without the key. Base64 requires nothing. You pipe it through base64 -d and you have the original value. The Kubernetes documentation says this explicitly: “Kubernetes Secrets are, by default, stored as base64-encoded unencrypted data.” Most engineers I have worked with have not read that sentence.

I have found production database passwords in Kubernetes Secrets that were committed to Git repositories. I have found Stripe API keys in ConfigMaps, which are not even secrets, because someone did not know the difference. I have found clusters where every developer had cluster-admin and could read every secret in every namespace, including the production database credentials, the JWT signing key, and the third-party API tokens for services that could move money.

This is not a rare problem. It is the default state of Kubernetes unless someone has done specific work to prevent it.

What Kubernetes Secrets actually are

A Kubernetes Secret is a Kubernetes object, stored in etcd, that happens to have a type field set to Opaque and a data field that holds base64-encoded values. That is it. The encoding exists so that the Secret can hold binary data, not so that it can hide it. The Kubernetes API server does not encrypt the values. It stores them in etcd as base64 strings.

If etcd is not encrypted at rest, and in most clusters it is not, then anyone who can read the etcd data directory on the node, or anyone who can port-forward to the etcd service, can read every secret in the cluster without going through the Kubernetes API. If etcd is encrypted at rest, which requires configuring the API server with a KMS provider or a static encryption key, then the secrets are encrypted in etcd but still transmitted in plaintext over the API. Anyone with get permissions on secrets can decode them with base64 -d.

The security model of Kubernetes Secrets is RBAC. If your RBAC is tight, secrets are as safe as the API server is. If your RBAC is loose, and in most early-stage clusters it is, secrets are effectively public within the cluster.

The question is not whether Kubernetes Secrets are secure. They are a mechanism. The question is whether your cluster is configured to make them secure, and for most teams, the answer is no.

The three problems

When I audit a cluster’s secret management, I look for three problems. Every cluster I have audited has at least two of them.

Problem one: secrets in Git

Teams store Kubernetes Secrets in Git. They do this because GitOps requires manifests in Git, and Secrets are manifests, so they commit them. The Secret manifest contains the base64-encoded value. Anyone with read access to the repository can decode it. The CI pipeline has access. The contributor who was added last week has access. The bot account that was granted repo access for a migration has access.

# This is what gets committed. It is not secret.
apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
  namespace: production
type: Opaque
data:
  password: c2VjcmV0LXBhc3N3b3JkLWRvLW5vdC1jb21taXQ=
  # echo 'c2VjcmV0LXBhc3N3b3JkLWRvLW5vdC1jb21taXQ=' | base64 -d
  # => secret-password-do-not-commit

I have run git log -p -- '**/secret*.yaml' in repositories and found three years of rotated database passwords, all in plaintext after a trivial base64 decode. Every rotation was a new secret committed to version control. Every old password is still in the git history. Rotating the secret in Kubernetes did not remove it from Git.

Problem two: RBAC that grants secret access to everyone

The default RBAC in a cluster without explicit namespace isolation grants developers broad access. In clusters where everyone has cluster-admin, which is more common than anyone admits, every developer can read every secret in every namespace. The production database password is readable by the intern who started on Monday.

# Check who can read secrets in production
kubectl auth can-i get secrets -n production --as=developer@company.com
# yes

When I run this in audits, the answer is almost always yes. The fix is scoped RBAC: a Role that grants access only to the secrets in the developer’s namespace, not to production secrets. But scoped RBAC requires namespace isolation, and namespace isolation requires a design decision that most early-stage teams have not made.

Problem three: no rotation

Secrets are static. The database password was set when the cluster was created. It has not changed since. The JWT signing key has been the same since the first deploy. The third-party API tokens are the same ones that were pasted into a Slack channel eight months ago by a contractor who no longer works at the company.

Rotation is not a Kubernetes feature. It is a process, and the process requires tooling. Without a secret management system that supports rotation, the only way to rotate a secret is to generate a new one, update the Kubernetes Secret, restart the pods that use it, and verify that everything works. Most teams do not do this because it is manual, it is risky, and nobody has time. So the secrets sit. For years.

What actually works without Vault

HashiCorp Vault is the answer most teams think they need. Vault is excellent. It is also a distributed system that requires its own high availability, its own backup strategy, its own access management, and its own operational expertise. I have seen teams spend three weeks standing up a Vault cluster and then leave the auto-unseal configured with a single key because nobody wanted to deal with the Shamir’s secret sharing ceremony. The Vault cluster was less secure than the Kubernetes Secrets it replaced.

For most teams, especially early-stage ones, there are two patterns that work, do not require running a secrets management server, and solve the three problems above.

Pattern one: Sealed Secrets

Sealed Secrets is a controller by Bitnami that encrypts secrets with a public key. You encrypt a Kubernetes Secret with the controller’s public key, and the result is a SealedSecret manifest that is safe to commit to Git. The controller, running in the cluster, holds the private key and decrypts the SealedSecret into a regular Kubernetes Secret.

# Install the controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v2.15.0/controller.yaml

# Create a sealed secret from a regular secret
echo -n 'my-actual-password' | kubectl create secret generic db-password \
  --dry-run=client --from-file=password=/dev/stdin -o yaml | \
  kubeseal --controller-namespace=kube-system -o yaml > db-password-sealed.yaml

The output db-password-sealed.yaml is safe to commit to Git. The encrypted value cannot be decrypted without the private key, which lives only in the cluster. When you apply the SealedSecret, the controller decrypts it and creates a regular Kubernetes Secret that pods can mount.

This solves problem one: secrets in Git. The committed manifest is encrypted. It does not solve problem two or three directly, but it makes GitOps compatible with secrets without requiring a separate secrets store.

The tradeoff is rotation. When you need to rotate a secret, you generate a new SealedSecret and commit it. The controller updates the underlying Secret. If the private key is compromised, every SealedSecret in the repository is compromised, and you need to re-encrypt all of them. The private key is stored in the cluster as a regular Kubernetes Secret, so if someone has cluster-admin, they can extract it and decrypt everything.

Sealed Secrets is a good first step. It is better than committing base64-encoded secrets to Git. It is not a complete solution.

Pattern two: External Secrets Operator

External Secrets Operator (ESO) is a controller that fetches secrets from an external secret store and creates Kubernetes Secrets from them. The external store can be AWS Secrets Manager, Google Secret Manager, Azure Key Vault, HashiCorp Vault, or any of the fifteen backends ESO supports.

The pattern works like this. Your secrets live in AWS Secrets Manager. You do not commit them to Git. In Git, you commit an ExternalSecret manifest that references the AWS secret by name. The controller, running in the cluster, authenticates to AWS using an IAM role, fetches the secret value, and creates a Kubernetes Secret from it.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: database-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: production/database/password
        property: password

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-central-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

The service account external-secrets-sa is annotated with an IAM role ARN via IRSA (IAM Roles for Service Accounts). Only the External Secrets Controller uses this service account. Developers do not have access to it. Developers commit ExternalSecret manifests to Git. They never see the actual secret values. The secret values live in AWS Secrets Manager, where they can be rotated, audited, and access-controlled independently of the Kubernetes cluster.

This solves all three problems. Secrets are not in Git. RBAC can be scoped so developers cannot read the generated Kubernetes Secrets directly, only the controller can. Rotation happens in AWS Secrets Manager, and ESO refreshes the Kubernetes Secret on its refreshInterval, so pods pick up new values on the next restart or when the controller updates the mounted secret.

The tradeoff is that you need a cloud provider’s secret management service, or Vault. If you are running on AWS, Secrets Manager is the path of least resistance. It costs $0.40 per secret per month. For a startup with 30 secrets, that is $12 per month. The cost is not the issue. The issue is that someone has to put the secrets there in the first place, and that someone has to have access to both the cloud console and the application. That is an operational process, not a tooling problem.

The audit I ran last month

A fintech in Tallinn asked me to audit their cluster. They had 14 namespaces, 6 of them production. They were using Helm to deploy, and every Helm chart had a values file with secrets committed to Git. The Helm values files were in a private repository, which the CTO said made them “secure enough.” I ran git log -p -- 'values*.yaml' and found 47 secrets committed over 18 months. Database passwords, API keys, a private signing key for their JWT tokens. I decoded every one of them in under five minutes.

The CTO said: “But the repo is private.” I asked how many people had access. He checked. 23 people, including a contractor who had been removed from the organization but whose SSH key had not been revoked from the GitHub settings. The contractor had left eight months ago. His access was still active.

I replaced their Helm-based secret management with External Secrets Operator in two days. The secrets moved to AWS Secrets Manager. The Helm values files now reference ExternalSecret manifests. No secret values are in Git. The CI pipeline never sees secret values. Developers can deploy without ever touching credentials.

The JWT signing key was the hardest part. It was used by every service for token validation. Rotating it required a coordinated deploy of all services, with a grace period where both the old and new keys were accepted. That rotation took a week. It should have been done on day one. Instead, the same signing key had been in use for 18 months, committed to a Git repository that 23 people could read.

The baseline I install

When I set up a cluster for a team, the secret management baseline is four things:

1. etcd encryption at rest. Enable it on the API server. It is a one-time configuration during cluster creation. If the cluster already exists, it requires a two-step rotation: enable encryption, then re-encrypt all existing secrets. It is not optional.

# API server flags
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml

# encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <32-byte-base64-key>
      - identity: {}

2. External Secrets Operator with a cloud backend. Secrets live in the cloud provider’s secret manager. Git contains only ExternalSecret references. No secret values in version control.

3. RBAC that denies developers access to secrets in production namespaces. Developers get access to the secrets in their own namespace, typically dev or staging. Production secrets are readable only by the External Secrets Controller and by service accounts used by application pods. Humans do not need to read production secrets. If they do, they use a break-glass account that is audited.

4. Rotation policy. Database passwords rotate quarterly. API keys rotate on the provider’s schedule. The JWT signing key rotates annually. ESO’s refreshInterval ensures the Kubernetes Secrets stay current with the external store. Rotation is a calendar event, not an emergency.

None of this requires Vault. None of it requires a platform team. It requires a controller, a cloud secret manager, and discipline. The discipline is the part most teams skip, and skipping it is why I keep finding database passwords in Git repositories.

What I tell teams

The next time you run kubectl get secret <name> -o jsonpath='{.data.*}' | base64 -d, look at what comes out. That is the same data that lives in etcd, in your Git history, and in your CI logs. If any of those are acceptable places for that data to be, you are fine. If any of them are not, you have work to do. The work is not complicated. It is a controller, a secret store, and an RBAC policy. The complicated part is accepting that the default was never secure and that base64 was never a lock.

It was always a box with the key taped to the side.