Safe storage of Kubernetes Secrets with Mozilla SOPS and IaC

Origin story

If you have used Kubernetes for more than a few weeks, you have probably encountered a situation where you needed to modify a Secret, let’s say - a password for a database user required by your deployed application. You need to launch a client like kubectl or Lens, find the correct Secret, decipher the base64 value, change it, encode and update the k8s Secret. But in reality, these secrets are not safe at all. Base64 values are reversibly encoded, a shorter form of their decoded variant. If it gets leaked or otherwise accessed by a bad actor, they will obtain this data. On top of that, if you want to follow best practices such as infrastructure-as-code, you must also store the passwords with your config. Things get complicated fast, don’t they?

Usually, you would not keep a password in a Git repository, after all, keeping secrets in plaintext is a Bad Idea™. Considering that all changes made in Git are saved in history, all previous passwords are also held there. That makes things even worse… and again, Base64 doesn’t help there in any measure.

We need a way to encrypt our sensitive data in a way that will be safe and usable by other tools and systems. Enter Mozilla SOPS.

Introduction

SOPS stands for Secrets OPerationS and is a CLI tool used for the encryption, and decryption of files in YAML, JSON, BINARY, ENV, and INI formats using various backends, including PGP, age, AWS KMS, GCP KMS, Azure Key Vault and HashiCorp Vault. That makes it cloud-agnostic. Works on Windows, Linux, and macOS.

Where SOPS truly shines, in my opinion, is when it is used for keeping confidential data in the Git repo to keep your application config complete and centralized.

For example, if you want to deploy PostgreSQL alongside your app using Helm on AWS, you don’t need to separate your sensitive data from insensitive ones. Let’s consider this values.yaml file example:

  enabled: true
  postgresqlUsername: "postgres"
  postgresqlPassword: "pass"
  postgresqlDatabase: "pass"
  service:
    port: 5432

Let’s assume that we only need to encrypt postgresqlPassword. We can leave the rest as it is by preparing config for SOPS, where we tell SOPS that it only needs to encrypt values of keys that match a specific regular expression (regex):

creation_rules:
  - path_regexp: \.(yml|yaml)$
    encrypted_regex: "^(.*[Pp]ass.*)$"
    key_groups:
      - kms:
          - arn: "arn:aws:kms:eu-central-1:999999999999:key/<key_uuid>"
            role: "arn:aws:iam::888888888888:role/sops-data-key-encryption"

Here we specify that we only want to encrypt files ending with .yml and .yaml. Inside them, we want to keep all the contents as they were, except the values of keys that have “pass” or “Pass” somewhere in the name. We want to use an AWS KMS key along with an IAM role that allows us to use it.

As a result of sops -e values.yaml command, we would receive something like this:

  enabled: true
  postgresqlUsername: "postgres"
  postgresqlPassword: ENC[<redacted>]
  postgresqlDatabase: "db"
  service:
    port: 5432
sops:
    kms:
        - arn: arn:aws:kms:eu-central-1:999999999999:key/<key_uuid>
          role: arn:aws:iam::888888888888:role/sops-encryption
          created_at: "2023-01-09T10:25:12Z"
          enc: <redacted>
          aws_profile: ""
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2023-01-09T10:25:12Z"
    mac: <redacted>
    pgp: []
    encrypted_regex: ^(.*[Pp]ass.*)$
    version: 3.7.3

As you can see, the file has grown quite a bit. Let’s go through it.

First, we have our postgresql section which stays almost the same, except that the PostgreSQL password has been encrypted. That’s exactly what we wanted. What we didn’t expect was to see a whole new sops section. It’s metadata based on the .sops.yaml config file that SOPS adds to all encrypted files. Besides timedates of file creation and modification and SOPS version, there is also a copy of the encryption/decryption key included used for decryption later on. You can save it to the new file by redirecting output in a standard Unix manner:

Once the file is encrypted, you can open the file. To do that you only need a valid AWS profile and permissions. Type sops values.enc.yaml command to launch your default text editor and see decrypted contents of the file.

Comparison to Sealed Secrets

SOPS is a tool that works similarly to Sealed Secrets, another Kubernetes Secret encryption tool. The main difference lies in the place where encryption happens. SOPS encrypts and decrypts the files client-side, while Sealed Secrets' encryption/decryption takes place on the server side, on the cluster itself. This may become a security issue for SOPS because if we’d like to use it with CD pipelines (which is probably what you want sooner than later) you need to grant access to the key to be able to deploy the app. CI/CD solutions like Flux or ArgoCD support SOPS and it is less likely to be a real issue. Sealed Secrets is not prone to that problem.

On the other hand, Sealed Secrets can only encrypt the whole file, thus making it hard to read for humans. Also, it doesn’t support any cloud key management system, the keys are taken care of by a dedicated Kubernetes controller.

Summary

SOPS is a very versatile tool for managing encrypted secrets. It works amazingly with YAML and JSON files, but can also work with others, including binaries. CI/CD pipelines can work with it easily, but it’s even better if you use a CD tool like Flux, which seamlessly integrates SOPS.

This is just an introduction, in another article, I am going to show how to use it in the GitOps environment based on Flux.

Tech review: Łukasz Lenart