Running Mastodon server on Kubernetes

Recently all eyes of the world have been on Twitter as the company is undergoing a major transformation. At the same time, Mastodon is gaining popularity since more and more people have been looking for an alternative to Twitter.

Here at SoftwareMill, we decided to follow the trend and give Mastodon a chance - create an account for the company, similar to the one we have at Twitter. We didn’t want the account to be associated with somebody else's server and domain - so we decided to run our own Mastodon server.

When I dived deeper into Mastodon’s server configuration, I found some parts of the documentation unclear. As a Mastodon server administrator, you have to fulfil multiple requirements, like owning the domain and configuring the email provider. Mastodon’s architecture consists of components such as PostgreSQL, Redis or Elasticsearch. Launching your server can be a lot to take - especially if you want to follow best practices to ensure high availability, database and storage backup and replication.

In this article, I gathered instructions on how to run your Mastodon server on Kubernetes. Also, I shared some tips on more troublesome tasks while launching your instance so be sure to follow along with the rest of the article. Let’s get started!

Mastodon’s server prerequisites

You want to run the Mastodon server on Kubernetes? Great, the community provides the Helm Chart. So you only have to configure the values.yaml file and that’s it?

If you want to create the server just for testing or development - sure, that would be enough. But in real case scenario when you want to launch the server and allow users to create accounts within your domain as the administrator you have some responsibilities:

• PostgreSQL database backup - here all of the data is persisted: users' accounts, posts, and followers.
• Application secrets - must be safely stored, lost secrets will cause the unavailability of 2FA or logging out users.
• Storage backup - here the media and files uploaded by users are persisted.

For tips about scaling the Mastodon instance and overview of the architecture, head over to Adam’s article. Here, I will focus primarily on the configuration and backup.

In my configuration, database, storage and Kubernetes cluster are hosted on the Google Cloud Platform.

The PostgreSQL database instance is running as a part of Cloud SQL. Create the PostgreSQL instance, then create the mastodon database and mastodon user.

For the storage, you can configure any Object Storage Provider that exposes an S3-compatible API. In GCP create the Storage Bucket with the bucket location specified to Multi-region, since it has the highest availability. You can enable the storage interoperability by generating the HMAC keys and associating them with the Service Account. First, create the Service Account and bind the appropriate role to the Service Account. Then, generate the HMAC key for the Service Account.

Here is the Terraform module for the S3 configuration. In the directory /modules/mastodon/ create file main.tf and variables.tf

main.tf

resource "google_storage_bucket" "mastodon" {
  name     = var.bucket_name
  location = "EU"
}

resource "google_service_account" "mastodon" {
  account_id   = "service-account-mastodon"
  display_name = "A service account for Mastodon server to access storage bucket"
}

/**
 * Binding role objectAdmin to service account and bucket
 */

resource "google_storage_bucket_iam_binding" "binding" {
  bucket = google_storage_bucket.mastodon.name
  role   = "roles/storage.admin"
  members = [
    "serviceAccount:${google_service_account.mastodon.email}"
  ]
}

resource "google_storage_hmac_key" "mastodon" {
  service_account_email = google_service_account.mastodon.email
}

resource "kubernetes_secret" "mastodon" {
  metadata {
    name = "mastodon-production-s3"
  }

  data = {
    AWS_ACCESS_KEY_ID     = google_storage_hmac_key.mastodon.access_id
    AWS_SECRET_ACCESS_KEY = google_storage_hmac_key.mastodon.secret
  }

  type = "Opaque"
}

variables.tf

variable "bucket_name" {
  description = "The name of the GCP bucket"
  type        = string
}

Then, in your root module call the child module mastodon and provide the bucket_name:

module "mastodon" {
  source      = "../modules/mastodon"
  bucket_name = "mastodon-production"
}

In the source argument specify the path to a local directory containing the Mastodon module's configuration files.

Other prerequisites for launching the server are:

• Buying the domain for your server - in your DNS provider configuration, add the A record information with the public IP address of the Load Balancer.
• Preparing the email hosting provider.

Prepare the environment

Before you start configuring the Mastodon server, you need to create the Kubernetes cluster. Here are some tools that I use in the daily work I recommend installing in your cluster if you want to follow best practices.

You can use other tools for example Argo CD instead of Flux - it's just a suggestion.

You can install in Kubernetes cluster:

• Flux CD - bootstrap Flux to Kubernetes cluster. With Flux you will deploy Mastodon following GitOps practices.
• Bitnami’s sealed secrets controller - running the Mastodon server requires providing multiple sensitive secrets. With sealed secrets, you can store your secrets safely in the Git repository.
• Nginx Ingress Controller - ingress will act as a load balancer and a reverse proxy.
• Cert-manager - configure the Cluster Issuer. Cert-manager will automatically issue the TLS certificate.

Server configuration

You can store Mastodon server configuration in the Git repository. Checkout my Git repo where I put code with sample configuration.

Flux CD with reconciliation will ensure that with every change made to the master branch, the application in the cluster is synced. The whole configuration consists of those files:

• source.yaml
• release.yaml
• kustomization.yaml
• kustomizeconfig.yaml
• values.yaml
• sealed-secrets-postgres.yaml
• sealed-secrets-mastodon-server.yaml

Let’s get through this configuration step by step. Initialise the Git repository. Create the source.yaml - here the source of the Helm Chart is defined. Create the GitRepository source with url pointing to Mastodon’s official GitHub Repository. The ref checkout strategy is set to a specified tag. The ignore field excludes all files except for the chart directory.

source.yaml

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: mastodon
  namespace: default
spec:
  interval: 5m
  url: https://github.com/mastodon/mastodon
  ref:
    tag: v4.0.2
  ignore: |
    exclude all
    !/chart/

Now define the HelmRelase. Create the release.yaml.

release.yaml.

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: mastodon
  namespace: default
spec:
  interval: 5m
  chart:
    spec:
      chart: ./chart
      sourceRef:
        kind: GitRepository
        name: mastodon
      interval: 5m
  valuesFrom:
    - kind: ConfigMap
      name: mastodon-values

HelmRelease will be built from the earlier defined GitRepository source. Mastodon server configuration for Helm Chart is quite long and inserting those values directly into the HelmRelase or the separate ConfigMap would be unclear.

Let’s generate the ConfigMap with Kustomize. Helm release upgrade will be triggered every time values.yaml file is updated.

Create the kustomizeconfig.yaml to patch the ConfigMap mastodon-values.

kustomizeconfig.yaml

nameReference:
  - kind: ConfigMap
    version: v1
    fieldSpecs:
      - path: spec/valuesFrom/name
        kind: HelmRelease

Create a kustomization.yaml that generates the ConfigMap:

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: default
resources:
  - source.yaml
  - release.yaml
  - sealed-secret-postgres.yaml
  - sealed-secret-mastodon-server.yaml
configMapGenerator:
  - name: mastodon-values
    files:
      - values.yaml
configurations:
  - kustomizeconfig.yaml

Finally, create the values.yaml. Important fields in this section that you have to provide:

• SMTP - configure the SMTP section according to the settings of your e-mail provider.
• S3 - add your bucket name
• Ingress - configure the domain name and annotations: ingressClassName and cert-manager cluster issuer name.
• External database - specify the database name, username, hostname and port.
• Elasticsearch - in this configuration a single node of Elasticsearch is deployed. In Mastodon, Elasticsearch is used for full-text search, so one node should be enough.
• Redis - by default, a cluster of one master and three replicas are deployed. In this configuration the standalone mode with a single master node is enabled.
• Secrets - provide the correct name of existing secrets. See the instructions below on how to configure secrets with sealed secrets.
• Resources - don’t forget to specify the requests and limits for your environment.

Default values for the Helm Chart are from: https://github.com/mastodon/mastodon/blob/v4.0.2/chart/values.yaml
replicaCount: 2
image:
  tag: "v4.0.2"
  pullPolicy: IfNotPresent
mastodon:
  createAdmin:
    enabled: true
    username: admin
    email: example@email.pl
  local_domain: example.com
  s3:
    enabled: true
    existingSecret: "mastodon-server-secret"
    bucket: "your-bucket-name"
    endpoint: https://storage.googleapis.com
    hostname: storage.googleapis.com
    region: "EU"
  secrets:
    existingSecret: "mastodon-server-secret"
  smtp:
    auth_method: plain
    ca_file: /etc/ssl/certs/ca-certificates.crt
    delivery_method: smtp
    domain: 
    enable_starttls: "auto"
    from_address: example@email.pl
    openssl_verify_mode: peer
    port: 587
    reply_to: example@email.pl
    server: 
    tls: true
    login: example@email.pl
    existingSecret: mastodon-server-secret
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-issuer
    nginx.org/client-max-body-size: 40m
  ingressClassName:
  hosts:
    - host: example.com
      paths:
        - path: "/"
  tls:
    - secretName: mastodon-tls
      hosts:
        - example.com
elasticsearch:
  master:
    masterOnly: false
    replicaCount: 1
  data:
    replicaCount: 0
  coordinating:
    replicaCount: 0
  ingest:
    replicaCount: 0
postgresql:
  enabled: false
  postgresqlHostname: your-psql
  postgresqlPort: 5432
  auth:
    database: mastodon
    username: mastodon
    existingSecret: "mastodon-postgres-secret"
redis:
  architecture: "standalone"
  auth:
    existingSecret: "mastodon-server-secret"
resources: {}

Encrypt the secrets

Use Bitnami’s Sealed Secret to safely store secrets in the Git repository.
Create a secret for the postgres with the key password.

kubectl -n default create secret generic mastodon-postgres-secret \
--from-literal=password=change-me \
--dry-run=client \
-o yaml > basic-auth.yaml

Note: Remember to specify the –dry-run flag - it’s important not to apply the unencrypted secret to the cluster.

Encrypt the created secret with the sealed secret controller running in your cluster. Specify the name of the controller and the namespace:

kubeseal --format=yaml  \
--controller-name sealed-secrets \
--controller-namespace sealed-secrets \
< basic-auth.yaml > basic-auth-sealed.yaml

Delete the unencrypted secret basic-auth.yaml. Now you can safely store the encrypted secret in the Git repo. In values.yaml, set postgresql.auth.ExistingSecret value to the name of your secret.

Other required secrets are: OTP_SECRET, SECRET_KEY_BASE, VAPID_PRIVATE_KEY and VAPID_PUBLIC_KEY. To encrypt them with sealed-secrets they have to be generated manually. The easiest way to do so is to launch locally the docker container with the image mastodon and generate secrets with rake :

docker run -it tootsuite/mastodon:latest sh

Generate the OTP_SECRET:
bundle exec rake secret

Generate the SECRET_KEY_BASE:
bundle exec rake secret

Generate the VAPID_PRIVATE_KEY and VAPID_PUBLIC_KEY:
bundle exec rake mastodon:webpush:generate_vapidkey

Prepare the redis-password, login and password for the SMTP e-mail provider.

Once you have all the necessary secrets proceed with the following steps:

kubectl -n default create secret generic mastodon-server-secret \
--from-literal=login=change-me \
--from-literal=password=change-me \
--from-literal=redis-password=change-me \
--from-literal=OTP_SECRET=change-me \
--from-literal=SECRET_KEY_BASE=change-me \
--from-literal=VAPID_PRIVATE_KEY=change-me \
--from-literal=VAPID_PUBLIC_KEY=change-me \
--dry-run=client \
-o yaml > basic-auth.yaml

Encrypt the secret with the kubeseal in the same way as above. Remember to delete the unencrypted secret. Place your sealed secrets in Git repository.

What about the monitoring?

Mastodon enables StatsD metrics publishing. One solution for monitoring the infrastructure is to combine StatsD metrics and Graphite. I prefer to use Prometheus and Grafana for monitoring and visualisation of the data. To use StatsD metrics, you have to convert them into Prometheus metrics - you can use statsd-exporter as the solution.

Summary

Wrapping up - running your own Mastodon server comes with configuring multiple technologies.

If you decide to launch your instance, keep in mind that you are the administrator and the moderator of the server - after installing Mastodon you have to fill in the server information along with the terms of use.

For me, launching the server was a good opportunity to understand how our internal infrastructure works.

You can follow the SoftwareMill account softwaremill@softwaremill.social on Mastodon.