Migrating Mastodon Helm Chart to Bitnami

A few months back, I wrote an article on how to deploy Mastodon on Kubernetes using the official Mastodon Helm Chart. However, the Mastodon community recently relocated their Helm Chart to a different repository. Unfortunately, this Helm Chart isn’t well maintained, and when we tried to use it, we bumped into a few blockers. Consequently, we opted to utilize the Mastodon Helm Chart provided by Bitnami, which is a more reliable and well-maintained solution.

Bitnami offers an extensive library of Helm Charts for popular applications. While many of these Helm Charts receive regular updates, our experience with the Mastodon Helm Chart was not without its challenges. During the migration process, we encountered several issues, some of which were caused by incorrectly defined templates.

You might wonder if it’s necessary to migrate the Helm Chart to a different repository that's better maintained. The answer is yes. Recently, Mastodon released information about a few vulnerabilities affecting older versions. Unfortunately, these critical security patches are not included in Mastodon's official Helm Chart. Fortunately, the Bitnami team takes the necessary steps to ensure their Helm Chart remains up-to-date.

With that said, here is the tutorial on how to migrate the Mastodon Helm Chart to the one packaged by Bitnami. Let's begin!

Prepare Mastodon instance for the migration

As you might remember from the previous article about deploying Mastodon on Kubernetes, I am using Flux CD to deploy the Helm Chart with the GitOps approach. In this migration process, the goal is to minimize modifications to our existing configurations. The following steps are required:

• Update HelmRelease and adjust values for the Helm Chart.
• Update the HelmRepository.
• Create a new secret for Redis.

Please note that this migration will involve a temporary downtime. Make sure to prepare your values.yaml file accordingly. I described some sections that may be problematic below. Here is the link to the Mastodon Helm Chart from Bitnami repository.

Apache Helm Chart and Ingress

To use Ingress, you need to enable Apache Helm Chart, which is a dependency. It's worth noting that as of October 25th, Mastodon Helm Chart version 2.1.0 uses Apache Helm Chart as a dependency in version 9.x.x.

Configuring values for Apache Helm Chart was also a bit tricky - version 9.6.5 states that ingress.certManager is deprecated, however when you look into the Helm Chart templates - this value is still required in order to enable TLS in ingress. Also, there was an issue with Apache ConfigMap, but fortunately, it was merged and closed. Need I mention that the migration was quite a challenge?

Apache Helm Chart 10.0.0 release updated the whole Ingress logic. Keep in mind that if you're following this article at a later date, you may need to make adjustments to the Ingress configuration I've described here.

initJob

Init job performs the database and elasticsearch migration, checks the connection to S3, and precompiles the assets.

initJob.createAdmin flag must be set to false - otherwise, the init job can’t finish - it is necessary to start up the Mastodon pods.

WebSocket connection

useSecureWebSocket must be set to true - we bumped into a problem when the website fails to create a WebSocket connection to streaming. The issue is still open here.

Storage

For our storage, we are using Google Cloud Storage. During the migration, the S3 bucket tried to use the http protocol even though externalS3.protocol was set to https. Additionally, pods couldn’t find the correct bucket name after the Mastodon start up, so the configuration below was necessary.

forceHttpsS3Protocol: true
s3AliasHost: storage.googleapis.com/<bucket-name>

Enable metrics

Later in this tutorial, I will also share the configuration required to collect metrics with Prometheus Statsd Exporter and visualize them in a Grafana dashboard. Add here the name of your Prometheus Statsd Exporter service name.

extraConfig:
  STATSD_ADDR: <name of statds exporter service>.<namespace>:9125

Redis secret

Currently, the Mastodon Helm Chart logic allows you to specify existing secrets for Redis only if you are using external Redis. If you are using Redis as a dependency, based on the Helm Chart templates, the secret name is the same as the Redis Helm Chart release.

Move your existing password to the new secret named: <your-helm-release>-redis, for example, mastodon-bitnami-redis.

Edit your Helm Chart values

After wrapping everything up, your values file should look like this:

# Default values for the Helm Chart are from: https://github.com/bitnami/charts/blob/main/bitnami/mastodon/values.yaml

adminUser: admin
adminEmail: admin@email.com
existingSecret: existing-secret-name
extraConfig:
  STATSD_ADDR: <name of statds exporter service>.<namespace>:9125

localDomain: your.domain
useSecureWebSocket: true

enableS3: true
forceHttpsS3Protocol: true
s3AliasHost: storage.googleapis.com/your-bucket-name
externalS3:
  host: storage.googleapis.com
  existingSecret: existing-secret-name
  protocol: https
  bucket: your-bucket-name
  region: region-name
  existingSecretAccessKeyIDKey: AWS_ACCESS_KEY_ID
  existingSecretKeySecretKey: AWS_SECRET_ACCESS_KEY

smtp:
  server: smtp-server
  from_address: your-smtp-email@com
  domain: your.domain
  delivery_method: smtp
  openssl_verify_mode: peer
  enable_starttls_auto: true
  auth_method: plain
  existingSecret: existing-secret-name

web:
  replicaCount: 1
  resources:
    limits:
      cpu: 1
      memory: 1280Mi
    requests:
      cpu: 250m
      memory: 768Mi

sidekiq:
  replicaCount: 1
  resources:
    limits:
      cpu: 1
      memory: 768Mi
    requests:
      cpu: 250m
      memory: 512Mi

streaming:
  replicaCount: 1
  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 250m
      memory: 128Mi

initJob:
  createAdmin: false

externalDatabase:
  user: mastodon
  database: mastodon
  existingSecret: existing-postgres-secret-name
  existingSecretPasswordKey: password

redis:
  enabled: true
  architecture: standalone
  existingSecret: mastodon-bitnami-redis
  master:
    persistence:
      size: 2Gi
    resources:
      requests:
        cpu: 100m
        memory: 1Gi
      limits:
        cpu: 1
        memory: 2Gi
  auth:
    existingSecret: mastodon-bitnami-redis

postgresql:
  enabled: false

minio:
  enabled: false

elasticsearch:
  master:
    masterOnly: false
    replicaCount: 1
    resources:
      requests:
        cpu: 100m
        memory: 1Gi
      limits:
        cpu: 1
        memory: 2Gi
    heapSize: 250m
    persistence:
      size: 4Gi
  coordinating:
    replicaCount: 0
  data:
    replicaCount: 0
  ingest:
    replicaCount: 0

apache:
  enabled: true
  ingress:
    enabled: true
    hostname: your.domain
    path: /
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-issuer
    tls:
      - hosts:
          - your.domain
        secretName: your.domain-tls
    certManager: true

Note that resource requests and limits are just examples - they work for me, but you might want to adjust them. If you want to know how to prepare basics for Mastodon Helm Chart deployment, like the secrets I mentioned earlier, check out my previous post on how to deploy Mastodon in Kubernetes.

Prepare the HelmRepository:

Update your HelmRepository file:

kind: HelmRepository
metadata:
  name: mastodon
  namespace: default
spec:
  type: oci
  interval: 5m
  url: oci://registry-1.docker.io/bitnamicharts

Prepare the HelmRelease:

Update your HelmRelease file:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: mastodon-bitnami
  namespace: default
spec:
  interval: 5m
  chart:
    spec:
      chart: mastodon
      version: 3.0.5
      sourceRef:
        kind: HelmRepository
        name: mastodon
      interval: 5m
  valuesFrom:
    - kind: ConfigMap
      name: mastodon-values
    - kind: ConfigMap
      name: pgsql-14-configuration
      valuesKey: DB_HOST
      targetPath: externalDatabase.host
    - kind: ConfigMap
      name: pgsql-14-configuration
      valuesKey: DB_PORT
      targetPath: externalDatabase.port

Note that I’m not adding values directly to the HelmRelease file - instead, I’m referring to a ConfigMap created with kustomize. Link to the Flux CD documentation is here.

Start the migration

Create a new HelmRelease with a different name, for example, mastodon-bitnami. This is important - we can’t just update the current Mastodon HelmRelease, we need to create a separate one. The InitJob mentioned before has post-install and pre-upgrade hooks. The latter means that the InitJob will be executed before any resources are updated. Consequently, any changes done to existing configmaps wouldn't be taken into account. The InitJob would continue to use outdated vars and continuously fail.

Suspend your old Mastodon HelmRelease. Scale your old Mastodon deployments to 0 replicas - we want to turn down the server but keep the current configuration and Redis setup, just in case..

Add new HelmRelease - observe the initJob. When it succeeds, the Streaming, Web, and Sidekiq pods should be up along with the Redis, Elastichsearch, and Apache. Check your ingress - cert-manager should have created the new certificate by now.

Mastodon monitoring with Prometheus and Grafana

Mastodon enables StatsD metrics. To fetch those metrics with Prometheus and to visualize them in Grafana they have to be converted first. Statsd-exporter from the Prometheus Community translates StatsD metrics to Prometheus metrics via configured mapping rules.

Install Statsd-exporter Helm Chart in your monitoring namespace.

Add HelmRepository:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: prometheus-community
  namespace: flux-system
spec:
  interval: 5m0s
  url: https://prometheus-community.github.io/helm-charts

Add HelmRelease:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: prometheus-statsd-exporter
spec:
  chart:
    spec:
      chart: prometheus-statsd-exporter
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
        namespace: flux-system
      version: 0.9.1
  interval: 5m0s
  releaseName: prometheus-statsd-exporter
  valuesFrom:
    - kind: ConfigMap
      name: prometheus-statsd-exporter-values
      valuesKey: prometheus-statsd-exporter-values.yaml

Refer to the values in ConfigMap created with Kustomize - this will keep our HelmRelease nice and clean.

To translate metrics from the Statsd format to the one that will be readable by Prometheus, we need to create mappings. I found a very useful article on how to monitor Mastodon. The author of the blog shares his mapping configuration and also a Grafana Dashboard which works perfectly.

Add mappings to the values.yaml:

serviceMonitor:
  enabled: true
  namespace: monitoring
  endpoint: "/metrics"
statsd:
  mappingConfig: |-
    # add here mappings

Don’t forget to add the name of your Prometheus Statsd exporter service to the Mastodon Helm Chart values under the STATS_ADDR env. Add your dashboard to Grafana, and you should be able to collect statistics and monitor your instance.

Summary

By now, you should have completed a successful migration of your Mastodon instance with as little downtime as possible. While this Helm Chart may still require a few pull requests and fixes, it's reassuring to know that the Bitnami Github repository is well-maintained and receives regular updates. Enjoy a smooth experience managing your Mastodon server!

reviewed by: Paweł Maszota