Migrating Mastodon Helm Chart to Bitnami
A few months back, I wrote an article on how to deploy Mastodon on Kubernetes using the official Mastodon Helm Chart. However, the Mastodon community recently relocated their Helm Chart to a different repository. Unfortunately, this Helm Chart isn’t well maintained, and when we tried to use it, we bumped into a few blockers. Consequently, we opted to utilize the Mastodon Helm Chart provided by Bitnami, which is a more reliable and well-maintained solution.
Bitnami offers an extensive library of Helm Charts for popular applications. While many of these Helm Charts receive regular updates, our experience with the Mastodon Helm Chart was not without its challenges. During the migration process, we encountered several issues, some of which were caused by incorrectly defined templates.
You might wonder if it’s necessary to migrate the Helm Chart to a different repository that's better maintained. The answer is yes. Recently, Mastodon released information about a few vulnerabilities affecting older versions. Unfortunately, these critical security patches are not included in Mastodon's official Helm Chart. Fortunately, the Bitnami team takes the necessary steps to ensure their Helm Chart remains up-to-date.
With that said, here is the tutorial on how to migrate the Mastodon Helm Chart to the one packaged by Bitnami. Let's begin!
Prepare Mastodon instance for the migration
As you might remember from the previous article about deploying Mastodon on Kubernetes, I am using Flux CD to deploy the Helm Chart with the GitOps approach. In this migration process, the goal is to minimize modifications to our existing configurations. The following steps are required:
- Update HelmRelease and adjust values for the Helm Chart.
- Update the HelmRepository.
- Create a new secret for Redis.
Please note that this migration will involve a temporary downtime. Make sure to prepare your
values.yaml file accordingly. I described some sections that may be problematic below. Here is the link to the Mastodon Helm Chart from Bitnami repository.
Apache Helm Chart and Ingress
To use Ingress, you need to enable Apache Helm Chart, which is a dependency. It's worth noting that as of October 25th, Mastodon Helm Chart version 2.1.0 uses Apache Helm Chart as a dependency in version 9.x.x.
Configuring values for Apache Helm Chart was also a bit tricky - version 9.6.5 states that ingress.certManager is deprecated, however when you look into the Helm Chart templates - this value is still required in order to enable TLS in ingress. Also, there was an issue with Apache ConfigMap, but fortunately, it was merged and closed. Need I mention that the migration was quite a challenge?
Apache Helm Chart 10.0.0 release updated the whole Ingress logic. Keep in mind that if you're following this article at a later date, you may need to make adjustments to the Ingress configuration I've described here.
Init job performs the database and elasticsearch migration, checks the connection to S3, and precompiles the assets.
initJob.createAdmin flag must be set to false - otherwise, the init job can’t finish - it is necessary to start up the Mastodon pods.
useSecureWebSocket must be set to true - we bumped into a problem when the website fails to create a WebSocket connection to streaming. The issue is still open here.
For our storage, we are using Google Cloud Storage. During the migration, the S3 bucket tried to use the http protocol even though externalS3.protocol was set to https. Additionally, pods couldn’t find the correct bucket name after the Mastodon start up, so the configuration below was necessary.
forceHttpsS3Protocol: true s3AliasHost: storage.googleapis.com/<bucket-name>
Later in this tutorial, I will also share the configuration required to collect metrics with Prometheus Statsd Exporter and visualize them in a Grafana dashboard. Add here the name of your Prometheus Statsd Exporter service name.
extraConfig: STATSD_ADDR: <name of statds exporter service>.<namespace>:9125
Currently, the Mastodon Helm Chart logic allows you to specify existing secrets for Redis only if you are using external Redis. If you are using Redis as a dependency, based on the Helm Chart templates, the secret name is the same as the Redis Helm Chart release.
Move your existing password to the new secret named:
<your-helm-release>-redis, for example,
Edit your Helm Chart values
After wrapping everything up, your values file should look like this:
# Default values for the Helm Chart are from: https://github.com/bitnami/charts/blob/main/bitnami/mastodon/values.yaml adminUser: admin adminEmail: email@example.com existingSecret: existing-secret-name extraConfig: STATSD_ADDR: <name of statds exporter service>.<namespace>:9125 localDomain: your.domain useSecureWebSocket: true enableS3: true forceHttpsS3Protocol: true s3AliasHost: storage.googleapis.com/your-bucket-name externalS3: host: storage.googleapis.com existingSecret: existing-secret-name protocol: https bucket: your-bucket-name region: region-name existingSecretAccessKeyIDKey: AWS_ACCESS_KEY_ID existingSecretKeySecretKey: AWS_SECRET_ACCESS_KEY smtp: server: smtp-server from_address: your-smtp-email@com domain: your.domain delivery_method: smtp openssl_verify_mode: peer enable_starttls_auto: true auth_method: plain existingSecret: existing-secret-name web: replicaCount: 1 resources: limits: cpu: 1 memory: 1280Mi requests: cpu: 250m memory: 768Mi sidekiq: replicaCount: 1 resources: limits: cpu: 1 memory: 768Mi requests: cpu: 250m memory: 512Mi streaming: replicaCount: 1 resources: limits: cpu: 500m memory: 512Mi requests: cpu: 250m memory: 128Mi initJob: createAdmin: false externalDatabase: user: mastodon database: mastodon existingSecret: existing-postgres-secret-name existingSecretPasswordKey: password redis: enabled: true architecture: standalone existingSecret: mastodon-bitnami-redis master: persistence: size: 2Gi resources: requests: cpu: 100m memory: 1Gi limits: cpu: 1 memory: 2Gi auth: existingSecret: mastodon-bitnami-redis postgresql: enabled: false minio: enabled: false elasticsearch: master: masterOnly: false replicaCount: 1 resources: requests: cpu: 100m memory: 1Gi limits: cpu: 1 memory: 2Gi heapSize: 250m persistence: size: 4Gi coordinating: replicaCount: 0 data: replicaCount: 0 ingest: replicaCount: 0 apache: enabled: true ingress: enabled: true hostname: your.domain path: / ingressClassName: nginx annotations: cert-manager.io/cluster-issuer: letsencrypt-issuer tls: - hosts: - your.domain secretName: your.domain-tls certManager: true
Note that resource requests and limits are just examples - they work for me, but you might want to adjust them. If you want to know how to prepare basics for Mastodon Helm Chart deployment, like the secrets I mentioned earlier, check out my previous post on how to deploy Mastodon in Kubernetes.
Prepare the HelmRepository:
Update your HelmRepository file:
kind: HelmRepository metadata: name: mastodon namespace: default spec: type: oci interval: 5m url: oci://registry-1.docker.io/bitnamicharts
Prepare the HelmRelease:
Update your HelmRelease file:
apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: mastodon-bitnami namespace: default spec: interval: 5m chart: spec: chart: mastodon version: 3.0.5 sourceRef: kind: HelmRepository name: mastodon interval: 5m valuesFrom: - kind: ConfigMap name: mastodon-values - kind: ConfigMap name: pgsql-14-configuration valuesKey: DB_HOST targetPath: externalDatabase.host - kind: ConfigMap name: pgsql-14-configuration valuesKey: DB_PORT targetPath: externalDatabase.port
Note that I’m not adding values directly to the HelmRelease file - instead, I’m referring to a ConfigMap created with kustomize. Link to the Flux CD documentation is here.
Start the migration
Create a new HelmRelease with a different name, for example, mastodon-bitnami. This is important - we can’t just update the current Mastodon HelmRelease, we need to create a separate one. The InitJob mentioned before has post-install and pre-upgrade hooks. The latter means that the InitJob will be executed before any resources are updated. Consequently, any changes done to existing configmaps wouldn't be taken into account. The InitJob would continue to use outdated vars and continuously fail.
Suspend your old Mastodon HelmRelease. Scale your old Mastodon deployments to 0 replicas - we want to turn down the server but keep the current configuration and Redis setup, just in case..
Add new HelmRelease - observe the initJob. When it succeeds, the Streaming, Web, and Sidekiq pods should be up along with the Redis, Elastichsearch, and Apache. Check your ingress - cert-manager should have created the new certificate by now.
Mastodon monitoring with Prometheus and Grafana
Mastodon enables StatsD metrics. To fetch those metrics with Prometheus and to visualize them in Grafana they have to be converted first. Statsd-exporter from the Prometheus Community translates StatsD metrics to Prometheus metrics via configured mapping rules.
Install Statsd-exporter Helm Chart in your monitoring namespace.
apiVersion: source.toolkit.fluxcd.io/v1beta1 kind: HelmRepository metadata: name: prometheus-community namespace: flux-system spec: interval: 5m0s url: https://prometheus-community.github.io/helm-charts
apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: prometheus-statsd-exporter spec: chart: spec: chart: prometheus-statsd-exporter sourceRef: kind: HelmRepository name: prometheus-community namespace: flux-system version: 0.9.1 interval: 5m0s releaseName: prometheus-statsd-exporter valuesFrom: - kind: ConfigMap name: prometheus-statsd-exporter-values valuesKey: prometheus-statsd-exporter-values.yaml
Refer to the values in ConfigMap created with Kustomize - this will keep our HelmRelease nice and clean.
To translate metrics from the Statsd format to the one that will be readable by Prometheus, we need to create mappings. I found a very useful article on how to monitor Mastodon. The author of the blog shares his mapping configuration and also a Grafana Dashboard which works perfectly.
Add mappings to the
serviceMonitor: enabled: true namespace: monitoring endpoint: "/metrics" statsd: mappingConfig: |- # add here mappings
Don’t forget to add the name of your Prometheus Statsd exporter service to the Mastodon Helm Chart values under the
STATS_ADDR env. Add your dashboard to Grafana, and you should be able to collect statistics and monitor your instance.
By now, you should have completed a successful migration of your Mastodon instance with as little downtime as possible. While this Helm Chart may still require a few pull requests and fixes, it's reassuring to know that the Bitnami Github repository is well-maintained and receives regular updates. Enjoy a smooth experience managing your Mastodon server!
reviewed by: Paweł Maszota