Migrating Mastodon Helm Chart to Bitnami
A few months back, I wrote an article on how to deploy Mastodon on Kubernetes using the official Mastodon Helm Chart. However, the Mastodon community recently relocated their Helm Chart to a different repository. Unfortunately, this Helm Chart isn’t well maintained, and when we tried to use it, we bumped into a few blockers. Consequently, we opted to utilize the Mastodon Helm Chart provided by Bitnami, which is a more reliable and well-maintained solution.
Bitnami offers an extensive library of Helm Charts for popular applications. While many of these Helm Charts receive regular updates, our experience with the Mastodon Helm Chart was not without its challenges. During the migration process, we encountered several issues, some of which were caused by incorrectly defined templates.
You might wonder if it’s necessary to migrate the Helm Chart to a different repository that's better maintained. The answer is yes. Recently, Mastodon released information about a few vulnerabilities affecting older versions. Unfortunately, these critical security patches are not included in Mastodon's official Helm Chart. Fortunately, the Bitnami team takes the necessary steps to ensure their Helm Chart remains up-to-date.
With that said, here is the tutorial on how to migrate the Mastodon Helm Chart to the one packaged by Bitnami. Let's begin!
Prepare Mastodon instance for the migration
As you might remember from the previous article about deploying Mastodon on Kubernetes, I am using Flux CD to deploy the Helm Chart with the GitOps approach. In this migration process, the goal is to minimize modifications to our existing configurations. The following steps are required:
- Update HelmRelease and adjust values for the Helm Chart.
- Update the HelmRepository.
- Create a new secret for Redis.
Please note that this migration will involve a temporary downtime. Make sure to prepare your values.yaml
file accordingly. I described some sections that may be problematic below. Here is the link to the Mastodon Helm Chart from Bitnami repository.
Apache Helm Chart and Ingress
To use Ingress, you need to enable Apache Helm Chart, which is a dependency. It's worth noting that as of October 25th, Mastodon Helm Chart version 2.1.0 uses Apache Helm Chart as a dependency in version 9.x.x.
Configuring values for Apache Helm Chart was also a bit tricky - version 9.6.5 states that ingress.certManager is deprecated, however when you look into the Helm Chart templates - this value is still required in order to enable TLS in ingress. Also, there was an issue with Apache ConfigMap, but fortunately, it was merged and closed. Need I mention that the migration was quite a challenge?
Apache Helm Chart 10.0.0 release updated the whole Ingress logic. Keep in mind that if you're following this article at a later date, you may need to make adjustments to the Ingress configuration I've described here.
initJob
Init job performs the database and elasticsearch migration, checks the connection to S3, and precompiles the assets.
initJob.createAdmin flag must be set to false - otherwise, the init job can’t finish - it is necessary to start up the Mastodon pods.
WebSocket connection
useSecureWebSocket must be set to true - we bumped into a problem when the website fails to create a WebSocket connection to streaming. The issue is still open here.
Storage
For our storage, we are using Google Cloud Storage. During the migration, the S3 bucket tried to use the http protocol even though externalS3.protocol was set to https. Additionally, pods couldn’t find the correct bucket name after the Mastodon start up, so the configuration below was necessary.
forceHttpsS3Protocol: true
s3AliasHost: storage.googleapis.com/<bucket-name>
Enable metrics
Later in this tutorial, I will also share the configuration required to collect metrics with Prometheus Statsd Exporter and visualize them in a Grafana dashboard. Add here the name of your Prometheus Statsd Exporter service name.
extraConfig:
STATSD_ADDR: <name of statds exporter service>.<namespace>:9125
Redis secret
Currently, the Mastodon Helm Chart logic allows you to specify existing secrets for Redis only if you are using external Redis. If you are using Redis as a dependency, based on the Helm Chart templates, the secret name is the same as the Redis Helm Chart release.
Move your existing password to the new secret named: <your-helm-release>-redis
, for example, mastodon-bitnami-redis
.
Edit your Helm Chart values
After wrapping everything up, your values file should look like this:
# Default values for the Helm Chart are from: https://github.com/bitnami/charts/blob/main/bitnami/mastodon/values.yaml
adminUser: admin
adminEmail: admin@email.com
existingSecret: existing-secret-name
extraConfig:
STATSD_ADDR: <name of statds exporter service>.<namespace>:9125
localDomain: your.domain
useSecureWebSocket: true
enableS3: true
forceHttpsS3Protocol: true
s3AliasHost: storage.googleapis.com/your-bucket-name
externalS3:
host: storage.googleapis.com
existingSecret: existing-secret-name
protocol: https
bucket: your-bucket-name
region: region-name
existingSecretAccessKeyIDKey: AWS_ACCESS_KEY_ID
existingSecretKeySecretKey: AWS_SECRET_ACCESS_KEY
smtp:
server: smtp-server
from_address: your-smtp-email@com
domain: your.domain
delivery_method: smtp
openssl_verify_mode: peer
enable_starttls_auto: true
auth_method: plain
existingSecret: existing-secret-name
web:
replicaCount: 1
resources:
limits:
cpu: 1
memory: 1280Mi
requests:
cpu: 250m
memory: 768Mi
sidekiq:
replicaCount: 1
resources:
limits:
cpu: 1
memory: 768Mi
requests:
cpu: 250m
memory: 512Mi
streaming:
replicaCount: 1
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 250m
memory: 128Mi
initJob:
createAdmin: false
externalDatabase:
user: mastodon
database: mastodon
existingSecret: existing-postgres-secret-name
existingSecretPasswordKey: password
redis:
enabled: true
architecture: standalone
existingSecret: mastodon-bitnami-redis
master:
persistence:
size: 2Gi
resources:
requests:
cpu: 100m
memory: 1Gi
limits:
cpu: 1
memory: 2Gi
auth:
existingSecret: mastodon-bitnami-redis
postgresql:
enabled: false
minio:
enabled: false
elasticsearch:
master:
masterOnly: false
replicaCount: 1
resources:
requests:
cpu: 100m
memory: 1Gi
limits:
cpu: 1
memory: 2Gi
heapSize: 250m
persistence:
size: 4Gi
coordinating:
replicaCount: 0
data:
replicaCount: 0
ingest:
replicaCount: 0
apache:
enabled: true
ingress:
enabled: true
hostname: your.domain
path: /
ingressClassName: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-issuer
tls:
- hosts:
- your.domain
secretName: your.domain-tls
certManager: true
Note that resource requests and limits are just examples - they work for me, but you might want to adjust them. If you want to know how to prepare basics for Mastodon Helm Chart deployment, like the secrets I mentioned earlier, check out my previous post on how to deploy Mastodon in Kubernetes.
Prepare the HelmRepository:
Update your HelmRepository file:
kind: HelmRepository
metadata:
name: mastodon
namespace: default
spec:
type: oci
interval: 5m
url: oci://registry-1.docker.io/bitnamicharts
Prepare the HelmRelease:
Update your HelmRelease file:
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: mastodon-bitnami
namespace: default
spec:
interval: 5m
chart:
spec:
chart: mastodon
version: 3.0.5
sourceRef:
kind: HelmRepository
name: mastodon
interval: 5m
valuesFrom:
- kind: ConfigMap
name: mastodon-values
- kind: ConfigMap
name: pgsql-14-configuration
valuesKey: DB_HOST
targetPath: externalDatabase.host
- kind: ConfigMap
name: pgsql-14-configuration
valuesKey: DB_PORT
targetPath: externalDatabase.port
Note that I’m not adding values directly to the HelmRelease file - instead, I’m referring to a ConfigMap created with kustomize. Link to the Flux CD documentation is here.
Start the migration
Create a new HelmRelease with a different name, for example, mastodon-bitnami. This is important - we can’t just update the current Mastodon HelmRelease, we need to create a separate one. The InitJob mentioned before has post-install and pre-upgrade hooks. The latter means that the InitJob will be executed before any resources are updated. Consequently, any changes done to existing configmaps wouldn't be taken into account. The InitJob would continue to use outdated vars and continuously fail.
Suspend your old Mastodon HelmRelease. Scale your old Mastodon deployments to 0 replicas - we want to turn down the server but keep the current configuration and Redis setup, just in case..
Add new HelmRelease - observe the initJob. When it succeeds, the Streaming, Web, and Sidekiq pods should be up along with the Redis, Elastichsearch, and Apache. Check your ingress - cert-manager should have created the new certificate by now.
Mastodon monitoring with Prometheus and Grafana
Mastodon enables StatsD metrics. To fetch those metrics with Prometheus and to visualize them in Grafana they have to be converted first. Statsd-exporter from the Prometheus Community translates StatsD metrics to Prometheus metrics via configured mapping rules.
Install Statsd-exporter Helm Chart in your monitoring namespace.
Add HelmRepository:
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: prometheus-community
namespace: flux-system
spec:
interval: 5m0s
url: https://prometheus-community.github.io/helm-charts
Add HelmRelease:
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: prometheus-statsd-exporter
spec:
chart:
spec:
chart: prometheus-statsd-exporter
sourceRef:
kind: HelmRepository
name: prometheus-community
namespace: flux-system
version: 0.9.1
interval: 5m0s
releaseName: prometheus-statsd-exporter
valuesFrom:
- kind: ConfigMap
name: prometheus-statsd-exporter-values
valuesKey: prometheus-statsd-exporter-values.yaml
Refer to the values in ConfigMap created with Kustomize - this will keep our HelmRelease nice and clean.
To translate metrics from the Statsd format to the one that will be readable by Prometheus, we need to create mappings. I found a very useful article on how to monitor Mastodon. The author of the blog shares his mapping configuration and also a Grafana Dashboard which works perfectly.
Add mappings to the values.yaml
:
serviceMonitor:
enabled: true
namespace: monitoring
endpoint: "/metrics"
statsd:
mappingConfig: |-
# add here mappings
Don’t forget to add the name of your Prometheus Statsd exporter service to the Mastodon Helm Chart values under the STATS_ADDR
env. Add your dashboard to Grafana, and you should be able to collect statistics and monitor your instance.
Summary
By now, you should have completed a successful migration of your Mastodon instance with as little downtime as possible. While this Helm Chart may still require a few pull requests and fixes, it's reassuring to know that the Bitnami Github repository is well-maintained and receives regular updates. Enjoy a smooth experience managing your Mastodon server!
reviewed by: Paweł Maszota