> ## Content Index
> Fetch the complete content index at: https://prod-0-dol-blog-zrh1.dol.ch/llms.txt
> Use this file to discover other available public pages before exploring further.

# Installing Mastodon 4.6 on Kubernetes
- URL: https://prod-0-dol-blog-zrh1.dol.ch/en/blog/mastodon-4-6-kubernetes-en/
- Published: 2026-07-28T00:00:00.000Z
- Updated: 2026-09-10T10:08:08.000Z
- Description: A complete guide for your own instance: Postgres 18 through an operator instead of a Deployment, Valkey 9 with persistence, Sidekiq with an autoscaler, Envoy Gateway instead of the retired ingress-nginx, sealed secrets and backups with point in time recovery. Second attempt, this time with the lesso
- Author: dima
- Tags: en, Guide · Kubernetes, #pair-mastodon-4-6-kubernetes

In September 2025 I already wrote up how to install Mastodon on Kubernetes. That blog no longer exists, and the old guide survives only [in the Web Archive](https://web.archive.org/web/20251107070914/https://www.dol.ch/how-to-install-mastodon-4-4-5-on-kubernetes/). A text about backups that only exists as somebody else's copy is at least staying on topic. My own instance ran on that setup until March 2026, when I moved Mastodon off Kubernetes on cost grounds. Almost everything in the old guide held up over that time. One part was wrong all the same, and it was the most important one: Postgres as a `Deployment` with a PVC next to it.

This is the whole guide rewritten: the configuration as it last ran on that instance, with the version pins brought up to today. Finished, tested, and the reason I know where the sharp edges are. You do not need to know the old guide, everything is here from an empty namespace to a tested backup. Where something changed, I explain it at the point where the decision gets made.

Same rules as last time: no Helm, everything by hand, because I want to understand what is running. Mine ran as GitOps through ArgoCD, but every file here works just as well with `kubectl apply`.

## My setup

Three nodes with 4 GB RAM, 2 cores and 20 GB SSD each, plus a load balancer. The guide is written against Kubernetes 1.36\. The persistent volumes are network volumes from the provider's storage class, the local SSD only holds the system and images. Envoy Gateway for the traffic, cert-manager for certificates, kubeseal for secrets. No Elasticsearch, it does not pay off at this size. I explain further down why the 2025 nginx ingress is gone.

The versions this guide is written against:

| Component      | Version | Note                              |
| -------------- | ------- | --------------------------------- |
| Mastodon       | v4.6.4  | requires Postgres 14+, Redis 7.0+ |
| PostgreSQL     | 18.4    | through CloudNativePG             |
| CloudNativePG  | 1.30.0  | the operator                      |
| Valkey         | 9.1.1   | Redis fork, protocol compatible   |
| Sealed Secrets | v0.38.4 | controller and CLI                |
| cert-manager   | v1.21.0 | for Let's Encrypt                 |
| Envoy Gateway  | v1.8.3  | brings Gateway API 1.5.1 with it  |

Valkey instead of Redis is a deliberate choice. Mastodon documents Redis 7.0 and newer, and [Valkey](https://valkey.io/?ref=prod-0-dol-blog-zrh1.dol.ch) is a protocol compatible fork of it. The environment variables are still called `REDIS_*`, only the image changes.

## Tools

You need kubeseal locally for the secrets. On macOS:

```bash
brew install kubeseal

```

On Linux:

```bash
export KUBESEAL_VERSION="0.38.4"
curl -OL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz"
tar -xvzf kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

```

Verify with `kubeseal --version`. You also need Docker or Podman locally: the Mastodon keys further down are generated in a throwaway container.

## Namespace and controllers

```bash
kubectl create namespace mastodon

```

The Sealed Secrets controller, if you do not have one yet:

```bash
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.38.4/controller.yaml

```

This manifest puts the controller into `kube-system` as `sealed-secrets-controller`, which is exactly what the `kubeseal` calls further down assume. If your controller lives somewhere else, say from a Helm chart in its own namespace, adjust `--controller-namespace` and `--controller-name` there.

cert-manager, if it is not running yet:

```bash
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.21.0/cert-manager.yaml

kubectl -n cert-manager rollout status deployment/cert-manager-webhook

```

And [CloudNativePG](https://cloudnative-pg.io/?ref=prod-0-dol-blog-zrh1.dol.ch), the Postgres operator. The `--server-side` flag is not optional here, the CRDs are too large for the classic annotation:

```bash
kubectl apply --server-side -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-1.30.0.yaml

kubectl -n cnpg-system rollout status deployment/cnpg-controller-manager

```

## The database

### What was wrong with the Deployment

This is the part I got wrong in 2025\. Back then it was a `Deployment` with `replicas: 1`, a PVC, and a `pg_dump` CronJob at four in the morning. That runs fine for months and has three problems, all of which surface at the least convenient time.

A `Deployment` rolls out with `RollingUpdate` by default, starting the new pod before the old one is gone. On a ReadWriteOnce volume the rollout then sits there until somebody intervenes. Kubernetes also sends `SIGTERM` on shutdown, which Postgres reads as a smart shutdown: wait until all clients disconnect on their own. Sidekiq does not disconnect, so after 30 seconds comes `SIGKILL` and on the next start a crash recovery. And a nightly dump means a mistake at 14:00 costs you half a day of federation.

### The CNPG cluster

The operator takes all of that off your hands. First the credentials for the backup store:

```bash
kubectl create secret generic mastodon-backup-creds \
  --namespace mastodon \
  --from-literal=ACCESS_KEY_ID=your_key \
  --from-literal=ACCESS_SECRET_KEY=your_secret \
  --dry-run=client -o yaml > backup-creds.yaml

kubeseal --controller-namespace kube-system \
  --controller-name sealed-secrets-controller \
  --format yaml < backup-creds.yaml > sealed-backup-creds.yaml

kubectl apply -f sealed-backup-creds.yaml && rm backup-creds.yaml

```

Then the cluster:

```yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: mastodon-db
  namespace: mastodon
spec:
  instances: 2
  imageName: ghcr.io/cloudnative-pg/postgresql:18.4

  # On three small nodes, "preferred" is the difference between
  # "running" and "Pending, because the rule cannot be satisfied"
  affinity:
    enablePodAntiAffinity: true
    topologyKey: kubernetes.io/hostname
    podAntiAffinityType: preferred

  storage:
    size: 20Gi
  walStorage:
    size: 10Gi

  postgresql:
    parameters:
      shared_buffers: "256MB"
      effective_cache_size: "768MB"
      work_mem: "8MB"
      maintenance_work_mem: "128MB"

  bootstrap:
    initdb:
      database: mastodon_production
      owner: mastodon

  backup:
    barmanObjectStore:
      destinationPath: s3://your-bucket/mastodon
      endpointURL: https://your-s3-endpoint
      s3Credentials:
        accessKeyId:
          name: mastodon-backup-creds
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: mastodon-backup-creds
          key: ACCESS_SECRET_KEY
      wal:
        compression: gzip
      data:
        compression: gzip
    retentionPolicy: "30d"

  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      memory: "1Gi"

```

Four points in there need explaining.

`walStorage` as its own volume separates WAL from data, and the difference shows when one of the two fills up. Data volume full: writes fail, but Postgres keeps running, the WAL archive stays intact, and you grow the volume at your leisure. WAL volume full, classically because archiving to object storage is stuck: Postgres stops hard with a PANIC, but the data next to it stays untouched. On a shared volume, both cases end in the hard variant.

The `backup` block does continuous WAL archiving: instead of one state per night, you can restore to any point in time since the last base backup. A note on versions, because this is moving right now: this built in `barmanObjectStore` form has been deprecated since CNPG 1.26, and the [1.30 release notes](https://cloudnative-pg.io/releases/cloudnative-pg-1-30.0-released/?ref=prod-0-dol-blog-zrh1.dol.ch) say it will be removed in 1.31\. On 1.30 it still works and is still the default. If you are installing anything newer, start with the [Barman Cloud plugin](https://cloudnative-pg.io/plugin-barman-cloud/?ref=prod-0-dol-blog-zrh1.dol.ch) right away instead of migrating twice. And for this setup it means: plan the move to the plugin before you lift the operator to 1.31.

You do not have to copy `instances: 2`. On three nodes with 4 GB each, the second instance costs memory that Sidekiq would happily use, and it tightens scheduling: with a node down or in maintenance, both Postgres pods compete with everything else for the remaining two. With `instances: 1` you still get backups, point in time recovery, controlled updates and declarative configuration, just no automatic failover when a node dies. For a personal instance that is a legitimate trade, and I see no shame in running it that way.

The fourth point only shows if you go looking for it: there is no `CREATE EXTENSION` anywhere. Mastodon 4.6 needs nothing beyond `plpgsql`, which every Postgres database has out of the box. Do not take my word for it, check the `enable_extension` lines in the [db/schema.rb](https://github.com/mastodon/mastodon/blob/v4.6.4/db/schema.rb?ref=prod-0-dol-blog-zrh1.dol.ch) of your version, that is where I look before setting up. If an extension ever shows up there, `postInitApplicationSQL` under `initdb` is the place to create it, and since Postgres 13 extensions marked trusted can also be created by the database owner during the migration, no superuser needed.

### Automatic backups

The scheduled base backup that goes with it:

```yaml
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: mastodon-db-daily
  namespace: mastodon
spec:
  # Six fields, not five. The first one is seconds.
  schedule: "0 0 3 * * *"
  backupOwnerReference: self
  cluster:
    name: mastodon-db

```

The cron expression caught me on the first attempt. CNPG uses a Go library with a seconds field, so a five field expression copied from a normal `CronJob` means something other than you intended.

### Services and credentials

The operator now creates three services: `mastodon-db-rw` always points at the current primary, `mastodon-db-ro` at the replicas, `mastodon-db-r` at all of them. That is exactly what a handwritten Service with a fixed selector cannot do, because after a failover it would point at the wrong pod.

The operator generates the application user's password itself and stores it in the `mastodon-db-app` secret. It never gets copied anywhere: the deployments further down read it straight from there via `valueFrom`. That leaves exactly one source of truth, and if the operator ever rotates the password, nothing quietly keeps running with a stale copy. In case you still want to connect with `psql` by hand:

```bash
kubectl -n mastodon get secret mastodon-db-app \
  -o jsonpath='{.data.password}' | base64 -d

```

## Valkey

For Mastodon, Valkey holds more than the cache: the Sidekiq queues live there too. Lose those and you lose in-flight jobs, meaning outgoing deliveries, push notifications and media processing. So it gets a `StatefulSet` with its own volume, not a `Deployment` with a PVC attached like I had in 2025.

```yaml
apiVersion: v1
kind: Service
metadata:
  name: valkey
  namespace: mastodon
spec:
  clusterIP: None
  selector:
    app: valkey
  ports:
    - port: 6379
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: valkey
  namespace: mastodon
spec:
  serviceName: valkey
  replicas: 1
  selector:
    matchLabels:
      app: valkey
  template:
    metadata:
      labels:
        app: valkey
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: valkey
          image: valkey/valkey:9.1-alpine
          args: ["--appendonly", "yes", "--appendfsync", "everysec"]
          ports:
            - containerPort: 6379
          volumeMounts:
            - name: data
              mountPath: /data
          readinessProbe:
            exec:
              command: ["valkey-cli", "ping"]
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              cpu: "100m"
              memory: "256Mi"
            limits:
              memory: "512Mi"
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 5Gi

```

The grace period is set to 60 seconds because Valkey needs time to write its data out on shutdown, and the 30 second default is tight for that.

## Secrets

These values get generated once and never changed. `SECRET_KEY_BASE` encrypts sessions, `OTP_SECRET` is tied to two factor authentication, the VAPID keys to push notifications. Swap them later and you throw away every login, every 2FA enrolment and every push subscription. For `LOCAL_DOMAIN` this holds even more strictly: the domain is baked into the identity of every account, it is the name every other server knows you by. Change it later and the federation for the old identity is gone for good, and nothing can repair that.

```bash
# Active Record encryption keys
docker run --rm ghcr.io/mastodon/mastodon:v4.6.4 \
  bin/rails db:encryption:init

# SECRET_KEY_BASE and OTP_SECRET, run twice
docker run --rm ghcr.io/mastodon/mastodon:v4.6.4 bundle exec rake secret

# VAPID key pair
docker run --rm ghcr.io/mastodon/mastodon:v4.6.4 \
  bundle exec rake mastodon:webpush:generate_vapid_key

```

Those go into the secret. `DB_HOST` points at the operator's `-rw` service, and there is deliberately no `DB_PASS`: that comes straight from `mastodon-db-app` via `valueFrom` in a moment:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: mastodon-env
  namespace: mastodon
type: Opaque
stringData:
  LOCAL_DOMAIN: "your-domain.ch"
  SINGLE_USER_MODE: "true"
  RAILS_ENV: "production"

  SECRET_KEY_BASE: "..."
  OTP_SECRET: "..."
  VAPID_PRIVATE_KEY: "..."
  VAPID_PUBLIC_KEY: "..."
  ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY: "..."
  ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT: "..."
  ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY: "..."

  DB_HOST: "mastodon-db-rw"
  DB_NAME: "mastodon_production"
  DB_USER: "mastodon"

  REDIS_HOST: "valkey"
  REDIS_PORT: "6379"
  CACHE_REDIS_URL: "redis://valkey:6379/1"

  SMTP_SERVER: "smtp.example.ch"
  SMTP_PORT: "587"
  SMTP_LOGIN: "mail@your-domain.ch"
  SMTP_PASSWORD: "..."
  SMTP_FROM_ADDRESS: "mail@your-domain.ch"

  S3_ENABLED: "true"
  S3_BUCKET: "your-bucket"
  S3_REGION: "..."
  S3_ENDPOINT: "https://your-s3-endpoint"
  S3_FORCE_PATH_STYLE: "true"
  S3_ALIAS_HOST: "cdn.your-domain.ch"
  AWS_ACCESS_KEY_ID: "..."
  AWS_SECRET_ACCESS_KEY: "..."

```

`CACHE_REDIS_URL` moves the Rails cache into database 1\. That way you can flush the cache in an emergency without touching the Sidekiq queues in database 0\. One limit of this double duty is worth knowing: the eviction policy applies per instance, not per database. The queues only tolerate `noeviction`, the default, so the cache here must never be capped with `allkeys-lru` either. As long as memory holds out this does not matter. Once the instance outgrows it, the cache moves into its own small Valkey.

Seal it and delete the plaintext file:

```bash
kubeseal --controller-namespace kube-system \
  --controller-name sealed-secrets-controller \
  --format yaml < mastodon-env.yaml > sealed-mastodon-env.yaml

kubectl apply -f sealed-mastodon-env.yaml
rm mastodon-env.yaml

```

## Migrations

The schema has to exist before the first start. As a Job, not an init container, so it does not run again on every pod restart:

```yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: mastodon-db-migrate
  namespace: mastodon
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: migrate
          image: ghcr.io/mastodon/mastodon:v4.6.4
          command: ["bundle", "exec", "rails", "db:migrate"]
          envFrom:
            - secretRef:
                name: mastodon-env
          env:
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  name: mastodon-db-app
                  key: password

```

This is the first appearance of the pattern for the database password: `envFrom` pulls everything from `mastodon-env`, and the single `env` entry takes `DB_PASS` straight from the operator secret. The same block appears from now on in every manifest that talks to the database. If your `mastodon-env` still carries an old `DB_PASS`: individual `env` entries beat `envFrom`, the direct reference wins.

On the first run the job replays Mastodon's entire migration history, a few hundred migrations, so give it a few minutes.

You need this same job after every upgrade, and since Jobs are immutable, delete the old one first: `kubectl -n mastodon delete job mastodon-db-migrate`, then apply again. Check the release notes too. Bigger upgrades split their migrations: the part before the rollout runs with `SKIP_POST_DEPLOYMENT_MIGRATIONS=true`, the rest after it.

## Web, streaming and Sidekiq

### Web

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mastodon-web
  namespace: mastodon
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mastodon-web
  template:
    metadata:
      labels:
        app: mastodon-web
    spec:
      containers:
        - name: web
          image: ghcr.io/mastodon/mastodon:v4.6.4
          command: ["bundle", "exec", "puma", "-C", "config/puma.rb"]
          envFrom:
            - secretRef:
                name: mastodon-env
          env:
            - name: WEB_CONCURRENCY
              value: "1"
            - name: MAX_THREADS
              value: "3"
            - name: DB_POOL
              value: "5"
            - name: MALLOC_ARENA_MAX
              value: "2"
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  name: mastodon-db-app
                  key: password
          ports:
            - containerPort: 3000
          readinessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 20
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 60
            periodSeconds: 30
            timeoutSeconds: 10
            failureThreshold: 5
          resources:
            requests:
              cpu: "300m"
              memory: "600Mi"
            limits:
              memory: "1.2Gi"

```

The PVC for `/mastodon/public/system` from my old guide is gone here. With `S3_ENABLED: true` the media live in object storage, the local directory is not needed, and that also removes the init container that used to fix up permissions.

The readiness probe was missing entirely in 2025\. Without it the gateway sends requests to a pod where Puma is still starting. There is now also a liveness probe, deliberately sluggish: a Puma that is genuinely hung is still hung after five failed checks 30 seconds apart, and one that is merely busy gets enough slack before anyone restarts it for no reason.

The resources follow a pattern that runs through every manifest in this guide, the Postgres cluster included: limits for memory, none for CPU. Memory is not compressible, and without a limit a growing process eventually takes the node down with it. CPU is different: when it gets scarce, the scheduler divides it up according to the requests, and a CPU limit would throttle Puma even while the node has nothing else to do.

### Streaming

Streaming is a separate process with its own image:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mastodon-streaming
  namespace: mastodon
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mastodon-streaming
  template:
    metadata:
      labels:
        app: mastodon-streaming
    spec:
      containers:
        - name: streaming
          image: ghcr.io/mastodon/mastodon-streaming:v4.6.4
          envFrom:
            - secretRef:
                name: mastodon-env
          env:
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  name: mastodon-db-app
                  key: password
          ports:
            - containerPort: 4000
          readinessProbe:
            httpGet:
              path: /api/v1/streaming/health
              port: 4000
            initialDelaySeconds: 10
          livenessProbe:
            httpGet:
              path: /api/v1/streaming/health
              port: 4000
            initialDelaySeconds: 30
            periodSeconds: 30
            timeoutSeconds: 10
            failureThreshold: 5
          resources:
            requests:
              cpu: "100m"
              memory: "200Mi"
            limits:
              memory: "400Mi"

```

### Sidekiq and the scheduler

Sidekiq does the federation work. Two values matter on shutdown here: Kubernetes waits up to `terminationGracePeriodSeconds` after `SIGTERM`, but Sidekiq has its own timeout and by default pushes unfinished jobs back onto the queue after 25 seconds. Only the `-t 110` below makes use of the full 120 seconds. Nothing is lost without the flag, the jobs simply run again later, they just start over from the beginning:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mastodon-sidekiq
  namespace: mastodon
spec:
  selector:
    matchLabels:
      app: mastodon-sidekiq
  template:
    metadata:
      labels:
        app: mastodon-sidekiq
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: sidekiq
          image: ghcr.io/mastodon/mastodon:v4.6.4
          command:
            - bundle
            - exec
            - sidekiq
            - -c
            - "5"
            - -t
            - "110"
            - -q
            - default,8
            - -q
            - push,6
            - -q
            - ingress,4
            - -q
            - mailers,2
            - -q
            - pull,1
          envFrom:
            - secretRef:
                name: mastodon-env
          env:
            - name: DB_POOL
              value: "5"
            - name: MALLOC_ARENA_MAX
              value: "2"
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  name: mastodon-db-app
                  key: password
          resources:
            requests:
              cpu: "500m"
              memory: "300Mi"
            limits:
              memory: "600Mi"

```

One queue is deliberately missing from that list. [Mastodon's documentation](https://docs.joinmastodon.org/admin/scaling/?ref=prod-0-dol-blog-zrh1.dol.ch) is blunt about `scheduler`: it must never run in more than one process at a time. The deployment above is about to get an autoscaler, so the scheduler gets its own deployment with exactly one replica and no autoscaling:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mastodon-sidekiq-scheduler
  namespace: mastodon
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mastodon-sidekiq-scheduler
  template:
    metadata:
      labels:
        app: mastodon-sidekiq-scheduler
    spec:
      containers:
        - name: sidekiq
          image: ghcr.io/mastodon/mastodon:v4.6.4
          command: ["bundle", "exec", "sidekiq", "-c", "1", "-q", "scheduler"]
          envFrom:
            - secretRef:
                name: mastodon-env
          env:
            - name: DB_POOL
              value: "2"
            - name: MALLOC_ARENA_MAX
              value: "2"
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  name: mastodon-db-app
                  key: password
          resources:
            requests:
              cpu: "50m"
              memory: "300Mi"
            limits:
              memory: "512Mi"

```

The scheduler triggers the periodic work: publishing scheduled posts, refreshing trends, cleaning up old data. Run it twice and all of that happens twice, including your scheduled posts going out twice.

### Autoscaling and connections

Plus the autoscaler for the main worker, so that under load you get several small pods instead of one large one:

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: mastodon-sidekiq
  namespace: mastodon
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: mastodon-sidekiq
  minReplicas: 1
  maxReplicas: 3
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80

```

This is why the Sidekiq deployment above has no `replicas` field. Once the HPA manages the count, every repeated `kubectl apply` and every ArgoCD sync would write the value from the file back and flatten the scaling again. Without the field the deployment starts with one replica, and after that the HPA has the last word.

Scaling on CPU has a catch: during federation Sidekiq spends most of its time waiting on other people's servers, so it is often not CPU bound. Queue length would be the better signal, but that costs you a custom metrics adapter. For a small instance CPU is good enough, you should just know it is an approximation.

One more thing about `DB_POOL`: the value applies per process. One web pod with five, up to three Sidekiq pods with five each, the scheduler with its two, plus streaming with its default pool of ten and the occasional migration job: around forty connections, which the Postgres default of 100 covers with room to spare. If you ever run significantly more processes, do not just raise `max_connections`, every connection costs memory in a database container that has little to spare, put [PgBouncer](https://www.pgbouncer.org/?ref=prod-0-dol-blog-zrh1.dol.ch) in front instead. That adds one easily missed line: in transaction pooling mode `PREPARED_STATEMENTS` must be `false`, otherwise Rails' prepared statements collide with shared connections.

## Services and the gateway

### Why Gateway API

```yaml
apiVersion: v1
kind: Service
metadata:
  name: mastodon-web
  namespace: mastodon
spec:
  selector:
    app: mastodon-web
  ports:
    - port: 80
      targetPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: mastodon-streaming
  namespace: mastodon
spec:
  selector:
    app: mastodon-streaming
  ports:
    - port: 4000
      targetPort: 4000

```

In 2025 this section was an nginx `Ingress`, and I would have happily kept it. But the Kubernetes project [announced the retirement of ingress-nginx](https://www.kubernetes.dev/blog/2025/11/12/ingress-nginx-retirement/?ref=prod-0-dol-blog-zrh1.dol.ch) in November 2025 and stopped maintenance in March 2026\. Existing installations keep running, only no patches are coming anymore, including for the next CVE. I am not willing to run the component that parses every byte from the open internet on those terms.

The question of what replaces it has an official answer: [Gateway API](https://gateway-api.sigs.k8s.io/?ref=prod-0-dol-blog-zrh1.dol.ch) instead of Ingress. As the implementation I picked [Envoy Gateway](https://gateway.envoyproxy.io/?ref=prod-0-dol-blog-zrh1.dol.ch). CNCF project, Envoy underneath, and it installs like everything else in this guide, one manifest, no Helm:

```bash
kubectl apply --server-side -f \
  https://github.com/envoyproxy/gateway/releases/download/v1.8.3/install.yaml

kubectl -n envoy-gateway-system rollout status deployment/envoy-gateway

```

`--server-side` for the same reason as with CNPG, and the manifest already brings the Gateway API CRDs (v1.5.1) with it, you do not install those separately.

### Adjusting cert-manager

A word on compatibility, since Kubernetes 1.36 is what I put at the top: Envoy Gateway v1.8 is officially tested against 1.32 through 1.35, and 1.36 was not in the [matrix](https://gateway.envoyproxy.io/news/releases/matrix/?ref=prod-0-dol-blog-zrh1.dol.ch) yet when I wrote this. It does not depend on any API that changed in 1.36, and the next release should close the gap. If you want to stay strictly inside the tested matrix, stay on Kubernetes 1.35, which is [supported until February 2027](https://kubernetes.io/releases/?ref=prod-0-dol-blog-zrh1.dol.ch) and behaves exactly the same in this guide.

cert-manager can answer HTTP-01 challenges through a Gateway, it just does not look for Gateways by default. The switch is called `--enable-gateway-api`:

```bash
kubectl -n cert-manager patch deployment cert-manager --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--enable-gateway-api"}]'

```

The patch restarts the controller, which it needs anyway to see the new CRDs. Since cert-manager above comes from the static manifest, the flag is gone again on the next re-apply, so it belongs in your copy in Git.

The ClusterIssuer changes too. The old one solved HTTP-01 through an Ingress, this one builds a temporary HTTPRoute per challenge:

```yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: mail@your-domain.ch
    privateKeySecretRef:
      name: letsencrypt-prod-account
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: mastodon
                namespace: mastodon
                kind: Gateway

```

### The gateway and the routes

Then the gateway itself:

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: mastodon
  namespace: mastodon
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  gatewayClassName: envoy
  listeners:
    - name: http
      protocol: HTTP
      port: 80
    - name: https
      protocol: HTTPS
      port: 443
      hostname: your-domain.ch
      tls:
        mode: Terminate
        certificateRefs:
          - name: your-domain-tls

```

The annotation does what it used to do on the Ingress: cert-manager sees the Gateway, issues the certificate for the listener's hostname and writes it into the secret the listener references.

Two routes attach to this. The first one only redirects, from the http listener to https:

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mastodon-redirect
  namespace: mastodon
spec:
  parentRefs:
    - name: mastodon
      sectionName: http
  rules:
    - filters:
        - type: RequestRedirect
          requestRedirect:
            scheme: https
            statusCode: 301

```

The second one does the actual routing:

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mastodon
  namespace: mastodon
spec:
  parentRefs:
    - name: mastodon
      sectionName: https
  hostnames:
    - your-domain.ch
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api/v1/streaming
      backendRefs:
        - name: mastodon-streaming
          port: 4000
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: mastodon-web
          port: 80

```

A few details, for once all of them pleasant. The order of the rules looks meaningful and is not: Gateway API specifies that the longest matching path wins, however you sort them. With nginx that was true as well, but only as an implementation detail. Here it is contract. If you are wondering whether the redirect route swallows Let's Encrypt's challenges: it does not. cert-manager's challenge route matches a longer path, and longer wins, also by spec. `proxy-body-size` has no successor here, on purpose: Envoy streams request bodies instead of buffering them, so there is no default limit to raise and no 413 in the middle of a video upload. And websockets for streaming work without any extra configuration.

### LoadBalancer and DNS

Envoy Gateway creates its own LoadBalancer service for the gateway in `envoy-gateway-system`. That only works if your cluster can give LoadBalancer services an external IP: on managed providers the load balancer from my setup at the top takes care of it, on bare metal you need something like [MetalLB](https://metallb.io/?ref=prod-0-dol-blog-zrh1.dol.ch), otherwise the address simply stays empty. Your DNS record points there from now on:

```bash
kubectl -n mastodon get gateway mastodon

```

The ADDRESS column is what goes into DNS, and PROGRAMMED should say True. If you are coming from the ingress setup: build the gateway and routes alongside it, switch DNS, and only remove ingress-nginx once no traffic shows up in its logs anymore.

## The first account

With `SINGLE_USER_MODE` the registration form stays closed, and as long as no account exists the front page has nothing to point at. So create the owner account directly in the web pod:

```bash
kubectl -n mastodon exec deploy/mastodon-web -- \
  bin/tootctl accounts create dima \
  --email mail@your-domain.ch --confirmed --role Owner

```

The command prints a temporary password. Log in with it, change it, set up 2FA, done.

## Cleaning up media

Without cleanup, object storage grows without bound, because every federated file gets cached.

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: mastodon-media-cleanup
  namespace: mastodon
spec:
  schedule: "0 4 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: cleanup
              image: ghcr.io/mastodon/mastodon:v4.6.4
              envFrom:
                - secretRef:
                    name: mastodon-env
              env:
                - name: DB_PASS
                  valueFrom:
                    secretKeyRef:
                      name: mastodon-db-app
                      key: password
              command:
                - /bin/bash
                - -c
                - |
                  bin/tootctl media remove --days=7
                  bin/tootctl media remove-orphans
                  bin/tootctl preview_cards remove --days=7
                  bin/tootctl statuses remove --days=14
                  bin/tootctl accounts prune

```

Four in the morning is deliberate, an hour after the base backup, so the two do not overlap. `concurrencyPolicy: Forbid` was missing in my old version. If a run takes longer than 24 hours, the next one otherwise starts alongside it, and two parallel cleanup jobs against the same database are not a good idea.

## The backup that stopped hurting

In 2025 this was the longest and most unpleasant section of the whole guide. I had built my own Docker image: Ubuntu with AWS CLI, rclone and the Postgres client, plus a bash script with error checks after every step and a registry secret so the cluster could pull the image. It worked, and it was a lot of moving parts for a job that is really standard.

The `ScheduledBackup` above replaces all of it. The database goes to object storage continuously through WAL archiving, the base backup runs at night, and `retentionPolicy` handles how long things are kept.

The media no longer need their own backup if they already live in S3\. What they need is versioning or replication on the provider side, and that is a bucket setting rather than a nightly `rclone sync`.

You can check the state like this:

```bash
kubectl -n mastodon get backups
kubectl -n mastodon get cluster mastodon-db \
  -o jsonpath='{.status.firstRecoverabilityPoint}'

```

And the part I did not have before: restoring to a point in time. It creates a new cluster while the old one keeps running untouched, so you can compare and then switch over:

```yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: mastodon-db-restore
  namespace: mastodon
spec:
  instances: 1
  imageName: ghcr.io/cloudnative-pg/postgresql:18.4
  storage:
    size: 20Gi
  walStorage:
    size: 10Gi
  bootstrap:
    recovery:
      source: mastodon-db
      recoveryTarget:
        targetTime: "2026-07-27 14:00:00+02"
  externalClusters:
    - name: mastodon-db
      barmanObjectStore:
        destinationPath: s3://your-bucket/mastodon
        endpointURL: https://your-s3-endpoint
        s3Credentials:
          accessKeyId:
            name: mastodon-backup-creds
            key: ACCESS_KEY_ID
          secretAccessKey:
            name: mastodon-backup-creds
            key: ACCESS_SECRET_KEY

```

The restored cluster has no `backup` block, and that is on purpose: it is a rehearsal, so it has no business writing to object storage. If you ever promote one of these to the real thing, give it its own `destinationPath` or `serverName` before you add backups back. Point it at the old path and it archives its WAL straight over the history of the cluster you were trying to rescue.

Please do this once

Walk through this restore before you need it. A backup whose restore you have never rehearsed is a guess with a file size.

## If you are coming from the old guide

If Postgres is still running as a `Deployment` for you: CNPG can import from an existing database as part of bootstrapping, instead of you juggling `pg_dump` and `pg_restore`.

```yaml
spec:
  bootstrap:
    initdb:
      database: mastodon_production
      owner: mastodon
      import:
        type: microservice
        databases:
          - mastodon_production
        source:
          externalCluster: old-instance
  externalClusters:
    - name: old-instance
      connectionParameters:
        host: postgres
        user: mastodon
        dbname: mastodon_production
      password:
        name: mastodon-env
        key: DB_PASS

```

One quirk you need to know: the import runs exactly once, when the cluster is created. The only way to get a second pass is to delete the new cluster and create it again. So the sequence looks like this: create the cluster with `import` and do not touch Mastodon, that is the dress rehearsal. Compare row counts on the large tables against the old database, then delete the new cluster again. For the real move, scale web, streaming and Sidekiq to zero, downtime starts here, and create the cluster with `import` once more, this time without a gap. Point `DB_HOST` at `mastodon-db-rw` in the secret, scale back up, watch the logs. While you are at it, delete the old `DB_PASS` from the secret, the new manifests read the password straight from `mastodon-db-app`.

Leave the old deployment and its PVC in place for a week afterwards, even if it annoys you. Delete the way back only once you are sure you will not need it.

## What changed since 2025

| Then                         | Now                           | Why                                                 |
| ---------------------------- | ----------------------------- | --------------------------------------------------- |
| Postgres as a Deployment     | CloudNativePG                 | No clean rollout, no failover, no PITR              |
| pg\_dump via CronJob         | WAL archiving                 | RPO from 24 hours down to minutes                   |
| Custom backup image          | ScheduledBackup               | One image, one script, one registry secret fewer    |
| Redis as a Deployment        | Valkey as a StatefulSet       | Sidekiq queues survive restarts                     |
| Scheduler in the worker pool | Own single-replica Deployment | Periodic jobs must not run twice                    |
| No probes                    | Readiness and liveness        | No traffic to starting pods, restarts for hung ones |
| tootsuite/mastodon           | ghcr.io/mastodon/...          | Official registry, consistent tags                  |
| PVC for media                | S3 only                       | One volume and one init container fewer             |
| nginx Ingress                | Envoy Gateway + Gateway API   | ingress-nginx was retired in March 2026             |

What I did not change: Sealed Secrets, skipping Helm, no Elasticsearch on small nodes, the Sidekiq autoscaler, and the nightly media cleanup. That was already right.

And one warning that belongs here for honesty: two Postgres instances protect you from a dead node, not from a `DELETE` without a `WHERE`. That replicates cleanly to both. High availability and backups solve two different problems, and only one of them can be fixed at three in the morning with a coffee.

If you rebuild this and get stuck somewhere, write to me. Last time the best corrections came from people who simply tried what I had written down.