Mastodon and ActivityPub

I’ve started using Mastodon. It’s an open alternative to Twitter. “Open” meaning there’s no central controlling entity. Tech-savvy people can run their own server and servers can talk to each other (referred to as federation). If you’re interested in joining you can find a server to join, sign up, and follow me: @mark@kingant.net.

Why???

Part of me has always disliked posting on Facebook and Twitter. They’re walled gardens. I’d rather self host. I like keeping a record of my posts. Twitter allows downloading an archive of your posts and Facebook might, too, but I’d rather control the system of record myself. Also I don’t love being used by a company so they can make money from ads. Wise man once say, “when you’re not paying for the product you are the product.”

And then Elon Musk being a jerk¹ drove me to make a change.

The open/federated nature is appealing to me. It’s similar to how email works. And XMPP.

ActivityPub

ActivityPub is the important thing. There are other ActivityPub services that aren’t Mastodon. For example Pixelfed (an Instagram clone) and PeerTube (which is tailored for video sharing). It’s possible for them to be written such that they can interact with each other. For example on Mastodon I can follow a Pixelfed user. It’s great.

And actually I think it’s weird that these services are treated as separate things. Mastodon is just an ActivityPub service that looks like Twitter and Pixelfed is just an ActivityPub service that looks like Instagram. I’m not even sure the functionality is very different. It makes me wonder if they could be written as different UIs atop a common server. But whatever, I’m not super familiar with it. Maybe there are good reasons for them to be separate.

Running My Own Server

And because I’m a huge nerd I’m running my own server. I did it partially as a learning exercise. But also now my ID can be my email address (“@mark@kingant.net“) instead of something like “@markdoliner@someone-elses-server.example.com”

It took some work. The developers haven’t prioritized ease of installation (which is perfectly reasonable). It runs as two processes: The web/API service and a background job queue worker. It also requires PostgreSQL and Redis. As mentioned previously, I use Salt to manage my cloud servers. Initially I tried running everything directly on an EC2 instance, but Mastodon is written in Ruby and installing and managing Ruby and building Mastodon via Salt rules is a lot of work.

There are official Docker images so I looked into that instead. The docker-compose.yml file was immensely helpful. At first I tried running containers in Amazon Elastic Container Service (ECS). But configuring it is way more elaborate than you would imagine. And I’m not sure but it might have required paying for separate EC2 instances per container, though maybe it’s possible to avoid that if you go a step further and use Amazon Elastic Kubernetes Service (EKS). But of course that’s even more effort to configure.

What I did instead is run the containers myself on a single EC2 instance. Three containers, specifically: web/API, background job queue worker, and Redis. I’m running PostgreSQL directly on the host because it’s super easy. Obviously I could have used RDS for PostgreSQL and that’s certainly a nice managed option, but it’s also more expensive. I wanted to run Redis on the host but it’s hard to configure it to allow access from the Mastodon containers but disallow access from outside networks. Though even with PostgreSQL I ended up configuring it to accept connections from the world (disallowed via AWS security group of course, and PostgreSQL is configured such that users can only authenticate from localhost or the Docker network). So this feels like a decent compromise. The maintenance overhead is manageable and it’s fairly cheap. I’m paying around $3 to $4 per month ($83 up front for a t4g.micro 3 year reservation plus $1 to $2 per month for EBS storage).

I don’t want to make my Salt config repo public but here are the states for the Docker containers in case it helps anyone:

include:
  - mastodon/docker
  - mastodon/user

# A Docker network for containers to access the world.
mastodon_external:
  docker_network.present:
    - require:
      - sls: mastodon/docker

# A Docker network for intra-container communication.
mastodon_internal:
  docker_network.present:
    - internal: True
    - require:
      - sls: mastodon/docker

# Create directory for Mastodon. We leave ownership as the default
# (root) because the mastodon user shouldn't need to create files here
# and using root is a little more restrictive. It prevents the Mastodon
# processes (running in Docker as the mastodon user) from causing
# mischief (e.g. modifying the static web files, for example).
/srv/mastodon.kingant.net:
  file.directory: []

# Create private directory for Mastodon Redis data.
/srv/mastodon.kingant.net/redis:
  file.directory:
    - user: mastodon
    - group: mastodon
    - mode: 700
    - require:
      - sls: mastodon/user

# Redis Docker container.
# We're running it as a Docker container rather than on the host mostly
# because we would need to change the host Redis's config to bind to the
# Docker network IP (by default it only binds to localhost) and that
# requires modifying the config file from the package (there is no
# conf.d directory). That means we would stop getting automatic config
# file updates on new package versions, which is unfortunate.
# Of course if we ever want to change any other config setting then
# we'll have the same problem with the Docker contain. Though it's maybe
# still slightly cleaner having Redis accessible only over the
# mastodon_internal network, because Redis isn't big on data isolation.
# The options are:
# 1. Use "namespaces." Meaning "prefix your keys with a namespace string
#    of your choice, maybe with a trailing  colon."
# 2. Use a single server but have different apps use different DBs
#    within Redis. This is a thing... but seems problematic because
#    the DBs are numbered sequentially so what happens if you remove one
#    in the middle?
# 3. Use separate servers. This probably makes the most sense (and
#    provides the strongest isolation).
redis:
  docker_container.running:
    - image: redis:7-alpine
    - binds:
      - /srv/mastodon.kingant.net/redis:/data:rw
    # Having a healthcheck isn't important for Redis but it existed in
    # Mastodon's example docker-compose file so I included it here. The
    # times are in nanoseconds (currently 5 seconds).
    - healthcheck: {test: ["CMD", "redis-cli", "ping"], retries=10}
    - networks:
      - mastodon_internal
    - restart_policy: always
    # The UID and GID are hardcoded here and in user.present in
    # mastodon/user.sls because it's really hard to look them up. See
    # https://github.com/saltstack/salt/issues/63287#issuecomment-1377500491
    - user: 991:991
    - require:
      - sls: mastodon/docker
      - sls: mastodon/user
      - docker_network: mastodon_internal
      - file: /srv/mastodon.kingant.net/redis

# Create directory for Mastodon config file.
/srv/mastodon.kingant.net/config:
  file.directory: []

# Install the Mastodon config file. It's just a bunch of environment
# variables that get mounted in Docker containers and then sourced.
# Note: Secrets must be added to this file by hand. See the "solution
# for storing secrets" comment in TODO.md.
/srv/mastodon.kingant.net/config/env_vars:
  file.managed:
    - source: salt://mastodon/files/srv/mastodon.kingant.net/config/env_vars
    - user: mastodon
    - group: mastodon
    - mode: 600
    # Do not modify the file if it already exists. This allows Salt to
    # create the initial version of the file while being careful not to
    # overwrite it once the secrets have been added.
    - replace: False
    - require:
      - file: /srv/mastodon.kingant.net/config

# Docker container for Mastodon web.
mastodon-web:
  docker_container.running:
    - image: ghcr.io/mastodon/mastodon:v4.1
    - command: bash -c "set -o allexport && source /etc/opt/mastodon/env_vars && set +o allexport && rm -f /mastodon/tmp/pids/server.pid && bundle exec rails s -p 3000"
    - binds:
      - /srv/mastodon.kingant.net/config:/etc/opt/mastodon:ro
      - /srv/mastodon.kingant.net/www/system:/opt/mastodon/public/system:rw
    - extra_hosts:
      - host.docker.internal:host-gateway
    - healthcheck: {test: ["CMD-SHELL", "wget -q --spider --proxy=off http://localhost:3000/health || exit 1"], retries=10}
    - networks:
      - mastodon_external
      - mastodon_internal
    - port_bindings:
      - 3000:3000
    - restart_policy: always
    - skip_translate: extra_hosts # Because Salt was complaining that "host-gateway" wasn't a valid IP.
    - user: 991:991
    - require:
      - sls: mastodon/docker
      - sls: mastodon/user
      - docker_network: mastodon_external
      - docker_network: mastodon_internal
      - file: /srv/mastodon.kingant.net/config/env_vars
    - watch:
      - file: /srv/mastodon.kingant.net/config/env_vars

# Docker container for Mastodon streaming.
mastodon-streaming:
  docker_container.running:
    - image: ghcr.io/mastodon/mastodon:v4.1
    - command: bash -c "set -o allexport && source /etc/opt/mastodon/env_vars && set +o allexport && node ./streaming"
    - binds:
      - /srv/mastodon.kingant.net/config:/etc/opt/mastodon:ro
    - extra_hosts:
      - host.docker.internal:host-gateway
    - healthcheck: {test: ["CMD-SHELL", "wget -q --spider --proxy=off http://localhost:4000/api/v1/streaming/health || exit 1"], retries=10}
    - networks:
      - mastodon_external
      - mastodon_internal
    - port_bindings:
      - 4000:4000
    - restart_policy: always
    - skip_translate: extra_hosts # Because Salt was complaining that "host-gateway" wasn't a valid IP.
    - user: 991:991
    - require:
      - sls: mastodon/docker
      - sls: mastodon/user
      - docker_network: mastodon_external
      - docker_network: mastodon_internal
      - file: /srv/mastodon.kingant.net/config/env_vars
    - watch:
      - file: /srv/mastodon.kingant.net/config/env_vars

# Docker container for Mastodon sidekiq.
mastodon-sidekiq:
  docker_container.running:
    - image: ghcr.io/mastodon/mastodon:v4.1
    - command: bash -c "set -o allexport && source /etc/opt/mastodon/env_vars && set +o allexport && bundle exec sidekiq"
    - binds:
      - /srv/mastodon.kingant.net/config:/etc/opt/mastodon:ro
      - /srv/mastodon.kingant.net/www/system:/opt/mastodon/public/system:rw
    - extra_hosts:
      - host.docker.internal:host-gateway
    - healthcheck: {test: ["CMD-SHELL", "ps aux | grep '[s]idekiq\ 6' || false"], retries=10}
    - networks:
      - mastodon_external
      - mastodon_internal
    - restart_policy: always
    - skip_translate: extra_hosts # Because Salt was complaining that "host-gateway" wasn't a valid IP.
    - user: 991:991
    - require:
      - sls: mastodon/docker
      - sls: mastodon/user
      - docker_network: mastodon_external
      - docker_network: mastodon_internal
      - file: /srv/mastodon.kingant.net/config/env_vars
    - watch:
      - file: /srv/mastodon.kingant.net/config/env_vars

Future Thoughts

I’d love to see more providers offering ActivityPub-as-a-service. There are some already, but they’re a bit pricey. And it feels like an immature product space. I could see companies wanting to use their own domain for their support and marketing accounts. e.g. “@support@delta.com” instead of something like “@deltasupport@mastodon.social”

I’d love to see a lighter-weight alternative to Mastodon that is targeted toward small installations like me. Like removing Redis as a requirement. Maybe getting rid of on-disk storage of files and just using the DB. Doing a better job of automatically cleaning up old, cached meda. Maybe this is Pleroma. Or maybe one of the other options.