KingAnt.net

I’ve started using Mastodon. It’s an open alternative to Twitter. “Open” meaning there’s no central controlling entity. Tech-savvy people can run their own server and servers can talk to each other (referred to as federation). If you’re interested in joining you can find a server to join, sign up, and follow me: @mark@kingant.net.

Why???

Part of me has always disliked posting on Facebook and Twitter. They’re walled gardens. I’d rather self host. I like keeping a record of my posts. Twitter allows downloading an archive of your posts and Facebook might, too, but I’d rather control the system of record myself. Also I don’t love being used by a company so they can make money from ads. Wise man once say, “when you’re not paying for the product you are the product.”

And then Elon Musk being a jerk¹ drove me to make a change.

The open/federated nature is appealing to me. It’s similar to how email works. And XMPP.

ActivityPub

ActivityPub is the important thing. There are other ActivityPub services that aren’t Mastodon. For example Pixelfed (an Instagram clone) and PeerTube (which is tailored for video sharing). It’s possible for them to be written such that they can interact with each other. For example on Mastodon I can follow a Pixelfed user. It’s great.

And actually I think it’s weird that these services are treated as separate things. Mastodon is just an ActivityPub service that looks like Twitter and Pixelfed is just an ActivityPub service that looks like Instagram. I’m not even sure the functionality is very different. It makes me wonder if they could be written as different UIs atop a common server. But whatever, I’m not super familiar with it. Maybe there are good reasons for them to be separate.

Running My Own Server

And because I’m a huge nerd I’m running my own server. I did it partially as a learning exercise. But also now my ID can be my email address (“@mark@kingant.net“) instead of something like “@markdoliner@someone-elses-server.example.com”

It took some work. The developers haven’t prioritized ease of installation (which is perfectly reasonable). It runs as two processes: The web/API service and a background job queue worker. It also requires PostgreSQL and Redis. As mentioned previously, I use Salt to manage my cloud servers. Initially I tried running everything directly on an EC2 instance, but Mastodon is written in Ruby and installing and managing Ruby and building Mastodon via Salt rules is a lot of work.

There are official Docker images so I looked into that instead. The docker-compose.yml file was immensely helpful. At first I tried running containers in Amazon Elastic Container Service (ECS). But configuring it is way more elaborate than you would imagine. And I’m not sure but it might have required paying for separate EC2 instances per container, though maybe it’s possible to avoid that if you go a step further and use Amazon Elastic Kubernetes Service (EKS). But of course that’s even more effort to configure.

What I did instead is run the containers myself on a single EC2 instance. Three containers, specifically: web/API, background job queue worker, and Redis. I’m running PostgreSQL directly on the host because it’s super easy. Obviously I could have used RDS for PostgreSQL and that’s certainly a nice managed option, but it’s also more expensive. I wanted to run Redis on the host but it’s hard to configure it to allow access from the Mastodon containers but disallow access from outside networks. Though even with PostgreSQL I ended up configuring it to accept connections from the world (disallowed via AWS security group of course, and PostgreSQL is configured such that users can only authenticate from localhost or the Docker network). So this feels like a decent compromise. The maintenance overhead is manageable and it’s fairly cheap. I’m paying around $3 to $4 per month ($83 up front for a t4g.micro 3 year reservation plus $1 to $2 per month for EBS storage).

I don’t want to make my Salt config repo public but here are the states for the Docker containers in case it helps anyone:

include:
  - mastodon/docker
  - mastodon/user

# A Docker network for containers to access the world.
mastodon_external:
  docker_network.present:
    - require:
      - sls: mastodon/docker

# A Docker network for intra-container communication.
mastodon_internal:
  docker_network.present:
    - internal: True
    - require:
      - sls: mastodon/docker

# Create directory for Mastodon. We leave ownership as the default
# (root) because the mastodon user shouldn't need to create files here
# and using root is a little more restrictive. It prevents the Mastodon
# processes (running in Docker as the mastodon user) from causing
# mischief (e.g. modifying the static web files, for example).
/srv/mastodon.kingant.net:
  file.directory: []

# Create private directory for Mastodon Redis data.
/srv/mastodon.kingant.net/redis:
  file.directory:
    - user: mastodon
    - group: mastodon
    - mode: 700
    - require:
      - sls: mastodon/user

# Redis Docker container.
# We're running it as a Docker container rather than on the host mostly
# because we would need to change the host Redis's config to bind to the
# Docker network IP (by default it only binds to localhost) and that
# requires modifying the config file from the package (there is no
# conf.d directory). That means we would stop getting automatic config
# file updates on new package versions, which is unfortunate.
# Of course if we ever want to change any other config setting then
# we'll have the same problem with the Docker contain. Though it's maybe
# still slightly cleaner having Redis accessible only over the
# mastodon_internal network, because Redis isn't big on data isolation.
# The options are:
# 1. Use "namespaces." Meaning "prefix your keys with a namespace string
#    of your choice, maybe with a trailing  colon."
# 2. Use a single server but have different apps use different DBs
#    within Redis. This is a thing... but seems problematic because
#    the DBs are numbered sequentially so what happens if you remove one
#    in the middle?
# 3. Use separate servers. This probably makes the most sense (and
#    provides the strongest isolation).
redis:
  docker_container.running:
    - image: redis:7-alpine
    - binds:
      - /srv/mastodon.kingant.net/redis:/data:rw
    # Having a healthcheck isn't important for Redis but it existed in
    # Mastodon's example docker-compose file so I included it here. The
    # times are in nanoseconds (currently 5 seconds).
    - healthcheck: {test: ["CMD", "redis-cli", "ping"], retries=10}
    - networks:
      - mastodon_internal
    - restart_policy: always
    # The UID and GID are hardcoded here and in user.present in
    # mastodon/user.sls because it's really hard to look them up. See
    # https://github.com/saltstack/salt/issues/63287#issuecomment-1377500491
    - user: 991:991
    - require:
      - sls: mastodon/docker
      - sls: mastodon/user
      - docker_network: mastodon_internal
      - file: /srv/mastodon.kingant.net/redis

# Create directory for Mastodon config file.
/srv/mastodon.kingant.net/config:
  file.directory: []

# Install the Mastodon config file. It's just a bunch of environment
# variables that get mounted in Docker containers and then sourced.
# Note: Secrets must be added to this file by hand. See the "solution
# for storing secrets" comment in TODO.md.
/srv/mastodon.kingant.net/config/env_vars:
  file.managed:
    - source: salt://mastodon/files/srv/mastodon.kingant.net/config/env_vars
    - user: mastodon
    - group: mastodon
    - mode: 600
    # Do not modify the file if it already exists. This allows Salt to
    # create the initial version of the file while being careful not to
    # overwrite it once the secrets have been added.
    - replace: False
    - require:
      - file: /srv/mastodon.kingant.net/config

# Docker container for Mastodon web.
mastodon-web:
  docker_container.running:
    - image: ghcr.io/mastodon/mastodon:v4.1
    - command: bash -c "set -o allexport && source /etc/opt/mastodon/env_vars && set +o allexport && rm -f /mastodon/tmp/pids/server.pid && bundle exec rails s -p 3000"
    - binds:
      - /srv/mastodon.kingant.net/config:/etc/opt/mastodon:ro
      - /srv/mastodon.kingant.net/www/system:/opt/mastodon/public/system:rw
    - extra_hosts:
      - host.docker.internal:host-gateway
    - healthcheck: {test: ["CMD-SHELL", "wget -q --spider --proxy=off http://localhost:3000/health || exit 1"], retries=10}
    - networks:
      - mastodon_external
      - mastodon_internal
    - port_bindings:
      - 3000:3000
    - restart_policy: always
    - skip_translate: extra_hosts # Because Salt was complaining that "host-gateway" wasn't a valid IP.
    - user: 991:991
    - require:
      - sls: mastodon/docker
      - sls: mastodon/user
      - docker_network: mastodon_external
      - docker_network: mastodon_internal
      - file: /srv/mastodon.kingant.net/config/env_vars
    - watch:
      - file: /srv/mastodon.kingant.net/config/env_vars

# Docker container for Mastodon streaming.
mastodon-streaming:
  docker_container.running:
    - image: ghcr.io/mastodon/mastodon:v4.1
    - command: bash -c "set -o allexport && source /etc/opt/mastodon/env_vars && set +o allexport && node ./streaming"
    - binds:
      - /srv/mastodon.kingant.net/config:/etc/opt/mastodon:ro
    - extra_hosts:
      - host.docker.internal:host-gateway
    - healthcheck: {test: ["CMD-SHELL", "wget -q --spider --proxy=off http://localhost:4000/api/v1/streaming/health || exit 1"], retries=10}
    - networks:
      - mastodon_external
      - mastodon_internal
    - port_bindings:
      - 4000:4000
    - restart_policy: always
    - skip_translate: extra_hosts # Because Salt was complaining that "host-gateway" wasn't a valid IP.
    - user: 991:991
    - require:
      - sls: mastodon/docker
      - sls: mastodon/user
      - docker_network: mastodon_external
      - docker_network: mastodon_internal
      - file: /srv/mastodon.kingant.net/config/env_vars
    - watch:
      - file: /srv/mastodon.kingant.net/config/env_vars

# Docker container for Mastodon sidekiq.
mastodon-sidekiq:
  docker_container.running:
    - image: ghcr.io/mastodon/mastodon:v4.1
    - command: bash -c "set -o allexport && source /etc/opt/mastodon/env_vars && set +o allexport && bundle exec sidekiq"
    - binds:
      - /srv/mastodon.kingant.net/config:/etc/opt/mastodon:ro
      - /srv/mastodon.kingant.net/www/system:/opt/mastodon/public/system:rw
    - extra_hosts:
      - host.docker.internal:host-gateway
    - healthcheck: {test: ["CMD-SHELL", "ps aux | grep '[s]idekiq\ 6' || false"], retries=10}
    - networks:
      - mastodon_external
      - mastodon_internal
    - restart_policy: always
    - skip_translate: extra_hosts # Because Salt was complaining that "host-gateway" wasn't a valid IP.
    - user: 991:991
    - require:
      - sls: mastodon/docker
      - sls: mastodon/user
      - docker_network: mastodon_external
      - docker_network: mastodon_internal
      - file: /srv/mastodon.kingant.net/config/env_vars
    - watch:
      - file: /srv/mastodon.kingant.net/config/env_vars

Future Thoughts

I’d love to see more providers offering ActivityPub-as-a-service. There are some already, but they’re a bit pricey. And it feels like an immature product space. I could see companies wanting to use their own domain for their support and marketing accounts. e.g. “@support@delta.com” instead of something like “@deltasupport@mastodon.social”

I’d love to see a lighter-weight alternative to Mastodon that is targeted toward small installations like me. Like removing Redis as a requirement. Maybe getting rid of on-disk storage of files and just using the DB. Doing a better job of automatically cleaning up old, cached meda. Maybe this is Pleroma. Or maybe one of the other options.

Apparently I haven’t posted this anywhere so I just want to go on record: Let’s Encrypt is amazing and fantastic. In five years the percentage of web pages loaded by Firefox using HTTPS went from 30% to 80%. That’s huge! A really impressive accomplishment¹. And ACME, the protocol for validating domain ownership, is also great. A massive improvement over the old process of getting certificates. And all those certificates are only valid for 90 days, instead of the old standard of a year or more! So good.

A Quibble

I don’t like how Certbot is configured. To keep things in perspective let me say that this is just a minor usability complaint. It doesn’t stop Let’s Encrypt from being fantastic.

Certbot is a tool for obtaining certificates from Let’s Encrypt for https and other services. The configuration file is written automatically when the certbot command is run. Users are discouraged² from modifying the configuration file directly. I can’t recall any other tool with this behavior. I don’t get it. Like, why? Sysadmins are used to writing config files, why is it important for this to be different? Maybe it’s supposed to be easier for users? But is it?

I guess usually this behavior is fine and not a problem. I find it inconvenient because I manage my two cloud servers using Salt (a configuration-as-code tool). It’s easy to place a config file on the server and apply changes to it. But since that’s discouraged I settled on running the certbot command but only if the config file does not exist. This is what I did, in case it helps anyone:

{% macro configure_certbot(primary_domain, additional_domains = none, require_file = none) %}
{#
  @param require_file We use Certbot's "webroot" authentication plugin.
         Certbot puts a file in our web server's root directory. This
         means we need a working web server. This parameter should be a
         Salt state that ensures Nginx is up and running for this
         domain.
#}
{# Don't attempt this in VirtualBox because it will always fail. #}
{% if grains['virtual'] != 'VirtualBox' %}
run-certbot-{{ primary_domain }}:
  cmd.run:
    - name: "certbot certonly
        --non-interactive
        --quiet
        --agree-tos
        --email mark@kingant.net
        --no-eff-email
        --webroot
        --webroot-path /srv/{{ primary_domain }}/www
        --domains {{ primary_domain }}{{ ',' if additional_domains else '' }}{{ additional_domains|join(',') if additional_domains else '' }}
        --deploy-hook 'install --owner=root     --group=root     --mode=444 --preserve-timestamps ${RENEWED_LINEAGE}/fullchain.pem /etc/nginx/{{ primary_domain }}_cert.pem &&
                       install --owner=www-data --group=www-data --mode=400 --preserve-timestamps ${RENEWED_LINEAGE}/privkey.pem   /etc/nginx/{{ primary_domain }}_cert.key &&
                       service nginx reload'"
    # Certbot saves the above arguments into a conf file and
    # automatically renews the certificate as needed so we only need to
    # run this command once. This would need to be done differently if
    # we ever want to change the renewal config via Salt. The Certbot
    # docs[1] describe using --force-renewal to force the renewal conf
    # file to be updated. So we could figure out a way to do that once
    # via Salt. Or we could manage the renewal conf file directly via
    # Salt.
    #
    # [1] https://eff-certbot.readthedocs.io/en/stable/using.html#modifying-the-renewal-configuration-of-existing-certificates
    - unless:
      - fun: file.file_exists
        path: /etc/letsencrypt/renewal/{{ primary_domain }}.conf
    - require:
      - pkg: certbot
      - file: {{ require_file }}
{% endif %}
{% endmacro %}

{{ configure_certbot('kingant.net', additional_domains=['www.kingant.net'], require_file='/etc/nginx/sites-enabled/kingant.net') }}

It’s a bit messy and I don’t know what I’ll do when I need to change the config. If you want certbot to update the config file you have to use --force-renewal, but I certainly wouldn’t want to do that every time I apply my configuration state to my servers. I think I’ll have to either run certbot --force-renewal by hand (fine, but loses the benefits of configuration-as-code), or have Salt manage the config file (discouraged by the official docs). Either option is fine, it just feels like a dumb problem to have.

I’m not the only one who has been inconvenienced by this. A quick search turned up this question thread and this feature request ticket.

Anyway, but Let’s Encrypt really is fantastic! This one usability complaint for my atypical usage pattern is super minor.