Let’s Encrypt is Fantastic

Apparently I haven’t posted this anywhere so I just want to go on record: Let’s Encrypt is amazing and fantastic. In five years the percentage of web pages loaded by Firefox using HTTPS went from 30% to 80%. That’s huge! A really impressive accomplishment1. And ACME, the protocol for validating domain ownership, is also great. A massive improvement over the old process of getting certificates. And all those certificates are only valid for 90 days, instead of the old standard of a year or more! So good.

A Quibble

I don’t like how Certbot is configured. To keep things in perspective let me say that this is just a minor usability complaint. It doesn’t stop Let’s Encrypt from being fantastic.

Certbot is a tool for obtaining certificates from Let’s Encrypt for https and other services. The configuration file is written automatically when the certbot command is run. Users are discouraged2 from modifying the configuration file directly. I can’t recall any other tool with this behavior. I don’t get it. Like, why? Sysadmins are used to writing config files, why is it important for this to be different? Maybe it’s supposed to be easier for users? But is it?

I guess usually this behavior is fine and not a problem. I find it inconvenient because I manage my two cloud servers using Salt (a configuration-as-code tool). It’s easy to place a config file on the server and apply changes to it. But since that’s discouraged I settled on running the certbot command but only if the config file does not exist. This is what I did, in case it helps anyone:

{% macro configure_certbot(primary_domain, additional_domains = none, require_file = none) %}
  @param require_file We use Certbot's "webroot" authentication plugin.
         Certbot puts a file in our web server's root directory. This
         means we need a working web server. This parameter should be a
         Salt state that ensures Nginx is up and running for this
{# Don't attempt this in VirtualBox because it will always fail. #}
{% if grains['virtual'] != 'VirtualBox' %}
run-certbot-{{ primary_domain }}:
    - name: "certbot certonly
        --email mark@kingant.net
        --webroot-path /srv/{{ primary_domain }}/www
        --domains {{ primary_domain }}{{ ',' if additional_domains else '' }}{{ additional_domains|join(',') if additional_domains else '' }}
        --deploy-hook 'install --owner=root     --group=root     --mode=444 --preserve-timestamps ${RENEWED_LINEAGE}/fullchain.pem /etc/nginx/{{ primary_domain }}_cert.pem &&
                       install --owner=www-data --group=www-data --mode=400 --preserve-timestamps ${RENEWED_LINEAGE}/privkey.pem   /etc/nginx/{{ primary_domain }}_cert.key &&
                       service nginx reload'"
    # Certbot saves the above arguments into a conf file and
    # automatically renews the certificate as needed so we only need to
    # run this command once. This would need to be done differently if
    # we ever want to change the renewal config via Salt. The Certbot
    # docs[1] describe using --force-renewal to force the renewal conf
    # file to be updated. So we could figure out a way to do that once
    # via Salt. Or we could manage the renewal conf file directly via
    # Salt.
    # [1] https://eff-certbot.readthedocs.io/en/stable/using.html#modifying-the-renewal-configuration-of-existing-certificates
    - unless:
      - fun: file.file_exists
        path: /etc/letsencrypt/renewal/{{ primary_domain }}.conf
    - require:
      - pkg: certbot
      - file: {{ require_file }}
{% endif %}
{% endmacro %}

{{ configure_certbot('kingant.net', additional_domains=['www.kingant.net'], require_file='/etc/nginx/sites-enabled/kingant.net') }}

It’s a bit messy and I don’t know what I’ll do when I need to change the config. If you want certbot to update the config file you have to use --force-renewal, but I certainly wouldn’t want to do that every time I apply my configuration state to my servers. I think I’ll have to either run certbot --force-renewal by hand (fine, but loses the benefits of configuration-as-code), or have Salt manage the config file (discouraged by the official docs). Either option is fine, it just feels like a dumb problem to have.

I’m not the only one who has been inconvenienced by this. A quick search turned up this question thread and this feature request ticket.

Anyway, but Let’s Encrypt really is fantastic! This one usability complaint for my atypical usage pattern is super minor.


  1. Yeah sure, Let’s Encrypt isn’t solely responsible. There had been a push to encrypt more sites post-Snowden anyway (e.g. Cloudflare in 2014). And there’s no way to know how big of an impact Let’s Encrypt had. Buuuuut, I bet it was huge. And yeah, it wasn’t Let’s Encrypt by themselves, hosts like Squarespace, DigitalOcean, and WP Engine have also done their part.
  2. “it is also possible to manually modify the certificate’s renewal configuration file, but this is discouraged since it can easily break Certbot’s ability to renew your certificates.”
Posted in All, Computers | Leave a comment

Using Salt to Manage Server Configuration


I’ve been using Salt (for clarity and searchability it’s also sometimes referred to as Salt Project or Salt Stack) to manage the configuration of my web server since 2014. It’s similar to Puppet, Chef (I guess they call it “Progress Chef” now), and Ansible.

At Meebo we used Puppet to manage our server config. This was like maybe 2008 through 2012. It was ok. I don’t remember the specifics but I felt like it could have been better. I don’t remember if there were fundamental problems or if I just felt that it was hard to use.

Anyway, when we chose the configuration management tech at Honor in 2014 we looked for a better option. We made a list of the leading contenders. It was a toss up between Salt and Ansible. They both seemed great. I don’t remember why we chose Salt. Maybe it seemed a little easier to use?

I Like It

I started using it for my personal web server around the same time. I’ve been happy with it. The main benefit is that it’s easier to update to a newer version of Ubuntu LTS every 2 or 4 years. My process is basically:

  1. Use Vagrant to run the new Ubuntu version locally. Tweak the config as needed to get things working (update package names, change service files from SysV init to systemd, etc.), and test.
  2. Provision a new EC2 instance with the new Ubuntu version. Apply the Salt states to the new server.
  3. Migrate data from the old server to the new server and update the Elastic IP to point to the new server.
  4. Verify that everything is good then terminate the old server.

It’s an upfront time investment to write the Salt rules, but it makes the above migration fairly easy. I run it in a “masterless” configuration, so I’m not actually running the Salt server. Rather, I have my server-config repo checked out on my EC2 instance.


Salt does have weaknesses. Since the beginning I’ve felt that their documentation could be more clear. It’s hard for me to be objective since I’ve been using it for so long, but here are a few minor gripes:

  • Some of the terminology is unclear. For example, I think the things in this list are typically referred as “states,” but the top of that page calls them “state modules” even though there is a different set of things that are called modules. Additionally the rules that I write to dictate how my server should be configured are also referred to as states, I think? And it’s not clear what modules are or when or how you would use them. There are often modules with the same name as a state but with different functions.
  • This page about Salt Formulas has a ton of advice about good conventions for writing Salt states. That’s great, but why is it on this page? Shouldn’t it be in the “Using Salt” section of the documentation instead of here, in the documentation intended for people developing Salt itself?
  • Navigating the Salt documentation is hard. Additionally there’s at least one item that’s cut off at the bottom of the right hand nav menu on this page. The last item I see is “Windows” but I know there is at least a “Developing Salt” section. Possibly also Release Notes, Venafi Tools for Salt, and Glossary.
  • The term “high state” feels unnatural to me. I think it has some meaning and if you understand how Salt works then maybe there’s a moment of clarity where the pieces fit together. But mostly it feels jargony.
  • It’s hard to have a state key off the result of an arbitrary command that runs mid-way through applying your configuration. There’s a thing called “Slots” that tries to solve this problem but it’s hard to use.
  • Also, speaking of Slots, why is it called “Slots”? And why is the syntax so awkward? Also I found the documentation hard to read. Partially because it feels jargony. Also there’s some awkward repetition (“Slots extend the state syntax and allows you to do things right before the state function is executed. So you can make a decision in the last moment right before a state is executed.”) and clunky grammar (“There are some specifics in the syntax coming from the execution functions nature and a desire to simplify the user experience. First one is that you don’t need to quote the strings passed to the slots functions. The second one is that all arguments handled as strings.”).


So I’m happy with it and intend to keep using it. I suspect other options have their own flaws. I have a vague impression that maybe Ansible is a little more widely-used, which is useful to consider.

Also the modern approach to running your own applications in the cloud is:

  • Build Docker containers for your applications.
  • Either deploy your containers directly to a cloud provider like Amazon Elastic Container Service, or deploy them using Kubernetes.

So there’s less of a need to do configuration management like this on hosts. But it’s probably still valuable for anything you don’t want to run in a container (mail server? DB server? VPN server?).

Posted in Computers | Leave a comment

Firefox as a Snap package in Ubuntu 22.04

In Ubuntu 22.04 Firefox is installed from a Snap package and not a dpkg like most other things and the update experience is awful. I get this notification:

Screenshot of a notification that says "Pending update of 'firefox' snap. Close the app to avoid disruptions (11 days left)"

However, closing Firefox and reopening it does not cause it to be updated. Apparently you have to either close it and leave it closed for a few hours until the Snap service’s next periodic update, which maybe happens every 6 hours, or close it and run a command to cause the Snap to update.

This is a terrible user experience. Nothing in the messaging informs me about the above. Also it’s absolutely unreasonable to expect the user to leave their browser closed for 3 hours (on average) until an update happens, and expecting a user to run a command manually is a poor experience. See this Stack Exchange post to see other people who were confused and annoyed by this behavior. Lots of people pointed it out in the comments on this post, too. Auto-updates should just happen and all the user should need to do is restart the application (assuming the application isn’t able to dynamically reload the affected files).

I also don’t understand why the Snap isn’t updated while the application is running. Linux is generally able to modify files that are in-use. It’s something that I see as a huge advantage that Linux has over Windows (for more discussion see this Stack Exchange question, answers, and comments). It’s plausible that some applications could misbehave if their files are changed while running—maybe Firefox suffers from this? But then I wonder what the update experience would be like if the user isn’t the administrator. Does the app get killed at some point so the update can happen?

Posted in Computers | Leave a comment

NASA Deep Space Network (DSN)

A random thing I stumbled upon and thought was interesting: NASA runs a website where you can see the current send/receive status of each antenna in the DSN (Deep Space Network) as well as the spacecraft it’s communicating with.

The DSN is a network of antennae at three locations around the world (California, Spain, Australia) that send and receive messages from various satellites and other spacecraft. They’re roughly 120 degrees apart from each other to give complete coverage. The Wikipedia page has a ton more info.

Posted in All | Leave a comment


Update 2022-08-13: A few months ago the app started crashing and kept crashing so I uninstalled it. Looking at the comments on Play Store, other people have had the same problem. That was on a three and a half year old phone (a Google Pixel 2). I have since uploaded to a newer phone so maybe the app would work now, but I haven’t tried. One thought I had is that maintaining this at the state level is more work than doing it at the national level, which is an unfortunate situation. Also I wonder if exposure notifications work across states currently, e.g. if someone from New York traveled to California.

Original Post Follows

I installed North Carolina’s SlowCOVIDNC virus exposure notification app and if you live in NC I encourage you to install it, too. You can get it from Apple App Store or Google Play.

So if you’ve been hesitating and wondering “I wonder what my computer programmer and security conscious friends think about this app?” The answer is I think you should install it.

I try harder than most people to avoid installing apps on my phone. I’m wary of apps causing problems either through incompetence or malicious intent. I think the risks in this case are low (much lower than any other random app, like a game or weather app) and the potential to reduce virus spread rate exists (and I barely even go out), so it’s worth it.

Posted in Computers | Leave a comment

Comparing Signatures on Mail-In Ballots

Voting by mail? Did you know that 28 states use signature matching to verify ballots?

They compare the signature on your ballot with one they have on file from voter registrations, ballot applications or the D.M.V. California apparently does this. NC apparently does not (we have to have two witness signatures, instead (Edit: Actually just one witness. Wasn’t it two for the primary?? Definitely appears to be just one now)).

All but 4 of those states have a process to allow voters to fix a mismatched signature. This process is sometimes referred to as “curing.”

So if you’re voting by mail, you may wish to be careful with your signature. Or if you think your signature is inconsistent and you’re in a swing state (Edit: or you’re voting for any close race that you care about) and don’t have a lot of confidence in the cure process and want to be extra sure your vote is counted then you may wish to vote in person.

Wondering what your state does? There’s a color-coded map near the bottom of this NY Times article.

I had a thought that there was some federal decision stating that all states must give voters an opportunity to fix invalid ballots, but I can’t find anything indicating such. Somewhat related, this WRAL article discusses uncertainty about how NC will handle incomplete vote-by-mail ballots.

Posted in All | Leave a comment

Two NC Races in the November Election

For those of you who might otherwise vote a straight Republican ticket in NC, I’d like to draw your attention to two races.

State Auditor

An N&O article says this about the Republican candidate: “put on probation in connection with a stalking charge, and also has been accused of refusing to obey orders from police, causing a scene at a concert and threatening a man’s family over money.” To be fair the article also says he “has not been convicted on the criminal charges.” And I guess he has a master’s degree in public administration, which is maybe relevant. But! The Democratic candidate is so much better. Beth Wood has a degree in accounting and is a CPA. She’s the incumbent (first elected in 2008) and before that worked in the state auditor’s office as well as the state treasurer’s office. So she’s experienced and qualified. And political party shouldn’t even really come into play in this office!


Lieutenant Governor

Sure, you might like some of the things Mark Robinson says. He hits a lot of talking points Republicans care about. But he also has no experience in state politics. As a reminder, the lieutenant governor presides over the senate, and I’d argue that he is not qualified to do so. He makes a lot of offensive statements on Facebook, and also says weird things like “The looming pandemic I’m most worried about is SOCIALISM” and “When the TRULY innocent are murdered leftists could care less. In fact, they champion such” and “When will the Crips, Bloods, and Planned Parenthood start believing black lives matter?” He sounds unfocused and angry at everything. He does not sound like someone who would help a legislative body operate more effectively. The Democratic candidate Yvonne Lewis Holley, on the other hand, sounds fine. She has served in the NC house since 2013 so she’s familiar with the legislative process. The issues she cares about demonstrate compassion and I’d argue that that’s a valuable virtue.


Posted in All | Leave a comment

Camping With Kids

Emily and the kids and I went camping for one night only at Hanging Rock State Park last night. It was good! A lot of work, but I think we all had a good time and it was definitely nice to get out of the house.

I mostly just want to make a record of this so I remember what it was like at their ages. They both did good in the car. They might not have napped at all.

We had a campfire. With s’mores. The marshmallows were especially popular. Three deer meandered through the campsite next to ours. Emily and I thought it was cool. The kids mostly didn’t care. It rained overnight while we were in our tent and we all stayed dry, so it was pleasant without being inconvenient.

Upper Cascades Falls at Hanging Rock State Park

Ruby did a three mile hike yesterday and a one mile hike today, which impressed Emily and I. Edie stumbled a lot and I carried her in our hiking backpack most of the time. Ruby ate pasta out of a bowl without too much trouble. Edie had a hard time holding a bowl without spilling or dropping it.

I’m still very happy with the roof cargo box we bought (Yakima SkyBox 16 Carbonite). Good size. Works well. Super functional. And the built-in roof rails on our Subaru Outback are great, too. I didn’t factor factory roof rails into our decision process when we bought the car but I probably should have.

Posted in All | Leave a comment