Last year, I needed to move my services to a new Linux VPS. Previously, I used a mess of bash scripts to configure my server, with an aim of version-controlling the configuration. Whilst this was a good idea in theory, in practice this was incredibly complicated and brittle. I now use Ansible to provision my VPS and my Raspberry Pi.
My situation #
I self-host my personal website and various services for my hobby and open source projects. This includes Luanti’s ContentDB and Renewed Tab’s API. My server serves over a million requests per day to 100,000 unique visitors.
I host using a Linux VPS rather than cloud computing because it is more cost efficient at my scale and has predictable pricing. I can get higher performing hardware for less money than major cloud providers, and I don’t need to worry about losing my savings if I go viral. I’d much rather go offline than go broke!
For years, I used a collection of bash scripts to set up the server, version-controlled in a Git repo. The bash scripts installed software, linked configuration files, and created users. Whilst this was a good idea in theory, in practice I found it to be increasingly complicated and brittle. Because it was slow, I rarely ran the entire thing and did a lot of manual configuration on the server. This meant that the configuration repo did not fully capture the state of the server, limiting its usefulness to set up a new server.
The goal #
My goal was to switch to a new tool that would allow me to do Infrastructure-as-Code on the server, minimising manual operations. I wanted to be able to provision new servers quickly, if needed. If my VPS were to suddenly disappear, I’d like to be back online within 30 minutes using an automated provisioning process.
Another goal was to make the configuration modular such that I could provision multiple servers at a time, running different services on each. Currently, I have my VPS and a Raspberry Pi to download backups and run local services. In the future, I might set up a staging server, for example, or use dedicated VPSes for a particular service.
How I use Ansible #
Ansible is a tool for automating tasks against an inventory of target hosts. A playbook is a collection of tasks to achieve a goal.
Playbooks and roles #
I created an Ansible playbook to configure and set up the new Linux VPS. The playbook has tasks in three groups:
- The core play sets up essential software like curl, git, python, and NodeJS. It also configures the firewall (including to block abusive subnets) and hardens some settings.
- The services play sets up all the services I host.
- The monitoring play sets up prometheus, Grafana, and alerting.
To make the system modular, each of these parts is further split into roles to achieve its goal. A role is a reusable package of tasks, variables, files, and templates. Here’s the directory structure:
📄 ansible.cfg
📄 inventory.ini
📄 playbook.ansible.yml
📄 requirements.yml
📁 roles
| ------ core ------
├─ 📁 core
├─ 📁 firewall
├─ 📁 nginx
├─ 📁 backups
| ---- services ----
├─ 📁 contentdb
| ├─ 📁 files
| ├─ 📁 tasks
| ├─ 📁 templates
| └─ 📁 vars
├─ 📁 renewedtab
├─ 📁 doineedacoat
├─ 📁 blog_api
├─ 📁 quassel-core
| --- monitoring ---
└─ 📁 monitor
Each role inside a play is tagged, which allows me to run specific roles to
save time when reconfiguring a particular service. For example,
ansible-playbook playbook.ansible.yml -t renewedtab.
- name: Services
hosts: servers
roles:
- role: contentdb
tags: ["contentdb"]
- role: renewedtab
tags: ["renewedtab"]
- role: doineedacoat
tags: ["doineedacoat"]
- role: blog_api
tags: ["blog_api"]
- role: quassel-core
tags: ["quassel"]Services #
Each service I host has a dedicated role to configure it. A typical service role will create a user, clone a repo, install software, and spin up Docker containers.
I prefer to use Docker to containerise my services. It allows me to reduce the amount of software I need to install globally, which improves resilience, especially when I want to update software. A few of my services are hosted using global NodeJS and PM2 instances; I plan to move these to Docker in the future.
nginx is installed on the server outside of Docker and listens on HTTP/S ports.
Each service adds an .nginx file to sites-available for nginx to pick up. In
the future, I’m considering splitting this up into a lightweight reverse proxy
like HAProxy at the system level, and then a web server for each service to
handle specific configuration, caching, and ratelimiting.
Secrets management #
Several of my services require secrets to be available, stored as plaintext in a
config file such as config.cfg or .env. Using ContentDB as an example, I
created contentdb/vars/secrets.yml:
contentdb_postgres_password: ""
contentdb_secret_key: ""
contentdb_wtf_csrf_secret_key: ""
contentdb_github_client_id: ""
contentdb_github_client_secret: ""
contentdb_github_api_token: ""
contentdb_mail_username: ""
contentdb_mail_password: ""
Then the config file template at contentdb/templates/config.cfg.j2:
SECRET_KEY="{{ contentdb_secret_key }}"
WTF_CSRF_SECRET_KEY="{{ contentdb_wtf_csrf_secret_key }}"
SQLALCHEMY_DATABASE_URI="postgresql://contentdb:{{ contentdb_postgres_password }}@db:5432/contentdb"
GITHUB_CLIENT_ID="{{ contentdb_github_client_id }}"
GITHUB_CLIENT_SECRET="{{ contentdb_github_client_secret }}"
GITHUB_API_TOKEN="{{ contentdb_github_api_token }}"
MAIL_USERNAME="{{ contentdb_mail_username }}"
MAIL_PASSWORD="{{ contentdb_mail_password }}"
And then finally used in contentdb/tasks/main.yml:
- name: Include secrets
ansible.builtin.include_vars: secrets.yml
- name: Create config.cfg
ansible.builtin.template:
src: config.cfg.j2
dest: /home/contentdb/contentdb/config.cfg
owner: contentdb
group: contentdb
mode: "0600"I named the vars file secrets.yml so that all secrets files have the same name
and can easily be gitignored.
Ansible does support encrypting secrets using Ansible Vault. However, the secrets are stored in plaintext on the server so this would have limited value and would have increased friction when using Ansible.
Daily backups to my RPi #
I use a Raspberry Pi with an HDD to back up my server. To provision the RPi, I
use an rpi.ansible.yml playbook to run the core and backups roles.
When run on the VPS, the backups role will create a backups user and set up
/var/local/backups/ to store database dumps. When run on the RPi, the role
will create an SSH key and attach it to the VPS, and then set up a cron job to
download the backups daily. The contentdb role is responsible for creating the
database backups.
I encrypt the database backups using GPG because they contain sensitive user data. The Raspberry Pi can only download encrypted backups; it does not have access to the decryption key or the production data.
Future improvements #
I’d like the ability to run my playbook against a local Docker container or virtual machine, so that I can continuously verify that it can set up a fresh servers.
Opinions #
Things I like about Ansible #
I like how Ansible tracks changes and can skip unnecessary steps. I like the wide range of built-in modules. I like how you can use Jinja2 templates to generate configuration files from variables and facts. I like how you can use Ansible to provision multiple machines at a time.
Why Ansible is not my ideal solution, but it will do #
Ansible is an imperative configuration management tool where you write the steps to provision a server.
My ideal solution would be a declarative configuration management tool for setting up a server. Rather than specifying the imperative steps to set up a server, I’d like to be able to specify the end result and have the tool work out the steps. For example, I might declare a load balancer, a docker-compose, and a systemd service, and the tool would set up users and configure software to reach the end result.
The closest thing I’ve found to this was NixOS, which I plan to try out at some point in the future. I went with Ansible as it’s a more popular tool that could be useful in a future job.
However, Ansible is meeting my goals for Infrastructure-as-Code and there’s a point where messing around with configuration tools has diminishing returns. I’d much rather spend my time looking into other ways to improve my DevOps, such as cheap ways to achieve high availability.
Conclusion #
I’m interested to hear any feedback or handy things that I might have missed.
I’m planning to look into other tools and solutions for managing my services. I’d also like to investigate the use of cloud technologies as a small-scale service provider. What’s the best cloud setup I could get for less than $50 per month?
Comments