Anubis: Distrusting data centers and particular ASNs

My services receive AI scraper attacks primarily from data centers such as BytePlus^{footnote 1}. I started using Anubis to challenge requests to my service and protect it from going offline. My goal was for the system to be invisible to most users, and only challenge the most suspicious users. This article describes how to distrust particular network blocks in Anubis, such as data centers, for free - without using Anubis’ expensive add-on called Thoth.

My situation #

I am the creator and sole maintainer of ContentDB, a website to provide games, mods, and texture packs for the Luanti game-creation platform. This is an open-source hobby project I do in my own time, it’s not-for-profit.

In the last year, ContentDB has gone offline several times due to attacks from AI scrapers. Each time, I’ve needed to manually block specific data centers and IP ranges to stop the attack.

ContentDB is hosted on a £16 per month Virtual Private Server (VPS). I use a single Linux VPS because it provides predictable pricing. This is a volunteer and non-profit project; if ContentDB is suddenly hit by a lot of traffic, I would rather that it go offline than send me a huge bill due to cloud hosting.

So whilst adding a CDN like Cloudflare or Bunny would be a much better solution, it would almost double the monthly cost. Anubis is not perfect but does prevent these crawler attacks from increasing the costs of a volunteer non-profit project.

My goal #

My goal was for the system to be invisible to most users and only challenge the most suspicious users. As most of the attacks I received were from data centers, I wanted a way to distrust data centers. Under normal load, Anubis should challenge requests from data centers but allow other users in. Only when the system load increases should normal users start to see challenges.

This system means that 95% of my visitors will not see an Anubis challenge.

Setting up GeoLite2 ASN #

First, install libnginx-mod-http-geoip2 and geoipupdate. The former is an open source plugin, not to be confused with the paid-for Nginx Plus plugin.

Next, register for a MaxMind GeoLite2 licensekey at https://www.maxmind.com/en/geolite2/signup.

Place the account and licence key at /etc/GeoIP.conf (replace everything including the braces):

AccountID {{ nginx_maxmind_account_id }}
LicenseKey {{ nginx_maxmind_license_key }}
EditionIDs GeoLite2-ASN GeoLite2-City GeoLite2-Country

This can be automated using Ansible:

- name: Install nginx
  ansible.builtin.apt:
    pkg:
      - libnginx-mod-http-geoip2
      - geoipupdate
    state: latest
    update_cache: true

- name: Write GeoIP.conf
  ansible.builtin.template:
    src: GeoIP.conf.j2
    dest: /etc/GeoIP.conf
    owner: root
    group: root
    mode: "0644"

- name: Fetch MaxMind GeoLite2 databases
  ansible.builtin.command:
    cmd: geoipupdate
  register: nginx_geoipupdate_result
  changed_when: "'GeoIP2 database update complete' in nginx_geoipupdate_result.stdout"

Enable geoip2 in the root nginx config:

geoip2 /var/lib/GeoIP/GeoLite2-ASN.mmdb {
	auto_reload 60m;
	$geoip2_data_asn source=$remote_addr autonomous_system_number;
}

And expose it to Anubis in the location block:

location / {
	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Http-Version $server_protocol;
	proxy_set_header X-ASN $geoip2_data_asn;
	proxy_pass http://anubis;
}

Use in Bot policy #

You can now use the ASN in an Anubis rule:

- name: distrust-commercial-asns
  headers_regex:
    X-ASN: "^(1442|3223|3561|3722)$"
  action: WEIGH
  weight:
    adjust: 10

- name: high-load-average
  action: WEIGH
  expression: load_1m >= 9.0 # make sure to end the load comparison in a .0
  weight:
    adjust: 20

- name: low-load-average
  action: WEIGH
  expression: load_15m <= 4.0 # make sure to end the load comparison in a .0
  weight:
    adjust: -10

Automating ASN blocklist using Ansible #

I use a bad-asn-list from GitHub to distrust popular data centres.

Use Ansible to fetch and update the list:

- name: Download bad-asn-list CSV
  ansible.builtin.get_url:
    url: https://raw.githubusercontent.com/brianhama/bad-asn-list/master/bad-asn-list.csv
    dest: /tmp/bad-asn-list.csv
    mode: '0644'
    force: true

- name: Read bad-asn-list CSV
  community.general.read_csv:
    path: /tmp/bad-asn-list.csv
  register: anubis_bad_asn_csv

- name: Extract ASN numbers
  ansible.builtin.set_fact:
    anubis_bad_asns: "{{ anubis_bad_asn_csv.list | map(attribute=anubis_bad_asn_csv.list[0].keys() | list | first) | unique | list }}"

and use the list in the botPolicy like so:

- name: distrust-commercial-asns
  headers_regex:
    X-ASN: "^({{ anubis_bad_asns | join('|') }})$"
  action: WEIGH
  weight:
    adjust: 10