Developing ContentDB

In 2018, I had the opportunity to create a web app for University coursework, as a solo project. I chose to create a package repository for Luanti, an open-source project I help maintain.

Luanti (formerly Minetest) is an open-source game engine with millions of downloads and thousands of weekly players. The project has a very active modding community, and many available games to run. There was one big issue - you had to manually install mods and games by unzipping their files into a directory. This was a very poor user experience.

The project aimed to make a website to hold the metadata of different types of packages. The website needed to have both an HTML front-end, and a JSON REST API to be used by the Minetest Engine and other software. Authors of packages should be able to upload and maintain their packages, and Editors should be able to upload and maintain any package.

The community commonly receives attacks from bots and malicious users, so the system needed sufficient moderator tools. New users should have any uploads or changes checked before they’re published. There also needed be moderation tools, such as banning.

The website needed to be stable and easy-to-use for content creators and users alike. This means that uploading packages needed to be as frictionless as possible, which calls for interesting features such as importing metadata from GitHub and other VCSes.

Prior Attempts #

Creating a package repository for Minetest was by no means a brand new idea; there had been many prior attempts, a few of which by me.

2012: Minetest Extensions #

I created a PHP website called Minetest Extensions. It used a MySQL database and implemented several proprietary APIs required by existing Minetest package manager command-line tools. I wasn’t experienced with back-end development at the time, and so it suffered from bugs and security issues.

2013: Minetest Mod Database #

Another user created a Python / Django package repository. It was made official and hosted by celeron55.

It never had many mods due to a very manual data entry process, which could only be performed by the mod author.

It only supported mods and texture packs. Any mods were required to be mods to the strict definition; Mod packs were not allowed. This was a huge problem given that many of the most important mods are structured into mod packs.

The website went offline due to the database breaking and the host being unable to fix it. It didn’t use containerisation and was hard to set up and develop. The developer was also only sporadically available.

2015: Minetest Bower #

Another user created a package repository based on Bower, a Git-based package manager for the web. Using it for Lua-based packages was a bit of an abuse of the tool, but it did work.

The major issue was that it only supported content with a Git repository, and required the owner to update the repository to add a bower.json file. This resulted in even fewer mods being added than to the Minetest Mod Database.

2016: Minetest Mods Android App #

I created an Android app to install mods. It used a NodeJS backend that crawled the forums to get mod information. This worked to a degree, but the data was flawed and required me to manually override a lot of information that the crawler didn’t detect properly.

2017: NodeJS-based ContentDB #

I wrote a prototype using NodeJS, Sequelize, and PostgreSQL for a package repository that had goals similar to ContentDB. It had support for Asynchronous Tasks (Git importing and automatic releases), a user login system, and a REST API.

This project allowed me to assess the suitability of NodeJS for the task; I got negative reactions from other core developers. I also wasn’t a fan of Sequelize as Python’s SQLAlchemy was so much better.

2017: Minetest Content Database #

Another user created a package repository based on Python / Django. It supported mods, games, texture packs, skins, and servers.

Development faded out by the start of 2018.

Client Requirements #

The client for this project was Perttu Ahola, aka celeron55, the original creator of Minetest.

I started by interviewing to ascertain the requirements of such a package repository. The result was the following high-level requirements:

1. It must be easy to deploy.
2. It must have some kind of security model to spare most users from malicious mods.
3. It must be practical to implement an interface to it from Minetest.
4. It should be easy to add stuff, perhaps as simple as clicking a button to import from GitHub.
5. Should have good moderation tools and high attention to security.

…and the following specific features:

• MUST: Users have one of the following ranks, which have different permissions: new member, member, editor, moderator, admin.
• MUST: Users should be able to log in using username/password, GitHub, or the forums (phpBB).
• MUST: Packages are one of the following types: mod, game, texture pack.
• MUST: Packages can be created by any user but must be approved before they’re made public.
• MUST: Packages have releases, which are immutable download-able versions.
• MUST: Edits to packages may need to be reviewed depending on user rank and whether they’re the author.
• MUST: A REST API to allow access to the data.
• MUST: A database to store the data.
• SHOULD: Packages have tags that can be used to filter them in search.
• SHOULD: Package metadata can be imported from GitHub.
• SHOULD: Moderation tools such as banning.

Design #

With these requirements in mind, I created a design document. I made sure to keep celeron55 in the loop; he signed off on the final version.

Permissions and Ranks #

One of the most important system in ContentDB is the permission system. There are many named permissions, such as EDIT_PACKAGE, DELETE_RELEASE, and APPROVE_SCREENSHOT. When performing an action, ContentDB will check whether the user has the required permission on the object in question.

Each user has a rank. A rank determines the permissions of the user and include the permissions of lower ranks.

• New Members: need to have any edits or new packages approved.
• Members: can edit or upload packages by them, but edits or uploads for other authors need to be approved. Can approve edits to their packages.
• Editors: can edit any package, upload for any user, and approve changes to any package. Can approve any releases.
• Moderators: can ban users and delete mods.
• Admin: can change passwords and create users.

New packages must be approved by an Editor.

Technology #

I chose to use Python/Flask with SQLAlchemy. I had prior experience with these frameworks, and Minetest already had a web app using Flask.

Database #

I decided to use a relational database to store the data required for ContentDB to function. As data integrity was a concern, I spent extra effort on designing database-level validation using constraints.

Development for Coursework #

I was given five months to develop the project for the university coursework module.

I started by setting up the database. I chose to use PostgreSQL, as it has good support for validation constraints and migrations.

Once the database was created, I worked on implementing a front-end template to allow further progress. I chose to use my own CSS to get the best possible score. I wasn’t as good at CSS as I am today, and making it work was considered more important than making it pretty.

I used pre-existing libraries for user and login management for the coursework version.

Edit Requests were a feature that existed in early versions of ContentDB. It allowed users to create requests to edit a package, which can then be accepted or rejected by the package’s owner or an Editor. It was removed after the project was submitted due to changing priorities and difficulties in maintaining the implementation.

Asynchronous Tasks, Celery, and Git Support #

ContentDB uses Celery to run asynchronous tasks, to avoid blocking the server thread. These tasks include importing meta from Git, creating releases, and fetching user info from the forums.

Git support was the most tricky thing to implement, as it required acquiring a fairly in-depth knowledge of how Git works - and it’s not simple software. For example, one bug I had was a very weird error when attempting to clone a particular repo. It turned out that a tag existing on a commit that wasn’t on a branch was the cause. Through the use of integration tests, I was able to nail the implementation.

Deployment #

I deployed ContentDB using Docker onto my dedicated server. Docker allows local development and production to match, making development much easier.

I implemented support for ContentDB in Minetest’s main menu, allowing it to install and update packages.

I added Prometheus support with a Grafana dashboard to show statistics, such as downloads and total users.

Post-submission #

In the four years since ContentDB was submitted as coursework, I have continued to develop it. One of the first changes I made was switching from custom CSS to Bootstrap CSS, which made future development easier.

The submitted version just had a read-only API for getting information about packages and updates. I added support for API access tokens and extended the API to allow for editing packages and making releases.

Other improvements include:

• Threads and comments
• Ratings and reviews
• Email notifications
• Package videos
• Improved tools for Editors and users
• Package .zip validation
• Release creation on Webhooks from GitHub and GitLab
• Git update detection: ContentDB will check repos to see if there’s been any new commits or tags
• Support for dependency installation
• Translation / internationalisation

Conclusion #

One of the main problems with ContentDB was ensuring maintainability as it grew. Since submission, I’ve done several refactors to improve this, such as separating business logic from the app’s routes. This separation allows the front-end, REST API, and async tasks to share the same code, which reduces risk.

Another problem was insufficient UI tests. I used Flask’s test context feature to check the response of queries. There are tests for core behaviour, such as user logins and package pages, but they’re not particularly comprehensive. One reason is that the testing tools used were insufficient; I have since invested time in learning how to better test web front-ends, using Selenium, and have applied that to more recent projects.

ContentDB is one of the largest projects I’ve developed, and one of the longest running.