rubenwardy's blog
Welcome to my blog, where I write about my projects and various tech-related topics.
2024-03-08T00:00:00.000Z
https://blog.rubenwardy.com/feed.json
rubenwardy
Why I like and recommend using TypeScript
2024-03-08T00:00:00.000Z
2024-03-08T00:00:00.000Z
https://blog.rubenwardy.com/2024/03/08/typescript-is-awesome/
I find that TypeScript makes JavaScript more enjoyable to use. In this article, I'll explain why.
<p>I’ve recently gotten into using TypeScript when working with web technology, for
creating web apps and servers. I find that TypeScript makes JavaScript more
enjoyable to use. In this article, I’ll explain my reasons why.</p>
<h2 id="improves-the-developer-experience" tabindex="-1">Improves the developer experience <a class="header-anchor" href="#improves-the-developer-experience">#</a></h2>
<p>The number one benefit for me is that it massively improves the developer
experience without impacting the agility of an interpreted language. TypeScript
combined with a supported IDE like VSCode(ium) delivers a great experience.</p>
<h3 id="makes-the-ide-more-helpful" tabindex="-1">Makes the IDE more helpful <a class="header-anchor" href="#makes-the-ide-more-helpful">#</a></h3>
<p>TypeScript provides type information that helps the IDE help you. Whilst type
information is supported for JavaScript, it’s not as reliable or fully featured.</p>
<p>With type information, the IDE can show relevant suggestions for methods and
members (ie: IntelliSense), allowing you to see what is inside an object. This
increases productivity. TypeScript makes refactoring operations like renaming
considerably more reliable as it can accurately find all the uses of a symbol.</p>
<figure class="undefined"><a href="/static/img/Dmj8IEK8QO-1057.png"><picture><source type="image/webp" srcset="/static/img/vHZnJzxY5P-1057.webp 1057w"><img alt="VSCode listing the members of an interface" loading="lazy" decoding="async" src="/static/img/vHZnJzxY5P-1057.png" width="1057" height="169"></picture></a><figcaption>VSCode listing the members of an interface</figcaption></figure>
<h3 id="types-as-documentation" tabindex="-1">Types as documentation <a class="header-anchor" href="#types-as-documentation">#</a></h3>
<p>Defining types acts as a form of documentation. This doesn’t replace actual
documentation but instead supplements it - it tells you the types that an API
expects.</p>
<p>Here’s the signature of a function without any comments. Just the argument names
and the types are enough to get an idea of how to use it.</p>
<pre class="language-ts"><code class="language-ts"><span class="token function">findPath</span><span class="token punctuation">(</span>from<span class="token operator">:</span> Vector2<span class="token punctuation">,</span> to<span class="token operator">:</span> Vector2<span class="token punctuation">,</span> maxDistance<span class="token operator">?</span><span class="token operator">:</span> <span class="token builtin">number</span><span class="token punctuation">)</span><span class="token operator">:</span> Vector2<span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">;</span></code></pre>
<p>An alternative to TypeScript is JSDoc. This allows you to document the types in
JavaScript using comments instead of type annotations. This is a good stopgap if
you have an existing code base in JavaScript, but is quite verbose and not as
effective.</p>
<pre class="language-js"><code class="language-js"><span class="token comment">/**
* @param {Vector2} from
* @param {Vector2} to
* @param {Vector2} [maxDistance]
* @returns {Vector2[]} the path
*/</span>
<span class="token function">findPath</span><span class="token punctuation">(</span>from<span class="token punctuation">,</span> to<span class="token punctuation">,</span> maxDistance<span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h2 id="static-linting-errors-for-types-and-mistakes" tabindex="-1">Static linting: errors for types and mistakes <a class="header-anchor" href="#static-linting-errors-for-types-and-mistakes">#</a></h2>
<p>A secondary benefit for me is that TypeScript will emit compile errors when
types do not match. This is very useful for discovering a lot of different
problems.</p>
<p>One example of a situation where this helps is when dealing with optional types.
TypeScript will tell you if you try to access an optional type without first
checking for null/undefined. I find that in a lot of languages, nullability
causes a lot of issues. It’s a big issue in Java, for example, and one of the
reasons why I prefer Kotlin. Remember the types-as-documentation argument as
well: having nullability in the types tells you, as the developer, whether you
need to handle the null case.</p>
<p>Type checking is also very useful when refactoring code and updating
dependencies, as it gives you compile errors for invocations that don’t match
changes made.</p>
<p>If you’re doing development properly, you should have unit tests and code
reviews. These have a greater impact on code quality and correctness than
TypeScript does, but it doesn’t negate the benefit completely.</p>
<h3 id="compile-time-is-better-than-run-time" tabindex="-1">Compile-time is better than run-time <a class="header-anchor" href="#compile-time-is-better-than-run-time">#</a></h3>
<p>A common complaint II’ve read about TypeScript is that it only does compile-time
type checking and not runtime. It’s important to note that compile-time checks
are better than run-time checks. It’s good to catch as many errors as possible
at compile-time, as run-time is slower and less reliable. Compile-time errors
are fast enough to be shown in your IDE as you edit the code.</p>
<p>C and C++ are considered to be strongly typed languages and rely almost entirely
on compile-time type checking. The “static” in <em>static</em> type checking means that
the types are verified at compile-time. In C, the concept of types doesn’t even
exist at runtime. C++ has
<a href="https://en.wikipedia.org/wiki/Run-time_type_information">Run-Time Type Information (RTTI)</a>,
which allows you to do safe casts at run-time (<code>std::dynamic_cast</code>) and also get
type information (<code>typeid</code> and <code>std::type_info</code>). These features are useful but
not essential - it’s common for developers to disable RTTI to reduce program
size in embedded applications. Now, perhaps C/C++ isn’t the best role model when
it comes to type safety. Rust would be a better example, as it has a
well-written compiler and is memory-safe.</p>
<p>So, what’s the problem here with TypeScript? The problem isn’t that it only does
compile-time checks, the problem is that incorrect types can enter the program.
First, incorrect types can enter the program through third-party dependencies.
This is especially common when the dependencies are written in JavaScript.
Second, TypeScript has dangerous operations that can result in incorrect types.
For example, type assertions (<code>as T</code>) allow you to tell the compiler that a
symbol is a certain type even if this isn’t true. The use of <code>any</code> is quite
dangerous as well.</p>
<p>It’s possible for compile-time checking to be fully capable if you limit the use
of dangerous operations. You can use <code>eslint</code> to enforce this in a project.</p>
<h3 id="dealing-with-untyped-data" tabindex="-1">Dealing with untyped data <a class="header-anchor" href="#dealing-with-untyped-data">#</a></h3>
<p>Incorrect types can enter a TypeScript program when interacting with untyped
dependencies and data. One way to ensure this doesn’t happen is by using a
library like Zod. <a href="https://github.com/colinhacks/zod">Zod</a> is a TypeScript
validation library that allows you to check unknown data using a schema. It also
generates TypeScript types, allowing schemas to work seamlessly with TypeScript.</p>
<pre class="language-ts"><code class="language-ts"><span class="token keyword">const</span> User <span class="token operator">=</span> z<span class="token punctuation">.</span><span class="token function">object</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
username<span class="token operator">:</span> z<span class="token punctuation">.</span><span class="token function">string</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
full_name<span class="token operator">:</span> z<span class="token punctuation">.</span><span class="token function">string</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
age<span class="token operator">:</span> z<span class="token punctuation">.</span><span class="token function">number</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> user <span class="token operator">=</span> User<span class="token punctuation">.</span><span class="token function">parse</span><span class="token punctuation">(</span><span class="token constant">JSON</span><span class="token punctuation">.</span><span class="token function">parse</span><span class="token punctuation">(</span>json<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">// `user` will have the correct TypeScript type.</span></code></pre>
<p>Another cool thing is that Zod can transform data for you whilst validating. If
an API gives you a date timestamp in ISO format, you can use <code>z.date()</code> to
convert that string to a JS <code>Date</code> object. You can also provide custom transform
functions to validate and manipulate the data. This makes it a very powerful
tool for dealing with untyped data.</p>
<h2 id="drawbacks" tabindex="-1">Drawbacks <a class="header-anchor" href="#drawbacks">#</a></h2>
<p>Like any technology, TypeScript does have some drawbacks. The main one for me is
that it adds complexity to a project. You need to set up a build system to
process TypeScript into JavaScript. I don’t think this is a problem once a
project reaches a certain size, but if I’m just writing a few small scripts for
a website I’ll use plain JavaScript.</p>
<p>A lot of people say they have issues with TypeScript types, but I’m not too
worried about this. The majority of my code is fairly straightforward type wise.
I have had more issues with Webpack and NPM modules than TypeScript itself.</p>
<p>For another opinion, you might be interested in reading
<a href="https://medium.com/javascript-scene/the-typescript-tax-132ff4cb175b"
rel="nofollow noreferrer noopener">
The TypeScript Tax</a>.</p>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>I’m not saying that TypeScript is a perfect language - there’s no such thing,
only different tradeoffs. Whether or not TypeScript is worth it depends on your
project and priorities.</p>
<p>In my opinion, TypeScript is great when you’re working with web technology and
would otherwise be using JavaScript. It improves the developer experience and
increases resilience without a huge amount of cost. I find it enjoyable to use
and will continue using it for the foreseeable future.</p>
rubenwardy
My favourite articles of 2023
2024-01-28T00:00:00.000Z
2024-01-28T00:00:00.000Z
https://blog.rubenwardy.com/2024/01/28/interesting-reads-2023/
Articles are a great way of keeping up with programming and technology. Here are some articles I read in 2023 that I recommend reading.
<p>I read a lot of articles about programming and technology as a way of keeping up
with the industry and expanding my knowledge. Here are some articles I read in
2023 that I recommend reading.</p>
<h2 id="tech-industry" tabindex="-1">Tech industry <a class="header-anchor" href="#tech-industry">#</a></h2>
<h3 id="tweeting-for-10-000-years-ended-after-4-years-8-months" tabindex="-1">Tweeting for 10,000 years ended after 4 years, 8 months <a class="header-anchor" href="#tweeting-for-10-000-years-ended-after-4-years-8-months">#</a></h3>
<p>This article is an interesting read on software longevity, with a humorously
short outcome. It shows the risks of
<a href="/2023/10/10/hello-indieweb/">relying on private platforms</a> and shows how
fragile software is compared to other forms of engineering.</p>
<div class="col-md col-6 embed">
<a href="https://brandur.org/10000-years" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><picture><source type="image/webp" srcset="/static/img/isbaJpOY7r-375.webp 375w" sizes="375px"><img alt="Tweeting for 10,000 Years: An Experiment in Autonomous Software" loading="lazy" decoding="async" style="background-color: rgba(0, 0, 0, 1.0);" src="/static/img/isbaJpOY7r-375.png" width="375" height="188"></picture></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
Tweeting for 10,000 Years: An Experiment in Autonomous Software
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Brandur</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h3 id="stop-using-brave-browser" tabindex="-1">Stop using Brave browser <a class="header-anchor" href="#stop-using-brave-browser">#</a></h3>
<p>Here is a must-read for those considering using Brave. I support using whatever
software works best for you, but I consider Brave to be untrustworthy due to
the issues highlighted in this article.</p>
<div class="col-md col-6 embed">
<a href="https://www.spacebar.news/p/stop-using-brave-browser" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><picture><source type="image/webp" srcset="/static/img/kiOlIfWAvs-375.webp 375w" sizes="375px"><img alt="Stop using Brave Browser" loading="lazy" decoding="async" style="background-color: rgba(0, 0, 0, 1.0);" src="/static/img/kiOlIfWAvs-375.png" width="375" height="188"></picture></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
Stop using Brave Browser
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Corbin Davenport</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h2 id="programming" tabindex="-1">Programming <a class="header-anchor" href="#programming">#</a></h2>
<h3 id="selling-your-extension" tabindex="-1">Selling your extension <a class="header-anchor" href="#selling-your-extension">#</a></h3>
<p>A couple of years ago, I released a browser extension called
<a href="https://renewedtab.com">Renewed Tab</a> to the Firefox and Chrome stores. Since
then, I’ve been getting a lot of emails about selling my extension or adding
spyware to it. These spammers all have the same goal - to steal the search and
browsing data of my users, and sell it on to data brokers.</p>
<p>I would never sell out my users, but not every browser extension developer is
the same. My extension only had a few hundred installs when I first started
receiving emails, so I’m sure the vast majority of extension developers will be
receiving these “offers”.</p>
<div class="col-md col-6 embed">
<a href="https://ericwbailey.website/published/re-selling-your-extension/" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><picture><source type="image/webp" srcset="/static/img/NPgPbO0vLl-375.webp 375w" sizes="375px"><img alt="RE: Selling your extension" loading="lazy" decoding="async" style="background-color: rgba(0, 0, 0, 1.0);" src="/static/img/NPgPbO0vLl-375.png" width="375" height="197"></picture></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
RE: Selling your extension
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Eric Bailey</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h3 id="self-closing-tags-img" tabindex="-1">Self-closing tags (<img />) <a class="header-anchor" href="#self-closing-tags-img">#</a></h3>
<p>I found this article interesting as it taught me something new about HTML.</p>
<div class="col-md col-6 embed">
<a href="https://jakearchibald.com/2023/against-self-closing-tags-in-html/" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><picture><source type="image/webp" srcset="/static/img/hBW5rBCaqk-375.webp 375w" sizes="375px"><img alt="The case against self-closing tags in HTML" loading="lazy" decoding="async" style="background-color: rgba(0, 0, 0, 1.0);" src="/static/img/hBW5rBCaqk-375.png" width="375" height="195"></picture></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
The case against self-closing tags in HTML
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Jake Archibald</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h3 id="what-they-don-t-tell-you-when-you-translate-your-app" tabindex="-1">What they don’t tell you when you translate your app <a class="header-anchor" href="#what-they-don-t-tell-you-when-you-translate-your-app">#</a></h3>
<p>Software internationalisation is super important for the accessibility and reach
of your software. Even though I do not speak multiple languages, I try to
support translation in my applications.</p>
<div class="col-md col-6 embed">
<a href="https://ericwbailey.website/published/what-they-dont-tell-you-when-you-translate-your-app/" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><picture><source type="image/webp" srcset="/static/img/yS75Z6WBob-375.webp 375w" sizes="375px"><img alt="What they don't tell you when you translate your app" loading="lazy" decoding="async" style="background-color: rgba(0, 0, 0, 1.0);" src="/static/img/yS75Z6WBob-375.png" width="375" height="197"></picture></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
What they don't tell you when you translate your app
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Eric Bailey</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h2 id="work" tabindex="-1">Work <a class="header-anchor" href="#work">#</a></h2>
<h3 id="on-consultancies" tabindex="-1">On consultancies <a class="header-anchor" href="#on-consultancies">#</a></h3>
<p>I’ve been working at consultancies and software houses since I graduated.
Whilst I haven’t necessarily experienced all these things, I found this article
to be an interesting read.</p>
<div class="col-md col-6 embed">
<a href="https://underscorehao.net/programmers-limbo/on-consultancies/" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
On Consultancies
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Pavel Danov</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h2 id="blogging" tabindex="-1">Blogging <a class="header-anchor" href="#blogging">#</a></h2>
<h3 id="seven-reasons-why-i-don-t-write" tabindex="-1">Seven reasons why I don’t write <a class="header-anchor" href="#seven-reasons-why-i-don-t-write">#</a></h3>
<p>I found this article quite influential in changing how I think about blogging.
I try to write from my perspective and experiences, without worrying if it’s
been said before. I also try to write what I want to write, rather than trying
to go viral.</p>
<div class="col-md col-6 embed">
<a href="https://mxb.dev/blog/seven-reasons-why-i-dont-write/" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><picture><source type="image/webp" srcset="/static/img/1o-2IGM5Y3-375.webp 375w" sizes="375px"><img alt="7 reasons why I don't write" loading="lazy" decoding="async" style="background-color: rgba(0, 0, 0, 1.0);" src="/static/img/1o-2IGM5Y3-375.png" width="375" height="238"></picture></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
7 reasons why I don't write
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Max Böck</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h3 id="what-happened-to-blogging-for-the-hell-of-it" tabindex="-1">What happened to blogging for the hell of it <a class="header-anchor" href="#what-happened-to-blogging-for-the-hell-of-it">#</a></h3>
<p>The Internet sometimes feels like one grift after another. This article presents
an argument against that. You don’t need to blog to boost vanity metrics
or to increase your value to employers. Just blog for the hell of it.</p>
<div class="col-md col-6 embed">
<a href="https://whiona.weblog.lol/2023/10/what-happened-to-blogging-for-the-hell-of-it" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
What happened to blogging for the hell of it?
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Whiona</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h2 id="misc" tabindex="-1">Misc <a class="header-anchor" href="#misc">#</a></h2>
<h3 id="mechanical-watch" tabindex="-1">Mechanical Watch <a class="header-anchor" href="#mechanical-watch">#</a></h3>
<p>I love explorables - interactive articles that explain how something works.
Here is a great one about how mechanical watches work.</p>
<div class="col-md col-6 embed">
<a href="https://ciechanow.ski/mechanical-watch/" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><picture><source type="image/webp" srcset="/static/img/CSv38_jp_I-375.webp 375w" sizes="375px"><img alt="Mechanical Watch" loading="lazy" decoding="async" style="background-color: rgba(0, 0, 0, 1.0);" src="/static/img/CSv38_jp_I-375.png" width="375" height="197"></picture></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
Mechanical Watch
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Bartosz Ciechanowski</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>I hope you found some articles worth reading. What were your favourite reads
last year? Feel free to send article suggestions you think I might like.</p>
<p><a href="https://www.pexels.com/photo/clear-glass-teacup-with-coffee-beverage-2299028/">Cover Image</a>
© Engin Akyurt</p>
rubenwardy
Five reasons why I write
2023-11-18T00:00:00.000Z
2023-11-18T00:00:00.000Z
https://blog.rubenwardy.com/2023/11/18/why-i-write/
In 2022, I made a goal of writing at least twelve long-form articles on my blog each year. But why bother? In this article, I will explain why I write.
<p>In 2022, I made a <a href="/stats/#goal">goal</a> of writing at least twelve long-form
articles on my blog each year. But why bother? In this article, I will explain
the reasons why I write.</p>
<h2 id="to-get-better-at-writing" tabindex="-1">To get better at writing <a class="header-anchor" href="#to-get-better-at-writing">#</a></h2>
<p>I write to improve my technical writing. Technical writing is a very useful
skill as a software engineer; communicating complex concepts is an essential
part of my day job. Writing articles regularly is a great way to practice my
planning and writing skills.</p>
<p>I’ve gotten pretty good at the <em>process</em> of writing articles, but I’m not sure
whether I’m actually getting better at writing. To get better at something, you
need timely and relevant feedback<sup class="footnote-ref"><a href="#fn1" id="fnref1"><span class="sr-only">footnote </span>1</a></sup>. I don’t have an editor to point
out issues and readers don’t tend to comment on writing quality unless it is
awful. It’s hard to improve at writing without feedback. The type of writing I
do also isn’t especially ambitious; perhaps creative writing or poetry would
provide better practice, although it’s not really what I want to be doing.</p>
<p>For the last ten years, I’ve been working on an online book to teach Minetest
modding and game creation called the
<a href="https://rubenwardy.com/minetest_modding_book/">Minetest Modding Book</a>. One day,
I’d like to self-publish a physical book on this subject. This will be a huge
undertaking - and you can’t remotely patch books once they are printed. Getting
better at writing helps towards this goal and will make the end result better.</p>
<h2 id="to-document-my-projects" tabindex="-1">To document my projects <a class="header-anchor" href="#to-document-my-projects">#</a></h2>
<p>I learnt to program because I wanted to create things. Websites, games,
world-ending artificial intelligences, and apps. I’m the sort of person who has
a lot of projects on the go at any one time.</p>
<p>Writing about my <a href="/tags/projects/">projects</a> is a way of sharing them with the
world. Whilst I do publish many of my projects as open source, some are less
accessible. For example, you can’t use my <a href="/tags/electronics/">electronics</a> or
<a href="/tags/3d_printing/">3D printing</a> projects unless you have electronics skills
and a 3D printer. It would also be quite time-consuming to replicate these
projects.</p>
<p>Writing whilst my memory is fresh allows me to archive my thoughts about a
project for the future, and share the lessons I learned. This is quite useful as
something to refer back to.</p>
<h2 id="to-get-feedback-and-learn" tabindex="-1">To get feedback and learn <a class="header-anchor" href="#to-get-feedback-and-learn">#</a></h2>
<p>Recently, I’ve been learning electronics. My eventual goal with electronics is
to create autonomous robots and drones; I’d like to make a quadcopter with my
own flight controller that can take off, land, and follow a target. This will be
quite an ambitious project, and I’m nowhere near capable enough for that yet.</p>
<p>To get better at electronics, I’ve been working on
<a href="/tags/electronics/">easier projects</a> to develop the skills I will need. Writing
about these projects and the problems I’ve faced has allowed me to get
suggestions from those experienced with electronics. This has been invaluable
for my self-development.</p>
<p>You don’t know what you don’t know. Even if I have solved all the problems and
finished an electronics project, I still benefit from suggestions for
improvements and better ways of doing things.</p>
<h2 id="to-share-knowledge-and-solutions" tabindex="-1">To share knowledge and solutions <a class="header-anchor" href="#to-share-knowledge-and-solutions">#</a></h2>
<p>There’s been a lot of cases where I’ve had to work out how to do something
myself. By publishing <a href="/tags/tutorials/">tutorials</a>, I can hopefully save the
next searcher from struggling with the same problem.</p>
<figure class="undefined"><a href="/static/img/y0GvM1uTpI-485.png"><picture><source type="image/webp" srcset="/static/img/-CauQE5j61-485.webp 485w"><img alt="Web comic.
The left is text that says:
Never have I felt so close to another soul, and yet so helplessly alone, as when I google an error, and there's one result. A thread by someone with the same problem and no answer. Last posted to in 2003.
The right half shows a stick figure shaking a monitor saying:
"Who were you DenverCoder9? WHAT DID YOU SEE?"" loading="lazy" decoding="async" title="All long help threads should have a sticky globally-editable post at the top saying 'DEAR PEOPLE FROM THE FUTURE: Here's what we've figured out so far ...'" src="/static/img/-CauQE5j61-485.png" width="485" height="270"></picture></a><figcaption>A relevant <a href="https://xkcd.com/979/">XKCD</a>: "Wisdom of the Ancients"</figcaption></figure>
<h2 id="to-think-and-explore-opinions" tabindex="-1">To think and explore opinions <a class="header-anchor" href="#to-think-and-explore-opinions">#</a></h2>
<p>Writing blog posts is a great way to develop opinions about subjects. As part of
my writing process, I break down concepts, make notes, and figure out which
points are the most important. By attempting to explain something clearly, you
can often identify weaknesses in your arguments or knowledge. Writing articles
allows me to brainstorm, weigh up my thoughts, and build/challenge opinions.</p>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>I write to get better at writing, to document my projects, to get feedback, to
share knowledge, and to develop opinions.</p>
<hr class="footnotes-sep">
<section class="footnotes">
<ol class="footnotes-list">
<li id="fn1" class="footnote-item"><p><a href="https://www.youtube.com/watch?v=5eW6Eagr9XA">The 4 things it takes to be an expert</a> by
Veritasium is a good explainer on how to get better at something. <a href="#fnref1" class="footnote-backref">↩︎</a></p>
</li>
</ol>
</section>
rubenwardy
Better word counts and reading time in Eleventy (11ty)
2023-10-29T00:00:00.000Z
2023-10-29T00:00:00.000Z
https://blog.rubenwardy.com/2023/10/29/eleventy-wordcount/
A short tutorial on how to count words and determine the reading time using Eleventy.
<p>I recently <a href="/2023/10/27/switched-to-eleventy/">switched to using Eleventy</a> to
generate my blog. As part of this process, I needed to implement word counts for
each of my posts. I made my own plugin for this as I was unable to find an
existing one that met my requirements - they either used regex to parse HTML or
included non-text like scripts and code snippets in the count.</p>
<p>The below code should work with any template engine, including Liquid and
Nunjucks.</p>
<h2 id="what-exactly-is-a-word" tabindex="-1">What exactly is a word? <a class="header-anchor" href="#what-exactly-is-a-word">#</a></h2>
<p>First of all, we need to work out exactly how we are going to count words.</p>
<p>A word is a unit of prose or writing. We need to make sure we exclude code
snippets and scripts from the word count.</p>
<p>A naive approach would be to split text by spaces and count all the parts.
However, this misses words that are joined together like “Dog/Cat”. Another
approch would be to split text by all punctuation, but this would count words
like “self-hosting” as two words.</p>
<p>To validate my word counter, I made a page where I dumped all the detected words
from a post. I used this to refine the list of delimators.</p>
<h2 id="counting-words" tabindex="-1">Counting words <a class="header-anchor" href="#counting-words">#</a></h2>
<h3 id="dependencies" tabindex="-1">Dependencies <a class="header-anchor" href="#dependencies">#</a></h3>
<p>You need to install JSDom:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">npm</span> <span class="token function">install</span> <span class="token parameter variable">--save</span> jsdom</code></pre>
<h3 id="eleventy-js" tabindex="-1">.eleventy.js <a class="header-anchor" href="#eleventy-js">#</a></h3>
<p>In the eleventy config, you need to add our new plugin:</p>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> pluginWordcount <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"./plugins/wordcount.js"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
module<span class="token punctuation">.</span><span class="token function-variable function">exports</span> <span class="token operator">=</span> <span class="token keyword">function</span><span class="token punctuation">(</span><span class="token parameter">eleventyConfig</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
eleventyConfig<span class="token punctuation">.</span><span class="token function">addPlugin</span><span class="token punctuation">(</span>pluginWordcount<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">// You can only have one module.exports in a configuration file,</span>
<span class="token comment">// so make sure you add the above line to your existing one.</span>
<span class="token punctuation">}</span></code></pre>
<h3 id="wordcount-js-plugin" tabindex="-1">wordcount.js plugin <a class="header-anchor" href="#wordcount-js-plugin">#</a></h3>
<p>This is the file for the plugin. It contains <code>extractText</code> to get all the text
from HTML, and countWords to count the words in a piece of text.</p>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> <span class="token punctuation">{</span> UserConfig <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"@11ty/eleventy"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> <span class="token punctuation">{</span> <span class="token constant">JSDOM</span> <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"jsdom"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> <span class="token constant">TO_STRIP</span> <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token string">"code"</span><span class="token punctuation">,</span>
<span class="token string">"pre code"</span><span class="token punctuation">,</span>
<span class="token string">"script"</span><span class="token punctuation">,</span>
<span class="token string">".header-anchor"</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">;</span>
<span class="token keyword">function</span> <span class="token function">extractText</span><span class="token punctuation">(</span><span class="token parameter">html</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> dom <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">JSDOM</span><span class="token punctuation">(</span>html<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> document <span class="token operator">=</span> dom<span class="token punctuation">.</span>window<span class="token punctuation">.</span>document<span class="token punctuation">;</span>
<span class="token comment">// Remove non-text elements</span>
document<span class="token punctuation">.</span><span class="token function">querySelectorAll</span><span class="token punctuation">(</span><span class="token constant">TO_STRIP</span><span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span><span class="token string">", "</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token parameter">child</span> <span class="token operator">=></span> child<span class="token punctuation">.</span><span class="token function">remove</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">return</span> document<span class="token punctuation">.</span>body<span class="token punctuation">.</span>textContent<span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">const</span> cache <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">;</span>
<span class="token keyword">function</span> <span class="token function">countWords</span><span class="token punctuation">(</span><span class="token parameter">value</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>cache<span class="token punctuation">[</span>value<span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> cache<span class="token punctuation">[</span>value<span class="token punctuation">]</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">const</span> result <span class="token operator">=</span> <span class="token function">extractText</span><span class="token punctuation">(</span>value<span class="token punctuation">)</span>
<span class="token punctuation">.</span><span class="token function">split</span><span class="token punctuation">(</span><span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">[\s;/\\]</span><span class="token regex-delimiter">/</span></span><span class="token punctuation">)</span>
<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">x</span> <span class="token operator">=></span> x<span class="token punctuation">.</span><span class="token function">trim</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment">// Word is non-empty with at least one letter or number</span>
<span class="token punctuation">.</span><span class="token function">filter</span><span class="token punctuation">(</span><span class="token parameter">x</span> <span class="token operator">=></span> x<span class="token punctuation">.</span><span class="token function">match</span><span class="token punctuation">(</span><span class="token regex"><span class="token regex-delimiter">/</span><span class="token regex-source language-regex">.*[a-z0-9].*</span><span class="token regex-delimiter">/</span><span class="token regex-flags">i</span></span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token punctuation">.</span>length<span class="token punctuation">;</span>
cache<span class="token punctuation">[</span>value<span class="token punctuation">]</span> <span class="token operator">=</span> result<span class="token punctuation">;</span>
<span class="token keyword">return</span> result<span class="token punctuation">;</span>
<span class="token punctuation">}</span>
module<span class="token punctuation">.</span><span class="token function-variable function">exports</span> <span class="token operator">=</span> <span class="token parameter">eleventyConfig</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
eleventyConfig<span class="token punctuation">.</span><span class="token function">addFilter</span><span class="token punctuation">(</span><span class="token string">"wordcount"</span><span class="token punctuation">,</span> countWords<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre>
<h3 id="inside-post-layout" tabindex="-1">Inside post layout <a class="header-anchor" href="#inside-post-layout">#</a></h3>
<p>Here’s how you might use the wordcount filter inside a post layout that uses
liquid:</p>
<pre class="language-liquid"><code class="language-liquid"><span class="token liquid language-liquid"><span class="token delimiter punctuation">{%</span> <span class="token keyword">assign</span> wordcount <span class="token operator">=</span> content <span class="token operator">|</span> <span class="token function filter">wordcount</span> <span class="token delimiter punctuation">%}</span></span>
<span class="token liquid language-liquid"><span class="token delimiter punctuation">{{</span> wordcount <span class="token operator">|</span> <span class="token function filter">divided_by</span><span class="token operator">:</span> <span class="token number">238</span> <span class="token operator">|</span> <span class="token function filter">round</span> <span class="token delimiter punctuation">}}</span></span> min read
(<span class="token liquid language-liquid"><span class="token delimiter punctuation">{{</span> wordcount <span class="token delimiter punctuation">}}</span></span> words)</code></pre>
<h3 id="unit-tests" tabindex="-1">Unit tests <a class="header-anchor" href="#unit-tests">#</a></h3>
<p>As a bonus, here’s the unit tests I used when writing the word counter:</p>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> <span class="token punctuation">{</span> describe <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"mocha"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> <span class="token punctuation">{</span> expect <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"chai"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> <span class="token punctuation">{</span> count <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"./wordcount"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> parameterisedTests <span class="token operator">=</span> <span class="token punctuation">{</span>
<span class="token string-property property">"empty"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">html</span><span class="token operator">:</span> <span class="token string">""</span><span class="token punctuation">,</span>
<span class="token literal-property property">expected</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token string-property property">"just symbols"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">html</span><span class="token operator">:</span> <span class="token string">". . -//!\"$%^&*()\\`"</span><span class="token punctuation">,</span>
<span class="token literal-property property">expected</span><span class="token operator">:</span> <span class="token number">0</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token string-property property">"single word paragraph"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">html</span><span class="token operator">:</span> <span class="token string">"<p> Hey! </p>"</span><span class="token punctuation">,</span>
<span class="token literal-property property">expected</span><span class="token operator">:</span> <span class="token number">1</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token string-property property">"punctuation"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">html</span><span class="token operator">:</span> <span class="token string">"<p>Hello world! This is a test, of the word/counter</p>"</span><span class="token punctuation">,</span>
<span class="token literal-property property">expected</span><span class="token operator">:</span> <span class="token number">10</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token string-property property">"strips scripts"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">html</span><span class="token operator">:</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">
<p>Hello world! This is a test, of the word/counter</p>
<script>
alert("Hello world!")
</script>
</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">expected</span><span class="token operator">:</span> <span class="token number">10</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token string-property property">"strips code blocks"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">html</span><span class="token operator">:</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">
<p>Hello world! This is a test, of the word/counter</p>
<pre>
<code>
alert("Hello world!")
</code>
</pre>
</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">expected</span><span class="token operator">:</span> <span class="token number">10</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token string-property property">"strips inline code"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">html</span><span class="token operator">:</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">
<p>Hello world! This is a <code>test</code>, of the word/counter</p>
</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">expected</span><span class="token operator">:</span> <span class="token number">9</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token string-property property">"strips heading anchors"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">html</span><span class="token operator">:</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">
<h2>A heading</h2>
<a class="header-anchor">1</a>
<p>Hello world! One two</p>
</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">expected</span><span class="token operator">:</span> <span class="token number">6</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token string-property property">"counts numbers but not symbols"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">html</span><span class="token operator">:</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">
<p>Hello world! You are 26.0 today - or so</p>
</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">expected</span><span class="token operator">:</span> <span class="token number">8</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token string-property property">"words can contain hyphens"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token literal-property property">html</span><span class="token operator">:</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">
<p>Hello world! One-two three</p>
</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span>
<span class="token literal-property property">expected</span><span class="token operator">:</span> <span class="token number">4</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span>
<span class="token function">describe</span><span class="token punctuation">(</span><span class="token string">"countWords"</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
Object<span class="token punctuation">.</span><span class="token function">entries</span><span class="token punctuation">(</span>parameterisedTests<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter"><span class="token punctuation">[</span>key<span class="token punctuation">,</span> data<span class="token punctuation">]</span></span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token function">it</span><span class="token punctuation">(</span>key<span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token function">expect</span><span class="token punctuation">(</span><span class="token function">count</span><span class="token punctuation">(</span>data<span class="token punctuation">.</span>html<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span>to<span class="token punctuation">.</span><span class="token function">equal</span><span class="token punctuation">(</span>data<span class="token punctuation">.</span>expected<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
rubenwardy
How to find backlinks using Eleventy (11ty)
2023-10-29T00:00:00.000Z
2023-10-29T00:00:00.000Z
https://blog.rubenwardy.com/2023/10/29/eleventy-backlinks/
A short tutorial on how to find backlinks (pages that link to the current page) using Eleventy
<p>I recently <a href="/2023/10/27/switched-to-eleventy/">switched to using Eleventy</a> to
generate my blog. Because Eleventy is very easy to extend, I was able to add
more features to my blog - including backlinks.</p>
<p>In a post’s sidebar, there is a list of other posts that link to the current
post. This is an effective way to allow readers to find related content.</p>
<p>To implement this, I created a small plugin that adds a filter to find which
items in a collection link to the given URL. To find links in HTML, I used the
JSDom library.</p>
<p>Whilst there are existing backlink plugins, they didn’t meet my purposes.
<code>eleventy-plugin-backlinks</code>, for example, only finds links made using
wikilinks-style markup (ie: <code>[[Other Post Name]])</code>). I wanted backlinks to
work with any link in a post - whether the post is markdown, HTML, or something
else.</p>
<p>The below code should work with any template engine, including Liquid and
Nunjucks.</p>
<h2 id="finding-backlinks" tabindex="-1">Finding Backlinks <a class="header-anchor" href="#finding-backlinks">#</a></h2>
<h3 id="dependencies" tabindex="-1">Dependencies <a class="header-anchor" href="#dependencies">#</a></h3>
<p>You need to install JSDom:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">npm</span> <span class="token function">install</span> <span class="token parameter variable">--save</span> jsdom</code></pre>
<h3 id="eleventy-js" tabindex="-1">.eleventy.js <a class="header-anchor" href="#eleventy-js">#</a></h3>
<p>In the eleventy config, you need to add our new plugin:</p>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> pluginBacklinks <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"./plugins/backlinks.js"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
module<span class="token punctuation">.</span><span class="token function-variable function">exports</span> <span class="token operator">=</span> <span class="token keyword">function</span><span class="token punctuation">(</span><span class="token parameter">eleventyConfig</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
eleventyConfig<span class="token punctuation">.</span><span class="token function">addPlugin</span><span class="token punctuation">(</span>pluginBacklinks<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">// You can only have one module.exports in a configuration file,</span>
<span class="token comment">// so make sure you add the above line to your existing one.</span>
<span class="token punctuation">}</span></code></pre>
<h3 id="backlinks-js-plugin" tabindex="-1">backlinks.js plugin <a class="header-anchor" href="#backlinks-js-plugin">#</a></h3>
<p>This is the file for the plugin. It contains <code>getLinks</code> to extract links from
HTML and a plugin function to register a filter.</p>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> <span class="token punctuation">{</span> UserConfig <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"@11ty/eleventy"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> <span class="token punctuation">{</span> <span class="token constant">JSDOM</span> <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"jsdom"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> hostname <span class="token operator">=</span> <span class="token string">"blog.rubenwardy.com"</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> cache <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">;</span>
<span class="token comment">/**
* Extract links from html, not including hash parts
*/</span>
<span class="token keyword">function</span> <span class="token function">getLinks</span><span class="token punctuation">(</span><span class="token parameter">html</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>cache<span class="token punctuation">[</span>html<span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> cache<span class="token punctuation">[</span>html<span class="token punctuation">]</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">const</span> dom <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">JSDOM</span><span class="token punctuation">(</span>html<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> document <span class="token operator">=</span> dom<span class="token punctuation">.</span>window<span class="token punctuation">.</span>document<span class="token punctuation">;</span>
<span class="token keyword">const</span> result <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Set</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token operator">...</span>document<span class="token punctuation">.</span><span class="token function">querySelectorAll</span><span class="token punctuation">(</span><span class="token string">"a[href]"</span><span class="token punctuation">)</span><span class="token punctuation">]</span>
<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">x</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token keyword">let</span> href <span class="token operator">=</span> x<span class="token punctuation">.</span><span class="token function">getAttribute</span><span class="token punctuation">(</span><span class="token string">"href"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">// Normalise internal links</span>
<span class="token keyword">const</span> url <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">URL</span><span class="token punctuation">(</span>href<span class="token punctuation">,</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">https://</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>hostname<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>url<span class="token punctuation">.</span>hostname <span class="token operator">==</span> hostname<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> url<span class="token punctuation">.</span>pathname<span class="token punctuation">;</span>
<span class="token punctuation">}</span>
url<span class="token punctuation">.</span>hash <span class="token operator">=</span> <span class="token string">""</span><span class="token punctuation">;</span>
<span class="token keyword">return</span> url<span class="token punctuation">.</span><span class="token function">toString</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
cache<span class="token punctuation">[</span>html<span class="token punctuation">]</span> <span class="token operator">=</span> result<span class="token punctuation">;</span>
<span class="token keyword">return</span> result<span class="token punctuation">;</span>
<span class="token punctuation">}</span>
module<span class="token punctuation">.</span><span class="token function-variable function">exports</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token parameter">eleventyConfig</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
eleventyConfig<span class="token punctuation">.</span><span class="token function">addFilter</span><span class="token punctuation">(</span><span class="token string">"links_to"</span><span class="token punctuation">,</span> <span class="token keyword">async</span> <span class="token keyword">function</span><span class="token punctuation">(</span><span class="token parameter">collection<span class="token punctuation">,</span> target</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> collection<span class="token punctuation">.</span><span class="token function">filter</span><span class="token punctuation">(</span><span class="token parameter">item</span> <span class="token operator">=></span> <span class="token function">getLinks</span><span class="token punctuation">(</span>item<span class="token punctuation">.</span>content<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">has</span><span class="token punctuation">(</span>target<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre>
<h3 id="inside-post-layout" tabindex="-1">Inside post layout <a class="header-anchor" href="#inside-post-layout">#</a></h3>
<p>This is how you use the <code>links_to</code> filter to get backlinks inside a post layout
that uses liquid:</p>
<pre class="language-liquid"><code class="language-liquid"><span class="token liquid language-liquid"><span class="token delimiter punctuation">{%</span> <span class="token keyword">assign</span> backlinks <span class="token operator">=</span> collections<span class="token punctuation">.</span>post <span class="token operator">|</span> <span class="token function filter">links_to</span><span class="token operator">:</span> <span class="token object">page</span><span class="token punctuation">.</span>url <span class="token delimiter punctuation">%}</span></span>
<span class="token comment"><!-- An empty list isn't false-y in Eleventy liquid --></span>
<span class="token liquid language-liquid"><span class="token delimiter punctuation">{%</span> <span class="token keyword">assign</span> backlinks_count <span class="token operator">=</span> backlinks <span class="token operator">|</span> <span class="token function filter">size</span> <span class="token delimiter punctuation">%}</span></span>
<span class="token liquid language-liquid"><span class="token delimiter punctuation">{%</span> <span class="token keyword">if</span> backlinks_count <span class="token operator">></span> <span class="token number">0</span> <span class="token delimiter punctuation">%}</span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>aside</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>backlinks<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>h3</span><span class="token punctuation">></span></span>Links here<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>h3</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>ul</span><span class="token punctuation">></span></span>
<span class="token liquid language-liquid"><span class="token delimiter punctuation">{%-</span> <span class="token keyword">for</span> post <span class="token keyword">in</span> backlinks <span class="token delimiter punctuation">-%}</span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>li</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>a</span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span><span class="token liquid language-liquid"><span class="token delimiter punctuation">{{</span> post<span class="token punctuation">.</span>url <span class="token delimiter punctuation">}}</span></span><span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token liquid language-liquid"><span class="token delimiter punctuation">{{</span> post<span class="token punctuation">.</span>data<span class="token punctuation">.</span>title <span class="token delimiter punctuation">}}</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>a</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>li</span><span class="token punctuation">></span></span>
<span class="token liquid language-liquid"><span class="token delimiter punctuation">{%-</span> <span class="token keyword">endfor</span> <span class="token delimiter punctuation">-%}</span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>div</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>aside</span><span class="token punctuation">></span></span>
<span class="token liquid language-liquid"><span class="token delimiter punctuation">{%</span> <span class="token keyword">endif</span> <span class="token delimiter punctuation">%}</span></span></code></pre>
rubenwardy
I've switched to Eleventy (11ty): my thoughts
2023-10-27T00:00:00.000Z
2023-10-27T00:00:00.000Z
https://blog.rubenwardy.com/2023/10/27/switched-to-eleventy/
In this article, I will talk about why I switched, how I did it, and my thoughts about Eleventy as a new user.
<p>After nine years of using Jekyll, I’ve switched to using
<a href="https://www.11ty.dev/">Eleventy</a> (11ty) and redesigned my blog whilst I was at
it. I was finding that Jekyll was too limiting and wanted something more
capable. In this article, I will talk about why I switched, how I did it, and my
thoughts about Eleventy as a new user.</p>
<h2 id="jekyll-and-why-i-decided-to-switch" tabindex="-1">Jekyll, and why I decided to switch <a class="header-anchor" href="#jekyll-and-why-i-decided-to-switch">#</a></h2>
<p>I chose to use Jekyll in 2014 because it was the only static site generator that
GitHub Pages supported. It allowed me to use multiple pages without having
to duplicate all the layout markup, making it easier to grow the size of my
website.</p>
<p>Recently, I’ve been finding Jekyll increasingly painful to use and too limited
for my purposes. Since 2022, my goal has been to write at least twelve posts a
year. This means that I’ve been working on posts much more frequently than I
have done in the past. Ease of writing posts is much more important now.</p>
<p>One of my biggest annoyances with writing a post was creating thumbnails for my
images - this was quite time-consuming; I wanted to fully automate it in a
manner that I don’t even need to think about when writing a post.</p>
<p>Another thing was that the files were split into different places - the text was
placed in <code>_posts</code> and the images were in <code>static</code>. I wanted the ability to
make a single folder for an article and get the generator to automatically move
the assets to where they are needed.</p>
<figure class="mx-0">
<div class="row row-wrap gap-6">
<div class="col-md">
<pre><code>📁 static
├─ 📁 hello-indieweb
| └─ 🖼️ cover.png
└─ 📁 switched-to-eleventy
├─ 🖼️ cover.png
└─ 🖼️ schematic.png
📁 _posts
├─ 📄 2023-10-10-hello-indieweb.md
└─ 📄 2023-10-27-switched-to-eleventy.md</code></pre>
</div>
<div class="col-md">
<pre><code>📁 posts
├─ 📁 2023-10-10-hello-indieweb
| ├─ 📄 index.md
| └─ 🖼️ cover.png
└─ 📁 2023-10-27-switched-to-eleventy
├─ 📄 index.md
├─ 🖼️ cover.png
└─ 🖼️ schematic.png
</code></pre>
</div>
</div>
<figcaption class="text-center mt-4">Which would you prefer?</figcaption>
</figure>
<p>Whilst I could have hacked together more scripts to fix these issues, my setup
was already quite convoluted. To generate my <a href="/stats/">writing statistics page</a>,
I had a mess of Python scripts and complicated layout code. Liquid isn’t the
most capable language to use by itself. So, I decided to look for other static
site generators.</p>
<h2 id="my-requirements" tabindex="-1">My requirements <a class="header-anchor" href="#my-requirements">#</a></h2>
<p>My first requirement was that it must be a static site generator. A static site
is the easiest type of website to host - it can be hosted on any web host, even
a Content Delivery Network (CDN). This gives a lot of flexibility. It also
increases longevity - if, for whatever reason, I cease maintaining my website,
it can continue to be hosted as a low-tech static website for a long time. To
get dynamic features like commenting, I have
<a href="/2023/10/10/hello-indieweb/#comment-form">a NodeJS server</a> hosted on a separate
subdomain.</p>
<p>The second requirement was that it should not <em>require</em> any client-side
JavaScript. Requiring JavaScript is bad for SEO and accessibility and bloats the
webpage. Client-side JavaScript should only be used to
<a href="https://developer.mozilla.org/en-US/docs/Glossary/Progressive_Enhancement">enhance</a>
the behaviour.</p>
<p>The third requirement was that the generator must be easy to extend and
customise to match my workflow. I’m not fond of Ruby or Bundler; I found
Bundler quite painful to use. I felt like JavaScript/TypeScript would be the
better language to use as it is a web technology.</p>
<p>The final requirement is the generator must be fully open source. Using
open-source tools is vital to achieve flexibility and longevity.</p>
<h2 id="switching-to-eleventy" tabindex="-1">Switching to Eleventy <a class="header-anchor" href="#switching-to-eleventy">#</a></h2>
<p>After looking into the options, I decided to give Eleventy a go. I started by
setting up an Eleventy project and porting a single blog post. To derisk the
process, I focused on implementing and prototyping the unknowns first, such as
thumbnails, SCSS, post/tags, and writing statistics. This allowed me to quickly
verify whether Eleventy was a good fit before spending a lot of effort porting
all my content across.</p>
<p>I also decided to use this opportunity to redesign my blog as well. I used to
use Bootstrap; I wanted to go with something a bit more custom and unique this
time. I also wanted a light/dark mode switcher.</p>
<p>Before I published the new Eleventy version to production, I wrote a simple
script to make sure that no URLs had changed or were missing.
<a href="https://www.w3.org/Provider/Style/URI">Cool URLs never change</a>, so I considered
verifying this important.</p>
<h2 id="the-good" tabindex="-1">The good <a class="header-anchor" href="#the-good">#</a></h2>
<h3 id="eleventy-image" tabindex="-1">Eleventy-Image <a class="header-anchor" href="#eleventy-image">#</a></h3>
<p><a href="https://www.11ty.dev/docs/plugins/image/">Eleventy-Image</a> is an official plugin
for Eleventy that allows you to resize images and create thumbnails.</p>
<p>Using Eleventy-Image, I defined custom template functions to create image
thumbnails based on how they are used. For example, the <code>figure</code> function
creates a 540px width thumbnail and then renders a <code><figure></code> element to the
HTML.</p>
<pre class="language-liquid"><code class="language-liquid"><span class="token liquid language-liquid"><span class="token delimiter punctuation">{%</span> figure <span class="token string">"./schematic.png"</span><span class="token punctuation">,</span>
<span class="token string">"The schematic for my plant watering system's circuit. Created using KiCAD."</span> <span class="token delimiter punctuation">%}</span></span></code></pre>
<p>The source image is stored in the post’s directory. The <code>figure</code> function saves
resized versions to the <code>static</code> directory. This allows me to keep the text and
images of a post together, and only create specific image sizes when needed.</p>
<h3 id="flexibility" tabindex="-1">Flexibility <a class="header-anchor" href="#flexibility">#</a></h3>
<p>Eleventy is very flexible, powerful, and fairly easy to customise. Using
JavaScript, you can define custom template filters and functions. You can even
build entire pages using JS - or JSX with an additional library.</p>
<p>It was incredibly easy to add features such as a table of contents, writing
statistics, and backlinks. For a table of contents, I used an existing plugin
and only needed to add <code>{{ toc }}</code> to place it. I
implemented writing statistics, <a href="/2023/10/29/eleventy-wordcount/">word counts</a>,
and <a href="/2023/10/29/eleventy-backlinks/">backlinks</a> by creating plugins to add
template functions using JavaScript.</p>
<pre class="language-js"><code class="language-js"><span class="token comment">// Usage:</span>
<span class="token comment">// {% assign backlinks = collections.post | links_to: page.url %}</span>
eleventyConfig<span class="token punctuation">.</span><span class="token function">addFilter</span><span class="token punctuation">(</span><span class="token string">"links_to"</span><span class="token punctuation">,</span> <span class="token keyword">async</span> <span class="token keyword">function</span><span class="token punctuation">(</span><span class="token parameter">posts<span class="token punctuation">,</span> targetURL</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> posts<span class="token punctuation">.</span><span class="token function">filter</span><span class="token punctuation">(</span><span class="token parameter">post</span> <span class="token operator">=></span> <span class="token function">getLinks</span><span class="token punctuation">(</span>post<span class="token punctuation">.</span>content<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">has</span><span class="token punctuation">(</span>targetURL<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<p>Another thing is that Eleventy allows you to customise or provide your own
Markdown parser, or use different formats entirely. To get heading anchors
(<code>#</code>), I just needed to enable that option.</p>
<p>Finally, Eleventy has support for JavaScript data files, which is great for
fetching data from the network before building the site. I used this feature to
load <a href="/2023/10/10/hello-indieweb/#webmentions">Webmentions</a> from the
Webmention.io REST API.</p>
<h2 id="the-bad" tabindex="-1">The bad <a class="header-anchor" href="#the-bad">#</a></h2>
<h3 id="learning-curve-and-inconsistencies" tabindex="-1">Learning curve and inconsistencies <a class="header-anchor" href="#learning-curve-and-inconsistencies">#</a></h3>
<p>Eleventy has quite a lot of quirks that make learning it quite difficult. It
took much longer than I expected to switch to Eleventy.</p>
<p>First, Eleventy uses variables inconsistently. At the top of each page, there’s
metadata known as front matter:</p>
<pre class="language-yml"><code class="language-yml"><span class="token punctuation">---</span>
<span class="token key atrule">title</span><span class="token punctuation">:</span> I've redesigned my blog and switched to Eleventy
<span class="token key atrule">description</span><span class="token punctuation">:</span> <span class="token string">""</span>
<span class="token key atrule">tags</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> fullstack
<span class="token punctuation">-</span> reviews
<span class="token punctuation">---</span></code></pre>
<p>To access the title and URL inside a template, you do <code>title</code> and <code>page.url</code>.
Inside JavaScript or when reading a collection, it’s <code>item.data.title</code>,
<code>item.url</code>, or <code>item.page.url</code>.</p>
<p>Having a lot of global variables inside the template is quite confusing. It’s
not just pages - Liquid allows you to include other liquid files. In Jekyll, you
access parameters by writing <code>include.param1</code>. In Eleventy, the parameter is
just <code>param1</code>.</p>
<p>The Eleventy liquid syntax is also slightly different from Jekyll liquid, for some
reason. To include a directory, you need to quote the file name and use <code>:</code>
instead of <code>=</code>:</p>
<pre class="language-liquid"><code class="language-liquid"><span class="token comment"><!-- Eleventy --></span>
<span class="token liquid language-liquid"><span class="token delimiter punctuation">{%</span> <span class="token keyword">include</span> <span class="token string">"youtube.html"</span><span class="token punctuation">,</span> id<span class="token operator">:</span> <span class="token string">"dQw4w9WgXcQ"</span><span class="token punctuation">,</span> caption<span class="token operator">:</span> <span class="token string">"Funny cat videos"</span> <span class="token delimiter punctuation">%}</span></span>
<span class="token comment"><!-- Jekyll --></span>
<span class="token liquid language-liquid"><span class="token delimiter punctuation">{%</span> <span class="token keyword">include</span> youtube<span class="token punctuation">.</span>html id<span class="token operator">=</span><span class="token string">"dQw4w9WgXcQ"</span> caption<span class="token operator">=</span><span class="token string">"Funny cat videos"</span> <span class="token delimiter punctuation">%}</span></span></code></pre>
<p>But custom shortcodes do not support named parameters, you can only pass them in
by order:</p>
<pre class="language-liquid"><code class="language-liquid"><span class="token liquid language-liquid"><span class="token delimiter punctuation">{%</span> youtube <span class="token string">"dQw4w9WgXcQ"</span><span class="token punctuation">,</span> <span class="token string">"Funny cat videos"</span> <span class="token delimiter punctuation">%}</span></span></code></pre>
<h3 id="painful-to-read-error-messages" tabindex="-1">Painful to read error messages <a class="header-anchor" href="#painful-to-read-error-messages">#</a></h3>
<p>The error messages produced by Eleventy are very long and <a href="https://github.com/11ty/eleventy/issues/3076">point to the wrong
lines</a> in templates. It looks like
the line numbers don’t include the height of the front matter.</p>
<pre><code>[11ty] Problem writing Eleventy templates: (more in DEBUG output)
[11ty] 1. Having trouble rendering liquid template ./file.md (via TemplateContentRenderError)
[11ty] 2. tag "doesnotexist" not found, file:./file.md, line:2, col:1 (via ParseError)
[11ty] 3. tag "doesnotexist" not found (via AssertionError)
[11ty]
[11ty] Original error stack trace: AssertionError: tag "doesnotexist" not found
[11ty] at assert (/home/ruben/dev/tmp/eleventy_repro/node_modules/liquidjs/dist/liquid.node.cjs.js:319:15)
[11ty] at Parser.parseToken (/home/ruben/dev/tmp/eleventy_repro/node_modules/liquidjs/dist/liquid.node.cjs.js:2284:17)
[11ty] at Parser.parseTokens (/home/ruben/dev/tmp/eleventy_repro/node_modules/liquidjs/dist/liquid.node.cjs.js:2276:33)
[11ty] at Parser.parse (/home/ruben/dev/tmp/eleventy_repro/node_modules/liquidjs/dist/liquid.node.cjs.js:2270:21)
[11ty] at Liquid.parse (/home/ruben/dev/tmp/eleventy_repro/node_modules/liquidjs/dist/liquid.node.cjs.js:3618:28)
[11ty] at Liquid.compile (/home/ruben/dev/tmp/eleventy_repro/node_modules/@11ty/eleventy/src/Engines/Liquid.js:255:28)
[11ty] at Markdown.compile (/home/ruben/dev/tmp/eleventy_repro/node_modules/@11ty/eleventy/src/Engines/Markdown.js:68:28)
[11ty] at TemplateRender.getCompiledTemplate (/home/ruben/dev/tmp/eleventy_repro/node_modules/@11ty/eleventy/src/TemplateRender.js:269:26)
[11ty] at Template.compile (/home/ruben/dev/tmp/eleventy_repro/node_modules/@11ty/eleventy/src/TemplateContent.js:362:42)
[11ty] at async Template._render (/home/ruben/dev/tmp/eleventy_repro/node_modules/@11ty/eleventy/src/TemplateContent.js:486:16)
[11ty] Wrote 0 files in 0.06 seconds (v2.0.1)
</code></pre>
<p>Eleventy would have been much easier to use if the error message was at the
bottom and it gave a link to the exact line in the template. Providing the
template path as <code>path/to/file.h:123</code> like the rest of the paths would also make
it clickable in IDEs.</p>
<h3 id="typing-and-ide-support" tabindex="-1">Typing and IDE support <a class="header-anchor" href="#typing-and-ide-support">#</a></h3>
<p>Another thing I didn’t like is that Eleventy does not support TypeScript. I like
TypeScript because it makes my IDE (VSCodium) more helpful. IntelliSense allows
me to inspect what methods are available on an object, and linting shows errors
when I use something incorrectly.</p>
<p>I tried setting up TypeScript with Eleventy, but there were no types available.
Eleventy does come with <code>index.d.ts</code>, but it only documents a single thing and
not the entire API.</p>
<p>One thing I did discover is that you can use JSDoc to get IntelliSense to work a
little bit. JSDoc does feel a bit like “TypeScript at home” though. It’s very
verbose and doesn’t work as well.</p>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> <span class="token punctuation">{</span> UserConfig <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">"@11ty/eleventy/src/UserConfig"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">/**
* @param {UserConfig} eleventyConfig
* @returns {void}
*/</span>
module<span class="token punctuation">.</span><span class="token function-variable function">exports</span> <span class="token operator">=</span> <span class="token keyword">function</span><span class="token punctuation">(</span><span class="token parameter">eleventyConfig</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token comment">// configure here</span>
<span class="token punctuation">}</span></code></pre>
<h3 id="old-style-imports-commonjs" tabindex="-1">Old-style imports: CommonJS <a class="header-anchor" href="#old-style-imports-commonjs">#</a></h3>
<p>As you may have seen in the above example, Eleventy uses CommonJS <code>require</code>
rather than the more modern ES6 <code>import</code>. This is not great, but will
<a href="https://github.com/11ty/eleventy/issues/836">be fixed</a> in Eleventy 3.0.0.</p>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>Whilst I’m not 100% happy with Eleventy, it is more capable than Jekyll and
better for my purposes. Many of the issues I have are fixable, and now that I
know how to use Eleventy, it should be easier to do things in the future.</p>
rubenwardy
I have joined the IndieWeb
2023-10-10T00:00:00.000Z
2023-10-10T00:00:00.000Z
https://blog.rubenwardy.com/2023/10/10/hello-indieweb/
The collapse of Twitter last year got me thinking about closed platforms and reducing the hold that privately-owned silos have over the Internet.
<p>The collapse of Twitter last year got me thinking about closed platforms and
reducing the hold that privately owned platforms have over the Internet.</p>
<p>I’ve been blogging for nine years now on my personal website. I like owning my
own domain as it allows me to retain control and stay independent of particular
services. Private platforms have a tendency to be bought out and/or ruined by
commercial interests, especially now with tech growth slowing down and investors
getting uneasy.</p>
<p>However, there are some benefits to closed blogging platforms. Medium provides a
network effect that small blogs don’t have. It has an algorithm that promotes
posts that users may find interesting. This allows the blogs to organically gain
new readers. Additionally, Medium makes it super easy to like, comment, and
reply to a post, resulting in a platform that feels a lot more like a social
network than your standard cloud blogging service.</p>
<p>Last year, I started looking into ways independent blogs could communicate, just
like on Medium. I considered making my blog ActivityPub-compatible, as that
would allow users on Mastodon and the Fediverse to like, share, and comment on
articles. And then, I stumbled upon the IndieWeb.</p>
<h2 id="what-is-indieweb" tabindex="-1">What is IndieWeb? <a class="header-anchor" href="#what-is-indieweb">#</a></h2>
<p>IndieWeb.org describes the IndieWeb as:</p>
<figure class="figure ">
<blockquote class="mb-1" cite="https://indieweb.org/"><p><strong>The IndieWeb is a community of independent & personal websites</strong> connected by simple standards, based on the principles of: owning your domain & using it as your primary identity, publishing on your own site (optionally syndicating elsewhere), and owning your data.</p>
</blockquote>
<figcaption class="pl-4 mt-3">
—<a href="https://indieweb.org/">
IndieWeb.org
</a></figcaption>
</figure>
<p>To phrase it another way, IndieWeb is about posting the things you make on your
personal website and domain, to keep control of your data and stay independent
from private platforms (aka silos). You may still post to silos but you should
post to your personal website first.</p>
<p>IndieWeb isn’t just about blogging. You might post Twitter-like microposts,
photos, location check-ins, reviews, replies to other sites, and more.</p>
<h3 id="posse-and-backfeeding" tabindex="-1">POSSE and Backfeeding <a class="header-anchor" href="#posse-and-backfeeding">#</a></h3>
<p>An important concept is “<strong>P</strong>ublish on your <strong>O</strong>wn <strong>S</strong>ite; <strong>S</strong>yndicate
<strong>E</strong>lsewhere” (<a href="https://indieweb.org/POSSE">POSSE</a>). This means that you should
post the original version on your own website and then share links or copies of
your content with relevant social media communities. This is simpler and more
flexible than adding ActivityPub support to my blog, and is so obvious that I’ve
already been doing it without realising it.</p>
<p>You may be thinking that POSSE is pretty obvious and a bit of a cop-out. But
where POSSE truly shines is when combined with <em>backfeeding</em>. A <em>Backfeed</em> is a
list of replies, likes, and mentions for the current page. Combined with POSSE,
this allows you to see replies to the current page across all different private
silos. For example, you might see comments from Mastodon and Reddit at the
bottom of a blog post, as well as replies from other IndieWeb websites.</p>
<p>Together, POSSE and backfeeding strike a good compromise between owning your
own presence and participating in silos. They improve discoverability and allow
for reader interaction.</p>
<h2 id="webmentions" tabindex="-1">Webmentions <a class="header-anchor" href="#webmentions">#</a></h2>
<p>The IndieWeb community has authored several standards that allow IndieWeb
websites to communicate.</p>
<p><a href="https://indieweb.org/Webmention">Webmentions</a> allow websites to be notified
when another site links to them. By receiving a notification, a site can know
about replies and mentions without having to maintain impractical web crawlers
or subscribe to a backlinking service.</p>
<h3 id="receiving-webmentions" tabindex="-1">Receiving Webmentions <a class="header-anchor" href="#receiving-webmentions">#</a></h3>
<p>I started by
<a href="https://indieweb.org/webmention-implementation-guide">implementing support</a> for
receiving Webmentions. This was super easy, I just needed to add a couple of
<code>link</code> tags to the top of all pages:</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>link</span> <span class="token attr-name">rel</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>webmention<span class="token punctuation">"</span></span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://webmention.io/example.com/webmention<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>link</span> <span class="token attr-name">rel</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>pingback<span class="token punctuation">"</span></span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://webmention.io/example.com/xmlrpc<span class="token punctuation">"</span></span><span class="token punctuation">></span></span></code></pre>
<p><a href="https://webmention.io/">WebMention.io</a> is a cloud service for receiving
Webmentions. You might think it’s odd to use a cloud service for this, but it’s
not a problem as I’m still using my own domain for the pages and could switch
the Webmentions service at any time. IndieWeb isn’t about self-hosting, it’s
about owning your identity and data.</p>
<h3 id="sending-webmentions" tabindex="-1">Sending Webmentions <a class="header-anchor" href="#sending-webmentions">#</a></h3>
<p>I currently send Webmentions manually using
<a href="https://telegraph.p3k.io/">Telegraph</a> or <a href="https://indiewebify.me/">IndieWebify</a>.</p>
<p>My blog is statically hosted and is built using GitLab CI. As the site is only
published when CI finishes, it would be impossible to include sending web
mentions as part of the same CI pipeline. In the future, I’ll probably look into
using Brid.gy or some other tool to send Webmentions by monitoring my web feeds.</p>
<h2 id="microformats2" tabindex="-1">Microformats2 <a class="header-anchor" href="#microformats2">#</a></h2>
<p>Personal websites can contain a variety of content. Long-form articles,
Twitter-like notes, location check-ins, reviews, and replies.
<a href="https://microformats.org/">Microformats2</a> is a way of marking up the content of
web pages so that machines can understand it better. This is a powerful thing
when combined with Webmentions as it allows the receiving website to understand
what is linking to it and why.</p>
<p>Microformats2 works by adding classes to elements representing content:</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>article</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>h-entry<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>h2</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>p-name<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>Hello world!<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>h2</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>a</span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>/tags/a-tag/<span class="token punctuation">"</span></span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>p-category<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
A tag
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>a</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>div</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>e-content<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
This is the article's content.
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>div</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>article</span><span class="token punctuation">></span></span></code></pre>
<p>Implementing support for <a href="https://microformats.org">Microformats2</a> (mf2) was a
huge pain, I cannot overstate how much so. The documentation was very fragmented
and inconsistent, and the tools I found to test mf2 didn’t match the
documentation.</p>
<p>The most extreme problem I had was with
<a href="https://indieweb.org/authorship">authorship</a> - authorship is how you find out
who is the author of a piece of content. The documentation says that you should
be able to just include a link to the homepage in each piece of content, and
tools should fetch the author info:</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>article</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>h-entry<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>div</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>e-content<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>This is an example note<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>div</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>a</span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://rubenwardy.com<span class="token punctuation">"</span></span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>u-author<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>a</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>article</span><span class="token punctuation">></span></span></code></pre>
<p>However, this did not work at all. Most of the tools I found didn’t make further
requests and only looked at the current page. This makes sense I guess, but it’s
annoying that the documentation said it was possible.</p>
<p>The next thing I looked at was including the authorship information in the
footer of each page, and then referencing it from each piece of content like so:</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>article</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>h-entry<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>div</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>e-content<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>This is an example note<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>div</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>a</span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>/<span class="token punctuation">"</span></span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>u-author<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>a</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>article</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>footer</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>a</span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>/<span class="token punctuation">"</span></span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>h-card<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>img</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>u-photo<span class="token punctuation">"</span></span> <span class="token attr-name">src</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>/me.jpg<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>span</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>p-name<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>Author Name<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>span</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>a</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>footer</span><span class="token punctuation">></span></span></code></pre>
<p>Unfortunately, this didn’t work with any of the tools either. The only thing I
found that worked was to include the authorship information in full in every
single piece of content.</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>article</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>h-entry<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>div</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>e-content<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>This is an example note<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>div</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>div</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>p-author h-card d-none<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>a</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>u-url p-name<span class="token punctuation">"</span></span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://rubenwardy.com/<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>rubenwardy<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>a</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>img</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>u-photo<span class="token punctuation">"</span></span> <span class="token attr-name">src</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>/me.jpg<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>div</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>article</span><span class="token punctuation">></span></span></code></pre>
<p>Thank you to users on the IndieWeb IRC channels for pointing towards useful
tools and documentation, and asking my newbie questions. Without them, I
wouldn’t have been able to implement support at all. I believe that they have
improved the documentation a bit based on my feedback, although the authorship
page still mentions the methods I tried that didn’t work.</p>
<p>The three main tools I used for testing Microformats2 were
<a href="https://indiewebify.me/">IndieWebify</a>, <a href="https://pin13.net/mf2/">pin13.net mf2</a>,
<a href="https://waterpigs.co.uk/php-mf2/">Waterpigs mf2</a>.</p>
<h2 id="backfeeding" tabindex="-1">Backfeeding <a class="header-anchor" href="#backfeeding">#</a></h2>
<p>A <em>Backfeed</em> is a list of replies, likes, and mentions for the current page. For
example, you might see comments from Mastodon and Reddit at the bottom of a blog
post, as well as replies from other IndieWeb websites.</p>
<h3 id="backfeeding-likes" tabindex="-1">Backfeeding likes <a class="header-anchor" href="#backfeeding-likes">#</a></h3>
<p>My blog shows likes from Mastodon and other social platforms using
<a href="https://brid.gy/">Brid.gy</a> and Webmentions. When I post a link to my blog on
social media, Brid.gy monitors activity and sends Webmentions. JavaScript on my
blog fetches Webmentions and updates the counter. My blog also caches like
counts at build time. In the future, I’ll make it so that the JS only fetches
activity since the blog was last built, reducing the amount of work the Web
Mentions API needs to do.</p>
<h3 id="backfeeding-replies" tabindex="-1">Backfeeding replies <a class="header-anchor" href="#backfeeding-replies">#</a></h3>
<p>I decided not to implement the backfeeding of comments from social media as I’m
concerned about the privacy implications. Just because someone decides to reply
publicly on social media silos doesn’t mean that they want their post and their
profile picture to appear on my website. Additionally, Webmentions can’t be
deleted meaning that the comment may continue to appear on my website even after
the author deletes it on the silo.</p>
<p>Two good articles discussing the ethics and privacy challenges of backfeeding
include
<a href="https://shkspr.mobi/blog/2022/12/the-ethics-of-syndicating-comments-using-webmentions/">“The ethics of syndicating comments using WebMentions”</a>
and
<a href="https://sebastiangreger.net/2018/05/indieweb-privacy-challenge-webmentions-backfeeds-gdpr/">“The IndieWeb privacy challenge”</a>.</p>
<p>I may reconsider this in the future. I’d need to make it sufficiently clear to
commenters and allow them to opt-out. I’d also need to make sure that deleting
the comment on the silo also deletes it from my website.</p>
<h3 id="comment-form" tabindex="-1">Comment form <a class="header-anchor" href="#comment-form">#</a></h3>
<p>I added a comment form to the bottom of posts on my blog. My blog is statically
hosted. To collect comments, I have a service running on another subdomain that
collects any comments and sends them to me. Users can also choose to send
comments by email or another method. All comments are moderated before showing
on my blog.</p>
<p>To avoid spam, the comment form has a “username” form hidden using CSS. Most
spam bots don’t bother applying the CSS so will see the field and fill it in.
This is called a honeypot field and is surprisingly effective - I was receiving
multiple spam comments a day, but since adding the field I’ve only received a
single spam comment.</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>style</span><span class="token punctuation">></span></span><span class="token style"><span class="token language-css">
<span class="token selector">input[name="username"]</span> <span class="token punctuation">{</span>
<span class="token property">display</span><span class="token punctuation">:</span> none<span class="token punctuation">;</span>
<span class="token punctuation">}</span>
</span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>style</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>input</span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>text<span class="token punctuation">"</span></span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>username<span class="token punctuation">"</span></span><span class="token punctuation">></span></span></code></pre>
<p>You can find
<a href="https://gitlab.com/rubenwardy/blog_api">the source code behind commenting on GitLab</a>.</p>
<h2 id="thoughts-on-the-indieweb" tabindex="-1">Thoughts on the IndieWeb <a class="header-anchor" href="#thoughts-on-the-indieweb">#</a></h2>
<p>IndieWeb standards are fairly obscure and don’t seem to have been adopted much
yet. Of all the posts I’ve made since adding IndieWeb support, this is probably
the only one that will actually find websites linked to that can receive
Webmentions. If a popular Content Management System, like WordPress, added
built-in support for Webmentions and mf2, I could see it suddenly becoming a lot
more popular.</p>
<p>Whilst Webmentions are pretty cool, Microformats2 is pretty complicated and was
pretty annoying to implement. I know that the IndieWeb crowd will have strong
opinions on this, but I quite like how simple <a href="https://json-ld.org/">JSON-LD</a>
was to add support for and that it’s JSON.</p>
<p>As for the community side of IndieWeb, personal websites have seen a big
resurgence since the fall of Twitter. I think we’re in a new golden age for RSS
and personal websites. I’ve been encouraging a lot of my friends to take up
blogging.</p>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>I’m certainly a more technical user than the average blogger. I don’t mind being
an early adopter of technology and appreciate the goals of IndieWeb. I like how
they try to focus on the people before the technology, even though Microformats2
leaves much to be desired.</p>
<p>I currently only post blog posts on my website. I don’t plan to post notes on my
website as I prefer to use Mastodon directly. But I might start posting my
photography here.</p>
<p>I’m undecided as to whether I’ll stick with IndieWeb technology in the long
term, but I’ll certainly continue to own and publish on my own domain.</p>
rubenwardy
Ghosts live in the ground: debugging an electronic circuit
2023-08-29T00:00:00.000Z
2023-08-29T00:00:00.000Z
https://blog.rubenwardy.com/2023/08/29/ghosts-live-in-the-ground/
Whilst working on a plant watering system, I spent hours debugging an issue that turned out to be Electromagnetic Interference (EMI)
<p>I’m making a device to water my basil plants. Whilst working on this, I
encountered an issue where running the pumps whilst connected to WiFi would
crash the microcontroller. After investigation, I found the cause was poor
grounding and Electromagnetic Interference (EMI).</p>
<h2 id="plant-watering-system" tabindex="-1">Plant watering system <a class="header-anchor" href="#plant-watering-system">#</a></h2>
<p>Previously, I created a <a href="/2022/03/17/plant-monitor/">plant monitor</a> that tracked
soil moisture and sent notifications when the plant needed watering. But wouldn’t
it be nice if it could water the plant instead of sending a notification? This
new project will do exactly that.</p>
<p>I’m designing the plant watering system to have support for four plants, with a
soil moisture sensor and pump for each plant. The pumps will draw water from a
water tank, which contains a water level sensor. I’m using an ESP32
microcontroller as the brains, as it comes with Internet connectivity and
low-power deep sleeping.</p>
<p>I designed a circuit and then made a prototype on a breadboard, which worked as
expected. Making a prototype allows me to find any issues with the circuit
design before I bring out the soldering iron. Or, at least, it’s supposed to.</p>
<figure class="undefined"><a href="/static/img/9sXPrJA3X0-3507.png"><picture><source type="image/webp" srcset="/static/img/LiESHuP_Q0-540.webp 540w"><img alt="The schematic for my plant watering system's circuit. Created using KiCAD." loading="lazy" decoding="async" src="/static/img/LiESHuP_Q0-540.png" width="540" height="381"></picture></a><figcaption>The schematic for my plant watering system's circuit. Created using KiCAD.</figcaption></figure>
<h2 id="the-problem" tabindex="-1">The problem <a class="header-anchor" href="#the-problem">#</a></h2>
<p>I started working on the production version, soldered on to stripboard.
Stripboard is a material for making circuits; it comes pre-formed with a grid of
holes and parallel strips of copper traces. I use it because it’s an easy way to
make semi-permanent circuits for my projects.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/LO71Lq9Spt-540.jpeg"><picture><source type="image/webp" srcset="/static/img/9af6zTL_J7-540.webp 540w"><img alt="The front of the stripboard." loading="lazy" decoding="async" src="/static/img/9af6zTL_J7-540.jpeg" width="540" height="372"></picture></a><figcaption>The front of the stripboard.</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/N_rVBVkMyh-540.jpeg"><picture><source type="image/webp" srcset="/static/img/eYXZMPQpuo-540.webp 540w"><img alt="The back of the stripboard, showing copper traces." loading="lazy" decoding="async" src="/static/img/eYXZMPQpuo-540.jpeg" width="540" height="372"></picture></a><figcaption>The back of the stripboard, showing copper traces.</figcaption></figure></div>
</div>
<p>I decided to work on the circuit in parts, so I could test the circuit as I
went. I started by soldering on the pump circuits and connecting them to the
microcontroller. It is at this point that I discovered a big issue - when I ran
a pump, the microcontroller would crash. This would result in the pump staying
on until power was cut.</p>
<blockquote>
<p>Guru Meditation Error: Core 0 panic’ed (IllegalInstruction). Exception was
unhandled.</p>
</blockquote>
<p>I noticed that if I disconnected the microcontroller from WiFi, the crash would
no longer happen.</p>
<h2 id="investigating-the-issue" tabindex="-1">Investigating the issue <a class="header-anchor" href="#investigating-the-issue">#</a></h2>
<h3 id="power-transistor-drawing-too-much-current" tabindex="-1">Power transistor drawing too much current? <a class="header-anchor" href="#power-transistor-drawing-too-much-current">#</a></h3>
<p>I’m using power transistors to allow the microcontroller to turn the pumps on
and off. This is required because the pumps would draw way too much current if
powered directly by the microcontroller. I’m using two power supplies. The pumps
are powered by a 5V USB plug, the kind commonly used to charge phones. The
microcontroller is powered by a micro USB cable to my computer so that I can
program and debug it.</p>
<p>At first, I thought that the power transistors were still drawing too much
current from the microcontroller and causing the crash. The fact that the crash
only happened when the WiFi was running strongly hinted at this, as WiFi
requires a lot of current itself. To test this theory, I replaced the
transistors with a push button. The microcontroller still crashed, showing this
wasn’t the cause.</p>
<h3 id="adjusted-capacitors" tabindex="-1">Adjusted capacitors <a class="header-anchor" href="#adjusted-capacitors">#</a></h3>
<p>After discussing the issue with a friend, I moved the capacitor closer to the
pump and added another, smaller, capacitor. This didn’t fix the issue.</p>
<h3 id="severed-common-ground" tabindex="-1">Severed common ground <a class="header-anchor" href="#severed-common-ground">#</a></h3>
<p>At this point, the only connection to the microcontroller was a ground rail. My
friend now suspected that the issue might be due to a difference in voltage
between the grounds of the two power supplies causing current to flow along the
common ground. I severed the ground rail but the issue still happened. This was
very strange as there was now no connection at all between the pump and the
microcontroller?!</p>
<h3 id="last-hopes" tabindex="-1">Last hopes <a class="header-anchor" href="#last-hopes">#</a></h3>
<p>I decided to create the stripboard again from scratch to make sure it wasn’t a
soldering issue. I did this, and the problem still happened.</p>
<p>At this point, I suspected the circuit was either haunted or there was some kind
of EM interference. I broke the ground rail closer to the ESP32 and it started
working again. It turns out that the ground rail was acting like an antenna. A
current was being induced by the noisy pump and caused the ESP32 to crash
despite not being connected.</p>
<figure class="undefined"><a href="/static/img/Dn4DU3Jd_c-669.jpeg"><picture><source type="image/webp" srcset="/static/img/yZPdTBLGGn-540.webp 540w"><img alt="Photo showing where I broke the ground rail closer to the ESP32." loading="lazy" decoding="async" src="/static/img/yZPdTBLGGn-540.jpeg" width="540" height="817"></picture></a><figcaption>Photo showing where I broke the ground rail closer to the ESP32.</figcaption></figure>
<h2 id="achieving-good-grounding" tabindex="-1">Achieving good grounding <a class="header-anchor" href="#achieving-good-grounding">#</a></h2>
<p>The lesson here is that stripboards have pretty terrible grounding and
Electromagnetic Compatibility (EMC) properties.</p>
<p>Someone on the BPS Space Discord recommended
<a href="https://www.youtube.com/live/ySuUZEjARPY">a video</a> on proper grounding in
circuits. This was very interesting, I learned a lot from it and it will improve
my circuit designs in the future.</p>
<p>While watching that video in a coffee shop, a plant fell on me from a shelf
above, covering me in soil. I guess it wanted to teach me the difference between
grounding and earthing.</p>
<p>To continue with this project, I have decided to bite the bullet and make a
Printed Circuit Board (PCB) for it. I have been looking for an excuse to make a
PCB for a while. PCBs allow you to get vastly better grounding, which will avoid
EMI issues like this in the future.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/7tMKsLMwHL-1125.png"><picture><source type="image/webp" srcset="/static/img/wjIAEcZCjT-540.webp 540w"><img alt="My Work In Progress Printed Circuit Board (PCB) design. Created using KiCAD." loading="lazy" decoding="async" src="/static/img/wjIAEcZCjT-540.png" width="540" height="458"></picture></a><figcaption>My Work In Progress Printed Circuit Board (PCB) design. Created using KiCAD.</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/5zdnRETV9d-908.png"><picture><source type="image/webp" srcset="/static/img/VRyH7zT4yP-540.webp 540w"><img alt="The source of my PCB, showing footprints and traces. Created using KiCAD." loading="lazy" decoding="async" src="/static/img/VRyH7zT4yP-540.png" width="540" height="459"></picture></a><figcaption>The source of my PCB, showing footprints and traces. Created using KiCAD.</figcaption></figure></div>
</div>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>Thanks to my friend and people on the BPS Discord for helping me with this
issue.</p>
<p>Well, this was certainly an… educational problem to have. This problem, and
the switch to making a PCB, has delayed this project by a month at least. But I
will learn a lot more and gain more value from this project. I just hope my
basil can survive in the meantime.</p>
rubenwardy
Ten years of Capture The Flag
2023-08-16T00:00:00.000Z
2023-08-16T00:00:00.000Z
https://blog.rubenwardy.com/2023/08/16/minetest-ctf-is-10/
CTF is a PvP game made using Minetest. This article covers the history of CTF, the lessons I learned, and the changes I made along the way.
<p>Ten years ago today, I uploaded the first version of Capture The Flag. Capture
The Flag is a multiplayer game where two teams of players battle to claim the
other team’s flag whilst defending their own. Capture The Flag is played in a
destructible voxel environment, allowing players to build defences and place
traps.</p>
<p>Capture The Flag started life as a persistent kingdoms game but quickly pivoted
to a match-based team game. It was developed iteratively, taking into account
player feedback. I hosted a server for the game for many years and a community
formed around it. In 2021, I handed over the reins to CTF to very capable hands;
it remains <a href="https://www.minetest.net/">Minetest</a>’s most popular server to this
day.</p>
<p>This article covers the history of CTF, the lessons I learned, and the changes I
made along the way.</p>
<h2 id="2013-city-vs-city-game" tabindex="-1">2013: City vs City game <a class="header-anchor" href="#2013-city-vs-city-game">#</a></h2>
<p>I originally created Capture The Flag to be used on a persistent map, where
teams were countries and players worked together to defend castles. Any player
could create a new country and place flags. Flags were placed at the top of
castles and allowed a country to claim ownership of the surrounding land.</p>
<p>There was a diplomacy system that allowed countries to declare war and peace,
and form alliances. Players could place turrets that automatically attacked
enemy players. If an enemy country claimed a flag, it would gain ownership of
the surrounding land - including any turrets.</p>
<figure class="undefined"><a href="/static/img/Md1sPSK36P-936.png"><picture><source type="image/webp" srcset="/static/img/EZDm1V5gUS-540.webp 540w"><img alt="A Minetest GUI. There's two rows. The first says "red, peace", the second says "blue, war"." loading="lazy" decoding="async" src="/static/img/EZDm1V5gUS-540.png" width="540" height="373"></picture></a><figcaption>The diplomacy tab in the team GUI.</figcaption></figure>
<p>I created a private server to test the game with friends. I hosted this at home,
so it wasn’t online that much. I implemented the basics - countries, flags, land
ownership, and turrets - but never really got to the point where I felt like it
was ready to host publicly.</p>
<p>When I first started working on the game, I knew it would be running on a server
and didn’t want it to crash. I added a lot of if-statements for things that
should never happen, and in the process caused a lot of bugs to become subtle
behaviour issues rather than crashes. The lesson I learned here is that it’s
better to crash than to obscure a bug or end up in an unknown state.</p>
<p>From the start, I wanted the core mods to support being used in any team-based
games, even without flags. A lot of features were added in additional mods, for
example, flags were added by <code>ctf_flags</code>. There were also a lot of configuration
options that were never really needed. I learned the principle of
<a href="https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it">You Aren’t Gonna Need It (YAGNI)</a>
the hard way.</p>
<h2 id="2015-switch-to-match-based-capture-the-flag" tabindex="-1">2015: Switch to match-based Capture The Flag <a class="header-anchor" href="#2015-switch-to-match-based-capture-the-flag">#</a></h2>
<p>In July 2015, after a while of not working on the game, I decided to pivot to
making a match-based game instead. I still planned to work on a persistent
kingdoms game eventually, but doing a match-based game first would allow me to
develop the core code and test it. I split the game into a reusable mod pack
called CTF PvP Engine and a game called Capture The Flag.</p>
<p>Around this time, I played
<a href="https://store.steampowered.com/app/219830/King_Arthurs_Gold/">King Arthur’s Gold</a> -
a 2d sidescrolling capture-the-flag game where players build castles and then
attack each other. The game’s structural integrity mechanic made sieges pretty
awesome. I quite liked the defensive building part of the game and felt that a
voxel world was especially well-suited for this kind of gameplay. King Arthur’s
Gold was Capture The Flag’s main inspiration.</p>
<figure class="undefined"><a href="/static/img/npEmqtDlgO-1920.jpeg"><picture><source type="image/webp" srcset="/static/img/a2QG9JSTTH-540.webp 540w"><img alt="Screenshot of a 2d sidescrolling game. On each side of the image, there's castles built out of stone. Red and blue players are approach each other from opposite sides, firing arrows and raising shields." loading="lazy" decoding="async" src="/static/img/a2QG9JSTTH-540.jpeg" width="540" height="303"></picture></a><figcaption>King Arthur's Gold (source: <a href="https://store.steampowered.com/app/219830/King_Arthurs_Gold/">Steam page</a>)</figcaption></figure>
<h2 id="2015-public-server" tabindex="-1">2015: Public server <a class="header-anchor" href="#2015-public-server">#</a></h2>
<p>In September 2015, I started the official Capture The Flag server, hosted on
DigitalOcean. The server was mostly dead for a few months but then suddenly
become popular - I guess it had reached the critical number of players that
allowed the game to become fun.</p>
<p>By this point, I had been making mods for Minetest for a couple of years but
hadn’t created a public server before. One thing I learned was just how good
running a server is at finding bugs - players do a lot of things that you don’t
expect. One example is when players discovered that dropping sand on flags
caused them to be dropped as items, allowing players to place multiple flags for
their team. This made it much harder for the other team to win as they would
have needed to capture all the flags.</p>
<p>I found that another benefit of running a public server is getting access to a
community of players and feedback. From this point onwards, I was able to
iteratively develop the game based on how the players played it and what
feedback they gave. For example, the game contained team chests that allowed
players to share weapons and items. I discovered that a lot of team members were
intentionally destroying unused weapons rather than placing them in the team
chest, to prevent newbie players from picking them up, dying, and delivering the
weapons to the enemy team. To fix this issue, I made it so part of the chest was
restricted to players who had at least a certain amount of in-game experience.</p>
<figure class="undefined"><a href="/static/img/d9hrCqOul9-762.png"><picture><source type="image/webp" srcset="/static/img/FNlkIyfTCH-540.webp 540w"><img alt="Screenshot of a chest inventory GUI. The inventory is split in half. On the left, there's "Any player can take from here, including enemies." On the right, it says "You need more score (200+)"" loading="lazy" decoding="async" src="/static/img/FNlkIyfTCH-540.png" width="540" height="574"></picture></a><figcaption>The pro section in the team chest. An early version allowed enemies to take from the chest too, although this was changed later on.</figcaption></figure>
<p>I delegated the day-to-day running of the server to moderators. I wasn’t
interested in being a moderator or server owner, for me the fun was in creating
an experience and seeing how the players react, rather than enforcing rules. I
chose new moderators based on the recommendation of existing moderators - I felt
that the existing moderators were better placed to judge character. A lot of
people asked to become a moderator, but I rarely granted this as asking to
become a moderator is a red flag.</p>
<p>The DigitalOcean server I was using was quite underpowered. In April 2016, a
community member called Calinou offered to host the server and did so until they
stopped it in November 2016.</p>
<h2 id="2017-restarted-the-server-and-development" tabindex="-1">2017: Restarted the server and development <a class="header-anchor" href="#2017-restarted-the-server-and-development">#</a></h2>
<p>In October 2017, I decided to restart the server and development on Capture the
Flag due to popular demand. This time, I hosted the server using a dedicated
machine on Kimsufi - a brand owned by OVH. With hindsight, hosting on a cheap
dedicated server wasn’t good value for money, but it was faster than the
DigitalOcean VPS.</p>
<p>My aim for development at this point was to improve the variety of matches,
increase strategy, and incentivise co-op. It was too easy for a match to be won
by a lone hero.</p>
<h2 id="2018-multiple-maps" tabindex="-1">2018: Multiple maps <a class="header-anchor" href="#2018-multiple-maps">#</a></h2>
<p>After months of work in secret, I released support for multiple maps. Instead of
always playing on the same map, the map was now different for each match.
Players could use a dedicated tool to create new maps and then submit them for
inclusion.</p>
<p>The feature shipped with three maps - Two Hills and Two Lakes (the previous map
used), The Bridge, and Caverns. Caverns was my favourite map. It’s set
underground, with three caverns and a small dungeon to explore. All of these
were disconnected; players needed to tunnel to reach the enemy team. I liked how
this added a stealth aspect to the game.</p>
<p>Notice how none of these maps came with prebuilt structures. The aim was still
to encourage players to build their own defences and use the destructible
environment to their advantage. But that doesn’t mean I was against it
completely - some of the new maps contributed after the feature was released
included entire towns.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/K30MIKRBHv-987.png"><picture><source type="image/webp" srcset="/static/img/zKpj7-UZxx-540.webp 540w"><img alt="Screenshot of a voxel world. To the top and the bottom of the image are grassy hills. Between them are two lakes and sand." loading="lazy" decoding="async" src="/static/img/zKpj7-UZxx-540.png" width="540" height="378"></picture></a><figcaption><b>Two Hills, Two Lakes:</b> The classic CTF map. Photo by Shara.</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/S0GvN3Ty9R-1920.png"><picture><source type="image/webp" srcset="/static/img/dYcBxWIoBL-540.webp 540w"><img alt="Screenshot of a voxel world. A canyon with a river splits the terrain. There's a one meter wide bridge from one side to the other." loading="lazy" decoding="async" src="/static/img/dYcBxWIoBL-540.png" width="540" height="303"></picture></a><figcaption><b>The Bridge:</b> Very little cover and no apple trees. A river separates the two teams.</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/DHG8561lPB-1920.png"><picture><source type="image/webp" srcset="/static/img/4DCq8EFlrh-540.webp 540w"><img alt="Screenshot of a voxel cavern light by torches. In the center, there's a blue flag." loading="lazy" decoding="async" src="/static/img/4DCq8EFlrh-540.png" width="540" height="303"></picture></a><figcaption><b>Caverns:</b> Each team has a cavern. Lots of ores and tunnelling.</figcaption></figure></div>
</div>
<h2 id="2019-apple-cheating-and-medkits" tabindex="-1">2019: Apple cheating and medkits <a class="header-anchor" href="#2019-apple-cheating-and-medkits">#</a></h2>
<p>Apples were used to heal the player in CTF. Someone made a cheat that would eat
apples automatically when they took damage, by emulating player input.</p>
<p>The cheat was programmed quite poorly and was vulnerable to a
<a href="https://en.wikipedia.org/wiki/Arbitrary_code_execution">Remote Code Execution (RCE)</a>
bug. This could have allowed the server and other players to run non-sandboxed
shell commands, taking over the cheater’s computer. I didn’t exploit this for
obvious reasons.</p>
<p>I wrote a mod to detect the use of this cheat. At the end of each match, a GUI
window is shown with the match results. I dealt damage to all players when this
window was shown and then listened to key presses using a hidden text field. If
a bunch of numbers were pressed (ie: 3333331), then I’d know that they were
using the apple cheat. The anti-cheat mod would wait a random period, 5-10
minutes, and then ban them with an anti-cheat message.</p>
<p>This anti-cheat wasn’t a permanent solution but was enough to catch and
frustrate the cheat writer, which was the main goal. We eventually replaced
apples with medkits and bandages. Players can use medkits to heal themselves but
have to remain still whilst it acts, leaving them vulnerable. Players can use
bandages to instantly heal their teammates but not themselves.</p>
<h2 id="2020-classes-and-ctf-2-0" tabindex="-1">2020: Classes and CTF 2.0 <a class="header-anchor" href="#2020-classes-and-ctf-2-0">#</a></h2>
<p>In 2020, the COVID pandemic meant that I was able to find time again for the
game. I worked on a classes feature inspired by King Arthur’s Gold. It allowed
players to select a class at their team’s flag. Classes come with different
abilities and initial items. The aim was to increase variety and allow other
styles of play to become more viable, as to this point the most successful
tactic was to spam with a steel sword.</p>
<p>I also refactored a lot of the code to remove unused features left over from the
City vs City days. I released this and classes as Capture The Flag 2.0.</p>
<figure class="undefined"><a href="/static/img/xu6unFc-aZ-1122.png"><picture><source type="image/webp" srcset="/static/img/Iv_HbeSSAR-540.webp 540w"><img alt="A Minetest GUI. There's four columns, one for each class. Knight, Sharp Shooter, Medic, and Rocketeer." loading="lazy" decoding="async" src="/static/img/Iv_HbeSSAR-540.png" width="540" height="178"></picture></a><figcaption>Players could change classes by right-clicking their team's flag. RIP Rocketeer.</figcaption></figure>
<h2 id="2021-handover-to-landarvargan" tabindex="-1">2021+: Handover to LandarVargan <a class="header-anchor" href="#2021-handover-to-landarvargan">#</a></h2>
<p>By 2021, I had fully lost interest in working on Capture The Flag. I had limited
time and there were many other projects I wanted to be working on. I chose
<a href="https://fosstodon.org/@Lone_Wolf">LandarVargan</a> to become the new project lead
as he had been significantly contributing to CTF for a while.</p>
<p>After taking over, LandarVargan added savilli to the development team. They
worked together on Capture the Flag 3.0, a rewrite to get rid of eight years of
technical debt. Many new gameplay features have been added to 3.0 including
support for multiple game modes. These game modes are another way of increasing
variety.</p>
<p>During a testing session for the 3.0 rewrite, some players got ahold of
near-infinite stacks of grenades. This formed the inspiration for the Nade Fight
game mode added by savilli. In this mode, players fight with an unlimited
supply of 3 different grenades. Each one has a unique use and a cooldown after
throwing. Void grenades trap players, firecrackers damage them, and yellow
grenades allow the player to move quickly through the air.</p>
<p>Unlike me, LandarVargan plays existing competitive PvP games in his own time,
including Albion Online and Brawlhalla. He was interested in what about those
games and CTF was fun and attracted their player base. In studying them, he got
a lot of ideas for how he could improve CTF.</p>
<figure class="figure ">
<blockquote class="mb-1" ><p>CTF’s player base is small if you compare it to other games, but that also
means it’s much easier for devs to stay in touch with the community and
listen to their feedback. Most people who play Minetest also have some
experience programming too, and have access to CTF’s open-source code, so
feedback is much higher quality than it is for most games.</p>
<p>It’s extremely helpful to have staff that can handle the day-to-day
running of the server and social media platforms (ie: Discord) for you.
I was semi-aware of the difficulties of running a server and community
before taking over CTF but greatly underestimated it, and the ever-ongoing
process of learning how to do/delegate that has probably been the most
valuable experience I’ve gotten out of working on CTF.</p>
<p>In the future, I don’t plan to make many more big additions - like I
already have with the cosmetics system, multiple modes system, or Elysium
(A hub area outside of the main gameplay loop where players can hang out/
practice PvP after capturing the flag) and hope to get to a point where
I’m just playing the game, polishing/improving existing things, and adding
new modes, which most of my remaining ideas revolve around.</p>
</blockquote>
<figcaption class="pl-4 mt-3">
—LandarVargan</figcaption>
</figure>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>I never ended up continuing my City vs City game, but someone else used the
CTF PvP Engine to make a similar game called
<a href="https://forum.minetest.net/viewtopic.php?t=14587">Persistent Kingdoms</a>.</p>
<p>Capture The Flag feels slightly bittersweet to me. Whilst it has been popular
and liked by a lot of players, I feel like it never became the game I wanted it
to be. I was never that interested in first-person shooters; I wanted it to be a
game of strategy and castle building and sieging. To get it to that I probably
would have had to implement some difficult features like structural integrity
and pivot the game’s features more to that goal. Not sure whether the player
base would have wanted this. Capture The Flag was ultimately developed by
watching how it was played.</p>
<p>Whilst I am no longer interested in continuing CTF myself, I am glad that the
community grew to a point where it was able to attract contributors and survive
without me. I think this is the best indicator of the project’s success.</p>
<h3 id="join-the-celebrations" tabindex="-1">Join the celebrations <a class="header-anchor" href="#join-the-celebrations">#</a></h3>
<p>The 8th anniversary of the server will be the 1st of September. Fancy
celebrating? From today until the 1st of September, there will be fireworks and
party hats on the server. You can play Capture The Flag by joining
<a href="https://ctf.rubenwardy.com/">ctf.rubenwardy.com</a>:30001 using a Minetest client.</p>
<figure class="undefined"><a href="/static/img/AWPOfFx3qU-1115.png"><picture><source type="image/webp" srcset="/static/img/o4wv1rDVVY-540.webp 540w"><img alt="A Minetest GUI. There's four columns, one for each class. Knight, Sharp Shooter, Medic, and Rocketeer." loading="lazy" decoding="async" src="/static/img/o4wv1rDVVY-540.png" width="540" height="155"></picture></a><figcaption>Fireworks will spawn in treasure chests. You can get an Anniversary Hat by launching a rocket into the sky. Screenshot by LandarVargan.</figcaption></figure>
<!--
## Timeline
* 2013, Aug: start of code
* Was originally titled "Capture The Flag / City vs City combat" in a repo called capture_the_flag
* Persistent kingdoms-style gameplay
* Turrets, diplomacy (war, peace, alliance), ...
* Had private (invite-only) server for testing
* ?: Someone else hosted a public City vs City server, forget who
* 2015, Jul: Split into mod pack (ctf_pvp_engine) and game (capturetheflag)
* 2015, Jul: Switched to match-based CTF
* 2015, Sep: Started hosting my own server
* 2015, Nov: Guns! Started with kaeza's firearms mod, was buggy, switched to shooter
* 2015, Dec: Team chests
* 2015, Dec: Stats / league table
* 2016, Feb: Two hills and two lakes map
* 2016, Apr: Calinou takes over hosting CTF
* 2016, Apr: kill history
* 2016, between Nov-Jan: Server stopped
* 2017, Oct: Server restarted after someone asked
* 2018, Jan: sprinting, team doors
* 2018, Jan: [Multiple maps](https://forum.minetest.net/viewtopic.php?p=309834#p309834)
* 2018/19?: ANAND as a developer
* 2018, May: Discord
* 2019, Mar: Apples replaced with medkits and bandages
* ANAND leaves
* 2020, Mar: Classes
* LW as a developer
* 2021, Nov: hand over to LW
* 2021, Nov: v3
-->
rubenwardy
Make games not engines: why I'll be using Godot engine
2023-07-14T00:00:00.000Z
2023-07-14T00:00:00.000Z
https://blog.rubenwardy.com/2023/07/14/make-games-not-game-engines/
This article is an exploration of my journey in game dev, a discussion of what motivates me, and a promise for the future.
<p>One of the reasons I learned how to program was to make games. Games are a
unique form of creative medium, combining art, interactive storytelling, and
vibrant worlds. But as a software engineer, it’s easy to lose sight of my goals
and get trapped by the technical details. It’s common for software engineers in
game dev to roll their own engine, which I believe reduces productivity and is
ultimately a distraction to making a game.</p>
<p>Note that I’m not just referring to making reusable or generic game engines; for this
article, I consider using low-level technology like OpenGL, SFML, or SDL to make
games to include the act of rolling your own game engine, even if the focus is
specific. It’s more manageable, but you still end up reinventing the wheel and
having to solve many of the same problems.</p>
<p>There are plenty of other articles about whether or not to make your own game
engine. This article is personal to me; it’s an exploration of my journey in
game dev, a discussion of what motivates me, and a promise for the future.</p>
<h2 id="my-journey" tabindex="-1">My Journey <a class="header-anchor" href="#my-journey">#</a></h2>
<h3 id="learning-to-program" tabindex="-1">Learning to Program <a class="header-anchor" href="#learning-to-program">#</a></h3>
<p>One of the first programming languages I learned was Scratch. I created a lot of
small games, the most impressive of which was
<a href="https://scratch.mit.edu/projects/1162598/">Spirit Walker</a>, an adaption of my
favourite book series at the time.</p>
<p>Not long afterwards, I was given a book called “Game Programming for Teens”,
which taught me how to make games using the Blitz3D game engine. I created a series
of competitive games, mostly with split-screen multiplayer, called
<a href="https://github.com/rubenwardy/ProSeriesMiniGames">Pro Series</a>.</p>
<p>Neither of these tools were perfect: they were pretty limited, and the code I
wrote sucked almost as much as the art I made. But those tools allowed seeing
results immediately and that is very important for motivation. I was able to
create a lot of impressive things (for my age) in a fairly short time.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/wETt-G5wBT-480.png"><picture><source type="image/webp" srcset="/static/img/T6FP5e00J8-480.webp 480w"><img alt="Spirit Walker (2010)" loading="lazy" decoding="async" src="/static/img/T6FP5e00J8-480.png" width="480" height="360"></picture></a><figcaption>Spirit Walker (2010)</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/zccLmJBG0q-1326.png"><picture><source type="image/webp" srcset="/static/img/af4ybxP4w2-540.webp 540w"><img alt="Pro Series: Dogfight - two-player plane battle (2011)" loading="lazy" decoding="async" src="/static/img/af4ybxP4w2-540.png" width="540" height="303"></picture></a><figcaption>Pro Series: Dogfight - two-player plane battle (2011)</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/vyuTMmUdhv-1024.jpeg"><picture><source type="image/webp" srcset="/static/img/1-aOmjY9YJ-540.webp 540w"><img alt="Pro Series: Obstacle - adventure obstacle course (2011)" loading="lazy" decoding="async" src="/static/img/1-aOmjY9YJ-540.jpeg" width="540" height="405"></picture></a><figcaption>Pro Series: Obstacle - adventure obstacle course (2011)</figcaption></figure></div>
</div>
<h3 id="rolling-my-own-engine" tabindex="-1">Rolling My Own Engine <a class="header-anchor" href="#rolling-my-own-engine">#</a></h3>
<p>It wasn’t long before I was running up against the limitations of Scratch and
Blitz3D. I briefly tried Unity, but I wasn’t a fan of how it worked.</p>
<p>I googled “programming languages for game dev” and found C++. Perhaps not the
best advice for a beginner. From there, I found
<a href="https://irrlicht.sourceforge.io/">Irrlicht</a>, a 3d rendering library on top of
which you can make your own engines. Whilst Irrlicht is largely obsolete and
dead these days, at the time it had a thriving community and wasn’t too
outdated. I discovered <a href="https://www.minetest.net">Minetest</a> on the Irrlicht
forums; I’ve been contributing to Minetest since this time, eventually making
core developer in 2015.</p>
<p>I made a few simple game prototypes with Irrlicht, including a zombie survival
game. I also made <a href="https://rubenwardy.com/NodeBoxEditor/">NodeBoxEditor</a>, a
voxel 3d modelling program for Minetest.</p>
<p>Whilst these projects served as great learning exercises, I wasn’t able to
finish many projects as much of the focus was on the <em>tech</em> rather than on the
<em>games</em>. One exception during this time was my HTML5 projects, I was able to
complete quite a few small projects including
<a href="https://physics.rubenwardy.com">Physics Simulation Kit</a>.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/8jo9Zp_sV1-1024.png"><picture><source type="image/webp" srcset="/static/img/P7Pmoo8-NB-540.webp 540w"><img alt="A zombie survival game that was never finished, C++ Irrlicht (2012)" loading="lazy" decoding="async" src="/static/img/P7Pmoo8-NB-540.png" width="540" height="405"></picture></a><figcaption>A zombie survival game that was never finished, C++ Irrlicht (2012)</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/APK9Mum01X-756.png"><picture><source type="image/webp" srcset="/static/img/j42mWXwWSD-540.webp 540w"><img alt="A government politics sim prototype, HTML5 (2014)" loading="lazy" decoding="async" src="/static/img/j42mWXwWSD-540.png" width="540" height="389"></picture></a><figcaption>A government politics sim prototype, HTML5 (2014)</figcaption></figure></div>
</div>
<h3 id="the-final-straw-rvwp" tabindex="-1">The Final Straw: RVWP <a class="header-anchor" href="#the-final-straw-rvwp">#</a></h3>
<p><a href="https://rubenwardy.com/rvwp/">Ruben’s Virtual World Project (RVWP)</a> was a
hybrid top-down shooter and base builder, where you manage a group of rebels
hiding from a dystopian government. In “base builder” mode, the player would
build a base and manage a group of characters similar to RimWorld. Then they
would switch to “Hero” mode, and go on missions in a top-down shooter.</p>
<p>RVWP was written using C++ and SFML; I started the project to learn low-level
multiplayer networking but transitioned to making a game after a while. I spent
five years (on and off) working on it. In the end, I implemented very basic
top-down gameplay, a build mode, and some
<a href="/2022/07/17/game-ai-for-colonists/">NPC AI</a>. The amount of time I spent on
gameplay was massively dwarfed by the amount of time working on the engine; I
implemented multiplayer networking, voxel lighting, entities, behaviour trees,
and Lua scripting. By the time I got to this point, I had mostly lost interest
in the game idea.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/OTyw-uIQ7E-400.gif"><img alt="Z-Layer system in RVWP, C++ SFML (2015-2021)" loading="lazy" decoding="async" src="/static/img/ogXwDoCf6g-400.gif" width="400" height="43200"></a><figcaption>Z-Layer system in RVWP, C++ SFML (2015-2021)</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/z5GG_qT3fM-814.png"><picture><source type="image/webp" srcset="/static/img/24zabsrqPU-540.webp 540w"><img alt="Base building in RVWP, C++ SFML (2015-2021)" loading="lazy" decoding="async" src="/static/img/24zabsrqPU-540.png" width="540" height="425"></picture></a><figcaption>Base building in RVWP, C++ SFML (2015-2021)</figcaption></figure></div>
</div>
<h2 id="thoughts" tabindex="-1">Thoughts <a class="header-anchor" href="#thoughts">#</a></h2>
<h3 id="what-did-i-learn" tabindex="-1">What did I learn? <a class="header-anchor" href="#what-did-i-learn">#</a></h3>
<p>When I first started making games, most of my time was spent implementing
gameplay. As I became a better programmer, solutions became increasingly more
programming-focused, with less time being spent on gameplay.</p>
<p>It all comes down to priorities. Creating my own game engines taught me a lot
about programming and game development. I implemented scene graphs, entity
systems, sounds, multiplayer networking, physics, tools, and more. I became
better at C++ and programming overall. But I’m no longer interested in
programming as a learning exercise, I want to actually make games.</p>
<p>Time spent rolling my own tech is time I could be working on gameplay; how good
a game is depends a lot on the number of iterations and feedback cycles it goes
through. Modern game engines make it much easier to prototype and get a game to
a playable state quickly. I’m not planning on starting an indie dev company, but
if I were - time is money and using a game engine is a competitive advantage.</p>
<p>I’m not saying that no one should write game engines. Someone needs to, and they
are amazing learning experiences. But if you want to make a game, make a game,
don’t make an engine as well. And if you seriously want to make a game engine,
make sure you have a team and a limited scope (ideally a game in mind).</p>
<figure class="figure ">
<blockquote class="mb-1" cite="https://web.archive.org/web/20201215065909/https://twitter.com/epyoncf/status/1123155254760038400"><p>Give a man a game engine, and he’ll have fun making a game. Teach a man to
create an engine and… I dunno, he’s still creating the, what is it, fifth
iteration of his engine?</p>
</blockquote>
<figcaption class="pl-4 mt-3">
—<a href="https://web.archive.org/web/20201215065909/https://twitter.com/epyoncf/status/1123155254760038400">
Kornel Kisielewicz (@epyoncf)
</a></figcaption>
</figure>
<h3 id="why-did-i-write-my-own-engine" tabindex="-1">Why did I write my own engine? <a class="header-anchor" href="#why-did-i-write-my-own-engine">#</a></h3>
<p>So why did I ever decide to write my own engines? Part of it is the learning
exercise excuse, RVWP was started so I could learn about multiplayer networking
at a low level.</p>
<p>Another part is being able to know and own all my code. Before I started using
C++ and Irrlicht, I tried making games with Unity. I ran into several bugs in
the engine - given that Unity is closed-source, I was frustrated at not being
able to fix the bugs or have any visibility into why they were happening. This
is not as much of a problem with Godot, as it <s>has zero bugs</s> is open-source.</p>
<p>Another part of it is dogma - I felt like existing engines were impure, buggy,
and bloated. As a software engineer, it was much more fun to write my own tech,
especially to begin with when the project is new. It was also easy to justify -
this is just a learning exercise! The most important thing when making a game is
to make the game. Players don’t care if the game is a few extra megabytes -
games are regularly in the gigabytes these days! Nowadays, I care more about the
outcome than always aiming for ‘perfect’ tech. This quote comes to mind:</p>
<figure class="figure ">
<blockquote class="mb-1" cite="https://www.polygon.com/2020/1/13/21064100/vvvvvv-source-code-game-development-terry-cavanagh-release"><p>There’s a word for games where the code is barely hanging together, with
stupid layout, utterly unscaleable fixes and workarounds on top of
workarounds.</p>
<p>“Shipped”</p>
</blockquote>
<figcaption class="pl-4 mt-3">
—<a href="https://www.polygon.com/2020/1/13/21064100/vvvvvv-source-code-game-development-terry-cavanagh-release">
Mike Bithell
</a></figcaption>
</figure>
<p>Obviously, the above is taken to the extreme. I’m not planning to become a
<a href="https://en.wikipedia.org/wiki/Flying_Spaghetti_Monster">Pastafarian</a>, but I am
now willing to take a more pragmatic approach to creating technology. My goal
for the future is to focus less on the perfect tech and more on the thing I’m
creating. I will use established game engines and not be afraid to write
imperfect code.</p>
<h2 id="a-new-hope-godot" tabindex="-1">A New Hope: Godot <a class="header-anchor" href="#a-new-hope-godot">#</a></h2>
<p>I tried out Godot quite a while ago, but it was still quite immature. Godot is a
very capable engine now, especially Godot 4. I’ve started using it for new game
projects. I really didn’t like Phaser.JS, so ended up converting my
<a href="/2022/03/19/ld48-tin-mining/">Tin Mining</a> game jam entry from that to Godot
as well.</p>
<p>Godot isn’t perfect. When I first used it, I wasn’t a fan of how it used
inheritance rather than something more component-based. But this is an example
of ‘purity’ again, there are plenty of ways to do composition in Godot and avoid
all the issues with OOP-based designs.</p>
<p>Premade game engines can have a bit of a learning curve, but it’s worth it in
the long run. Godot’s
<a href="https://docs.godotengine.org/en/stable/index.html">documentation</a> is very good
and there are a lot of third-party tutorials and support available.</p>
<figure class="undefined"><a href="/static/img/JPiMw1C2RY-900.jpeg"><picture><source type="image/webp" srcset="/static/img/VGFWe5QjaR-540.webp 540w"><img alt="My <a href="/2022/03/19/ld48-tin-mining/">Tin Mining</a> game running on Steam Deck. This took almost no time to set up, thanks to Godot.
" loading="lazy" decoding="async" src="/static/img/VGFWe5QjaR-540.jpeg" width="540" height="211"></picture></a><figcaption>My <a href="/2022/03/19/ld48-tin-mining/">Tin Mining</a> game running on Steam Deck. This took almost no time to set up, thanks to Godot.
</figcaption></figure>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>I’m a software engineer. It’s easy to fall into the trap of writing code for the
fun of it, rather than to actually make something. But this is a distraction, I
like making things and want to actually finish a game.</p>
<p>I’ve been learning Godot and enjoy using it. One day, hopefully soon, I’d like
to finish a game and publish it to Steam. I’m not expecting to make the next
bestseller, just something that I’m proud of enough to publish even if niche.</p>
<h3 id="further-reading" tabindex="-1">Further reading <a class="header-anchor" href="#further-reading">#</a></h3>
<ul>
<li><a href="https://pikuma.com/blog/why-make-a-game-engine">Should I Write a Game Engine or use an Existing One?</a> -
this article contains a lot of pros and cons of making your own engine.</li>
<li><a href="https://geometrian.com/programming/tutorials/write-games-not-engines/">Write Games, Not Engines</a> -
the author defines ‘engine’ as strictly reusable code, which is a more narrow
definition (but potentially more accurate) than what I use. This is a good
article to read if you do want to make engines, though, as it describes the
best approach - make your engine with a game in mind.</li>
<li><a href="https://duerrenberger.dev/blog/2023/04/11/should-i-develop-a-game-or-an-engine/">Should I Develop a Game or an Engine?</a></li>
<li><a href="https://www.polygon.com/2020/1/13/21064100/vvvvvv-source-code-game-development-terry-cavanagh-release">The truth is that many games are held together by duct tape - Polygon</a></li>
</ul>
rubenwardy
Conquer: a deep-dive into my Minetest mod jam entry
2023-05-31T00:00:00.000Z
2023-05-31T00:00:00.000Z
https://blog.rubenwardy.com/2023/05/31/conquer/
Conquer is an RTS minigame created for the 2020 mod jam. It received first place. This article will discuss some interesting challenges that it needed to solve.
<p>In 2020, the Minetest Discord community ran a mod-making competition with
“combat” as the theme. Participants had one week to create a mod with all the
code written within the time, but pre-existing art was allowed. I made a
Real-Time Strategy (RTS) minigame called
<a href="https://content.minetest.net/packages/rubenwardy/conquer/">Conquer</a>;
it received first place.</p>
<p>In this article, I will discuss some of the interesting challenges that Conquer
needed to solve. I believe that Conquer is a great example to learn from as it
demonstrates best practices for Minetest modding; it is well-structured and
unit-tested.</p>
<h2 id="what-is-conquer" tabindex="-1">What is Conquer? <a class="header-anchor" href="#what-is-conquer">#</a></h2>
<p><a href="https://content.minetest.net/packages/rubenwardy/conquer/">Conquer</a> is a mod
that adds RTS gameplay. It allows players to start Conquer mini-games, where
they can place buildings, train units, and fight other players.</p>
<p>Whilst I created Conquer as a mod, it was designed with plans to eventually
convert it into a game, with custom map generation and more in-depth gameplay.</p>
<p>Players select units by left-clicking them. To issue commands, the player uses
the scroll wheel to select a command and then right-click to perform it.
Commands include move, melee attack, and ranged attack. Archers automatically
aim at nearby enemies, and all units will defend themselves from melee attacks.</p>
<p>Units can damage buildings using melee or ranged attacks. When a player’s keep
is destroyed, that player is out of the game. The appearance of a building
reflects how much damage it has taken.</p>
<figure>
<iframe class="youtube-video"
src="https://www.youtube.com/embed/jB_sFZGRUgw"
title="YouTube video player"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
allowfullscreen>
</iframe><figcaption>Conquer gameplay trailer</figcaption></figure>
<h2 id="my-process" tabindex="-1">My Process <a class="header-anchor" href="#my-process">#</a></h2>
<p>I used GitLab projects to organise my tasks. I decided what the Minimal Viable
Product (MVP) was and created issues for the core features. During development,
I created new issues to break down high-level goals into smaller steps.</p>
<p>I only had seven days to make quite a complicated mod. I needed to cut a lot of
features that would have been desirable, such as squad movement, walls, and
siege engines. I pushed features that weren’t needed in the MVP to the bottom of
the backlog and resisted implementing them.</p>
<figure class="undefined"><a href="/static/img/yFodKQw_Gw-1191.png"><picture><source type="image/webp" srcset="/static/img/n_9UJDJMpZ-540.webp 540w"><img alt="Three columns with 4 issues in each column" loading="lazy" decoding="async" src="/static/img/n_9UJDJMpZ-540.png" width="540" height="235"></picture></a><figcaption>Current GitLab project board for conquer. Tasks to work on next are on the right</figcaption></figure>
<h2 id="unit-ai-behaviour" tabindex="-1">Unit AI Behaviour <a class="header-anchor" href="#unit-ai-behaviour">#</a></h2>
<p>Units need to know how to do some tasks without being micromanaged by the
player. They need to understand how to navigate the world, melee attack other
units, and arrows at moving targets. To tell the units how to do these things, I
chose to use Behaviour Trees.</p>
<h3 id="behaviour-trees" tabindex="-1">Behaviour Trees <a class="header-anchor" href="#behaviour-trees">#</a></h3>
<p>Explaining Behavior Trees in full is a bit out of scope for this article; I
highly recommend reading
<a href="https://www.gamedeveloper.com/programming/behavior-trees-for-ai-how-they-work">“Behavior trees for AI: How they work”</a>
by Chris Simpson. However, I will try to explain the basics.</p>
<p>Behaviour trees allow you to control an NPC’s decision-making by combining
reusable nodes rather than writing code for specific states. They work best when
the nodes are small and specific, for example, you might have nodes to check
conditions or walk to a position.</p>
<p>Behaviour trees are basically a programming language in themselves, but for game
AI. Execution starts at the top of the tree, and then works downwards based on
the rules of different nodes. A node is either running, succeeded, or failed.</p>
<figure class="undefined"><a href="/static/img/h2NpJ4FboX-450.png"><picture><source type="image/webp" srcset="/static/img/_POkdAl5wF-450.webp 450w"><img alt="A behaviour tree with 3 nodes. The root is a sequence node, and the two children are "Go to target" and "Attack target"" loading="lazy" decoding="async" src="/static/img/_POkdAl5wF-450.png" width="450" height="237"></picture></a><figcaption>The behaviour tree for melee attacks. The unit first moves to be next to the target, and then performs the attack</figcaption></figure>
<p>Using behaviour trees for Conquer unit AI was probably overkill. Behaviour trees
are great for structuring complex behaviour, but Conquer’s unit AI is fairly
simple. I decided to use behaviour trees as I wanted to make a Lua
implementation to use in other projects anyway - I ended up using it
<a href="/2022/07/17/game-ai-for-colonists/">in another project to create worker NPCs that can build</a>.</p>
<p>The nodes implemented in Conquer include:</p>
<ul>
<li>Selector: run one child after another until one succeeds</li>
<li>Sequence: run all children in order, stop if one fails</li>
<li>FollowPath: cause the unit to follow a path</li>
<li>MoveToNearTarget: cause the unit to move to be near a target</li>
<li>MeleeAttack and RangedAttack.</li>
</ul>
<h3 id="aiming-arrows" tabindex="-1">Aiming arrows <a class="header-anchor" href="#aiming-arrows">#</a></h3>
<figure class="undefined"><a href="/static/img/pyZl7dtMq3-960.webp"><img alt="Archers shooting a volley of arrows" loading="lazy" decoding="async" src="/static/img/LJabCsl4rV-960.webp" width="960" height="25920"></a><figcaption>Archers shooting a volley of arrows</figcaption></figure>
<p>Archers need to know what direction to fire an arrow to hit a moving target. I
originally tried to derive the maths for this myself, but ultimately found
<a href="https://en.wikipedia.org/wiki/Projectile_motion#Angle_%CE%B8_required_to_hit_coordinate_(x,_y)">an equation on Wikipedia</a>
that worked. This was by far the thing I found most challenging whilst creating
Conquer, it took a while to get it to work as expected.</p>
<p>Here’s the final code. It’s just the equation converted to Lua.</p>
<pre class="language-lua"><code class="language-lua"><span class="token keyword">local</span> <span class="token keyword">function</span> <span class="token function">calculate_projectile_direction</span><span class="token punctuation">(</span>from<span class="token punctuation">,</span> to<span class="token punctuation">,</span> initial_speed<span class="token punctuation">,</span> gravity<span class="token punctuation">)</span>
<span class="token comment">-- Turn this into a 2D problem by considering just the plane</span>
<span class="token keyword">local</span> delta <span class="token operator">=</span> to <span class="token operator">-</span> from
<span class="token keyword">local</span> x <span class="token operator">=</span> math<span class="token punctuation">.</span><span class="token function">sqrt</span><span class="token punctuation">(</span>delta<span class="token punctuation">.</span>x<span class="token operator">*</span>delta<span class="token punctuation">.</span>x <span class="token operator">+</span> delta<span class="token punctuation">.</span>z<span class="token operator">*</span>delta<span class="token punctuation">.</span>z<span class="token punctuation">)</span>
<span class="token keyword">local</span> y <span class="token operator">=</span> delta<span class="token punctuation">.</span>y
<span class="token keyword">local</span> x2 <span class="token operator">=</span> x<span class="token operator">*</span>x
<span class="token keyword">local</span> v2 <span class="token operator">=</span> initial_speed<span class="token operator">*</span>initial_speed
<span class="token comment">-- If there's an imaginary number, no solution is possible</span>
<span class="token keyword">local</span> square <span class="token operator">=</span> v2<span class="token operator">*</span>v2 <span class="token operator">-</span> gravity<span class="token operator">*</span><span class="token punctuation">(</span>gravity<span class="token operator">*</span>x2 <span class="token operator">+</span> <span class="token number">2</span><span class="token operator">*</span>y<span class="token operator">*</span>v2<span class="token punctuation">)</span>
<span class="token keyword">if</span> square <span class="token operator"><=</span> <span class="token number">0</span> <span class="token keyword">then</span>
<span class="token keyword">return</span> <span class="token keyword">nil</span>
<span class="token keyword">end</span>
<span class="token keyword">local</span> t_x <span class="token operator">=</span> gravity <span class="token operator">*</span> x
<span class="token keyword">local</span> t_y <span class="token operator">=</span> v2 <span class="token operator">-</span> math<span class="token punctuation">.</span><span class="token function">sqrt</span><span class="token punctuation">(</span>square<span class="token punctuation">)</span>
<span class="token keyword">local</span> factor <span class="token operator">=</span> t_x <span class="token operator">/</span> x
<span class="token keyword">local</span> u_y <span class="token operator">=</span> t_y
<span class="token keyword">return</span> vector<span class="token punctuation">.</span><span class="token function">normalize</span><span class="token punctuation">(</span>vector<span class="token punctuation">.</span><span class="token function">new</span><span class="token punctuation">(</span>delta<span class="token punctuation">.</span>x <span class="token operator">*</span> factor<span class="token punctuation">,</span> u_y<span class="token punctuation">,</span> delta<span class="token punctuation">.</span>z <span class="token operator">*</span> factor<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">end</span></code></pre>
<p>The above function returns the direction to fire an arrow to hit a stationary
target. To hit a moving target, I predict the target’s position at a set time
in the future:</p>
<pre class="language-lua"><code class="language-lua"><span class="token keyword">local</span> guessed_time <span class="token operator">=</span> vector<span class="token punctuation">.</span><span class="token function">distance</span><span class="token punctuation">(</span>from<span class="token punctuation">,</span> target<span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">0.11</span>
<span class="token keyword">local</span> offset_target <span class="token operator">=</span> target <span class="token operator">+</span> target_velocity <span class="token operator">*</span> guessed_time</code></pre>
<p>This is a simple approach that works a lot of the time but is flawed as the way
it estimates the flight time using distance isn’t accurate. It’s good enough for
my purposes, however; archers occasionally missing could be considered a feature!</p>
<h2 id="buildings-and-nodeobjectref" tabindex="-1">Buildings and NodeObjectRef <a class="header-anchor" href="#buildings-and-nodeobjectref">#</a></h2>
<p>In Conquer, barracks produce new units, farms produce food, and the keep must be
defended at all costs. The appearance of each building changes based on its
health. Buildings regenerate health when no enemy units are nearby.</p>
<figure class="undefined"><a href="/static/img/hJrVSykgiK-1218.png"><picture><source type="image/webp" srcset="/static/img/C9s_OFtc2K-540.webp 540w"><img alt="Screenshot. A damaged keep with a tower missing, and a damage barracks." loading="lazy" decoding="async" src="/static/img/C9s_OFtc2K-540.png" width="540" height="359"></picture></a><figcaption>A damaged keep and barracks</figcaption></figure>
<h3 id="node-objectref" tabindex="-1">Node ObjectRef <a class="header-anchor" href="#node-objectref">#</a></h3>
<p>An object in Minetest is a moving thing with a position and velocity. For
example, both players and Conquer units are objects in the world. In contrast, a
node is a static thing in the voxel grid. Grass, keeps, and barracks are
examples of nodes.</p>
<p>To interact with an object using the Lua API, you use an ObjectRef. ObjectRefs
have methods giving you access to the position, velocity, and hp. Nodes do not
have ObjectRefs as they are not objects.</p>
<p>Conquer units need to be able to attack enemy units and nodes. Whilst I could
have implemented these abilities separately, I decided instead to create a Node
ObjectRef, a bit of Lua code that allows interacting with a building node
as if it were an object.</p>
<p>This is a nice abstraction that vastly simplifies the code. I intentionally only
implemented a subset of the ObjectRef interface: <code>get_pos</code>, <code>get_velocity</code>,
<code>get_hp</code>, and <code>punch</code>.</p>
<pre class="language-lua"><code class="language-lua"><span class="token keyword">local</span> node_object <span class="token operator">=</span> conquer<span class="token punctuation">.</span><span class="token function">create_target</span><span class="token punctuation">(</span>vector<span class="token punctuation">.</span><span class="token function">new</span><span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment">-- Can now interact with the node like an object</span>
node_object<span class="token punctuation">:</span><span class="token function">get_pos</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
node_object<span class="token punctuation">:</span><span class="token function">get_velocity</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
node_object<span class="token punctuation">:</span><span class="token function">punch</span><span class="token punctuation">(</span>unit_entity<span class="token punctuation">,</span> <span class="token number">0.5</span><span class="token punctuation">,</span> caps<span class="token punctuation">)</span></code></pre>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>The mod jam received 16 entries. After the deadline, the organisers set up a
server to play all the mods - it was chaos and a lot of fun. I ran a server just
for Conquer as a flat area was needed to play on. I also didn’t get around to
implementing persistence before the jam deadline, so other mods crashing the
shared server would have caused ongoing sessions to be lost.</p>
<!-- Conquer was 4,200 lines of code written in a week. -->
<p>I continued to develop Conquer after the jam, adding multi-select and other
quality-of-life features. The next big thing to implement is squad movement,
allowing a selection of units to move as one and avoid occupying the same
position.</p>
<p>Thanks to <a href="https://www.greenxenith.com/">GreenXenith</a> and others for organising
the mod jam. It was a pretty cool experience and had a good atmosphere to it. It
was a lot more successful than I expected.
You can <a href="https://content.minetest.net/packages/?tag=jam_combat_mod">find other jam entries on ContentDB</a>.</p>
<p>After the success of the mod jam, GreenXenith and others arranged game jams in
2021 and 2022. These were open to the wider Minetest community and proved even
more successful. I was a judge on the game jams and didn’t enter.</p>
<!--
Order of work:
* Sessions
* Keep and UnitEntity
* Unit selection and movement
* Pathfinding
* Unit types: knight vs archer
* HUD and actions
* Behaviour trees
* Melee
* Keep GUI
* Training units
* Build mode HUD
* Arrows and ranged attack
* Units defend themselves
* Archer auto attack
* Damaging buildings
* Defeating players
Post jam
* Multiselection
* Farm building
* Help GUI
* Buildings cost resources
* Persistance
* Healthbars
-->
rubenwardy
Thoughts on native Android development
2023-05-04T00:00:00.000Z
2023-05-04T00:00:00.000Z
https://blog.rubenwardy.com/2023/05/04/android/
This article is a reflection on the projects I've worked on, the problems I have, and what I want to try in the future.
<p>I love making apps. I like making stuff and it’s even better when I can interact
with the stuff I make. App development feels especially real as I can hold and
touch what I made. So why don’t I have many Android hobby projects? This article
is a reflection on the projects I’ve worked on, the problems I have, and what I
want to try in the future.</p>
<h2 id="my-background-in-android-dev" tabindex="-1">My background in Android dev <a class="header-anchor" href="#my-background-in-android-dev">#</a></h2>
<h3 id="minetest-mods" tabindex="-1">Minetest Mods <a class="header-anchor" href="#minetest-mods">#</a></h3>
<figure class="float-right mt-0 mr-0 ml-6 mb-2"><a href="/static/img/vM0G7svjPn-164.png"><picture><source type="image/webp" srcset="/static/img/PiOqU28Q7s-164.webp 164w"><img alt="App screenshot, a list of mod titles and descriptions" loading="lazy" decoding="async" src="/static/img/PiOqU28Q7s-164.png" width="164" height="164"></picture></a><figcaption>Minetest Mods App</figcaption></figure>
<p>I learned native Android development in 2016 to create an
<a href="/2022/10/22/Minetest-mods-app/">app to install mods for Minetest</a>. I published
Minetest Mods to the Play Store and maintained it there for several years,
unpublishing it only after I added built-in support for downloading mods inside
<a href="https://www.minetest.net">Minetest</a>.</p>
<p>I took great pride in ensuring the app followed
<a href="https://m2.material.io/design/introduction">Material Design</a> and had a good
user experience, and was rewarded with an average rating of 4.3 stars. The
rating was reduced by the fact that it only supported mods, and not games or
texture packs. There were also a lot of issues with flawed data,
<a href="/2022/03/24/contentdb/">I created a new backend called ContentDB</a> to solve this
issue, ultimately leading to the obsolescence of the app.</p>
<div class="clearfix"></div>
<h3 id="first-job" tabindex="-1">First job <a class="header-anchor" href="#first-job">#</a></h3>
<figure class="float-right mt-0 mr-0 ml-6 mb-2"><a href="/static/img/8ZsWHt9vGa-164.png"><picture><source type="image/webp" srcset="/static/img/xxQW6bbZo6-164.webp 164w"><img alt="App screenshot, a cow banner image and a list of news articles" loading="lazy" decoding="async" src="/static/img/xxQW6bbZo6-164.png" width="164" height="164"></picture></a><figcaption>BCVA App</figcaption></figure>
<p>My first job was doing Android, iOS, and backend development at a small software
house alongside my studies. It was awesome to get paid for doing the thing I
loved! Compare that to my job before, where I worked in hospitality for not much
money.</p>
<p>I worked on six different client projects, including two iOS apps and five
Android apps. Most of the team had a background in web design, so I was able to
play a leading role in the technical direction of the projects including pushing
for the adoption of best practices like continuous integration and improved
authentication (OAuth2).</p>
<div class="clearfix"></div>
<h3 id="hobby-apps" tabindex="-1">Hobby apps <a class="header-anchor" href="#hobby-apps">#</a></h3>
<figure class="float-right mt-0 mr-0 ml-6 mb-2"><a href="/static/img/8Ad-811h2d-164.png"><picture><source type="image/webp" srcset="/static/img/Q1qxcyKhPv-164.webp 164w"><img alt="App screenshot, a pie chart and a graph showing spending" loading="lazy" decoding="async" src="/static/img/Q1qxcyKhPv-164.png" width="164" height="164"></picture></a><figcaption>Analytics for Monzo</figcaption></figure>
<p>I did other hobby Android apps but ultimately didn’t complete them due to
issues.</p>
<p>Analytics for Monzo was an app to show pretty charts for Monzo bank users, but I
was unable to finish it due to a restricted API. With a banking API? Who would
have thought?!</p>
<p><a href="/2020/09/13/return-to-android-dev/">ForumMate</a> was a forum aggregator for
old-style bulletin board forums, but problems with getting it to work with all
forums and concerns about legality led me to abandon the project. I started the
project as an excuse to get back up to speed with the latest Android SDK
constructs, so it did serve its purpose.</p>
<div class="clearfix"></div>
<h3 id="current-job" tabindex="-1">Current job <a class="header-anchor" href="#current-job">#</a></h3>
<p>My current project at work is an Android app used by millions of people. It’s
written mostly in Java with a mixture of modern and legacy APIs. We are slowly
rewriting the app into Kotlin.</p>
<p>One of the things that stands out about this project is that it’s the first time
I’ve worked on something where I know people that use it. At previous jobs, I
mostly worked on niche apps and internal tools for aerospace/transport.</p>
<p>The large user base also allows small changes to make a huge difference to
users. I noticed a UX pain point in the app - an icon being used incorrectly. I
proposed and implemented the fix for this, which increased the discoverability
of an essential feature from a success rate of 26% to 92%. That’s a huge
difference from such a small change.</p>
<h2 id="so-what-s-the-problem-then" tabindex="-1">So, what’s the problem then? <a class="header-anchor" href="#so-what-s-the-problem-then">#</a></h2>
<p>I’m very project-driven; I’ve rarely learned technology just for the sake of it,
it’s always because I have an idea I want to make. I learned React to create
<a href="https://renewedtab.com">Renewed Tab</a> because I couldn’t find a New Tab
extension that worked how I wanted. The fact is that there are not many things
that need an Android app rather than a web app. Web apps can be used on a much
wider range of platforms, including phones. Web apps are also much easier to
make.</p>
<p>This cross-platform issue is a big one - I rarely write applications for just
Linux or Windows, I always prefer to use cross-platform toolkits. Even if I only
publish to Linux, the platform I use, I still choose toolkits that are
cross-platform. So why do I treat mobile app development differently?</p>
<p>Another issue is that the Android SDK is especially painful to deal with. It’s
complicated and there are so many ways to do things; it’s not always clear which
should be used. Java as a language leads to boilerplate code, the introduction
of Kotlin made a big difference in cutting this down. I do enjoy native Android
development, the costs are just high compared to other options.</p>
<h2 id="to-the-future" tabindex="-1">To the future <a class="header-anchor" href="#to-the-future">#</a></h2>
<h3 id="cross-platform-toolkits" tabindex="-1">Cross-platform toolkits <a class="header-anchor" href="#cross-platform-toolkits">#</a></h3>
<p>Cross-platform toolkits allow you to develop for Android and iOS from the same
codebase. Some even allow targetting desktop as well. This is appealing as it
lowers the development effort when creating apps and reduces the time to market.</p>
<p>I have historically been quite sceptical of cross-platform app development
toolkits as I got the impression that they gave subpar results.</p>
<p>First, I saw these toolkits as being quite janky and hard to make a good user
experience in. This may just be because my exposure has been with thin wrappers
over webviews, on badly coded apps. Modern toolkits like React Native and
Flutter make it much easier to create good user experiences.</p>
<p>Second, I didn’t think these toolkits would be able to match the appearance and
functionality of native apps. This is something I’d like to look into, but I
will note that it is very common for Android apps to break material guidelines
and instead look like iOS apps. Companies like to avoid doing the same work
multiple times. Just because an app is native doesn’t mean that it provides a
perfect native experience, it may be easier to provide a good experience using
cross-platform toolkits.</p>
<figure class="figure ">
<blockquote class="mb-1" cite="https://flutter.dev/multi-platform/mobile"><p>Flutter empowers you to create tailored experiences for iOS and Android users
without compromise and with fewer resources.</p>
</blockquote>
<figcaption class="pl-4 mt-3">
—<a href="https://flutter.dev/multi-platform/mobile">
flutter.dev
</a></figcaption>
</figure>
<h3 id="ideas-that-work-well" tabindex="-1">Ideas that work well <a class="header-anchor" href="#ideas-that-work-well">#</a></h3>
<p>I need to look out for use cases and ideas I have that would suit an app. I
don’t want to start making things just for the sake of it. Apps are best suited
for ideas that require deeper integration with the device or a user experience
that web apps are unable to provide.</p>
<p>In 2019, I started the <a href="/2020/09/13/return-to-android-dev/">ForumMate</a> project
as an excuse to get back up to speed with the latest Android SDK constructs -
including MVVM, Room, repositories, Dagger, data binding, and Jetpack
navigation. I choose this idea as it had enough complexity to be an interesting
project.</p>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>Hopefully this article wasn’t too negative, it’s not meant to be. I do enjoy
native Android dev, but I am wary of the costs especially when working on
projects by myself. Native development may still have its place on bigger
projects.</p>
<p>The next time I have an idea for an app, I’ll use it as an opportunity to learn
a cross-platform toolkit like React Native or Flutter. If you have any thoughts
on React Native, Flutter, or other cross-platform toolkits, it would be great to
see replies.</p>
rubenwardy
Bing (and DuckDuckGo) shadow banned my website
2023-04-26T00:00:00.000Z
2023-04-26T00:00:00.000Z
https://blog.rubenwardy.com/2023/04/26/bing-ban/
My domain was shadow banned by Bing and DuckDuckGo last year, and I don't know why. My root domain has since been unbanned, but my blog remains banned.
<p class="alert is-secondary">
Bing has now unbanned my blog. My site is still penalised/low in the
results, but at least it's appearing now. See the bottom of this article
for their response.
</p>
<p>My domain was shadow banned by Bing and DuckDuckGo last year, and I don’t know
why. Since then, my root domain (rubenwardy.com) has been unbanned but my blog
remains banned. The event also negatively impacted the search placement of my
root domain; another site that reuploaded some of my content is appearing as the
first result when searching for it.</p>
<h2 id="events" tabindex="-1">Events <a class="header-anchor" href="#events">#</a></h2>
<h3 id="june-2022-discovered-my-domain-was-banned" tabindex="-1">June 2022: Discovered my domain was banned <a class="header-anchor" href="#june-2022-discovered-my-domain-was-banned">#</a></h3>
<p>Last year, I received a message from a user saying that my website wasn’t
appearing on DuckDuckGo. Looking into it, I discovered that DuckDuckGo uses Bing
and my website wasn’t appearing there either.</p>
<figure class="undefined"><a href="/static/img/8XQapg-2g2-821.png"><picture><source type="image/webp" srcset="/static/img/OtTYdHO5ff-540.webp 540w"><img alt="There are no results for site:blog.rubenwardy.com" loading="lazy" decoding="async" src="/static/img/OtTYdHO5ff-540.png" width="540" height="147"></picture></a><figcaption>There are no results for site:blog.rubenwardy.com</figcaption></figure>
<p>The first thing I did was sign all my domains up to Bing Webmaster. The
dashboards were empty, and using the URL inspection tool resulted in the
following generic error:</p>
<figure class="undefined"><a href="/static/img/77Nib4ZmP7-997.png"><picture><source type="image/webp" srcset="/static/img/Xvl4hknYGk-540.webp 540w"><img alt="Screenshot of Bing URL inspection for my blog. The page says: "Discovered but not crawled. URL cannot appear on Bing. The inspected URL is known to Bing but has some issues which are preventing indexation. We recommend you to follow Bing Webmaster Guidelines to increase your chances of indexation."" loading="lazy" decoding="async" src="/static/img/Xvl4hknYGk-540.png" width="540" height="371"></picture></a><figcaption>Unhelpful webmaster URL inspection</figcaption></figure>
<p>I’ve read through the Bing Webmaster Guidelines and I’m certain that my website
follows them.</p>
<p>rubenwardy.com hosts <a href="https://rubenwardy.com/">my portfolio</a>, blog, and several
open-source projects - including
<a href="https://rubenwardy.com/minetest_modding_book/en/index.html">the book</a> I wrote
about creating mods for Minetest. At this point last year, my best guess for the
cause of the ban was that Bing incorrectly thought that I was hosting stolen
content, as another domain had reuploaded the book. Unfortunately, the book is
licensed under CC BY-SA 4.0, which means I am unable to use copyright to take
it down.</p>
<h3 id="june-july-contacted-bing-support" tabindex="-1">June-July: Contacted Bing support <a class="header-anchor" href="#june-july-contacted-bing-support">#</a></h3>
<p>At this point, I sent an email to Bing support explaining the situation
(UCM000004402991). After a week, I got the following response:</p>
<blockquote>
<p>Thank you for writing to Bing Webmaster Support. I have reviewed your site,
and I have now reached out to our Product Review Group for further analysis.</p>
<p>In the meantime, you can go through our Bing Webmaster guidelines to check if
your site is violating any of the rules cited in Bing Webmaster Guidelines,
especially the section Things to Avoid, and thoroughly check your site for any
deliberately or accidentally employed SEO techniques that may have adversely
affected your standing in Bing and Bing-powered search results.</p>
</blockquote>
<p>Googling this, I’ve found the exact same text sent to someone else, so it’s a
copy-paste response.</p>
<p>I once again went through the Bing Webmaster Guidelines, but could not find
anything that I was doing wrong. After a month, I chased up the email and got
the following:</p>
<blockquote>
<p>Our engineering team is still looking into this issue. I will keep you posted
on this case. Please allow some time on this request</p>
</blockquote>
<h3 id="october-rubenwardy-com-unbanned-blog-rubenwardy-com-remains-banned" tabindex="-1">October: rubenwardy.com unbanned, blog.rubenwardy.com remains banned <a class="header-anchor" href="#october-rubenwardy-com-unbanned-blog-rubenwardy-com-remains-banned">#</a></h3>
<p>Without any update from Bing support, rubenwardy.com started appearing in the
results albeit heavily penalised. My blog has since remained banned, with the
same error message.</p>
<p>Given this, it seems like the problem may be with the blog rather than my root
domain and the reuploaded content issue. I’m not sure what my blog could be
doing wrong.</p>
<p>I followed up with Bing support, and they replied saying they have been
following up with the engineering team and will keep me posted, but I never
received an update.</p>
<h2 id="ideas-for-why" tabindex="-1">Ideas for why <a class="header-anchor" href="#ideas-for-why">#</a></h2>
<h3 id="reuploaded-content" tabindex="-1">Reuploaded content <a class="header-anchor" href="#reuploaded-content">#</a></h3>
<p>My original idea was that it was due to another site reuploading my modding book
and being considered the original version. But given that the root domain was
unbanned and the blog domain wasn’t, this looks less likely.</p>
<p>I have now added canonical tags to most of my pages. Unfortunately, the owner of
the other site is malicious so I cannot ask them to fix the problem. The book is
licensed under CC BY-SA 4.0, which means I am unable to use copyright to take it
down.</p>
<h3 id="lots-of-inbound-links" tabindex="-1">Lots of inbound links <a class="header-anchor" href="#lots-of-inbound-links">#</a></h3>
<p>I’m quite active online and tend to put my website in my profiles and forum
signatures. This has resulted in a lot of inbound links, especially from
minetest.net, which may have resulted in Bing considering it link spam.</p>
<p>I <a href="/2022/03/24/contentdb/">developed and host ContentDB</a>, Minetest’s website to
find games, mods, and texture packs. I placed a link to rubenwardy.com in
ContentDB’s footer as a copyright attribution without realising the SEO
implications. As ContentDB has over 14,000 pages, this has resulted in a huge
number of inbound links. I have now replaced that footer link with one to an
<a href="https://content.minetest.net/about/">about page</a> that links to my site,
this will result in a single quality link rather than thousands of poor-quality
links. But these links were to rubenwardy.com, not blog.rubenwardy.com - this
doesn’t explain why my blog is still banned.</p>
<figure class="undefined"><a href="/static/img/-FD924P4v0-442.png"><picture><source type="image/webp" srcset="/static/img/FYNyPQH8Wt-442.webp 442w"><img alt="Google Search Console showing a large number of inbound links from minetest.net." loading="lazy" decoding="async" src="/static/img/FYNyPQH8Wt-442.png" width="442" height="159"></picture></a><figcaption>Google Search Console showing a large number of inbound links from minetest.net.</figcaption></figure>
<h3 id="html5-validation-issues" tabindex="-1">HTML5 Validation issues <a class="header-anchor" href="#html5-validation-issues">#</a></h3>
<p>I ran an HTML5 validator on my blog and main website, and fixed a couple of
small mistakes.</p>
<h3 id="weird-urls" tabindex="-1">Weird URLs <a class="header-anchor" href="#weird-urls">#</a></h3>
<p>Google Search Console alerted me about a large number of Not Found (404) URLs.
These URLs are bizarre and look like spam. Another site must be creating these
links to me, perhaps they’re hoping my site is badly programmed and will return
a soft not found with the links present. After searching, I found some
<a href="https://support.google.com/webmasters/thread/205198696/email-from-google-page-indexing-issue?hl=en">threads which confirmed this</a>.
As I correctly return a 404, there’s nothing for me to do here.</p>
<figure class="undefined"><a href="/static/img/sGXEdnOYGk-933.png"><picture><source type="image/webp" srcset="/static/img/T6yQsH2qe8-540.webp 540w"><img alt="Screenshot of Google Search Console showing a large number of links with Chinese text and a URL" loading="lazy" decoding="async" src="/static/img/T6yQsH2qe8-540.png" width="540" height="272"></picture></a><figcaption>Chinese link spam in Google Search Console</figcaption></figure>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>My blog (blog.rubenwardy.com) remains banned, and the event has negatively
impacted the search placement of my root domain; another site that reuploaded
some of my content is appearing as the first result when searching for it. This
is especially annoying as it’s the most popular thing on my website, many users
go looking for it and instead find an outdated version on a dodgy domain.</p>
<p>It appears that Bing banning small sites may be a common issue, I’ve found other
people reporting that their site was similarly effected:</p>
<ul>
<li><a href="https://daverupert.com/2023/01/shadow-banned-by-duckduckgo-and-bing/">I’m Shadow Banned by DuckDuckGo (and Bing) - Dave Rupert</a></li>
<li><a href="https://dariusz.wieckiewicz.org/en/bing-jail/">Bing Jail - Dariusz Więckiewicz</a></li>
</ul>
<p>There are potentially over a billion websites on the Internet, Bing needs to
index and rank them whilst avoiding abuse. Some websites contain illegal or
harmful content, others attempt to manipulate the search results. Given the
scale of the problem, it is understandable for Bing to sometimes make mistakes.</p>
<p>In the end, Dave Rupert discovered that Bing was
<a href="https://daverupert.com/2023/02/solved-the-case-of-the-bing-ban-theory/">incorrectly classifying his site as a spam blog</a>.
A spam blog is a website created purely to link to and promote other websites,
often with useless information. Dave was able to fix his problem are blogging
about it and raising awareness, so hopefully this post will be able to resolve
my problems. I’d appreciate suggestions and help resolving this issue.</p>
<h2 id="update-my-blog-has-been-unbanned" tabindex="-1">Update: My blog has been unbanned <a class="header-anchor" href="#update-my-blog-has-been-unbanned">#</a></h2>
<p>Bing has now unbanned my blog. My site is still penalised/low in the results,
but at least it’s appearing now</p>
<blockquote>
<p>Good day and thank you for your patience while we investigated the issue.</p>
<p>Firstly, I am happy to provide you information that our Product Review Group
succeeded in resolving the issue which prevented your site from showing in our
index. After submitting your site to be reviewed, the team has decided to lift
the block. Allow up to two to three weeks for your site to be crawled,
indexed, and serving again.</p>
<p>I am unable to provide you the specifics of the block, as our Product Review
team does not share the details of the block. However, I wish to help you with
a few > links and help docs, which will not only help in avoiding blocks, but
also help you to rank better in the Bing index.</p>
<ul>
<li><a href="https://www.bing.com/webmasters/help/why-is-my-site-not-in-the-index-2141dfab">Why is My Site Not in the Index?</a></li>
<li><a href="https://blogs.bing.com/webmaster/2014/05/09/10-seo-myths-reviewed">10 SEO Myths Reviewed</a></li>
<li><a href="https://blogs.bing.com/webmaster/2014/04/24/is-seo-the-future-no-and-heres-why">Is SEO The Future? No, And Here’s Why</a></li>
<li><a href="https://blogs.bing.com/webmaster/2014/10/17/building-authority-setting-expectations">Building Authority & Setting Expectations</a></li>
<li><a href="https://blogs.bing.com/search-quality-insights/2014/12/08/the-role-of-content-quality-in-bing-ranking">The Role of Content Quality in Bing Ranking</a></li>
</ul>
<p>When Bing crawls and indexes pages on the Internet, it makes decisions about
which pages will most likely resonate with searchers, so we look at these
signals relative to all other sites on the web as part of our selection
criteria. Always ensure you follow webmaster guidelines, especially the
section <a href="https://www.bing.com/webmasters/help/webmasters-guidelines-30fba23a">Things to Avoid</a>.</p>
<p>Let me know if you have any follow-up questions or concerns regarding this
issue and I would be more than happy to assist.</p>
<p>Have a wonderful day ahead!</p>
</blockquote>
rubenwardy
3D-printed IoT lock box using ESP32
2023-03-16T00:00:00.000Z
2023-03-16T00:00:00.000Z
https://blog.rubenwardy.com/2023/03/16/lockbox/
In this article, I will cover how I created my lock box - from the problems I had, the iterations I made, and the final design I settled on.
<p>My eventual goal with electronics is to create autonomous robots and drones; I’d
like to make a quadcopter with my own flight controller that can take off, land,
and follow a target. This will be quite an ambitious project, and I’m nowhere
near capable enough for that yet.</p>
<p>Previously, I created a simple <a href="/2022/03/17/plant-monitor/">plant monitor</a> that
reported stats to an online dashboard. This allowed me to learn soldering,
stripboards, and 3d printing. To work on future projects, I need to
be able to produce ever more complicated circuits and mechanical designs.</p>
<p>After watching some Lock Picking Lawyer on YouTube, I was inspired to think
about different locking mechanisms. A locking box would be a good experiment
with mechanical design, and if combined with IoT, would be good for electronics
too.</p>
<p>In this article, I will cover how I created my lock box - from the problems I
had, the iterations I made, and the final design I settled on. It’s a bit of a
random project, I didn’t have set goals in mind. I started by exploring
different locking mechanisms, and then moved to focusing on the electronics and
getting it working.</p>
<h2 id="initial-servo-designs" tabindex="-1">Initial servo designs <a class="header-anchor" href="#initial-servo-designs">#</a></h2>
<h3 id="pin" tabindex="-1">Pin <a class="header-anchor" href="#pin">#</a></h3>
<p>My initial idea was to use a servo to move a pin to unlock the box. Whilst I
could have used the servo head as part of the lock mechanism directly, I didn’t
want the servo to be a load-bearing part of the lock.</p>
<p>My first design was to use a pin, a paperclip pushrod, and a two-part slot. I
found it quite hard to get the pin to move straight without falling out at the
end. The slightest bit of misalignment in the two latches also caused a lot of
friction on the pin. Due to these problems, I decided to change to a design that
didn’t need as much horizontal movement.</p>
<figure class="undefined"><a href="/static/img/yaE0GVZnFj-640.jpeg"><picture><source type="image/webp" srcset="/static/img/jwcbPAuq_1-540.webp 540w"><img alt="Servo-based lock design using a sliding pin." loading="lazy" decoding="async" src="/static/img/jwcbPAuq_1-540.jpeg" width="540" height="405"></picture></a><figcaption>Servo-based lock design using a sliding pin.</figcaption></figure>
<h3 id="e-shaped-gripper" tabindex="-1">E-shaped gripper <a class="header-anchor" href="#e-shaped-gripper">#</a></h3>
<p>My second design used a servo to move an E-shaped gripper. When in the locked
position, the gripper blocks the lid from being removed by colliding with tabs
on the side of the lid.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/npmYo2I-wK-640.webp"><img alt="Servo-based lock design using E-shaped gripper." loading="lazy" decoding="async" src="/static/img/ywwmIAcSO1-640.webp" width="640" height="10440"></a><figcaption>Servo-based lock design using E-shaped gripper.</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/VlxKWXmdbw-224.jpeg"><picture><source type="image/webp" srcset="/static/img/F61v5Olkba-224.webp 224w"><img alt="Tabs on the lid." loading="lazy" decoding="async" src="/static/img/F61v5Olkba-224.jpeg" width="224" height="360"></picture></a><figcaption>Tabs on the lid.</figcaption></figure></div>
</div>
<p>The nice thing about this design is that it didn’t need much calibration, the
servo just needed to be in the right place to move the gripper.</p>
<p>However, the servo I used was pretty unreliable - it would frequently do
360° rotations and break the lock. I also had problems with removing the
lid, there was way too much flex in the lid and base, causing it to get stuck at
weird angles.</p>
<p>Another problem is that I designed the case before the electronics, and
underestimated the amount of space I’d need for them. Increasing the size of the
case to fix the electronics would exacerbate the problems further, I need to
come up with a design with fewer moving parts.</p>
<h2 id="solenoid-design" tabindex="-1">Solenoid design <a class="header-anchor" href="#solenoid-design">#</a></h2>
<p>I had bought a
<a href="https://thepihut.com/products/small-lock-style-solenoid-6vdc-600mah-with-2-pin-jst">cheap solenoid lock from PiHut</a>
a few months ago, so I decided to redesign the box using it. I also decided to
get the full electronic design done first and then design around that, allowing
the case to correctly fit the electronics.</p>
<h3 id="electronics" tabindex="-1">Electronics <a class="header-anchor" href="#electronics">#</a></h3>
<p>The electronics were soldered onto two stripboards, with 22-gauge wire
connecting them.</p>
<p>The main board has the ESP32 microcontroller, Real-Time Clock, MicroSD card
reader, and solenoid driver. I included an SDCard reader to store configuration
data and for logging; this is something I wanted to learn how to do as it will
be much more useful in future projects. The solenoid driver consists of a power
MOSFET and a flyback diode.</p>
<p>I included a separate MicroUSB port to act as the power supply. Using the
microcontroller’s MicroUSB port would limit the amount of current available to
the solenoid and would also allow the lock to be bypassed by reprogramming the
microcontroller. The MicroUSB module is connected to the main board using
22-gauge wire.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/wE64tFXAym-1650.png"><picture><source type="image/webp" srcset="/static/img/M5gB6FdZWU-540.webp 540w"><img alt="Electronics schematic, created using KiCAD. The ESP32 has builtin pull-up resistors." loading="lazy" decoding="async" src="/static/img/M5gB6FdZWU-540.png" width="540" height="443"></picture></a><figcaption>Electronics schematic, created using KiCAD. The ESP32 has builtin pull-up resistors.</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/b-H81bRaN8-360.jpeg"><picture><source type="image/webp" srcset="/static/img/WatW1D0CcQ-360.webp 360w"><img alt="Inside the lockbox." loading="lazy" decoding="async" src="/static/img/WatW1D0CcQ-360.jpeg" width="360" height="480"></picture></a><figcaption>Inside the lockbox.</figcaption></figure></div>
</div>
<h3 id="screen" tabindex="-1">Screen <a class="header-anchor" href="#screen">#</a></h3>
<p>The screen board contains an SSD1306 OLED screen and four push buttons.
3D-printed button caps rest on the buttons and allow them to be pressed through
the case. The lid has holes for the screens and the four button caps.</p>
<p>Creating the screen board was by far the hardest part of the project - space was
tight and once the screen was on it blocked access to the rest of the screen
board. At one point, the cheap soldering iron I was using overheated - it
started glowing red hot, indicating it was over 200°C hotter than it should be!
I bought a new soldering station, a Hakko FX-888D, which made soldering so much
easier.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/_Bdx-u19IT-640.jpeg"><picture><source type="image/webp" srcset="/static/img/w6S0xe4a4P-540.webp 540w"><img alt="The screen board." loading="lazy" decoding="async" src="/static/img/w6S0xe4a4P-540.jpeg" width="540" height="405"></picture></a><figcaption>The screen board.</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/8B5Gq29YQu-640.jpeg"><picture><source type="image/webp" srcset="/static/img/U3BYXeJcWM-540.webp 540w"><img alt="Underside of the lid with 4 button caps and the screen hole." loading="lazy" decoding="async" src="/static/img/U3BYXeJcWM-540.jpeg" width="540" height="405"></picture></a><figcaption>Underside of the lid with 4 button caps and the screen hole.</figcaption></figure></div>
</div>
<h3 id="case-and-lock" tabindex="-1">Case and lock <a class="header-anchor" href="#case-and-lock">#</a></h3>
<p>Rather than making the lid part of the lock mechanism, I added a sliding drawer
to hold the contents. Using a smaller moving part prevents the issues with the
plastic flexing. The drawer has a slot on the bottom for the solenoid lock, and
a lip to prevent picking.</p>
<p>The lid slides onto the base using tabs, locking it into place vertically. The
drawer slides into rails on the lid and base, locking the whole thing together.
The drawer prevents the lid from sliding horizontally and provides extra
strength vertically. Using the drawer to lock the lid in place is a simple and
elegant way to prevent access to the electronics whilst locked - it avoids
needing to use screws.</p>
<p>The case has a hole for the power cable and a small platform on the inside to
mount the MicroUSB module using screws.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/FNdQnmWSV_-860.png"><picture><source type="image/webp" srcset="/static/img/N2DkdlDZCq-540.webp 540w"><img alt="After maintenance, the lid is lowered and pushed horizontally to lock into place." loading="lazy" decoding="async" src="/static/img/N2DkdlDZCq-540.png" width="540" height="303"></picture></a><figcaption>After maintenance, the lid is lowered and pushed horizontally to lock into place.</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/qaHmR2PHcD-860.png"><picture><source type="image/webp" srcset="/static/img/0KYps6H3_s-540.webp 540w"><img alt="The drawer slides into the slot, preventing the lid from being removed horizontally." loading="lazy" decoding="async" src="/static/img/0KYps6H3_s-540.png" width="540" height="303"></picture></a><figcaption>The drawer slides into the slot, preventing the lid from being removed horizontally.</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/5qYAq21RiH-860.png"><picture><source type="image/webp" srcset="/static/img/gs7SmzhDqz-540.webp 540w"><img alt="The drawer also has tabs that prevent the lid from being removed vertically." loading="lazy" decoding="async" src="/static/img/gs7SmzhDqz-540.png" width="540" height="303"></picture></a><figcaption>The drawer also has tabs that prevent the lid from being removed vertically.</figcaption></figure></div>
</div>
<h2 id="software" tabindex="-1">Software <a class="header-anchor" href="#software">#</a></h2>
<p>The software is pretty minimal. ESP32 uses the Arduino C++ SDK so I was able to
benefit from existing libraries for components such as the display (SSD1306),
Real-Time Clock, and MicroSD card reader. One of my other projects is built
using the Pico C++ SDK and required me to write my own firmware for this
hardware.</p>
<p>When connected to a WiFi network, the microcontroller will synchronise the
current time using the Network Time Protocol. The time is stored in the
Real-Time Clock module, allowing the time to be known whilst offline.</p>
<p>The user can lock the device using a pin or set up a time-based lock. The device
could also use a REST API, but I didn’t get around to exploring this.</p>
<p>One thing that surprised me about working with the MicroSD card reader is just
how slow it was. It took 0.65 seconds to open a file, write a few bytes, and
then close it. Appending to a file was super fast, however, likely because the
cards are optimised for sequential access. This is probably just a software
problem. I used SD.h from the Arduino SDK, which uses sdfatlib, with a 32GB
MicroSD card formatted to FAT32.</p>
<h2 id="future-improvements" tabindex="-1">Future Improvements <a class="header-anchor" href="#future-improvements">#</a></h2>
<h3 id="increase-storage-space" tabindex="-1">Increase storage space <a class="header-anchor" href="#increase-storage-space">#</a></h3>
<p>The drawer is too small to be useful for much - it only really fits a key or
note, and not much else. This is just a prototype and a learning exercise, so it
doesn’t matter that much. A larger version could have quite a few use cases,
such as rewarding good habits.</p>
<p>The solenoid design is 15% larger than the servo design but has 70% less storage
space. Now, the old design didn’t really fit the electronics so it’s not a fair
comparison, but it highlights a problem with the size of the box vs the size of
the lockable storage space.</p>
<figure class="undefined"><a href="/static/img/I5jJo5PlPI-640.jpeg"><picture><source type="image/webp" srcset="/static/img/6yTrQnA7cX-540.webp 540w"><img alt="The two designs side-by-side." loading="lazy" decoding="async" src="/static/img/6yTrQnA7cX-540.jpeg" width="540" height="405"></picture></a><figcaption>The two designs side-by-side.</figcaption></figure>
<p>One way to solve this would be to miniaturize the electronics. A Pi Pico W would
have a smaller footprint and also has a built-in RTC, so the external RTC could
be removed. Making a PCB could make the electronics smaller, especially if you
embed the various breakouts onto it. Glueing the screen rather than using a
holder plate to screw it in would also save space.</p>
<p>Another option would be to make the whole thing bigger. The same amount of
electronics is required, no matter the size. So making it bigger will allow the
storage efficiency to be higher.</p>
<!--
old vol: 65533.46 mm³
new vol: 75292.361 mm³
old loc: 86904 71*68*(25-(13.2-6.2))
new loc: 24064 38.9*38.9*16.2-10*10*4.5
-->
<h3 id="battery" tabindex="-1">Battery <a class="header-anchor" href="#battery">#</a></h3>
<p>Another possible future improvement would be to add a small li-ion battery,
allowing the box to become more portable.</p>
<h3 id="limit-switch" tabindex="-1">Limit switch <a class="header-anchor" href="#limit-switch">#</a></h3>
<p>A limit switch could be used to automatically detect when the drawer is in the
box. This would be a small user-experience improvement.</p>
<h3 id="custom-locks" tabindex="-1">Custom locks <a class="header-anchor" href="#custom-locks">#</a></h3>
<p>It would be good to get my custom mechanical designs based on servos working.
Whilst there isn’t a huge amount of benefit to them, I am disappointed that I
wasn’t able to get them working correctly.</p>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>Well, I definitely learned a lot in this project. I probably won’t continue to
work on it as I’ve already spent more time on it than I was planning and I don’t
have much use for it.</p>
<p>I’ve been working on a new robotic car platform using a Pi Pico. The aim is for
it to be a platform for me to experiment with various robotic techniques,
including dead reckoning and
<a href="https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping">Simultaneous Localization and Mapping (SLAM)</a>.
I’d also like to learn how to use lipo batteries and RF transceivers.</p>
<p>Another project is likely to be an automatic watering station for my basil,
which needs to be watered daily. This would be a nice continuation of my
<a href="/2022/03/17/plant-monitor/">plant monitor</a> project.</p>
rubenwardy
My favourite programming and game dev articles
2023-02-17T00:00:00.000Z
2023-02-17T00:00:00.000Z
https://blog.rubenwardy.com/2023/02/17/interesting-reads/
Articles are a great way to expand your knowledge by learning from the experiences of others. Here is a collection of articles I recommend reading.
<p>Articles are a great way to expand your knowledge as a programmer, by learning
from the experiences of others. A good article is also interesting to read.
Here is a collection of articles I recommend reading.</p>
<h2 id="game-dev" tabindex="-1">Game dev <a class="header-anchor" href="#game-dev">#</a></h2>
<h3 id="the-simulation-dream" tabindex="-1">The Simulation Dream <a class="header-anchor" href="#the-simulation-dream">#</a></h3>
<p>Tynan Sylvester’s
<a href="https://tynansylvester.com/2013/06/the-simulation-dream/">The Simulation Dream</a>
explains how to create a deep simulated world without super complex systems that
would confuse players. It makes the point that the player’s perception of the
game is more important than what the game is actually doing.</p>
<p>Tynan is the creator of <a href="https://rimworldgame.com/">RimWorld</a>, my favourite game
of all time.</p>
<div class="col-md col-6 embed">
<a href="https://tynansylvester.com/2013/06/the-simulation-dream/" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><img class="my-0" src="/static/img/qvPDXNADPu-270.png" loading="lazy"
alt="The Simulation Dream"></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
The Simulation Dream
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Tynan Sylvester</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h3 id="behaviour-trees" tabindex="-1">Behaviour Trees <a class="header-anchor" href="#behaviour-trees">#</a></h3>
<p>Behaviour Trees are a nice way to construct game AIs. They allow you to control
an NPC’s decision-making by combining reusable nodes rather than creating new
states in a state machine. They’re great for making extensible
<a href="/2022/07/17/game-ai-for-colonists/">worker NPCs</a>. Below are the two articles
I used to learn how behaviour trees work.</p>
<div class="row gap-3">
<div class="col-md col-6 embed">
<a href="https://www.gamedeveloper.com/programming/behavior-trees-for-ai-how-they-work" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><img class="my-0" src="/static/img/FzwH3ULtVE-270.png" loading="lazy"
alt="Behavior trees for AI: How they work"></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
Behavior trees for AI: How they work
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Chris Simpson- GameDeveloper.com</p>
</div>
</div>
</div>
</div>
</a>
</div>
<p><div class="col-md col-6 embed">
<a href="https://takinginitiative.wordpress.com/2014/02/17/synchronized-behavior-trees/" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><img class="my-0" src="/static/img/4rQnFoVDDw-270.png" loading="lazy"
alt="Synchronized Behavior Trees"></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
Synchronized Behavior Trees
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Bobby Anguelov- TakingInitiative</p>
</div>
</div>
</div>
</div>
</a></p>
</div>
</div>
<h3 id="enemy-ai-in-mafia-iii" tabindex="-1">Enemy AI in Mafia III <a class="header-anchor" href="#enemy-ai-in-mafia-iii">#</a></h3>
<p>Here is an interesting article about how enemy NPCs work in Mafia III, with
stealth, cover, perception, tactics, and combat.</p>
<div class="col-md col-6 embed">
<a href="https://www.gamedeveloper.com/design/open-world-enemy-ai-in-mafia-iii" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><img class="my-0" src="/static/img/WhOlEhUJpx-270.png" loading="lazy"
alt="Open-world enemy AI in Mafia III"></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
Open-world enemy AI in Mafia III
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Jiri Holba- GameDeveloper.com</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h3 id="physics-for-game-programmers" tabindex="-1">Physics for Game Programmers <a class="header-anchor" href="#physics-for-game-programmers">#</a></h3>
<p>This is actually a talk rather than an article, but it was too good to leave out.
The creator of Box2D talks about how to implement a character controller with
swept collision and an iterative solver. It’s dense but super interesting and
the best resource I’ve found about building your own physics engine.</p>
<div class="col-md col-6 embed">
<a href="https://youtu.be/SHinxAhv1ZE" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><img class="my-0" src="/static/img/oJK0DQgegJ-270.png" loading="lazy"
alt="Physics for Game Programmers: Understanding Constraints"></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
Physics for Game Programmers: Understanding Constraints
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Erin Catto- YouTube</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h3 id="minetest-miniforts" tabindex="-1">Minetest MiniForts <a class="header-anchor" href="#minetest-miniforts">#</a></h3>
<p>This is one of the most technically impressive things I’ve seen made in Minetest.
It’s a base-building game where you command minions to build up the base and
defend it from attackers. Without spoiling too much, they use TypeScript,
their own JSX GUI API, data binding, and a hierarchical pathfinder.</p>
<p>I’m a big fan of RimWorld and have
<a href="/2022/07/17/game-ai-for-colonists/">incorporated mechanics from it in my own games</a>
before, so this is pretty cool to see.</p>
<div class="col-md col-6 embed">
<a href="https://leanrada.com/works/miniforts.html" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><img class="my-0" src="/static/img/lxxJpd6pNl-361.png" loading="lazy"
alt="MiniForts: A game about building and defending a fort"></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
MiniForts: A game about building and defending a fort
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Lean Rada</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h2 id="programming" tabindex="-1">Programming <a class="header-anchor" href="#programming">#</a></h2>
<h3 id="stack-overflow" tabindex="-1">Stack Overflow <a class="header-anchor" href="#stack-overflow">#</a></h3>
<p>Stack Overflow has a reputation for being unfriendly to those asking questions,
but it turns out that this is by design - SO’s primary purpose is to be a wiki
for programming answers, rather than a place to ask questions. This interesting
article by one of Stack Overflow’s founders explains the reasons behind the
site’s design.</p>
<div class="col-md col-6 embed">
<a href="https://blog.codinghorror.com/what-does-stack-overflow-want-to-be-when-it-grows-up/" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><img class="my-0" src="/static/img/qpnua-gFnT-270.png" loading="lazy"
alt="What does Stack Overflow want to be when it grows up?"></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
What does Stack Overflow want to be when it grows up?
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Jeff Atwood- Coding Horror</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h3 id="the-rewriting-curse" tabindex="-1">The Rewriting Curse <a class="header-anchor" href="#the-rewriting-curse">#</a></h3>
<p>Written by the other founder of Stack Overflow, this article discusses the
dangers of rewriting software from scratch.</p>
<div class="col-md col-6 embed">
<a href="https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><img class="my-0" src="/static/img/b3XBXAp83j-270.png" loading="lazy"
alt="Things You Should Never Do, Part I"></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
Things You Should Never Do, Part I
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Joel Spolsky- Joel on Software</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h3 id="yaml" tabindex="-1">YAML <a class="header-anchor" href="#yaml">#</a></h3>
<p>YAML is a nice, simple format for configuration, right? Wrong. YAML has a
shocking number of edge cases and foot guns.</p>
<div class="col-md col-6 embed">
<a href="https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell" class="hover-enlarge d-block no-underline">
<div class="card">
<div class="row align-items-center">
<div class="col col-basis-66 ratio-4x3 h-100"><img class="my-0" src="/static/img/tuGfWvbH_D-270.png" loading="lazy"
alt="The yaml document from hell"></div>
<div class="col">
<div class="card-body px-4 py-2">
<h5 class="card-title my-0">
The yaml document from hell
</h5>
<p class="card-text mb-0 mt-2 text-muted">
Ruud van Asseldonk</p>
</div>
</div>
</div>
</div>
</a>
</div>
<h3 id="honourable-mentions" tabindex="-1">Honourable Mentions <a class="header-anchor" href="#honourable-mentions">#</a></h3>
<ul>
<li><a href="https://findthatmeme.com/blog/2023/01/08/image-stacks-and-iphone-racks-building-an-internet-scale-meme-search-engine-Qzrz7V6T.html">Image Stacks and iPhone Racks - Building an Internet Scale Meme Search Engine</a></li>
<li><a href="https://nolanlawson.com/2019/11/05/what-ive-learned-about-accessibility-in-spas/">What I’ve learned about accessibility in SPAs</a></li>
<li><a href="https://ericwbailey.website/published/aria-label-is-a-code-smell/">aria-label is a code smell</a></li>
</ul>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>I hope you found these interesting. Feel free to reply with any articles or
talks you liked or think I might like.</p>
<p><a href="https://www.pexels.com/photo/clear-glass-teacup-with-coffee-beverage-2299028/">Cover Image</a>
© Engin Akyurt</p>
rubenwardy
SDL_GameController: Making gamepads just work
2023-01-24T00:00:00.000Z
2023-01-24T00:00:00.000Z
https://blog.rubenwardy.com/2023/01/24/using_sdl_gamecontroller/
When implementing controller support in a game, you want gamepads to just work. The solution is SDL_GameController, which allows you to target a large number of gamepads.
<p>When implementing controller support in a game, it’s desirable for gamepads to
just work without a lot of user configuration. Platform APIs are pretty useless
for this, the solution is an API like SDL_GameController that allows you to
target a large number of gamepads without much effort.</p>
<p>Each operating system has its own API for gamepad input. Windows has XInput, and
Linux has the joystick and evdev APIs. When a gamepad button is pressed,
applications will receive a button id. This is a number, there’s no OS way to
know which button id corresponds with which button. The ids for a button are not
the same on different gamepads and platforms, making it super hard to support
more than a couple of devices.</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">SDL_JoystickGetButton</span><span class="token punctuation">(</span>joystick<span class="token punctuation">,</span> <span class="token number">8</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
std<span class="token double-colon punctuation">::</span>cerr <span class="token operator"><<</span> <span class="token string">"no idea what button 8 is"</span> <span class="token operator"><<</span> std<span class="token double-colon punctuation">::</span>endl<span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>One thing platforms do give you is the name, model, and manufacturer
of the game controller. If you test with a large number of gamepads, you can
create a database from gamepad name to layout. Luckily, SDL_GameController
has already done this for you. Instead of a random number, you can use a named
button that will work no matter the gamepad and platform:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">SDL_GameControllerGetButton</span><span class="token punctuation">(</span>controller<span class="token punctuation">,</span> SDL_GameControllerButton<span class="token double-colon punctuation">::</span>SDL_CONTROLLER_BUTTON_X<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
std<span class="token double-colon punctuation">::</span>cerr <span class="token operator"><<</span> <span class="token string">"X was pressed!"</span> <span class="token operator"><<</span> std<span class="token double-colon punctuation">::</span>endl<span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<h2 id="what-is-sdl-gamecontroller" tabindex="-1">What is SDL_GameController? <a class="header-anchor" href="#what-is-sdl-gamecontroller">#</a></h2>
<p>SDL_GameController is an abstraction that allows you to program input based on
an Xbox-like controller layout, and have it work with a huge variety of devices.
It’s a layer built on top of the raw SDL_Joystick API.</p>
<p>Xbox-like controllers have a DPAD, two analog sticks, 4 buttons on the right
(often called A/B/X/Y), shoulder buttons, and 3 buttons in the middle
(start/back/logo). Examples include PlayStation DualShock, Nintendo Switch, and
Steam Deck (Neptune).</p>
<h3 id="mapping" tabindex="-1">Mapping <a class="header-anchor" href="#mapping">#</a></h3>
<p>SDL2 comes with a database of game controllers, mapping from controller id to
layout information. Users can also provide custom mappings, which is supported
without you needing to do anything.</p>
<p>Steam also comes with built-in support for SDL_GameController, allowing users to
remap their controllers for your game in Steam. If Steam supports the
controller, your game will too.</p>
<h2 id="using-sdl-gamecontroller" tabindex="-1">Using SDL_GameController <a class="header-anchor" href="#using-sdl-gamecontroller">#</a></h2>
<h3 id="setup" tabindex="-1">Setup <a class="header-anchor" href="#setup">#</a></h3>
<p>When initializing your SDL2 device, add the <code>SDL_INIT_GAMECONTROLLER</code> flag:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">SDL_Init</span><span class="token punctuation">(</span>SDL_INIT_VIDEO <span class="token operator">|</span> SDL_INIT_GAMECONTROLLER<span class="token punctuation">)</span> <span class="token operator"><</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
std<span class="token double-colon punctuation">::</span>cerr <span class="token operator"><<</span> <span class="token string">"SDL could not initialize! SDL Error: "</span> <span class="token operator"><<</span> <span class="token function">SDL_GetError</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator"><<</span> std<span class="token double-colon punctuation">::</span>endl<span class="token punctuation">;</span>
<span class="token keyword">return</span> <span class="token number">1</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>There are events for controllers connecting and disconnecting, but they’re
not fired for controllers that are already connected. So, before starting
the game loop, you’ll need to check for existing controllers:</p>
<pre class="language-cpp"><code class="language-cpp">SDL_GameController <span class="token operator">*</span><span class="token function">findController</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">int</span> i <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">;</span> i <span class="token operator"><</span> <span class="token function">SDL_NumJoysticks</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> i<span class="token operator">++</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">SDL_IsGameController</span><span class="token punctuation">(</span>i<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token function">SDL_GameControllerOpen</span><span class="token punctuation">(</span>i<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token keyword">return</span> <span class="token keyword">nullptr</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>For simplicity, we’ll only be setting up and tracking a single controller at a
time. But it’s a similar process to handle multiple controllers.</p>
<pre class="language-cpp"><code class="language-cpp">SDL_GameController <span class="token operator">*</span>controller <span class="token operator">=</span> <span class="token function">findController</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h3 id="controller-connection-and-removal" tabindex="-1">Controller connection and removal <a class="header-anchor" href="#controller-connection-and-removal">#</a></h3>
<p>Next, you’ll need to listen for the SDL_CONTROLLERDEVICEADDED and
SDL_CONTROLLERDEVICEREMOVED events in the SDL2 event handler:</p>
<pre class="language-c"><code class="language-c"><span class="token keyword">switch</span> <span class="token punctuation">(</span>event<span class="token punctuation">.</span>type<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">case</span> SDL_CONTROLLERDEVICEADDED<span class="token operator">:</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span>controller<span class="token punctuation">)</span> <span class="token punctuation">{</span>
controller <span class="token operator">=</span> <span class="token function">SDL_GameControllerOpen</span><span class="token punctuation">(</span>event<span class="token punctuation">.</span>cdevice<span class="token punctuation">.</span>which<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">break</span><span class="token punctuation">;</span>
<span class="token keyword">case</span> SDL_CONTROLLERDEVICEREMOVED<span class="token operator">:</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>controller <span class="token operator">&&</span> event<span class="token punctuation">.</span>cdevice<span class="token punctuation">.</span>which <span class="token operator">==</span> <span class="token function">SDL_JoystickInstanceID</span><span class="token punctuation">(</span>
<span class="token function">SDL_GameControllerGetJoystick</span><span class="token punctuation">(</span>controller<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token function">SDL_GameControllerClose</span><span class="token punctuation">(</span>controller<span class="token punctuation">)</span><span class="token punctuation">;</span>
controller <span class="token operator">=</span> <span class="token function">findController</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">break</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>The controller removed event gives us the Joystick Instance ID as which. To
check whether this is the same controller, we first need to get the SDL_Joystick
and then the ID for the controller.</p>
<h3 id="handling-input" tabindex="-1">Handling Input <a class="header-anchor" href="#handling-input">#</a></h3>
<p>There are two ways of receiving input; you can either use events or you can
poll for input state. The following events are available:</p>
<ul>
<li><a href="https://wiki.libsdl.org/SDL2/SDL_ControllerDeviceEvent">Device Events</a>:
<ul>
<li><code>SDL_CONTROLLERDEVICEADDED</code>: A controller was added.</li>
<li><code>SDL_CONTROLLERDEVICEREMOVED</code>: A controller was removed.</li>
<li><code>SDL_CONTROLLERDEVICEREMAPPED</code>: A controller was remapped, you can mostly
ignore this event unless you use raw joysticks.</li>
</ul>
</li>
<li><a href="https://wiki.libsdl.org/SDL2/SDL_ControllerButtonEvent">Button Events</a>:
<ul>
<li><code>SDL_CONTROLLERBUTTONDOWN</code>: A button was pressed on a controller.</li>
<li><code>SDL_CONTROLLERBUTTONUP</code>: A button was released on a controller.</li>
</ul>
</li>
<li><a href="https://wiki.libsdl.org/SDL2/SDL_ControllerAxisEvent">Axis Events</a>:
<ul>
<li><code>SDL_CONTROLLERAXISMOTION</code>: An axis was moved, such as a thumbstick or analog trigger.</li>
</ul>
</li>
</ul>
<p>Here’s an example of handling the X button using the event:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">case</span> SDL_CONTROLLERBUTTONDOWN<span class="token operator">:</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>controller <span class="token operator">&&</span> event<span class="token punctuation">.</span>cdevice<span class="token punctuation">.</span>which <span class="token operator">==</span> <span class="token function">SDL_JoystickInstanceID</span><span class="token punctuation">(</span>
<span class="token function">SDL_GameControllerGetJoystick</span><span class="token punctuation">(</span>controller<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">switch</span> <span class="token punctuation">(</span>event<span class="token punctuation">.</span>cbutton<span class="token punctuation">.</span>button<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">case</span> SDL_GameControllerButton<span class="token double-colon punctuation">::</span>SDL_CONTROLLER_BUTTON_X<span class="token operator">:</span>
std<span class="token double-colon punctuation">::</span>cerr <span class="token operator"><<</span> <span class="token string">"X pressed!"</span> <span class="token operator"><<</span> std<span class="token double-colon punctuation">::</span>endl<span class="token punctuation">;</span>
<span class="token keyword">break</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token keyword">break</span><span class="token punctuation">;</span></code></pre>
<p>and by polling for input state:</p>
<pre class="language-c"><code class="language-c"><span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">SDL_GameControllerGetButton</span><span class="token punctuation">(</span>controller<span class="token punctuation">,</span> SDL_GameControllerButton<span class="token operator">::</span>SDL_CONTROLLER_BUTTON_X<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token function">printf</span><span class="token punctuation">(</span><span class="token string">"X was pressed!\n"</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span></code></pre>
<!-- todo: axes and thresholds -->
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>Using SDL_GameController, it’s possible to target a large number of gamepads on
different platforms without much effort. How to manage multiple controllers,
keyboard and mouse, and input binding is definitely a story for another time.</p>
<p>The
<a href="https://gitlab.com/rubenwardy/sdl_gamecontroller_example">complete example code for SDL_GameController</a>
is available for
<a href="https://gitlab.com/rubenwardy/sdl_gamecontroller_example/-/archive/main/sdl_gamecontroller_example-main.zip">download</a>
from GitLab.</p>
<h3 id="sources" tabindex="-1">Sources <a class="header-anchor" href="#sources">#</a></h3>
<p><a href="https://www.pexels.com/photo/flat-lay-photo-of-black-sony-ps4-game-controller-on-blue-background-2520829/">Image</a>
© 2019 Stas Knop</p>
<ul>
<li><a href="https://wiki.libsdl.org/SDL2/CategoryGameController">https://wiki.libsdl.org/SDL2/CategoryGameController</a></li>
<li><a href="https://stackoverflow.com/questions/50022316/what-is-sdl-joystick-and-what-is-sdl-gamecontroller-what-are-the-relationships">https://stackoverflow.com/questions/50022316/what-is-sdl-joystick-and-what-is-sdl-gamecontroller-what-are-the-relationships</a></li>
</ul>
rubenwardy
Large spike in Minetest traffic caused by YouTube video
2022-12-08T00:00:00.000Z
2022-12-08T00:00:00.000Z
https://blog.rubenwardy.com/2022/12/08/contentdb-youtuber-finds-minetest/
A month ago, ContentDB received a large spike in traffic. Bobicraft, a Spanish-language Youtube channel with 5M subscribers, posted a video about Minetest.
<p>Last month, ContentDB received a large spike in traffic.
<a href="/2022/03/24/contentdb/">ContentDB</a> is the website
I created that allows you to install mods/games/texture packs from within
<a href="https://www.minetest.net">Minetest’s</a> main menu. On Saturday 5th, Bobicraft,
a Spanish-language Youtube channel with 5M subscribers, posted
<a href="https://www.youtube.com/watch?v=g74i2CwuUp4">a video</a> about Minetest.
At the time of writing, it has reached 1.3 million views and is now the
most-watched Minetest video.</p>
<p>In this article, I will show the data about this increase in traffic. As the
tools that I use to monitor ContentDB are ephemeral, this will keep the data
around for future reference.</p>
<h2 id="the-data" tabindex="-1">The data <a class="header-anchor" href="#the-data">#</a></h2>
<p>Saturday 5th saw an overall 3.4x increase in unique visitors and a 10x increase in
total bandwidth used. At its peak at 15:18 UTC, ContentDB saw a 46x spike in
downloads.</p>
<figure class="undefined"><a href="/static/img/3tlJiwKHIl-733.png"><picture><source type="image/webp" srcset="/static/img/c8PielAq9X-540.webp 540w"><img alt="Screenshot of ContentDB's <a href="https://monitor.rubenwardy.com/d/3ELzFy3Wz/contentdb">monitoring dashboard</a>, based on Grafana." loading="lazy" decoding="async" src="/static/img/c8PielAq9X-540.png" width="540" height="618"></picture></a><figcaption>Screenshot of ContentDB's <a href="https://monitor.rubenwardy.com/d/3ELzFy3Wz/contentdb">monitoring dashboard</a>, based on Grafana.</figcaption></figure>
<p>It usually has 2TB downloads in a month, but that weekend alone used 0.86TB. If
this level of activity was experienced for the whole month, it would use 16TB.
My hosting quota is around 100TB a month so I still have a while to go.</p>
<figure class="undefined"><a href="/static/img/0e6vmitoKq-757.png"><picture><source type="image/webp" srcset="/static/img/Ps9mTYwwmC-540.webp 540w"><img alt="Bandwidth usage from Nginx static files." loading="lazy" decoding="async" src="/static/img/Ps9mTYwwmC-540.png" width="540" height="378"></picture></a><figcaption>Bandwidth usage from Nginx static files.</figcaption></figure>
<h3 id="per-package-statistics" tabindex="-1">Per-package Statistics <a class="header-anchor" href="#per-package-statistics">#</a></h3>
<p>A few days after this event, I implemented per-package statistics on ContentDB,
showing daily downloads. I ingested Nginx logs to get data covering the
event and several weeks prior. Package statistics are public, you can go to
any package page and click “Statistics” to see them.</p>
<figure class="undefined"><a href="/static/img/pZf25nek6i-845.png"><picture><source type="image/webp" srcset="/static/img/wt0jaYJ0PP-540.webp 540w"><img alt="Downloads per day of i3, ContentDB's most downloaded mod, from the <a href="https://content.minetest.net/packages/jp/i3/stats/">new statistics page</a>." loading="lazy" decoding="async" src="/static/img/wt0jaYJ0PP-540.png" width="540" height="331"></picture></a><figcaption>Downloads per day of i3, ContentDB's most downloaded mod, from the <a href="https://content.minetest.net/packages/jp/i3/stats/">new statistics page</a>.</figcaption></figure>
<h2 id="the-impact" tabindex="-1">The Impact <a class="header-anchor" href="#the-impact">#</a></h2>
<p>ContentDB was easily able to handle the load, I didn’t notice any performance
degradation. ContentDB is hosted on a Virtual Dedicated Server (VDS) from
netCup (RS 2000 G9.5, 6-core 3.35GHz, 16GB RAM), and so has dedicated compute
resources.</p>
<p>Earlier this year, we saw a huge spike after AntVenom released a video on Minetest.
This resulted in quite a big growth in our community, with the Discord server
gaining a few hundred new users.</p>
<figure class="undefined"><a href="/static/img/AvXlvpOOG1-640.png"><picture><source type="image/webp" srcset="/static/img/N0t0D11dPP-540.webp 540w"><img alt="Users joining the Discord server after the AntVenom video (By GreenXenith)." loading="lazy" decoding="async" src="/static/img/N0t0D11dPP-540.png" width="540" height="405"></picture></a><figcaption>Users joining the Discord server after the AntVenom video (By GreenXenith).</figcaption></figure>
<p>Despite this new video exceeding the number of views, we haven’t seen anything
like the level of growth seen after the AntVenom video. Not many new users
joined the Discord server, and there was no noticeable uptick in online players.
That being said, there wasn’t an increase in online players after AntVenom either.</p>
<figure class="undefined"><a href="/static/img/IiUFUID0vU-913.png"><picture><source type="image/webp" srcset="/static/img/940rnwLrr0-540.webp 540w"><img alt="No increase in players playng online." loading="lazy" decoding="async" src="/static/img/940rnwLrr0-540.png" width="540" height="237"></picture></a><figcaption>No increase in players playng online.</figcaption></figure>
<p>The latter half of the video focused on ContentDB and the packages available
from it, so perhaps a lot of the viewers decided to download and try them out in
singleplayer.</p>
<p>Another reason may be that the video is Spanish language and our communities are
mostly English language.</p>
<p>The AntVenom video accurately portrayed Minetest and our values in a
more serious manner, which may have led to the correct users finding Minetest
and joining our communities.</p>
<h2 id="the-future" tabindex="-1">The Future <a class="header-anchor" href="#the-future">#</a></h2>
<p>Now that Minetest has been noticed by a couple of big YouTubers, we might start
seeing more YouTubers and media sources checking out Minetest.</p>
<p>One of the things we’ve been working towards is an improved main menu redesign.
This is vital to attract and retain users, as the current main menu doesn’t do a
very good job of promoting Minetest or its values as an open game creation
platform.</p>
<p>Another thing we’ll be doing soon is listing Minetest on Steam. It’ll start in a
“coming soon” state whilst we improve the main menu and other things, allowing
players to wishlist it.</p>
rubenwardy
Minetest on Steam Deck: usage and review
2022-12-02T00:00:00.000Z
2022-12-02T00:00:00.000Z
https://blog.rubenwardy.com/2022/12/02/minetest-steam-deck/
In this article, I will explain how to set up Minetest on the deck, and review the controls, performance, and experience.
<p>A few months ago, my Steam Deck arrived. So, obviously, one of the first things
I did was play Minetest. Whilst Minetest isn’t available on Steam yet, it is
available on Flatpak and so can be installed from desktop mode.</p>
<p>Minetest plays very well on the Steam Deck, it performs great and Steam Input
makes setting up controls easy. The Deck is an open console, so there’s full
support for modding and you can cross-play with other platforms.</p>
<p>In this article, I will explain how to set up Minetest on the deck, and review
the controls, performance, and experience. I’ve already written
<a href="/2022/10/14/steam-deck-is-awesome/">my thoughts about the Deck</a>
in another article, and recommend checking that out.</p>
<h2 id="setting-up-minetest-on-steam-deck" tabindex="-1">Setting Up Minetest on Steam Deck <a class="header-anchor" href="#setting-up-minetest-on-steam-deck">#</a></h2>
<h3 id="install-minetest" tabindex="-1">Install Minetest <a class="header-anchor" href="#install-minetest">#</a></h3>
<p>As Minetest isn’t available on Steam yet, you’ll need to boot into the Deck’s
Desktop Mode and install Minetest from Flatpak. Press and hold the power button
and then select Switch to Desktop Mode.</p>
<p>Next, open up the Discover app. To show the onscreen keyboard, you’ll need to
press Steam and X at the same time. Search for Minetest and install it.</p>
<p>Once installed, open up the start menu, find Minetest, and right-click on it by
pressing the left trigger (L2). Select “add to Steam” and wait for it to finish.</p>
<p>You can now boot back into game mode. You’ll find Minetest in Library >
Non-Steam Games.</p>
<figure class="undefined"><a href="/static/img/t3wD9g42hO-1280.png"><picture><source type="image/webp" srcset="/static/img/9EO2qe_WA2-540.webp 540w"><img alt="After installing using Discover, open up the start menu and add to Steam." loading="lazy" decoding="async" src="/static/img/9EO2qe_WA2-540.png" width="540" height="337"></picture></a><figcaption>After installing using Discover, open up the start menu and add to Steam.</figcaption></figure>
<h3 id="add-cover-art-optional" tabindex="-1">Add Cover Art (optional) <a class="header-anchor" href="#add-cover-art-optional">#</a></h3>
<p>The Steam Deck’s UI shows art for each game. It’s likely that you’ll want to add
art to Minetest so that it’s not just a grey square.</p>
<p>Run Minetest at least once in Game Mode, stop it, and return to Desktop Mode.</p>
<p>Now you need to download the cover art you’d like to use. I’ll be posting some
example images at some point. In the meantime, you’ll need to provide horizontal
(460x215px) and vertical (600x300px) images in the correct resolutions.</p>
<p>Launch Steam and go to the Library. Find Minetest’s tile on the right side of
the window, then right-click > Manage > Set Custom Art > Select the vertical
image.</p>
<p>Scroll down until you see the blank landscape image of the app.
Right-click > Manage > Select artwork > Choose the horizontal image.</p>
<p>If you’re stuck, <a href="https://www.youtube.com/watch?v=kI8kSws7aS0">here’s the tutorial I followed</a>
on YouTube.</p>
<h2 id="controls" tabindex="-1">Controls <a class="header-anchor" href="#controls">#</a></h2>
<h3 id="setting-up-a-control-layout" tabindex="-1">Setting up a Control Layout <a class="header-anchor" href="#setting-up-a-control-layout">#</a></h3>
<p>Unfortunately, Minetest doesn’t have built-in support for
<a href="https://wiki.minetest.net/Gamepads">gamepads</a>.
There is some basic code for it, but it’s very broken and doesn’t support
user interfaces at all. I’m planning to fix this at some point, but in the
meantime, we’ll need to make do without it.</p>
<p>When a game doesn’t have built-in controller support, you can use Steam Input to
bind the controls to fake mouse movement and key presses.</p>
<p>I’ve already made a config for Minetest. Select the gamepad icon, and then click
the layout button. Go to Community and select “Minetest Semi-official Steam
Deck” by rubenwardy. You can now customise my layout as desired, or
alternatively make your own layout based on Templates > Keyboard (WASD) and
Mouse.</p>
<figure class="undefined"><a href="/static/img/UV6gQHq_jN-881.png"><picture><source type="image/webp" srcset="/static/img/jGLwV8iVkN-540.webp 540w"><img alt="You can find my Steam Deck layout in the Community Layouts tab." loading="lazy" decoding="async" src="/static/img/jGLwV8iVkN-540.png" width="540" height="172"></picture></a><figcaption>You can find my Steam Deck layout in the Community Layouts tab.</figcaption></figure>
<p>If, at any time, you’ve forgotten the controls or want to change them, you can
press the Steam button and go to controller options. It shows you the controls
and allows you to edit them whilst the game is running.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/Hspt2iQZwX-1280.png"><picture><source type="image/webp" srcset="/static/img/HI3IUjW4EJ-540.webp 540w"><img alt="Check or edit your layout whilst playing using the Steam menu." loading="lazy" decoding="async" src="/static/img/HI3IUjW4EJ-540.png" width="540" height="285"></picture></a><figcaption>Check or edit your layout whilst playing using the Steam menu.</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/p3rd6uQgvC-1280.png"><picture><source type="image/webp" srcset="/static/img/d2zVloIe_H-540.webp 540w"><img alt="My controller layout." loading="lazy" decoding="async" src="/static/img/d2zVloIe_H-540.png" width="540" height="285"></picture></a><figcaption>My controller layout.</figcaption></figure></div>
</div>
<h3 id="playing-the-game" tabindex="-1">Playing the Game <a class="header-anchor" href="#playing-the-game">#</a></h3>
<p>The game plays well in classic creative/survival gameplay, it’s easy to place
blocks and move around. Playing PvP or games that require parkour is harder, but
there are tricks to making it playable - you can use the touchpad for finer
movement or the gyro to aim.</p>
<p>One problem with going from a keyboard+mouse game to a gamepad is that a gamepad
has way fewer buttons. Steam Deck comes with extra buttons on the back, which
are great for extra actions. You can also use the left touchpad as a radial
menu, selecting options by dragging your finger to an angle and pressing to
select.</p>
<figure class="undefined"><a href="/static/img/8QXYz6VTdF-843.png"><picture><source type="image/webp" srcset="/static/img/xP4MUQbPcZ-540.webp 540w"><img alt="A radial menu can be accessed using the left trackpad." loading="lazy" decoding="async" src="/static/img/xP4MUQbPcZ-540.png" width="540" height="337"></picture></a><figcaption>A radial menu can be accessed using the left trackpad.</figcaption></figure>
<h3 id="user-interfaces" tabindex="-1">User Interfaces <a class="header-anchor" href="#user-interfaces">#</a></h3>
<p>You can use the right touchpad or thumbstick to move and click the mouse.
Because “place” is right-click, you can use the left trigger to right-click in
the inventory, allowing you to place a single item in a slot. You can also use
sneak to shift+click.</p>
<p>Steam Deck does come with a touchscreen, but because Minetest was built in
desktop mode it just emulates a mouse. You can’t use the touch gestures
available on the Android port.</p>
<p>To type, you’ll need to manually open the keyboard by pressing Steam+X.</p>
<p>Interacting with Minetest’s User Interfaces is perfectly usable but is a bit
clunky. It would be nice to have intelligent focus using the left thumbstick,
but this will require engine changes to fully support gamepads.</p>
<h2 id="performance" tabindex="-1">Performance <a class="header-anchor" href="#performance">#</a></h2>
<p>It’s hard to get an objective measure of the performance whilst playing Minetest
as it depends a lot on what you’re doing. How much of the world and what you can
see makes a big difference - viewing a large dense area of the map with lots of
trees, walls, and entities will drop the FPS lower than viewing a similar area
of grassy plains.</p>
<p>Starting a brand new world, I got a stable 60FPS walking around a dense forest
with default settings (200 view range). Flying above the forest, the FPS drops
down to 25FPS, which is still perfectly playable.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/ir6WESSADS-1280.png"><picture><source type="image/webp" srcset="/static/img/mGJc8pBqgy-540.webp 540w"><img alt="Chilling on a tree. 200 view range, 60FPS" loading="lazy" decoding="async" src="/static/img/mGJc8pBqgy-540.png" width="540" height="337"></picture></a><figcaption>Chilling on a tree. 200 view range, 60FPS</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/xu5X6j8DfM-1280.png"><picture><source type="image/webp" srcset="/static/img/fi8OJ9KgfT-540.webp 540w"><img alt="Flying high. 500 view range, 25FPS" loading="lazy" decoding="async" src="/static/img/fi8OJ9KgfT-540.png" width="540" height="337"></picture></a><figcaption>Flying high. 500 view range, 25FPS</figcaption></figure></div>
</div>
<p>Walking around my farm with default settings, I get 56-60FPS with occasional
drops down to 50FPS. The town has a lot of underground tunnels with mesecons and
pipeworks networks. There are also a lot of mobs roaming around, it is a farm
after all. Turning on medium shadows drops the FPS to 37-45.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/X0h4ZAPY3v-1280.png"><picture><source type="image/webp" srcset="/static/img/y4Zg0CIkkL-540.webp 540w"><img alt="200 view range, default settings, 56-60FPS" loading="lazy" decoding="async" src="/static/img/y4Zg0CIkkL-540.png" width="540" height="337"></picture></a><figcaption>200 view range, default settings, 56-60FPS</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/R-MHITfDBj-1280.png"><picture><source type="image/webp" srcset="/static/img/DE2NI1ZAAu-540.webp 540w"><img alt="200 view range with medium shadows, 37-45FPS" loading="lazy" decoding="async" src="/static/img/DE2NI1ZAAu-540.png" width="540" height="337"></picture></a><figcaption>200 view range with medium shadows, 37-45FPS</figcaption></figure></div>
</div>
<p>Minetest is largely CPU bound. The Steam Deck has an AMD RDNA 2, which is quite
a good GPU in terms of power to performance; it certainly outperforms the iGPU
on <a href="https://rubenwardy.com/uses/#laptop">my laptop</a>. If Minetest were to receive
the attention needed to upgrade to make use of modern pipelines and the GPU, I’m
sure the Deck would benefit and perform better.</p>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>Steam Deck is an awesome new way to play Minetest. Its openness makes it the
best console to play Minetest on.</p>
<p>In the future, I plan on adding full gamepad support to Minetest, which will
make using GUIs much easier. I also plan on releasing Minetest to Steam once
we have a new main menu; this will make it easier to install Minetest on the
Deck.</p>
<p>This article was originally going to be a YouTube video before I discovered that
recording the screen was painful, almost as painful as recording the voice-over.</p>
rubenwardy
Setting up fingerprint auth on Kubuntu (Thinkpad X1)
2022-11-16T00:00:00.000Z
2022-11-16T00:00:00.000Z
https://blog.rubenwardy.com/2022/11/16/thinkpad-x1-fingerprint-auth/
Unlock your computer or use sudo with your fingerprint on Ubuntu KDE.
<p>Using a fingerprint to unlock your computer or elevate privileges (sudo) is a
nice convenience, saving time during daily use of a computer. This article will
show you how to set up fingerprint authentication on Ubuntu KDE. You’ll still
need to enter your password to login, but once logged in you will be able to use
a fingerprint to unlock or use sudo.</p>
<p>This guide is based on a Thinkpad X1 running Kubuntu 22.04, but it should work
for most laptops with supported fingerprint sensors. You can check whether your
fingerprint sensor is supported by searching for “Linux fingerprint” and your
computer’s make/model. The Arch wiki is
<a href="https://wiki.archlinux.org/title/Lenovo_ThinkPad_X1_Carbon_(Gen_7)#Fingerprint_sensor">especially useful</a>
for this sort of thing, even if you don’t use Arch.</p>
<p>It goes without saying that this comes without warranty; backup your data, and
have a LiveUSB ready. There are some pointers on recovering from mistakes at the
bottom of this page. Luckily, I never needed to use a LiveUSB, I could easily
recover from a virtual console.</p>
<h2 id="steps" tabindex="-1">Steps <a class="header-anchor" href="#steps">#</a></h2>
<h3 id="install-fprint" tabindex="-1">Install fprint <a class="header-anchor" href="#install-fprint">#</a></h3>
<p>Run <code>lsusb</code> and check that you can see your fingerprint sensor.
For a Thinkpad X1, this will be <code>06cb:00bd Synaptics, Inc. Prometheus MIS Touch Fingerprint Reader</code>.</p>
<p>If you don’t, then your firmware may be outdated or the sensor may not be supported.
You can use <code>fwupdmgr</code> to update your firmware.</p>
<p>Next install fprint: <code>sudo apt install -y fprintd libpam-fprintd</code></p>
<h3 id="enroll-fingerprint" tabindex="-1">Enroll Fingerprint <a class="header-anchor" href="#enroll-fingerprint">#</a></h3>
<p>You now need to enroll at least one finger so that fprint can recognise you.</p>
<p>You can do that using the GUI, by going to Settings > Users >
Configure Fingerprint Authentication.</p>
<p>Alternatively, you can use the CLI: Run <code>fprintd-enroll $USER</code> and swipe your
finger over the sensor 3 times.</p>
<h3 id="set-up-the-authentication-module-pam" tabindex="-1">Set up the Authentication Module (PAM) <a class="header-anchor" href="#set-up-the-authentication-module-pam">#</a></h3>
<p>Enable the fprint pam module by running <code>sudo pam-auth-update</code>, checking Fingerprint,
and selecting OK.</p>
<h3 id="manual-changes" tabindex="-1">Manual Changes <a class="header-anchor" href="#manual-changes">#</a></h3>
<p>In theory, the above should be all that you need to do to use fingerprints for
login and sudo auth, but I found that logins were broken. So we’ll need to do a
few more steps to fix that.</p>
<p>The Pluggable Authentication Module (PAM) configuration files can be found at
<code>/etc/pam.d/</code>, and are used to determine how to authenticate the user. I needed
to change the order of the authentication methods to get it to work.</p>
<p>First, comment out the <code>fprint</code> line in <code>/etc/pam.d/common-auth</code> if present, to
prevent fprint from always being used.</p>
<p>To use fingerprints for sudo, add the following lines to <code>/etc/pam.d/sudo</code> above
<code>@include common-auth</code>:</p>
<pre><code>auth [success=2 default=ignore] pam_fprintd.so max_tries=2 timeout=10
</code></pre>
<p>To use fingerprints to unlock, create a new file <code>/etc/pam.d/kde</code>:</p>
<pre><code>auth sufficient pam_unix.so try_first_pass likeauth nullok
auth sufficient pam_fprintd.so max_tries=2 timeout=10
</code></pre>
<h3 id="test-it" tabindex="-1">Test It <a class="header-anchor" href="#test-it">#</a></h3>
<p>Lock your screen, and check you can still use your password to unlock - this is
useful when the fingerprint sensor fails. Now check that the fingerprint works;
press Enter on the password field and then touch the sensor.</p>
<h2 id="recovering-from-mistakes" tabindex="-1">Recovering from Mistakes <a class="header-anchor" href="#recovering-from-mistakes">#</a></h2>
<p>When changing authentication settings, there’s always a possibility of being
locked out; this happened a few times before I figured out how to make this
work.</p>
<h3 id="using-virtual-consoles" tabindex="-1">Using Virtual Consoles <a class="header-anchor" href="#using-virtual-consoles">#</a></h3>
<p>You can switch to a Virtual Console by pressing Ctrl + Alt + F2. This is a
fullscreen console that allows you to log into the system without going through
the lock screen. From here, you can edit the configuration files using nano or
your CLI editor of choice.</p>
<p>See <a href="https://www.makeuseof.com/what-are-linux-virtual-consoles/">Linux Virtual Consoles Explained</a> for more info.</p>
<h3 id="using-a-liveusb" tabindex="-1">Using a LiveUSB <a class="header-anchor" href="#using-a-liveusb">#</a></h3>
<p>As a last resort, it’s possible to use a LiveUSB/LiveCD to edit the
authentication configuration files. This is possible for both unencrypted and
encrypted drives, as you can
<a href="https://askubuntu.com/questions/653408/mounting-encrypted-luks-partition-from-live-cd">mount encrypted partitions using cryptsetup</a>.</p>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<h3 id="sources" tabindex="-1">Sources <a class="header-anchor" href="#sources">#</a></h3>
<p><a href="https://www.pexels.com/photo/black-and-silver-laptop-computer-3550482/">Image</a>
© 2019 Olena Bohovyk</p>
<ul>
<li><a href="https://unix.stackexchange.com/questions/655203/how-do-i-implement-fprintd-into-login-in-kubuntu">https://unix.stackexchange.com/questions/655203/how-do-i-implement-fprintd-into-login-in-kubuntu</a></li>
<li><a href="https://wiki.archlinux.org/title/fprint">https://wiki.archlinux.org/title/fprint</a></li>
<li><a href="https://www.reddit.com/r/kde/comments/sohmso/the_new_fingerprint_feature_is_not_working/">https://www.reddit.com/r/kde/comments/sohmso/the_new_fingerprint_feature_is_not_working/</a></li>
</ul>
rubenwardy
Minetest Mods: my first Android app
2022-10-22T00:00:00.000Z
2022-10-22T00:00:00.000Z
https://blog.rubenwardy.com/2022/10/22/Minetest-mods-app/
In 2016, I created an app to install mods for Minetest's Android port. It taught me a great deal about Android dev and helped me get my first programming job.
<p>In 2016, I created an app to install mods for Minetest’s Android port. It
was my first Android app; it taught me a great deal about Android development
and also helped me get my first programming job.</p>
<p><a href="https://www.minetest.net">Minetest</a> is an open-source game engine with millions
of downloads and thousands of weekly players. The project has a very active
modding community, with many games available to play. Before I created the app,
users had to manually install content by unzipping their files into a directory;
this was a poor user experience, especially on Android, so I created the app
to make this easier.</p>
<p>Minetest now has <a href="/2022/03/24/contentdb/">ContentDB</a>, a platform I created to
browse and install mods and games within Minetest. Because of this, the app is
now obsolete and is no longer available for download. That doesn’t mean this app
is fully gone - the lessons I learned live on in ContentDB.</p>
<h2 id="getting-data" tabindex="-1">Getting data <a class="header-anchor" href="#getting-data">#</a></h2>
<p>For the app to function, it needs a list of mods that can be installed.
In 2017, the closest thing to this list was
<a href="https://krock-works.uk.to/minetest/modSearch.php">Krock’s Mod Search</a>, an index
of mod topics on the forum, but this was super unreliable. It was common for
there to be no download links or git repo, and a lot of the metadata was plain
wrong.</p>
<p>My first challenge before I could even start on the app was to create a backend
API with validated data. I did this by using Krock’s Mod Search as a base and
then performing extra checks. I fetched forum topics to extract more data and
links. I fetched downloads and checked their contents, making sure that they
contained a valid mod with the correct mod name. I was able to manually override
the data where the automatic crawler failed.</p>
<p>One of the most important parts of this validation system was good tooling. I
created a simple web front-end that allowed you to find mods and see any problems
that were preventing them from appearing in the app.</p>
<h2 id="the-app" tabindex="-1">The app <a class="header-anchor" href="#the-app">#</a></h2>
<p>The app’s homepage had a list of mods, with installed mods appearing pinned at
the top. By tapping a mod you go to the mod’s detail screen, where you can see
more information about the mod and install it.</p>
<p>Mod installation was done in a background service, allowing the user to navigate
away from the mod page whilst it was being installed. A notification showed the
current download progress. Once a mod was installed, the app could then prompt
the user to install missing dependencies.</p>
<figure class="undefined"><a href="/static/img/WH7sjxzqGx-2304.png"><picture><source type="image/webp" srcset="/static/img/J8RTMnkJNT-540.webp 540w"><img alt="The app running on a tablet" loading="lazy" decoding="async" src="/static/img/J8RTMnkJNT-540.png" width="540" height="337"></picture></a><figcaption>The app running on a tablet</figcaption></figure>
<h2 id="google-play-and-post-release" tabindex="-1">Google Play and post-release <a class="header-anchor" href="#google-play-and-post-release">#</a></h2>
<p>I published the app on Google Play and F-Droid. It was received with favourable
reviews, averaging 4.3 stars.</p>
<p>I maintained it for 3 years, adding features to improve user experience such as
dependency installation and world configuration, and rewriting parts as my
knowledge of Android dev increased.</p>
<p>As the app predated Android Kotlin and Jetpack by over a year, it was originally
written in Java. Towards the end I rewrote parts of the app in Kotlin using
more modern techniques. With the introduction of ContentDB, I ended up
sunsetting the app and removing it from the Play Store.</p>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>The Minetest Mods app was my first Android app; it taught me a great deal about
Android development. I’m happy with the result; I think it looked good and had a
good user experience. One of the reasons I like making user interfaces is being
able to create quality user experiences that I can be proud of.</p>
<p>One of the primary flaws in the app was the data. It was quite unreliable; a lot
of the mods failed to installed due to changes that the crawler didn’t pick up.
I made a lot of changes to the backend to improve this situation, and they did
help. But the problem was ultimately solved by ditching the crawler and using
mods from ContentDB instead.</p>
<p>Another flaw is that the app was designed just for mods, and not for other
content like games and texture packs. Towards the end, I started to work on
extending the app to support games and texture packs - but I was never happy
with the result, and so it never made it into a release.</p>
<p>The app also got me my first programming job at a software house, creating
Android and iOS apps for clients. Being able to show my app on Google Play in
the interview was a big selling point.</p>
rubenwardy
Devkit CLI: Upload your game to Steam Deck from your dev computer
2022-10-16T00:00:00.000Z
2022-10-16T00:00:00.000Z
https://blog.rubenwardy.com/2022/10/16/steam-deck-upload-game/
How to upload and run your games on the Steam Deck from your development computer, using a VSCode task or a script you can use in any editor.
<p><a href="/2022/10/14/steam-deck-is-awesome/">Steam Deck</a> is a new portable gaming
console from Valve. The Deck is a very open platform as it’s a full Linux PC,
making it very easy to use third-party games and stores. All Decks can act as a
devkit machine, and come with devkit tools preinstalled. There was a Steam Deck
Devkit edition, but that was a preproduction version of the hardware to allow
game developers to test their games.</p>
<p>This article will show you how to upload and run your games on the Steam Deck
from your development computer, using a VSCode task or a script you can use in
any editor. Whilst you could develop on the Deck directly, it would require
setting up a new dev environment. Being able to develop on your computer and
test on the Deck allows you to reuse your current dev environment.</p>
<h2 id="set-up-steamos-devkit" tabindex="-1">Set up SteamOS Devkit <a class="header-anchor" href="#set-up-steamos-devkit">#</a></h2>
<p>You’ll need to follow the “Install tools” and “Connecting to Deck” instructions
found in the
<a href="https://partner.steamgames.com/doc/steamdeck/loadgames">Steamworks documentation</a>.</p>
<h2 id="how-does-devkit-upload-to-the-deck" tabindex="-1">How does DevKit upload to the Deck? <a class="header-anchor" href="#how-does-devkit-upload-to-the-deck">#</a></h2>
<p>Before we continue with the process, it’s important to explain what the DevKit
GUI does when you upload a game. The GUI is quite complicated and there’s no
official CLI or library, so it would be nice if we could simplify the process.</p>
<p>It performs the following steps:</p>
<ol>
<li>Uploads some utility scripts</li>
<li>Creates a game on the Deck</li>
<li>Uploads the game files</li>
<li>Creates a shortcut for the game on the Deck</li>
</ol>
<p>Steps 1, 2, and 4 don’t need to be done every time. Once the game is registered
using the GUI, you only need to upload the files to upload a new version.</p>
<h2 id="create-your-game-using-the-devkit-client-gui" tabindex="-1">Create your game using the Devkit Client GUI <a class="header-anchor" href="#create-your-game-using-the-devkit-client-gui">#</a></h2>
<p>With your Deck connected, go to Title Upload and fill in the fields:</p>
<ul>
<li><strong>Name:</strong> Programmatic name, no spaces are allowed. Ex: <code>MyGame</code></li>
<li><strong>Local folder:</strong> Path to the build folder. Ex: <code>/home/user/dev/games/mygame/build/</code>.</li>
<li><strong>Upload filtering:</strong> You can filter what files are uploaded to the deck here.</li>
<li>Check “Delete remote files not present in local folder”</li>
<li><strong>Start command:</strong> Executable name, relative to local folder. Ex: <code>mygame.x84_64</code></li>
<li>If your game is a Windows executable, you’ll need to enable Steam Play.</li>
<li>Save config</li>
<li>Click <strong>Upload</strong></li>
</ul>
<p>You should now go to Library > Non-Steam Games and run your game. If it fails
to open, you should check the logs - you can do that using the “Device Logs” tab
of the Devkit Client GUI.</p>
<h2 id="the-script" tabindex="-1">The Script <a class="header-anchor" href="#the-script">#</a></h2>
<p>The script uses
<a href="https://www.digitalocean.com/community/tutorials/how-to-use-rsync-to-sync-local-and-remote-directories">rsync</a>
to upload the files to the deck. rsync is a common program used to copy entire
directories; in this case, we’re using it to upload over SSH. If you’re using
Windows, you may need to use WSL.</p>
<p>Save the following as <code>./utils/deck.sh</code> in your project and customise the variables
at the top.</p>
<pre class="language-bash"><code class="language-bash"><span class="token shebang important">#!/bin/bash</span>
<span class="token builtin class-name">set</span> <span class="token parameter variable">-e</span>
<span class="token comment"># Details as provided to the DevKit client</span>
<span class="token assign-left variable">GAME_NAME</span><span class="token operator">=</span><span class="token string">"MyGame"</span>
<span class="token assign-left variable">BUILD_DIR</span><span class="token operator">=~</span>/dev/games/mygame/build/
<span class="token assign-left variable">EXE</span><span class="token operator">=</span><span class="token string">"mygame.x86_64"</span>
<span class="token comment"># Deck IP address</span>
<span class="token assign-left variable">IP_ADDRESS</span><span class="token operator">=</span><span class="token string">"192.168.0.110"</span>
<span class="token comment"># No need to edit these</span>
<span class="token assign-left variable">TARGET_PATH</span><span class="token operator">=</span><span class="token string">"/home/deck/devkit-game/<span class="token variable">$GAME_NAME</span>/"</span>
<span class="token assign-left variable">RSA_KEY_PATH</span><span class="token operator">=~</span>/.config/steamos-devkit/devkit_rsa
<span class="token comment"># Upload game files</span>
<span class="token function">rsync</span> <span class="token parameter variable">-av</span> <span class="token parameter variable">--chmod</span><span class="token operator">=</span>Du<span class="token operator">=</span>rwx,Dgo<span class="token operator">=</span>rx,Fu<span class="token operator">=</span>rwx,Fog<span class="token operator">=</span>rx <span class="token punctuation">\</span>
<span class="token parameter variable">-e</span> <span class="token string">'ssh -o StrictHostKeyChecking=no -i $RSA_KEY'</span> <span class="token punctuation">\</span>
<span class="token parameter variable">--update</span> <span class="token parameter variable">--delete</span> --delete-excluded --delete-delay <span class="token punctuation">\</span>
<span class="token parameter variable">--checksum</span> <span class="token string">"<span class="token variable">$BUILD_DIR</span>"</span> <span class="token string">"deck@<span class="token variable">$IP_ADDRESS</span>:<span class="token variable">$TARGET_PATH</span>"</span></code></pre>
<p>You can add this as a VSCode task by adding the following to
<code>.vscode/tasks.json</code>:</p>
<pre class="language-js"><code class="language-js"><span class="token punctuation">{</span>
<span class="token comment">// See https://go.microsoft.com/fwlink/?LinkId=733558</span>
<span class="token comment">// for the documentation about the tasks.json format</span>
<span class="token string-property property">"version"</span><span class="token operator">:</span> <span class="token string">"2.0.0"</span><span class="token punctuation">,</span>
<span class="token string-property property">"tasks"</span><span class="token operator">:</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span>
<span class="token string-property property">"label"</span><span class="token operator">:</span> <span class="token string">"Run on Steam Deck"</span><span class="token punctuation">,</span>
<span class="token string-property property">"type"</span><span class="token operator">:</span> <span class="token string">"shell"</span><span class="token punctuation">,</span>
<span class="token string-property property">"command"</span><span class="token operator">:</span> <span class="token string">"./utils/deck.sh"</span><span class="token punctuation">,</span>
<span class="token string-property property">"presentation"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
<span class="token string-property property">"reveal"</span><span class="token operator">:</span> <span class="token string">"always"</span><span class="token punctuation">,</span>
<span class="token string-property property">"panel"</span><span class="token operator">:</span> <span class="token string">"shared"</span><span class="token punctuation">,</span>
<span class="token string-property property">"showReuseMessage"</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span>
<span class="token string-property property">"clear"</span><span class="token operator">:</span> <span class="token boolean">true</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token punctuation">]</span>
<span class="token punctuation">}</span></code></pre>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>The fact that the Deck is a Linux PC made this super easy to do. In the future,
I’d like to make a devkit CLI that allows you to create games as well as update.
I’d also like to be able to start and stop the game, this would be super useful
during testing.</p>
rubenwardy
Steam Deck is Awesome
2022-10-14T00:00:00.000Z
2022-10-14T00:00:00.000Z
https://blog.rubenwardy.com/2022/10/14/steam-deck-is-awesome/
My Steam Deck arrived two months ago, and I've been having a blast ever since.
<p>My Steam Deck arrived two months ago, and I’ve been having a blast ever since.
I’m not much of a gamer; RimWorld and Skyrim are the only two games I’ve spent
more than 100 hours in. I find it hard to play a game without wanting to do game
dev instead. But with the Deck, I’ve been playing the games that have been on my
play list for a while.</p>
<p>This article contains my thoughts about the Steam Deck. It’s not meant to be
a review applicable to everyone; I’m not your average user nor do I claim to
be. I recommend reading a wide range of reviews before buying something
expensive.</p>
<h2 id="why-i-like-the-steam-deck" tabindex="-1">Why I Like the Steam Deck <a class="header-anchor" href="#why-i-like-the-steam-deck">#</a></h2>
<h3 id="the-steam-store-and-my-steam-library" tabindex="-1">The Steam Store and My Steam library <a class="header-anchor" href="#the-steam-store-and-my-steam-library">#</a></h3>
<p>I was given an Xbox 360 when it first came out and I sold my original Xbox.
While I had over 30 games for the original Xbox, I only had three games for the
Xbox 360 and didn’t have money to buy more. Because of this, I didn’t play the
Xbox 360 much.</p>
<p>The variety and cost of games are very important for a console. The Deck comes
with Steam by default, giving access to a huge number of varied and inexpensive
games. Steam regularly runs sales, making it even less expensive to get good
games. I recently bought The Witcher 3, with all DLCs, for £6; that’s a
brilliant game with easily more than 100 hours of content.</p>
<p>The Steam store is the leading store for PC gamers. Over the years, I’ve bought
a lot of games for cheap on Steam sales but haven’t had time to play them. With
the Steam Deck, I am finally playing these games. Recent examples include Portal
1 and The Stanley Parable.</p>
<h3 id="super-open-device" tabindex="-1">Super Open Device <a class="header-anchor" href="#super-open-device">#</a></h3>
<p>Steam Deck is a full Linux PC. You can enter desktop mode and install apps using
Flatpak. Don’t want to install from Steam? You can install other game stores,
such as Heroic Games (for Epic Game Store), Itch.io, and GOG. You can even use
Xbox Game Pass.</p>
<p>I was able to get one of my games working in under 30 minutes. I exported my
game from Godot and used the SteamOS DevKit to upload it to the Deck. The DevKit
just uses rsync, a standard Linux file transfer utility, so it’s perfectly
possible to upload in the same way using a custom script. The trickier bit is
adding the game to Steam and creating the shortcuts, the devkit has a few long
scripts to do this.</p>
<figure class="undefined"><a href="/static/img/JPiMw1C2RY-900.jpeg"><picture><source type="image/webp" srcset="/static/img/VGFWe5QjaR-540.webp 540w"><img alt="My <a href='/2022/03/19/ld48-tin-mining/'>Tin Mining</a> game running on Steam Deck." loading="lazy" decoding="async" src="/static/img/VGFWe5QjaR-540.jpeg" width="540" height="211"></picture></a><figcaption>My <a href='/2022/03/19/ld48-tin-mining/'>Tin Mining</a> game running on Steam Deck.</figcaption></figure>
<h3 id="controls-and-steam-input" tabindex="-1">Controls and Steam Input <a class="header-anchor" href="#controls-and-steam-input">#</a></h3>
<p>One of the main benefits of the Steam Deck is the amazing controls. You get
full-size thumbsticks with ergonomic triggers and grips.</p>
<p>Steam Input allows you to easily rebind controls and add gamepad support to
games that don’t otherwise support them.
<a href="/2022/12/02/minetest-steam-deck/">Minetest</a> doesn’t
support gamepads by default, but it was super easy to set up Steam Input to fake
key presses and mouse movement to control Minetest. The experience is largely
indistinguishable from the game itself supporting gamepads.</p>
<figure class="undefined"><a href="/static/img/08ygh99Yt3-700.png"><picture><source type="image/webp" srcset="/static/img/jwe2n5JCX1-540.webp 540w"><img alt="Steam Input allows you to rebind the controls on any game, even those that don't support gamepads." loading="lazy" decoding="async" src="/static/img/jwe2n5JCX1-540.png" width="540" height="337"></picture></a><figcaption>Steam Input allows you to rebind the controls on any game, even those that don't support gamepads.</figcaption></figure>
<p>As well as the two thumbsticks, the Deck has two trackpads. The right trackpad
is commonly used to control the mouse or move the right thumbstick more
precisely. The left thumbstick is commonly used as a Virtual Menu / Radial Menu,
allowing you to press and select options quickly. The radial menu feature is
great in games that have a lot of keyboard shortcuts, such as RimWorld.</p>
<figure class="undefined"><a href="/static/img/fZQ0IMIsyQ-1280.png"><picture><source type="image/webp" srcset="/static/img/ZbNfNz_NjF-540.webp 540w"><img alt="A radial menu in RimWorld, accessed using the left trackpad." loading="lazy" decoding="async" src="/static/img/ZbNfNz_NjF-540.png" width="540" height="337"></picture></a><figcaption>A radial menu in RimWorld, accessed using the left trackpad.</figcaption></figure>
<p>The Deck also has 4 extra buttons on the back. Not many games use these
buttons - Aperture Desk Job and RimWorld were the only games I found
that did. You can set these buttons to perform your own actions, or just
ignore them.</p>
<h3 id="linux-and-open-source" tabindex="-1">Linux and Open Source <a class="header-anchor" href="#linux-and-open-source">#</a></h3>
<p>I’ve been using Linux as my primary OS for over 10 years. Linux performs better,
is more flexible, and provides an alternative to the current duopoly of
operating systems.</p>
<p>When I first started using Linux, I dual-booted as most of my games didn’t work.
Valve’s work on Proton/Wine in preparation for Steam Deck has made Linux viable
as a gaming platform. Now, all the games I’m interested in playing work on
Linux.</p>
<p>SteamOS is a very user-friendly introduction
to Linux; you don’t need to use the terminal or any desktop features, and the
read-only distro makes updating unlikely to cause issues.</p>
<p>From Valve’s perspective, Linux provides an alternative to Windows that reduces
their dependence on Microsoft. Valve started working on improving Linux as a
gaming platform after Windows 8 required the Windows Store to install apps.</p>
<!-- page.quotes.gabe -->
<h3 id="a-dedicated-machine-for-gaming" tabindex="-1">A Dedicated Machine for Gaming <a class="header-anchor" href="#a-dedicated-machine-for-gaming">#</a></h3>
<p>I work on my laptop. Having a dedicated machine makes gaming easier and puts me
in the right mood for it. It’s portable but also easy to dock to a TV.</p>
<h2 id="downsides" tabindex="-1">Downsides <a class="header-anchor" href="#downsides">#</a></h2>
<p>But the Deck isn’t perfect. It’s still clearly a work in progress, I’ve found
quite a few bugs in the Steam UI while using it. Valve releases updates pretty
regularly, and all the bugs I’ve found have since been fixed.</p>
<p>Another sticking point is that it’s not very clear what resolutions are being
used when using an external display, but since I started writing this post Valve
has released an update to fix this.</p>
<p>The Deck produces quite a lot of heat when playing demanding games, but this
doesn’t actually matter much as it doesn’t appear to be throttling or causing
issues.</p>
<p>The battery life varies depending on the game but generally tends to be from
2.5 hours - 5 hours. This is fine for playing on the go, as I don’t tend to be
on public transport or a cafe for long, and at home I tend to play docked to my
TV and charging.</p>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>The Steam Deck is the perfect gaming device for me. It’s open, flexible, and fun
to use. Even if you don’t use Linux, Valve’s work for the Deck will benefit all
Linux gamers. It’s truly the year of Linux on the <s>desktop</s> bus.</p>
rubenwardy
Why I created Renewed Tab, my New Tab extension
2022-07-21T00:00:00.000Z
2022-07-21T00:00:00.000Z
https://blog.rubenwardy.com/2022/07/21/renewed-tab/
I wrote an article on the Renewed Tab blog where I explained what led me to create a New Tab extension and what the design requirements were.
<p>I created <a href="https://renewedtab.com/">Renewed Tab</a> last year to fulfil my personal
needs, and have since expanded on it based on user feedback, focusing on rich
widgets, a clean user experience, and customisation.</p>
<p>I wrote
<a href="https://renewedtab.com/blog/2022/07/21/why-I-created-renewed-tab/">an article on the Renewed Tab blog</a>
where I explained what led me to create a New Tab extension and what the
design requirements were.</p>
rubenwardy
Creating worker NPCs using behavior trees
2022-07-17T00:00:00.000Z
2022-07-17T00:00:00.000Z
https://blog.rubenwardy.com/2022/07/17/game-ai-for-colonists/
A couple of my games have had NPCs that can build and perform work directed by the player, similar to RimWorld. In this article, I will explain how I implemented this.
<p>I’m a huge fan of <a href="https://rimworldgame.com/">RimWorld</a>, a base building game
where you manage a group of colonists. Rather than directly controlling the
colonists, you place blueprints for buildings, mark trees for cutting, and
animals for hunting. NPCs will then decide what to do automatically, based on
their skills and priorities.</p>
<p>I’ve made two games recently with similar mechanics. The first was
<a href="https://rubenwardy.com/rvwp/">Ruben’s Virtual World Project (RVWP)</a>, a hybrid
basebuilder/topdown shooter. The second was
<a href="/2022/03/19/ld48-tin-mining/">Tin Mining</a>, a mining sim created as an entry to
Ludum Dare 48. Both of these games allowed placing building plans that NPC
workers would then build out.</p>
<p>In this article, I will explain how I implemented the NPC AIs, and the problems
I faced.</p>
<h2 id="choosing-a-game-ai-framework" tabindex="-1">Choosing a Game AI framework <a class="header-anchor" href="#choosing-a-game-ai-framework">#</a></h2>
<h3 id="state-machines" tabindex="-1">State Machines <a class="header-anchor" href="#state-machines">#</a></h3>
<p>A common way to implement game AI is to use a finite state machine. Different
things an NPC could do would be modelled as states, and the NPC would transition
between states based on conditions.</p>
<figure class="undefined"><a href="/static/img/tzA3MR9Bix-358.svg"><picture><source type="image/webp" srcset="/static/img/VR-Cl38wXf-540.webp 540w"><img alt="An example of a state machine" loading="lazy" decoding="async" src="/static/img/VR-Cl38wXf-358.svg" width="358" height="207"></picture></a><figcaption>An example of a state machine</figcaption></figure>
<p>One problem with state machines is that you need to program and design for all
the transitions. What if an NPC dies whilst performing work? It shouldn’t keep
working like a zombie until it’s finished.</p>
<p>State machines work very well for simple behaviour, like that of an animal or
a dumb guard. But my NPCs need to be able to perform complex work, often with
multiple steps. The states would end up being pretty complicated.
I wanted a more capable framework.</p>
<p>Another concern was reuse. Lots of work involves the same actions - moving to a
location, picking up an object, running an animation. It’s possible for states
to reuse behaviour using utility functions, but this can be a bit painful.
I wanted my game AI framework to make reuse as easy as possible.</p>
<p>After researching game AI, the obvious choice for me was behavior trees.</p>
<h3 id="behavior-trees-behaviour-trees" tabindex="-1">Behavior Trees / Behaviour Trees <a class="header-anchor" href="#behavior-trees-behaviour-trees">#</a></h3>
<p>Explaining Behavior Trees in full is a bit out of scope for this article;
I highly recommend reading
<a href="https://www.gamedeveloper.com/programming/behavior-trees-for-ai-how-they-work">“Behavior trees for AI: How they work”</a>
by Chris Simpson.
However, I will try to explain the basics.</p>
<p>Behaviour trees allow you to control an NPC’s decision making by combining
reusable nodes rather than creating new states. They work best when the nodes
are small and specific, for example, you might have nodes to check conditions or
walk to a position.</p>
<p>Behaviour trees are basically a programming language in themselves, but for game
AI. Execution starts at the top of the tree, and then works downwards based on
the rules of different nodes. A node is either running, succeed, or failed.</p>
<figure class="undefined"><a href="/static/img/0bObbIgJWS-448.png"><picture><source type="image/webp" srcset="/static/img/MvgcjKcxIa-448.webp 448w"><img alt="A behaviour tree to make the NPC walk to a random position. Created using <a href="https://github.com/pruttned/owl-bt">owl-bt</a>, a browser-based BT editor." loading="lazy" decoding="async" src="/static/img/MvgcjKcxIa-448.png" width="448" height="186"></picture></a><figcaption>A behaviour tree to make the NPC walk to a random position. Created using <a href="https://github.com/pruttned/owl-bt">owl-bt</a>, a browser-based BT editor.</figcaption></figure>
<p>In the above tree, there is a Sequence node with two children. Sequences will
run each child one after another, until either a child fails or they have all
succeeded. The first child finds a random position, and writes it to a variable
called <code>$target</code>. The second child walks the NPC to that position. If the NPC
successfully finds a position and walks to it, then the sequence succeeds. If
either child fails, for example if there is something blocking the path, then
the sequence will also fail.</p>
<p>To implement logic, you can use decorators to check conditions. If the
condition is true, execution continues to the decorator’s child. if the
condition is false, the decorator marks itself as failed and returns to the
parent. This is very powerful when combined with a selector, a node that runs
its children until the first succeeds.</p>
<figure class="undefined"><a href="/static/img/8f04VuaM1N-716.png"><picture><source type="image/webp" srcset="/static/img/c6RGS7FtCR-540.webp 540w"><img alt="A behaviour tree to make the NPC do work, but prioritise dying or sleeping based on stats." loading="lazy" decoding="async" src="/static/img/c6RGS7FtCR-540.png" width="540" height="293"></picture></a><figcaption>A behaviour tree to make the NPC do work, but prioritise dying or sleeping based on stats.</figcaption></figure>
<p>The above tree will check whether the NPC’s needs, such as health and energy,
before doing work. This will be checked every time the tree updates, which
makes it possible for death or low energy to interrupt work.</p>
<p>It’s possible for behavior trees to include other behavior trees. The
“FindWork” node above does this in order to perform work; each work type has a
tree. Here’s the tree for construction work:</p>
<figure class="undefined"><a href="/static/img/1DL2mKP6Uq-777.png"><picture><source type="image/webp" srcset="/static/img/Uq9orLBiq1-540.webp 540w"><img alt="Tree for construction work. "Place Blueprint" replaces the player order with a building plan, and the "Work" node plays an animation whilst incrementing a loading bar.
" loading="lazy" decoding="async" src="/static/img/Uq9orLBiq1-540.png" width="540" height="255"></picture></a><figcaption>Tree for construction work. "Place Blueprint" replaces the player order with a building plan, and the "Work" node plays an animation whilst incrementing a loading bar.
</figcaption></figure>
<p>I implemented behavior trees in Tin Mining using the
<a href="https://godotengine.org/asset-library/asset/837">Behavior Tree</a> Godot
plugin. I wrote my own implementation for RVWP as I couldn’t find a good Lua
implementation, it was fairly easy to implement.</p>
<figure class="undefined"><a href="/static/img/F7tGriAsaF-274.png"><picture><source type="image/webp" srcset="/static/img/Bfb8jqEBnC-274.webp 274w"><img alt="A behaviour tree in the Godot Engine for moving items" loading="lazy" decoding="async" src="/static/img/Bfb8jqEBnC-274.png" width="274" height="161"></picture></a><figcaption>A behaviour tree in the Godot Engine for moving items</figcaption></figure>
<h3 id="goal-oriented-action-planning-goap" tabindex="-1">Goal-Oriented Action Planning (GOAP) <a class="header-anchor" href="#goal-oriented-action-planning-goap">#</a></h3>
<p>Just a quick mention of another framework that I considered.
<a href="https://gamedevelopment.tutsplus.com/tutorials/goal-oriented-action-planning-for-a-smarter-ai--cms-20793">GOAP</a>
is an AI system that allows NPCs to work out how to achieve a goal based on
possible actions. For example, the NPC knows that building a house requires
wood, and cutting trees makes wood. GOAP allows it to put these two things
together, and cut trees to get wood to build a house.</p>
<p>I think this isn’t necessarily mutually exclusive to behavior trees, it
operates at a higher level. You may use GOAP to decide what actions to do, and
then implement those actions using behavior trees.</p>
<h2 id="work-manager" tabindex="-1">Work Manager <a class="header-anchor" href="#work-manager">#</a></h2>
<h3 id="allocating-work" tabindex="-1">Allocating Work <a class="header-anchor" href="#allocating-work">#</a></h3>
<p>Now that we have a framework for performing work, we need a way to decide what
work an NPC should pick up next.</p>
<p>When a worker isn’t currently working on a task, it periodically asks the
Work Manager for an available task. The Work Manager allocates tasks based on
distance and a heuristic weighting of the task. For example, moving dropped
items to the stockpile is weighted higher than mining or building, to avoid
items building up and clogging the walkways.</p>
<p>In order to avoid NPCs working on the same thing, the Work Manager has a
reservation system that allows NPCs to lock a tile, entity, or piece of work.</p>
<h3 id="reachability" tabindex="-1">Reachability <a class="header-anchor" href="#reachability">#</a></h3>
<p>It’s important that NPCs are only allocated to work that they can reach - this
is called a reachability check, and is typically implemented using pathfinding.
Pathfinding tends to be quite expensive, but there are some quick checks that
can skip the effort in a lot of cases. My Tin Mining game uses a graph to
represent all possible paths; if a tile isn’t walkable, it’s not on the graph.
This means that you can check whether work is probably unreachable by seeing
if it’s on the graph, an <code>O(1)</code> check.</p>
<figure class="undefined"><a href="/static/img/0ZOaKd8uOQ-1182.png"><picture><source type="image/webp" srcset="/static/img/FhW3MpsJHq-540.webp 540w"><img alt="Tin Mining's pathfinding graph" loading="lazy" decoding="async" src="/static/img/FhW3MpsJHq-540.png" width="540" height="448"></picture></a><figcaption>Tin Mining's pathfinding graph</figcaption></figure>
<p>Pathfinding is still needed because it’s possible to have multiple disconnected
subgraphs on the map. Pathfinding to every possible work would be quite expensive,
so I defer pathfinding until after the work is allocated.</p>
<p>One problem with this approach is that it doesn’t know how close the work
really is to the NPC. The NPC might be three tiles from the work but 100 tiles
to walk there due to obstacles in the way. A future improvement will be to use
the actual walking distance rather than the direct distance.</p>
<h3 id="work-theft" tabindex="-1">Work theft <a class="header-anchor" href="#work-theft">#</a></h3>
<p>When an NPC finishes mining a tile, it often reveals more tiles to be mined.
In the time between the NPC mining the tile and finishing the work, another NPC
may have been allocated to the neighbouring tile. This results in a lot of
inefficiency, especially if the other worker is far away.</p>
<figure class="undefined"><a href="/static/img/9iZpPBVStF-700.webp"><img alt="Two NPCs that keep stealing each other's jobs" loading="lazy" decoding="async" src="/static/img/gW7bOdHkbC-700.webp" width="700" height="74900"></a><figcaption>Two NPCs that keep stealing each other's jobs</figcaption></figure>
<p>At this time, I don’t have a good fix for this due to the simple nature of my
work allocator. My workaround for the meantime is to avoid gaps between the tile
being mined and the worker looking for work - but this doesn’t work with
multiple-step work, such as digging a tile and then placing a ladder.</p>
<p>In the future, I’d like to implement a global work allocator algorithm. Instead
of considering each worker’s request independently, it should keep track of idle
workers in each frame and allocate them all in one go.</p>
<p>Another option would be to add some form of work queue or reservation. This is
how RimWorld resolves this problem.</p>
<h3 id="adjusting-priorities" tabindex="-1">Adjusting Priorities <a class="header-anchor" href="#adjusting-priorities">#</a></h3>
<p>Because the Work Manager uses a heuristic to allocate work, it’s possible to
change the heuristic to change how work is allocated.</p>
<p>In my tin mining game, I was having an issue where the workers would keep mining
tiles and leave all the rubble and ore on the ground. Ideally, workers mine for
a bit and then haul items back to the surface.</p>
<p>The first change I made to fix this was to make the dropped entities appear
20% closer than minable tiles, but also add an offset of 3 tiles so that
workers prioritise close tiles. This mostly worked, but when mining long
sections you can still end up with a lot of dropped items.</p>
<p>The second change was to make that 20% multiplier vary based on how many dropped
items there are. When there are not many dropped items, hauling items is a
lower priority. The more dropped items there are per worker, the higher the
priority becomes.</p>
<pre class="language-py"><code class="language-py"><span class="token comment"># Calculate multiplier based on number of dropped items</span>
targetDroppedItems <span class="token operator">=</span> <span class="token builtin">min</span><span class="token punctuation">(</span><span class="token number">3</span> <span class="token operator">*</span> numberOfWorkers<span class="token punctuation">,</span> <span class="token number">200</span><span class="token punctuation">)</span>
droppedItemsMultiplier <span class="token operator">=</span> <span class="token number">0.95</span>
<span class="token keyword">if</span> <span class="token builtin">len</span><span class="token punctuation">(</span>droppedItems<span class="token punctuation">)</span> <span class="token operator">></span> <span class="token number">3</span><span class="token operator">*</span>targetDroppedItems<span class="token punctuation">:</span>
droppedItemsMultiplier <span class="token operator">=</span> <span class="token number">0.4</span>
<span class="token keyword">elif</span> <span class="token builtin">len</span><span class="token punctuation">(</span>droppedItems<span class="token punctuation">)</span> <span class="token operator">></span> <span class="token number">2</span><span class="token operator">*</span>targetDroppedItems<span class="token punctuation">:</span>
droppedItemsMultiplier <span class="token operator">=</span> <span class="token number">0.65</span>
<span class="token keyword">elif</span> <span class="token builtin">len</span><span class="token punctuation">(</span>droppedItems<span class="token punctuation">)</span> <span class="token operator">></span> <span class="token number">1</span><span class="token operator">*</span>targetDroppedItems<span class="token punctuation">:</span>
droppedItemsMultiplier <span class="token operator">=</span> <span class="token number">0.8</span>
<span class="token comment"># Calculate weight based on distance, the multiplier,</span>
<span class="token comment"># and an offset to prioritise nearby tiles</span>
weight <span class="token operator">=</span> droppedItemsMultiplier <span class="token operator">*</span> \
worker<span class="token punctuation">.</span>position<span class="token punctuation">.</span>distance_squared_to<span class="token punctuation">(</span>item<span class="token punctuation">.</span>global_position<span class="token punctuation">)</span> <span class="token operator">+</span> <span class="token number">3</span><span class="token operator">*</span><span class="token number">3</span></code></pre>
<h2 id="debugging" tabindex="-1">Debugging <a class="header-anchor" href="#debugging">#</a></h2>
<p>One of the hardest parts of designing complex systems is making it easy to
debug. You want to know what an NPC is ‘thinking’, and be able to trace why it
did something at a certain time. I use a combination of logging and UI debug
tools to do this.</p>
<p>I have UI debug tools for the pathfinder, for inspecting work and locks on a
tile, for showing the NPC behavior tree and current work, and more.</p>
<p>RVWP has an immediate mode based debug API that allows adding lines and labels
to the world. This is very useful when designing game AIs, and can be seen in
the RVWP animation in the above section.</p>
<pre class="language-lua"><code class="language-lua"><span class="token keyword">local</span> debug <span class="token operator">=</span> rvwp<span class="token punctuation">.</span><span class="token function">get_debug</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
debug<span class="token punctuation">:</span><span class="token function">draw_line</span><span class="token punctuation">(</span>from<span class="token punctuation">,</span> next_pos<span class="token punctuation">,</span> <span class="token string">"#fff"</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> i<span class="token operator">=</span>self<span class="token punctuation">.</span>path_i <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">,</span> <span class="token operator">#</span>self<span class="token punctuation">.</span>path <span class="token keyword">do</span>
debug<span class="token punctuation">:</span><span class="token function">draw_line</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>path<span class="token punctuation">[</span>i <span class="token operator">-</span> <span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">,</span> self<span class="token punctuation">.</span>path<span class="token punctuation">[</span>i<span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token string">"#999"</span><span class="token punctuation">)</span>
<span class="token keyword">end</span></code></pre>
<h2 id="so-many-bugs" tabindex="-1">So many bugs <a class="header-anchor" href="#so-many-bugs">#</a></h2>
<p>Creating complex systems from simple rules is a great way to get a lot of random
bugs.</p>
<p>Running the pathfinder every frame would be expensive, so instead paths are
cached by the NPC. This resulted in NPCs not being aware of map changes, causing
them to fall down holes or get stuck behind new walls. The fix was to validate
the cached path against the navigation graph whilst moving.</p>
<figure class="undefined"><a href="/static/img/rqBOfWwQWK-696.png"><picture><source type="image/webp" srcset="/static/img/UECPtCLvFm-540.webp 540w"><img alt="Workers falling down a hole that was created after pathfinding" loading="lazy" decoding="async" src="/static/img/UECPtCLvFm-540.png" width="540" height="295"></picture></a><figcaption>Workers falling down a hole that was created after pathfinding</figcaption></figure>
<p>Another problem I kept running into were NPCs just doing nothing. One of the
times this happened was because NPCs kept being allocated to work that would
fail, perhaps it wasn’t reachable or wasn’t possible in some other way, Whilst I
could make the work manager check every precondition, it wouldn’t be very
flexible. I’d rather preconditions be implemented by decorators on the behavior
tree for each work type. My solution was to introduce failure lock outs: if a
piece of work fails, the NPC won’t retry it for 10 seconds.</p>
<h2 id="rimworld-regions" tabindex="-1">RimWorld Regions <a class="header-anchor" href="#rimworld-regions">#</a></h2>
<p>The task of finding the nearest work by walking distance can be expensive.
RimWorld has a fairly clever algorithm for doing this which I may investigate
in the future.</p>
<p>RimWorld uses a system of “regions” to make looking for work based on walking
distance super fast. Regions are essentially a higher-level pathfinding system -
instead of pathfinding based on nodes, it groups the map into regions at most
16x16 tiles in size, but further divided by walls. It remembers how regions are
connected. To find the closest work, you can iterate through the current and
nearby regions. The developer of RimWorld made
<a href="https://www.youtube.com/watch?v=RMBQn_sg7DA">an excellent video explaining regions</a>,
I highly recommend watching it.</p>
<figure class="undefined"><a href="/static/img/n6HAYv4Xdv-926.png"><picture><source type="image/webp" srcset="/static/img/PvC9R6TMcG-540.webp 540w"><img alt="RimWorld's <a href="https://www.youtube.com/watch?v=RMBQn_sg7DA">Region System</a>. Regions are at most 16x16 tiles, and then subdivided by walls. Connections between regions are stored.
" loading="lazy" decoding="async" src="/static/img/PvC9R6TMcG-540.png" width="540" height="408"></picture></a><figcaption>RimWorld's <a href="https://www.youtube.com/watch?v=RMBQn_sg7DA">Region System</a>. Regions are at most 16x16 tiles, and then subdivided by walls. Connections between regions are stored.
</figcaption></figure>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>My system is surprisingly effective despite being made out of simple rules and
components. While in the future I’ll probably implement more complicated work
allocation algorithms and a region system, it works well enough for now, allowing
me to focus on implementing other systems and game play mechanics.</p>
<h2 id="sources" tabindex="-1">Sources <a class="header-anchor" href="#sources">#</a></h2>
<ul>
<li><a href="https://www.gamedeveloper.com/programming/behavior-trees-for-ai-how-they-work">Behavior trees for AI: How they work</a></li>
<li><a href="https://www.youtube.com/watch?v=RMBQn_sg7DA">RimWorld Technology - Region System</a></li>
</ul>
rubenwardy
ChatGPT: An AI that can write Minetest mods... kinda
2022-06-22T00:00:00.000Z
2022-06-22T00:00:00.000Z
https://blog.rubenwardy.com/2022/06/22/gpt-3-minetest-modding/
An exploration of GPT-3/GitHub Copilot's knowledge of Minetest modding, with node registration, formspec manipulation, mod creation, and Minecraft to Minetest mod conversion.
<p>OpenAI’s <a href="https://en.wikipedia.org/wiki/GPT-3">GPT-3</a> is a powerful new
Artificial Intelligence model that uses deep learning to produce human-like
text. It was trained on a large body of text, with sources including websites,
Wikipedia, and books. It doesn’t just understand natural language, it can also
work with programming languages.</p>
<p>This topic is especially relevant with the recent introduction of
<a href="https://github.com/features/copilot">GitHub Copilot</a>.
Copilot is an “AI pair programmer” for your IDE that suggests code and entire
new functions. It’s based on same technology as GPT-3, but with a model derived
from GPT-3 and optimised for code called Codex. This article will use GPT-3 and
Codex, as I wasn’t able to get GitHub Copilot, but the results will be
identical.</p>
<p>In this article, I will ask GPT-3 to write Minetest code and explore how much it
knows about modding, creating simple and advanced Minetest mods. I will finish
by using it to convert Minecraft mods to Minetest.</p>
<p>So, what is <a href="https://www.minetest.net/">Minetest</a>? Let’s ask GPT-3 to explain:</p>
<p class="prompt">
Explain what Minetest is and what its benefits are
</p>
<blockquote>
<p>Minetest is a free and open-source voxel game engine with easy modding and
game creation capabilities. It has a strong emphasis on community-created
content and user-friendliness. Some of Minetest’s benefits include being able
to run on a wide variety of operating systems and hardware, having a fully
customizable game engine, and being able to create and share mods and other
content easily.</p>
</blockquote>
<p>And how is GPT-3 able to write Minetest code?</p>
<p class="prompt">
Explain how GPT-3 knows and is able to write Minetest code
</p>
<blockquote>
<p>GPT-3 has been trained on a large number of code examples, including code for
Minetest. As a result, it is able to identify patterns in the code and
generate new code that is similar to the examples it has seen.</p>
</blockquote>
<p>GPT-3 has a random element, and so can generate a different answer each time.
This is expressed as temperature - the higher the <em>temperature</em>, the more
randomness and the more creative the answers will be. But higher temperatures
are also less accurate.</p>
<p>A common bias when people assess the output of GPT-3 is to select the best
answers. I’ll be using the first answer in most cases, and will try to keep
temperature low.</p>
<h2 id="generating-simple-minetest-mods" tabindex="-1">Generating simple Minetest mods <a class="header-anchor" href="#generating-simple-minetest-mods">#</a></h2>
<h3 id="registering-a-node" tabindex="-1">Registering a node <a class="header-anchor" href="#registering-a-node">#</a></h3>
<p>A node in Minetest is a position in the 3d grid, it’s known as a Block in
other games. For example, a node could be air, grass, dirt, or something else.</p>
<p>Most Minetest modding tutorials start with registering a type of node, which
means that there will be a lot of documentation on the Internet about doing
this. It seems likely that some of this information found its way into GPT-3’s
training data.</p>
<p class="prompt">
Write Lua code that adds a node in Minetest
</p>
<blockquote class="my-0">
Write Lua code that adds a node in Minetest<mark>'s world</mark>
</blockquote>
<pre class="language-lua"><code class="language-lua">minetest<span class="token punctuation">.</span><span class="token function">add_node</span><span class="token punctuation">(</span><span class="token punctuation">{</span>x<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">,</span> y<span class="token operator">=</span><span class="token number">2</span><span class="token punctuation">,</span> z<span class="token operator">=</span><span class="token number">3</span><span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span>name<span class="token operator">=</span><span class="token string">"default:dirt"</span><span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<p>Huh. It modified the prompt to be about adding a node to Minetest’s world,
rather than registering a node. I’m using GPT-3 in autocomplete mode here, where
it attempts to finish the prompt you give it. In this situation, it decided that
changing the request was the more likely completion. One way to fix this is by
adding a full stop.</p>
<p>Running the prompt again resulted in a valid node registration, so it can work
out the meaning, but not very reliably. It doesn’t help that “add” is a bit
ambiguous. I changed “adds” to “registers” to be a bit more specific:</p>
<p class="prompt">
Write Lua code that registers a node in Minetest
</p>
<pre class="language-lua"><code class="language-lua">minetest<span class="token punctuation">.</span><span class="token function">register_node</span><span class="token punctuation">(</span><span class="token string">"my_node"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span>
description <span class="token operator">=</span> <span class="token string">"My Node"</span><span class="token punctuation">,</span>
tiles <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">"my_node.png"</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
groups <span class="token operator">=</span> <span class="token punctuation">{</span>cracky<span class="token operator">=</span><span class="token number">3</span><span class="token punctuation">}</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<p>OK, cool it can do that. The node name doesn’t follow conventions, but that’s
forgivable.</p>
<h3 id="a-node-that-quacks" tabindex="-1">A node that quacks <a class="header-anchor" href="#a-node-that-quacks">#</a></h3>
<p>But how much does it actually understand it? This code is very simple, and it
would be easy for it to just echo it from tutorials. Let’s change the spec a
little bit, to make it more unique:</p>
<p class="prompt">
Write Lua code that registers a node in Minetest that says quack when you punch it
</p>
<pre class="language-lua"><code class="language-lua">minetest<span class="token punctuation">.</span><span class="token function">register_node</span><span class="token punctuation">(</span><span class="token string">"quack"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span>
description <span class="token operator">=</span> <span class="token string">"Quack"</span><span class="token punctuation">,</span>
tiles <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">"default_wood.png"</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
on_punch <span class="token operator">=</span> <span class="token keyword">function</span><span class="token punctuation">(</span>pos<span class="token punctuation">,</span> node<span class="token punctuation">,</span> player<span class="token punctuation">,</span> pointed_thing<span class="token punctuation">)</span>
minetest<span class="token punctuation">.</span><span class="token function">chat_send_player</span><span class="token punctuation">(</span>player<span class="token punctuation">:</span><span class="token function">get_player_name</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">"Quack!"</span><span class="token punctuation">)</span>
<span class="token keyword">end</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<figure class="undefined"><a href="/static/img/wpbqVWJIbQ-640.png"><picture><source type="image/webp" srcset="/static/img/YccDTlAZB0-540.webp 540w"><img alt="The quack node generated by GPT-3." loading="lazy" decoding="async" src="/static/img/YccDTlAZB0-540.png" width="540" height="291"></picture></a><figcaption>The quack node generated by GPT-3.</figcaption></figure>
<p>That’s cool. It’s able to understand that “say” means a chat message, and has
also inferred a node name and description based on the prompt. The node name
still doesn’t match the naming conventions, but the description is correctly in
title case. GPT-3 is designed for finding patterns, so this isn’t too
surprising.</p>
<h3 id="duck-goose-node" tabindex="-1">Duck-Goose node <a class="header-anchor" href="#duck-goose-node">#</a></h3>
<p>This is still quite a simple example, let’s see how it does with a more
complicated problem:</p>
<p class="prompt">
Write Lua code that registers a node in Minetest that says duck when you punch
it once, and goose when you punch it twice
</p>
<pre class="language-lua"><code class="language-lua"><span class="token keyword">function</span> <span class="token function">duck_goose</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> clicker<span class="token punctuation">)</span>
<span class="token keyword">if</span> <span class="token keyword">not</span> clicker <span class="token keyword">or</span> <span class="token keyword">not</span> clicker<span class="token punctuation">:</span><span class="token function">is_player</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token keyword">then</span>
<span class="token keyword">return</span>
<span class="token keyword">end</span>
<span class="token keyword">if</span> self<span class="token punctuation">.</span>object<span class="token punctuation">:</span><span class="token function">get_luaentity</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>ducks <span class="token operator">==</span> <span class="token number">1</span> <span class="token keyword">then</span>
self<span class="token punctuation">.</span>object<span class="token punctuation">:</span><span class="token function">set_properties</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
textures <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">"goose.png"</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
self<span class="token punctuation">.</span>object<span class="token punctuation">:</span><span class="token function">get_luaentity</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>ducks <span class="token operator">=</span> <span class="token number">2</span>
<span class="token keyword">elseif</span> self<span class="token punctuation">.</span>object<span class="token punctuation">:</span><span class="token function">get_luaentity</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>ducks <span class="token operator">==</span> <span class="token number">2</span> <span class="token keyword">then</span>
self<span class="token punctuation">.</span>object<span class="token punctuation">:</span><span class="token function">set_properties</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
textures <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">"duck.png"</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
self<span class="token punctuation">.</span>object<span class="token punctuation">:</span><span class="token function">get_luaentity</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>ducks <span class="token operator">=</span> <span class="token number">1</span>
<span class="token keyword">end</span>
<span class="token keyword">end</span>
minetest<span class="token punctuation">.</span><span class="token function">register_node</span><span class="token punctuation">(</span><span class="token string">"example:duck"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span>
description <span class="token operator">=</span> <span class="token string">"Duck/Goose"</span><span class="token punctuation">,</span>
tiles <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">"duck.png"</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
groups <span class="token operator">=</span> <span class="token punctuation">{</span>choppy<span class="token operator">=</span><span class="token number">2</span><span class="token punctuation">,</span>dig_immediate<span class="token operator">=</span><span class="token number">2</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
on_punch <span class="token operator">=</span> duck_goose<span class="token punctuation">,</span>
on_rightclick <span class="token operator">=</span> duck_goose<span class="token punctuation">,</span>
ducks <span class="token operator">=</span> <span class="token number">1</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<p>OK, this is interesting: it’s cargo-culting on steroids.
<a href="https://en.wikipedia.org/wiki/Cargo_cult_programming">Cargo-culting</a> is when
a programmer copies and combines code without understanding it. It’s mixed up
the entity and node APIs big time, and has also gotten “duck” and “goose” the
wrong way around. It’s also changing the texture rather than saying something.
Theoretically, the code almost makes sense though.</p>
<p>A big part of the problem is <em>how</em> the prompt is phrased. Making the prompt a
lot less ambiguous makes a big difference, unsurprisingly. I also switched to
Edit mode, which I found worked a bit better.</p>
<p class="prompt">
<b>Write this into Lua code for Minetest</b><br>
A node that alternatives between saying duck and goose when punched
</p>
<pre class="language-lua"><code class="language-lua">minetest<span class="token punctuation">.</span><span class="token function">register_node</span><span class="token punctuation">(</span><span class="token string">"duck_goose:duck_goose"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span>
description <span class="token operator">=</span> <span class="token string">"Duck Goose"</span><span class="token punctuation">,</span>
tiles <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">"duck_goose.png"</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
groups <span class="token operator">=</span> <span class="token punctuation">{</span>cracky<span class="token operator">=</span><span class="token number">3</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
on_punch <span class="token operator">=</span> <span class="token keyword">function</span><span class="token punctuation">(</span>pos<span class="token punctuation">,</span> node<span class="token punctuation">,</span> puncher<span class="token punctuation">)</span>
<span class="token keyword">local</span> meta <span class="token operator">=</span> minetest<span class="token punctuation">.</span><span class="token function">get_meta</span><span class="token punctuation">(</span>pos<span class="token punctuation">)</span>
<span class="token keyword">local</span> say <span class="token operator">=</span> meta<span class="token punctuation">:</span><span class="token function">get_string</span><span class="token punctuation">(</span><span class="token string">"say"</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> say <span class="token operator">==</span> <span class="token string">"duck"</span> <span class="token keyword">then</span>
minetest<span class="token punctuation">.</span><span class="token function">chat_send_all</span><span class="token punctuation">(</span><span class="token string">"Goose!"</span><span class="token punctuation">)</span>
meta<span class="token punctuation">:</span><span class="token function">set_string</span><span class="token punctuation">(</span><span class="token string">"say"</span><span class="token punctuation">,</span> <span class="token string">"goose"</span><span class="token punctuation">)</span>
<span class="token keyword">else</span>
minetest<span class="token punctuation">.</span><span class="token function">chat_send_all</span><span class="token punctuation">(</span><span class="token string">"Duck!"</span><span class="token punctuation">)</span>
meta<span class="token punctuation">:</span><span class="token function">set_string</span><span class="token punctuation">(</span><span class="token string">"say"</span><span class="token punctuation">,</span> <span class="token string">"duck"</span><span class="token punctuation">)</span>
<span class="token keyword">end</span>
<span class="token keyword">end</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<p>This works, although isn’t entirely optimal. It would be better to use an
integer rather than a string to keep track.</p>
<p>And yes, I did make a typo in the prompt.</p>
<h2 id="explaining-code" tabindex="-1">Explaining code <a class="header-anchor" href="#explaining-code">#</a></h2>
<h3 id="explaining-the-quack-node" tabindex="-1">Explaining the Quack node <a class="header-anchor" href="#explaining-the-quack-node">#</a></h3>
<p>It would be nice to find out just how well GPT-3 can understand the code. GPT-3
is very good at noticing patterns and regurgitating, but is it able to explain
what the code is doing?</p>
<p>Using edit mode, I asked it to add comments to the first Quack node:</p>
<p class="prompt">
Add comments to the code<br>
<i class="snip"><Quack node code></i>
</p>
<pre class="language-lua"><code class="language-lua"><span class="token comment">-- create a duck block</span>
minetest<span class="token punctuation">.</span><span class="token function">register_node</span><span class="token punctuation">(</span><span class="token string">"quack"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span>
<span class="token comment">-- set the description to 'Quack'</span>
description <span class="token operator">=</span> <span class="token string">"Quack"</span><span class="token punctuation">,</span>
<span class="token comment">-- use the default wood texture</span>
tiles <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">"default_wood.png"</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token comment">-- when the player punches the block, the chat will send them the string 'Quack!'</span>
on_punch <span class="token operator">=</span> <span class="token keyword">function</span><span class="token punctuation">(</span>pos<span class="token punctuation">,</span> node<span class="token punctuation">,</span> player<span class="token punctuation">,</span> pointed_thing<span class="token punctuation">)</span>
minetest<span class="token punctuation">.</span><span class="token function">chat_send_player</span><span class="token punctuation">(</span>player<span class="token punctuation">:</span><span class="token function">get_player_name</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">"Quack!"</span><span class="token punctuation">)</span>
<span class="token keyword">end</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<p>The results of this prompt varied quite a lot, from detailed comments to a
single comment for the code. The above was the result of the first run, but in
other runs GPT-3 was wrong (saying the node made a duck sound) or just unhelpful
(writing “this is a comment”).</p>
<p>Next, I thought I’d try getting it to explain the code as prose:</p>
<p class="prompt">
Explain what this code does<br>
<i class="snip"><Quack node code></i>
</p>
<blockquote>
<p>This code registers a new node type, “quack”, which has the default wood
texture and sends the player the message “Quack!” when punched</p>
</blockquote>
<p>Full marks there, I guess.</p>
<h3 id="explaining-actual-mod-code" tabindex="-1">Explaining actual mod code <a class="header-anchor" href="#explaining-actual-mod-code">#</a></h3>
<p>The Quack node is a fairly simple example, exactly the sort of thing you’d find
in a Minetest tutorial. How well does GPT-3 do on actual mod code?</p>
<figure class="undefined"><a href="/static/img/m-yjvXntFI-640.png"><picture><source type="image/webp" srcset="/static/img/OMZEW6dnKn-540.webp 540w"><img alt="Units fighting in Conquer, my RTS mod for Minetest." loading="lazy" decoding="async" src="/static/img/OMZEW6dnKn-540.png" width="540" height="289"></picture></a><figcaption>Units fighting in Conquer, my RTS mod for Minetest.</figcaption></figure>
<p>I asked GPT-3 to explain the
<a href="https://gitlab.com/rubenwardy/conquer/-/blob/master/src/units/HealthBarEntity.lua">code</a>
for health bars in
<a href="https://content.minetest.net/packages/rubenwardy/conquer/">Conquer</a>, my RTS
mod for Minetest.</p>
<p class="prompt">
Explain what this code does to a beginner<br>
<a href="https://gitlab.com/rubenwardy/conquer/-/blob/master/src/units/HealthBarEntity.lua" class="snip"><Conquer code></a>
</p>
<blockquote>
<p>This code creates an entity called a “healthbar” which is used to show a
player’s health in the game “conquer”. The entity is a sprite, which is an
image that is displayed in the game, and it is immortal, meaning it can not be
killed. The healthbar is updated based on the health of the player it is
attached to. If the player’s health changes, the healthbar will change to
reflect that. If the player dies, the healthbar will be removed.</p>
</blockquote>
<p>This is almost entirely correct, the only inaccuracy is that it’s attached to
a unit entity and not a player. There’s absolutely no way it could have known
this though, given the code snippet doesn’t mention units at all.</p>
<h2 id="working-with-guis-formspecs" tabindex="-1">Working with GUIs (Formspecs) <a class="header-anchor" href="#working-with-guis-formspecs">#</a></h2>
<h3 id="generating-a-simple-formspec" tabindex="-1">Generating a simple formspec <a class="header-anchor" href="#generating-a-simple-formspec">#</a></h3>
<p>Lua is a fairly common programming language, used by many more games and
programs than Minetest. But what about formspecs, a domain-specific language for
defining GUIs in Minetest?</p>
<p class="prompt">
Write a Minetest formspec with a field and two buttons
</p>
<pre class="language-lua"><code class="language-lua">form <span class="token operator">=</span> <span class="token string">"size[8,8]"</span> <span class="token operator">..</span>
<span class="token comment">-- titlebar</span>
<span class="token string">"button_exit[0,0;1,1;exit;X]"</span> <span class="token operator">..</span>
<span class="token comment">-- text input field</span>
<span class="token string">"field[0.5,1.5;7,1;text;Type your text here:;]"</span> <span class="token operator">..</span>
<span class="token comment">-- buttons</span>
<span class="token string">"button[3,3;2,1;button1;Button 1]"</span> <span class="token operator">..</span>
<span class="token string">"button[3,4;2,1;button2;Button 2]"</span></code></pre>
<figure class="undefined"><a href="/static/img/qcELBTK5FW-684.png"><picture><source type="image/webp" srcset="/static/img/Y2N-z6W6hN-540.webp 540w"><img alt="A simple formspec generated by GPT-3." loading="lazy" decoding="async" src="/static/img/Y2N-z6W6hN-540.png" width="540" height="313"></picture></a><figcaption>A simple formspec generated by GPT-3.</figcaption></figure>
<p>It took some creative liberty by adding a close button to the top, but it has
fulfilled the requirements.</p>
<h3 id="manipulating-formspecs" tabindex="-1">Manipulating formspecs <a class="header-anchor" href="#manipulating-formspecs">#</a></h3>
<p>What about formspec code that doesn’t have surrounding text? I provided it with
the following code snippet:</p>
<pre class="language-lua"><code class="language-lua">size<span class="token punctuation">[</span><span class="token number">8</span><span class="token punctuation">,</span><span class="token number">3</span><span class="token punctuation">]</span>
label<span class="token punctuation">[</span><span class="token number">0.5</span><span class="token punctuation">,</span><span class="token number">0.1</span><span class="token punctuation">;</span>Hello everyone!<span class="token punctuation">]</span>
field<span class="token punctuation">[</span><span class="token number">0.5</span><span class="token punctuation">,</span><span class="token number">1</span><span class="token punctuation">;</span><span class="token number">3.5</span><span class="token punctuation">,</span><span class="token number">1</span><span class="token punctuation">;</span>name<span class="token punctuation">;</span>Name<span class="token punctuation">;</span><span class="token punctuation">]</span></code></pre>
<p>I started by wondering if it could detect the language:</p>
<p class="prompt">
Write the name of the language in a comment<br>
<i class="snip"><formspec code></i>
</p>
<pre class="language-c"><code class="language-c"><span class="token comment">// Minetest Formspec</span>
<span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></code></pre>
<p>That’s cool! Weird that it choose <code>//</code>, given that formspecs are written inside
Lua which uses <code>--</code>. I guess <code>//</code> is more common.</p>
<p>What about manipulating the formspec?</p>
<p class="prompt">
Add another text box for age twice the width of the name field<br>
<i class="snip"><formspec code></i>
</p>
<pre class="language-lua"><code class="language-lua">size<span class="token punctuation">[</span><span class="token number">8</span><span class="token punctuation">,</span><span class="token number">3</span><span class="token punctuation">]</span>
label<span class="token punctuation">[</span><span class="token number">0.5</span><span class="token punctuation">,</span><span class="token number">0.1</span><span class="token punctuation">;</span>Hello everyone!<span class="token punctuation">]</span>
field<span class="token punctuation">[</span><span class="token number">0.5</span><span class="token punctuation">,</span><span class="token number">1</span><span class="token punctuation">;</span><span class="token number">3.5</span><span class="token punctuation">,</span><span class="token number">1</span><span class="token punctuation">;</span>name<span class="token punctuation">;</span>Name<span class="token punctuation">;</span><span class="token punctuation">]</span>
field<span class="token punctuation">[</span><span class="token number">0.5</span><span class="token punctuation">,</span><span class="token number">2</span><span class="token punctuation">;</span><span class="token number">7</span><span class="token punctuation">,</span><span class="token number">1</span><span class="token punctuation">;</span>age<span class="token punctuation">;</span>Age<span class="token punctuation">;</span><span class="token punctuation">]</span></code></pre>
<figure class="undefined"><a href="/static/img/e6dAsFhlda-693.png"><picture><source type="image/webp" srcset="/static/img/SQAWk4iWkG-540.webp 540w"><img alt="GPT-3 added an age field with double the width of Name." loading="lazy" decoding="async" src="/static/img/SQAWk4iWkG-540.png" width="540" height="232"></picture></a><figcaption>GPT-3 added an age field with double the width of Name.</figcaption></figure>
<p>That’s correct, it’s also placed it without overlap.
I used “text box” in the prompt to avoid biasing it by saying “field”.</p>
<p>Does it understand directions?</p>
<p class="prompt">
Add another field for age to the left of the name field<br>
<i class="snip"><formspec code></i>
</p>
<pre class="language-lua"><code class="language-lua">size<span class="token punctuation">[</span><span class="token number">8</span><span class="token punctuation">,</span><span class="token number">3</span><span class="token punctuation">]</span>
label<span class="token punctuation">[</span><span class="token number">0.5</span><span class="token punctuation">,</span><span class="token number">0.25</span><span class="token punctuation">;</span>Hello everyone!<span class="token punctuation">]</span>
field<span class="token punctuation">[</span><span class="token number">0.5</span><span class="token punctuation">,</span><span class="token number">1</span><span class="token punctuation">;</span><span class="token number">1.5</span><span class="token punctuation">,</span><span class="token number">1</span><span class="token punctuation">;</span>age<span class="token punctuation">;</span>Age<span class="token punctuation">;</span><span class="token punctuation">]</span>
field<span class="token punctuation">[</span><span class="token number">2</span><span class="token punctuation">,</span><span class="token number">1</span><span class="token punctuation">;</span><span class="token number">3.5</span><span class="token punctuation">,</span><span class="token number">1</span><span class="token punctuation">;</span>name<span class="token punctuation">;</span>Name<span class="token punctuation">;</span><span class="token punctuation">]</span></code></pre>
<figure class="undefined"><a href="/static/img/igia820Wl7-699.png"><picture><source type="image/webp" srcset="/static/img/gQMm5J7Fd--540.webp 540w"><img alt="GPT-3 moved the name field and added the Age field to left of Name." loading="lazy" decoding="async" src="/static/img/gQMm5J7Fd--540.png" width="540" height="236"></picture></a><figcaption>GPT-3 moved the name field and added the Age field to left of Name.</figcaption></figure>
<p>It even moved the name field out of the way, rather than going into the
negatives.</p>
<h3 id="guessing-game" tabindex="-1">Guessing game <a class="header-anchor" href="#guessing-game">#</a></h3>
<p>Next, I wondered if I could get it to generate a guessing game formspec, with
a formspec submission callback.</p>
<p class="prompt">
<b>Write into Lua as a Minetest mod</b><br>
Create a guessing game formspec
</p>
<pre class="language-lua"><code class="language-lua">minetest<span class="token punctuation">.</span><span class="token function">register_on_player_receive_fields</span><span class="token punctuation">(</span><span class="token keyword">function</span><span class="token punctuation">(</span>player<span class="token punctuation">,</span> formname<span class="token punctuation">,</span> fields<span class="token punctuation">)</span>
<span class="token keyword">if</span> formname <span class="token operator">~=</span> <span class="token string">"guess"</span> <span class="token keyword">then</span>
<span class="token keyword">return</span>
<span class="token keyword">end</span>
<span class="token keyword">if</span> fields<span class="token punctuation">.</span>quit <span class="token keyword">then</span>
<span class="token keyword">return</span>
<span class="token keyword">end</span>
<span class="token keyword">local</span> guess <span class="token operator">=</span> <span class="token function">tonumber</span><span class="token punctuation">(</span>fields<span class="token punctuation">.</span>guess<span class="token punctuation">)</span>
<span class="token keyword">if</span> guess <span class="token operator">==</span> <span class="token keyword">nil</span> <span class="token keyword">then</span>
minetest<span class="token punctuation">.</span><span class="token function">chat_send_player</span><span class="token punctuation">(</span>player<span class="token punctuation">:</span><span class="token function">get_player_name</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">"Please enter a number."</span><span class="token punctuation">)</span>
<span class="token keyword">return</span>
<span class="token keyword">end</span>
<span class="token keyword">if</span> guess <span class="token operator"><</span> <span class="token number">1</span> <span class="token keyword">or</span> guess <span class="token operator">></span> <span class="token number">100</span> <span class="token keyword">then</span>
minetest<span class="token punctuation">.</span><span class="token function">chat_send_player</span><span class="token punctuation">(</span>player<span class="token punctuation">:</span><span class="token function">get_player_name</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">"Please enter a number between 1 and 100."</span><span class="token punctuation">)</span>
<span class="token keyword">return</span>
<span class="token keyword">end</span>
<span class="token keyword">if</span> guess <span class="token operator">==</span> <span class="token number">42</span> <span class="token keyword">then</span>
minetest<span class="token punctuation">.</span><span class="token function">chat_send_player</span><span class="token punctuation">(</span>player<span class="token punctuation">:</span><span class="token function">get_player_name</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">"You win!"</span><span class="token punctuation">)</span>
<span class="token keyword">else</span>
minetest<span class="token punctuation">.</span><span class="token function">chat_send_player</span><span class="token punctuation">(</span>player<span class="token punctuation">:</span><span class="token function">get_player_name</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">"Sorry, try again."</span><span class="token punctuation">)</span>
<span class="token keyword">end</span>
<span class="token keyword">end</span><span class="token punctuation">)</span>
minetest<span class="token punctuation">.</span><span class="token function">register_chatcommand</span><span class="token punctuation">(</span><span class="token string">"guess"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span>
params <span class="token operator">=</span> <span class="token string">""</span><span class="token punctuation">,</span>
description <span class="token operator">=</span> <span class="token string">"Play a guessing game"</span><span class="token punctuation">,</span>
func <span class="token operator">=</span> <span class="token keyword">function</span><span class="token punctuation">(</span>name<span class="token punctuation">,</span> param<span class="token punctuation">)</span>
<span class="token keyword">local</span> player <span class="token operator">=</span> minetest<span class="token punctuation">.</span><span class="token function">get_player_by_name</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span>
<span class="token keyword">if</span> player <span class="token operator">==</span> <span class="token keyword">nil</span> <span class="token keyword">then</span>
<span class="token comment">-- just a check to prevent the server crashing</span>
<span class="token keyword">return</span> <span class="token keyword">false</span><span class="token punctuation">,</span> <span class="token string">"Player not found"</span>
<span class="token keyword">end</span>
minetest<span class="token punctuation">.</span><span class="token function">show_formspec</span><span class="token punctuation">(</span>name<span class="token punctuation">,</span> <span class="token string">"guess"</span><span class="token punctuation">,</span>
<span class="token string">"size[8,4]"</span> <span class="token operator">..</span>
<span class="token string">"label[0,0;I'm thinking of a number between 1 and 100.]"</span> <span class="token operator">..</span>
<span class="token string">"field[2.5,2;3,1;guess;Guess;]"</span> <span class="token operator">..</span>
<span class="token string">"button_exit[2.5,3;3,1;quit;Quit]"</span><span class="token punctuation">)</span>
<span class="token keyword">end</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<figure class="undefined"><a href="/static/img/H39fA2q44Q-702.png"><picture><source type="image/webp" srcset="/static/img/eO0D14Yg6z-540.webp 540w"><img alt="A guessing game generated by GPT-3. It's missing a submit button to make a guess, but works when this is added." loading="lazy" decoding="async" src="/static/img/eO0D14Yg6z-540.png" width="540" height="292"></picture></a><figcaption>A guessing game generated by GPT-3. It's missing a submit button to make a guess, but works when this is added.</figcaption></figure>
<p>OK, now it’s scaring me.</p>
<p>It’s missing a way to submit the guess - there’s only a quit button. It works if
you add this, though.</p>
<p>I found the “just a check to prevent the server crashing” comment quite
interesting. I wondered whether this was taken straight out of Minetest mods.
A quick <a href="https://github.com/search?l=Lua&q=just+a+check+to+prevent+the+server+crashing&type=Code">search</a>
on GitHub revealed that this exact comment is used quite frequently by mods
in chat commands.</p>
<h3 id="list-of-fruit-editor" tabindex="-1">List of fruit editor <a class="header-anchor" href="#list-of-fruit-editor">#</a></h3>
<p>Guessing games are fairly common, what about something more complicated?</p>
<p class="prompt">
<b>Write into Lua as a Minetest mod</b><br>
Create a formspec that shows and allows maintaining a list of fruit
</p>
<pre class="language-lua"><code class="language-lua">minetest<span class="token punctuation">.</span><span class="token function">register_on_player_receive_fields</span><span class="token punctuation">(</span><span class="token keyword">function</span><span class="token punctuation">(</span>player<span class="token punctuation">,</span> formname<span class="token punctuation">,</span> fields<span class="token punctuation">)</span>
<span class="token keyword">if</span> formname <span class="token operator">~=</span> <span class="token string">"fruit_list"</span> <span class="token keyword">then</span>
<span class="token keyword">return</span>
<span class="token keyword">end</span>
<span class="token keyword">if</span> fields<span class="token punctuation">.</span>add <span class="token keyword">then</span>
<span class="token keyword">local</span> fruit <span class="token operator">=</span> fields<span class="token punctuation">.</span>fruit
<span class="token keyword">if</span> fruit <span class="token operator">~=</span> <span class="token string">""</span> <span class="token keyword">then</span>
table<span class="token punctuation">.</span><span class="token function">insert</span><span class="token punctuation">(</span>fruit_list<span class="token punctuation">,</span> fruit<span class="token punctuation">)</span>
<span class="token keyword">end</span>
<span class="token keyword">elseif</span> fields<span class="token punctuation">.</span>remove <span class="token keyword">then</span>
<span class="token keyword">local</span> fruit <span class="token operator">=</span> fields<span class="token punctuation">.</span>fruit
<span class="token keyword">if</span> fruit <span class="token operator">~=</span> <span class="token string">""</span> <span class="token keyword">then</span>
<span class="token keyword">for</span> i<span class="token punctuation">,</span> v <span class="token keyword">in</span> <span class="token function">ipairs</span><span class="token punctuation">(</span>fruit_list<span class="token punctuation">)</span> <span class="token keyword">do</span>
<span class="token keyword">if</span> v <span class="token operator">==</span> fruit <span class="token keyword">then</span>
table<span class="token punctuation">.</span><span class="token function">remove</span><span class="token punctuation">(</span>fruit_list<span class="token punctuation">,</span> i<span class="token punctuation">)</span>
<span class="token keyword">break</span>
<span class="token keyword">end</span>
<span class="token keyword">end</span>
<span class="token keyword">end</span>
<span class="token keyword">end</span>
<span class="token function">show_formspec</span><span class="token punctuation">(</span>player<span class="token punctuation">)</span>
<span class="token keyword">end</span><span class="token punctuation">)</span>
<span class="token keyword">function</span> <span class="token function">show_formspec</span><span class="token punctuation">(</span>player<span class="token punctuation">)</span>
<span class="token keyword">local</span> formspec <span class="token operator">=</span> <span class="token string">"size[8,9]"</span> <span class="token operator">..</span>
<span class="token string">"label[0,0;Fruit List]"</span> <span class="token operator">..</span>
<span class="token string">"field[0.3,1;4,1;fruit;Fruit;]"</span> <span class="token operator">..</span>
<span class="token string">"button[4.3,0.5;1,1;add;Add]"</span> <span class="token operator">..</span>
<span class="token string">"button[5.3,0.5;1,1;remove;Remove]"</span>
<span class="token keyword">for</span> i<span class="token punctuation">,</span> v <span class="token keyword">in</span> <span class="token function">ipairs</span><span class="token punctuation">(</span>fruit_list<span class="token punctuation">)</span> <span class="token keyword">do</span>
formspec <span class="token operator">=</span> formspec <span class="token operator">..</span> <span class="token string">"label[0,"</span> <span class="token operator">..</span> <span class="token punctuation">(</span>i <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">)</span> <span class="token operator">..</span> <span class="token string">";"</span> <span class="token operator">..</span> v <span class="token operator">..</span> <span class="token string">"]"</span>
<span class="token keyword">end</span>
minetest<span class="token punctuation">.</span><span class="token function">show_formspec</span><span class="token punctuation">(</span>player<span class="token punctuation">:</span><span class="token function">get_player_name</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">"fruit_list"</span><span class="token punctuation">,</span> formspec<span class="token punctuation">)</span>
<span class="token keyword">end</span>
minetest<span class="token punctuation">.</span><span class="token function">register_chatcommand</span><span class="token punctuation">(</span><span class="token string">"fruit"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span>
params <span class="token operator">=</span> <span class="token string">""</span><span class="token punctuation">,</span>
description <span class="token operator">=</span> <span class="token string">"Show fruit list"</span><span class="token punctuation">,</span>
func <span class="token operator">=</span> <span class="token keyword">function</span><span class="token punctuation">(</span>name<span class="token punctuation">,</span> param<span class="token punctuation">)</span>
<span class="token keyword">local</span> player <span class="token operator">=</span> minetest<span class="token punctuation">.</span><span class="token function">get_player_by_name</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span>
<span class="token keyword">if</span> player <span class="token operator">==</span> <span class="token keyword">nil</span> <span class="token keyword">then</span>
<span class="token keyword">return</span> <span class="token keyword">false</span><span class="token punctuation">,</span> <span class="token string">"Player not found"</span>
<span class="token keyword">end</span>
<span class="token function">show_formspec</span><span class="token punctuation">(</span>player<span class="token punctuation">)</span>
<span class="token keyword">return</span> <span class="token keyword">true</span><span class="token punctuation">,</span> <span class="token string">"Fruit list shown"</span>
<span class="token keyword">end</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<figure class="undefined"><a href="/static/img/4zttYByVsz-693.png"><picture><source type="image/webp" srcset="/static/img/wmiIk-JTDw-540.webp 540w"><img alt="A fruit list formspec generated by GPT-3, it works." loading="lazy" decoding="async" src="/static/img/wmiIk-JTDw-540.png" width="540" height="300"></picture></a><figcaption>A fruit list formspec generated by GPT-3, it works.</figcaption></figure>
<p>Wow. It’s just missing a definition of <code>fruit_list</code>, and should be using
<code>minetest.formspec_escape</code> to handle the case where a fruit name has special
characters. I think it’s interesting how it went with labels instead of a text
list.</p>
<h2 id="minecraft-to-minetest-conversion" tabindex="-1">Minecraft to Minetest conversion <a class="header-anchor" href="#minecraft-to-minetest-conversion">#</a></h2>
<p>One thing that could indicate deep understanding would be the ability to convert
between different modding APIs. I found the following hello world example
in a
<a href="https://medium.com/@programmingfun2017/minecraft-modding-with-forge-tutorial-hello-world-828ece8db9a2">Minecraft Forge modding tutorial</a>:</p>
<pre class="language-java"><code class="language-java"><span class="token keyword">package</span> <span class="token namespace">minecraftfun</span><span class="token punctuation">;</span>
<span class="token keyword">import</span> <span class="token import"><span class="token namespace">net<span class="token punctuation">.</span>minecraft<span class="token punctuation">.</span>util<span class="token punctuation">.</span>text<span class="token punctuation">.</span></span><span class="token class-name">TextComponentString</span></span><span class="token punctuation">;</span>
<span class="token keyword">import</span> <span class="token import"><span class="token namespace">net<span class="token punctuation">.</span>minecraftforge<span class="token punctuation">.</span>event<span class="token punctuation">.</span>world<span class="token punctuation">.</span></span><span class="token class-name">BlockEvent</span><span class="token punctuation">.</span><span class="token class-name">PlaceEvent</span></span><span class="token punctuation">;</span>
<span class="token keyword">import</span> <span class="token import"><span class="token namespace">net<span class="token punctuation">.</span>minecraftforge<span class="token punctuation">.</span>fml<span class="token punctuation">.</span>common<span class="token punctuation">.</span>eventhandler<span class="token punctuation">.</span></span><span class="token class-name">SubscribeEvent</span></span><span class="token punctuation">;</span>
<span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">BlockPlaceHandler</span> <span class="token punctuation">{</span>
<span class="token annotation punctuation">@SubscribeEvent</span>
<span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">onPlaceEvent</span><span class="token punctuation">(</span><span class="token class-name">PlaceEvent</span> event<span class="token punctuation">)</span> <span class="token punctuation">{</span>
event<span class="token punctuation">.</span><span class="token function">getPlayer</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">sendMessage</span><span class="token punctuation">(</span>
<span class="token keyword">new</span> <span class="token class-name">TextComponentString</span><span class="token punctuation">(</span><span class="token string">"You placed a block : "</span> <span class="token operator">+</span> event<span class="token punctuation">.</span><span class="token function">getPlacedBlock</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">getBlock</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">getLocalizedName</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>The first prompt I tried ended up just changing the package name in the first
line to <code>minetest</code> rather than rewriting it into Lua.</p>
<p class="prompt">
Rewrite the following code into Minetest<br>
<i class="snip"><Java code></i>
</p>
<pre class="language-java"><code class="language-java"><span class="token keyword">package</span> <span class="token namespace">minetestfun</span><span class="token punctuation">;</span>
<span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></code></pre>
<p>The second time, I explicitly mentioned Lua:</p>
<p class="prompt">
Write into Lua as a Minetest mod<br>
<i class="snip"><Java code></i>
</p>
<pre class="language-java"><code class="language-java"><span class="token operator">--</span><span class="token punctuation">[</span><span class="token punctuation">[</span>
<span class="token class-name">This</span> is a <span class="token class-name">Minetest</span> mod that writes into a file when a block is <span class="token class-name"><span class="token namespace">placed<span class="token punctuation">.</span></span>
It</span> is written in <span class="token class-name">Lua</span><span class="token punctuation">.</span>
<span class="token punctuation">]</span><span class="token punctuation">]</span>
<span class="token keyword">import</span> <span class="token import"><span class="token namespace">net<span class="token punctuation">.</span>minecraft<span class="token punctuation">.</span>util<span class="token punctuation">.</span>text<span class="token punctuation">.</span></span><span class="token class-name">TextComponentString</span></span><span class="token punctuation">;</span>
<span class="token keyword">import</span> <span class="token import"><span class="token namespace">net<span class="token punctuation">.</span>minecraftforge<span class="token punctuation">.</span>event<span class="token punctuation">.</span>world<span class="token punctuation">.</span></span><span class="token class-name">BlockEvent</span><span class="token punctuation">.</span><span class="token class-name">PlaceEvent</span></span><span class="token punctuation">;</span>
<span class="token keyword">import</span> <span class="token import"><span class="token namespace">net<span class="token punctuation">.</span>minecraftforge<span class="token punctuation">.</span>fml<span class="token punctuation">.</span>common<span class="token punctuation">.</span>eventhandler<span class="token punctuation">.</span></span><span class="token class-name">SubscribeEvent</span></span><span class="token punctuation">;</span>
<span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">BlockPlaceHandler</span> <span class="token punctuation">{</span>
<span class="token annotation punctuation">@SubscribeEvent</span>
<span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">onPlaceEvent</span><span class="token punctuation">(</span><span class="token class-name">PlaceEvent</span> event<span class="token punctuation">)</span> <span class="token punctuation">{</span>
event<span class="token punctuation">.</span><span class="token function">getPlayer</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">sendMessage</span><span class="token punctuation">(</span>
<span class="token keyword">new</span> <span class="token class-name">TextComponentString</span><span class="token punctuation">(</span><span class="token string">"You placed a block : "</span> <span class="token operator">+</span> event<span class="token punctuation">.</span><span class="token function">getPlacedBlock</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">getBlock</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">getLocalizedName</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>No, adding a comment saying “It is written in Lua” doesn’t make it written in
Lua. The comment about writing to a file is also completely incorrect.</p>
<p>Next, I went back to saying “Rewrite”:</p>
<div class="prompt">
Rewrite the following into Lua as a Minetest mod<br>
<i class="snip"><Java code></i>
</div>
<pre class="language-lua"><code class="language-lua">minetest<span class="token punctuation">.</span><span class="token function">register_on_placenode</span><span class="token punctuation">(</span><span class="token keyword">function</span><span class="token punctuation">(</span>pos<span class="token punctuation">,</span> newnode<span class="token punctuation">,</span> placer<span class="token punctuation">,</span> oldnode<span class="token punctuation">,</span> itemstack<span class="token punctuation">,</span> pointed_thing<span class="token punctuation">)</span>
<span class="token keyword">if</span> placer<span class="token punctuation">:</span><span class="token function">is_player</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token keyword">then</span>
minetest<span class="token punctuation">.</span><span class="token function">chat_send_player</span><span class="token punctuation">(</span>placer<span class="token punctuation">:</span><span class="token function">get_player_name</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">"You placed a block : "</span><span class="token operator">..</span>newnode<span class="token punctuation">.</span>name<span class="token punctuation">)</span>
<span class="token keyword">end</span>
<span class="token keyword">end</span><span class="token punctuation">)</span></code></pre>
<p>Woah! That’s exactly what I wanted. It even checks that <code>placer</code> is a player,
which is something that many modders forget to do.</p>
<p>The only thing it does do wrong is <code>newnode.name</code>. It should get the node’s
description from the node definition, this can either be done using
<code>ItemStack(newnode.name):get_description()</code> or by looking up the node def from
<code>minetest.registered_nodes</code>. This mistake is understandable though given that
the Java code uses “name” for what Minetest calls “description”.</p>
<p>As mentioned, GPT-3 includes randomness; it doesn’t always produce the same
result. I ran it again, with a lower temperature, and it made up an API function
called <code>send_message</code>:</p>
<div class="prompt">
Rewrite the following into Lua as a Minetest mod<br>
<i class="snip"><Java code></i>
</div>
<pre class="language-lua"><code class="language-lua">minetest<span class="token punctuation">.</span><span class="token function">register_on_placenode</span><span class="token punctuation">(</span><span class="token keyword">function</span><span class="token punctuation">(</span>pos<span class="token punctuation">,</span> newnode<span class="token punctuation">,</span> placer<span class="token punctuation">,</span> oldnode<span class="token punctuation">,</span> itemstack<span class="token punctuation">,</span> pointed_thing<span class="token punctuation">)</span>
placer<span class="token punctuation">:</span><span class="token function">send_message</span><span class="token punctuation">(</span><span class="token string">"You placed a block: "</span><span class="token operator">..</span>newnode<span class="token punctuation">.</span>name<span class="token punctuation">)</span>
<span class="token keyword">end</span><span class="token punctuation">)</span></code></pre>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>GPT-3/Codex is surprisingly intelligent and can produce working Minetest mods and
formspecs. However, it frequently produces results that look like
<a href="https://en.wikipedia.org/wiki/Cargo_cult_programming">cargo-culting</a> and making
stuff up, and it can also produce vulnerable code. It also requires effort from
a human to write good prompts and identify problems.</p>
<p>GitHub’s Copilot uses another AI model from OpenAI called Codex. It’s very
closely related to GPT-3 but specialises in code rather than natural language.
Copilot uses this model in IDEs, such as VSCode, to suggest code.</p>
<p>I’ll probably look into GitHub Copilot in the future, but I imagine it’ll
produce very similar results just with better IDE integration.</p>
<p>GPT-3/Codex learned how to write Minetest code by reading code on the Internet.
This code may or may not be open source, and may or may not be permissively
licensed. These models tend to regurgitate code, which leads to license
laundering - open source code being turned into proprietary code, without
credit. These products benefit from the unpaid labour of the open-source
community. So whilst this technology is interesting, I’m not sure how much I
agree with it ethically.</p>
<p>You can try out GPT-3 and Codex for free on
<a href="https://beta.openai.com/playground">OpenAI’s website</a>, and
<a href="https://github.com/features/copilot/">GitHub Copilot</a> is now available
publicly.</p>
<h2 id="tl-dr" tabindex="-1">TL;DR <a class="header-anchor" href="#tl-dr">#</a></h2>
<ul>
<li>It can state some facts about Minetest</li>
<li>It can create small mods based on human language descriptions:
<ul>
<li><a href="#a-node-that-quacks">A node that says quack when you punch it</a></li>
<li><a href="#duck-goose-node">A node that alternates between saying duck and goose when punched</a></li>
<li><a href="#guessing-game">A guessing game GUI</a></li>
<li><a href="#list-of-fruit-editor">A GUI to manage and view a list of fruit</a></li>
</ul>
</li>
<li>It can <a href="#explaining-code">explain what Minetest code does</a> in comments or prose</li>
<li>It can understand Minetest’s domain-specific language for GUIs (formspecs)
and <a href="#working-with-guis-formspecs">can manipulate them</a></li>
<li>It can <a href="#minecraft-to-minetest-conversion">convert Minecraft Java code into Minetest Lua code</a></li>
</ul>
rubenwardy
Developing ContentDB
2022-03-24T00:00:00.000Z
2022-03-24T00:00:00.000Z
https://blog.rubenwardy.com/2022/03/24/contentdb/
How I created ContentDB, the website that allows players to install mods, games, and texture packs from within Minetest.
<p>In 2018, I had the opportunity to create a
<a href="https://content.minetest.net/">web app</a> for University coursework, as a solo
project. I chose to create a package repository for Minetest, an open-source
project I help maintain.</p>
<p><a href="https://www.minetest.net/">Minetest</a> is an open-source game engine with
millions of downloads and thousands of weekly players. The project has a very
active modding community, and many available games to run. There was one big
issue - you had to manually install mods and games by unzipping their files into
a directory. This was a very poor user experience.</p>
<p>The project aimed to make a website to hold the metadata of different types of
packages. The website needed to have both an HTML front-end, and a JSON REST API
to be used by the Minetest Engine and other software. Authors of packages should
be able to upload and maintain their packages, and Editors should be able to
upload and maintain any package.</p>
<p>The community commonly receives attacks from bots and malicious users, so the
system needed sufficient moderator tools. New users should have any uploads or
changes checked before they’re published. There also needed be moderation tools,
such as banning.</p>
<p>The website needed to be stable and easy-to-use for content creators and users
alike. This means that uploading packages needed to be as frictionless as
possible, which calls for interesting features such as importing metadata from
GitHub and other VCSes.</p>
<h2 id="prior-attempts" tabindex="-1">Prior Attempts <a class="header-anchor" href="#prior-attempts">#</a></h2>
<p>Creating a package repository for Minetest was by no means a brand new idea;
there had been many prior attempts, a few of which by me.</p>
<h3 id="2012-minetest-extensions" tabindex="-1">2012: Minetest Extensions <a class="header-anchor" href="#2012-minetest-extensions">#</a></h3>
<p>I created a PHP website called Minetest Extensions. It used a MySQL
database and implemented several proprietary APIs required by existing Minetest
package manager command-line tools. I wasn’t experienced with back-end
development at the time, and so it suffered from bugs and security issues.</p>
<h3 id="2013-minetest-mod-database" tabindex="-1">2013: Minetest Mod Database <a class="header-anchor" href="#2013-minetest-mod-database">#</a></h3>
<p>Another user created a Python / Django package repository. It was made
official and hosted by celeron55.</p>
<p>It never had many mods due to a very manual data entry process, which could only
be performed by the mod author.</p>
<p>It only supported mods and texture packs. Any mods were required to be mods to
the strict definition; Mod packs were not allowed. This was a huge problem given
that many of the most important mods are structured into mod packs.</p>
<p>The website went offline due to the database breaking and the host being unable
to fix it. It didn’t use containerisation and was hard to set up and develop.
The developer was also only sporadically available.</p>
<h3 id="2015-minetest-bower" tabindex="-1">2015: Minetest Bower <a class="header-anchor" href="#2015-minetest-bower">#</a></h3>
<p>Another user created a package repository based on <a href="https://bower.io">Bower</a>, a Git-based
package manager for the web. Using it for Lua-based packages was a bit of an
abuse of the tool, but it did work.</p>
<p>The major issue was that it only supported content with a Git repository, and
required the owner to update the repository to add a bower.json file. This
resulted in even fewer mods being added than to the Minetest Mod Database.</p>
<h3 id="2016-minetest-mods-android-app" tabindex="-1">2016: Minetest Mods Android App <a class="header-anchor" href="#2016-minetest-mods-android-app">#</a></h3>
<p>I created <a href="/2022/10/22/Minetest-mods-app/">an Android app</a> to
install mods. It used a NodeJS backend that crawled the forums to get mod
information. This worked to a degree, but the data was flawed and required me to
manually override a lot of information that the crawler didn’t detect properly.</p>
<h3 id="2017-nodejs-based-contentdb" tabindex="-1">2017: NodeJS-based ContentDB <a class="header-anchor" href="#2017-nodejs-based-contentdb">#</a></h3>
<p>I wrote a prototype using NodeJS, Sequelize, and PostgreSQL for a package
repository that had goals similar to ContentDB. It had support for Asynchronous
Tasks (Git importing and automatic releases), a user login system, and a
REST API.</p>
<p>This project allowed me to assess the suitability of NodeJS for the task; I
got negative reactions from other core developers. I also wasn’t a fan of
Sequelize as Python’s SQLAlchemy was so much better.</p>
<h3 id="2017-minetest-content-database" tabindex="-1">2017: Minetest Content Database <a class="header-anchor" href="#2017-minetest-content-database">#</a></h3>
<p>Another user created a package repository based on Python / Django.
It supported mods, games, texture packs, skins, and servers.</p>
<p>Development faded out by the start of 2018.</p>
<h2 id="client-requirements" tabindex="-1">Client Requirements <a class="header-anchor" href="#client-requirements">#</a></h2>
<p>The client for this project was Perttu Ahola, aka celeron55, the original
creator of Minetest.</p>
<p>I started by interviewing to ascertain the requirements of such a package
repository. The result was the following high-level requirements:</p>
<ol>
<li>It must be easy to deploy.</li>
<li>It must have some kind of security model to spare most users from malicious mods.</li>
<li>It must be practical to implement an interface to it from Minetest.</li>
<li>It should be easy to add stuff, perhaps as simple as clicking a button to
import from GitHub.</li>
<li>Should have good moderation tools and high attention to security.</li>
</ol>
<p>…and the following specific features:</p>
<ul>
<li>MUST: Users have one of the following ranks, which have different permissions:
new member, member, editor, moderator, admin.</li>
<li>MUST: Users should be able to log in using username/password, GitHub, or the
forums (phpBB).</li>
<li>MUST: Packages are one of the following types: mod, game, texture pack.</li>
<li>MUST: Packages can be created by any user but must be approved before they’re
made public.</li>
<li>MUST: Packages have releases, which are immutable download-able versions.</li>
<li>MUST: Edits to packages may need to be reviewed depending on user rank and whether they’re the author.</li>
<li>MUST: A REST API to allow access to the data.</li>
<li>MUST: A database to store the data.</li>
<li>SHOULD: Packages have tags that can be used to filter them in search.</li>
<li>SHOULD: Package metadata can be imported from GitHub.</li>
<li>SHOULD: Moderation tools such as banning.</li>
</ul>
<h2 id="design" tabindex="-1">Design <a class="header-anchor" href="#design">#</a></h2>
<p>With these requirements in mind, I created a design document. I made sure to
keep celeron55 in the loop; he signed off on the final version.</p>
<h3 id="permissions-and-ranks" tabindex="-1">Permissions and Ranks <a class="header-anchor" href="#permissions-and-ranks">#</a></h3>
<p>One of the most important system in ContentDB is the permission system. There
are many named permissions, such as <code>EDIT_PACKAGE</code>, <code>DELETE_RELEASE</code>, and
<code>APPROVE_SCREENSHOT</code>. When performing an action, ContentDB will check whether
the user has the required permission on the object in question.</p>
<p>Each user has a rank. A rank determines the permissions of the user and
include the permissions of lower ranks.</p>
<ul>
<li><strong>New Members</strong>: need to have any edits or new packages approved.</li>
<li><strong>Members</strong>: can edit or upload packages by them, but edits or uploads for other authors need to be approved. Can approve edits to their packages.</li>
<li><strong>Editors</strong>: can edit any package, upload for any user, and approve changes to any package. Can approve any releases.</li>
<li><strong>Moderators</strong>: can ban users and delete mods.</li>
<li><strong>Admin</strong>: can change passwords and create users.</li>
</ul>
<p>New packages must be approved by an Editor.</p>
<h3 id="technology" tabindex="-1">Technology <a class="header-anchor" href="#technology">#</a></h3>
<p>I chose to use Python/Flask with SQLAlchemy. I had prior experience with these
frameworks, and Minetest already had a web app using Flask.</p>
<h3 id="database" tabindex="-1">Database <a class="header-anchor" href="#database">#</a></h3>
<p>I decided to use a relational database to store the data required for ContentDB
to function. As data integrity was a concern, I spent extra effort on designing
database-level validation using constraints.</p>
<figure class="undefined"><a href="/static/img/Yg8iy9JKGn-750.png"><picture><source type="image/webp" srcset="/static/img/-A65xsysQf-540.webp 540w"><img alt="The current database relationship diagram" loading="lazy" decoding="async" src="/static/img/-A65xsysQf-540.png" width="540" height="261"></picture></a><figcaption>The current database relationship diagram</figcaption></figure>
<h2 id="development-for-coursework" tabindex="-1">Development for Coursework <a class="header-anchor" href="#development-for-coursework">#</a></h2>
<p>I was given five months to develop the project for the university coursework
module.</p>
<p>I started by setting up the database. I chose to use PostgreSQL, as it has
good support for validation constraints and migrations.</p>
<p>Once the database was created, I worked on implementing a front-end template
to allow further progress. I chose to use my own CSS to get the best possible
score. I wasn’t as good at CSS as I am <a href="https://renewedtab.com/">today</a>, and
making it work was considered more important than making it pretty.</p>
<p>I used pre-existing libraries for user and login management for the coursework
version.</p>
<p>Edit Requests were a feature that existed in early versions of ContentDB.
It allowed users to create requests to edit a package, which can then be
accepted or rejected by the package’s owner or an Editor. It was removed after
the project was submitted due to changing priorities and difficulties in
maintaining the implementation.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/79HTaqUh4C-1290.jpeg"><picture><source type="image/webp" srcset="/static/img/yD06cEhobb-540.webp 540w"><img alt="The ContentDB homepage when submitted" loading="lazy" decoding="async" src="/static/img/yD06cEhobb-540.jpeg" width="540" height="597"></picture></a><figcaption>The ContentDB homepage when submitted</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/9CUviqzaiA-1123.png"><picture><source type="image/webp" srcset="/static/img/Php0MyIi4Y-540.webp 540w"><img alt="An early version of the package page" loading="lazy" decoding="async" src="/static/img/Php0MyIi4Y-540.png" width="540" height="397"></picture></a><figcaption>An early version of the package page</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/KxZxMjMiIh-1115.png"><picture><source type="image/webp" srcset="/static/img/m48ZBA7foV-540.webp 540w"><img alt="The Edit Request feature" loading="lazy" decoding="async" src="/static/img/m48ZBA7foV-540.png" width="540" height="228"></picture></a><figcaption>The Edit Request feature</figcaption></figure></div>
</div>
<h3 id="asynchronous-tasks-celery-and-git-support" tabindex="-1">Asynchronous Tasks, Celery, and Git Support <a class="header-anchor" href="#asynchronous-tasks-celery-and-git-support">#</a></h3>
<p>ContentDB uses Celery to run asynchronous tasks, to avoid blocking the server
thread. These tasks include importing meta from Git, creating releases, and
fetching user info from the forums.</p>
<p>Git support was the most tricky thing to implement, as it required acquiring a
fairly in-depth knowledge of how Git works - and it’s not simple software. For
example, one bug I had was a very weird error when attempting to clone a
particular repo. It turned out that a tag existing on a commit that wasn’t on a
branch was the cause. Through the use of integration tests, I was able to nail
the implementation.</p>
<h2 id="deployment" tabindex="-1">Deployment <a class="header-anchor" href="#deployment">#</a></h2>
<p>I deployed ContentDB using Docker onto my dedicated server.
As well as production, development is also done using Docker.</p>
<p>I implemented support for ContentDB in Minetest’s main menu, allowing
it to install and update packages.</p>
<figure class="undefined"><a href="/static/img/EumUIGqAHH-700.png"><picture><source type="image/webp" srcset="/static/img/E9PBiIazxs-540.webp 540w"><img alt="The ContentDB integration in Minetest's mainmenu, using the REST API" loading="lazy" decoding="async" src="/static/img/E9PBiIazxs-540.png" width="540" height="332"></picture></a><figcaption>The ContentDB integration in Minetest's mainmenu, using the REST API</figcaption></figure>
<p>I added Prometheus support with a
<a href="https://monitor.rubenwardy.com/d/3ELzFy3Wz/contentdb">Grafana dashboard</a>
to show statistics, such as downloads and total users.</p>
<h2 id="post-submission" tabindex="-1">Post-submission <a class="header-anchor" href="#post-submission">#</a></h2>
<p>In the four years since ContentDB was submitted as coursework, I have continued
to develop it. One of the first changes I made was switching from custom CSS to
Bootstrap CSS, which made future development easier.</p>
<figure class="undefined"><a href="/static/img/Gzlzxf7l2_-1920.png"><picture><source type="image/webp" srcset="/static/img/FbXcLoaS63-540.webp 540w"><img alt="The current package page design, using bootstrap" loading="lazy" decoding="async" src="/static/img/FbXcLoaS63-540.png" width="540" height="303"></picture></a><figcaption>The current package page design, using bootstrap</figcaption></figure>
<p>The submitted version just had a read-only API for getting information about
packages and updates. I added support for API access tokens and extended
<a href="https://content.minetest.net/help/api/">the API</a> to allow for editing packages
and making releases.</p>
<p>Other improvements include:</p>
<ul>
<li>Threads and comments</li>
<li>Ratings and reviews</li>
<li>Email notifications</li>
<li>Package videos</li>
<li>Improved tools for Editors and users</li>
<li>Package .zip validation</li>
<li>Release creation on Webhooks from GitHub and GitLab</li>
<li>Git update detection: ContentDB will check repos to see if there’s been any
new commits or tags</li>
<li>Support for dependency installation</li>
<li>Translation / internationalisation</li>
</ul>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>One of the main problems with ContentDB was ensuring maintainability as it grew.
Since submission, I’ve done several refactors to improve this, such as
separating business logic from the app’s routes. This separation allows the
front-end, REST API, and async tasks to share the same code, which reduces risk.</p>
<p>Another problem was insufficient UI tests. I used Flask’s test context feature
to check the response of queries. There are tests for core behaviour, such as
user logins and package pages, but they’re not particularly comprehensive. One
reason is that the testing tools used were insufficient; I have since invested
time in learning how to better test web front-ends, using Selenium, and have
applied that to more recent projects.</p>
<p><a href="https://content.minetest.net/">ContentDB</a> is one of the largest projects I’ve
developed, and one of the longest running.</p>
rubenwardy
Tin Mining - Ludum Dare 48 post-mortem
2022-03-19T00:00:00.000Z
2022-03-19T00:00:00.000Z
https://blog.rubenwardy.com/2022/03/19/ld48-tin-mining/
In April 2021, I participated in my first game jam. I made a Cornish tin mining simulation game, where you place out orders that workers follow.
<p>In April 2021, I participated in my first game jam, Ludum Dare 48.
Ludum Dare is a popular online game jam; this event received over 3800
submissions. The theme was “Deeper and Deeper,” and I created a game
where you manage a tin mine.</p>
<p>The year is 1790, and the Cornish tin industry is booming. You are a
businessperson who has just secured investment to build a mine. The area is
known to be rich in tin, which is in high demand.</p>
<p>Rather than controlling your workers directly, you drag out plans for tiles to
be mined and built. The workers will mine tunnels and build where ordered.
They will carry mined resources to the surface to be sold.</p>
<figure class="undefined"><a href="/static/img/6czJ2rVSyu-960.gif"><img alt="Gif of the Tin Mining game." loading="lazy" decoding="async" src="/static/img/6EVfY4m0n6-960.gif" width="960" height="65340"></a><figcaption>Gif of the Tin Mining game.</figcaption></figure>
<h2 id="development" tabindex="-1">Development <a class="header-anchor" href="#development">#</a></h2>
<p>I started this a day late and had to work on Monday, so I was only able
to implement the bare minimum viable gameplay. I had planned to add hazards,
buildings to prevent these hazards, and a tech tree. For example, gas explosions,
toxic gas, and flooding were huge problems in mining.</p>
<p>The map is procedurally-generated, using a combination of multiple simplex
noises. The ores are randomly placed in veins, with the likelihood based on
depth.</p>
<p>I made all the art myself, using GIMP and the
<a href="https://lospec.com/palette-list/endesga-32">Endesga 32</a> color palette.
I choose to make the art super low resolution - 8x8 pixels - to make the art
easier to make, and hopefully hide my lack of artistic skills.</p>
<figure class="undefined"><a href="/static/img/tsuH3VY2dO-1920.png"><picture><source type="image/webp" srcset="/static/img/2m1NnZ_3tW-540.webp 540w"><img alt="Mining deeper" loading="lazy" decoding="async" src="/static/img/2m1NnZ_3tW-540.png" width="540" height="303"></picture></a><figcaption>Mining deeper</figcaption></figure>
<p>I created the game using TypeScript and Phaser 3. Phaser 3 was an absolute pain
to use. It was poorly documented, and I spent most of the time fighting against
the API. One of the main things I’ve learned from doing this game is that I
never want to use Phaser 3 again.</p>
<h3 id="game-ai" tabindex="-1">Game AI <a class="header-anchor" href="#game-ai">#</a></h3>
<p>I choose to use
<a href="https://www.gamedeveloper.com/programming/behavior-trees-for-ai-how-they-work">behaviour trees</a>
to implement the worker AI. Behaviour trees make it easier to implement extendable
NPC behaviour, which will be useful when working on more complex behaviour in the
post-jam version.</p>
<p>When a worker isn’t currently working on a task, it periodically asks the
WorkManager for an available task. The WorkManager allocates tasks based on
distance and a heuristic weighting of the task. For example, moving dropped
items to the stockpile is weighted higher than mining or building, to avoid
items building up and clogging the walkways.</p>
<p>Workers shouldn’t get stuck on work that isn’t possible, such as trying to mine
tiles that are not yet exposed. The way I implemented this was a bit hacky:
workers have “work failure lockouts.” If a worker attempts work and fails, then
that worker won’t attempt that piece of work again for 10 seconds. The work
manager will allocate tasks without checking their accessibility, and the worker
is responsible for triggering a lockout if the work isn’t reachable.</p>
<p>There were many bugs in the AI system, some of which were only found after
submission. Luckily, Ludum Dare allows releasing bug fixes after the deadline,
so I was able to patch most of the issues. The remaining issues just couldn’t be
fixed due to architectural issues, or performance issues; the pathfinder was
pretty naive and slow.</p>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>The main reason I entered a game jam is that I wanted to finish a game,
at least to the minimum viable gameplay. It forces you to focus on making the
game, rather than making the code perfect, which is something I’ve not been
the best at in the past. This project certainly achieved these goals. Whilst
I didn’t implement all the features I wanted, I still finished it sufficiently
for a small game.</p>
<p>I decided to learn a new framework for making games for the next game jam. I
chose to learn the <a href="https://godotengine.org/">Godot Engine</a>, and have since
wrote several games in it, including a post-jam version of Tin Miner and an
entry for Ludum Dare 49 - which I’ll write about at some point.</p>
<p>I look forward to working on the post-jam game in the future. I think this idea
may be a good game to work towards publishing on Steam, due to the smaller scope
compared to some of the other games I’ve made.</p>
<h2 id="post-jam-version" tabindex="-1">Post-jam version <a class="header-anchor" href="#post-jam-version">#</a></h2>
<p>As mentioned, Phaser 3 was an absolute pain to use. Therefore, I decided to
rewrite the game using the <a href="https://godotengine.org/">Godot Engine</a>. This was a
pleasant experience, as Godot’s API is very nice to use.</p>
<figure class="undefined"><a href="/static/img/65GuyNBJ2W-1920.png"><picture><source type="image/webp" srcset="/static/img/TBtb4vevF--540.webp 540w"><img alt="A post-jam version of the game, written using Godot." loading="lazy" decoding="async" src="/static/img/TBtb4vevF--540.png" width="540" height="337"></picture></a><figcaption>A post-jam version of the game, written using Godot.</figcaption></figure>
<p>It took a bit longer to make, as I focused more on doing things correctly.
Whilst this was mostly differences in maintainability, I also snuck in some
usability improvements:</p>
<figure class="undefined"><a href="/static/img/Fcf0-qBlOq-700.png"><picture><source type="image/webp" srcset="/static/img/tMJJhoBcvG-540.webp 540w"><img alt="Post-jam user improvements" loading="lazy" decoding="async" src="/static/img/tMJJhoBcvG-540.png" width="540" height="344"></picture></a><figcaption>Post-jam user improvements</figcaption></figure>
<p>The pathfinder in the LD version reads the tiles directly, which is slow as
there needs to be a lot of navigation logic. The new Godot version stores a
navigation graph to represent possible routes, which increases performance by
a large factor.</p>
<p>One nice thing about the new graph-based pathfinder is that it’s quick to check
whether a tile is on the network. This acts as a nice optimization for checking
whether a tile can be accessed to perform work.</p>
<p>Another difference is the new version contains better debug tools.
For example, a debug menu to edit the world, navigation graph debugging,
and graphs.</p>
<figure class="undefined"><a href="/static/img/CZV7NZVgIT-1335.png"><picture><source type="image/webp" srcset="/static/img/KEW6gRDkd7-540.webp 540w"><img alt="Post-jam navigation mesh and debug graphs. Hovering over a tile with a job highlights the allocated NPC with a circle." loading="lazy" decoding="async" src="/static/img/KEW6gRDkd7-540.png" width="540" height="332"></picture></a><figcaption>Post-jam navigation mesh and debug graphs. Hovering over a tile with a job highlights the allocated NPC with a circle.</figcaption></figure>
<h2 id="play-the-game" tabindex="-1">Play the game <a class="header-anchor" href="#play-the-game">#</a></h2>
<p>You can play the Ludum Dare version
<a href="https://rubenwardy.com/ld48_cornish_mines/">online</a>.
The post-jam version is currently in development and will be made available
when ready.</p>
<h3 id="controls" tabindex="-1">Controls <a class="header-anchor" href="#controls">#</a></h3>
<ul>
<li>Pan: Right-click drag / middle-click drag / arrow keys to pan</li>
<li>Zoom: scroll wheel</li>
<li>Select tools from build menu, and then:
<ul>
<li>Left-click and drag to place orders.</li>
<li>Esc to cancel the tool.</li>
</ul>
</li>
<li>Tip: you can order workers to mine and build a ladder on the same tile, and they will mine and then build a ladder.</li>
</ul>
rubenwardy
IoT Plant Monitor using ESP32
2022-03-17T00:00:00.000Z
2022-03-17T00:00:00.000Z
https://blog.rubenwardy.com/2022/03/17/plant-monitor/
I made a plant monitor, which measures soil moisture, temperature, and humidity, and reports these things to a cloud IoT service called Thinger.io.
<p>I have a lot of houseplants, but I often forget to water them.
I’ve been getting into electronics and thought this would be a great
opportunity to make something.</p>
<p>I made a plant monitor, which measures soil moisture, temperature, and humidity,
and reports these things to a cloud IoT service called Thinger.io.</p>
<h2 id="the-case" tabindex="-1">The Case <a class="header-anchor" href="#the-case">#</a></h2>
<p>I wanted the monitor to be subtle and hidden, so I designed a case to fit
between the inner planter and the outer decorative pot. I printed it using my
brand new Prusa Mini+. It took many iterations to get it perfect. The
electronics fit snug in the case, with ports for sensors and a microUSB charger.</p>
<p>The case was the first thing I made using Computer-Aided Design (CAD),
and was a particularly painful task as the case is curved to fit the 23cm
diameter pot.</p>
<p>I’ve put the files up on
<a href="https://www.thingiverse.com/thing:5234381">Thingiverse</a> and
<a href="https://www.prusaprinters.org/prints/130542-clip-on-plant-sensor-case">PrusaPrinters</a>.</p>
<figure class="undefined"><a href="/static/img/xFr84pXF5T-537.png"><picture><source type="image/webp" srcset="/static/img/NdYJB3nloy-537.webp 537w"><img alt="The curved case in CAD" loading="lazy" decoding="async" src="/static/img/NdYJB3nloy-537.png" width="537" height="699"></picture></a><figcaption>The curved case in CAD</figcaption></figure>
<h2 id="electronics" tabindex="-1">Electronics <a class="header-anchor" href="#electronics">#</a></h2>
<h3 id="components" tabindex="-1">Components <a class="header-anchor" href="#components">#</a></h3>
<p>The first problem I had was finding working soil moisture sensors. I bought
some capacitive soil moisture sensors from eBay, but they were very slow
reacting - they took over 3 minutes to respond to changes in moisture.
After some research, I found out that some low-quality sensors from China
contain a capacitor in the wrong place, which smooths out the output
voltage to an unusable degree. I replaced the sensors with ones bought from
the Pi Hut, which worked considerably better - they respond immediately to
changes in moisture. This event put me off buying from eBay for a while.</p>
<figure class="undefined"><a href="/static/img/jZ-3-HLHgX-700.png"><picture><source type="image/webp" srcset="/static/img/teFu5CzC7t-540.webp 540w"><img alt="Graph showing the slow reaction of the faulty soil moisture sensor" loading="lazy" decoding="async" src="/static/img/teFu5CzC7t-540.png" width="540" height="293"></picture></a><figcaption>Graph showing the slow reaction of the faulty soil moisture sensor</figcaption></figure>
<p>I decided to use an ESP32 as the microcontroller, as they seemed easy-to-use,
abundant, and cheap. I choose a cheap ESP32-WROOM-32 devkit from a trusted
eBay seller.</p>
<h3 id="making-a-permanent-circuit-using-stripboard" tabindex="-1">Making a permanent circuit using stripboard <a class="header-anchor" href="#making-a-permanent-circuit-using-stripboard">#</a></h3>
<p>This is my first proper electronics project, and the first time I’ve made
a ‘permanent’ circuit. The soldering was a bit messy but functional
(I blame the flux for spitting at me, but really it’s just a lack of practice).</p>
<p>In order to use stripboard, you need to be able to break the copper tracks to
allow the circuit to be correctly formed. I bought a Stripboard Track Cutter
from the Pi Hut for this job, and it turned out to be incredibly ill-suited.
It took 40 seconds of hard pressure to break a track. I’ve since bought a new
track cutter from eBay that only takes a few seconds of basically no pressure,
so I’ll definitely be using that for future projects.</p>
<figure class="undefined"><a href="/static/img/assG3FbzhC-1000.jpeg"><picture><source type="image/webp" srcset="/static/img/CtYOGKf-EF-540.webp 540w"><img alt="The electronics, on a stripboard" loading="lazy" decoding="async" src="/static/img/CtYOGKf-EF-540.jpeg" width="540" height="268"></picture></a><figcaption>The electronics, on a stripboard</figcaption></figure>
<p>The soil moisture sensor is a simple analog 3-pin device - two for power, one for a
voltage reading. The output voltage decreases with moisture.
This sensor connects to the board using a 3-pin male header.</p>
<p>The temperature/humidity sensor uses Inter-Integrated Circuit (I2C), a low-level
serial communication bus, to communicate with the microcontroller. I2C requires
4-pins - two for power, and two for the bus.</p>
<p>I included connectors for two I2C devices, with space on the board for up to four
devices. I only use one I2C device currently, but I’m likely to add more devices
such as a light intensity (lux) sensor in the future.</p>
<p>I bought some 4-pin JST PH connectors for I2C, but ended up using
female headers instead as JST PH’s 2mm spacing doesn’t fit the stripboard’s
3mm spacing. Perhaps in the future I’ll either make a PCB or create an adapter,
allowing JST PH to be used.</p>
<figure class="undefined"><a href="/static/img/tfbB-H3Kob-1308.png"><picture><source type="image/webp" srcset="/static/img/crYP__TrLA-540.webp 540w"><img alt="Electronics schematic, created using KiCAD" loading="lazy" decoding="async" src="/static/img/crYP__TrLA-540.png" width="540" height="326"></picture></a><figcaption>Electronics schematic, created using KiCAD</figcaption></figure>
<h2 id="internet-of-things-with-thinger-io" tabindex="-1">Internet of Things with Thinger.IO <a class="header-anchor" href="#internet-of-things-with-thinger-io">#</a></h2>
<p>I was tempted to implement my own cloud IoT platform, but then I reminded myself
that the point of this project was to learn electronics and 3d printing, not
to spend hours reinventing the wheel.</p>
<p>I did some research on existing IoT platforms, and choose Thinger.IO as it is
open source and seemed to have the right balance of simplicity vs capability.</p>
<p>Integrating with Thinger.IO with the microcontroller was as simple as including
their SDK and writing up a few lines to publish the sensor values.</p>
<p>Setting up the dashboard on Thinger.IO was a bit more complicated. It took a
while to work out how to record data rather than just show it, I needed to use
a data bucket with manually posted values, and then write to it every 5 minutes
from the microcontroller. I would have preferred that the microcontroller
just expose the sensor values, and not care about what the IoT platform does
with it.</p>
<figure class="undefined"><a href="/static/img/w3K6NY3gYR-700.jpeg"><picture><source type="image/webp" srcset="/static/img/Gy_qZ4yCjg-540.webp 540w"><img alt="The dashboard on Thinger.io, showing the collected data" loading="lazy" decoding="async" src="/static/img/Gy_qZ4yCjg-540.jpeg" width="540" height="465"></picture></a><figcaption>The dashboard on Thinger.io, showing the collected data</figcaption></figure>
<h2 id="source-code" tabindex="-1">Source code <a class="header-anchor" href="#source-code">#</a></h2>
<p>The source code is pretty basic, this was definitely more of an electronics
project than a programming one!</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string"><Arduino.h></span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string"><Adafruit_SHT31.h></span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">define</span> <span class="token macro-name">THINGER_SERIAL_DEBUG</span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string"><ThingerESP32.h></span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string">"secrets.hpp"</span></span>
ThingerESP32 <span class="token function">thing</span><span class="token punctuation">(</span>USERNAME<span class="token punctuation">,</span> DEVICE_ID<span class="token punctuation">,</span> DEVICE_CREDENTIAL<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">define</span> <span class="token macro-name">MOIS_LOW</span> <span class="token expression"><span class="token number">1.8f</span></span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">define</span> <span class="token macro-name">MOIS_HIGH</span> <span class="token expression"><span class="token number">2.2f</span></span></span>
<span class="token keyword">int</span> moistureReadPin <span class="token operator">=</span> <span class="token number">35</span><span class="token punctuation">;</span>
<span class="token keyword">int</span> moistureEnPin <span class="token operator">=</span> <span class="token number">32</span><span class="token punctuation">;</span>
Adafruit_SHT31 sht31 <span class="token operator">=</span> <span class="token function">Adafruit_SHT31</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">float</span> <span class="token function">measureSoil</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
Serial<span class="token punctuation">.</span><span class="token function">println</span><span class="token punctuation">(</span><span class="token string">"Measuring soil"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token function">digitalWrite</span><span class="token punctuation">(</span>moistureEnPin<span class="token punctuation">,</span> HIGH<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token function">delay</span><span class="token punctuation">(</span><span class="token number">1000</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">int</span> raw <span class="token operator">=</span> <span class="token function">analogRead</span><span class="token punctuation">(</span>moistureReadPin<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">float</span> moisture <span class="token operator">=</span> raw <span class="token operator">*</span> <span class="token number">3.3f</span> <span class="token operator">/</span> <span class="token number">4095.0f</span><span class="token punctuation">;</span>
<span class="token function">digitalWrite</span><span class="token punctuation">(</span>moistureEnPin<span class="token punctuation">,</span> LOW<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">return</span> <span class="token number">100.f</span> <span class="token operator">-</span> <span class="token number">100.f</span> <span class="token operator">*</span> <span class="token punctuation">(</span>moisture <span class="token operator">-</span> MOIS_LOW<span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token punctuation">(</span>MOIS_HIGH <span class="token operator">-</span> MOIS_LOW<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">void</span> <span class="token function">setup</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
Serial<span class="token punctuation">.</span><span class="token function">begin</span><span class="token punctuation">(</span><span class="token number">9600</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token function">pinMode</span><span class="token punctuation">(</span>moistureReadPin<span class="token punctuation">,</span> INPUT<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token function">pinMode</span><span class="token punctuation">(</span>moistureEnPin<span class="token punctuation">,</span> OUTPUT<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span>sht31<span class="token punctuation">.</span><span class="token function">begin</span><span class="token punctuation">(</span><span class="token number">0x44</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
Serial<span class="token punctuation">.</span><span class="token function">println</span><span class="token punctuation">(</span><span class="token string">"Couldn't find SHT31"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">while</span> <span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token function">delay</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
sht31<span class="token punctuation">.</span><span class="token function">heater</span><span class="token punctuation">(</span><span class="token boolean">false</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
thing<span class="token punctuation">.</span><span class="token function">add_wifi</span><span class="token punctuation">(</span>SSID<span class="token punctuation">,</span> SSID_PASSWORD<span class="token punctuation">)</span><span class="token punctuation">;</span>
thing<span class="token punctuation">[</span><span class="token string">"data"</span><span class="token punctuation">]</span> <span class="token operator">>></span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">(</span>pson <span class="token operator">&</span>out<span class="token punctuation">)</span> <span class="token punctuation">{</span>
out<span class="token punctuation">[</span><span class="token string">"temperature"</span><span class="token punctuation">]</span> <span class="token operator">=</span> sht31<span class="token punctuation">.</span><span class="token function">readTemperature</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
out<span class="token punctuation">[</span><span class="token string">"humidity"</span><span class="token punctuation">]</span> <span class="token operator">=</span> sht31<span class="token punctuation">.</span><span class="token function">readHumidity</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
out<span class="token punctuation">[</span><span class="token string">"soil_moisture"</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token function">measureSoil</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">;</span>
thing<span class="token punctuation">.</span><span class="token function">handle</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
thing<span class="token punctuation">.</span><span class="token function">write_bucket</span><span class="token punctuation">(</span><span class="token string">"Plants"</span><span class="token punctuation">,</span> <span class="token string">"data"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
ESP<span class="token punctuation">.</span><span class="token function">deepSleep</span><span class="token punctuation">(</span><span class="token number">60</span> <span class="token operator">*</span> <span class="token number">1000000</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">void</span> <span class="token function">loop</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token comment">// Never executed due to deep sleep</span>
<span class="token punctuation">}</span></code></pre>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>This was a very nice project to learn electronics and 3d printed, I certainly
learned a lot. I still forget to water my plants, but at least I do so in a
high-tech fashion!</p>
<p>I look forward to working on more complicated projects in the future, from
electronics to mechanical design.</p>
<p>In the future, I’d like to add a light intensity (lux) sensor to the monitor. I
need to think about where to put the sensor, as putting it on top of the pot
will result in an inaccurate reading. This is due to sunlight being obstructed
by the pot, and the difference in height to the leaves.</p>
<p>This project has seeded many ideas for future projects, such as self-watering
plants, a weather station, a monitored indoor greenhouse, and hydroponics. I
plan to seed basil and a few other herbs soon, having a mini indoors controlled
greenhouse with grow lights could be a cool project.</p>
rubenwardy
Securing Markdown user content with Mozilla Bleach
2021-05-08T00:00:00.000Z
2021-05-08T00:00:00.000Z
https://blog.rubenwardy.com/2021/05/08/mozilla-bleach-markdown/
When accepting user content, it's important to sanitize user input to avoid Cross Site Scripting (XSS) and other attacks.
<p>Markdown is a common choice for rich text formatting due to its readability and
ease-of-use. Unlike a lot of markup, it aims to match natural text. It’s even
easy for beginner users, and there are WYSIWYG editors available.</p>
<p>We will be using the <a href="https://python-markdown.github.io/">Python Markdown</a>
library to convert Markdown to HTML. Markdown doesn’t have a well-defined
standard. The library aims to comply with what little is defined by the
<a href="https://daringfireball.net/projects/markdown/syntax">Markdown syntax specification</a>,
meaning that it is also often stricter than other parsers.</p>
<p>To convert Markdown to HTML:</p>
<pre class="language-py"><code class="language-py"><span class="token keyword">from</span> markdown <span class="token keyword">import</span> Markdown
md <span class="token operator">=</span> Markdown<span class="token punctuation">(</span>extensions<span class="token operator">=</span><span class="token punctuation">[</span><span class="token string">"fenced_code"</span><span class="token punctuation">,</span> <span class="token string">"tables"</span><span class="token punctuation">]</span><span class="token punctuation">,</span> output_format<span class="token operator">=</span><span class="token string">"html5"</span><span class="token punctuation">)</span>
html <span class="token operator">=</span> md<span class="token punctuation">.</span>convert<span class="token punctuation">(</span>source<span class="token punctuation">)</span></code></pre>
<p>You can use another library to interpret Markdown, if you wish.
The rest of the code will be dealing with the HTML output, so is independent of
Markdown parsers.</p>
<h3 id="avoid-xss-attacks" tabindex="-1">Avoid XSS attacks <a class="header-anchor" href="#avoid-xss-attacks">#</a></h3>
<p>When allowing user submitted content, it’s important to sanitise it to avoid
Cross-Site Scripting attacks (XSS). If you don’t sanitise user input, then
an attacker will be able to add HTML tags to run JavaScript when other users
view your website. This can be used to steal login credentials, run bitcoin
mining malware, or deface your website. So not ideal.</p>
<p><a href="https://github.com/mozilla/bleach">Bleach</a>, by Mozilla, is library to sanitised
untrusted HTML. It works based on a whitelist of tags and their attributes.
I have based my list on the
<a href="https://github.com/Wenzil/mdx_bleach/blob/master/mdx_bleach/whitelist.py">mdx_bleach extension</a>,
which you could use directly with the markdown library - but I prefer to use
the Bleach library directly after generating the HTML as dependencies have a
tendency to break.</p>
<p>Another thing that Bleach does is safely linkify text - it can convert text
resembling a URL into a link.</p>
<pre class="language-py"><code class="language-py"><span class="token comment"># List of allowed HTML tags</span>
ALLOWED_TAGS <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token string">"h1"</span><span class="token punctuation">,</span> <span class="token string">"h2"</span><span class="token punctuation">,</span> <span class="token string">"h3"</span><span class="token punctuation">,</span> <span class="token string">"h4"</span><span class="token punctuation">,</span> <span class="token string">"h5"</span><span class="token punctuation">,</span> <span class="token string">"h6"</span><span class="token punctuation">,</span> <span class="token string">"hr"</span><span class="token punctuation">,</span>
<span class="token string">"ul"</span><span class="token punctuation">,</span> <span class="token string">"ol"</span><span class="token punctuation">,</span> <span class="token string">"li"</span><span class="token punctuation">,</span> <span class="token string">"p"</span><span class="token punctuation">,</span> <span class="token string">"br"</span><span class="token punctuation">,</span>
<span class="token string">"pre"</span><span class="token punctuation">,</span> <span class="token string">"code"</span><span class="token punctuation">,</span> <span class="token string">"blockquote"</span><span class="token punctuation">,</span>
<span class="token string">"strong"</span><span class="token punctuation">,</span> <span class="token string">"em"</span><span class="token punctuation">,</span> <span class="token string">"a"</span><span class="token punctuation">,</span> <span class="token string">"img"</span><span class="token punctuation">,</span> <span class="token string">"b"</span><span class="token punctuation">,</span> <span class="token string">"i"</span><span class="token punctuation">,</span>
<span class="token string">"table"</span><span class="token punctuation">,</span> <span class="token string">"thead"</span><span class="token punctuation">,</span> <span class="token string">"tbody"</span><span class="token punctuation">,</span> <span class="token string">"tr"</span><span class="token punctuation">,</span> <span class="token string">"th"</span><span class="token punctuation">,</span> <span class="token string">"td"</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span>
<span class="token comment"># A map of HTML tags to allowed attributes</span>
<span class="token comment"># If a tag isn't here, then no attributes are allowed</span>
ALLOWED_ATTRIBUTES <span class="token operator">=</span> <span class="token punctuation">{</span>
<span class="token string">"h1"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">"id"</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token string">"h2"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">"id"</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token string">"h3"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">"id"</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token string">"h4"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">"id"</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token string">"a"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">"href"</span><span class="token punctuation">,</span> <span class="token string">"title"</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token string">"img"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">"src"</span><span class="token punctuation">,</span> <span class="token string">"title"</span><span class="token punctuation">,</span> <span class="token string">"alt"</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span>
<span class="token comment"># Allowed protocols in links.</span>
ALLOWED_PROTOCOLS <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">"http"</span><span class="token punctuation">,</span> <span class="token string">"https"</span><span class="token punctuation">,</span> <span class="token string">"mailto"</span><span class="token punctuation">]</span>
md <span class="token operator">=</span> Markdown<span class="token punctuation">(</span>output_format<span class="token operator">=</span><span class="token string">"html5"</span><span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">render_markdown</span><span class="token punctuation">(</span>source<span class="token punctuation">)</span><span class="token punctuation">:</span>
html <span class="token operator">=</span> md<span class="token punctuation">.</span>convert<span class="token punctuation">(</span>source<span class="token punctuation">)</span>
cleaner <span class="token operator">=</span> Cleaner<span class="token punctuation">(</span>
tags<span class="token operator">=</span>ALLOWED_TAGS<span class="token punctuation">,</span>
attributes<span class="token operator">=</span>ALLOWED_ATTRIBUTES<span class="token punctuation">,</span>
protocols<span class="token operator">=</span>ALLOWED_PROTOCOLS<span class="token punctuation">,</span>
filters<span class="token operator">=</span><span class="token punctuation">[</span>partial<span class="token punctuation">(</span>LinkifyFilter<span class="token punctuation">,</span> callbacks<span class="token operator">=</span>bleach<span class="token punctuation">.</span>linkifier<span class="token punctuation">.</span>DEFAULT_CALLBACKS<span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> cleaner<span class="token punctuation">.</span>clean<span class="token punctuation">(</span>html<span class="token punctuation">)</span></code></pre>
<h3 id="supporting-code-highlighting" tabindex="-1">Supporting code highlighting <a class="header-anchor" href="#supporting-code-highlighting">#</a></h3>
<p>The CodeHilite extension for Python-Markdown uses
<a href="https://pygments.org/">Pygments</a> to provide syntax highlighting.
You can enable the extension by adding it to the extensions list.</p>
<pre class="language-py"><code class="language-py">md <span class="token operator">=</span> Markdown<span class="token punctuation">(</span>extensions<span class="token operator">=</span><span class="token punctuation">[</span><span class="token string">"fenced_code"</span><span class="token punctuation">,</span> <span class="token string">"tables"</span><span class="token punctuation">,</span> <span class="token string">"codehilite"</span><span class="token punctuation">]</span><span class="token punctuation">,</span> output_format<span class="token operator">=</span><span class="token string">"html5"</span><span class="token punctuation">)</span></code></pre>
<p>You will also need to provide the <code>.css</code> files for the
<a href="https://pygments.org/docs/styles/#builtin-styles">style</a>
(<a href="https://pygments.org/demo/">demo</a>) you choose.
I ended up going with <a href="https://draculatheme.com/pygments/">Darcula</a> due to
personal preference.</p>
<p>Bleach will strip attributes that aren’t whitelisted, including the class names
needed by code highlighting. It’s important that you don’t just allow any
<code>class</code> attribute values to be used, as this would allow malicious users to use
any CSS class and deface your website. Instead, we will provide a function to
the <code>ALLOWED_ATTRIBUTES</code> dictionary, which will check whether the provided
values are allowed.</p>
<pre class="language-py"><code class="language-py">ALLOWED_TAGS <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token comment"># ...</span>
<span class="token string">"div"</span><span class="token punctuation">,</span> <span class="token string">"span"</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span>
ALLOWED_CSS_CLASSES <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token string">"highlight"</span><span class="token punctuation">,</span> <span class="token string">"codehilite"</span><span class="token punctuation">,</span>
<span class="token string">"hll"</span><span class="token punctuation">,</span> <span class="token string">"c"</span><span class="token punctuation">,</span> <span class="token string">"err"</span><span class="token punctuation">,</span> <span class="token string">"g"</span><span class="token punctuation">,</span> <span class="token string">"k"</span><span class="token punctuation">,</span> <span class="token string">"l"</span><span class="token punctuation">,</span> <span class="token string">"n"</span><span class="token punctuation">,</span> <span class="token string">"o"</span><span class="token punctuation">,</span> <span class="token string">"x"</span><span class="token punctuation">,</span> <span class="token string">"p"</span><span class="token punctuation">,</span> <span class="token string">"ch"</span><span class="token punctuation">,</span> <span class="token string">"cm"</span><span class="token punctuation">,</span> <span class="token string">"cp"</span><span class="token punctuation">,</span> <span class="token string">"cpf"</span><span class="token punctuation">,</span> <span class="token string">"c1"</span><span class="token punctuation">,</span> <span class="token string">"cs"</span><span class="token punctuation">,</span>
<span class="token string">"gd"</span><span class="token punctuation">,</span> <span class="token string">"ge"</span><span class="token punctuation">,</span> <span class="token string">"gr"</span><span class="token punctuation">,</span> <span class="token string">"gh"</span><span class="token punctuation">,</span> <span class="token string">"gi"</span><span class="token punctuation">,</span> <span class="token string">"go"</span><span class="token punctuation">,</span> <span class="token string">"gp"</span><span class="token punctuation">,</span> <span class="token string">"gs"</span><span class="token punctuation">,</span> <span class="token string">"gu"</span><span class="token punctuation">,</span> <span class="token string">"gt"</span><span class="token punctuation">,</span> <span class="token string">"kc"</span><span class="token punctuation">,</span> <span class="token string">"kd"</span><span class="token punctuation">,</span> <span class="token string">"kn"</span><span class="token punctuation">,</span> <span class="token string">"kp"</span><span class="token punctuation">,</span> <span class="token string">"kr"</span><span class="token punctuation">,</span>
<span class="token string">"kt"</span><span class="token punctuation">,</span> <span class="token string">"ld"</span><span class="token punctuation">,</span> <span class="token string">"m"</span><span class="token punctuation">,</span> <span class="token string">"s"</span><span class="token punctuation">,</span> <span class="token string">"na"</span><span class="token punctuation">,</span> <span class="token string">"nb"</span><span class="token punctuation">,</span> <span class="token string">"nc"</span><span class="token punctuation">,</span> <span class="token string">"no"</span><span class="token punctuation">,</span> <span class="token string">"nd"</span><span class="token punctuation">,</span> <span class="token string">"ni"</span><span class="token punctuation">,</span> <span class="token string">"ne"</span><span class="token punctuation">,</span> <span class="token string">"nf"</span><span class="token punctuation">,</span> <span class="token string">"nl"</span><span class="token punctuation">,</span> <span class="token string">"nn"</span><span class="token punctuation">,</span> <span class="token string">"nx"</span><span class="token punctuation">,</span>
<span class="token string">"py"</span><span class="token punctuation">,</span> <span class="token string">"nt"</span><span class="token punctuation">,</span> <span class="token string">"nv"</span><span class="token punctuation">,</span> <span class="token string">"ow"</span><span class="token punctuation">,</span> <span class="token string">"w"</span><span class="token punctuation">,</span> <span class="token string">"mb"</span><span class="token punctuation">,</span> <span class="token string">"mf"</span><span class="token punctuation">,</span> <span class="token string">"mh"</span><span class="token punctuation">,</span> <span class="token string">"mi"</span><span class="token punctuation">,</span> <span class="token string">"mo"</span><span class="token punctuation">,</span> <span class="token string">"sa"</span><span class="token punctuation">,</span> <span class="token string">"sb"</span><span class="token punctuation">,</span> <span class="token string">"sc"</span><span class="token punctuation">,</span> <span class="token string">"dl"</span><span class="token punctuation">,</span> <span class="token string">"sd"</span><span class="token punctuation">,</span>
<span class="token string">"s2"</span><span class="token punctuation">,</span> <span class="token string">"se"</span><span class="token punctuation">,</span> <span class="token string">"sh"</span><span class="token punctuation">,</span> <span class="token string">"si"</span><span class="token punctuation">,</span> <span class="token string">"sx"</span><span class="token punctuation">,</span> <span class="token string">"sr"</span><span class="token punctuation">,</span> <span class="token string">"s1"</span><span class="token punctuation">,</span> <span class="token string">"ss"</span><span class="token punctuation">,</span> <span class="token string">"bp"</span><span class="token punctuation">,</span> <span class="token string">"fm"</span><span class="token punctuation">,</span> <span class="token string">"vc"</span><span class="token punctuation">,</span> <span class="token string">"vg"</span><span class="token punctuation">,</span> <span class="token string">"vi"</span><span class="token punctuation">,</span> <span class="token string">"vm"</span><span class="token punctuation">,</span> <span class="token string">"il"</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span>
<span class="token keyword">def</span> <span class="token function">allow_class</span><span class="token punctuation">(</span>_tag<span class="token punctuation">,</span> name<span class="token punctuation">,</span> value<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> name <span class="token operator">==</span> <span class="token string">"class"</span> <span class="token keyword">and</span> value <span class="token keyword">in</span> ALLOWED_CSS_CLASSES
ALLOWED_ATTRIBUTES <span class="token operator">=</span> <span class="token punctuation">{</span>
<span class="token comment"># etc</span>
<span class="token string">"code"</span><span class="token punctuation">:</span> allow_class<span class="token punctuation">,</span>
<span class="token string">"div"</span><span class="token punctuation">:</span> allow_class<span class="token punctuation">,</span>
<span class="token string">"span"</span><span class="token punctuation">:</span> allow_class<span class="token punctuation">,</span>
<span class="token punctuation">}</span>
md <span class="token operator">=</span> Markdown<span class="token punctuation">(</span>output_format<span class="token operator">=</span><span class="token string">"html5"</span><span class="token punctuation">)</span></code></pre>
<p>And there you are! You can now render untrusted user markdown safely, with
code highlighting and linkify.</p>
rubenwardy
ForumMate: My return to Android app development
2020-09-13T00:00:00.000Z
2020-09-13T00:00:00.000Z
https://blog.rubenwardy.com/2020/09/13/return-to-android-dev/
I created a forum aggregator app to relearn Android devevelopment, including databinding, View Models, LiveData, and more.
<p>I worked as an Android developer just over two years ago, creating native apps
for clients using Java and Kotlin. During that time, Kotlin was gaining
prominence and had just been made official by Google. Google also introduced
Architecture Components that year, later renamed to JetPack. Since then, the
Android ecosystem has changed significantly, with Kotlin and JetPack gaining
significant maturity and development. Out with Realm, Activities, and
Model-View-Presenter (MVP), in with Room, fragment-based architecture, and MVVM.
<a href="/2020/05/06/android-complete-recyclerview-data-binding/">Data-binding</a> and
MVVM are pretty awesome and breathe a whole new life into Android app development.</p>
<h2 id="what-is-forummate" tabindex="-1">What is ForumMate? <a class="header-anchor" href="#what-is-forummate">#</a></h2>
<p>In order to get back up to speed with Android development, I needed a project to
work on. I decided to make a forum/bulletin board aggregator app, which uses
HTML parsing to get data from legacy forum websites and display it using a nice
user interface.</p>
<p><a href="https://gitlab.com/rubenwardy/forummate">ForumMate</a> allows the user to log into
a phpBB forum using a WebView, where the app captures the session cookies. The
user can either select a forum from a list fetched using a REST API, or log into
any forum by navigating to it on Google search.</p>
<p>These session cookies are stored and used in future HTTP requests, where they are
parsed using JSoup. Results are stored in a Room database, with the interface
subscribing to LiveData.</p>
<h3 id="screenshots" tabindex="-1">Screenshots <a class="header-anchor" href="#screenshots">#</a></h3>
<p>The user interface is a still a work in progress, but it works.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/MbopOpLpvX-255.png"><picture><source type="image/webp" srcset="/static/img/SFIlXoCVRj-255.webp 255w"><img alt="List of available forum topics" loading="lazy" decoding="async" src="/static/img/SFIlXoCVRj-255.png" width="255" height="453"></picture></a><figcaption>List of available forum topics</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/KUKc2q3LXK-255.png"><picture><source type="image/webp" srcset="/static/img/Mm8-doWTE0-255.webp 255w"><img alt="Unread Topics" loading="lazy" decoding="async" src="/static/img/Mm8-doWTE0-255.png" width="255" height="453"></picture></a><figcaption>Unread Topics</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/PNsJIFd7li-255.png"><picture><source type="image/webp" srcset="/static/img/odDk8SyiX4-255.webp 255w"><img alt="Posting a comment to a thread" loading="lazy" decoding="async" src="/static/img/odDk8SyiX4-255.png" width="255" height="453"></picture></a><figcaption>Posting a comment to a thread</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/hP4fjEoiLY-255.png"><picture><source type="image/webp" srcset="/static/img/EIcedpq1Wf-255.webp 255w"><img alt="Accounts" loading="lazy" decoding="async" src="/static/img/EIcedpq1Wf-255.png" width="255" height="453"></picture></a><figcaption>Accounts</figcaption></figure></div>
</div>
<h2 id="architecture-and-technical-information" tabindex="-1">Architecture and Technical information <a class="header-anchor" href="#architecture-and-technical-information">#</a></h2>
<p>The app is architectured using MVVM, with
<a href="/2020/05/06/android-complete-recyclerview-data-binding/">two-way data-binding</a>,
dependency injection, and Navigation components.</p>
<p>The code has 5 main zones: Views (ie: fragments), ViewModels, Repositories,
Database (ie: Room), and the Backend. The backend consists of interfaces with an
implementation per forum software, used to interact with forums. The
ForumLoginStrategy interface is used to detect the forum software using the
<a href="https://en.wikipedia.org/wiki/Strategy_pattern">Strategy Pattern</a> and obtain
access information.</p>
<figure class="undefined"><a href="/static/img/o4BdQ4E-74-1308.png"><picture><source type="image/webp" srcset="/static/img/K1dSqW5Gub-540.webp 540w"><img alt="Diagram showing the zones used by the login flow. Repositories are also used to access the database." loading="lazy" decoding="async" src="/static/img/K1dSqW5Gub-540.png" width="540" height="54"></picture></a><figcaption>Diagram showing the zones used by the login flow. Repositories are also used to access the database.</figcaption></figure>
<h2 id="why-i-haven-t-published-it-on-google-play" tabindex="-1">Why I haven’t published it on Google Play <a class="header-anchor" href="#why-i-haven-t-published-it-on-google-play">#</a></h2>
<p>I’ve decided not to publish ForumMate for two reasons:</p>
<ul>
<li><strong>Crawling raises legal concerns:</strong>
Website owners are unlikely to like their sites being access by automated
tools. There’s also a risk of it being considered copyright infringement,
I’d have to look into this.</li>
<li><strong>The difficulty of crawling takes too much focus:</strong>
Forum software varies massively, with different themes and backing software,
which makes the parsing software quite brittle. Fixing this would take too
much focus away from app development and places it on HTML parsing, which is
not what I want to achieve with this project.</li>
</ul>
<p>I’ve achieved what I intended to do with ForumMate - I’ve gotten back up to
speed with Android development.</p>
rubenwardy
Writing a Lua sandbox using sol2 / sol3
2020-07-26T00:00:00.000Z
2020-07-26T00:00:00.000Z
https://blog.rubenwardy.com/2020/07/26/sol3-script-sandbox/
Sandboxing Lua protects the user's computer from malicious or buggy scripts. This tutorial shows how to do this using sol2 v3.0.
<p>Sandboxing can protect the user’s computer from malicious or buggy scripts.
But sandboxes are difficult to get right; you need to be very careful with what
you expose, and make sure you test for vulnerabilities.
The <a href="http://lua-users.org/wiki/SandBoxes">Sandboxes</a> on the Lua wiki is required
reading, as it contains very helpful advice.</p>
<h2 id="environments" tabindex="-1">Environments <a class="header-anchor" href="#environments">#</a></h2>
<p>An environment is a table that stores the global variables available to a function.
Each function will have an environment assigned to it, and we can use this to sandbox
code. To manage environments in sol3, you will use
<a href="https://sol2.readthedocs.io/en/latest/api/environment.html">sol::environment</a>.</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token comment">// lua is a `sol::state` or `sol::state_view`</span>
<span class="token comment">// Create new blank environment</span>
<span class="token keyword">auto</span> env <span class="token operator">=</span> sol<span class="token double-colon punctuation">::</span><span class="token function">environment</span><span class="token punctuation">(</span>lua<span class="token punctuation">,</span> sol<span class="token double-colon punctuation">::</span>create<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">// Set global variable for globals</span>
env<span class="token punctuation">[</span><span class="token string">"_G"</span><span class="token punctuation">]</span> <span class="token operator">=</span> env<span class="token punctuation">;</span></code></pre>
<p>To sandbox, you will want to create an environment with only whitelisted,
safe functions. Let’s list all the safe global functions:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">const</span> std<span class="token double-colon punctuation">::</span>vector<span class="token operator"><</span>std<span class="token double-colon punctuation">::</span>string<span class="token operator">></span> whitelisted <span class="token operator">=</span> <span class="token punctuation">{</span>
<span class="token string">"assert"</span><span class="token punctuation">,</span>
<span class="token string">"error"</span><span class="token punctuation">,</span>
<span class="token string">"ipairs"</span><span class="token punctuation">,</span>
<span class="token string">"next"</span><span class="token punctuation">,</span>
<span class="token string">"pairs"</span><span class="token punctuation">,</span>
<span class="token string">"pcall"</span><span class="token punctuation">,</span>
<span class="token string">"print"</span><span class="token punctuation">,</span>
<span class="token string">"select"</span><span class="token punctuation">,</span>
<span class="token string">"tonumber"</span><span class="token punctuation">,</span>
<span class="token string">"tostring"</span><span class="token punctuation">,</span>
<span class="token string">"type"</span><span class="token punctuation">,</span>
<span class="token string">"unpack"</span><span class="token punctuation">,</span>
<span class="token string">"_VERSION"</span><span class="token punctuation">,</span>
<span class="token string">"xpcall"</span><span class="token punctuation">,</span>
<span class="token comment">// These functions are unsafe as they can bypass or change metatables,</span>
<span class="token comment">// but they are required to implement classes.</span>
<span class="token string">"rawequal"</span><span class="token punctuation">,</span>
<span class="token string">"rawget"</span><span class="token punctuation">,</span>
<span class="token string">"rawset"</span><span class="token punctuation">,</span>
<span class="token string">"setmetatable"</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre>
<p>Now, let’s copy the whitelisted globals into the environment:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">const</span> <span class="token keyword">auto</span> <span class="token operator">&</span>name <span class="token operator">:</span> whitelisted<span class="token punctuation">)</span> <span class="token punctuation">{</span>
env<span class="token punctuation">[</span>name<span class="token punctuation">]</span> <span class="token operator">=</span> lua<span class="token punctuation">[</span>name<span class="token punctuation">]</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>Next, you’ll want to define and copy whitelisted modules.
We didn’t include these in the above list, as we want to copy the tables themselves.
This prevents changes that untrusted code makes to modules from affecting trusted code.</p>
<pre class="language-cpp"><code class="language-cpp">std<span class="token double-colon punctuation">::</span>vector<span class="token operator"><</span>std<span class="token double-colon punctuation">::</span>string<span class="token operator">></span> safeLibraries <span class="token operator">=</span> <span class="token punctuation">{</span>
<span class="token string">"coroutine"</span><span class="token punctuation">,</span> <span class="token string">"string"</span><span class="token punctuation">,</span> <span class="token string">"table"</span><span class="token punctuation">,</span> <span class="token string">"math"</span><span class="token punctuation">}</span><span class="token punctuation">;</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">const</span> <span class="token keyword">auto</span> <span class="token operator">&</span>name <span class="token operator">:</span> safeLibraries<span class="token punctuation">)</span> <span class="token punctuation">{</span>
sol<span class="token double-colon punctuation">::</span>table <span class="token function">copy</span><span class="token punctuation">(</span>lua<span class="token punctuation">,</span> sol<span class="token double-colon punctuation">::</span>create<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">auto</span> pair <span class="token operator">:</span> lua<span class="token punctuation">[</span>name<span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token comment">// first is the name of a function in module, second is the function</span>
copy<span class="token punctuation">[</span>pair<span class="token punctuation">.</span>first<span class="token punctuation">]</span> <span class="token operator">=</span> pair<span class="token punctuation">.</span>second<span class="token punctuation">;</span>
<span class="token punctuation">}</span>
env<span class="token punctuation">[</span>name<span class="token punctuation">]</span> <span class="token operator">=</span> copy<span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>Finally, you’ll want to partially copy modules that contain unsafe functions:</p>
<pre class="language-cpp"><code class="language-cpp">sol<span class="token double-colon punctuation">::</span>table <span class="token function">os</span><span class="token punctuation">(</span>lua<span class="token punctuation">,</span> sol<span class="token double-colon punctuation">::</span>create<span class="token punctuation">)</span><span class="token punctuation">;</span>
os<span class="token punctuation">[</span><span class="token string">"clock"</span><span class="token punctuation">]</span> <span class="token operator">=</span> lua<span class="token punctuation">[</span><span class="token string">"os"</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">"clock"</span><span class="token punctuation">]</span><span class="token punctuation">;</span>
os<span class="token punctuation">[</span><span class="token string">"date"</span><span class="token punctuation">]</span> <span class="token operator">=</span> lua<span class="token punctuation">[</span><span class="token string">"os"</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">"date"</span><span class="token punctuation">]</span><span class="token punctuation">;</span>
os<span class="token punctuation">[</span><span class="token string">"difftime"</span><span class="token punctuation">]</span> <span class="token operator">=</span> lua<span class="token punctuation">[</span><span class="token string">"os"</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">"difftime"</span><span class="token punctuation">]</span><span class="token punctuation">;</span>
os<span class="token punctuation">[</span><span class="token string">"time"</span><span class="token punctuation">]</span> <span class="token operator">=</span> lua<span class="token punctuation">[</span><span class="token string">"os"</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">"time"</span><span class="token punctuation">]</span><span class="token punctuation">;</span>
env<span class="token punctuation">[</span><span class="token string">"os"</span><span class="token punctuation">]</span> <span class="token operator">=</span> os<span class="token punctuation">;</span></code></pre>
<h2 id="safe-loadstring-loadfile-and-dofile" tabindex="-1">Safe loadstring, loadfile, and dofile <a class="header-anchor" href="#safe-loadstring-loadfile-and-dofile">#</a></h2>
<h3 id="what" tabindex="-1">What? <a class="header-anchor" href="#what">#</a></h3>
<p>Let’s first revise what each function does:</p>
<ul>
<li><a href="https://www.lua.org/manual/5.1/manual.html#pdf-loadstring">loadstring</a>(string [, chunkname])
<ul>
<li>Attempts to load <code>string</code> as a function.</li>
<li>Returns either the function, or <code>nil, error_msg</code>.</li>
</ul>
</li>
<li><a href="https://www.lua.org/manual/5.1/manual.html#pdf-loadfile">loadfile</a>(path)
<ul>
<li>Attempts to load file <code>file</code> as a function.</li>
<li>Returns either the function, or <code>nil, error_msg</code>.</li>
</ul>
</li>
<li><a href="https://www.lua.org/manual/5.1/manual.html#pdf-dofile">dofile</a>(path)
<ul>
<li>Loads from the file, and executes it. Returns the return value of the file.</li>
<li>If there is an error, it is not caught by <code>dofile</code>.</li>
</ul>
</li>
</ul>
<p>We need to provide safe implementations of each of these functions.
We will do this be making sure that the following things are checked:</p>
<ol>
<li>The environment is set on any functions. The default <code>loadstring</code> will set the
global environment on the returned function, which allows escaping the sandbox.</li>
<li>Bytecode cannot be loaded, as it can be used to escape the sandbox.</li>
<li>In loadfile, we check that the file path is within the sandbox path.</li>
</ol>
<h3 id="loadstring" tabindex="-1">loadstring <a class="header-anchor" href="#loadstring">#</a></h3>
<pre class="language-cpp"><code class="language-cpp">std<span class="token double-colon punctuation">::</span>tuple<span class="token operator"><</span>sol<span class="token double-colon punctuation">::</span>object<span class="token punctuation">,</span> sol<span class="token double-colon punctuation">::</span>object<span class="token operator">></span> <span class="token class-name">LuaSecurity</span><span class="token double-colon punctuation">::</span><span class="token function">loadstring</span><span class="token punctuation">(</span>
<span class="token keyword">const</span> std<span class="token double-colon punctuation">::</span>string <span class="token operator">&</span>str<span class="token punctuation">,</span> <span class="token keyword">const</span> std<span class="token double-colon punctuation">::</span>string <span class="token operator">&</span>chunkname<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span>str<span class="token punctuation">.</span><span class="token function">empty</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">&&</span> str<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">==</span> LUA_SIGNATURE<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> std<span class="token double-colon punctuation">::</span><span class="token function">make_tuple</span><span class="token punctuation">(</span>sol<span class="token double-colon punctuation">::</span>nil<span class="token punctuation">,</span>
sol<span class="token double-colon punctuation">::</span><span class="token function">make_object</span><span class="token punctuation">(</span>lua<span class="token punctuation">,</span> <span class="token string">"Bytecode prohibited by Lua sandbox"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
sol<span class="token double-colon punctuation">::</span>load_result result <span class="token operator">=</span> lua<span class="token punctuation">.</span><span class="token function">load</span><span class="token punctuation">(</span>str<span class="token punctuation">,</span> chunkname<span class="token punctuation">,</span> sol<span class="token double-colon punctuation">::</span>load_mode<span class="token double-colon punctuation">::</span>text<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>result<span class="token punctuation">.</span><span class="token function">valid</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
sol<span class="token double-colon punctuation">::</span>function func <span class="token operator">=</span> result<span class="token punctuation">;</span>
env<span class="token punctuation">.</span><span class="token function">set_on</span><span class="token punctuation">(</span>func<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">return</span> std<span class="token double-colon punctuation">::</span><span class="token function">make_tuple</span><span class="token punctuation">(</span>func<span class="token punctuation">,</span> sol<span class="token double-colon punctuation">::</span>nil<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> std<span class="token double-colon punctuation">::</span><span class="token function">make_tuple</span><span class="token punctuation">(</span>
sol<span class="token double-colon punctuation">::</span>nil<span class="token punctuation">,</span> sol<span class="token double-colon punctuation">::</span><span class="token function">make_object</span><span class="token punctuation">(</span>lua<span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">(</span>sol<span class="token double-colon punctuation">::</span>error<span class="token punctuation">)</span>result<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">what</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p><code>LUA_SIGNATURE</code> is the character used to indicate that some source code is
bytecode.</p>
<p><code>env.set_on(func)</code> is used to set the environment.</p>
<h3 id="loadfile" tabindex="-1">loadfile <a class="header-anchor" href="#loadfile">#</a></h3>
<pre class="language-cpp"><code class="language-cpp">std<span class="token double-colon punctuation">::</span>tuple<span class="token operator"><</span>sol<span class="token double-colon punctuation">::</span>object<span class="token punctuation">,</span> sol<span class="token double-colon punctuation">::</span>object<span class="token operator">></span> <span class="token class-name">LuaSecurity</span><span class="token double-colon punctuation">::</span><span class="token function">loadfile</span><span class="token punctuation">(</span>
<span class="token keyword">const</span> std<span class="token double-colon punctuation">::</span>string <span class="token operator">&</span>path<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token function">checkPath</span><span class="token punctuation">(</span>path<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> std<span class="token double-colon punctuation">::</span><span class="token function">make_tuple</span><span class="token punctuation">(</span>sol<span class="token double-colon punctuation">::</span>nil<span class="token punctuation">,</span>
sol<span class="token double-colon punctuation">::</span><span class="token function">make_object</span><span class="token punctuation">(</span>
lua<span class="token punctuation">,</span> <span class="token string">"Path is not allowed by the Lua sandbox"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
std<span class="token double-colon punctuation">::</span>ifstream <span class="token function">t</span><span class="token punctuation">(</span>path<span class="token punctuation">)</span><span class="token punctuation">;</span>
std<span class="token double-colon punctuation">::</span>string <span class="token function">str</span><span class="token punctuation">(</span><span class="token punctuation">(</span>std<span class="token double-colon punctuation">::</span><span class="token generic-function"><span class="token function">istreambuf_iterator</span><span class="token generic class-name"><span class="token operator"><</span><span class="token keyword">char</span><span class="token operator">></span></span></span><span class="token punctuation">(</span>t<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
std<span class="token double-colon punctuation">::</span><span class="token generic-function"><span class="token function">istreambuf_iterator</span><span class="token generic class-name"><span class="token operator"><</span><span class="token keyword">char</span><span class="token operator">></span></span></span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">return</span> <span class="token function">loadstring</span><span class="token punctuation">(</span>str<span class="token punctuation">,</span> <span class="token string">"@"</span> <span class="token operator">+</span> path<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>checkPath is a method that will be used later to verify that <code>path</code> is allowed,
for now it can be defined to always return true.</p>
<h3 id="dofile" tabindex="-1">dofile <a class="header-anchor" href="#dofile">#</a></h3>
<pre class="language-cpp"><code class="language-cpp">sol<span class="token double-colon punctuation">::</span>object <span class="token class-name">LuaSecurity</span><span class="token double-colon punctuation">::</span><span class="token function">dofile</span><span class="token punctuation">(</span><span class="token keyword">const</span> std<span class="token double-colon punctuation">::</span>string <span class="token operator">&</span>path<span class="token punctuation">)</span> <span class="token punctuation">{</span>
std<span class="token double-colon punctuation">::</span>tuple<span class="token operator"><</span>sol<span class="token double-colon punctuation">::</span>object<span class="token punctuation">,</span> sol<span class="token double-colon punctuation">::</span>object<span class="token operator">></span> ret <span class="token operator">=</span> <span class="token function">loadfile</span><span class="token punctuation">(</span>path<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>std<span class="token double-colon punctuation">::</span><span class="token generic-function"><span class="token function">get</span><span class="token generic class-name"><span class="token operator"><</span><span class="token number">0</span><span class="token operator">></span></span></span><span class="token punctuation">(</span>ret<span class="token punctuation">)</span> <span class="token operator">==</span> sol<span class="token double-colon punctuation">::</span>nil<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">throw</span> sol<span class="token double-colon punctuation">::</span><span class="token function">error</span><span class="token punctuation">(</span>std<span class="token double-colon punctuation">::</span><span class="token generic-function"><span class="token function">get</span><span class="token generic class-name"><span class="token operator"><</span><span class="token number">1</span><span class="token operator">></span></span></span><span class="token punctuation">(</span>ret<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token generic-function"><span class="token function">as</span><span class="token generic class-name"><span class="token operator"><</span>std<span class="token double-colon punctuation">::</span>string<span class="token operator">></span></span></span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
sol<span class="token double-colon punctuation">::</span>unsafe_function func <span class="token operator">=</span> std<span class="token double-colon punctuation">::</span><span class="token generic-function"><span class="token function">get</span><span class="token generic class-name"><span class="token operator"><</span><span class="token number">0</span><span class="token operator">></span></span></span><span class="token punctuation">(</span>ret<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">return</span> <span class="token function">func</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p><code>dofile</code> will need to check the load result, and run the function in unsafe mode.</p>
<h3 id="adding-to-the-environment" tabindex="-1">Adding to the environment <a class="header-anchor" href="#adding-to-the-environment">#</a></h3>
<p>Don’t forget to actually set them on the environment!</p>
<pre class="language-cpp"><code class="language-cpp">env<span class="token punctuation">.</span><span class="token function">set_function</span><span class="token punctuation">(</span><span class="token string">"loadstring"</span><span class="token punctuation">,</span> <span class="token operator">&</span>LuaSecurity<span class="token double-colon punctuation">::</span>loadstring<span class="token punctuation">,</span> <span class="token keyword">this</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
env<span class="token punctuation">.</span><span class="token function">set_function</span><span class="token punctuation">(</span><span class="token string">"loadfile"</span><span class="token punctuation">,</span> <span class="token operator">&</span>LuaSecurity<span class="token double-colon punctuation">::</span>loadfile<span class="token punctuation">,</span> <span class="token keyword">this</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
env<span class="token punctuation">.</span><span class="token function">set_function</span><span class="token punctuation">(</span><span class="token string">"dofile"</span><span class="token punctuation">,</span> <span class="token operator">&</span>LuaSecurity<span class="token double-colon punctuation">::</span>dofile<span class="token punctuation">,</span> <span class="token keyword">this</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h2 id="running-scripts-safely" tabindex="-1">Running scripts safely <a class="header-anchor" href="#running-scripts-safely">#</a></h2>
<p>The easiest way to run a script safely is to pass the environment into <code>script_file</code>:</p>
<pre class="language-cpp"><code class="language-cpp">lua<span class="token punctuation">.</span><span class="token function">script_file</span><span class="token punctuation">(</span><span class="token string">"mods/mymod/init.lua"</span><span class="token punctuation">,</span> security<span class="token operator">-></span><span class="token function">getEnvironment</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h2 id="setting-the-global-environment" tabindex="-1">Setting the global environment <a class="header-anchor" href="#setting-the-global-environment">#</a></h2>
<p>In order to safely execute our scripts, we need to remember to set the safe
environment. Wouldn’t it be nicer to change the default environment in Lua?</p>
<p>The default environment in Lua is stored in a registry value, and so can be assigned
like so:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">if</span> <span class="token expression">LUA_VERSION_NUM <span class="token operator">>=</span> <span class="token number">502</span></span></span>
<span class="token comment">// Get environment registry index</span>
<span class="token function">lua_rawgeti</span><span class="token punctuation">(</span>lua<span class="token punctuation">,</span> LUA_REGISTRYINDEX<span class="token punctuation">,</span> env<span class="token punctuation">.</span><span class="token function">registry_index</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">// Set the global environment</span>
<span class="token function">lua_rawseti</span><span class="token punctuation">(</span>lua<span class="token punctuation">,</span> LUA_REGISTRYINDEX<span class="token punctuation">,</span> LUA_RIDX_GLOBALS<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">else</span></span>
<span class="token comment">// Get main thread</span>
<span class="token keyword">int</span> is_main <span class="token operator">=</span> <span class="token function">lua_pushthread</span><span class="token punctuation">(</span>lua<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token function">assert</span><span class="token punctuation">(</span>is_main<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">int</span> thread <span class="token operator">=</span> <span class="token function">lua_gettop</span><span class="token punctuation">(</span>lua<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">// Get environment registry index</span>
<span class="token function">lua_rawgeti</span><span class="token punctuation">(</span>lua<span class="token punctuation">,</span> LUA_REGISTRYINDEX<span class="token punctuation">,</span> env<span class="token punctuation">.</span><span class="token function">registry_index</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">// Set the global environment</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token function">lua_setfenv</span><span class="token punctuation">(</span>lua<span class="token punctuation">,</span> thread<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">throw</span> <span class="token function">ModException</span><span class="token punctuation">(</span>
<span class="token string">"Security: Unable to set environment of the main Lua thread!"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">;</span>
<span class="token function">lua_pop</span><span class="token punctuation">(</span>lua<span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// Pop thread</span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">endif</span></span></code></pre>
<p>Unfortunately, the preprocessors to support multiple versions of Lua makes this
ugly. If you’re only targeting a specific version, you can remove the unused branch.</p>
<p>We can now safely load scripts directly, without specifying the environment:</p>
<pre class="language-cpp"><code class="language-cpp">lua<span class="token punctuation">.</span><span class="token function">script_file</span><span class="token punctuation">(</span><span class="token string">"mods/mymod/init.lua"</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h2 id="checking-file-paths" tabindex="-1">Checking file paths <a class="header-anchor" href="#checking-file-paths">#</a></h2>
<p>Ideally, you’d not allow any file system access to untrusted scripts. You can
use virtual file systems to load all allowed resources into memory, and then only
read from memory.</p>
<p>However, sometimes the scripts aren’t totally untrusted, and you would like to allow
some access to the file system. To do this, you can check the path to make sure it’s
in an allowed location. Note that this isn’t completely safe, symlinks can be used
to escaped the allowed path - however, if scripts can’t make symlinks then
it’s the user’s stupidity.</p>
<p>C++17’s filesystem provides useful path-parsing methods:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">bool</span> <span class="token class-name">LuaSecurity</span><span class="token double-colon punctuation">::</span><span class="token function">checkPath</span><span class="token punctuation">(</span><span class="token keyword">const</span> std<span class="token double-colon punctuation">::</span>string <span class="token operator">&</span>filepath<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>basePath<span class="token punctuation">.</span><span class="token function">empty</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token boolean">false</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">auto</span> base <span class="token operator">=</span> std<span class="token double-colon punctuation">::</span>filesystem<span class="token double-colon punctuation">::</span><span class="token function">absolute</span><span class="token punctuation">(</span>basePath<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">lexically_normal</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">auto</span> path <span class="token operator">=</span> std<span class="token double-colon punctuation">::</span>filesystem<span class="token double-colon punctuation">::</span><span class="token function">absolute</span><span class="token punctuation">(</span>filepath<span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">lexically_normal</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">auto</span> <span class="token punctuation">[</span>rootEnd<span class="token punctuation">,</span> nothing<span class="token punctuation">]</span> <span class="token operator">=</span>
std<span class="token double-colon punctuation">::</span><span class="token function">mismatch</span><span class="token punctuation">(</span>base<span class="token punctuation">.</span><span class="token function">begin</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> base<span class="token punctuation">.</span><span class="token function">end</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> path<span class="token punctuation">.</span><span class="token function">begin</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">return</span> rootEnd <span class="token operator">==</span> base<span class="token punctuation">.</span><span class="token function">end</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>You may wish to extend this to add multiple base paths, and also add a way
to make some paths readonly.</p>
<h2 id="preventing-infinite-loops" tabindex="-1">Preventing infinite loops <a class="header-anchor" href="#preventing-infinite-loops">#</a></h2>
<p>You can use <code>lua_sethook</code> to run a callback after a number of instructions,
and then raise a Lua error.</p>
<p><a href="https://stackoverflow.com/questions/2777527/stopping-a-runaway-lua-subprocess">https://stackoverflow.com/questions/2777527/stopping-a-runaway-lua-subprocess</a></p>
<h2 id="summary" tabindex="-1">Summary <a class="header-anchor" href="#summary">#</a></h2>
<p>I hope you found this article useful. This doesn’t cover all possible exploits -
untrusted code may still crash or freeze the program - but it aims to at least
protect the host from the code.</p>
<p>I’d like to finish by reminding you to add sandbox unit tests to make sure that
it’s working correctly, and you don’t accidentally break it. This can be as
simple as some <code>assert</code>s in a builtin Lua file somewhere.</p>
rubenwardy
Extending sol3's implicit type conversion
2020-07-22T00:00:00.000Z
2020-07-22T00:00:00.000Z
https://blog.rubenwardy.com/2020/07/22/sol3-vector-extension/
How to let sol3 automatically create Lua types based on C++ types when binding to a function. This is super useful for a Vector3 type.
<p>Many APIs in my game push <code>Vector3</code>s to and from Lua. It’s such a common operation,
that most of my functions used to look like this:</p>
<pre class="language-cpp"><code class="language-cpp">sol<span class="token double-colon punctuation">::</span>table <span class="token function">add</span><span class="token punctuation">(</span>sol<span class="token double-colon punctuation">::</span>table tPos<span class="token punctuation">)</span> <span class="token punctuation">{</span>
Vector3f pos <span class="token operator">=</span> <span class="token function">TableToPos</span><span class="token punctuation">(</span>tPos<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">// do something</span>
<span class="token keyword">return</span> <span class="token function">PosToTable</span><span class="token punctuation">(</span>pos<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>One of the benefits of sol is that it is able to bind Lua arguments to C++ function
arguments, converting types implicitly. Having to convert from a table to a vector
ourselves is quite annoying. It would be much nicer to have sol do it for us.
Luckily, sol allows you to customise how types are retrieved and pushed to Lua
using <a href="https://sol2.readthedocs.io/en/latest/tutorial/customization.html">Customisation Points</a>.</p>
<p>When trying to convert a type from Lua to C++, sol will call certain templated functions.
We will be customisating sol’s behaviour using a technique called template specialization,
which allows us to specialise a specific instance of the templated functions and structs.
By the end of this article, we’ll be able to use <code>Vector3</code> directly when using sol,
allowing the above code to be turned into this:</p>
<pre class="language-cpp"><code class="language-cpp">Vector3f <span class="token function">add</span><span class="token punctuation">(</span>Vector3f pos<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token comment">// do something</span>
<span class="token keyword">return</span> pos<span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<h2 id="sol-lua-get" tabindex="-1">sol_lua_get <a class="header-anchor" href="#sol-lua-get">#</a></h2>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">namespace</span> sol <span class="token punctuation">{</span>
<span class="token keyword">template</span> <span class="token operator"><</span><span class="token keyword">typename</span> <span class="token class-name">T</span><span class="token operator">></span>
<span class="token keyword">inline</span> Vector3<span class="token operator"><</span>T<span class="token operator">></span> <span class="token function">sol_lua_get</span><span class="token punctuation">(</span>sol<span class="token double-colon punctuation">::</span>types<span class="token operator"><</span>Vector3<span class="token operator"><</span>T<span class="token operator">>></span><span class="token punctuation">,</span> lua_State <span class="token operator">*</span>L<span class="token punctuation">,</span> <span class="token keyword">int</span> index<span class="token punctuation">,</span>
sol<span class="token double-colon punctuation">::</span>stack<span class="token double-colon punctuation">::</span>record <span class="token operator">&</span>tracking<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">int</span> absoluteIndex <span class="token operator">=</span> <span class="token function">lua_absindex</span><span class="token punctuation">(</span>L<span class="token punctuation">,</span> index<span class="token punctuation">)</span><span class="token punctuation">;</span>
sol<span class="token double-colon punctuation">::</span>table table <span class="token operator">=</span> sol<span class="token double-colon punctuation">::</span>stack<span class="token double-colon punctuation">::</span><span class="token generic-function"><span class="token function">get</span><span class="token generic class-name"><span class="token operator"><</span>sol<span class="token double-colon punctuation">::</span>table<span class="token operator">></span></span></span><span class="token punctuation">(</span>L<span class="token punctuation">,</span> absoluteIndex<span class="token punctuation">)</span><span class="token punctuation">;</span>
T x <span class="token operator">=</span> table<span class="token punctuation">[</span><span class="token string">"x"</span><span class="token punctuation">]</span><span class="token punctuation">;</span>
T y <span class="token operator">=</span> table<span class="token punctuation">[</span><span class="token string">"y"</span><span class="token punctuation">]</span><span class="token punctuation">;</span>
T z <span class="token operator">=</span> table<span class="token punctuation">[</span><span class="token string">"z"</span><span class="token punctuation">]</span><span class="token punctuation">;</span>
tracking<span class="token punctuation">.</span><span class="token function">use</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">return</span> <span class="token punctuation">{</span> x<span class="token punctuation">,</span> y<span class="token punctuation">,</span> z <span class="token punctuation">}</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p><code>sol_lua_get</code> is the function used to convert from Lua to C++.</p>
<p>The first argument of this function is a dummy argument used by the meta-programming
to select the correct function to use. The tracking record argument is used to tell
sol what you did in this function. In this particular case, we only read a single
argument from the stack, so we call:</p>
<pre class="language-cpp"><code class="language-cpp">tracking<span class="token punctuation">.</span><span class="token function">use</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<p>The example in the sol tutorial uses two arguments, and so calls <code>use(2)</code>.</p>
<p>Note that it’s important to do this specialisation inside the same namespace
as the original templated function. This is something that’s missed inside the
sol tutorial.</p>
<h2 id="sol-lua-push" tabindex="-1">sol_lua_push <a class="header-anchor" href="#sol-lua-push">#</a></h2>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">template</span> <span class="token operator"><</span><span class="token keyword">typename</span> <span class="token class-name">T</span><span class="token operator">></span>
<span class="token keyword">inline</span> <span class="token keyword">int</span> <span class="token function">sol_lua_push</span><span class="token punctuation">(</span>sol<span class="token double-colon punctuation">::</span>types<span class="token operator"><</span>Vector3<span class="token operator"><</span>T<span class="token operator">>></span><span class="token punctuation">,</span> lua_State <span class="token operator">*</span>L<span class="token punctuation">,</span> <span class="token keyword">const</span> Vector3<span class="token operator"><</span>T<span class="token operator">></span> <span class="token operator">&</span>pos<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token function">lua_createtable</span><span class="token punctuation">(</span>L<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token function">lua_getglobal</span><span class="token punctuation">(</span>L<span class="token punctuation">,</span> <span class="token string">"Vector"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token function">lua_setmetatable</span><span class="token punctuation">(</span>L<span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">2</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
sol<span class="token double-colon punctuation">::</span>stack_table <span class="token function">vec</span><span class="token punctuation">(</span>L<span class="token punctuation">)</span><span class="token punctuation">;</span>
vec<span class="token punctuation">[</span><span class="token string">"x"</span><span class="token punctuation">]</span> <span class="token operator">=</span> pos<span class="token punctuation">.</span>x<span class="token punctuation">;</span>
vec<span class="token punctuation">[</span><span class="token string">"y"</span><span class="token punctuation">]</span> <span class="token operator">=</span> pos<span class="token punctuation">.</span>y<span class="token punctuation">;</span>
vec<span class="token punctuation">[</span><span class="token string">"z"</span><span class="token punctuation">]</span> <span class="token operator">=</span> pos<span class="token punctuation">.</span>z<span class="token punctuation">;</span>
<span class="token keyword">return</span> <span class="token number">1</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p><code>sol_lua_push</code> is the function used to convert from C++ to Lua.</p>
<p>Notice how stack_table is used to modify the table created by <code>lua_createtable</code>.
This code also sets the global Vector as a metatable on the table, this is useful
if you have a Lua Vector class.</p>
<h2 id="lua-type-of" tabindex="-1">lua_type_of <a class="header-anchor" href="#lua-type-of">#</a></h2>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">template</span> <span class="token operator"><</span><span class="token keyword">typename</span> <span class="token class-name">T</span><span class="token operator">></span>
<span class="token keyword">struct</span> <span class="token class-name">lua_type_of</span><span class="token operator"><</span>Vector3<span class="token operator"><</span>T<span class="token operator">>></span>
<span class="token operator">:</span> std<span class="token double-colon punctuation">::</span>integral_constant<span class="token operator"><</span>sol<span class="token double-colon punctuation">::</span>type<span class="token punctuation">,</span> sol<span class="token double-colon punctuation">::</span>type<span class="token double-colon punctuation">::</span>table<span class="token operator">></span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre>
<p>This is a <a href="https://sol2.readthedocs.io/en/latest/api/types.html#type-traits">type trait</a>
used to tell sol that the Lua type should be a table.</p>
<h2 id="sol-lua-check" tabindex="-1">sol_lua_check <a class="header-anchor" href="#sol-lua-check">#</a></h2>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">template</span> <span class="token operator"><</span><span class="token keyword">typename</span> <span class="token class-name">Handler</span><span class="token punctuation">,</span> <span class="token keyword">typename</span> <span class="token class-name">T</span><span class="token operator">></span>
<span class="token keyword">inline</span> <span class="token keyword">bool</span> <span class="token function">sol_lua_check</span><span class="token punctuation">(</span>sol<span class="token double-colon punctuation">::</span>types<span class="token operator"><</span>Vector3<span class="token operator"><</span>T<span class="token operator">>></span><span class="token punctuation">,</span> lua_State <span class="token operator">*</span>L<span class="token punctuation">,</span> <span class="token keyword">int</span> index<span class="token punctuation">,</span>
Handler <span class="token operator">&&</span>handler<span class="token punctuation">,</span> sol<span class="token double-colon punctuation">::</span>stack<span class="token double-colon punctuation">::</span>record <span class="token operator">&</span>tracking<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">int</span> absoluteIndex <span class="token operator">=</span> <span class="token function">lua_absindex</span><span class="token punctuation">(</span>L<span class="token punctuation">,</span> index<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span>stack<span class="token double-colon punctuation">::</span><span class="token generic-function"><span class="token function">check</span><span class="token generic class-name"><span class="token operator"><</span>sol<span class="token double-colon punctuation">::</span>table<span class="token operator">></span></span></span><span class="token punctuation">(</span>L<span class="token punctuation">,</span> absoluteIndex<span class="token punctuation">,</span> handler<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
tracking<span class="token punctuation">.</span><span class="token function">use</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">return</span> <span class="token boolean">false</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
sol<span class="token double-colon punctuation">::</span>stack<span class="token double-colon punctuation">::</span><span class="token function">get_field</span><span class="token punctuation">(</span>L<span class="token punctuation">,</span> <span class="token string">"x"</span><span class="token punctuation">,</span> absoluteIndex<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">bool</span> x <span class="token operator">=</span> sol<span class="token double-colon punctuation">::</span>stack<span class="token double-colon punctuation">::</span><span class="token generic-function"><span class="token function">check</span><span class="token generic class-name"><span class="token operator"><</span><span class="token keyword">float</span><span class="token operator">></span></span></span><span class="token punctuation">(</span>L<span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
sol<span class="token double-colon punctuation">::</span>stack<span class="token double-colon punctuation">::</span><span class="token function">get_field</span><span class="token punctuation">(</span>L<span class="token punctuation">,</span> <span class="token string">"y"</span><span class="token punctuation">,</span> absoluteIndex<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">bool</span> y <span class="token operator">=</span> sol<span class="token double-colon punctuation">::</span>stack<span class="token double-colon punctuation">::</span><span class="token generic-function"><span class="token function">check</span><span class="token generic class-name"><span class="token operator"><</span><span class="token keyword">float</span><span class="token operator">></span></span></span><span class="token punctuation">(</span>L<span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
sol<span class="token double-colon punctuation">::</span>stack<span class="token double-colon punctuation">::</span><span class="token function">get_field</span><span class="token punctuation">(</span>L<span class="token punctuation">,</span> <span class="token string">"z"</span><span class="token punctuation">,</span> absoluteIndex<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">bool</span> z <span class="token operator">=</span> sol<span class="token double-colon punctuation">::</span>stack<span class="token double-colon punctuation">::</span><span class="token generic-function"><span class="token function">check</span><span class="token generic class-name"><span class="token operator"><</span><span class="token keyword">float</span><span class="token operator">></span></span></span><span class="token punctuation">(</span>L<span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
sol<span class="token double-colon punctuation">::</span>stack<span class="token double-colon punctuation">::</span><span class="token function">pop_n</span><span class="token punctuation">(</span>L<span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
tracking<span class="token punctuation">.</span><span class="token function">use</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">return</span> x <span class="token operator">&&</span> y <span class="token operator">&&</span> z<span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p><code>sol_lua_check</code> is the function used to determine whether a stack value is
of the correct type, and can be converted. In this case, we check that it’s
a table and it has the required fields.</p>
<h2 id="wait-how-do-i-actually-use-this" tabindex="-1">Wait, how do I actually use this? <a class="header-anchor" href="#wait-how-do-i-actually-use-this">#</a></h2>
<p>It should all be defined in a header after you include <code>sol.hpp</code>.</p>
<p>I like to have a <code>Lua.hpp</code> header with the following content:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">pragma</span> <span class="token expression">once</span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">define</span> <span class="token macro-name">SOL_ALL_SAFETIES_ON</span> <span class="token expression"><span class="token number">1</span></span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string"><sol/sol.hpp></span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string"><types/Vector3.hpp></span></span>
<span class="token keyword">namespace</span> sol <span class="token punctuation">{</span>
<span class="token comment">// All the specialisations here:</span>
<span class="token comment">// sol_lua_get, sol_lua_push sol_lua_check, lua_type_of</span>
<span class="token punctuation">}</span>
</code></pre>
<p>Instead of including <code>sol.hpp</code> in other files, I include <code>Lua.hpp</code>. This makes sure
that sol receives the same defines and the same specialisations each time.</p>
rubenwardy
Drop shadows in SFML
2020-05-12T00:00:00.000Z
2020-05-12T00:00:00.000Z
https://blog.rubenwardy.com/2020/05/12/sfml-drop-shadow/
Drop shadows are created using a Gaussian blur drawn underneath the original element. There's two methods to do this in SFML, depending on your requirements.
<p>Drop shadows are created using a
<a href="https://en.wikipedia.org/wiki/Gaussian_blur">Gaussian blur</a>
drawn underneath the original element. You can do this using either a fixed
texture or a post-processing fragment shader.</p>
<h2 id="texture-based-approach" tabindex="-1">Texture-based Approach <a class="header-anchor" href="#texture-based-approach">#</a></h2>
<p>If you do not need to have dynamic drop shadows, you should consider baking the
shadow into a texture and directly rendering that. However, a static shadow is
not suitable for things like text drop shadows, which will change drastically
depending on user input.</p>
<h2 id="shader-based-approach" tabindex="-1">Shader-based Approach <a class="header-anchor" href="#shader-based-approach">#</a></h2>
<p>The blur is done in two passes - horizontal and vertical. A pass blurs each
pixel using the weighted sum of its neighbours along a direction, up to some
radius. Each pass is done using the same shader, which is given different values
for direction.</p>
<figure class="undefined"><a href="/static/img/6dQ-80Q1rs-537.png"><picture><source type="image/webp" srcset="/static/img/cIZdCJBTdw-537.webp 537w"><img alt="A text drawn with and without a drop shadow" loading="lazy" decoding="async" src="/static/img/cIZdCJBTdw-537.png" width="537" height="288"></picture></a><figcaption>A text drawn with and without a drop shadow</figcaption></figure>
<pre class="language-glsl"><code class="language-glsl"><span class="token keyword">uniform</span> <span class="token keyword">sampler2D</span> source<span class="token punctuation">;</span>
<span class="token keyword">uniform</span> <span class="token keyword">vec4</span> mask<span class="token punctuation">;</span> <span class="token comment">// shadow color</span>
<span class="token keyword">uniform</span> <span class="token keyword">vec2</span> direction<span class="token punctuation">;</span> <span class="token comment">// step size</span>
<span class="token keyword">void</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">vec2</span> textureCoordinates <span class="token operator">=</span> gl_TexCoord<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>xy<span class="token punctuation">;</span>
<span class="token keyword">vec4</span> color <span class="token operator">=</span> <span class="token keyword">vec4</span><span class="token punctuation">(</span><span class="token number">0.0</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
color <span class="token operator">+=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>source<span class="token punctuation">,</span> textureCoordinates <span class="token operator">-</span> <span class="token number">4.0</span> <span class="token operator">*</span> direction<span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">0.0162162162</span><span class="token punctuation">;</span>
color <span class="token operator">+=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>source<span class="token punctuation">,</span> textureCoordinates <span class="token operator">-</span> <span class="token number">3.0</span> <span class="token operator">*</span> direction<span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">0.0540540541</span><span class="token punctuation">;</span>
color <span class="token operator">+=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>source<span class="token punctuation">,</span> textureCoordinates <span class="token operator">-</span> <span class="token number">2.0</span> <span class="token operator">*</span> direction<span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">0.1216216216</span><span class="token punctuation">;</span>
color <span class="token operator">+=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>source<span class="token punctuation">,</span> textureCoordinates <span class="token operator">-</span> direction<span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">0.1945945946</span><span class="token punctuation">;</span>
color <span class="token operator">+=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>source<span class="token punctuation">,</span> textureCoordinates<span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">0.2270270270</span><span class="token punctuation">;</span>
color <span class="token operator">+=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>source<span class="token punctuation">,</span> textureCoordinates <span class="token operator">+</span> direction<span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">0.1945945946</span><span class="token punctuation">;</span>
color <span class="token operator">+=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>source<span class="token punctuation">,</span> textureCoordinates <span class="token operator">+</span> <span class="token number">2.0</span> <span class="token operator">*</span> direction<span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">0.1216216216</span><span class="token punctuation">;</span>
color <span class="token operator">+=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>source<span class="token punctuation">,</span> textureCoordinates <span class="token operator">+</span> <span class="token number">3.0</span> <span class="token operator">*</span> direction<span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">0.0540540541</span><span class="token punctuation">;</span>
color <span class="token operator">+=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>source<span class="token punctuation">,</span> textureCoordinates <span class="token operator">+</span> <span class="token number">4.0</span> <span class="token operator">*</span> direction<span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">0.0162162162</span><span class="token punctuation">;</span>
color <span class="token operator">=</span> mask <span class="token operator">*</span> <span class="token keyword">vec4</span><span class="token punctuation">(</span><span class="token number">100.0</span><span class="token punctuation">,</span> <span class="token number">100.0</span><span class="token punctuation">,</span> <span class="token number">100.0</span><span class="token punctuation">,</span> color<span class="token punctuation">[</span><span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token number">100.0</span><span class="token punctuation">;</span>
gl_FragColor <span class="token operator">=</span> color<span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>This needs to be done using post-processing. The way to do this in SFML
is to render your scene to a RenderTexture, which you then render to the screen
using a fragment shader like so:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token comment">// This is an example, `texture` is a RenderTexture</span>
sf<span class="token double-colon punctuation">::</span>Sprite <span class="token function">sprite</span><span class="token punctuation">(</span>texture<span class="token punctuation">.</span><span class="token function">getTexture</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
shadowShader<span class="token operator">-></span><span class="token function">setUniform</span><span class="token punctuation">(</span><span class="token string">"mask"</span><span class="token punctuation">,</span> sf<span class="token double-colon punctuation">::</span><span class="token class-name">Glsl</span><span class="token double-colon punctuation">::</span><span class="token function">Vec4</span><span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">50</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
shadowShader<span class="token operator">-></span><span class="token function">setUniform</span><span class="token punctuation">(</span><span class="token string">"direction"</span><span class="token punctuation">,</span> sf<span class="token double-colon punctuation">::</span><span class="token function">Vector2f</span><span class="token punctuation">(</span><span class="token number">0.f</span><span class="token punctuation">,</span> pixelSize<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
sf<span class="token double-colon punctuation">::</span>RenderStates states <span class="token operator">=</span> baseStates<span class="token punctuation">;</span>
states<span class="token punctuation">.</span>shader <span class="token operator">=</span> shadowShader<span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
states<span class="token punctuation">.</span>transform<span class="token punctuation">.</span><span class="token function">translate</span><span class="token punctuation">(</span>pos<span class="token punctuation">)</span><span class="token punctuation">;</span>
target<span class="token punctuation">.</span><span class="token function">draw</span><span class="token punctuation">(</span>sprite<span class="token punctuation">,</span> states<span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<p>The source drawable should be rendered to a temporary RenderTexture. This
RenderTexture is then blurred in one direction to another RenderTexture.
This final RenderTexture is then blurred in the other direction to the screen.
The reason that you render the drawable to a RenderTexture <em>before</em> you first
blur it is to make the fragment shader act like a post-processing shader - you
don’t want to act on the source drawable’s faces, but on the resulting pixels.</p>
<p>Full example source code can be found here, with a helper class:</p>
rubenwardy
Android: Complete, generic data-binding RecyclerView adapter
2020-05-06T00:00:00.000Z
2020-05-06T00:00:00.000Z
https://blog.rubenwardy.com/2020/05/06/android-complete-recyclerview-data-binding/
Data binding greatly reduces the amount of code you need to connect user-interfaces with ViewModels.
<p>Data binding greatly reduces the amount of code you need to connect user-interfaces
with ViewModels. It keeps Activity and Fragment code small, and makes it easier
to manage lifecycles.</p>
<pre class="language-xml"><code class="language-xml"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>EditText</span>
<span class="token attr-name"><span class="token namespace">android:</span>id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@+id/username<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">android:</span>layout_height</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>wrap_content<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">android:</span>layout_width</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>match_parent<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">android:</span>text</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@={viewModel.username}<span class="token punctuation">"</span></span><span class="token punctuation">/></span></span></code></pre>
<p>I discovered that there was no attribute to bind the elements in a RecyclerView,
due to the fact that a RecyclerView needs an adapter to be able to create
element views. It would also be nice to automatically use data binding to create
the viewholders. There are a number of guides to do both of these halves, but I
now present the code to do the whole.</p>
<p>This guide assumes you have at least a beginner’s knowledge of data binding,
and have it enabled for your project.</p>
<h2 id="1-generic-data-binding-recyclerview-adapter" tabindex="-1">1. Generic data-binding RecyclerView adapter <a class="header-anchor" href="#1-generic-data-binding-recyclerview-adapter">#</a></h2>
<p>The BindingRecyclerAdapter is a generic class which allows binding item
holder views to the item ViewModel, and has a property to set the contained data.
This property will be used by data binding later.</p>
<pre class="language-kotlin"><code class="language-kotlin"><span class="token keyword">class</span> BindingRecyclerAdapter<span class="token operator"><</span>T<span class="token punctuation">,</span> V<span class="token operator">></span><span class="token punctuation">(</span><span class="token keyword">val</span> viewModel<span class="token operator">:</span> V<span class="token punctuation">,</span> <span class="token annotation builtin">@LayoutRes</span> <span class="token keyword">val</span> layout<span class="token operator">:</span> Int<span class="token punctuation">)</span><span class="token operator">:</span> RecyclerView<span class="token punctuation">.</span>Adapter<span class="token operator"><</span>BindingRecyclerAdapter<span class="token punctuation">.</span>ViewHolder<span class="token operator">></span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">var</span> <span class="token keyword">data</span><span class="token operator">:</span> List<span class="token operator"><</span>T<span class="token operator">></span> <span class="token operator">=</span> <span class="token function">emptyList</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">set</span><span class="token punctuation">(</span>v<span class="token punctuation">)</span> <span class="token punctuation">{</span>
field <span class="token operator">=</span> v
<span class="token function">notifyDataSetChanged</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token keyword">class</span> <span class="token function">ViewHolder</span><span class="token punctuation">(</span><span class="token keyword">val</span> binding<span class="token operator">:</span> ViewDataBinding<span class="token punctuation">)</span> <span class="token operator">:</span> RecyclerView<span class="token punctuation">.</span><span class="token function">ViewHolder</span><span class="token punctuation">(</span>binding<span class="token punctuation">.</span>root<span class="token punctuation">)</span> <span class="token punctuation">{</span><span class="token punctuation">}</span>
<span class="token keyword">override</span> <span class="token keyword">fun</span> <span class="token function">onCreateViewHolder</span><span class="token punctuation">(</span>parent<span class="token operator">:</span> ViewGroup<span class="token punctuation">,</span> viewType<span class="token operator">:</span> Int<span class="token punctuation">)</span><span class="token operator">:</span> ViewHolder <span class="token punctuation">{</span>
<span class="token keyword">val</span> layoutInflater <span class="token operator">=</span> LayoutInflater<span class="token punctuation">.</span><span class="token function">from</span><span class="token punctuation">(</span>parent<span class="token punctuation">.</span>context<span class="token punctuation">)</span>
<span class="token keyword">val</span> binding <span class="token operator">=</span> DataBindingUtil<span class="token punctuation">.</span><span class="token function">inflate</span><span class="token punctuation">(</span>layoutInflater<span class="token punctuation">,</span> layout<span class="token punctuation">,</span> parent<span class="token punctuation">,</span> <span class="token boolean">false</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> <span class="token function">ViewHolder</span><span class="token punctuation">(</span>binding<span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token keyword">override</span> <span class="token keyword">fun</span> <span class="token function">getItemCount</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">:</span> Int <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token keyword">data</span><span class="token punctuation">.</span>size
<span class="token punctuation">}</span>
<span class="token keyword">override</span> <span class="token keyword">fun</span> <span class="token function">onBindViewHolder</span><span class="token punctuation">(</span>holder<span class="token operator">:</span> ViewHolder<span class="token punctuation">,</span> position<span class="token operator">:</span> Int<span class="token punctuation">)</span> <span class="token punctuation">{</span>
holder<span class="token punctuation">.</span>binding<span class="token punctuation">.</span><span class="token function">setVariable</span><span class="token punctuation">(</span>BR<span class="token punctuation">.</span>item<span class="token punctuation">,</span> <span class="token keyword">data</span><span class="token punctuation">.</span><span class="token function">getOrNull</span><span class="token punctuation">(</span>position<span class="token punctuation">)</span><span class="token punctuation">)</span>
holder<span class="token punctuation">.</span>binding<span class="token punctuation">.</span><span class="token function">setVariable</span><span class="token punctuation">(</span>BR<span class="token punctuation">.</span>viewModel<span class="token punctuation">,</span> viewModel<span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<h2 id="2-binding-adapter" tabindex="-1">2. Binding adapter <a class="header-anchor" href="#2-binding-adapter">#</a></h2>
<p>Now, we need to create a binding adapter to handle the <code>data</code> attribute:</p>
<pre class="language-kotlin"><code class="language-kotlin"><span class="token annotation builtin">@BindingAdapter</span><span class="token punctuation">(</span><span class="token string-literal singleline"><span class="token string">"data"</span></span><span class="token punctuation">)</span>
<span class="token keyword">fun</span> <span class="token operator"><</span>T<span class="token operator">></span> <span class="token function">setRecyclerViewProperties</span><span class="token punctuation">(</span>recyclerView<span class="token operator">:</span> RecyclerView<span class="token punctuation">,</span> <span class="token keyword">data</span><span class="token operator">:</span> List<span class="token operator"><</span>T<span class="token operator">></span><span class="token operator">?</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token keyword">data</span> <span class="token operator">==</span> <span class="token keyword">null</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span>
<span class="token punctuation">}</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>recyclerView<span class="token punctuation">.</span>adapter <span class="token keyword">is</span> BindingRecyclerAdapter<span class="token operator"><</span><span class="token operator">*</span><span class="token punctuation">,</span> <span class="token operator">*</span><span class="token operator">></span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token punctuation">(</span>recyclerView<span class="token punctuation">.</span>adapter <span class="token keyword">as</span> BindingRecyclerAdapter<span class="token operator"><</span>T<span class="token punctuation">,</span> <span class="token operator">*</span><span class="token operator">></span><span class="token punctuation">)</span><span class="token punctuation">.</span>data <span class="token operator">=</span> <span class="token keyword">data</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<h2 id="3-parent-view" tabindex="-1">3. Parent View <a class="header-anchor" href="#3-parent-view">#</a></h2>
<p>First, you will need to add the ViewModel to the layout data section.
Layouts with data-binding need to have <code><layout></code> as the outer-most view.</p>
<pre class="language-xml"><code class="language-xml"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>layout</span> <span class="token attr-name"><span class="token namespace">xmlns:</span>android</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>http://schemas.android.com/apk/res/android<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">xmlns:</span>tools</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>http://schemas.android.com/tools<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>data</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>variable</span>
<span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>viewModel<span class="token punctuation">"</span></span>
<span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>com.example.app.viewmodels.MyListViewModel<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>data</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>androidx.recyclerview.widget.RecyclerView</span>
<span class="token attr-name"><span class="token namespace">android:</span>id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@+id/my_list<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">android:</span>scrollbars</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>vertical<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">android:</span>layout_width</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>match_parent<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">android:</span>layout_height</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>match_parent<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">app:</span>data</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@{viewModel.items}<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>layout</span><span class="token punctuation">></span></span></code></pre>
<p>Next, let’s set up our RecyclerView to use the the new BindingRecyclerAdapter:</p>
<pre class="language-kotlin"><code class="language-kotlin"><span class="token keyword">override</span> <span class="token keyword">fun</span> <span class="token function">onViewCreated</span><span class="token punctuation">(</span>view<span class="token operator">:</span> View<span class="token punctuation">,</span> savedInstanceState<span class="token operator">:</span> Bundle<span class="token operator">?</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">super</span><span class="token punctuation">.</span><span class="token function">onViewCreated</span><span class="token punctuation">(</span>view<span class="token punctuation">,</span> savedInstanceState<span class="token punctuation">)</span>
binding<span class="token punctuation">.</span><span class="token function">apply</span> <span class="token punctuation">{</span>
myList<span class="token punctuation">.</span><span class="token function">apply</span> <span class="token punctuation">{</span>
<span class="token function">setHasFixedSize</span><span class="token punctuation">(</span><span class="token boolean">true</span><span class="token punctuation">)</span>
adapter <span class="token operator">=</span> BindingRecyclerAdapter<span class="token operator"><</span>Item<span class="token operator">></span><span class="token punctuation">(</span>R<span class="token punctuation">.</span>layout<span class="token punctuation">.</span>item<span class="token punctuation">)</span>
layoutManager <span class="token operator">=</span> <span class="token function">LinearLayoutManager</span><span class="token punctuation">(</span>context<span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token comment">// This is needed to subscribe to LiveData updates</span>
lifecycleOwner <span class="token operator">=</span> <span class="token keyword">this</span><span class="token label symbol">@MyListFragment</span>
viewModel <span class="token operator">=</span> myListViewModel
<span class="token function">invalidateAll</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<h2 id="4-item-layout" tabindex="-1">4. Item Layout <a class="header-anchor" href="#4-item-layout">#</a></h2>
<pre class="language-xml"><code class="language-xml"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>layout</span> <span class="token attr-name"><span class="token namespace">xmlns:</span>android</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>http://schemas.android.com/apk/res/android<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">xmlns:</span>app</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>http://schemas.android.com/apk/res-auto<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">xmlns:</span>tools</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>http://schemas.android.com/tools<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>data</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>import</span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>android.view.View<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>variable</span>
<span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>item<span class="token punctuation">"</span></span>
<span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>com.example.app.models.My<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>variable</span>
<span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>viewModel<span class="token punctuation">"</span></span>
<span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>com.example.app.viewmodels.MyListViewModel<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>data</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>TextView</span>
<span class="token attr-name"><span class="token namespace">android:</span>id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@+id/user<span class="token punctuation">"</span></span>
<span class="token special-attr"><span class="token attr-name">style</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span><span class="token value css language-css">?attr/titleTextAppearance</span><span class="token punctuation">"</span></span></span>
<span class="token attr-name"><span class="token namespace">android:</span>layout_width</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>match_parent<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">android:</span>layout_height</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>wrap_content<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">android:</span>lines</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>1<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">android:</span>layout_marginStart</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>16dp<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">android:</span>text</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>@{item.author}<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">android:</span>textStyle</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>bold<span class="token punctuation">"</span></span>
<span class="token attr-name"><span class="token namespace">tools:</span>text</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>Username<span class="token punctuation">"</span></span> <span class="token punctuation">/></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>androidx.constraintlayout.widget.ConstraintLayout</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>layout</span><span class="token punctuation">></span></span></code></pre>
<h2 id="appendix-a-extending-to-add-an-empty-view" tabindex="-1">Appendix A: Extending to add an Empty View <a class="header-anchor" href="#appendix-a-extending-to-add-an-empty-view">#</a></h2>
<p>You may find it useful to add an empty view.</p>
<p>Add the following property to BindingRecyclerAdapter</p>
<pre class="language-kotlin"><code class="language-kotlin"><span class="token keyword">class</span> BindingRecyclerAdapter<span class="token operator"><</span>T<span class="token punctuation">,</span> V<span class="token operator">></span><span class="token punctuation">(</span><span class="token keyword">val</span> viewModel<span class="token operator">:</span> V<span class="token punctuation">,</span> <span class="token annotation builtin">@LayoutRes</span> <span class="token keyword">val</span> layout<span class="token operator">:</span> Int<span class="token punctuation">)</span><span class="token operator">:</span> RecyclerView<span class="token punctuation">.</span>Adapter<span class="token operator"><</span>BindingRecyclerAdapter<span class="token punctuation">.</span>ViewHolder<span class="token operator">></span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">var</span> emptyView<span class="token operator">:</span> View<span class="token operator">?</span> <span class="token operator">=</span> <span class="token keyword">null</span>
<span class="token keyword">set</span><span class="token punctuation">(</span>v<span class="token punctuation">)</span> <span class="token punctuation">{</span>
field <span class="token operator">=</span> v
<span class="token function">updateEmptyView</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token keyword">var</span> <span class="token keyword">data</span><span class="token operator">:</span> List<span class="token operator"><</span>T<span class="token operator">></span> <span class="token operator">=</span> <span class="token function">emptyList</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">set</span><span class="token punctuation">(</span>v<span class="token punctuation">)</span> <span class="token punctuation">{</span>
field <span class="token operator">=</span> v
<span class="token function">notifyDataSetChanged</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token function">updateEmptyView</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token keyword">private</span> <span class="token keyword">fun</span> <span class="token function">updateEmptyView</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
emptyView<span class="token operator">?</span><span class="token punctuation">.</span>visibility <span class="token operator">=</span> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token keyword">data</span><span class="token punctuation">.</span><span class="token function">isEmpty</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> View<span class="token punctuation">.</span>VISIBLE <span class="token keyword">else</span> View<span class="token punctuation">.</span>GONE
<span class="token punctuation">}</span></code></pre>
rubenwardy
Why I switched from GitHub to GitLab
2020-05-06T00:00:00.000Z
2020-05-06T00:00:00.000Z
https://blog.rubenwardy.com/2020/05/06/why-i-switched-to-gitlab/
Since switching to GitLab, a few people have asked me why. There are several draws towards GitLab, and several pushes from GitHub.
<p>Since switching to GitLab, a few people have asked me why. There are several
draws towards GitLab, and several pushes from GitHub.</p>
<h2 id="1-pipelines" tabindex="-1">1. Pipelines <a class="header-anchor" href="#1-pipelines">#</a></h2>
<h3 id="1-a-superior-continuous-integration-ci-delivery-cd" tabindex="-1">1.a. Superior Continuous Integration (CI) / Delivery (CD) <a class="header-anchor" href="#1-a-superior-continuous-integration-ci-delivery-cd">#</a></h3>
<p>I don’t usually like it when services are bloated by having a large amount of
functionality that would be better as separate programs. But I am a
pragmatist, and the ease of use of GitLab more than makes it up for me.
The ability to use <a href="https://gitlab.com/rubenwardy/docker-sfml-rvwp">Docker images</a>
in CI is so much better than what I was doing before.</p>
<p>I’ve experimented with a few different CI/CD and GitLab Pipelines are by far the
easiest I’ve found to set up and get working. It is also a lot faster than
Travis, and allows you to host your own private Runners for projects, which is
super nice.</p>
<h3 id="1-b-better-static-pages" tabindex="-1">1.b. Better Static Pages <a class="header-anchor" href="#1-b-better-static-pages">#</a></h3>
<p>GitLab Pages are published using their Pipelines system. This means that you
have a complete Linux container available to generate your static website,
allowing custom plugins and code far beyond what GitHub Pages allows you. I have
a dedicated server, which means that I don’t need GitLab/GitHub pages as a host,
instead, I use them as a nice way to generate and make updating my static sites
easy.</p>
<h2 id="2-free-and-open-source" tabindex="-1">2. Free and Open Source <a class="header-anchor" href="#2-free-and-open-source">#</a></h2>
<p>GitLab is free and open source; GitHub is not. I prefer to use free software
where possible, which is reason enough. With GitLab, I can export my information
in full from GitLab and move it to my own instance. Whilst GitHub does allow
this using its API, it would not be able to migrate to a self-hosted instance of
the same software.</p>
<h2 id="3-i-ll-be-getting-used-to-gitlab-anyway" tabindex="-1">3. I’ll be getting used to GitLab, anyway <a class="header-anchor" href="#3-i-ll-be-getting-used-to-gitlab-anyway">#</a></h2>
<p>A common argument against migrating is centered around GitHub being perceived as
easier to use. Whilst the GitLab interface is slightly more cluttered, I think
that the perceived difference in use is much more due to users being used to the
GitHub interface. This is no longer an issue for me as I will be getting used to
it at my job.</p>
<h2 id="4-no-microsoft" tabindex="-1">4. No Microsoft <a class="header-anchor" href="#4-no-microsoft">#</a></h2>
<p>My main website and blog are in private repositories, and I wanted to keep it
that way. This would require paying for the GitHub Pro plan, but I do not want
to directly contribute to Microsoft’s wealth. This was the final push towards
GitLab.</p>
rubenwardy
A Comparison of SFML GUI Libraries: TGUI vs SFGUI vs ImGui
2020-01-21T00:00:00.000Z
2020-01-21T00:00:00.000Z
https://blog.rubenwardy.com/2020/01/21/creating-game-guis-in-sfml/
An in-depth comparison of the options for making Graphical User Interfaces (GUIs/UIs) in SFML, discussing their pros and cons.
<p>SFML is an excellent library that can be used to create 2D games and similar
applications in C++. It’s an abstraction over OpenGL and various system APIs,
presenting a consistent and easy-to-use interface.</p>
<p>Providing a Graphical User Interface (GUI / UI) API is out of scope for SFML.
GUIs are complicated, and there’s no single good way to implement them.
The S in SFML stands for Simple but GUI code rarely is.</p>
<p>There are many different options to choose from when making GUIs. This article
is an in-depth comparison of the options for making GUIs in SFML, discussing
their pros and cons.</p>
<h2 id="homegrown-solution" tabindex="-1">Homegrown Solution <a class="header-anchor" href="#homegrown-solution">#</a></h2>
<ul class="factlist"><li class="good">
Simple
</li><li class="bad">
Reinventing the wheel
</li></ul>
<p>A full GUI library is likely to be overkill for most simple uses, such as those
consisting of buttons and text boxes. These can be implemented rather simply
using SFML’s graphics and input APIs. SFML already provides a nice
cross-platform API for text input and clipboards.</p>
<a class="btn is-secondary" href="https://www.binpress.com/creating-city-building-game-with-sfml-gui-system/">
Creating a GUI system with SFML
</a>
<h2 id="dear-imgui" tabindex="-1">Dear ImGui <a class="header-anchor" href="#dear-imgui">#</a></h2>
<ul class="factlist"><li class="good">
Easy API
</li><li class="good">
Well-maintained
</li><li class="warning">
Poor theming
</li><li class="info">
Designed for prototypes and tools
</li></ul>
<p>ImGui is a very easy-to-use library designed for use in prototyping and tools,
but not for GUIs which are used by a typical end user, such as those that appear
during gameplay.</p>
<p>It enables fast iteration, and prefers ease-of-use and simplicity over
performance and customisability.</p>
<p>The <a href="https://en.wikipedia.org/wiki/Immediate_mode_GUI">Immediate Mode</a> GUI
pattern combines the rendering and the event handling of a GUI element. This is
in contrast to event-based GUI libraries, where you first set up the elements
and then subscribe to receive events on them.</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token class-name">ImGui</span><span class="token double-colon punctuation">::</span><span class="token function">Text</span><span class="token punctuation">(</span><span class="token string">"Hello, world %d"</span><span class="token punctuation">,</span> <span class="token number">123</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token class-name">ImGui</span><span class="token double-colon punctuation">::</span><span class="token function">Button</span><span class="token punctuation">(</span><span class="token string">"Save"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token function">MySaveFunction</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token class-name">ImGui</span><span class="token double-colon punctuation">::</span><span class="token function">InputText</span><span class="token punctuation">(</span><span class="token string">"string"</span><span class="token punctuation">,</span> buf<span class="token punctuation">,</span> <span class="token function">IM_ARRAYSIZE</span><span class="token punctuation">(</span>buf<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token class-name">ImGui</span><span class="token double-colon punctuation">::</span><span class="token function">SliderFloat</span><span class="token punctuation">(</span><span class="token string">"float"</span><span class="token punctuation">,</span> <span class="token operator">&</span>f<span class="token punctuation">,</span> <span class="token number">0.0f</span><span class="token punctuation">,</span> <span class="token number">1.0f</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<p class="btn-row">
<a class="btn is-secondary" href="https://edw.is/using-imgui-with-sfml-pt1/">ImGui Tutorial</a>
<a class="btn is-secondary" href="https://github.com/ocornut/imgui">GitHub</a>
</p>
<h2 id="simple-fast-gui-sfgui" tabindex="-1">Simple Fast GUI (SFGUI) <a class="header-anchor" href="#simple-fast-gui-sfgui">#</a></h2>
<ul class="factlist"><li class="good">
Well-designed API
</li><li class="warning">
Poor documentation
</li><li class="bad">
Unmaintained
</li><li class="info">
Container-based
</li></ul>
<p>SFGUI provides a fully-functional automatic layouting system, with element
bounds being allocated using an approach using
<a href="https://github.com/TankOs/SFGUI/wiki/Size-policies">requisition and allocation</a>.
Put simply: child elements request a minimum size called a requisition, and then
parents allocate the final size for their children. This approach makes making
responsive GUIs much easier.</p>
<p>SFGUI doesn’t provide much documentation other than the tutorials. The source
has Doxygen documentation, but this isn’t hosted anywhere. Additionally, SFGUI
only receives the bare-minimum maintenance to remain working. The last update
which implemented a feature or fixed a bug was in June 2018. Admittedly, SFGUI
is pretty stable with very few noticeable bugs, but has some gaping omissions
such as tooltip and copy+paste support.</p>
<p>I didn’t get far enough to investigate the theming capabilities of SFGUI.</p>
<p class="btn-row">
<a class="btn is-secondary" href="https://github.com/TankOs/SFGUI/">GitHub</a>
</p>
<h2 id="texus-gui-tgui" tabindex="-1">Texus GUI (TGUI) <a class="header-anchor" href="#texus-gui-tgui">#</a></h2>
<ul class="factlist"><li class="good">
Easy API
</li><li class="good">
WYSIWYG editor
</li><li class="warning">
Bad theming
</li><li class="warning">
Custom DSLs
</li><li class="warning">
Poor documentation
</li><li class="warning">
Single maintainer
</li><li class="info">
Constraint-based
</li></ul>
<p>While Texus GUI does also provide some container-based formatting, such as a
grid element, it is predominantly a constraint-based method of implementing
GUIs. The position and size of elements is controlled using a custom
domain-specific language (DSL).</p>
<pre class="language-cpp"><code class="language-cpp">widget<span class="token operator">-></span><span class="token function">setPosition</span><span class="token punctuation">(</span><span class="token punctuation">{</span><span class="token string">"ButtonName.right + 50"</span><span class="token punctuation">,</span> <span class="token string">"ButtonName.top"</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
widget<span class="token operator">-></span><span class="token function">setSize</span><span class="token punctuation">(</span><span class="token punctuation">{</span><span class="token string">"min(&.w, &.h * 4/3)"</span><span class="token punctuation">,</span> <span class="token string">"min(&.h, &.w * 3/4)"</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<p>Styling is done using stylesheets written using another custom domain-specific
language. These stylesheets set up
<a href="https://tgui.eu/tutorials/0.9/renderers/">renderers</a> to contain properties
about the widget’s appearance.</p>
<pre class="language-lua"><code class="language-lua"><span class="token function">Button</span> <span class="token punctuation">{</span>
Texture <span class="token operator">=</span> <span class="token string">"button.png"</span> <span class="token function">Part</span><span class="token punctuation">(</span><span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">104</span><span class="token punctuation">,</span> <span class="token number">190</span><span class="token punctuation">,</span> <span class="token number">49</span><span class="token punctuation">)</span> <span class="token function">Middle</span><span class="token punctuation">(</span><span class="token number">10</span><span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">,</span> <span class="token number">170</span><span class="token punctuation">,</span> <span class="token number">29</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
TextureHover <span class="token operator">=</span> <span class="token string">"button.png"</span> <span class="token function">Part</span><span class="token punctuation">(</span><span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">155</span><span class="token punctuation">,</span> <span class="token number">190</span><span class="token punctuation">,</span> <span class="token number">45</span><span class="token punctuation">)</span> <span class="token function">Middle</span><span class="token punctuation">(</span><span class="token number">10</span><span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">,</span> <span class="token number">170</span><span class="token punctuation">,</span> <span class="token number">25</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
TextureDown <span class="token operator">=</span> <span class="token string">"button.png"</span> <span class="token function">Part</span><span class="token punctuation">(</span><span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">202</span><span class="token punctuation">,</span> <span class="token number">190</span><span class="token punctuation">,</span> <span class="token number">45</span><span class="token punctuation">)</span> <span class="token function">Middle</span><span class="token punctuation">(</span><span class="token number">10</span><span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">,</span> <span class="token number">170</span><span class="token punctuation">,</span> <span class="token number">25</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
TextureFocused <span class="token operator">=</span> <span class="token string">"button.png"</span> <span class="token function">Part</span><span class="token punctuation">(</span><span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">155</span><span class="token punctuation">,</span> <span class="token number">190</span><span class="token punctuation">,</span> <span class="token number">45</span><span class="token punctuation">)</span> <span class="token function">Middle</span><span class="token punctuation">(</span><span class="token number">10</span><span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">,</span> <span class="token number">170</span><span class="token punctuation">,</span> <span class="token number">25</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
TextureDisabled <span class="token operator">=</span> <span class="token string">"button.png"</span> <span class="token function">Part</span><span class="token punctuation">(</span><span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">195</span><span class="token punctuation">,</span> <span class="token number">49</span><span class="token punctuation">)</span> <span class="token function">Middle</span><span class="token punctuation">(</span><span class="token number">10</span><span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">,</span> <span class="token number">175</span><span class="token punctuation">,</span> <span class="token number">29</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
TextColor <span class="token operator">=</span> <span class="token function">rgb</span><span class="token punctuation">(</span><span class="token number">250</span><span class="token punctuation">,</span> <span class="token number">250</span><span class="token punctuation">,</span> <span class="token number">250</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
TextColorDisabled <span class="token operator">=</span> <span class="token function">rgb</span><span class="token punctuation">(</span><span class="token number">100</span><span class="token punctuation">,</span> <span class="token number">100</span><span class="token punctuation">,</span> <span class="token number">100</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>TGUI does appear to lack some code reuse for each element, which can make
theming hit and miss. For example, I discovered that not all elements which have
backgrounds support the same style properties to customise the background.</p>
<p>TGUI also only has a single active maintainer, the original creator Texus, who
is very responsive to bug and feature requests.</p>
<p>TGUI follows the source code approach to documentation - the docs are generated
from Doxygen comments in the source code, detailing what each thing does.</p>
<p class="btn-row">
<a class="btn is-secondary" href="https://tgui.eu/">Website</a>
<a class="btn is-secondary" href="https://github.com/texus/TGUI">GitHub</a>
<a class="btn is-secondary" href="https://tgui.eu/documentation/0.8/classtgui_1_1Grid.html">Doxygen</a>
<a class="btn is-secondary" href="https://tgui.eu/tutorials/0.9/">Tutorials</a>
<a class="btn is-secondary" href="https://discordapp.com/invite/Msf4vyx">Discord</a>
</p>
<h2 id="crazy-eddy-gui-cegui" tabindex="-1">Crazy Eddy GUI (CEGUI) <a class="header-anchor" href="#crazy-eddy-gui-cegui">#</a></h2>
<ul class="factlist"><li class="good">
Well-maintained
</li><li class="bad">
Complex
</li><li class="bad">
Ugly, Old C++
</li></ul>
<p>CEGUI is a library which isn’t specific to SFML, and because of this has a much
larger user base than any of the other options. It has a team of maintainers,
rather than a single person.</p>
<p>CEGUI uses outdated C++, and it shows in the API. There’s a lot of manual
pointer use and a lot of ugly APIs. This was an immediate killer for me, so I
didn’t go further than this when investigating this library.</p>
<p class="btn-row">
<a class="btn is-secondary" href="https://github.com/cegui/cegui">BitBucket</a>
<a class="btn is-secondary" href="http://static.cegui.org.uk/docs/0.8.7/">Docs</a>
</p>
<h2 id="opengl-gui-libraries" tabindex="-1">OpenGL GUI Libraries <a class="header-anchor" href="#opengl-gui-libraries">#</a></h2>
<p>It’s possible to make use of a much larger ecosystem by looking for GUI
libraries that target OpenGL. You can use a <code>sf::RenderTexture</code> as a target for
the raw OpenGL calls, and then render that texture to the screen.</p>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="#conclusion">#</a></h2>
<p>Whilst I prefer the container-based requisition-allocation approach of SFGUI,
the benefits of a better-maintained library are apparent which is why I use
TGUI. TGUI feels badly designed and incomplete.</p>
<p>I am considering switching either switching back to SFGUI, and maintaining one
myself, or switching the graphics backend completely in order to use a
better-maintained library like nanogui, but fear that this is too much of a
distraction from making my game.</p>
<p>In this post, I tried to collect and convey the merits of each approach but I
may have missed some things out. Feel free to comment below.</p>
rubenwardy
Rendering a topdown world with layers and z-levels using SFML
2019-04-24T00:00:00.000Z
2019-04-24T00:00:00.000Z
https://blog.rubenwardy.com/2019/04/24/rendering-multi-z-layer-topdown-map-using-SFML/
A topdown game with Z-levels is 3d despite being rendered in 2d. This technique has 3d lighting and normal mapping, with multiple layers per tile for effects.
<p><a href="https://rubenwardy.com/rvwp/">Ruben’s Virtual World Project</a>
is a game I’ve been working on for almost 4 years now.
Recently I rewrote the rendering code to support voxel lighting and multiple
z-level - heights of the map.</p>
<figure class="undefined"><a href="/static/img/OTyw-uIQ7E-400.gif"><img alt="Demonstration of multiple z-layers. The world is rendered as a cross-section, with lower layers shown with a slight fade. This is why you can see below the grass when the players drops down a layer." loading="lazy" decoding="async" src="/static/img/ogXwDoCf6g-400.gif" width="400" height="43200"></a><figcaption>Demonstration of multiple z-layers. The world is rendered as a cross-section, with lower layers shown with a slight fade. This is why you can see below the grass when the players drops down a layer.</figcaption></figure>
<h2 id="world-layers" tabindex="-1">World Layers <a class="header-anchor" href="#world-layers">#</a></h2>
<figure class="float-right mt-0 mr-0 ml-6 mb-2"><a href="/static/img/55gltIwd2x-200.png"><picture><source type="image/webp" srcset="/static/img/T1kby6HgoT-200.webp 200w"><img alt="The two layers" loading="lazy" decoding="async" src="/static/img/T1kby6HgoT-200.png" width="200" height="92"></picture></a><figcaption>The two layers</figcaption></figure>
<p>Each z-level has two layers - a tile layer and a floor/terrain layer.
Each of these layers has a mesh (VertexArray) which are created in slightly
different ways.
The floor layer is totally populated, meaning that every position has a quad
representing it. The tile layer is sparsely populated, meaning that only positions
which have a tile have a matching mesh quad.</p>
<p>Multiple z-levels are rendered one after another, with hidden z-levels not
rendered at all.</p>
<h2 id="lighting-and-shaders" tabindex="-1">Lighting and Shaders <a class="header-anchor" href="#lighting-and-shaders">#</a></h2>
<p>Lighting is performed by a shader on each mesh, and takes in a
three different textures - diffuse, normal, and lightmap.
The mesh contains UV co-ordinates which are used to index the diffuse map
and normal map, as both are dependent on the type of tile and not the position.
The lightmap is indexed using the position.</p>
<figure class="undefined"><a href="/static/img/9zEcfJbFKO-526.png"><picture><source type="image/webp" srcset="/static/img/YNGRSXbNBi-526.webp 526w"><img alt="The result and the 3 sources to the lighting shader. The weird breaks on the dirt wall is due to the walls not yet connecting themselves to each other." loading="lazy" decoding="async" src="/static/img/YNGRSXbNBi-526.png" width="526" height="389"></picture></a><figcaption>The result and the 3 sources to the lighting shader. The weird breaks on the dirt wall is due to the walls not yet connecting themselves to each other.</figcaption></figure>
<p>Each position has 3x3 pixels in the lightmap, representing the lighting above
and from each of the four sides.</p>
<p>A vertex shader is needed to export a relative position to the fragment shader:</p>
<pre class="language-glsl"><code class="language-glsl"><span class="token keyword">varying</span> <span class="token keyword">vec4</span> relativePosition<span class="token punctuation">;</span>
<span class="token keyword">void</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
relativePosition <span class="token operator">=</span> gl_Vertex<span class="token punctuation">;</span>
gl_Position <span class="token operator">=</span> gl_ModelViewProjectionMatrix <span class="token operator">*</span> gl_Vertex<span class="token punctuation">;</span>
gl_TexCoord<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">=</span> gl_TextureMatrix<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">*</span> gl_MultiTexCoord0<span class="token punctuation">;</span>
gl_FrontColor <span class="token operator">=</span> gl_Color<span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>The bulk of the work is done in the fragment shader:</p>
<pre class="language-glsl"><code class="language-glsl"><span class="token keyword">uniform</span> <span class="token keyword">sampler2D</span> source<span class="token punctuation">;</span> <span class="token comment">// diffuse map</span>
<span class="token keyword">uniform</span> <span class="token keyword">sampler2D</span> lightmap<span class="token punctuation">;</span>
<span class="token keyword">uniform</span> <span class="token keyword">sampler2D</span> normalmap<span class="token punctuation">;</span>
<span class="token keyword">uniform</span> <span class="token keyword">float</span> factor<span class="token punctuation">;</span> <span class="token comment">// Lower z-levels from cross-section have higher factors</span>
<span class="token keyword">varying</span> <span class="token keyword">vec4</span> relativePosition<span class="token punctuation">;</span>
<span class="token comment">// From a normal, calculate how much comes from each direction</span>
<span class="token keyword">vec4</span> <span class="token function">getComponents</span><span class="token punctuation">(</span><span class="token keyword">vec4</span> normal<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">vec2</span> rot <span class="token operator">=</span> normal<span class="token punctuation">.</span>xy<span class="token operator">*</span><span class="token number">2.f</span> <span class="token operator">-</span> <span class="token number">1.f</span><span class="token punctuation">;</span>
<span class="token keyword">return</span> <span class="token keyword">vec4</span><span class="token punctuation">(</span>
<span class="token function">clamp</span><span class="token punctuation">(</span><span class="token operator">-</span>rot<span class="token punctuation">.</span>y<span class="token punctuation">,</span> <span class="token number">0.f</span><span class="token punctuation">,</span> <span class="token number">1.f</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token function">clamp</span><span class="token punctuation">(</span>rot<span class="token punctuation">.</span>x<span class="token punctuation">,</span> <span class="token number">0.f</span><span class="token punctuation">,</span> <span class="token number">1.f</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token function">clamp</span><span class="token punctuation">(</span>rot<span class="token punctuation">.</span>y<span class="token punctuation">,</span> <span class="token number">0.f</span><span class="token punctuation">,</span> <span class="token number">1.f</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token function">clamp</span><span class="token punctuation">(</span><span class="token operator">-</span>rot<span class="token punctuation">.</span>x<span class="token punctuation">,</span> <span class="token number">0.f</span><span class="token punctuation">,</span> <span class="token number">1.f</span><span class="token punctuation">)</span>
<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">void</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token comment">// Get light values for each direction</span>
<span class="token keyword">vec2</span> rel <span class="token operator">=</span> <span class="token function">floor</span><span class="token punctuation">(</span>relativePosition<span class="token punctuation">.</span>xy <span class="token operator">/</span> <span class="token number">64.f</span><span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token number">16.f</span><span class="token punctuation">;</span>
<span class="token keyword">vec4</span> lightAbove <span class="token operator">=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>lightmap<span class="token punctuation">,</span> rel <span class="token operator">+</span> <span class="token number">0.5f</span> <span class="token operator">/</span> <span class="token number">16.f</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">vec4</span> lightUp <span class="token operator">=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>lightmap<span class="token punctuation">,</span> rel <span class="token operator">+</span> <span class="token keyword">vec2</span><span class="token punctuation">(</span><span class="token number">0.5f</span><span class="token punctuation">,</span> <span class="token number">0.f</span><span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token number">16.f</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">vec4</span> lightRight <span class="token operator">=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>lightmap<span class="token punctuation">,</span> rel <span class="token operator">+</span> <span class="token keyword">vec2</span><span class="token punctuation">(</span><span class="token number">0.8f</span><span class="token punctuation">,</span> <span class="token number">0.5f</span><span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token number">16.f</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">vec4</span> lightDown <span class="token operator">=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>lightmap<span class="token punctuation">,</span> rel <span class="token operator">+</span> <span class="token keyword">vec2</span><span class="token punctuation">(</span><span class="token number">0.5f</span><span class="token punctuation">,</span> <span class="token number">0.8f</span><span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token number">16.f</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">vec4</span> lightLeft <span class="token operator">=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>lightmap<span class="token punctuation">,</span> rel <span class="token operator">+</span> <span class="token keyword">vec2</span><span class="token punctuation">(</span><span class="token number">0.f</span><span class="token punctuation">,</span> <span class="token number">0.5f</span><span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token number">16.f</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">// Get normal and weighting for each direction</span>
<span class="token keyword">vec4</span> normal <span class="token operator">=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>normalmap<span class="token punctuation">,</span> gl_TexCoord<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>xy<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">vec4</span> rot <span class="token operator">=</span> <span class="token function">getComponents</span><span class="token punctuation">(</span>normal<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">// Leak sides to the above, to make underground wall tops visible</span>
lightAbove <span class="token operator">=</span> lightAbove <span class="token operator">+</span> <span class="token function">clamp</span><span class="token punctuation">(</span>lightUp <span class="token operator">+</span> lightRight <span class="token operator">+</span> lightDown <span class="token operator">+</span> lightLeft<span class="token punctuation">,</span> <span class="token number">0.f</span><span class="token punctuation">,</span> <span class="token number">0.6f</span><span class="token punctuation">)</span>
lightAbove <span class="token operator">=</span> <span class="token function">clamp</span><span class="token punctuation">(</span>lightAbove<span class="token punctuation">,</span> <span class="token number">0.f</span><span class="token punctuation">,</span> <span class="token number">1.f</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">// Calculate final light level</span>
<span class="token keyword">vec4</span> lightV <span class="token operator">=</span> rot<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token operator">*</span>lightUp <span class="token operator">+</span> rot<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token operator">*</span>lightRight <span class="token operator">+</span> rot<span class="token punctuation">[</span><span class="token number">2</span><span class="token punctuation">]</span><span class="token operator">*</span>lightDown <span class="token operator">+</span> rot<span class="token punctuation">[</span><span class="token number">3</span><span class="token punctuation">]</span><span class="token operator">*</span>lightLeft <span class="token operator">+</span> <span class="token punctuation">(</span>normal<span class="token punctuation">.</span>z<span class="token operator">*</span><span class="token number">2.f</span> <span class="token operator">-</span> <span class="token number">1.f</span><span class="token punctuation">)</span><span class="token operator">*</span>lightAbove<span class="token punctuation">;</span>
<span class="token comment">// Just support 1D lighting for now</span>
<span class="token keyword">float</span> light <span class="token operator">=</span> lightV<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">;</span>
<span class="token comment">// Calculate color</span>
<span class="token keyword">vec4</span> color <span class="token operator">=</span> <span class="token function">texture2D</span><span class="token punctuation">(</span>source<span class="token punctuation">,</span> gl_TexCoord<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>xy<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>color<span class="token punctuation">[</span><span class="token number">3</span><span class="token punctuation">]</span> <span class="token operator"><</span> <span class="token number">0.1</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
gl_FragColor <span class="token operator">=</span> color<span class="token punctuation">;</span>
<span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>
<span class="token keyword">float</span> u <span class="token operator">=</span> <span class="token number">1.0</span> <span class="token operator">-</span> <span class="token function">clamp</span><span class="token punctuation">(</span>factor<span class="token punctuation">,</span> <span class="token number">0.0</span><span class="token punctuation">,</span> <span class="token number">1.0</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> <span class="token keyword">vec4</span> BLUE <span class="token operator">=</span> <span class="token keyword">vec4</span><span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">0.75</span><span class="token punctuation">,</span> <span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token number">1.0</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
gl_FragColor <span class="token operator">=</span> <span class="token function">mix</span><span class="token punctuation">(</span>BLUE<span class="token punctuation">,</span> color<span class="token punctuation">,</span> u<span class="token punctuation">)</span> <span class="token operator">*</span> u <span class="token operator">*</span> u <span class="token operator">*</span> light <span class="token operator">*</span> light<span class="token punctuation">;</span>
gl_FragColor<span class="token punctuation">[</span><span class="token number">3</span><span class="token punctuation">]</span> <span class="token operator">=</span> color<span class="token punctuation">[</span><span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>This probably isn’t the best way to do it. This is one of the first shaders
I’ve ever written, and graphics isn’t my thing.</p>
rubenwardy
Getting rid of YouTube Music's "Are you still listening?" dialog
2019-04-16T00:00:00.000Z
2019-04-16T00:00:00.000Z
https://blog.rubenwardy.com/2019/04/16/youtube-music-are-you-still-listening-fix/
After listening for a while, YouTube will keep pausing the music. You can suppress this using a browser plugin and some JavaScript
<p>YouTube Music is a great way to listen for music for free, and with no adverts
if you use an adblocker. There is one annoying problem however: after listening
for a while, Youtube will keep pausing the music to show a dialog which says
“Are you still listening?”. This article will show how to automatically confirm
the dialog.</p>
<p>First, you need to install a plugin to allow running custom JavaScript on
web pages. I used Greasemonkey/Tampermonkey
(<a href="https://addons.mozilla.org/en-GB/firefox/addon/greasemonkey/">Firefox</a> |
<a href="https://chrome.google.com/webstore/detail/tampermonkey/dhdgffkkebhmkfjojejmpbldmpobfkfo?hl=en">Chrome</a>),
it seems to be the most popular plugin for this task.</p>
<p>Open up the plugin’s menu, and click “new user script”.
Open up the menu again, select the script and click “edit”.
Paste the following code as the source:</p>
<pre class="language-js"><code class="language-js"><span class="token comment">// ==UserScript==</span>
<span class="token comment">// @name Youtube Music Listening Clicker</span>
<span class="token comment">// @version 1</span>
<span class="token comment">// @grant none</span>
<span class="token comment">// ==/UserScript==</span>
<span class="token function">setInterval</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span>document<span class="token punctuation">.</span><span class="token function">querySelector</span><span class="token punctuation">(</span><span class="token string">"paper-dialog.ytmusic-popup-container"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">getAttribute</span><span class="token punctuation">(</span><span class="token string">"aria-hidden"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
document<span class="token punctuation">.</span><span class="token function">querySelector</span><span class="token punctuation">(</span><span class="token string">".ytmusic-you-there-renderer .style-blue-text .size-default"</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">click</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token number">1000</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">"Youtube Music Listening Clicker loaded"</span><span class="token punctuation">)</span></code></pre>
<p>Save and go back to the plugin’s menu, select the script, and click “User script options”.
Add “music.youtube.com” to the scripts include textbox and check “Only use these” below.</p>
<p>Reload youtube music if you have it open. Enjoy no more annoying dialogs!</p>
<p>Please note: the music may still pause for a second or so if the tab isn’t
focused, due to Firefox/Chrome limiting JavaScript in the backend. There’s
no real fix for this. One way to reduce the dialog from appearing at all is
to just play another playlist when the dialog first appears, otherwise it will
just keep asking.</p>
rubenwardy
C++: Self-registering functions using macros for test libraries
2019-02-17T00:00:00.000Z
2019-02-17T00:00:00.000Z
https://blog.rubenwardy.com/2019/02/17/cpp-self-registering-test-macros/
It's important that unit tests are easy to write. Rather than having to register every unit test, it would be nice to have them detected automatically.
<p>Google’s C++ testing library has a nice syntax for registering tests,
without needing to remember to add the tests to some central index.
This article will show how to use macros to allow the creation of tests
using only the following code:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token function">Test</span><span class="token punctuation">(</span>IntegerComparisonTest<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">int</span> a <span class="token operator">=</span> <span class="token number">3</span><span class="token punctuation">;</span>
<span class="token function">assert</span><span class="token punctuation">(</span>a <span class="token operator">==</span> <span class="token number">3</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>Whilst this gives the appearance of GTest adding a new keyword,
<code>Test()</code> is actually a macro. After the preprocessor stage, the code
will look something like this:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">class</span> <span class="token class-name">Test_MyTest</span> <span class="token punctuation">{</span>
<span class="token keyword">public</span><span class="token operator">:</span>
<span class="token keyword">void</span> <span class="token function">execute</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">static</span> Test_MyTest <span class="token function">create</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token function">Test_MyTest</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span>
<span class="token keyword">static</span> <span class="token keyword">bool</span> registered <span class="token operator">=</span> <span class="token class-name">TestFactory</span><span class="token double-colon punctuation">::</span><span class="token function">Register</span><span class="token punctuation">(</span><span class="token string">"MyTest"</span><span class="token punctuation">,</span> <span class="token operator">&</span>Test_MyTest<span class="token double-colon punctuation">::</span>create<span class="token punctuation">)</span>
<span class="token punctuation">}</span><span class="token punctuation">;</span>
<span class="token keyword">void</span> <span class="token class-name">Test_MyTest</span><span class="token double-colon punctuation">::</span><span class="token function">execute</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">int</span> a <span class="token operator">=</span> <span class="token number">3</span><span class="token punctuation">;</span>
<span class="token function">assert</span><span class="token punctuation">(</span>a <span class="token operator">==</span> <span class="token number">3</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>The thing to note here is that the test body isn’t actually part of the macro,
and that it utilises a split declaration and definition.</p>
<h2 id="simpler-approach" tabindex="-1">Simpler Approach <a class="header-anchor" href="#simpler-approach">#</a></h2>
<p>Now, this may be necessary in GTest’s use case to store some additional meta data
in the class, but it’s not always necessary. It would be much cleaner to cut
out the static create function and the unnecessary class.
There’s a solution to this: function pointers!</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">bool</span> <span class="token function">test_MyTest</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">static</span> <span class="token keyword">bool</span> test_MyTest_registered <span class="token operator">=</span> <span class="token class-name">TestFactory</span><span class="token double-colon punctuation">::</span><span class="token function">Register</span><span class="token punctuation">(</span><span class="token string">"MyTest"</span><span class="token punctuation">,</span> <span class="token operator">&</span>test_MyTest<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">bool</span> <span class="token function">test_MyTest</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">int</span> a <span class="token operator">=</span> <span class="token number">3</span><span class="token punctuation">;</span>
<span class="token function">assert</span><span class="token punctuation">(</span>a <span class="token operator">==</span> <span class="token number">3</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<h2 id="implementation" tabindex="-1">Implementation <a class="header-anchor" href="#implementation">#</a></h2>
<pre class="language-cpp"><code class="language-cpp"><span class="token comment">// ---- .hpp ----</span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string"><functional></span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">define</span> <span class="token macro-name function">Test</span><span class="token expression"><span class="token punctuation">(</span>name<span class="token punctuation">)</span> </span><span class="token punctuation">\</span>
<span class="token expression"><span class="token keyword">void</span> test_</span><span class="token punctuation">##</span><span class="token expression"><span class="token function">name</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> </span><span class="token punctuation">\</span>
<span class="token expression"><span class="token keyword">static</span> <span class="token keyword">bool</span> test_</span><span class="token punctuation">##</span><span class="token expression">name</span><span class="token punctuation">##</span><span class="token expression">_registered <span class="token operator">=</span> <span class="token class-name">TestFactory</span><span class="token double-colon punctuation">::</span><span class="token function">Register</span><span class="token punctuation">(</span>#name<span class="token punctuation">,</span> <span class="token operator">&</span>test_</span><span class="token punctuation">##</span><span class="token expression">name<span class="token punctuation">)</span><span class="token punctuation">;</span> </span><span class="token punctuation">\</span>
<span class="token expression"><span class="token keyword">void</span> test_</span><span class="token punctuation">##</span><span class="token expression"><span class="token function">name</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span></span>
<span class="token keyword">class</span> <span class="token class-name">TestFactory</span> <span class="token punctuation">{</span>
<span class="token keyword">public</span><span class="token operator">:</span>
<span class="token keyword">static</span> <span class="token keyword">bool</span> <span class="token function">Register</span><span class="token punctuation">(</span>std<span class="token double-colon punctuation">::</span>string name<span class="token punctuation">,</span> std<span class="token double-colon punctuation">::</span>function<span class="token operator"><</span><span class="token keyword">bool</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">></span> func<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">private</span><span class="token operator">:</span>
<span class="token keyword">static</span> std<span class="token double-colon punctuation">::</span>map<span class="token operator"><</span>std<span class="token double-colon punctuation">::</span>string<span class="token punctuation">,</span> std<span class="token double-colon punctuation">::</span>function<span class="token operator"><</span><span class="token keyword">bool</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">>></span> Tests<span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token comment">// ---- .cpp ----</span>
std<span class="token double-colon punctuation">::</span>map<span class="token operator"><</span>std<span class="token double-colon punctuation">::</span>string<span class="token punctuation">,</span> std<span class="token double-colon punctuation">::</span>function<span class="token operator"><</span><span class="token keyword">bool</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">>></span> TestRunner<span class="token double-colon punctuation">::</span>Tests<span class="token punctuation">;</span>
<span class="token keyword">bool</span> <span class="token class-name">TestFactory</span><span class="token double-colon punctuation">::</span><span class="token function">Register</span><span class="token punctuation">(</span>std<span class="token double-colon punctuation">::</span>string name<span class="token punctuation">,</span> std<span class="token double-colon punctuation">::</span>function<span class="token operator"><</span><span class="token keyword">bool</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">></span> func<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">auto</span> it <span class="token operator">=</span> Tests<span class="token punctuation">.</span><span class="token function">find</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>it <span class="token operator">==</span> Tests<span class="token punctuation">.</span><span class="token function">end</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
Tests<span class="token punctuation">[</span>name<span class="token punctuation">]</span> <span class="token operator">=</span> std<span class="token double-colon punctuation">::</span><span class="token function">move</span><span class="token punctuation">(</span>func<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">return</span> <span class="token boolean">true</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">return</span> <span class="token boolean">false</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<h2 id="sources" tabindex="-1">Sources <a class="header-anchor" href="#sources">#</a></h2>
<ul>
<li><a href="https://www.bfilipek.com/2018/02/factory-selfregister.html">Factory With Self-Registering Types</a></li>
</ul>
rubenwardy
Porting C++ programs from Linux to Windows using vcpkg, CMake, and Visual Studio
2019-02-15T00:00:00.000Z
2019-02-15T00:00:00.000Z
https://blog.rubenwardy.com/2019/02/15/porting-cpp-to-windows-vcpkg/
Recently I had to reinstall Windows to debug a hardware issue. I decided to try to make the most of this by trying to build my game on Windows.
<p>Recently I had to reinstall Windows to debug a hardware issue.
I decided to try to make the most of this by trying to build
<a href="https://rubenwardy.com/rvwp/">my game</a> on Windows.</p>
<h2 id="installing-vcpkg" tabindex="-1">Installing vcpkg <a class="header-anchor" href="#installing-vcpkg">#</a></h2>
<p>First, you need to clone vcpkg.
I installed the git bash program from <a href="https://git-scm.com/downloads">the git website</a></p>
<pre class="language-sh"><code class="language-sh"><span class="token function">git</span> clone https://github.com/Microsoft/vcpkg.git
<span class="token builtin class-name">cd</span> <span class="token function">vcpkg</span>
./bootstrap-vcpkg.bat</code></pre>
<p>Next, you need to decide whether to install 32-bit (x86) or 64-bit (x64) dependencies,
or both, then run one or both of the following commands:</p>
<pre class="language-sh"><code class="language-sh"><span class="token assign-left variable">VCPKG_DEFAULT_TRIPLET</span><span class="token operator">=</span>x86-windows ./vcpkg.exe <span class="token function">install</span> DEPS
<span class="token assign-left variable">VCPKG_DEFAULT_TRIPLET</span><span class="token operator">=</span>x64-windows ./vcpkg.exe <span class="token function">install</span> DEPS</code></pre>
<p>where <code>DEPS</code> is a space-deliminated list of package names.</p>
<p>You can search for packages like so:</p>
<pre class="language-sh"><code class="language-sh">./vcpkg.exe search query</code></pre>
<p>If you’re unable to find a dependency, you may have to write your own “port”
file to add it to vcpkg.</p>
<p>Finally, to let Visual Studio know about the dependencies, run this:</p>
<pre class="language-sh"><code class="language-sh">./vcpkg.exe integrate <span class="token function">install</span></code></pre>
<h2 id="building" tabindex="-1">Building <a class="header-anchor" href="#building">#</a></h2>
<h3 id="toolchain" tabindex="-1">Toolchain <a class="header-anchor" href="#toolchain">#</a></h3>
<p>Import your program into Visual Studio as a cmake project.
When generating the CMake cache, you’ll probably receive “not found” errors.</p>
<p>Right click on CMakeLists.txt and select “Open CMake Settings”.
A file called CMakeSettings.json should appear, with content like the following:</p>
<pre class="language-json"><code class="language-json"><span class="token punctuation">{</span>
<span class="token property">"configurations"</span><span class="token operator">:</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span>
<span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"x64-Release"</span><span class="token punctuation">,</span>
<span class="token property">"generator"</span><span class="token operator">:</span> <span class="token string">"Ninja"</span><span class="token punctuation">,</span>
<span class="token property">"configurationType"</span><span class="token operator">:</span> <span class="token string">"RelWithDebInfo"</span><span class="token punctuation">,</span>
<span class="token property">"inheritEnvironments"</span><span class="token operator">:</span> <span class="token punctuation">[</span>
<span class="token string">"msvc_x64_x64"</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token property">"buildRoot"</span><span class="token operator">:</span> <span class="token string">"${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\build\\${name}"</span><span class="token punctuation">,</span>
<span class="token property">"installRoot"</span><span class="token operator">:</span> <span class="token string">"${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\install\\${name}"</span><span class="token punctuation">,</span>
<span class="token property">"cmakeCommandArgs"</span><span class="token operator">:</span> <span class="token string">""</span><span class="token punctuation">,</span>
<span class="token property">"buildCommandArgs"</span><span class="token operator">:</span> <span class="token string">"-v"</span><span class="token punctuation">,</span>
<span class="token property">"ctestCommandArgs"</span><span class="token operator">:</span> <span class="token string">""</span>
<span class="token punctuation">}</span>
<span class="token punctuation">]</span>
<span class="token punctuation">}</span></code></pre>
<p>Add the vcpkg toolchain file to <code>cmakeCommandArgs</code></p>
<pre class="language-json"><code class="language-json"><span class="token property">"cmakeCommandArgs"</span><span class="token operator">:</span> <span class="token string">"-DCMAKE_TOOLCHAIN_FILE=\"C:\\Users\\ruben\\vcpkg\\scripts\\buildsystems\\vcpkg.cmake\""</span></code></pre>
<p>Clear the CMake cache by doing CMake > Delete Cache Folders > CMakeLists.txt,
then click the Generate button on the CMake changes detected banner.</p>
<p>This <em>should</em> be enough to get Visual Studio to find the dependencies, but I
found that this wouldn’t work when you have custom CMake find files which
probably don’t support the toolchain.</p>
<h3 id="explicit-dependency-paths" tabindex="-1">Explicit dependency paths <a class="header-anchor" href="#explicit-dependency-paths">#</a></h3>
<p>I’ve yet to find out how to modify the find modules to use vcpkg’s toolchain
cleanly, but in the meantime I set the directories manually by adding
CMake flags and environment variables.</p>
<p>You should prefer more general settings to individual include/lib settings.
For example, findX.cmake files may allow you to specify a single environment
variable to the install root instead of separate <code>X_INCLUDE_DIR</code> and <code>X_LIBRARY</code>
CMake variables. This is shown below, with <code>ENETDIR</code> and <code>SFML_ROOT</code>.
Don’t be too afraid to modify any embedded find scripts to support environment
variables.</p>
<pre class="language-json"><code class="language-json"><span class="token punctuation">{</span>
<span class="token property">"environments"</span><span class="token operator">:</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span>
<span class="token property">"ENETDIR"</span><span class="token operator">:</span> <span class="token string">"C:\\Users\\ruben\\vcpkg\\installed\\x64-windows"</span><span class="token punctuation">,</span>
<span class="token property">"SFML_ROOT"</span><span class="token operator">:</span> <span class="token string">"C:\\Users\\ruben\\vcpkg\\installed\\x64-windows"</span><span class="token punctuation">,</span>
<span class="token property">"SFGUI_ROOT"</span><span class="token operator">:</span> <span class="token string">"C:\\Users\\ruben\\vcpkg\\installed\\x64-windows"</span><span class="token punctuation">,</span>
<span class="token property">"THOR_ROOT"</span><span class="token operator">:</span> <span class="token string">"C:\\Users\\ruben\\vcpkg\\installed\\x64-windows"</span><span class="token punctuation">,</span>
<span class="token property">"LUA_ROOT"</span><span class="token operator">:</span> <span class="token string">"C:\\Users\\ruben\\vcpkg\\installed\\x64-windows"</span>
<span class="token punctuation">}</span>
<span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token property">"configurations"</span><span class="token operator">:</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span>
<span class="token property">"cmakeCommandArgs"</span><span class="token operator">:</span> <span class="token string">"-DCMAKE_TOOLCHAIN_FILE=\"C:\\Users\\ruben\\vcpkg\\scripts\\buildsystems\\vcpkg.cmake\""</span>
<span class="token punctuation">}</span>
<span class="token punctuation">]</span>
<span class="token punctuation">}</span></code></pre>
<p>Note that you may need to clear the CMake cache and regenerate for changes to
take affect.</p>
<h2 id="common-mistakes" tabindex="-1">Common mistakes <a class="header-anchor" href="#common-mistakes">#</a></h2>
<h3 id="console-program-vs-windows-program" tabindex="-1">Console program vs Windows program <a class="header-anchor" href="#console-program-vs-windows-program">#</a></h3>
<p>By default, MSVC will compile your program as a console program.
This mode results in Windows allocating and showing a console for you when starting
the program up. This console will <strong>require a redraw on every std::cerr or std::cout print,
resulting in massive performance issues</strong>.</p>
<p>If you program shows a graphical window, then you should change it to a Windows
program.</p>
<p>There are three methods to do this.</p>
<ol>
<li>
<p>The first option is to set the executable type to WIN32 in CMake:</p>
<pre class="language-cmake"><code class="language-cmake"><span class="token keyword">if</span><span class="token punctuation">(</span><span class="token variable">WIN32</span><span class="token punctuation">)</span>
<span class="token keyword">add_executable</span><span class="token punctuation">(</span><span class="token punctuation">${</span>EXECUTABLE_NAME<span class="token punctuation">}</span> <span class="token variable">WIN32</span> <span class="token punctuation">${</span>SRC<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">add_executable</span><span class="token punctuation">(</span><span class="token punctuation">${</span>EXECUTABLE_NAME<span class="token punctuation">}</span> <span class="token punctuation">${</span>SRC<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token keyword">endif</span><span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
</li>
<li>
<p>Second, by setting linker flags using CMake:</p>
<pre class="language-cmake"><code class="language-cmake"><span class="token keyword">if</span><span class="token punctuation">(</span><span class="token variable">WIN32</span><span class="token punctuation">)</span>
<span class="token keyword">set</span><span class="token punctuation">(</span><span class="token variable">CMAKE_EXE_LINKER_FLAGS</span> <span class="token string">"<span class="token interpolation"><span class="token punctuation">${</span><span class="token variable">CMAKE_EXE_LINKER_FLAGS</span><span class="token punctuation">}</span></span> /SUBSYSTEM:WINDOWS /ENTRY:mainCRTStartup"</span><span class="token punctuation">)</span>
<span class="token keyword">endif</span><span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
</li>
<li>
<p>The final option is to set the linker flags using a pragma directive:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">ifdef</span> <span class="token expression">_WIN32</span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">pragma</span> <span class="token expression"><span class="token function">comment</span><span class="token punctuation">(</span>linker<span class="token punctuation">,</span> </span><span class="token string">"/SUBSYSTEM:WINDOWS /ENTRY:mainCRTStartup"</span><span class="token expression"><span class="token punctuation">)</span></span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">endif</span></span></code></pre>
</li>
</ol>
<p>The ENTRY parameter allows you to use the standard <code>main()</code> entrypoint instead
of the non-standard <code>WinMain()</code>.
You may need to clear and regenerate the CMake cache.</p>
<h2 id="profit" tabindex="-1">Profit! <a class="header-anchor" href="#profit">#</a></h2>
<p>Hopefully that should be enough to get it work.
Please <a href="https://rubenwardy.com/contact/">contact me</a> if you know of any ways to make this cleaner or more robust.</p>
rubenwardy
RimWorld: Create timelapse from Progress Renderer on Linux
2019-01-25T00:00:00.000Z
2019-01-25T00:00:00.000Z
https://blog.rubenwardy.com/2019/01/25/rimworld-progress-renderer-to-video/
Progress Renderer is a brilliant mod that takes a screenshot once a day, and dumps it into a folder. Here's a script to turn that into a video.
<p>Progress Renderer
(<a href="https://steamcommunity.com/sharedfiles/filedetails/?id=1438693028">Steam</a> |
<a href="https://github.com/Lanilor/Progress-Renderer">GitHub</a>)
is a brilliant mod that takes a screenshot once a day, and dumps it into a folder.</p>
<p>The following bash script can be used to generate an mp4 timelapse of the screenshots.
Simply specify a region to cut out of the screenshots, and some other parameters,
and run the script in the working directory.</p>
<pre class="language-bash"><code class="language-bash"><span class="token shebang important">#!/bin/bash</span>
<span class="token comment">###########################</span>
<span class="token comment"># Video resolution (must be even)</span>
<span class="token assign-left variable">SIZE</span><span class="token operator">=</span>2272x1472
<span class="token comment"># Video FPS</span>
<span class="token assign-left variable">FPS</span><span class="token operator">=</span><span class="token number">10</span>
<span class="token comment"># Region to export</span>
<span class="token assign-left variable">CROP</span><span class="token operator">=</span>4544x2944+4064+3360
<span class="token comment"># Output file</span>
<span class="token assign-left variable">OUT</span><span class="token operator">=~</span>/Videos/rimworldvid.mp4
<span class="token comment"># TMP Location</span>
<span class="token assign-left variable">TMP</span><span class="token operator">=</span>/tmp/rimworldvid
<span class="token comment">###########################</span>
<span class="token builtin class-name">echo</span> <span class="token string">"================= CONFIG ====================="</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Region: <span class="token variable">$CROP</span>"</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Output: <span class="token variable">$OUT</span>"</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Resolu: <span class="token variable">$SIZE</span> @ <span class="token variable">$FPS</span> fps"</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Tmp: <span class="token variable">$TMP</span>/"</span>
<span class="token builtin class-name">echo</span> <span class="token string">"============================================="</span>
<span class="token builtin class-name">echo</span> <span class="token string">""</span>
<span class="token function">mkdir</span> <span class="token parameter variable">-p</span> <span class="token variable">$TMP</span>
<span class="token keyword">for</span> <span class="token for-or-select variable">filename</span> <span class="token keyword">in</span> *.jpg<span class="token punctuation">;</span> <span class="token keyword">do</span>
<span class="token keyword">if</span> <span class="token punctuation">[</span> <span class="token operator">!</span> <span class="token parameter variable">-f</span> <span class="token string">"<span class="token variable">$TMP</span>/<span class="token variable">$filename</span>"</span> <span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Cropping <span class="token variable">$filename</span>"</span>
convert <span class="token variable">$filename</span> <span class="token parameter variable">-crop</span> <span class="token variable">$CROP</span> <span class="token parameter variable">-resize</span> <span class="token variable">$SIZE</span> <span class="token string">"<span class="token variable">$TMP</span>/<span class="token variable">$filename</span>"</span> <span class="token operator">||</span> <span class="token builtin class-name">exit</span> <span class="token variable">$?</span>
<span class="token keyword">fi</span>
<span class="token keyword">done</span> <span class="token operator">||</span> <span class="token builtin class-name">exit</span> <span class="token variable">$?</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Rendering mp4"</span>
<span class="token builtin class-name">cd</span> <span class="token variable">$TMP</span>
ffmpeg <span class="token parameter variable">-framerate</span> <span class="token variable">$FPS</span> <span class="token parameter variable">-pattern_type</span> glob <span class="token parameter variable">-i</span> <span class="token string">'*.jpg'</span> <span class="token string">"<span class="token variable">$OUT</span>"</span></code></pre>
rubenwardy
Raytracer and Rasteriser
2018-09-07T00:00:00.000Z
2018-09-07T00:00:00.000Z
https://blog.rubenwardy.com/2018/09/07/raytracer-rasteriser/
I wrote a raytracer and a rasteriser for a university module. The raytracer supported features such as indirect lighting, reflection, refraction, and a photon mapper
<p>I wrote a raytracer and a rasteriser as part of my university course.
The raytracer supported features such as indirect lighting, reflection,
refraction, and a photon mapper capable of simulating the final positions of
60,000,000 photons in a few minutes (and quite a few GBs of RAM).</p>
<h2 id="raytracer" tabindex="-1">Raytracer <a class="header-anchor" href="#raytracer">#</a></h2>
<h3 id="features" tabindex="-1">Features <a class="header-anchor" href="#features">#</a></h3>
<ul>
<li>Direct, specular, and ambient lighting.</li>
<li>Soft shadows.</li>
<li>Reflection.</li>
<li>Refraction with Snell’s law, Fresnel’s law,</li>
<li>Total Internal Reflection, and Beer’s law.</li>
<li>Material roughness and rough reflections.</li>
<li>Anti-aliasing.</li>
<li>Obj 3D model importer.</li>
<li>Ability to switch to and from rasteriser using F1 key. Other F keys can be used to enable/disable certain features, see README.</li>
<li>OpenMP support</li>
<li>Undersampling - render at a lower resolution than the window.</li>
</ul>
<h3 id="direct-illumination-shadows-and-reflection" tabindex="-1">Direct Illumination, Shadows, and Reflection <a class="header-anchor" href="#direct-illumination-shadows-and-reflection">#</a></h3>
<p>Once anti-aliasing is implemented, the triangle normal need only be randomised for each sample, in order to acquire roughness. For each sample, the rotation of the triangle normal - for the triangle which the ray intersects with - is given a random offset. The magnitude of this random offset is dictated by the roughness attribute of the triangle. This gives a rough, blurred, look to rough mirrors. It also makes specularity look more realistic.</p>
<p>Reflections can be implemented simply by firing another ray from the intersection
position in the right direction, and using the colour it returns as the pixel colour.
This colour is then blended with the diffuse color depending on material properties.
You get rough mirrors for free by having roughness.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/qKskBSYM8r-955.png"><picture><source type="image/webp" srcset="/static/img/nCjCgaX20d-540.webp 540w"><img alt="Direct illumination with shadows." loading="lazy" decoding="async" src="/static/img/nCjCgaX20d-540.png" width="540" height="410"></picture></a><figcaption>Direct illumination with shadows.</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/Z2NsALWgoE-860.png"><picture><source type="image/webp" srcset="/static/img/bCdDO1pQE2-540.webp 540w"><img alt="Rough reflection and soft shadows." loading="lazy" decoding="async" src="/static/img/bCdDO1pQE2-540.png" width="540" height="401"></picture></a><figcaption>Rough reflection and soft shadows.</figcaption></figure></div>
</div>
<h3 id="refraction" tabindex="-1">Refraction <a class="header-anchor" href="#refraction">#</a></h3>
<p>Refraction is achieved by firing rays through the material, accounting for the difference
in refractive index. The amount of light which is refracted vs diffused depends on the
transparency of the material.</p>
<p>If a ray hits a transparent triangle on the opposite side to the normal, then the
refractive index is <code>1/R</code>, where R is the index of the material. Otherwise it’s just
<code>R</code>.</p>
<p>Fresnel’s Law determines how much of a ray is reflected when it hits a change
in refractive index. Above a certain angle - the critical angle - the rays totally
reflect, causing Total Internal Reflection.</p>
<p>Beer’s Law describes a property of semi-transparent materials where colour is absorbed from
light per length it passes through the material. The application of Beer’s law gives a jelly-like
effect.</p>
<pre class="language-c"><code class="language-c">refractedColor <span class="token operator">=</span> incomingColor <span class="token operator">*</span> <span class="token function">exp</span><span class="token punctuation">(</span><span class="token operator">-</span>absorbedColor <span class="token operator">*</span> lengthInMaterial<span class="token punctuation">)</span></code></pre>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/Wp35LoxACg-857.png"><picture><source type="image/webp" srcset="/static/img/6A7saBzkyy-540.webp 540w"><img alt="Pure refraction with Snell's law, Fresnel's law, and TIR." loading="lazy" decoding="async" src="/static/img/6A7saBzkyy-540.png" width="540" height="400"></picture></a><figcaption>Pure refraction with Snell's law, Fresnel's law, and TIR.</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/RqWTJ_aR0d-857.png"><picture><source type="image/webp" srcset="/static/img/_rnPYcwqZi-540.webp 540w"><img alt="Refraction with beers law - colour of material seeps in depending on length of ray." loading="lazy" decoding="async" src="/static/img/_rnPYcwqZi-540.png" width="540" height="403"></picture></a><figcaption>Refraction with beers law - colour of material seeps in depending on length of ray.</figcaption></figure></div>
</div>
<h3 id="photon-mapper" tabindex="-1">Photon Mapper <a class="header-anchor" href="#photon-mapper">#</a></h3>
<p>Millions of photons are simulated after being emitted from the light,
and then their final positions are recorded. The record of all the final
positions is called a photon map. I used an OctTree
(like a quad tree, but 2x2x2 rather than 2x2) to enable efficient querying
of photon positions.</p>
<p>Hitting any surface will result in a certain percentage of the photon’s
color being reflected, which results in indirect lighting. Photons will keep
bouncing until their color is low enough to be negligible in which case they’ll
stop. The percentage of color which isn’t reflected will be stored in the
photon map as a final photon position (ie: diffuse) or sent split into colors
and refracted if the triangle is transparent.</p>
<p>Warning: the default settings for the photon map require 4GB of free RAM.
Reduce the photon count in drawRaytracer() if you don’t have that available.</p>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/GRlnqyx5nC-848.png"><picture><source type="image/webp" srcset="/static/img/96s1ZIyGKa-540.webp 540w"><img alt="Note the light from the sphere on the walls, and the double shadow from indirect lighting on the small cube." loading="lazy" decoding="async" src="/static/img/96s1ZIyGKa-540.png" width="540" height="408"></picture></a><figcaption>Note the light from the sphere on the walls, and the double shadow from indirect lighting on the small cube.</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/8-fg9cmF7H-859.png"><picture><source type="image/webp" srcset="/static/img/GSdU1ENv3z-540.webp 540w"><img alt="A re-production of Newton’s light experiment." loading="lazy" decoding="async" src="/static/img/GSdU1ENv3z-540.png" width="540" height="401"></picture></a><figcaption>A re-production of Newton’s light experiment.</figcaption></figure></div>
</div>
<h2 id="rasteriser" tabindex="-1">Rasteriser <a class="header-anchor" href="#rasteriser">#</a></h2>
<h3 id="features-1" tabindex="-1">Features <a class="header-anchor" href="#features-1">#</a></h3>
<ul>
<li>Direct and ambient lighting.</li>
<li>Perspective Correct Interpolation.</li>
<li>“Real Time” Shadow Mapping.</li>
<li>Obj 3D model importer.</li>
<li>Ability to switch to and from rasteriser using F1 key. Other F keys can be used to enable/disable certain features, see README.</li>
<li>Undersampling - render at a lower resolution than the window.</li>
</ul>
<figure class="undefined"><a href="/static/img/ty4GmDyk6X-1356.png"><picture><source type="image/webp" srcset="/static/img/YWv0Wp9NXX-540.webp 540w"><img alt="Rasteriser with Shadows" loading="lazy" decoding="async" src="/static/img/YWv0Wp9NXX-540.png" width="540" height="540"></picture></a><figcaption>Rasteriser with Shadows</figcaption></figure>
rubenwardy
Interactive Commit Sorter for Changelogs
2018-08-24T00:00:00.000Z
2018-08-24T00:00:00.000Z
https://blog.rubenwardy.com/2018/08/24/commit-sorter-changelog/
I wrote a Bash script to sort git commits into buckets, to be used as the first step of making a change log.
<p>I wrote a Bash script to sort git commits into buckets, to be used as the first
step of making a change log. It supports rewording commit messages, can be
stopped and resumed, and supports automatic filtering based on keywords.</p>
<p>License: MIT</p>
<h2 id="example" tabindex="-1">Example <a class="header-anchor" href="#example">#</a></h2>
<pre><code>$ commitsorter ~/Documents HEAD~10..HEAD
Outputting to /home/ruben/Documents/commitsorter
[bugfix / feature / csm / ssm / other / delete / reword]
Fix something (''rubenwardy'')
: b (auto)
Add foo (''rubenwardy'')
: f
Add bnrre mispelt (''rubenwardy'')
: r
New message: Add bar (''rubenwardy'')
Add bar (''rubenwardy'')
: f
Summary:
- Bugfixes: 1
- Features: 2
- CSM : 0
- SSM : 0
- Others : 0
- Deleted : 0
</code></pre>
<h2 id="the-script" tabindex="-1">The Script <a class="header-anchor" href="#the-script">#</a></h2>
<pre class="language-bash"><code class="language-bash"><span class="token shebang important">#!/bin/bash</span>
<span class="token keyword">if</span> <span class="token punctuation">[</span><span class="token punctuation">[</span> <span class="token variable">$#</span> <span class="token parameter variable">-ne</span> <span class="token number">2</span> <span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Usage: commitsorter output REF1..REF2"</span>
<span class="token builtin class-name">exit</span> <span class="token number">1</span>
<span class="token keyword">fi</span>
<span class="token comment">### Define new results here ... ###</span>
<span class="token function-name function">makebuckets</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token builtin class-name">echo</span> <span class="token parameter variable">-e</span> <span class="token string">"## Bug fixes and Improvements ##<span class="token entity" title="\n">\n</span>"</span> <span class="token operator">></span> <span class="token string">"<span class="token variable">$1</span>/b.txt"</span>
<span class="token builtin class-name">echo</span> <span class="token parameter variable">-e</span> <span class="token string">"## Features ##<span class="token entity" title="\n">\n</span>"</span> <span class="token operator">></span> <span class="token string">"<span class="token variable">$1</span>/f.txt"</span>
<span class="token builtin class-name">echo</span> <span class="token parameter variable">-e</span> <span class="token string">"## Client Modding ##<span class="token entity" title="\n">\n</span>"</span> <span class="token operator">></span> <span class="token string">"<span class="token variable">$1</span>/c.txt"</span>
<span class="token builtin class-name">echo</span> <span class="token parameter variable">-e</span> <span class="token string">"## Server Modding ##<span class="token entity" title="\n">\n</span>"</span> <span class="token operator">></span> <span class="token string">"<span class="token variable">$1</span>/m.txt"</span>
<span class="token builtin class-name">echo</span> <span class="token parameter variable">-e</span> <span class="token string">"## Other / Misc ##<span class="token entity" title="\n">\n</span>"</span> <span class="token operator">></span> <span class="token string">"<span class="token variable">$1</span>/o.txt"</span>
<span class="token builtin class-name">echo</span> <span class="token parameter variable">-e</span> <span class="token string">"## Deleted ##<span class="token entity" title="\n">\n</span>"</span> <span class="token operator">></span> <span class="token string">"<span class="token variable">$1</span>/d.txt"</span>
<span class="token punctuation">}</span>
<span class="token comment">### ... and also add them to help ###</span>
<span class="token function-name function">help</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token builtin class-name">echo</span> <span class="token parameter variable">-e</span> <span class="token string">"[<span class="token entity" title="\e">\e</span>[4mb<span class="token entity" title="\e">\e</span>[0mugfix / <span class="token entity" title="\e">\e</span>[4mf<span class="token entity" title="\e">\e</span>[0meature / <span class="token entity" title="\e">\e</span>[4mc<span class="token entity" title="\e">\e</span>[0msm / ss<span class="token entity" title="\e">\e</span>[4mm<span class="token entity" title="\e">\e</span>[0m / <span class="token entity" title="\e">\e</span>[4mo<span class="token entity" title="\e">\e</span>[0mther / <span class="token entity" title="\e">\e</span>[4md<span class="token entity" title="\e">\e</span>[0melete / <span class="token entity" title="\e">\e</span>[4mr<span class="token entity" title="\e">\e</span>[0meword]"</span>
<span class="token punctuation">}</span>
<span class="token assign-left variable">output</span><span class="token operator">=</span><span class="token string">"<span class="token variable">$1</span>/commitsorter"</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Outputting to <span class="token variable">$output</span>"</span>
<span class="token comment">### Uncomment to backup and start again instead of resuming ###</span>
<span class="token comment"># if [ -d "$output" ]; then</span>
<span class="token comment"># DATE=$(date +%Y_%m_%d_%H_%M_%S)</span>
<span class="token comment"># backup="${output}_$DATE"</span>
<span class="token comment"># echo "Backing up $output to $backup"</span>
<span class="token comment"># mv "$output" "$backup"</span>
<span class="token comment"># fi</span>
<span class="token comment"># Create directories and buckets</span>
<span class="token function">mkdir</span> <span class="token parameter variable">-p</span> <span class="token string">"<span class="token variable">$1</span>"</span>
<span class="token function">mkdir</span> <span class="token parameter variable">-p</span> <span class="token string">"<span class="token variable">$output</span>"</span>
<span class="token function">mkdir</span> <span class="token parameter variable">-p</span> <span class="token string">"<span class="token variable">$output</span>/cache"</span>
<span class="token keyword">if</span> <span class="token punctuation">[</span> <span class="token operator">!</span> <span class="token parameter variable">-f</span> <span class="token variable">$output</span>/b.txt <span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token keyword">then</span>
makebuckets <span class="token variable">$output</span>
<span class="token keyword">fi</span>
<span class="token comment"># Get log</span>
<span class="token assign-left variable">entries</span><span class="token operator">=</span><span class="token variable"><span class="token variable">`</span><span class="token function">git</span> log <span class="token parameter variable">--pretty</span><span class="token operator">=</span>format:<span class="token string">"%h %s (''%an'')"</span> <span class="token string">"<span class="token variable">$2</span>"</span><span class="token variable">`</span></span>
<span class="token comment"># Print help</span>
<span class="token builtin class-name">echo</span> <span class="token string">""</span>
<span class="token builtin class-name">help</span>
<span class="token comment"># Loop</span>
<span class="token assign-left variable"><span class="token environment constant">IFS</span></span><span class="token operator">=</span><span class="token string">$'<span class="token entity" title="\n">\n</span>'</span>
<span class="token keyword">for</span> <span class="token for-or-select variable">line</span> <span class="token keyword">in</span> <span class="token variable">$entries</span>
<span class="token keyword">do</span>
<span class="token assign-left variable">hash</span><span class="token operator">=</span><span class="token variable"><span class="token variable">$(</span><span class="token builtin class-name">echo</span> $line <span class="token operator">|</span> <span class="token function">cut</span> -c1-7<span class="token variable">)</span></span>
<span class="token assign-left variable">message</span><span class="token operator">=</span><span class="token variable"><span class="token variable">$(</span><span class="token builtin class-name">echo</span> $line <span class="token operator">|</span> <span class="token function">cut</span> <span class="token parameter variable">-c</span> <span class="token number">9</span>- <span class="token variable">)</span></span>
<span class="token builtin class-name">echo</span> <span class="token string">""</span>
<span class="token builtin class-name">echo</span> <span class="token string">"<span class="token variable">$message</span>"</span>
<span class="token keyword">if</span> <span class="token punctuation">[</span><span class="token punctuation">[</span> <span class="token parameter variable">-f</span> <span class="token variable">$output</span>/cache/<span class="token variable">$hash</span> <span class="token punctuation">]</span><span class="token punctuation">]</span> <span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token assign-left variable">input</span><span class="token operator">=</span><span class="token variable"><span class="token variable">$(</span><span class="token function">cat</span> $output/cache/$hash<span class="token variable">)</span></span>
<span class="token builtin class-name">echo</span> <span class="token parameter variable">-e</span> <span class="token string">": <span class="token variable">$input</span> (already sorted)"</span>
<span class="token keyword">elif</span> <span class="token builtin class-name">echo</span> <span class="token variable">$message</span> <span class="token operator">|</span> <span class="token function">grep</span> <span class="token parameter variable">-iq</span> <span class="token string">"fix\(ed\)\? "</span><span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token builtin class-name">echo</span> <span class="token string">": b (auto)"</span>
<span class="token builtin class-name">echo</span> <span class="token variable">$message</span> <span class="token operator">>></span> <span class="token variable">$output</span>/b.txt
<span class="token builtin class-name">echo</span> b <span class="token operator">></span> <span class="token variable">$output</span>/cache/<span class="token variable">$hash</span>
<span class="token keyword">elif</span> <span class="token builtin class-name">echo</span> <span class="token variable">$message</span> <span class="token operator">|</span> <span class="token function">grep</span> <span class="token parameter variable">-iq</span> <span class="token string">"translated using weblate"</span><span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token builtin class-name">echo</span> <span class="token string">": d (auto)"</span>
<span class="token builtin class-name">echo</span> <span class="token variable">$message</span> <span class="token operator">>></span> <span class="token variable">$output</span>/d.txt
<span class="token builtin class-name">echo</span> d <span class="token operator">></span> <span class="token variable">$output</span>/cache/<span class="token variable">$hash</span>
<span class="token keyword">else</span>
<span class="token keyword">while</span> <span class="token boolean">true</span><span class="token punctuation">;</span> <span class="token keyword">do</span>
<span class="token builtin class-name">read</span> <span class="token parameter variable">-p</span> <span class="token string">": "</span> <span class="token parameter variable">-rsn1</span> input
<span class="token builtin class-name">echo</span> <span class="token variable">$input</span>
<span class="token assign-left variable"><span class="token environment constant">LC_ALL</span></span><span class="token operator">=</span>C <span class="token comment"># Turn off locale.</span>
<span class="token keyword">if</span> <span class="token punctuation">[</span><span class="token punctuation">[</span> <span class="token variable">$input</span> <span class="token operator">==</span> r <span class="token punctuation">]</span><span class="token punctuation">]</span> <span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token builtin class-name">echo</span> <span class="token string">""</span>
<span class="token builtin class-name">read</span> <span class="token parameter variable">-p</span> <span class="token string">"New message: "</span> line2
<span class="token keyword">if</span> <span class="token punctuation">[</span><span class="token punctuation">[</span> <span class="token operator">!</span> <span class="token parameter variable">-z</span> <span class="token variable">$message2</span> <span class="token punctuation">]</span><span class="token punctuation">]</span> <span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token assign-left variable">line</span><span class="token operator">=</span><span class="token variable">$message2</span>
<span class="token keyword">fi</span>
<span class="token builtin class-name">echo</span> <span class="token string">""</span>
<span class="token builtin class-name">echo</span> <span class="token variable">$message</span>
<span class="token keyword">elif</span> <span class="token punctuation">[</span><span class="token punctuation">[</span> <span class="token variable">$input</span> <span class="token operator">==</span> <span class="token punctuation">[</span>a-z<span class="token punctuation">]</span> <span class="token punctuation">]</span><span class="token punctuation">]</span> <span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token keyword">if</span> <span class="token punctuation">[</span> <span class="token parameter variable">-f</span> <span class="token variable">$output</span>/<span class="token variable">$input</span>.txt <span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token builtin class-name">echo</span> <span class="token variable">$message</span> <span class="token operator">>></span> <span class="token variable">$output</span>/<span class="token variable">$input</span>.txt
<span class="token builtin class-name">echo</span> <span class="token variable">$input</span> <span class="token operator">></span> <span class="token variable">$output</span>/cache/<span class="token variable">$hash</span>
<span class="token builtin class-name">break</span>
<span class="token keyword">else</span>
<span class="token builtin class-name">help</span>
<span class="token keyword">fi</span>
<span class="token keyword">else</span>
<span class="token builtin class-name">help</span>
<span class="token keyword">fi</span>
<span class="token keyword">done</span>
<span class="token keyword">fi</span>
<span class="token keyword">done</span>
<span class="token builtin class-name">echo</span> <span class="token string">""</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Summary:"</span>
<span class="token keyword">for</span> <span class="token for-or-select variable">filename</span> <span class="token keyword">in</span> <span class="token variable">$output</span>/*.txt<span class="token punctuation">;</span> <span class="token keyword">do</span>
<span class="token assign-left variable">title</span><span class="token operator">=</span><span class="token variable"><span class="token variable">$(</span><span class="token function">head</span> <span class="token parameter variable">-n1</span> $filename <span class="token operator">|</span> <span class="token function">sed</span> <span class="token string">'s/^## \([^#]*\) ##$/\1/'</span><span class="token variable">)</span></span>
<span class="token assign-left variable">count</span><span class="token operator">=</span><span class="token variable"><span class="token variable">$(</span><span class="token function">cat</span> $filename <span class="token operator">|</span> <span class="token function">tail</span> -n+2 <span class="token operator">|</span> <span class="token function">sed</span> <span class="token string">'/^\s*$/d'</span> <span class="token operator">|</span> <span class="token function">wc</span> <span class="token parameter variable">-l</span><span class="token variable">)</span></span>
<span class="token builtin class-name">echo</span> <span class="token string">"- <span class="token variable">$title</span>: <span class="token variable">$count</span>"</span>
<span class="token keyword">done</span></code></pre>
rubenwardy
Finding and copying DLLs from MinGW-W64 directories to bin directory
2018-05-07T00:00:00.000Z
2018-05-07T00:00:00.000Z
https://blog.rubenwardy.com/2018/05/07/mingw-copy-dlls/
After cross-compiling your project for Windows, you find that it crashes due to missing DLLs. I will show how to detect and copy required DLLs.
<p>After cross-compiling your project for Windows, you find that it crashes due
to missing DLLs. I will show how to identify any required DLLs using objdump,
and copy them to your build directory.</p>
<p>mingw32-objdump can be used to analysis an executable or object. Among other
things, it lists the .DLLs which are required by the .EXE. Call it by passing
the path to the executable and the <code>-p</code> flag:</p>
<pre class="language-bash"><code class="language-bash">x86_64-w64-mingw32-objdump <span class="token parameter variable">-p</span> project/bin/project.exe</code></pre>
<p>This will output a ton of content, so you’ll want to pass the result
through grep and sed to get just the DLL names:</p>
<pre class="language-bash"><code class="language-bash">x86_64-w64-mingw32-objdump <span class="token parameter variable">-p</span> project/bin/project.exe <span class="token operator">|</span> <span class="token function">grep</span> <span class="token string">'DLL Name:'</span> <span class="token operator">|</span> <span class="token function">sed</span> <span class="token parameter variable">-e</span> <span class="token string">"s/<span class="token entity" title="\t">\t</span>*DLL Name: //g"</span></code></pre>
<p>The output will look a bit like this:</p>
<pre><code>libgcc_s_seh-1.dll
KERNEL32.dll
msvcrt.dll
libwinpthread-1.dll
libstdc++-6.dll
WSOCK32.dll
sfgui.dll
sfml-graphics-2.dll
sfml-network-2.dll
sfml-system-2.dll
sfml-window-2.dll
libthor.dll
</code></pre>
<p>Now we want to iterate over the result of this, and try and find the DLL
in a number of search paths. Here’s the full script to do that:</p>
<pre class="language-bash"><code class="language-bash"><span class="token shebang important">#!/bin/bash</span>
<span class="token assign-left variable">BINDIR</span><span class="token operator">=</span><span class="token string">"project/bin"</span>
<span class="token assign-left variable">EXE</span><span class="token operator">=</span><span class="token string">"<span class="token variable">$BINDIR</span>/project.exe"</span>
<span class="token assign-left variable">PREFIX</span><span class="token operator">=</span><span class="token string">"x86_64-w64-mingw32"</span>
<span class="token assign-left variable">paths</span><span class="token operator">=</span><span class="token punctuation">(</span><span class="token string">"/usr/local/mingw64/bin"</span>
<span class="token string">"/usr/local/mingw64/bin/x64"</span>
<span class="token string">"/usr/<span class="token variable">$PREFIX</span>/bin"</span>
<span class="token string">"/usr/lib/gcc/<span class="token variable">$PREFIX</span>/7.3-posix"</span>
<span class="token string">"/usr/<span class="token variable">$PREFIX</span>/lib"</span><span class="token punctuation">)</span>
<span class="token keyword">function</span> <span class="token function-name function">findAndCopyDLL</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">for</span> <span class="token for-or-select variable">i</span> <span class="token keyword">in</span> <span class="token string">"<span class="token variable">${paths<span class="token punctuation">[</span>@<span class="token punctuation">]</span>}</span>"</span>
<span class="token keyword">do</span>
<span class="token assign-left variable">FILE</span><span class="token operator">=</span><span class="token string">"<span class="token variable">$i</span>/<span class="token variable">$1</span>"</span>
<span class="token keyword">if</span> <span class="token punctuation">[</span> <span class="token parameter variable">-f</span> <span class="token variable">$FILE</span> <span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token function">cp</span> <span class="token variable">$FILE</span> <span class="token variable">$BINDIR</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Found <span class="token variable">$1</span> in <span class="token variable">$i</span>"</span>
copyForOBJ <span class="token variable">$FILE</span>
<span class="token builtin class-name">return</span> <span class="token number">0</span>
<span class="token keyword">fi</span>
<span class="token keyword">done</span>
<span class="token builtin class-name">return</span> <span class="token number">1</span>
<span class="token punctuation">}</span>
<span class="token keyword">function</span> <span class="token function-name function">copyForOBJ</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token assign-left variable">dlls</span><span class="token operator">=</span><span class="token variable"><span class="token variable">`</span>$PREFIX-objdump <span class="token parameter variable">-p</span> $1 <span class="token operator">|</span> <span class="token function">grep</span> <span class="token string">'DLL Name:'</span> <span class="token operator">|</span> <span class="token function">sed</span> <span class="token parameter variable">-e</span> <span class="token string">"s/<span class="token entity" title="\t">\t</span>*DLL Name: //g"</span><span class="token variable">`</span></span>
<span class="token keyword">while</span> <span class="token builtin class-name">read</span> <span class="token parameter variable">-r</span> filename<span class="token punctuation">;</span> <span class="token keyword">do</span>
findAndCopyDLL <span class="token variable">$filename</span> <span class="token operator">||</span> <span class="token builtin class-name">echo</span> <span class="token string">"Unable to find <span class="token variable">$filename</span>"</span>
<span class="token keyword">done</span> <span class="token operator"><<<</span> <span class="token string">"<span class="token variable">$dlls</span>"</span>
<span class="token punctuation">}</span>
copyForOBJ <span class="token variable">$EXE</span></code></pre>
<p>The output will look like this:</p>
<pre><code>Found libgcc_s_seh-1.dll in /usr/x86_64-w64-mingw32/bin
Unable to find KERNEL32.dll
Unable to find msvcrt.dll
Found libwinpthread-1.dll in /usr/x86_64-w64-mingw32/bin
Found libstdc++-6.dll in /usr/x86_64-w64-mingw32/bin
Unable to find WSOCK32.dll
Found sfgui.dll in /usr/local/mingw64/bin
Found sfml-graphics-2.dll in /usr/local/mingw64/bin
Found sfml-network-2.dll in /usr/local/mingw64/bin
Found sfml-system-2.dll in /usr/local/mingw64/bin
Found sfml-window-2.dll in /usr/local/mingw64/bin
Found libthor.dll in /usr/local/mingw64/bin
</code></pre>
<p>Some DLLs won’t be found as they’re provided as part of Windows.</p>
rubenwardy
Rubix: My Simple Kernel written in C for arm
2018-04-05T00:00:00.000Z
2018-04-05T00:00:00.000Z
https://blog.rubenwardy.com/2018/04/05/simple-kernel-in-c/
I created an kernel with support for multiprocessing, pipes, kill, and a simple filesystem.
<p>During the second year of university, I created a kernel for the ARMv7
instruction set. I went above and beyond what was required on this project,
achieving a clean design and features such as a blocked process queue, piping,
kill, and a simple filesystem. This was my favourite coursework so far. I found
it very interesting to learn about and implement the things that we take for granted
as programmers.</p>
<p>I tried to stick to POSIX as much as possible, and stuck to the Linux method of
having everything as either a file or process. Because pipes and standard in/out
were both “files”, I was able to implement both popen and piping of the output
of a process to another process.</p>
<h2 id="features" tabindex="-1">Features <a class="header-anchor" href="#features">#</a></h2>
<h3 id="system-calls-and-helper-functions" tabindex="-1">System Calls and Helper Functions <a class="header-anchor" href="#system-calls-and-helper-functions">#</a></h3>
<p>All system calls conform to the equivalent POSIX standard, except for
<code>close</code> which will close the appropriate resource (no need for <code>fclose</code> or <code>pclose</code>)
and <code>set_nonblocking</code> which is custom.</p>
<ul>
<li><code>yield</code> - ends the current time slice.</li>
<li><code>write</code> - writes to an open file descriptor.</li>
<li><code>read</code> - reads from an file descriptor. Returns length of read, 0 on EOF.
May blocking - see <code>set_nonblocking</code>.</li>
<li><code>close</code> - closes a file descriptor.</li>
<li><code>dup2</code> - duplicates fd from <code>old</code> to <code>new</code>. <code>new</code> is closed if it already exists.</li>
<li><code>pipe</code> - creates a pipe. fd[0] is read, fd[1] is write.</li>
<li><code>fopen</code> - open file. Not quite POSIX, as it’s non-blocking</li>
<li><code>fork</code> - clones process.
Return value is 0 if child, PID of child if parent, -1 if error.</li>
<li><code>exec</code> - replaces the process with another program. PID is kept.
Stack and FDs (except in/out/err) are destroyed.</li>
<li><code>exit</code> - exits with exit code.</li>
<li><code>wait</code> - waits for a child program to exit, and gives exit code.</li>
<li><code>kill</code> - sends a kill signal to a process. Killed processes will not return an exit code.
<code>signal</code> is not yet implemented.</li>
<li><code>setpriority</code> - set priority of child process.</li>
<li><code>set_nonblocking</code> - is not POSIX, unfortunately. Set pipe non-blocking.</li>
</ul>
<p>The following functions use system calls to provide a higher function:</p>
<ul>
<li><code>popen</code> - opens a process and returns a FD. Uses <code>fork</code>, <code>pipe</code>, <code>exec</code>, and <code>dup2</code>.</li>
<li><code>wait</code>/<code>waitpid</code> - both use the <code>wait</code> syscall.</li>
</ul>
<h3 id="processes" tabindex="-1">Processes <a class="header-anchor" href="#processes">#</a></h3>
<ul>
<li><strong>time slicing</strong> - timer based timer slices.</li>
<li><strong>priority-based</strong> - priority(P) = priority_base(P) + slices_since_last_ran(P)</li>
<li><strong>blocked queue</strong> - for processes waiting for a process or file resource.</li>
<li><strong>process ownership</strong> - processes have a parent, which can kill/wait them.</li>
<li><strong>process groups</strong> - a limited type of process group, where all processes
that share a parent and the parent itself are in a group.</li>
</ul>
<h3 id="files" tabindex="-1">Files <a class="header-anchor" href="#files">#</a></h3>
<p>The kernel allows the use of <strong>file descriptors</strong> to refer to resources. They
are implemented under the hood using function pointers, which means that the main
part of the kernel doesn’t even know what type of file they are. Can be blocking or not.</p>
<p>Types:</p>
<ul>
<li><strong>pipe</strong> - Pointed to by a FD.</li>
<li><strong>in/out/err</strong> - these are “files” too!</li>
<li><strong>filesystem</strong> - Files are limited to 256 bytes with a maximum of 10 files.</li>
</ul>
rubenwardy
Minetest at Freenode Live 2017
2017-10-29T00:00:00.000Z
2017-10-29T00:00:00.000Z
https://blog.rubenwardy.com/2017/10/29/freenode-live/
I ran an exhibit for Minetest at Freenode Live. Our stall had a computer where visitors could play Minetest. We talked about it and handed out leaflets.
<p>Shara and I exhibited on behalf of Minetest at Freenode #Live. Our stall had a
computer where visitors could play Minetest. We talked about it and handed out
leaflets.</p>
<p>If you’d like to support Minetest, then please <a href="https://www.minetest.net/get-involved/">get involved</a>
and consider donating:</p>
<ul>
<li><a href="https://www.minetest.net/get-involved/#donate">celeron55</a></li>
<li><a href="https://rubenwardy.com/donate/">rubenwardy (me)</a></li>
</ul>
<h2 id="photos" tabindex="-1">Photos <a class="header-anchor" href="#photos">#</a></h2>
<div class="row row-wrap gap-4 figure-grid">
<div class="col-md"><figure class="undefined"><a href="/static/img/T3VooHHCPq-500.jpeg"><picture><source type="image/webp" srcset="/static/img/gJH-XJUkED-500.webp 500w"><img alt="Our stall" loading="lazy" decoding="async" src="/static/img/gJH-XJUkED-500.jpeg" width="500" height="281"></picture></a><figcaption>Our stall</figcaption></figure></div><div class="col-md"><figure class="undefined"><a href="/static/img/D0M_23dKTc-500.jpeg"><picture><source type="image/webp" srcset="/static/img/aQCb_RmDrm-500.webp 500w"><img alt="People playing MT" loading="lazy" decoding="async" src="/static/img/aQCb_RmDrm-500.jpeg" width="500" height="281"></picture></a><figcaption>People playing MT</figcaption></figure></div>
</div>
<h2 id="resources" tabindex="-1">Resources <a class="header-anchor" href="#resources">#</a></h2>
<ul>
<li><a href="https://rwdy.uk/freenode-live/">Flip book</a></li>
<li><a href="/static/files/freenode-leaflet.pdf">Resources leaflet</a></li>
</ul>
<h2 id="useful-links" tabindex="-1">Useful Links <a class="header-anchor" href="#useful-links">#</a></h2>
<ul>
<li>General
<ul>
<li><a href="https://www.minetest.net">Minetest Website</a></li>
<li><a href="https://forum.minetest.net">Minetest Forums</a></li>
<li>#minetest on irc.freenode.net</li>
<li><a href="https://wiki.minetest.net/Setting_up_a_server">Setting up a server</a></li>
</ul>
</li>
<li>Modding and Development
<ul>
<li><a href="https://dev.minetest.net">Modding and Developer Wiki</a></li>
<li><a href="https://rubenwardy.com/minetest_modding_book">Minetest Modding Book</a></li>
<li><a href="https://api.minetest.net/">API Reference</a></li>
<li>#minetest-dev on irc.freenode.net</li>
</ul>
</li>
</ul>
<h2 id="licenses-and-credits" tabindex="-1">Licenses and Credits <a class="header-anchor" href="#licenses-and-credits">#</a></h2>
<p>Materials copyright © 2017 to rubenwardy and Shara.
License: <a href="https://creativecommons.org/licenses/by-sa/4.0/">CC-BY-SA 4.0</a></p>
<p>The following are the copyright owners of the screenshots,
not the mods involved.</p>
<ul>
<li>Flipbook Images
<ul>
<li>3D Armor © Stu.</li>
<li>Bags © Shara.</li>
<li>Caverealms Lite © Shara.</li>
<li>CTF © Shara.</li>
<li>Ethereal © Shara.</li>
<li>HomeDecor © Shara.</li>
<li>Lightning © sofar.</li>
<li>Mesecons © unk.</li>
<li>Mobs Redo © Shara.</li>
<li>MoreBlocks © Shara and shivajiva.</li>
<li>Other Worlds © Shara.</li>
<li>Protector © Shara.</li>
<li>Simple Skins © Shara.</li>
<li>SkyBlock © shivajiva.</li>
<li>Snow drift © paramat.</li>
<li>Technic © Mr_Pardison.</li>
<li>Unified Inventory © Shara.</li>
<li>Warp Potions © Shara.</li>
<li>WorldEdit © shivajiva</li>
</ul>
</li>
<li>Leaflet Images
<ul>
<li>Front/back cover photo © Shara, using Ethereal.</li>
<li>Inside left stone © Minetest, from minetest.net.</li>
<li>Small photo left © Shara, from RedCat Creative.</li>
<li>Small photo right © sfan5, from <a href="https://github.com/minetest/minetest.github.io/issues/78#issuecomment-266480046">a Github issue.</a></li>
<li>Inside right water © Shara, from RedCat Creative.</li>
</ul>
</li>
</ul>
<p>Have I missed you out? Please contact me.</p>
rubenwardy
Install Autodesk Maya on Ubuntu 17.04
2017-09-30T00:00:00.000Z
2017-09-30T00:00:00.000Z
https://blog.rubenwardy.com/2017/09/30/install-autodesk-maya-ubuntu-17-04/
Whilst Maya claims to have Linux support, it only officially supports RedHat and CentOS Linux. Here's how to install it on Debian and Ubuntu based systems.
<p>Whilst Maya claims to have Linux support, it only officially supports RedHat
and CentOS Linux. It is, however, still possible to install it on Debian and
Ubuntu based systems with some leg work.</p>
<h2 id="key-points" tabindex="-1">Key points <a class="header-anchor" href="#key-points">#</a></h2>
<p>To make this work, you need to:</p>
<ul>
<li>Install apt dependencies, including fonts.</li>
<li>Manually install the deb for libpng12, an obsolete package that Maya depends on.</li>
<li>Convert the .rpm files to .deb using <a href="https://help.ubuntu.com/community/RPM/AlienHowto">Alien</a>,
and install them separately from the GUI installer.
Please note that the conversion will probably take quite a while.</li>
<li>Add environment variables to stop segfaults and errors.</li>
</ul>
<h2 id="steps" tabindex="-1">Steps <a class="header-anchor" href="#steps">#</a></h2>
<h3 id="1-install-as-root" tabindex="-1">1. Install as Root <a class="header-anchor" href="#1-install-as-root">#</a></h3>
<p>If you have already downloaded maya, then <code>cd</code> to the containing folder.
If you haven’t, don’t worry. This script will download it.</p>
<p>Don’t worry if the Autodesk installer claims the installation was unsuccessful,
it <em>may</em> still have worked.</p>
<p>Run the following script as root:</p>
<pre class="language-bash"><code class="language-bash"><span class="token shebang important">#!/bin/bash</span>
<span class="token comment"># Make sure we’re running with root permissions.</span>
<span class="token keyword">if</span> <span class="token punctuation">[</span> <span class="token variable"><span class="token variable">`</span><span class="token function">whoami</span><span class="token variable">`</span></span> <span class="token operator">!=</span> root <span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Please run this script using sudo"</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Just type “sudo !!”"</span>
<span class="token builtin class-name">exit</span> <span class="token number">1</span>
<span class="token keyword">fi</span>
<span class="token comment"># Check for 64-bit arch</span>
<span class="token keyword">if</span> <span class="token punctuation">[</span>uname <span class="token parameter variable">-m</span> <span class="token operator">!=</span> x86_64<span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Maya will only run on 64-bit linux."</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Please install the 64-bit ubuntu and try again."</span>
<span class="token builtin class-name">exit</span> <span class="token number">1</span>
<span class="token keyword">fi</span>
<span class="token comment"># Detect existing download, else download from scratch</span>
<span class="token keyword">if</span> <span class="token punctuation">[</span> <span class="token parameter variable">-f</span> <span class="token string">"setup"</span> <span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Found existing Maya installer"</span>
<span class="token keyword">else</span>
<span class="token builtin class-name">echo</span> <span class="token string">"No Maya installer in current directory"</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Press [ENTER] to download and extract Maya 2017, or ctrl+C to cancel"</span>
<span class="token builtin class-name">read</span> <span class="token parameter variable">-n</span> <span class="token number">1</span> <span class="token parameter variable">-s</span> <span class="token parameter variable">-r</span> <span class="token parameter variable">-p</span> <span class="token string">""</span>
<span class="token comment">## Create Download Directory</span>
<span class="token function">mkdir</span> <span class="token parameter variable">-p</span> maya2017Install
<span class="token builtin class-name">cd</span> maya2017Install
<span class="token comment">## Download Maya Install Files</span>
<span class="token function">wget</span> https://edutrial.autodesk.com/NET17SWDLD/2017/MAYA/ESD/Autodesk_Maya_2017_EN_JP_ZH_Linux_64bit.tgz
<span class="token function">tar</span> xvf Autodesk_Maya_2017_EN_JP_ZH_Linux_64bit.tgz
<span class="token keyword">fi</span>
<span class="token comment"># Install Dependencies</span>
<span class="token function">wget</span> https://launchpadlibrarian.net/183708483/libxp6_1.0.2-2_amd64.deb
<span class="token function">apt-get</span> <span class="token function">install</span> <span class="token parameter variable">-y</span> libssl1.0.0 gcc libssl-dev libjpeg62 alien csh tcsh libaudiofile-dev libglw1-mesa elfutils libglw1-mesa-dev mesa-utils xfstt ttf-liberation xfonts-100dpi xfonts-75dpi ttf-mscorefonts-installer libfam0 libfam-dev libgstreamer-plugins-base0.10-0
<span class="token comment"># Fix .so libs</span>
<span class="token function">wget</span> https://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
<span class="token function">ln</span> <span class="token parameter variable">-s</span> /usr/lib/x86_64-linux-gnu/libtiff.so /usr/lib/x86_64-linux-gnu/libtiff.so.3
<span class="token function">ln</span> <span class="token parameter variable">-s</span> /usr/lib/x86_64-linux-gnu/libcrypto.so /usr/lib/x86_64-linux-gnu/libcrypto.so.10
<span class="token function">ln</span> <span class="token parameter variable">-s</span> /usr/lib/x86_64-linux-gnu/libssl.so /usr/lib/x86_64-linux-gnu/libssl.so.10
<span class="token comment"># Convert .rpm to .dev</span>
alien <span class="token parameter variable">-cv</span> *.rpm
<span class="token comment"># Install .deb</span>
dpkg <span class="token parameter variable">-i</span> *.deb
<span class="token comment"># Run installer (setup executable)</span>
<span class="token function">mv</span> /usr/bin/rpm /usr/bin/rpm_backup
<span class="token function">ln</span> <span class="token parameter variable">-s</span> /bin/true /usr/bin/rpm
<span class="token function">chmod</span> +x ./setup
./setup
<span class="token function">rm</span> /usr/bin/rpm
<span class="token function">mv</span> /usr/bin/rpm_backup /usr/bin/rpm
<span class="token comment"># Make tmp dir for Maya</span>
<span class="token function">mkdir</span> <span class="token parameter variable">-p</span> /usr/tmp
<span class="token comment"># This is fine, as it's just another /tmp dir (which is 777 too)</span>
<span class="token function">chmod</span> <span class="token number">777</span> /usr/tmp
<span class="token comment"># Maya Camera Modifier Key</span>
gsettings <span class="token builtin class-name">set</span> org.gnome.desktop.wm.preferences mouse-button-modifier <span class="token string">"<Super>"</span>
<span class="token comment"># Ensure that Fonts are Loaded</span>
xset +fp /usr/share/fonts/X11/100dpi/
xset +fp /usr/share/fonts/X11/75dpi/
xset fp rehash
<span class="token builtin class-name">echo</span> <span class="token string">"Maya was installed successfully."</span></code></pre>
<h3 id="2-modifications-for-each-user" tabindex="-1">2. Modifications for each user <a class="header-anchor" href="#2-modifications-for-each-user">#</a></h3>
<p>Then run the following script for each user you want to run Maya:</p>
<pre class="language-bash"><code class="language-bash"><span class="token shebang important">#!/bin/bash</span>
<span class="token keyword">if</span> <span class="token punctuation">[</span> <span class="token variable"><span class="token variable">`</span><span class="token function">whoami</span><span class="token variable">`</span></span> <span class="token operator">==</span> root <span class="token punctuation">]</span><span class="token punctuation">;</span> <span class="token keyword">then</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Please run this script using the user you want to run Maya with."</span>
<span class="token builtin class-name">echo</span> <span class="token string">"I really hope that isn't root!"</span>
<span class="token builtin class-name">exit</span> <span class="token number">1</span>
<span class="token keyword">fi</span>
<span class="token function">mkdir</span> <span class="token parameter variable">-p</span> ~/maya/2017/
<span class="token comment"># Fix Segmentation Fault Error</span>
<span class="token builtin class-name">echo</span> <span class="token string">"MAYA_DISABLE_CIP=1"</span> <span class="token operator">>></span> ~/maya/2017/Maya.env
<span class="token builtin class-name">echo</span> <span class="token string">"LC_ALL=C"</span> <span class="token operator">>></span> ~/maya/2017/Maya.env</code></pre>
<h2 id="troubleshooting" tabindex="-1">Troubleshooting <a class="header-anchor" href="#troubleshooting">#</a></h2>
<p>The above scripts are heavily dependent on your OS version and state,
so you’re likely to have one of the following errors.</p>
<h3 id="segmentation-fault-or-color-management-error-dialog" tabindex="-1">Segmentation Fault or Color Management Error Dialog <a class="header-anchor" href="#segmentation-fault-or-color-management-error-dialog">#</a></h3>
<p>Make sure you’re running Maya with the following environment variables.
You can do this by adding the following lines to <code>~/maya/2017/Maya.env</code>:</p>
<p>MAYA_DISABLE_CIP=1
LC_ALL=C</p>
<h3 id="lib-so-cannot-open-shared-object-file" tabindex="-1">lib*.so.*: cannot open shared object file <a class="header-anchor" href="#lib-so-cannot-open-shared-object-file">#</a></h3>
<p>This means the an compile lib object file cannot be found when running it.
To fix this:</p>
<ul>
<li>
<p>Check for an existing file with the same first segment in <code>/usr/lib/x86_64-linux-gnu</code>.
For example, if it complains about <code>libssl.so.12</code> then look for <code>libssl.so</code> in the
directory.</p>
</li>
<li>
<p>If the file exists, then create a soft symlink like so:</p>
<p>ln -s /usr/lib/x86_64-linux-gnu/EXISTING.so /usr/lib/x86_64-linux-gnu/NOTFOUND.so</p>
<p>For example,</p>
<p>ln -s /usr/lib/x86_64-linux-gnu/libssl.so /usr/lib/x86_64-linux-gnu/libssl.so.12</p>
</li>
</ul>
<h3 id="failed-to-install-maya-in-autodesk-gui" tabindex="-1">Failed to install Maya in Autodesk GUI <a class="header-anchor" href="#failed-to-install-maya-in-autodesk-gui">#</a></h3>
<p>Don’t worry about this, it may have still worked. Check it by running <code>maya</code> -
but make sure to click finish on the installer and run the user script first.</p>
<h2 id="references-and-sources" tabindex="-1">References and Sources <a class="header-anchor" href="#references-and-sources">#</a></h2>
<ul>
<li>Script modified from <a href="https://gist.github.com/borgfriend/b83467639cb8039dc79974bf780a4994">https://gist.github.com/borgfriend/b83467639cb8039dc79974bf780a4994</a>.</li>
<li><a href="https://forums.autodesk.com/t5/maya-forum/render-crash-on-linux/m-p/5608552">Render crash on Linux</a>, Autodesk forums.</li>
<li>libpng12 on packages.ubuntu.com</li>
</ul>
rubenwardy
RVWP: Multiplayer Topdown Sandbox Game in C++
2017-08-13T00:00:00.000Z
2017-08-13T00:00:00.000Z
https://blog.rubenwardy.com/2017/08/13/multiplayer-topdown-sandbox-game/
The game is a top-down sandbox with multiplayer support. I'm aiming towards a city-based game, where players can wander around a procedurally generated city.
<p>For the last two years, I have been working on a
<a href="https://rubenwardy.com/rvwp/">very ambitious game</a>.
The game is a top-down sandbox with multiplayer support. I’m aiming towards a
city-based game, where players can wander around a procedurally generated city.
One of the main reasons I started creating this game is to learn about multiplayer
networking at a low level - client-side prediction, server-side reconcilliation,
cheat preventation, and reducing the visual effect of latency.</p>
<figure class="undefined"><a href="/static/img/Tg1WyD7YGt-450.png"><picture><source type="image/webp" srcset="/static/img/WHovSDoaCg-450.webp 450w"><img alt="A recent video demonstrating the current state of the game." loading="lazy" decoding="async" src="/static/img/WHovSDoaCg-450.png" width="450" height="253"></picture></a><figcaption>A recent video demonstrating the current state of the game.</figcaption></figure>
<p>The project is written in C++ using SFML. It’s the biggest project I’ve ever created,
with 9,000 lines of code and 64 classes.</p>
<p>I decided to split the project into several milestones, each on iterating on the
last and moving it towards a more complete game.</p>
<h3 id="features" tabindex="-1">Features <a class="header-anchor" href="#features">#</a></h3>
<ul>
<li>Infinite map</li>
<li>Multiplayer</li>
<li>Block placement</li>
<li>Multiple map z-levels (can go up and down stairs)</li>
<li>Doors and stairs</li>
<li>Player chat</li>
<li>Game controller support (only tested with Xbox 360 controller)</li>
<li>Inventory and dual wielding</li>
<li>Weapons (guns and melee)</li>
<li>Main menu</li>
</ul>
<h3 id="milestone-1-foundations" tabindex="-1">Milestone 1 - Foundations <a class="header-anchor" href="#milestone-1-foundations">#</a></h3>
<p>MS1 adds a foundation for future work, including: networking,
MVC based architecture, basic entities, raytracer, materials to specify how to
draw something, collison, and loads more.</p>
<p>I started off by writing the basic world model and spatial paritioning. I then
wrote the client/server handshake.</p>
<figure class="undefined"><a href="/static/img/UKPZ4c3uWJ-1024.png"><picture><source type="image/webp" srcset="/static/img/O6lk_qGiac-540.webp 540w"><img alt="Just some basic code with socket communication." loading="lazy" decoding="async" src="/static/img/O6lk_qGiac-540.png" width="540" height="293"></picture></a><figcaption>Just some basic code with socket communication.</figcaption></figure>
<p>After that, I added rendering so I could see the world, and start adding
actual game features.</p>
<figure class="undefined"><a href="/static/img/zdu7QGC21q-1024.png"><picture><source type="image/webp" srcset="/static/img/_PY7dxDGIB-540.webp 540w"><img alt="The first ever screenshot, just after adding rendering." loading="lazy" decoding="async" src="/static/img/_PY7dxDGIB-540.png" width="540" height="343"></picture></a><figcaption>The first ever screenshot, just after adding rendering.</figcaption></figure>
<p>The next step is to add player movement and entity synchronisation, and then
node placement:</p>
<figure class="undefined"><a href="/static/img/CRyTOhcDFq-1024.png"><picture><source type="image/webp" srcset="/static/img/CoJoNnbndZ-540.webp 540w"><img alt="Player movement and node placement." loading="lazy" decoding="async" src="/static/img/CoJoNnbndZ-540.png" width="540" height="394"></picture></a><figcaption>Player movement and node placement.</figcaption></figure>
<p>I also decided to switch to Kenney’s texture pack during this time.</p>
<figure class="undefined"><a href="/static/img/l-AAtVi8-m-1024.png"><picture><source type="image/webp" srcset="/static/img/8KYRfzTZjx-540.webp 540w"><img alt="Switched to using one of Kenney's texture packs." loading="lazy" decoding="async" src="/static/img/8KYRfzTZjx-540.png" width="540" height="352"></picture></a><figcaption>Switched to using one of Kenney's texture packs.</figcaption></figure>
<h3 id="milestone-2-and-3-doors-entity-damage-and-interaction" tabindex="-1">Milestone 2 and 3 - Doors, Entity Damage, and Interaction <a class="header-anchor" href="#milestone-2-and-3-doors-entity-damage-and-interaction">#</a></h3>
<p>Adds the ability to open doors, highlight entities, and damage them.</p>
<h3 id="milestone-4-gui" tabindex="-1">Milestone 4 - GUI <a class="header-anchor" href="#milestone-4-gui">#</a></h3>
<p>Next I needed a way to interact with the inventory, so I added an MVC framework
to switch between windows and the game, and an inventory window to use the inventory.</p>
<figure class="undefined"><a href="/static/img/gCsWYpxvKw-779.jpeg"><picture><source type="image/webp" srcset="/static/img/1kIDr6cqm1-540.webp 540w"><img alt="Inventory GUI added, and the inv item position synchronisation." loading="lazy" decoding="async" src="/static/img/1kIDr6cqm1-540.jpeg" width="540" height="363"></picture></a><figcaption>Inventory GUI added, and the inv item position synchronisation.</figcaption></figure>
<p>I also added the ability for entities and players to die, and for players to be
respawned.</p>
<h3 id="milestone-5-weapons-and-tool-specs" tabindex="-1">Milestone 5 - Weapons and Tool Specs <a class="header-anchor" href="#milestone-5-weapons-and-tool-specs">#</a></h3>
<p>I allowed items to provide a ‘tool spec’. A tool spec is a struct which
specifies what functionalities an item has - for example, whether it can
damage tiles and entities, whether it can shoot bullets, and how much damage it
gives.</p>
<h3 id="milestone-6-playability" tabindex="-1">Milestone 6 - Playability <a class="header-anchor" href="#milestone-6-playability">#</a></h3>
<p>This was quite a big milestone. I added the following features:</p>
<ul>
<li>Support for multiple Z-levels (ie: going up and down stairs).</li>
<li>Infinite map by adding chunk loading, and keeping track of whether a player knows about a chunk.</li>
<li>A main menu with the ability to type in a server URL and port.</li>
<li>Chat.</li>
<li>Xbox controller support.</li>
<li>Visual bullets using particles.</li>
</ul>
<figure class="undefined"><a href="/static/img/zf-Dk4eMTR-1200.jpeg"><picture><source type="image/webp" srcset="/static/img/W4cVCdy-O3-540.webp 540w"><img alt="Controller support." loading="lazy" decoding="async" src="/static/img/W4cVCdy-O3-540.jpeg" width="540" height="303"></picture></a><figcaption>Controller support.</figcaption></figure>
<h3 id="where-now" tabindex="-1">Where now? <a class="header-anchor" href="#where-now">#</a></h3>
<p>Since the last milestone, I have been working on increasing code coverage.
The world logic is now at 97% covered, however the client (rendering, input, packet handling)
and the server (packet handling, profiles, interaction logic) still need tests.</p>
<p>The client and server are both designed in an MVP style, which means that it
shouldn’t be too hard to isolate and test them using dependency injection,
which is something I’d like to get around to.</p>
<p>As for actual game progress, I’d like to work on making the server more
scalable. I’m thinking of implementing a distributed architecture with
chunk workers - processes that assigned to be the “truth” of a group
of chunks, to update them based on logic, and to respond to any requests
relating to them.</p>
<p>Another thing I need to investigate is lighting, as it’s good example of an
algorithm that needs to propogate through chunk boundaries and so will help
with making sure my workers can handle it.</p>
<p class="alert is-secondary my-3">
Part of a series of blog posts about my game,
<a href="https://rubenwardy.com/rvwp/">RVWP</a>
</p>
rubenwardy
Mingw-w64 and CMake: unrecognised option -rdynamic on Ubuntu
2017-03-07T00:00:00.000Z
2017-03-07T00:00:00.000Z
https://blog.rubenwardy.com/2017/03/07/mingw-w64-cmake-unrecognised-option-rdynamic-ubuntu/
How to solve "error: unrecognized option '-rdynamic'" on Ubuntu.
<p>I had an issue where CMake was failing on a compiler test with the
following error:</p>
<pre><code>error: unrecognized option '-rdynamic'
</code></pre>
<p>The problem was that CMake caches settings such as compiler flags
in CMakeCache.txt, so you need to clear the cache when changing the
platform. Do this by deleting CMakeFiles and CMakeCache.txt</p>
rubenwardy
OpenCV: compile images (Mats) into a strip
2016-09-30T00:00:00.000Z
2016-09-30T00:00:00.000Z
https://blog.rubenwardy.com/2016/09/30/opencv-combine-images-into-strip/
I created a very short C++ snippet to accumulate a series of Mats into a single Mat strip.
<p>I created a very short C++ snippet to accumulate a series of <code>Mat</code>s into
a single <code>Mat</code> strip. It works like <code>acc = acc + m</code> - a new mat is added to the
accumulator each time, then stored in the accumulator again.</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">pragma</span> <span class="token expression">once</span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string"><opencv2/opencv.hpp></span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string"><opencv2/core/core.hpp></span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string"><opencv2/imgcodecs.hpp></span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string"><assert.h></span></span>
<span class="token keyword">using</span> <span class="token keyword">namespace</span> cv<span class="token punctuation">;</span>
<span class="token keyword">class</span> <span class="token class-name">MatStrip</span>
<span class="token punctuation">{</span>
<span class="token keyword">public</span><span class="token operator">:</span>
Mat current<span class="token punctuation">;</span>
<span class="token keyword">float</span> scale<span class="token punctuation">;</span>
<span class="token function">MatStrip</span><span class="token punctuation">(</span>Mat start<span class="token punctuation">,</span> <span class="token keyword">float</span> scale<span class="token operator">=</span><span class="token number">0.3f</span><span class="token punctuation">)</span><span class="token operator">:</span>
<span class="token function">scale</span><span class="token punctuation">(</span>scale<span class="token punctuation">)</span>
<span class="token punctuation">{</span>
<span class="token function">resize</span><span class="token punctuation">(</span>start<span class="token punctuation">,</span> current<span class="token punctuation">,</span> <span class="token function">Size</span><span class="token punctuation">(</span>start<span class="token punctuation">.</span>cols <span class="token operator">*</span> scale<span class="token punctuation">,</span> start<span class="token punctuation">.</span>rows <span class="token operator">*</span> scale<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token number">1.0</span><span class="token punctuation">,</span> INTER_CUBIC<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">void</span> <span class="token function">add</span><span class="token punctuation">(</span><span class="token keyword">const</span> Mat <span class="token operator">&</span>in_o<span class="token punctuation">)</span>
<span class="token punctuation">{</span>
Mat in<span class="token punctuation">;</span>
<span class="token function">resize</span><span class="token punctuation">(</span>in_o<span class="token punctuation">,</span> in<span class="token punctuation">,</span> <span class="token function">Size</span><span class="token punctuation">(</span>in_o<span class="token punctuation">.</span>cols <span class="token operator">*</span> scale<span class="token punctuation">,</span> in_o<span class="token punctuation">.</span>rows <span class="token operator">*</span> scale<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token number">1.0</span><span class="token punctuation">,</span> INTER_CUBIC<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token function">assert</span><span class="token punctuation">(</span>in<span class="token punctuation">.</span><span class="token function">type</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">==</span> current<span class="token punctuation">.</span><span class="token function">type</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
Size sz1 <span class="token operator">=</span> current<span class="token punctuation">.</span><span class="token function">size</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
Size sz2 <span class="token operator">=</span> in<span class="token punctuation">.</span><span class="token function">size</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
Mat <span class="token function">im3</span><span class="token punctuation">(</span>sz1<span class="token punctuation">.</span>height<span class="token punctuation">,</span> sz1<span class="token punctuation">.</span>width<span class="token operator">+</span>sz2<span class="token punctuation">.</span>width<span class="token punctuation">,</span> current<span class="token punctuation">.</span><span class="token function">type</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
Mat <span class="token function">left</span><span class="token punctuation">(</span>im3<span class="token punctuation">,</span> <span class="token function">Rect</span><span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> sz1<span class="token punctuation">.</span>width<span class="token punctuation">,</span> sz1<span class="token punctuation">.</span>height<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
current<span class="token punctuation">.</span><span class="token function">copyTo</span><span class="token punctuation">(</span>left<span class="token punctuation">)</span><span class="token punctuation">;</span>
Mat <span class="token function">right</span><span class="token punctuation">(</span>im3<span class="token punctuation">,</span> <span class="token function">Rect</span><span class="token punctuation">(</span>sz1<span class="token punctuation">.</span>width<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> sz2<span class="token punctuation">.</span>width<span class="token punctuation">,</span> sz2<span class="token punctuation">.</span>height<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
in<span class="token punctuation">.</span><span class="token function">copyTo</span><span class="token punctuation">(</span>right<span class="token punctuation">)</span><span class="token punctuation">;</span>
current <span class="token operator">=</span> im3<span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre>
rubenwardy
How to emit a single particle using SFML's Thor Particle System
2016-09-12T00:00:00.000Z
2016-09-12T00:00:00.000Z
https://blog.rubenwardy.com/2016/09/12/sfml-thor-emit-a-single-particle/
You must use a particle emitter to create particles, however this doesn't mean it's impossible to create single particles on command.
<p>Usecase: shells dropping in sync with firing, fake bullets, etc</p>
<p>You must use a particle emitter to create particles, however this doesn’t mean
it’s impossible to create single particles on command. You can create a particle
emitter which simply adds particles from a queue to the system</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">pragma</span> <span class="token expression">once</span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string"><SFML/Graphics.hpp></span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string"><Thor/Math.hpp></span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string"><Thor/Graphics.hpp></span></span>
<span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string"><Thor/Particles.hpp></span></span>
<span class="token keyword">namespace</span> sfext <span class="token punctuation">{</span>
<span class="token keyword">class</span> <span class="token class-name">SingleParticleEmitter</span>
<span class="token punctuation">{</span>
std<span class="token double-colon punctuation">::</span>vector<span class="token operator"><</span>thor<span class="token double-colon punctuation">::</span>Particle<span class="token operator">></span> <span class="token operator">*</span>particles<span class="token punctuation">;</span>
<span class="token keyword">public</span><span class="token operator">:</span>
<span class="token function">SingleParticleEmitter</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token function">SingleParticleEmitter</span><span class="token punctuation">(</span><span class="token keyword">const</span> SingleParticleEmitter<span class="token operator">&</span> that<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">void</span> <span class="token keyword">operator</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">(</span>thor<span class="token double-colon punctuation">::</span>EmissionInterface<span class="token operator">&</span> system<span class="token punctuation">,</span> sf<span class="token double-colon punctuation">::</span>Time dt<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">void</span> <span class="token function">push</span><span class="token punctuation">(</span>thor<span class="token double-colon punctuation">::</span>Particle particle<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">void</span> <span class="token function">push</span><span class="token punctuation">(</span>sf<span class="token double-colon punctuation">::</span>Vector2f position<span class="token punctuation">,</span> sf<span class="token double-colon punctuation">::</span>Vector2f velocity<span class="token punctuation">,</span> <span class="token keyword">float</span> lifetime<span class="token punctuation">,</span> <span class="token keyword">float</span> rot<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token keyword">float</span> rotsp<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">;</span> <span class="token comment">// end namesapce sfext</span></code></pre>
<p>The above allows you to use the push functions to emit particles.
Here is an example of it in use:</p>
<pre class="language-cpp"><code class="language-cpp">SingleParticleEmitter single_emitter<span class="token punctuation">;</span>
thor<span class="token double-colon punctuation">::</span>ParticleSystem particle_system<span class="token punctuation">;</span>
particle_system<span class="token punctuation">.</span><span class="token function">addEmitter</span><span class="token punctuation">(</span>single_emitter<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">// When you need to emit a shell:</span>
single_emitter<span class="token punctuation">.</span><span class="token function">push</span><span class="token punctuation">(</span>origin<span class="token punctuation">,</span> velocity<span class="token punctuation">,</span> <span class="token number">1.0f</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">// lifetime of the particle in seconds --^</span></code></pre>
<p>Here is the cpp file:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token macro property"><span class="token directive-hash">#</span><span class="token directive keyword">include</span> <span class="token string">"singleparticleemitter.hpp"</span></span>
<span class="token keyword">using</span> <span class="token keyword">namespace</span> sfext<span class="token punctuation">;</span>
<span class="token class-name">SingleParticleEmitter</span><span class="token double-colon punctuation">::</span><span class="token function">SingleParticleEmitter</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token punctuation">{</span>
particles <span class="token operator">=</span> <span class="token keyword">new</span> std<span class="token double-colon punctuation">::</span><span class="token generic-function"><span class="token function">vector</span><span class="token generic class-name"><span class="token operator"><</span>thor<span class="token double-colon punctuation">::</span>Particle<span class="token operator">></span></span></span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token class-name">SingleParticleEmitter</span><span class="token double-colon punctuation">::</span><span class="token function">SingleParticleEmitter</span><span class="token punctuation">(</span><span class="token keyword">const</span> SingleParticleEmitter<span class="token operator">&</span> that<span class="token punctuation">)</span>
<span class="token punctuation">{</span>
particles <span class="token operator">=</span> that<span class="token punctuation">.</span>particles<span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">void</span> <span class="token class-name">SingleParticleEmitter</span><span class="token double-colon punctuation">::</span><span class="token keyword">operator</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">(</span>thor<span class="token double-colon punctuation">::</span>EmissionInterface<span class="token operator">&</span> system<span class="token punctuation">,</span> sf<span class="token double-colon punctuation">::</span>Time dt<span class="token punctuation">)</span>
<span class="token punctuation">{</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">auto</span> particle <span class="token operator">:</span> <span class="token operator">*</span>particles<span class="token punctuation">)</span> <span class="token punctuation">{</span>
system<span class="token punctuation">.</span><span class="token function">emitParticle</span><span class="token punctuation">(</span>particle<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
particles<span class="token operator">-></span><span class="token function">clear</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">void</span> <span class="token class-name">SingleParticleEmitter</span><span class="token double-colon punctuation">::</span><span class="token function">push</span><span class="token punctuation">(</span>thor<span class="token double-colon punctuation">::</span>Particle particle<span class="token punctuation">)</span>
<span class="token punctuation">{</span>
particles<span class="token operator">-></span><span class="token function">push_back</span><span class="token punctuation">(</span>particle<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">void</span> <span class="token class-name">SingleParticleEmitter</span><span class="token double-colon punctuation">::</span><span class="token function">push</span><span class="token punctuation">(</span>sf<span class="token double-colon punctuation">::</span>Vector2f position<span class="token punctuation">,</span> sf<span class="token double-colon punctuation">::</span>Vector2f velocity<span class="token punctuation">,</span> <span class="token keyword">float</span> lifetime<span class="token punctuation">,</span> <span class="token keyword">float</span> rot<span class="token punctuation">,</span> <span class="token keyword">float</span> rotsp<span class="token punctuation">)</span>
<span class="token punctuation">{</span>
thor<span class="token double-colon punctuation">::</span>Particle <span class="token function">particle</span><span class="token punctuation">(</span>sf<span class="token double-colon punctuation">::</span><span class="token function">seconds</span><span class="token punctuation">(</span>lifetime<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
particle<span class="token punctuation">.</span>position <span class="token operator">=</span> position<span class="token punctuation">;</span>
particle<span class="token punctuation">.</span>velocity <span class="token operator">=</span> velocity<span class="token punctuation">;</span>
particle<span class="token punctuation">.</span>rotation <span class="token operator">=</span> rot<span class="token punctuation">;</span>
particle<span class="token punctuation">.</span>rotationSpeed <span class="token operator">=</span> rotsp<span class="token punctuation">;</span>
particle<span class="token punctuation">.</span>scale <span class="token operator">=</span> sf<span class="token double-colon punctuation">::</span><span class="token function">Vector2f</span><span class="token punctuation">(</span><span class="token number">1.f</span><span class="token punctuation">,</span> <span class="token number">1.f</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
particle<span class="token punctuation">.</span>color <span class="token operator">=</span> sf<span class="token double-colon punctuation">::</span>Color<span class="token double-colon punctuation">::</span>White<span class="token punctuation">;</span>
particle<span class="token punctuation">.</span>textureIndex <span class="token operator">=</span> <span class="token number">0u</span><span class="token punctuation">;</span>
<span class="token function">push</span><span class="token punctuation">(</span>particle<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
</code></pre>
rubenwardy
How to center a SFGUI window (sfg::Window)
2016-08-16T00:00:00.000Z
2016-08-16T00:00:00.000Z
https://blog.rubenwardy.com/2016/08/16/sfgui-center-window-container/
Simply get the SFGUI window size using GetAllocation, the sfml window size using getSize, then do some arithmetic.
<p>Simply get the SFGUI window size using GetAllocation,
the sfml window size using getSize, then do this arithmetic:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">auto</span> window <span class="token operator">=</span> sfg<span class="token double-colon punctuation">::</span><span class="token class-name">Window</span><span class="token double-colon punctuation">::</span><span class="token function">Create</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">auto</span> win_rect <span class="token operator">=</span> window<span class="token operator">-></span><span class="token function">GetAllocation</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
sf<span class="token double-colon punctuation">::</span>Vector2f <span class="token function">size</span><span class="token punctuation">(</span>win_rect<span class="token punctuation">.</span>width<span class="token punctuation">,</span> win_rect<span class="token punctuation">.</span>height<span class="token punctuation">)</span><span class="token punctuation">;</span>
window<span class="token operator">-></span><span class="token function">SetPosition</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">(</span>sf<span class="token double-colon punctuation">::</span>Vector2f<span class="token punctuation">)</span>rwindow<span class="token operator">-></span><span class="token function">getSize</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">-</span> size<span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token number">2.0f</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
rubenwardy
Setting up a MonoDevelop solution and project for RimWorld modding
2016-07-21T00:00:00.000Z
2016-07-21T00:00:00.000Z
https://blog.rubenwardy.com/2016/07/21/rimworld-setup-monodevelop/
This tutorial will show you how to create a new MonoDevelop solution and project, and how to properly connect it to RimWorld.
<p>This tutorial will show you how to create a new MonoDevelop solution and project,
and how to properly connect it to RimWorld. You’ll need to have an installation
of Mono that supports .NET 3.5 -
<a href="/2016/07/20/rimworld-install-monodevelop-with-dot-net-3.5/">here is a tutorial on how to install it</a>.</p>
<ul>
<li>Open Monodevelop (<a href="/2016/07/20/rimworld-install-monodevelop-with-dot-net-3.5/">see this guide to install with .NET 3.5 support</a>)</li>
<li>Create a new solution
<ul>
<li>Click File > New > Solution</li>
<li>Select the C# tab</li>
<li>Select Library (C#)</li>
<li>Enter your mod’s name into the <code>name</code> field.</li>
<li>Set the location to “Source” in your mod’s directory. Eg, RimWorld/mods/ModName/Source.</li>
<li><em>Optional:</em> Untick “Create a directory for solution”/“Create a project within the solution directory”.</li>
</ul>
</li>
<li>Add references to the RimWorld DLL
<ul>
<li>In the left panel, right click ModName > ModName > References. Select Edit References.</li>
<li>Go to the .NET Assembly tab</li>
<li>Click browse and go to RimWorld/RimWorldLinux_Data/Managed.
Select Assembly-CSharp.dll and UnityEngine.dll</li>
<li>Click open and then exit the edit references dialog.</li>
<li>For each reference, right click and decheck “local copy”.</li>
</ul>
</li>
<li>Set .NET framework to .NET 3.5
<ul>
<li>Right click project (ModName > Modname) in the left panel and select options</li>
<li>Go to Build > General</li>
<li>Select Mono / .NET 3.5 in the target framework</li>
</ul>
</li>
<li>Set output directory
<ul>
<li>Go to Build > Output</li>
<li>Set the output folder to RimWorld/mods/ModName/Assemblies.</li>
</ul>
</li>
<li>Profit!</li>
</ul>
<p>Some of this information was learned from
<a href="https://rimworldwiki.com/wiki/Modding_Tutorials/Setting_up_a_solution">this wiki page</a>.</p>
rubenwardy
Install Monodevelop to build for .NET 3.5 (using Mono 3.14) - RimWorld
2016-07-20T00:00:00.000Z
2016-07-20T00:00:00.000Z
https://blog.rubenwardy.com/2016/07/20/rimworld-install-monodevelop-with-dot-net-3.5/
This tutorial will show you how to install Mono and Monodevelop in order to develop .NET 3.5 projects.
<p>This tutorial will show you how to install Mono and Monodevelop in order to
develop .NET 3.5 projects. This is useful when writing C# assembly mods
for Unity engine based games, such as RimWorld, as they tend to require
.NET 3.5.</p>
<p>If you don’t use .NET 3.5, you may get errors including but not limited to:</p>
<p>System.TypeLoadException: Could not load type ‘System.Action’ from assembly ‘ModName’.</p>
<p>That error is commonly thrown when versions such as .NET or .DLLs mismatch.</p>
<p>Mono 4.0 and later <a href="https://www.mono-project.com/docs/about-mono/releases/4.0.0/#dropped-support-for-old-frameworks">don’t support .NET 3.5</a>.
Therefore, we’ll need to install an older version.</p>
<p>I couldn’t find many posts about this, doesn’t seem that many people develop RimWorld on Linux.</p>
<p>After this tutorial, MonoDevelop will support the following target frameworks:
Mono / .NET 2.0, Mono / .NET 3.0, .NET 3.5 Client Profile, Mono / .NET 3.5,
.NET 4.0 Client Profile, Mono / .NET 4.0, Mono / .NET 4.5, .NET Framework 4.5.1</p>
<h2 id="uninstall-any-installed-mono-versions" tabindex="-1">Uninstall any installed Mono versions <a class="header-anchor" href="#uninstall-any-installed-mono-versions">#</a></h2>
<p>Unfortunately I do not know of a way to keep multiple versions of Mono and
Monodevelop installed at a time, so you’ll need to uninstall them:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">sudo</span> <span class="token function">apt-get</span> remove mono-complete mono-devel monodevelop
<span class="token function">sudo</span> <span class="token function">apt</span> autoremove</code></pre>
<p>Now check that mono is not installed:</p>
<pre class="language-bash"><code class="language-bash">mono <span class="token parameter variable">--version</span>
<span class="token comment"># you should get something like command not found</span></code></pre>
<h2 id="add-ppa-for-mono-3-12-x" tabindex="-1">Add PPA for Mono 3.12.x <a class="header-anchor" href="#add-ppa-for-mono-3-12-x">#</a></h2>
<p>We need to add a snapshot PPA, as described
<a href="https://www.mono-project.com/docs/getting-started/install/linux/#accessing-older-releases">in the mono-project docs</a>.</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">sudo</span> apt-key adv <span class="token parameter variable">--keyserver</span> hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
<span class="token builtin class-name">echo</span> <span class="token string">"deb https://download.mono-project.com/repo/debian wheezy/snapshots/3.12.0 main"</span> <span class="token operator">|</span> <span class="token function">sudo</span> <span class="token function">tee</span> /etc/apt/sources.list.d/mono-xamarin.list
<span class="token function">sudo</span> <span class="token function">apt-get</span> update</code></pre>
<h2 id="install-mono-and-monodevelop" tabindex="-1">Install Mono and Monodevelop <a class="header-anchor" href="#install-mono-and-monodevelop">#</a></h2>
<p>Now we need to install Mono 3.14 and Monodevelop 5.7. We also need to lock the
versions of the installed packages, to stop apt-get upgrade from installing Mono 4.0+
and reversing our efforts. We use apt-mark hold to do this.</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">sudo</span> <span class="token function">apt-get</span> <span class="token function">install</span> build-essential cli-common libgtk2.0-cil-dev libglade2.0-cil-dev libgnome2.0-cil-dev libgconf2.0-cil-dev
<span class="token function">sudo</span> <span class="token function">apt-get</span> <span class="token function">install</span> <span class="token parameter variable">-t</span> wheezy/snapshots/3.12.0 mono-complete
<span class="token function">sudo</span> apt-mark hold mono-complete
<span class="token function">sudo</span> apt-mark hold mono-devel
<span class="token function">sudo</span> apt-mark hold libmono*
<span class="token function">sudo</span> <span class="token function">apt-get</span> <span class="token function">install</span> <span class="token parameter variable">-t</span> wheezy/snapshots/3.12.0 monodevelop
<span class="token function">sudo</span> apt-mark hold monodevelop</code></pre>
<p>For more information, see <a href="https://superuser.com/questions/124174/how-can-i-specify-the-repository-from-which-a-package-will-be-installed-emacs">this thread about installing from a specific repository</a>
and <a href="https://askubuntu.com/questions/18654/how-to-prevent-updating-of-a-specific-package">this thread about locking package versions</a>.</p>
<h2 id="profit" tabindex="-1">Profit! <a class="header-anchor" href="#profit">#</a></h2>
<pre class="language-bash"><code class="language-bash">mono <span class="token parameter variable">--version</span></code></pre>
<p>Should give something like:</p>
<p>Mono JIT compiler version 3.12.1 (tarball Fri Mar 6 19:12:47 UTC 2015)
Copyright © 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com</p>
<p>And monodevelop should be in the start menu.
Note that you may need to restart your computer, depending on your window manager.</p>
<p>Next: <a href="/2016/07/21/rimworld-setup-monodevelop/">Setting up a MonoDevelop solution and project for RimWorld modding</a>.</p>
<h2 id="appendix" tabindex="-1">Appendix <a class="header-anchor" href="#appendix">#</a></h2>
<p>Here is an error you may get when implementing a JobDriver using .NET 4.0+</p>
<p>Exception in SetupToil(pawn=Kimmy, job=Accident): System.TypeLoadException: Could not load type ‘System.Action’ from assembly ‘toilets’.
at Verse.AI.JobDriver.SetupToils()
Verse.Log:Error(string)
Verse.AI.JobDriver:SetupToil()
Verse.AI.Pawn_JobTracker:StatJob(Job, JobCondition, ThinkNode, Boolean, Boolean, ThinkTreeDef)
Verse.AI.Pawn_JobTracker:TryFindAndStartJob()
Verse.AI.Pawn_JobTracter:EndCurrentJob(JobCondition)</p>
rubenwardy
Move private repos from Bitbucket to GitHub
2016-06-02T00:00:00.000Z
2016-06-02T00:00:00.000Z
https://blog.rubenwardy.com/2016/06/02/move-private-repos-from-bitbucket-to-github/
Now that Github supports unlimited private repos in all plans, you might as well keep things all together
<p>Now that Github supports unlimited private repos in all plans, you might as well
keep things all together (although definitely have backups elsewhere incase
Github is DDOS’d again, dies or goes evil). Simply change “rubenwardy” to your
username and “XXXX” to a
<a href="https://github.com/settings/tokens">personal access token</a> with “repo” checked.
Also make sure you have an <a href="https://github.com/settings/keys">SSH key</a> for Github.</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># Clone from bitbucket</span>
<span class="token function">git</span> clone git@github.com:samkuehn/bitbucket-backup.git
<span class="token function">mkdir</span> bk
<span class="token builtin class-name">cd</span> bitbucket-backup
./bitbucket-backup <span class="token parameter variable">-u</span> rubenwardy <span class="token parameter variable">--mirror</span> <span class="token parameter variable">-l</span> <span class="token punctuation">..</span>/bk
<span class="token comment"># Upload to GitHub</span>
<span class="token builtin class-name">cd</span> <span class="token punctuation">..</span>/bk
<span class="token keyword">for</span> <span class="token for-or-select variable">D</span> <span class="token keyword">in</span> *<span class="token punctuation">;</span>
<span class="token keyword">do</span>
<span class="token builtin class-name">echo</span> <span class="token variable">$D</span>
<span class="token builtin class-name">cd</span> <span class="token variable">$D</span>
<span class="token assign-left variable">data</span><span class="token operator">=</span><span class="token string">"{<span class="token entity" title="\"">\"</span>name<span class="token entity" title="\"">\"</span>: <span class="token entity" title="\"">\"</span><span class="token variable">$D</span><span class="token entity" title="\"">\"</span>, <span class="token entity" title="\"">\"</span>auto_init<span class="token entity" title="\"">\"</span>: false, <span class="token entity" title="\"">\"</span>private<span class="token entity" title="\"">\"</span>: true }"</span>
<span class="token function">curl</span> <span class="token parameter variable">-i</span> <span class="token parameter variable">-H</span> <span class="token string">'Authorization: token XXXX'</span> <span class="token parameter variable">-d</span> <span class="token string">"<span class="token variable">$data</span>"</span> https://api.github.com/user/repos
<span class="token function">git</span> push <span class="token parameter variable">--mirror</span> git@github.com:rubenwardy/<span class="token variable">$D</span>.git
<span class="token builtin class-name">cd</span> <span class="token punctuation">..</span>/
<span class="token keyword">done</span>
<span class="token builtin class-name">cd</span> <span class="token punctuation">..</span>/</code></pre>
rubenwardy
Xubuntu / Xfce sync whisker menu favourites
2016-05-24T00:00:00.000Z
2016-05-24T00:00:00.000Z
https://blog.rubenwardy.com/2016/05/24/xubuntu-xfce-sync-whisker-menu-favourites/
When using multiple Xfce panels, it's annoying that the favourites aren't synchronised. Here's a script to fix that
<p>I use a dual monitor setup. For each monitor I have a panel, and each panel has
a whisker menu. I found that the favourites section of each of these panels
is not synchronised. To fix this, I wrote a simple script.</p>
<p>First, you have to decide which whisker config to keep.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="c"><span class="n">$ cd ~/.config/xfce4/panel/</span>
<span class="n">$ ls</span>
<span class="c1">whiskermenu-1.rc whiskermenu-9.rc</span></code></pre></figure>
<p>The favourites menu is stored at the top of each file, like this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="c"><span class="n">$ head -n1 whiskermenu-1.rc</span>
<span class="c1">favorites=firefox.desktop,google-chrome.desktop,hexchat.desktop,exo-file-manager.desktop,libreoffice-writer.desktop,libreoffice-calc.desktop,exo-terminal-emulator.desktop</span></code></pre></figure>
<p>Find the one you want to keep. From this point on,
replace <code>whiskermenu-1.rc</code> with the config you want to keep and
<code>whiskermenu-9.rc</code> with the one you want to ditch.</p>
<p>The script is as simple as this:</p>
<pre class="language-bash"><code class="language-bash"><span class="token shebang important">#! /bin/bash</span>
<span class="token function">rm</span> <span class="token parameter variable">-f</span> ~/.config/xfce4/panel/whiskermenu-9.rc
<span class="token function">cp</span> ~/.config/xfce4/panel/whiskermenu-1.rc ~/.config/xfce4/panel/whiskermenu-9.rc
xfce4-panel <span class="token parameter variable">-r</span></code></pre>
<p>If you save that to /usr/local/bin/syncwhisker, then you can use “syncwhisker”
in the terminal to keep your menus the same.</p>
rubenwardy
Dual boot Ubuntu on ASUS X555LA UEFI laptop alongside Windows 10
2016-05-21T00:00:00.000Z
2016-05-21T00:00:00.000Z
https://blog.rubenwardy.com/2016/05/21/dual-boot-ubuntu-on-asus-x555la-uefi-laptop/
I recently successfully dual booted Ubuntu on an ASUS X555LA laptop, the X555LAB variety.
<p>I recently successfully dual booted Ubuntu on an ASUS X555LA laptop,
the X555LAB variety. This guide will work on most Ubuntu derivatives,
I installed Xubuntu using these steps. The only things that are different
between them are window managers and preinstalled software.</p>
<p>Specs:</p>
<blockquote>
<p>Intel® Core™ i5-5200U CPU @ 2.20GHz, max: 2.70GHz<br>
Intel® HD Graphics 5500<br>
8GB RAM, 1TB HDD<br>
1920x1080 screen, 2xUSB3.0, 1xUSB2.0</p>
</blockquote>
<p>Here’s a step by step guide to what I needed to do.</p>
<h2 id="download-ubuntu" tabindex="-1">Download Ubuntu <a class="header-anchor" href="#download-ubuntu">#</a></h2>
<ol>
<li>You <strong>must</strong> use a 64bit version.</li>
<li>You <strong>should</strong> use 16.04 or later
(earlier versions may work but may require different instructions).</li>
</ol>
<h2 id="burn-to-a-dvd-r" tabindex="-1">Burn to a DVD-R <a class="header-anchor" href="#burn-to-a-dvd-r">#</a></h2>
<ol>
<li>Double click the .iso file.</li>
<li>Windows Disc Burner should open, if using Windows.</li>
<li>Burn to a disc, I suggest enabling the “verify disc after burn” option.</li>
</ol>
<h2 id="disable-fast-startup-in-windows-10" tabindex="-1">Disable fast startup in Windows 10 <a class="header-anchor" href="#disable-fast-startup-in-windows-10">#</a></h2>
<ol>
<li>Search for “power options” using search.</li>
<li>Click “change what the buttons do”.</li>
<li>Click something like “show hidden/disabled settings”. It will be a
link or button at the top with a UAC badge next to it.</li>
<li>Uncheck the fast startup checkbox near the bottom of the page.</li>
<li>Apply changes.</li>
</ol>
<h2 id="disable-fast-boot-in-uefi-settings" tabindex="-1">Disable fast boot in UEFI settings <a class="header-anchor" href="#disable-fast-boot-in-uefi-settings">#</a></h2>
<ol>
<li>Click Windows flag, click power options, click “restart” whilst holding shift.</li>
<li>Wait for computer to go into an options menu.</li>
<li>Click trouble shooting.</li>
<li>Click advanced.</li>
<li>Click change UEFI settings.</li>
<li>Go to boot tab.</li>
<li>Disable “fast boot”.</li>
<li>You do not need to disable secure boot.</li>
</ol>
<h2 id="check-boot-order" tabindex="-1">Check boot order <a class="header-anchor" href="#check-boot-order">#</a></h2>
<p>The only option should be: “Windows Boot Manager”.</p>
<h2 id="install-ubuntu" tabindex="-1">Install Ubuntu <a class="header-anchor" href="#install-ubuntu">#</a></h2>
<ol>
<li>Insert disc and restart computer</li>
<li>When booting, a blue menu should come up. It’ll have a Windows Boot Manager option,
a disc option, and an UEFI settings option. Select the disc option.
If this menu does not appear, restart and press esc during the “ASUS” logo.</li>
<li>A GRUB menu screen will now appear. <strong>If it does not appear, then the disc hasn’t
been loaded in UEFI mode.</strong></li>
<li>Select “Install Ubuntu”.</li>
<li>Install as normal. You’ll need to disable “install sources from third party providers”
as that will disable secure boot and break windows - don’t worry,
you can install many of these things later.</li>
<li>On the partitions manager, Files (sda5) is the Windows partition,
and Ubuntu (sda6) is the Ubuntu partition.
Note that they won’t add up to a full 1TB, as the laptop comes with two
partitions already.</li>
</ol>
<h2 id="problems" tabindex="-1">Problems <a class="header-anchor" href="#problems">#</a></h2>
<p>Here are some problems.</p>
<ul>
<li>The touchpad was super sensitive and I often clicked whilst typing, deleting text.
<ul>
<li>Solution: add touchpad delay (Settings -> Mouse -> Touchpad)</li>
</ul>
</li>
<li>Occasional WiFi problems
<ul>
<li><code>sudo service network-manager restart</code></li>
</ul>
</li>
<li>Fixed as of 16.10 <s>Locking causes the mouse to disappear when unlocking. Work arounds:</s>
<ul>
<li><s>ctrl+alt+f1 then ctrl+alt+f7</s></li>
<li><s>Move mouse between monitor boundaries if dual booting</s></li>
</ul>
</li>
<li>Fixed as of 16.10 <s>Occasionally when web browsing, tears appear making some text unreadable.</s>
<ul>
<li><s>A work around is to press ctrl+a, which causes the page to re-render.</s></li>
</ul>
</li>
</ul>
<p>These problems are made up for that fact that 1. you now have a real terminal
and 2. double tapping doesn’t break like on Windows (damn ASUS Smart Touch).</p>
<h2 id="see-also" tabindex="-1">See also <a class="header-anchor" href="#see-also">#</a></h2>
<p>Thanks for reading this. For more info, see:
<a href="https://help.ubuntu.com/community/UEFI">ubuntu.com/community/UEFI/</a>.</p>
rubenwardy
Linking a user account to an external phpBB forum
2016-05-07T00:00:00.000Z
2016-05-07T00:00:00.000Z
https://blog.rubenwardy.com/2016/05/07/2100-link-external-phpbb-forum/
This article will show you how to verify a user's identity by letting them associate their account with an external third party phpBB account.
<p>This article will show you how to verify a user’s identity by letting them
associate their account with an external third party phpBB account.
I used Python and Flask to achieve this, however any language and framework
should work, and shouldn’t be too hard to port to.</p>
<p>I wrote a <a href="/2016/05/07/1600-python-phpbb-parser/">Python library to parse phpBB profiles</a>
which I will be using throughout this tutorial.</p>
<p>First, you’ll need to install some packages:</p>
<p>pip install beautifulsoup4 phpbb-parser</p>
<p>I’m using Flask, Flask-User and Flask-SQLAlchemy.</p>
<p>pip install flask flask-user Flask-SQLAlchemy</p>
<p>You need to add a <code>forum</code> field to the user model:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">User</span><span class="token punctuation">(</span>db<span class="token punctuation">.</span>Model<span class="token punctuation">,</span> UserMixin<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token comment"># ...</span>
<span class="token comment"># Forum</span>
forum <span class="token operator">=</span> db<span class="token punctuation">.</span>Column<span class="token punctuation">(</span>db<span class="token punctuation">.</span>String<span class="token punctuation">(</span><span class="token number">100</span><span class="token punctuation">)</span><span class="token punctuation">,</span> nullable<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span></code></pre>
<h2 id="basic-idea-forum-signature-linking" tabindex="-1">Basic Idea: Forum signature linking <a class="header-anchor" href="#basic-idea-forum-signature-linking">#</a></h2>
<ol>
<li>Server generates random token.</li>
<li>User adds token to signature.</li>
<li>User gives forum username.</li>
<li>Server looks up signature from forum username.</li>
<li>If match, link accounts</li>
</ol>
<h2 id="template-link-forum-html" tabindex="-1">Template: link_forum.html <a class="header-anchor" href="#template-link-forum-html">#</a></h2>
<p>This is a page that gives the user the option to link to a forum.</p>
<pre class="language-html"><code class="language-html">
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>div</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>box box_grey<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>h2</span><span class="token punctuation">></span></span>Link forum account<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>h2</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>form</span> <span class="token attr-name">method</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>post<span class="token punctuation">"</span></span> <span class="token attr-name">action</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>{{ url_for('link_forum_page') }}<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>p</span><span class="token punctuation">></span></span>
Enter your forum username here:
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>p</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>input</span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>text<span class="token punctuation">"</span></span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>forum_username<span class="token punctuation">"</span></span>
<span class="token attr-name">value</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>{{ forum_username }}<span class="token punctuation">"</span></span>
<span class="token attr-name">required</span> <span class="token attr-name">placeholder</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>Forum username<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>p</span><span class="token punctuation">></span></span>
Go to
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>a</span> <span class="token attr-name">href</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>{{ forum_url }}/ucp.php?i=profile&mode=signature<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
User Control Panel > Profile > Edit signature
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>a</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>p</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>p</span><span class="token punctuation">></span></span>
Paste this into your signature:
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>p</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>input</span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>text<span class="token punctuation">"</span></span> <span class="token attr-name">value</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>{{ key }}<span class="token punctuation">"</span></span> <span class="token attr-name">readonly</span> <span class="token attr-name">size</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span>32</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>p</span><span class="token punctuation">></span></span>
Click next so we can check it.
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>p</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>p</span><span class="token punctuation">></span></span>
Don't worry, you can remove it after this is done.
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>p</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"><</span>input</span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>submit<span class="token punctuation">"</span></span> <span class="token attr-name">value</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>Next<span class="token punctuation">"</span></span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>form</span><span class="token punctuation">></span></span>
<span class="token tag"><span class="token tag"><span class="token punctuation"></</span>div</span><span class="token punctuation">></span></span>
</code></pre>
<h2 id="token-cache" tabindex="-1">Token cache <a class="header-anchor" href="#token-cache">#</a></h2>
<p>I used a memcache in order to temporarily associate an IP with a random token</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">from</span> werkzeug<span class="token punctuation">.</span>contrib<span class="token punctuation">.</span>cache <span class="token keyword">import</span> SimpleCache
cache <span class="token operator">=</span> SimpleCache<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<h2 id="view-link-forum-page" tabindex="-1">View: link_forum_page <a class="header-anchor" href="#view-link-forum-page">#</a></h2>
<pre class="language-python"><code class="language-python"><span class="token comment"># Project components</span>
<span class="token keyword">from</span> app <span class="token keyword">import</span> app
<span class="token keyword">from</span> models <span class="token keyword">import</span> <span class="token operator">*</span>
<span class="token comment"># Dependencies</span>
<span class="token keyword">from</span> flask <span class="token keyword">import</span> <span class="token operator">*</span>
<span class="token keyword">from</span> flask_user <span class="token keyword">import</span> <span class="token operator">*</span>
<span class="token keyword">import</span> phpbb_parser <span class="token keyword">as</span> parser
<span class="token comment"># GET link_forum_page</span>
<span class="token keyword">def</span> <span class="token function">link_forum_get</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token comment"># Used to automatically fill out the forms in the template with a forum username</span>
forum_username <span class="token operator">=</span> request<span class="token punctuation">.</span>args<span class="token punctuation">.</span>get<span class="token punctuation">(</span><span class="token string">"user"</span><span class="token punctuation">)</span> <span class="token keyword">or</span> <span class="token string">""</span>
<span class="token comment"># Create random token</span>
<span class="token keyword">import</span> uuid
key <span class="token operator">=</span> uuid<span class="token punctuation">.</span>uuid4<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token builtin">hex</span>
cache<span class="token punctuation">.</span><span class="token builtin">set</span><span class="token punctuation">(</span><span class="token string">"forum_claim_key_"</span> <span class="token operator">+</span> request<span class="token punctuation">.</span>remote_addr<span class="token punctuation">,</span> key<span class="token punctuation">,</span> <span class="token number">5</span><span class="token operator">*</span><span class="token number">60</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> render_template<span class="token punctuation">(</span><span class="token string">'link_forum.html'</span><span class="token punctuation">,</span> key<span class="token operator">=</span>key<span class="token punctuation">,</span>
forum_username<span class="token operator">=</span>forum_username<span class="token punctuation">,</span> forum_url<span class="token operator">=</span><span class="token string">"https://example.com"</span><span class="token punctuation">)</span>
<span class="token comment"># POST link_forum_page</span>
<span class="token keyword">def</span> <span class="token function">link_forum_post</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
forum_username <span class="token operator">=</span> request<span class="token punctuation">.</span>form<span class="token punctuation">[</span><span class="token string">"forum_username"</span><span class="token punctuation">]</span>
<span class="token comment"># Get profile</span>
profile <span class="token operator">=</span> parser<span class="token punctuation">.</span>get_profile<span class="token punctuation">(</span><span class="token string">"https://example.com"</span><span class="token punctuation">,</span> forum_username<span class="token punctuation">)</span>
<span class="token keyword">if</span> <span class="token keyword">not</span> profile<span class="token punctuation">:</span>
flash<span class="token punctuation">(</span><span class="token string">"Unable to find your forum user."</span><span class="token punctuation">,</span> <span class="token string">"error"</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> redirect<span class="token punctuation">(</span>url_for<span class="token punctuation">(</span><span class="token string">"link_forum_page"</span><span class="token punctuation">,</span> user<span class="token operator">=</span>forum_username<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># Get stored key</span>
stored_key <span class="token operator">=</span> cache<span class="token punctuation">.</span>get<span class="token punctuation">(</span><span class="token string">"forum_claim_key_"</span> <span class="token operator">+</span> request<span class="token punctuation">.</span>remote_addr<span class="token punctuation">)</span>
<span class="token keyword">if</span> <span class="token keyword">not</span> stored_key<span class="token punctuation">:</span>
flash<span class="token punctuation">(</span><span class="token string">"Failed to get key"</span><span class="token punctuation">,</span> <span class="token string">"error"</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> redirect<span class="token punctuation">(</span>url_for<span class="token punctuation">(</span><span class="token string">"link_forum_page"</span><span class="token punctuation">,</span> user<span class="token operator">=</span>forum_username<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># Get token in signature</span>
<span class="token keyword">if</span> <span class="token keyword">not</span> stored_key <span class="token keyword">in</span> profile<span class="token punctuation">.</span>signature<span class="token punctuation">.</span>text<span class="token punctuation">:</span>
flash<span class="token punctuation">(</span><span class="token string">"Could not find the key in your signature!"</span><span class="token punctuation">,</span> <span class="token string">"error"</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> redirect<span class="token punctuation">(</span>url_for<span class="token punctuation">(</span><span class="token string">"link_forum_page"</span><span class="token punctuation">,</span> user<span class="token operator">=</span>forum_username<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># Get user from database</span>
user <span class="token operator">=</span> User<span class="token punctuation">.</span>query<span class="token punctuation">.</span>filter_by<span class="token punctuation">(</span>username<span class="token operator">=</span>current_user<span class="token punctuation">.</span>username<span class="token punctuation">)</span><span class="token punctuation">.</span>first<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> <span class="token keyword">not</span> user<span class="token punctuation">:</span>
flash<span class="token punctuation">(</span><span class="token string">"Could not find a user of that name!"</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> redirect<span class="token punctuation">(</span>url_for<span class="token punctuation">(</span><span class="token string">"link_forum_page"</span><span class="token punctuation">,</span> user<span class="token operator">=</span>forum_username<span class="token punctuation">)</span><span class="token punctuation">)</span>
user<span class="token punctuation">.</span>forum <span class="token operator">=</span> forum_username
db<span class="token punctuation">.</span>session<span class="token punctuation">.</span>commit<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token comment"># SUCCESS! Redirect to index page</span>
<span class="token keyword">return</span> redirect<span class="token punctuation">(</span>url_for<span class="token punctuation">(</span><span class="token string">"index_page"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># Register link_forum_page</span>
<span class="token decorator annotation punctuation">@app<span class="token punctuation">.</span>route</span><span class="token punctuation">(</span><span class="token string">"/user/claim/"</span><span class="token punctuation">,</span> methods<span class="token operator">=</span><span class="token punctuation">[</span><span class="token string">"POST"</span><span class="token punctuation">,</span> <span class="token string">"GET"</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token decorator annotation punctuation">@login_required</span>
<span class="token keyword">def</span> <span class="token function">link_forum_page</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">if</span> request<span class="token punctuation">.</span>method <span class="token operator">==</span> <span class="token string">"GET"</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> link_forum_get<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> link_forum_post<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>Function split up to make it easier to read.</p>
<h2 id="signing-in-with-a-forum-account" tabindex="-1">Signing in with a forum account <a class="header-anchor" href="#signing-in-with-a-forum-account">#</a></h2>
<p>The above is only good for linking, and not for actually signing in.
To allow signing in, you could read public profile data off of the forum
profile and then use OAuth to sign in with those.</p>
<p>For example, if the user has their Github listed on their forum profile,
you can sign them in using that Github account.</p>
<ol>
<li>User gives forum username</li>
<li>Server looks up Github field in forum username</li>
<li>User signs in using Github to confirm identity
<ol>
<li>Server redirects user to github authentication.</li>
<li>Github redirects user to github callback with access token.</li>
<li>Server looks up Github username using access token.</li>
<li>Access token added to user model and accounts linked.</li>
<li>If not logged in, server logs user into the account associated with
that Github username.</li>
</ol>
</li>
</ol>
rubenwardy
Getting user profile data from external phpBB forums
2016-05-07T00:00:00.000Z
2016-05-07T00:00:00.000Z
https://blog.rubenwardy.com/2016/05/07/1600-python-phpbb-parser/
I recently wrote and released a python module to allow fetching of profile data.
<p>I recently wrote and released a python module to allow fetching of profile data.</p>
<p>You can install it using pip:</p>
<p>pip install beautifulsoup4 phpbb-parser</p>
<p>Here’s how you import and use a profile:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> phpbb_parser <span class="token keyword">as</span> parser
username <span class="token operator">=</span> <span class="token string">"rubenwardy"</span>
profile <span class="token operator">=</span> parser<span class="token punctuation">.</span>get_profile<span class="token punctuation">(</span><span class="token string">"https://forum.minetest.net"</span><span class="token punctuation">,</span> username<span class="token punctuation">)</span>
<span class="token keyword">if</span> profile<span class="token punctuation">:</span>
signature <span class="token operator">=</span> profile<span class="token punctuation">.</span>signature<span class="token punctuation">.</span>text
location <span class="token operator">=</span> profile<span class="token punctuation">.</span>get<span class="token punctuation">(</span><span class="token string">"location"</span><span class="token punctuation">)</span> <span class="token keyword">or</span> <span class="token string">"unknown"</span>
github <span class="token operator">=</span> profile<span class="token punctuation">.</span>get<span class="token punctuation">(</span><span class="token string">"github"</span><span class="token punctuation">)</span> <span class="token keyword">or</span> <span class="token string">"none"</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>username <span class="token operator">+</span> <span class="token string">" from "</span> <span class="token operator">+</span> location <span class="token operator">+</span> <span class="token string">" has github "</span> <span class="token operator">+</span> github<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"Signatue: "</span> <span class="token operator">+</span> signature<span class="token punctuation">.</span>text<span class="token punctuation">)</span>
<span class="token keyword">else</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"Could not get profile!"</span><span class="token punctuation">)</span></code></pre>
<p>profile.signature is a beautifulsoup4 object.</p>
<p>Next: <a href="/2016/05/07/2100-link-external-phpbb-forum/">Linking a user account to an external phpBB forum</a>.</p>
rubenwardy
1st Semester of University
2016-02-24T00:00:00.000Z
2016-02-24T00:00:00.000Z
https://blog.rubenwardy.com/2016/02/24/university/
To get more than 50% on our coursework, you had to submit extensions. Here are some of my favourite ones.
<p>To get more than 50% on our coursework, you had to submit extensions.
Here are some of my favourite ones.</p>
<h2 id="pong" tabindex="-1">Pong <a class="header-anchor" href="#pong">#</a></h2>
<figure class="undefined"><a href="/static/img/QtCuGu5nxO-640.png"><picture><source type="image/webp" srcset="/static/img/gX3zEW41Cd-540.webp 540w"><img alt="Screenshot of Pong" loading="lazy" decoding="async" src="/static/img/gX3zEW41Cd-540.png" width="540" height="405"></picture></a><figcaption>Screenshot of Pong</figcaption></figure>
<p>I created a Pong clone using C and SDL.</p>
<h2 id="chess-ai" tabindex="-1">Chess AI <a class="header-anchor" href="#chess-ai">#</a></h2>
<figure class="undefined"><a href="/static/img/bQtUGgNTu6-922.png"><picture><source type="image/webp" srcset="/static/img/9oDEZXA2vV-540.webp 540w"><img alt="Screenshot of Chess" loading="lazy" decoding="async" src="/static/img/9oDEZXA2vV-540.png" width="540" height="245"></picture></a><figcaption>Screenshot of Chess</figcaption></figure>
<p>I created a Chess game and Chess minimax implementation in C.</p>
<h2 id="infinite-and-generic-game-of-life-and-more" tabindex="-1">Infinite and Generic Game of Life, and more <a class="header-anchor" href="#infinite-and-generic-game-of-life-and-more">#</a></h2>
<figure class="undefined"><a href="/static/img/3Pygyqy5r1-640.png"><picture><source type="image/webp" srcset="/static/img/H4CxJQA2bV-540.webp 540w"><img alt="Screenshot of the Game of Life" loading="lazy" decoding="async" src="/static/img/H4CxJQA2bV-540.png" width="540" height="405"></picture></a><figcaption>Screenshot of the Game of Life</figcaption></figure>
<p>Features:</p>
<ul>
<li>Chunk based infinite universe. Automatically expands.</li>
<li>Interactive player and universe editor using SDL.</li>
<li>Can set the Born/Survive ruleset code (from command line).</li>
<li>Can choose the number to steps to take before opening the player
or outputting to stdout (from command line).</li>
<li>Has tests.</li>
</ul>
rubenwardy
Employee Task Management System
2015-12-10T00:00:00.000Z
2015-12-10T00:00:00.000Z
https://blog.rubenwardy.com/2015/12/10/etms/
I was contacted by a client to create a system which calculates the workload for employees based on their assignment to tasks and appointments.
<p>I was contacted by a client to create a system which calculates the workload
for employees based on their assignment to tasks and appointments.</p>
<p>The system needs to solve two problems:
Firstly, different staff members work different numbers of hours, which
makes it hard to allocate tasks fairly and proportionally.
Secondly, the client wanted to use the system to analyse past workloads
and to anticipate future workload, in order to improve her system of work.</p>
<h2 id="implementation" tabindex="-1">Implementation <a class="header-anchor" href="#implementation">#</a></h2>
<figure class="undefined"><a href="/static/img/Z_c9MEb_v--793.png"><picture><source type="image/webp" srcset="/static/img/SojqU7lw8u-540.webp 540w"><img alt="Screenshot of ETMS" loading="lazy" decoding="async" src="/static/img/SojqU7lw8u-540.png" width="540" height="247"></picture></a><figcaption>Screenshot of ETMS</figcaption></figure>
<p>The solution contains an algorithm to suggest new employees to tasks, based on
the type of tasks they are currently assigned to, how many hours they should
work a week, and how what they are doing at the time.</p>
<p>The solution was written as a web application, using Python, Flask and SQLAlchemy.</p>
<figure class="undefined"><a href="/static/img/pDfv1W1JnM-1281.png"><picture><source type="image/webp" srcset="/static/img/_GtUcTkAhv-540.webp 540w"><img alt="Server side rendered graphs in ETMS" loading="lazy" decoding="async" src="/static/img/_GtUcTkAhv-540.png" width="540" height="248"></picture></a><figcaption>Server side rendered graphs in ETMS</figcaption></figure>
<p>Originally I created the graphs using a HTML5 canvas. However, during the
first client feedback sesson, I found out they used Internet Explorer with
JavaScript disabled. This meant that I needed to move all of the graph
rendering to the server. I used the Python Imaging Library (PIL) to do this.</p>
rubenwardy
Convert folder of images to PDF with filenames
2015-04-07T00:00:00.000Z
2015-04-07T00:00:00.000Z
https://blog.rubenwardy.com/2015/04/07/convert-folder-of-images-to-pdf-with-filenames/
Here is a shell script specific for GNU/Linux based operating systems to use. On different operating systems the convert commands will be the same.
<p>Here is a shell script specific for GNU/Linux based operating systems to use.
On different operating systems the convert commands will be the same, but
the for loop will be different due to a different batch file syntax.</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">rm</span> /tmp/imageex <span class="token parameter variable">-r</span>
<span class="token function">mkdir</span> /tmp/imageex
<span class="token keyword">for</span> <span class="token for-or-select variable">filename</span> <span class="token keyword">in</span> *.png<span class="token punctuation">;</span> <span class="token keyword">do</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Processing <span class="token variable">$filename</span>"</span>
convert <span class="token variable">$filename</span> <span class="token parameter variable">-background</span> White <span class="token punctuation">\</span>
label:<span class="token variable">$filename</span> <span class="token parameter variable">-gravity</span> Center <span class="token punctuation">\</span>
<span class="token parameter variable">-append</span> <span class="token parameter variable">-pointsize</span> <span class="token number">14</span> <span class="token punctuation">\</span>
/tmp/imageex/<span class="token variable">$filename</span>.png
<span class="token keyword">done</span>
<span class="token builtin class-name">echo</span> <span class="token string">"Exporting to PDF..."</span>
convert /tmp/imageex/*.png output.pdf
<span class="token builtin class-name">echo</span> <span class="token string">"Done."</span></code></pre>
rubenwardy
3D Projection
2015-01-08T00:00:00.000Z
2015-01-08T00:00:00.000Z
https://blog.rubenwardy.com/2015/01/08/3d-projection/
Recently I have created an implementation of the 3d projection algorithm. It is just wireframe models. It works well, except it doesn't do frustum culling.
<p>Hello 2015! Recently I have created an implementation of the
<a href="https://en.wikipedia.org/wiki/3D_projection#Perspective_projection" target="_blank">3D projection</a> algorithm.
It is just wireframe models. It works pretty well, except it doesn’t do frustum culling.
You still see things that are behind you, but upside down.</p>
<p>The source code of this implementation is available under the WTFPL or CC0 licenses -
you can choose which one you want to use.
Use WASD to move, arrow keys to rotate, space to ascend and shift to descend.</p>
rubenwardy
Lotka Volterra - Predator Prey
2014-12-16T00:00:00.000Z
2014-12-16T00:00:00.000Z
https://blog.rubenwardy.com/2014/12/16/lotka-volterra/
This time I am going to show off a project I have had for quite a while - it is a cellular automaton which simulates the Lotka Volterra equations.
<p>Hi all! I’m back again for another post.
This time I am going to show off a project I have had for quite a while -
it is a cellular automaton which simulates the Lotka Volterra equations.</p>
<p><canvas id="foxrabbit" style="background: green;"></canvas></p>
<script src="/static/files/foxrabbit.js"></script>
<p>Orange for foxes, brown for rabbits.</p>
<p>The Lotka Volterra equations, also known as the predator prey equations,
are used to model the populations of two species over time - a predator
and a prey.</p>
<figure class="undefined"><a href="/static/img/HRnpJGAZSf-1512.png"><picture><source type="image/webp" srcset="/static/img/pFag-Z7ZtZ-540.webp 540w"><img alt="Graph of my result (left) next to the graph of the expected result (right)." loading="lazy" decoding="async" src="/static/img/pFag-Z7ZtZ-540.png" width="540" height="189"></picture></a><figcaption>Graph of my result (left) next to the graph of the expected result (right).</figcaption></figure>
rubenwardy
Lichess Widgets
2014-11-07T00:00:00.000Z
2014-11-07T00:00:00.000Z
https://blog.rubenwardy.com/2014/11/07/lichess-widgets/
Just a short post this time - I have created some widgets for the chess website Lichess.
<p>Just a short post this time - I have created some widgets for the chess website
Lichess. I was quite surprised that these did not exist yet, in any form.
Luckily Lichess exposes an API to use. You can have a look at them by clicking
the link below.</p>
rubenwardy
Lichess
2014-10-26T00:00:00.000Z
2014-10-26T00:00:00.000Z
https://blog.rubenwardy.com/2014/10/26/lichess/
Lichess is an online chess game and community. It is free and open source, ad-free and subscription-free.
<p>Lichess is an online chess game and community. It is free and open source,
ad-free and subscription-free. You can play against your friends, random
strangers and the computer. You can create teams to compete in tournaments.
There are tournaments going on all the time.</p>
<p>There are different game modes available, but I don’t really touch these.
Chess960, King of the Hill, Three-check, From position.
You can request that your game is analysed by a chess engine, it gives you
suggestions on what you should have done.
I like how it shows a graph of how much of an advantage one player has over
one another.</p>
<p>Lichess is also good for becoming better at chess.
Chess puzzles are available in the training section,
where you have to choose the best move in a given situation,
and follow it through to checkmate.
Players can create their own puzzles.</p>
<p>It is truly better than the alternatives.
Chess.com is cluttered with ads and has an uglier design - minimalism is the way forward.
Lichess is free and open source, and always will be.</p>
rubenwardy
Bytecode, Compilers and Interpreters
2014-10-22T00:00:00.000Z
2014-10-22T00:00:00.000Z
https://blog.rubenwardy.com/2014/10/22/bytecode-and-compilers/
Recently I have been looking at languages and compilation: VMs, parse trees, lexers, and interpreters
<p>Recently I have been looking at languages and compilation: VMs, parse trees,
lexers, and interpreters. <a href="https://www.nand2tetris.org">Nand to tetris</a> is a
pretty awesome guide to how the CPU executes programs - from logic gates to high
level languages.</p>
<h2 id="assembler-and-bytecode-vm" tabindex="-1">Assembler and Bytecode VM <a class="header-anchor" href="#assembler-and-bytecode-vm">#</a></h2>
<p>I created an assembler and Virtual Machine to run assembly style instructions.
I haven’t released it as it isn’t as complete as I want it yet.</p>
<h2 id="parse-trees-and-reverse-polish-notation-rpn" tabindex="-1">Parse Trees and Reverse Polish Notation (RPN) <a class="header-anchor" href="#parse-trees-and-reverse-polish-notation-rpn">#</a></h2>
<p>Turns a string such as “( 0 - (6) + ( 6 ^ 2 - 4 * 1 * 5 ) ^ (1 / 2) ) / ( 2 * 1)”
into a binary syntax tree, and then into Reverse Polish Notation, and then executes it.</p>
<h2 id="lexical-analyser" tabindex="-1">Lexical Analyser <a class="header-anchor" href="#lexical-analyser">#</a></h2>
<p>I have also experimented with a simple Lexical Analysisor, however it’s not at a state
that I’d like to release.</p>
<h2 id="rufunge" tabindex="-1">Rufunge <a class="header-anchor" href="#rufunge">#</a></h2>
<p>Befunge is an <a href="https://en.wikipedia.org/wiki/Esoteric_programming_language" target="_blank">esoteric programming language</a> -
a language which isn’t meant for pratical use, but rather to test programming
use. The language consists of a two dimensional grid of characters. Each
character is an individual instruction. I was disappointed that the authors
didn’t take it seriously enough, and didn’t add input and output to it. So, I
created my own! I called it ‘Rufunge’. It’s not finished yet, but can currently
run most Befunge programs.</p>
<h2 id="mesecode" tabindex="-1">Mesecode <a class="header-anchor" href="#mesecode">#</a></h2>
<p>Mesecode is a language which compiles to Lua. It’s purpose it to make
definitions in Minetest easier to read.</p>
<p>Instead of:</p>
<pre class="language-lua"><code class="language-lua">minetest<span class="token punctuation">.</span><span class="token function">register_node</span><span class="token punctuation">(</span><span class="token string">"default:stone"</span><span class="token punctuation">,</span> <span class="token punctuation">{</span>
description <span class="token operator">=</span> <span class="token string">"Stone"</span><span class="token punctuation">,</span>
tiles <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token string">"default_stone.png"</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
is_ground_content <span class="token operator">=</span> <span class="token keyword">true</span><span class="token punctuation">,</span>
groups <span class="token operator">=</span> <span class="token punctuation">{</span>cracky<span class="token operator">=</span><span class="token number">3</span><span class="token punctuation">,</span> stone<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
drop <span class="token operator">=</span> <span class="token string">'default:cobble'</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<p>You can write:</p>
<pre><code>mod default
node Stone
is ground, cracky=3, stone
drops Cobble
</code></pre>
rubenwardy