As a general thought, any idea if people have looked at something like (for example) using Proxmox on the physical hardware so the services can be put on VMs which can be migrated between hosts if there are problems?
They are indeed not equivalent, but for our use case this is sufficent, if we detect data corruption we can just throw away the files and download/regenerate them (this is a freely available dataset, if a bit large, https://en.wikipedia.org/wiki/Endgame_tablebase will explain it better than me). For this reason, it is also not backupped.
As mentionned elsewhere, we're renting most of our infra from OVH, and paying, monthly, for 40TB of SSDs or NVMes would simply explode our yearly budget.
Source: am président of the lichess charity (and also one of the sysadmins)
We're up to 2 employees now! The founder and a mobile dev.
The impact on costs is "not small", because as a rough estimate, the charity pays overall about twice what the dev gets in take-home money, because French employer taxes are high (keyword for the Frenchies reading us: URSSAF).
Source: am President of the Lichess charity and have the honour and pleasure of dealing with most of the French administrative paperwork.
It's not listed in this page because it varies month-by-month, but we publish our yearly books in a lichess blog post and on the government "Journal Officiel":
The TL;DR is that donations so far exceed our expenses, thanks to the generosity of our userbase, and thanks to the hundreds of volunteers doing work for free.
Nothing is modest at Lichess scale, we have a few million users, several thousand of them post in forums daily and chat with each other.
Also, we're a chess website with more than one million games per day, and some people like to cheat. We're classifying "fair play and anti-cheating" as "moderation" in our expenses because the two are intertwined.
So far France has given us exactly 0 euros and 0 cents, but has required us to hire and pay an accounting company to certify our books due to the level of donations we're receiving.
We could maybe argue that being allowed to exist as a non-profit is something France deserves kudos for, if you're in a good mood.
We're paying taxes on our employees, they're paid like regular French employees and therefore the employer is taxed accordingly. This is not income tax on our donations or swag sales.
I would prefer if the non-profit in France were "employer-taxed" less than for-profit companies, but no one seems to care.
We're self-hosting most of our infra on bare-metal, on the best price/quality hosting provider I know of (OVH). I doubt you could store ~1TB of metrics going back 4 or 5 years on fast NVMe storage for less than this in the cloud.
