The third administration: Welcome, dev & issue reports

[Ancient Crystal]

I'd say no -- for all I know I'd permanently lock out some old accounts with unavailable emails or the like. Better to let anyone who's still around change theirs manually.

[admin3]

Unscheduled downtime today, not sure for how long, probably a few days at most. The problem seems to have been the logs growing to several gigabytes, exhausting the server partition the web-facing containers are installed in. We're back to normal now that I've cleared them out, I'll have to install some sort of log rotation/truncation in the long term.

[admin3]

The disk-space problem keeps cropping up every now and then, I still need to do something about that. I've also cleared out the last batch of spambots and switched the forums to Google's recaptcha v3, which -- might perform better? I seem to recall v2 mostly didn't help, even back when I set this iteration of the forum up in 2020, and I've no idea how much of a step up v3 is. In the worst case we're back to rotating simple lore questions.

[Sublevel 114]

*karmic thumbs up*

[Sublevel 114]

I delayed with moving to Sublevel 114

can you help, please?

[Vortex]

Done!

[admin3]

The site had a bit of a health scare yesterday. Since we moved to the new hosting solution, there have been intermittent uptime losses due to the docker logs simply growing to clog the entire filesystem. These crashes (of a sort) have tended to be harmless, nothing a restart won't fix -- which perhaps is why I never took the time to correct the root cause -- but this time around I found the sessions table in the database damaged somehow, preventing a restart. What's more, though there's a standard way of correcting the issue, it needed more filesystem space than the server had on hand, since I've ran a lean setup from the beginning. This all meant it took a few hours to repair safely, plus however long the downtime went unnoticed.

The upside is, this should all be sorted now. Furthermore, I've finally re-engaged the scheduled backup system that initially didn't make it over to the new hosting solution, and I've found the configuration option to (hopefully) prevent logs from growing uncontrollably in the future. Both are things I should have done a year ago, but oh well.

[Sublevel 114]

(Looks like site's SSL certificate expired)

[admin3]

It did, after I failed to catch an error in the DNS configuration, which certificate renewal relies upon for the wildcard-type SANs (like *.pastelland.net).

[admin3]

Over the past week, the forum has (I suspect, from looking at the logs) been intermittently inaccessible due to high load, before becoming more-or-less completely inaccessible (I know, from manual testing) at some point in the past few days. The high load appears to come from one or more crude web scrapers (which have become far more prevalent as LLM training data becomes valuable) bombarding the site with queries. The server can in principle handle far higher loads in terms of bandwith, but apparently high-frequency access to the forum database eats CPU (which is the resource we were bottlenecked for) in a way that, say, video stream does not.

Fortuitously, the rise in web scrapers has been a known problem for some years now, and mitigation technologies are already somewhat mature. On the recommendation of Gil (hi) via Mat's discord, I've sought out and installed Anubis (git, site), which uses a combination of heuristics and proof-of-work to filter scrapers with reasonable effectiveness. (At least, the dumb scrapers, which is what we're dealing with.)

Internet users may already be familiar with the mascot, which going forward will also be guarding the pastel forums from undue intrusions into the subnet:

Screenshot from 2025-09-30 23-08-02.png (117.35 KiB) Viewed 54 times

(I also added the custom [width] bbcode tag, in which images can be put to keep them from taking up the whole page. Use it like [width=5,cm][/width] or [width=20,px][/width].)