I’m pleased to announce that Anime Herald is back online, after our extended downtime today. However, due to the circumstances, and the length of time, I want to be frank with you, and explain the incident that took us down, and its overall impact.
What Happened
At 5:45AM EDT, we received a report from our monitoring host informing us that Anime Herald was experiencing high ree site was down entirely. Upon logging into the instance, we discovered that it was entirely unresponsive, to the point that we could not run our typical diagnostics. We initiated a reboot at 6:30AM EDT
Upon reboot, we were unable to bring the site back online. According to the logs, a large, sustained attack from generative AI scraper bots had interrupted a system-critical update, forcing the server into an inoperable state on a functional level. Attempts to restore from backups proved ineffective, as these bots would immediately lock on to the IP address and cause a nearly immediate takedown.
I should note, that we actively block GenAI bots, because we are of the firm and unchanging belief that the systems actively steal from independent creators, and that anything generated by them is built upon wholesale theft. This is something we have codified in our Contributor Guidelines, and have been open about on our social media accounts.
As such, we have enabled all protections in our Cloudflare instance, and we are taking all steps possible to ensure that they do not steal from us. However, there has been a rise in bots that spoof user-agent strings, and rotate proxies to avoid such measures. These rogue bots actively ignore directives like robots.txt, thus breaking the social contract that’s been in-place since the dawn of the World Wide Web.
Our Course of Action
At 7:42 AM EDT, the decision was made to deploy a new VPS, with a new IP address, and restore our existing backup. However, to do so meant taking the time to restore more than fifteen years of images, articles, and other rended up taking several hours, keeping us offline until 12:37PM EDT
As our old VPS format was deprecated and grandfathered, we needed to upgrade. We are now running on a system with twice as much RAM and storage, along with multiple dedicated CPU cores. This has the knock-on effect of being more expensive, but hopefully, it will also prevent us from being overwhelmed for the foreseeable future.
Anime Herald, as a publication, is important to me – I want to do everything I can to ensure that it’s still here in five, ten, twenty years down the line. I would like to offer my personal apologies for our downtime, and thank you for putting your trust in us over the past fifteen years.
