NYAARRGH AAAGH BLAAAARGH AAAAAAAAAAAGH GGRRRAAAAARRR AAAGH. And so forth.

Yesterday morning there were 900 bounced email messages in my inbox, complaining that various folks with names like “mtahswud1423” had been sending spam through one of the domains on my webserver. Started around 3 am. This was a little unusual.

I mean, there are always a handful of these, just from people spoofing return addresses — everybody who has a domain with a catch-all address is used to seeing that. But this was way out of line. 900 bounces in six hours? No way. And some of the headers looked suspicious: instead of the usual imaginary return addresses, some of them were from ‘localhost’. And they were doing things like trying to subscribe to mailing lists, all sorts of behavior I’m not used to seeing from everyday spammers.

So I’m thinking, fuck. I’ve been hacked. Well, first time for everything.

So I spent the entire day backing up the data for the dozen or so sites on the server, wiping the entire machine clean, installing a new, up-to-date version of the OS, patching every patch I can find, checking through every script in the backed-up site data to make sure there weren’t any spam relays snuck in there (there weren’t), adding traps on the scripts that are supposed to send mail to notify me if they were being used too often, restoring the cleaned backups, testing again to make sure they’re running correctly, resetting permissions, notifying users of their new passwords, and so on.

(Thank you, Rachel, for letting me borrow your bandwidth; I’d still be working at it now if I’d tried doing all this over my satellite connection. Stupid bandwidth caps.)

This was a day I really could’ve more usefully spent doing, well, quite a number of other things. Work’s been backing up on me, a project I thought was finished has just re-appeared, there’s some household stuff that needs dealing with, etc. But I couldn’t just let my server sit there being a spam zombie, this needed to be taken care of before I get blacklisted. So I sucked it up, made my peace with the progress bar, and at least at the end of the day felt like I’d accomplished something. Server secured, cap’n. Take that, spammers.

This morning there were another 1700 bounced messages in my inbox. A large number of them refer to spam that was sent during the hour when I know for a fact my server was completely offline, shut down, incommunicado. I don’t know where this spam is coming from, but it ain’t me. And there’s nothing, absolutely nothing I can do about it.

To this I will respond with calm equanimity. NYAAAAAAAARRRRRGH.