...and not even the
ICG blow-up. That was minor compared to the rest of my day.
It's been tech hell week. No laptop Sunday thru Thursday afternoon. Antivirus upgrades on my servers on Thursday evening that took 3 hours instead of an hour and a half. Final tweaks getting taken care of from home at midnight while I watched the rerun of "Rose" (and caught the breast implant comment).
And then the phone rang at 6:50 this morning. Both my mail servers were down, and had been down for a while. I recycled them, but got a call a bit later saying they had crashed two more times.
The morning after I upgraded the antivirus software.
Can you guess that I spent an hour or two trying to diagnose the problem in the new antivirus software?
Wasn't the problem. Just coincidence.
The servers were getting slammed by a
dictionary attack phishing scheme. Caught that around 10:00am and put a block on the address.
That got the machines up and stable for my 11:00 meeting that I made it into the office just in time for.
Of course, it still took until 12:30 to work through the whole system to get mail routing and delivery working completely again. Then I could get something to eat. About 6 1/2 hours after I got up. I never go 6 1/2 hours after waking up without eating.
After lunch, it was time to drop stray messages back into the queue. 120,000 phish-mails deleted from the router files that I could recover. Maybe 1000 good messages in the queue. Two more queues I couldn't recover.
Joy. Rapture.
Fortunately I've got management (and most of my customers) who understand that if the downtime has been more than 15 minutes they should stop bothering me.
It's still less time than major outages outside our department.
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-userinfo.gif)
johnnyeponymous brings us Pacheco Progress 1, chock full of nutty goodness.
