At a previous employer that shall remain unnamed, someone started an email "reply to all" storm.
The problem started with the first email that included an Access database form we were all supposed to fill out & send back.
Except it wasn't a link to the form, it was the entire MF4ing form+database project file folder.
Since it was a corporate wide poll, they sent a copy to the entire corporation: every single address in a company with a few hundred local staff in the building itself, another thousand+ field techs, and a single address that was itself a distrubution list to the entire global company staff.
The poll was only supposed to be for the physical office staff, so everyone else hit that beloved ReplyToAll button & demanded to be removed from the list.
Thankfully I wasn't the poor sod that did the original email, but I was part of the IT group that had to clean up the mess afterwards.
All those emails, each with a multiple hundred megabyte attachment, getting cloned & recloned & rerererecloned, until the network just s4 itself & took most of us offline.
"Luckily" the field techs were all dial up & could just unplug to stop the storm in it's tracks.
But the rest of us hardwired on the network had no options besides take it or turn the computer off.
Being the PFY & lowest on the totem pole, I didn't have that option.
We had to write a script to find every email with an attachment of a specific set of files, strip the attachment, save the file diff to an $EmployeeName.diff.log file, delete the attachment, & move the email to an archive server.
Even with the script doing it as fast as the sacraficial interface computer could run it, it still took the better part of an entire day to finish the job.
My BOFH mentor had to manually call all the other BOFH's & tell them the script commands, verify that they had the code right, then sit on his thumbs while they ran it to ensure it at least started properly.
After about 5~10 minutes of hand holding, he would tell them to call him back if anything went wrong, or to tell him the job was done & they were ready to turn the network back on.
I'm sure some of the reps on the other side of the world just _LOVED_ getting Buttcrack-O-Dawn calls to fix an issue that was not of their making.
Eventually we cleaned the network, purged it of all the files/emails, and got everyone back up & running.
My BOFH then wrote an email rule that auto-rejected any email with an attachment;
the employees were told to save the file to the server, include just a link to the file in the email, so the ClusterF4 couldn't happen again.
In a strange twist of WTF?, the incident prompted the higher ups to invest in a faster corporate intranet.
Given the company was a Global Telecommunications Provider, it was always a mark of shame that we had some of the crappiest intranet capabilities you could imagine.
So for the high mucky mucks to "take it as a learning experience" was astonishing to say the least.
=-Jp |
