Been a hellish last few days!
The other day I noticed that my webserver was approaching its 18 gig capacity so I planned to add another three drives to my RAID. I scheduled the downtime with my users and went about this seemingly simple task. The server is an older Dell Poweredge so the drives aren’t hot swappable. So I shutdown the server, add the drives, reboot, reconfigure the container and reboot and immediately get errors. I really didn’t want to mess around with it so I decide just to go back to the original drives/configuration. Same dealio…shutdown, pull out the drives, reconfigure container, blah blah, still get errors. At this point I did a reconstruct which takes a few hours.
Anyways, while the drives were reconstructing, I go talk to the sys admin who handles backups…thinking worst case I’ll need to rebuild the server and reload from backup. To my surprise the tech tells me that my server isn’t being backed up!!! WOAH
So anyways I immediately get onto Dell tech support. Even though I didn’t have any customer tags or service numbers they wanted to help me. Props to them. Anyways, I’m verifying my processes and she’s confiming I did everything right and just to wait for the reconstruction to get done. So I thank her and everything and the last thing she says is just not to interrupt the reconstruct and I should be ok. WHEW
So times goes by and I notice that its taking its sweet time on 42%. 10 minutes go by…still 42%. I’m getting nervous. Hour goes by…still 42%. WHOA
Right back on the phone to Dell. Discuss the alternatives. Well, not really alternative, more like reboot and hope pigs fly. You know what, pigs don’t fly and the server didn’t reboot, not even to SCSI bios. By this time my fun meter is pegged and I pronounce it dead.
I call it a night and go home to drink and consider the quickest way to rebuild 18 gigs of web pages and apps. Can you say A Long Dark Road?
So I go to bed and in the morning I get a call from another tech saying they had found a week old backup. YAY
So I rebuild the server, reload from back up and enable SSL and everything is working great. Another night drinking in celebration.
Working on it today to recreate some of the past data/shares/etc and I notice that I’m getting a security alert on my certificate. I remember that I still needed to import a certificate chain to make the security alert go away. I get the chain all imported, reboot, and as my luck would have it SSL craps out on me.
I’ve worked 50 hours in the last 3 days and I’m still dorked.
Someone stick a fork in my neck!!
