Physician, Heal Thyself

May 4, 2012 by  

Web server. Head Crash. No Backup. Doh!

You’d think I’d learned my lesson about backing up critical data, but nope, not me. In an extreme case of overconfidence and hubris, I let a Mac mini that I had set up in my house as a web server for a genealogically oriented family association go without a full backup … for almost seven years! Yeah, yeah… I had planned on doing something when I had a little free time on my hands, but that time never came.

This server was running several specialized web applications, including the genealogy of the Grinnell family, configured in such a fashion that it is very difficult to start over from scratch. Oh, and did I mention the main membership database? Yup. Also gone. At least for that one, I think I’ve got a backup of that file somewhere, and because it’s membership renewal time, the membership chair had just made printouts of critical member information. Not that this should in anyway excuse what has happened.

Last Thursday (February, 2010), I got an email from the association president (I am the secretary and webmaster) noting the difficulty they were having in accessing the website. When I got home from work, I confirmed the problem and decided to start with the usual method of fixing things like this — reboot the server. Well, that’s where things got REALLY interesting/bad.

When I rebooted, I got a blank screen for about three minutes, followed by an alternating blank folder icon and a Mac OS X icon. I tried inserting a diagnostic CD, and it wouldn’t mount. Now, it won’t even eject!

I listened very closely to the drive, and it was making some really scary scraping noises. Not what you want to hear in a hard disk drive!

I removed the drive but was unable to get it to mount in any way, shape, matter, or form. Bad, bad news. Even worse news was that I had not performed a backup of the primary content for several years. Actually, I had, but I saved the backup files on the same hard disk drive that ultimately failed.

Suffice it to say, I have let down the organization I have been supporting with my technical expertise for over 20 years, and if things have gone as bad as I think they may have, I may have caused the loss of 6-9 months of data entry work performed by our genealogy committee chairperson in preparation for my task of writing and editing the next edition of the Grinnell family genealogy, a tome estimated to be over 2,000 pages, not to mention the entire website content, and the membership database. Here’s hoping it won’t come to that.

So, what have we learned here? First and foremost, make sure your data is backed up regularly, and if it absolutely, positively has to be available 24/7, make sure you have created at least one, and maybe even two bootable backups of your work. Next, if the data is that critical, move the site off your little computer at home, and if you are using software that permits it, move it to a hosted service, though even there, no matter whether they do regular backups or not, you must do your own backup of your critical data often. Ultimately, if you are the caretaker of the data, especially when it’s someone else’s data, it’s your responsibility to ensure that it remains available to those who need it, when they need it.

This is has been a very painful exercise in poor judgment that could easily have been avoided, and it could have all been done automatically with a few simple scripts.

Excuse me while I flog myself a little more before bedtime.

That was the story as of the early part of February, soon after the disaster began to unfold. Next, my dear friends, is the rest of the story:

My tech friend (president of the local Mac user group), known as “JG” (short for a very long German name), an interesting fellow who used to own several Apple stores in the Washington DC area (he sold his business just before Apple built their first Apple Store, directly across the street from his store, at Tysons Corners, near McLean, VA), helped me out quite a bit. I found an equivalent Mac mini for under $200 on eBay, which had an identical 40GB Seagate drive. The problem, however, was much, much worse, as he discovered that the drive in my Mac mini had suffered a terminal head crash. “JG” took the drive apart and found deep gouges on the media surface. Oh, did I mention I didn’t have a backup?

Anyhow, the good news (yes, Virginia, there was at least some good news) is that I was able to locate about 80% of the content in various forms and formats on other computers around the house, and so I have set up an account with a webhost ( and am in the process of building a Joomla!-based website for The person responsible for keeping up the online genealogy database (40,000 names, 15,000 families, etc.) kept a second copy of the database in Family Tree Maker (finally a Mac version after an almost 20 year absence!!!!!), so I should be able to recover all of that, put it back into The Next Generation of Genealogy Software (an interactive, web-based genealogy database), and tie it into Joomla! with single sign-on integration (one login works for both elements–premium content on the website and select access to the genealogy). The membership database, in FileMaker Pro, after we recover information on the most recent 30 or so members, is going to a FileMaker Pro host for 20 bucks a month.

Boy, did I learn a big, painful lesson! Backups are good. No backups are bad.

This experience was, at the very least, harrowing, frightening, sad, terrifying, humbling, humiliating, and more. I can’t think of a more perfect time to increase my medication.

(Portions reposted from Palm Beach with their kind permission)