When Full-Disk Encryption Goes Wrong
Last September I watched in horror as my MacBook Pro slowly died.
It started with a text from my wife:
“Hmmm,” I thought. “Maybe the System folder or something got corrupted. Should be a quick fix.” Little did I know that we had just lost every piece of data on that computer.
Side note: I love that the only thing the MacBook is displaying is an icon equivalent of ¯_(ツ)_/¯. No helpful error message, nothing. Just a “Yeah, I’m not even going to try.”
I’ve been a “computer person” for most of my life. I worked in tech support in high school and college, so I went through a pretty exhaustive list of things to attempt to get the computer to boot.
After a few unsuccessful attempts at getting the system to boot off the disk (and the usual exorcism of zapping the NVRAM a few times for good measure), I booted into Apple Hardware Test.
This probably isn’t going to end well. #shouldHaveBackedUp pic.twitter.com/p0VDfle5VY
— Tyler Stromberg (@aqua_geek) September 5, 2015
Of course, that found nothing wrong with my disk. On the plus side it satiated my nostalgia for Mac OS 9. If you ever find yourself missing those buttons hop on over to Apple Hardware Test.
Next I tried booting into the OS X Recovery System. That didn’t work — the Mac refused to find it. “Hmmm. That’s not good.”
Data Recovery #
At this point, I thought it best to take it into the Apple Store and have them look at it. I didn’t want to inadvertently wipe the drive and lose my data (ha, joke’s on me — it was already gone).
I was able to get an appointment at the Genius Bar the following day. They looked it over, ran some tests, and everything checked out. Their recommendation was to erase the drive and reinstall the OS. I wasn’t quite ready to part with my data just yet (again, ha — joke’s on me), so they referred me to a data recovery provider.
I reached out, got an initial estimate, and mailed the laptop off to them. After looking the device over they reached out with a quote: ~$2,000. ? We ultimately decided that we were willing to pay that to get our photos and documents back, so we gave them the go-ahead and waited to hear back.
They called a couple days later with the bad news: they weren’t able to recover anything. The master encryption key to the drive was corrupted, so they weren’t able to extract anything. ? On the plus side, we didn’t have to pay $2,000. But we were still back where we started: no photos and a dysfunctional laptop.
No, really, it’s gone #
I spent the next two weeks trying everything I could think of to either recover the key or erase the drive and move on.
First up was just getting the thing to boot. I mentioned earlier that it refused to find the OS X Recovery System. Luckily Apple provides functionality to boot into OS X Recovery via the Internet; unluckily it takes for.ev.er., even on a decent broadband connection.
This clock has absolutely no idea how to count. pic.twitter.com/9vXw4Ci1Ty
— Tyler Stromberg (@aqua_geek) September 22, 2015
For added fun, after several failed attempts to resurrect the drive and multiple reboots OS X Recovery just gave up altogether.
This just keeps getting worse. pic.twitter.com/31gOBojn0Z
— Tyler Stromberg (@aqua_geek) September 23, 2015
Once I finally succeeded at booting over the Internet I launched Disk Utility. At this point, we wrote off the drive and its contents as a loss and I just wanted to erase it and start salvaging data from wherever I could. The MacBook, however, had other ideas.
I… just, stop. Please. Just wipe the disk already and let me get on with trying to salvage what I can of my files. pic.twitter.com/6ea9E5fI1a
— Tyler Stromberg (@aqua_geek) September 23, 2015
Disk Utility would happily “erase” the drive, but it hung on “Waiting for the disks to reappear.”
Next I tried booting the laptop into Target Disk Mode. It refused to mount the disk, but I was able to image it. (You know, just in case we somehow figure out a way to reduce the time it would take to brute force the encryption to less than billions of years.) Still no dice getting the disk to erase, though.
While waiting for things to happen through all these attempts, I started reading through some docs on the Internet about the technical details behind FileVault. They directed me to an encrypted file on the recovery partition that contains the drive’s master key.
The last thing you want to see in an encrypted file is plaintext like this:
DYLD_NO_FIX_PREBINDING=1^@__CF_USER_TEXT_ENCODING=0x0:0:0
Somehow, the master key to my hard drive got overwritten with garbage. ?
I finally admitted defeat: everything on the drive is effectively lost but the MacBook won’t let it go.
I took it back to the Genius Bar to have Apple replace the hard drive, as it still wouldn’t allow me to wipe it. Over a month after this saga began I finally had a laptop with a working hard drive again. It was a painful lesson to learn, but we’re taking backups a lot more serious these days.
In Closing #
Bottom line: back up your data. And remember that a backup isn’t a backup until you’ve restored from it.
If you do decide to use FileVault for full-disk encryption, I would highly recommend you make a backup of your encryption key. Running the following command will make a backup of the file that contains the encrypted volume master key (among other things):
sudo csdiagnose
It will spit out a tarball of the file and other metadata — you’ll want to back that up somewhere safe (i.e. not on your encrypted hard drive).