Aug 18 2011

The panic and the data

To set the scene for those who are yet to know me well enough; I have a lot of technology. Not a lot by most people’s standards, a lot by the standards of a shop like BestBuy, PC World, Maplin…well you get the idea, to a normal human’s view, I could open a shop with the stuff I don’t use on a daily basis!

Among this ridiculous collection of crazy contraptions, shiny things and boxes that go “vvvzdt” there is a decent array of data storage. Some NAS, some just stacked in computers, some semi-portable, external, internal…you get the idea. Part of that collection comes in the form of two Synology Diskstations.

These are beautiful little white boxes, flashing lights, put them in a corner, magic tech toys. You put a hard drive in them, set them up and enjoy! I have been for quite a while! One has all of my work, my portfolio, my crucial data; the other houses all of my media, my photograph collection, all of my music (now leaving my 320 CD collection simply as the ‘backup’) and all of my video files (the missus and I both had huge collections before we merged them…that was a longer time ripping than the music!).

Now for those of you reading this without the benefit of 10 years experience in the IT industry, experience of IT forensics, data recovery, deep level computer hardware knowledge and/or a ‘Matrix’ style neural interface for rapidly learning any topic you like…I shall try and keep this as simple as I can…{and put explanations in these brackets}

The long and short of getting to this point now, the point at where I feel the desire to chronicle this expedition, was the result of a list of unfortunate events; events that went thus:
I was given a 2TB {rather large} hard drive as payment for knowledge. The drive however had a couple of bad sectors {broken bits}. This was only a problem for the person who gave me the drive as it was for their business and they had the redundancy in a RAID array anyway {they just swapped this one for a new one}.
I bought myself a nice new Diskstation to wrap around my nice new hard drive.
*insert a couple of months*
I carefully spent several weeks organising and sorting all of my data on my entire network to bring all of my media together onto the new drive.
Attempting to do a system update on the new Diskstation (let’s now call it ‘Mediastation’) I discovered there was a little issue.
A short while after discovering that the update wouldn’t take, our energy supplier had a bit of an issue. Without warning, all of the electricity on the street where I live vanished.

*here is the start of the fun*

When the power came back, I fired everything back up to check it was all ok (as I was in the middle of a couple of bits at the time too) I noticed that I couldn’t connect in the usual way to the Mediastation.

After investigation, the usual checks and then the last step of Synology’s own management software, it seemed that the Media station had lost its management information and believed itself to be a completely fresh, new system.

Crap! – I thought (what I said out loud was a little different but along similar lines)

Now one thing I already know about Synology is that part of the setup of the management software involves reformatting the entire drive {wiping in clear}. My data is on there! *starts to panic*…all 1.6TB of it {enough for storing 2200 films (at average size once ripped), or about 32,000 average music albums as mp3}. Once I got my pulse back to a steady 60bpm I thought logically and rationally.

Now I’ve been actively in the IT industry for the last 10 years, starting the first incarnation of my IT business at 15. I’ve also been doing data recovery since around that time too. Over the years I’ve tried many different pieces of software on the market, some good, some not so good. With careful selection I have my trusty collection of tools that normally serve me beautifully when it comes to any recovery I have to do these days.

Since learning the craft all the time ago, I’ve brought back information from broken drives, broken partitions {like C:\ and D:\}, corrupted file systems {how your computer reads the information}, and the usual suspects of accidental reformatting or deletion (initial mistake using fdisk all them years ago that drove me to learn in the first place). So I thought this shouldn’t be too difficult compared to all that…

When setting up one of these Synology systems, the drives are split with a system partition for the core software (as it’s basically a miniature Apple Mac), then the main data partition. It’s a Linux based operating system too, so it’s all formatted to ext3 {just smile and nod here}. Simple I thought, I will throw the drive into my spare Ubuntu machine, take the data off the drive, start again from scratch and move everything back over…

Problem number one!

It turns out that the drive didn’t have any visible partitions, it didn’t even have any invisible partitions either.

Not a problem! – I thought. I will just plug it into one of my Windows machines and get cracking the usual way…external drive, forensic software, make sure all the data is intact…

Problem number two!

The drive needed to be initialised in Windows; that’s normally only needed when you first plug in a new drive, as in a brand new, fresh out the packet new drive. Hmmmmm, never a good sign! Further investigation with all the usual toys, tricks and tools provided little in the way of encouragement. Short of my old faithful forensic package; a very powerful hex editor {again, smile and nod please} which at least showed me that the data was still all there, was intact (for the most part) and just needed to be brought back to life.

My tech instincts kicked in here at this point. I knew this sort of procedure could potentially be destructive so I needed to do this properly. Thankfully after some careful negotiation, I was fortunate enough to acquire another 2TB drive (long story, don’t ask) and after a fully day of anticipation, forensically cloned the entire drive. At least now if I try and recover from the new drive, break it or damage the data, I can still clone it again.

This is pretty much standard procedure for any forensic outfit simply for that reason, you leave the original drive and data untouched and you have the freedom to do anything needed to bring it back on the other drive, safe in the knowledge that things will be ok in the end. Needless to say, the average human isn’t in the position to go out and just buy another drive the same size…hence my very careful negotiation to get my new drive as a donation.

Thinking about the drive structure and how the data is laid out, logically, repairing the partitions would be the best place to start.

Problem number three!

As it happens, discovering after the best part of 2 full days (48 hours) of scanning, the majority of the partition information is actually not there. There are no partitions to find, to rebuild, it’s simply not there. Typically, of all the places for the bad sectors to choose, they seem to have decided to land right in the one place I didn’t want them too; right in the (metaphoric) page number section. The computer simply knows there are words in the book but can’t look for page numbers.

To make things as easy to understand as possible for this next part; think of a dictionary, all in order, you go to the start, it says it’s in alphabetical order. You want to find the word “monumental”, you flick to the section with words starting with the letter M, find the word, read the description. That’s how drives on Windows normally store data (I know, I know, it’s more involved than that but *points* you explain it to them!). With this drive (remember me mentioning ext3 before?) think of the dictionary again, you look at the start, it’s in a slightly different order. Rather than A-Z it’s in Superblock 1-26 (ish). You want to find the word “arse”, head to Superblock 1 (as it’s a good place to start), you see that there a number of things called ‘inodes’.

You look between Superblock 1 and 2, there is a big list of descriptions for all manner of words…just no words. Inode-3255:A round fruit, normally red or green, quite tasty, eaten daily is supposed to keep the doctor away. Inode-6342:Girls name, name of a play in which the sun will come out tomorrow. Inode-7345:Insect, 6 legs, carries lots of stuff and turns up in your kitchen during the summer. Ok so you get the idea! We flick back and have a look at the Superblock again to see Inode-3255:Apple, Inode-6342:Annie, Inode-7345:Ant.

Again, this is heavily simplified but I’m guessing you don’t quite think I’m the same level of crazy as you did before reading this. Moving on, several days later, several other pieces of software later and I reached a point of possible meltdown. Nothing would either find the containers for the files or the right data. I was really really starting to panic at this point; nearly a week of properly hard slogging and still coming up short. Then a breakthrough! Something found some data!…several hundred thousand files!!…all named “00001234”, “00001235”, “00001236” and so on!!!

Well, at the very least I knew I would be able to get my data, just that I would have to spend the next few years sorting it all up, finding the names for everything, putting songs back into order, that sort of thing. It was something, it was a glimmer of hope, a light at the end of the tunnel. Something to stop me from just giving up. I knew that if this was at least possible, then getting the superblocks back, listing the inode codes and then finding the appropriate file name for each inode could lead me back in the right direction. Again, it would take one hell of a long time but it was at the very least, possible!

Next came a day of research, there just had to be a way of doing this. Forum posts, computer scientists breakdowns of things that noone in their right minds should know about, learning things that I know nobody I’ve ever met will actually know about. It’s infuriating but exciting at the same time. Knowing that noone else in my group of friends is actually sad enough to know this sort of stuff. I’m elevating myself to a new level of geekdom purely in the name of geek! Granted I know it’s something I usually do on a daily basis too but this sort of thing is just how I get my kicks. Some people drink, smoke, take drugs, go extreme ironing or do other crazy things; I just geek till it MHz!

Megahurt it did, my brain was ready to dribble out of my ears. Noone seemed to know, nobody had actually documented this before in a way that was easy enough to find. There are a number of companies that you can send your drive too but with the amount of data I needed to recover the bill would have topped £9000.

Then I found it! The Holy Grail! Venus’s arms! Rocking horse poo! Perpetual motion! A piece of software that actually does what I’m trying to do!!!!!!!!!!!!

FUCKING BRILLIANT!!! – I thought, and quietly shouted at myself.

After a few hours of playing with the software, it knew the drive had data, it knew it had once held ext3 partitions, it knew I wanted to find the data that is on those ext3 partitions. All I had to do was set the big scan running for it to find the traces of the files, the traces of the superblocks, the list of inodes, folders, filenames, piece it all together and present me with a list of things to click and recover on screen. I pressed the button knowing this was going to be a huge weight off my mind. The videos of my childhood would be back, the copies of my Dad’s songs, he albums would be back in my possession, my photos of my Dad and step Mum’s wedding, pictures of my baby sister, my baby brother for that matter too, granted he’s 8 now but he was a baby when I took the pictures! I would have it all back!!! – Oh the irony that I used this Mediastation to temporarily sort everything out before I backed the main stuff up!

After a good 24 hours or so of scanning, the screen was showing promise, it was about a 6th of the way through the drive, but had pretty pictures, blocks of colour with a nice key at the bottom. It had found over 700 Superblocks, buckets of files, this was looking good. I frequently checked back on progress just for my own amusement, watching my data be found, one pink square on the screen at a time.

Problem number four!

I made the fatal mistake of having to work, to not be sat in front of the screen for the predicted 6 full days of scanning that it was due to take. I didn’t fancy 144 hours straight as a sitting. Granted, I’ve probably not been far off doing that a few times but this time I decided not too. I would just monitor it remotely every couple of hours from my laptop while I was doing other household duties or enjoying the evening time with my family. The evening of the second day, after 48 hours or so of scanning; I knew it was about a third of the way through because of the progress it had been making. But I couldn’t connect to the machine remotely, something was up. It was late at night, my daughter was sleep, just me and my good lady relaxing…or trying too, now I couldn’t, I had to go and check.

I ran into my computer room, my office type space, there it was, a black screen, just one line of white text:


I couldn’t believe it! It could have only been one possible thing. An automated Windows Update!!! That one thing that seems to know exactly the point at which you don’t want anything to happen on the computer, it decides that your computer should now be restarted!

Problem number five!

Now when you play about with very powerful software, something like a restart is something that has to be prepared for. Especially when you are playing with the very things that Windows needs to be able to run; correct partition information! Now because I was running 3 drives in this machine, one for Windows, 2 for the data recovery, I had to try and safeguard myself as much as possible. Was my data ok?…it should have been, I did the drive clone, I could still clone it again if I needed too. But either way it had set me back. I had to first fix the PC!

Bloody typical! – I thought (not quite what I actually said, but if you want the full rant then check my facebook page for that explicit little number)

To cut a very very long story only slightly shorter…After fixing windows I had to do a day long, high level scan just to make sure the data was still intact enough to try another long, low level scan and rebuild. Thankfully it was!
The progress is long, arduous, laborious, but actually working! As in really, properly, actually working! Data, filenames, folder structure, the lot!

At this point, it’s pretty much plain sailing now, I’ve managed to recover all 17,000 of my pictures and I’m plodding through the next stage, video!

Now I would like to take this final line to congratulate you on actually managing to read through (or just scroll down) to see this. So remember, always be nice to geeks, it’s their obsession with technology that fixes the things you break!