Mistakes you don’t want to make

Many years ago, a company switched from ArcServe to NetWorker. They did so around the time they made their end of year backups, the ones that they intended to keep ‘forever’ for legal requirements.

Fast-forward several years, and it was requested to recover Lotus Notes backups from those original end of year archives. That’s when the support call came through. You see, those end of year archives were done on a standalone tape drive, not a tape library, and both tapes had, say, ‘YEAR2002’ written on the label. There was a little “1” noted on the first label, and a little “2” noted on the second label. For convenience, we’ll call them the first and second tapes.

When they put the first tape into the library for recovery, their first issue was getting NetWorker to mount the tape, since it didn’t have a barcode. Some non-GUI commands later, the tape was in the drive, but NetWorker wouldn’t keep the tape mounted – every time they tried to mount the tape, NetWorker threw up an error saying that it was expecting tape YEAR2002 with a particular volume ID, not YEAR2002 with a different volume ID that wasn’t in the media database. The second YEAR2002 tape would mount though, but NetWorker couldn’t perform a recovery because all the media wasn’t available.

So, here’s what happened:

• The manual backup was run of a bunch of systems and Lotus Notes.
• A tape was labelled YEAR2002 within NetWorker, and the backup ran until the tape filled up.
• A new tape was put into the tape drive, and since they had no exposure to NetWorker, they labelled that tape as YEAR2002 as well and the backup went on its way.

I’ll qualify here – the Lotus Notes backup was done using the module.

Now here’s the thing – while NetWorker works on the volume ID being unique, it also works on the volume label being unique as well. It won’t support two volumes in the media database at the same time with the same label. It gets pretty strident about that if you try to label one tape with another tapes’ label, but I guess if you’re new to NetWorker it might just seem like there’s a bunch of confirmation boxes you have to click before you can label your next tape.

So the net result was that the backup was written to two pieces of media that couldn’t co-exist in the media database at the same time. Scanning the first necessitates removing the second from the media database, and because this isn’t a filesystem backup, there are limitations that couldn’t be stepped around in recovering from partial savesets.

For a regular filesystem backup as a last resort this still would not be impossible to recover from – using scanner and uasm you can still suck the data off the tape(s) without NetWorker needing both in the media database. Tedious, and not as good as just being able to select data in a recovery program, but it’s better than no recovery at all. But you can’t use scanner and uasm for a non-filesystem recovery

(You also can’t write a new tape label to a fresh tape, then dd the NetWorker data after the label on the other tape onto the newly labelled tape. The volume ID (or some other unique volume identification system) is written into the savestream, and transferring that savestream onto another volume sees NetWorker reject it if you subsequently attempt to scan it.)

Net result? Data that could not be recovered short of sending it off to a specialist forensics data recovery company.

NetWorker’s fault? No. There is after all, only so much that software can do in order to prevent you from shooting yourself in the foot.

Recovering with scanner and uasm

There are some types of recoveries that fall into the “last ditch effort” category – everything else has been tried, but the data just won’t come back. I would have to say that in 99.99% of cases this is due to one of the following two things:

1. The data was never properly backed up in the first place.
2. Cloning wasn’t done, and the only piece of media that holds a particular backup is broken.

Assuming the data was actually backed up, as a last thing to try*, you can recover filesystem data using scanner and uasm. This is documented in the man page for scanner – and also available as a PDF to Windows administrators in the command reference guide.

As per the man page, the way of running scanner and uasm to facilitate a recovery is as follows:

# scanner -S ssid device | uasm -r -v -m /source=/dest

or, if you prefer to avoid using the pipe,

# scanner -S ssid device -x uasm -r -v -m /source=/dest

Where in each of those commands:

• ssid is the saveset ID you want to recover from
• device is the path to the device where the volume holding that saveset can be accessed
• /source is the source path you’re recovering that you want to replace with:
• /dest is the replacement to the source path

Out of these, probably the “/source” and “/dest” makes the least sense, until you see an example.

Say for instance you have a backup of “/home” that you want to recover this way. Chances are, you don’t want to recover it on top of an existing /home; thus, you’d use “-m /home=/tmp/recovery”, or something like that. This instructs uasm to replace the “/home” path in the incoming data stream with “/tmp/recovery” when writing the data back out.

However, it’s not just a saveset ID argument that scanner takes in this mode; in fact, you can pass through most arguments in relation to restricting what is scanned – that is, clients, saveset names, and even file/record numbers.

If you’re dealing with potentially damaged media, the file/record numbers can sometimes be your saving grace. Say you’re doing a recovery and mid-way through the recovery NetWorker aborts saying there’s an error at file number 33 on the tape. At this point, you should at this point have a clone. Honestly, you really, really should have a clone.

Assuming though that for some reason you don’t have a clone, there’s a good chance your data is hosed** – recover, nwrecover, winworkr are not going to get you past that point.

Scanner might. There’s no guarantee, but it might. If the media error is limited to just a single portion of the tape, you may find that you can run a scanner/uasm combination that starts at the next file marker and hope that it gets your data back. Assuming we’re needing to get /home back for the client ‘baal’ on /dev/nst0***, this would make the command

# scanner -c baal -N /home -f 34 /dev/nst0 | uasm -r -v -m /home=/tmp/recovery

If you need to get even more data back, you could even use a starting record number. Say the recovery failed at file number 33, media record 1138; you might try the following:

# scanner -c baal -N /home -f 33 -r 1139 /dev/nst0 | uasm -r -v -m /home=/tmp/recovery

However, in each case, remember that the extent to which scanner and uasm will be able to recover your data will be limited to the amount of physical damage on the tape.

As I mentioned before, there’s no guarantees scanner is going to … ahem, save your bacon … if things have reached this point – but, what is there to lose by trying?

—
* This is also a valid technique where you have a decommissioned server and you just need to urgently pull back a few files without doing a full rebuild.

** And it’s not NetWorker’s fault of media or devices go bad, and it’s the only copy of the data!

*** If you’re needing to do this because of a media failure, make sure you use a different tape drive for your “last attempt”, just in case it was the tape drive that caused the failure.