Long term NetWare recovery

Are you still backing up Novell NetWare hosts? If you are, I hope you’re actively considering what you’re going to do in relation to NetWare recoveries in March 2010, when NetWare support ceases from both Novell and EMC.

I still have a lot of customers backing up NetWare hosts, and I’m sure my customer set isn’t unique. While Novell still tries to convince customers to switch from traditional NetWare services to NetWare on OES/SLES, a lot of companies are continuing to use NetWare until “the last minute”.

The “last minute” is of course, March 2010, when standard support for NetWare finishes.

Originally, NetWare support in NetWorker was scheduled to finish in March 2009, but partners and customers managed to convince EMC to extend the support to March 2010, to match Symantec and co-terminate with Novell’s end of standard support for NetWare as well.

Now it’s time we start considering what happens when that support finishes. Namely:

1. How will you recover long term NetWare backups?
2. How will you still run NetWare systems?
3. How will you manage NetWorker upgrades?

These are all fairly important questions. While we’re hopeful we might get some options for recovering NetWare backups on OES systems (i.e., pseudo cross-platform recoveries), there’s obviously no guarantees of that as yet.

So the question is – if you’re still using NetWare, how do you go about guaranteeing you can recover NetWare backups once NetWare has been phased out of existence?

The initial recommendation from Novell on this topic is: keep a NetWare box around.

I think this is a short-sighted recommendation on their part, and shows that they haven’t properly managed (internally) the transition from traditional NetWare to NetWare on OES/SLES. This is perhaps why there isn’t a 100% transition from one NetWare platform to the other. Being faced with unpalatable transition options, some Novell customers are instead considering alternate transitionary options.

Unfortunately, in the short term, I don’t see there being many options. I’m therefore inclined to recommend that:

1. Companies backing up traditional NetWare who only need to continue to recover a very small number of backups consider performing an old-school migration – recover the data to a host, and backup on an operating system that will continue to enjoy OS vendor and EMC support moving forward.
2. Companies backing up larger amounts of traditional NetWare should consider virtualising at least one, preferably a few more NetWare systems before end of support, and keeping good archival VM backups (to avoid having to do a reinstall), using those systems as recovery points for older NetWare data.

The longer-term concern is that the NetWare client in NetWorker has always been … interesting. Once NetWare support vanishes, the primary consideration for newer versions of NetWorker will be whether those newer versions actually support the old 7.2 NetWare client for recovery purposes.

With this in mind, it will become even more important to carefully review release notes and conduct test upgrades when new releases of NetWorker come out to confirm whether newer versions of the server software actually support communicating with the increasingly older NetWare client until such time as recovery from those NetWare backups is no longer required.

You may think this is a bit extreme, but bear in mind we don’t often see entire operating systems get phased out of existence, so it’s not a common problem. To be sure, individual iterations or releases may drop out of support (e.g., Solaris 6), but the entire operating system platform (e.g., Solaris, or even more generally, Unix) tends to stay in some level of support. In fact, the last time I think I recall an entire OS platform slipping out of NetWorker support was Banyan Vines, and the last client version released for that was 3 point something. (Data General Unix (DGUX) may have ceased being supported more recently, but overall the Unix platform has remained in support.)

If you’re still backing up NetWare servers and you’re not yet considering how you’re going to recover NetWare backups post March 2010, it’s time to give serious consideration to it.

Basics – Directed Recoveries

Hi,

This blog post can now be read at the NetWorker Information Hub.

Recovering with scanner and uasm

There are some types of recoveries that fall into the “last ditch effort” category – everything else has been tried, but the data just won’t come back. I would have to say that in 99.99% of cases this is due to one of the following two things:

1. The data was never properly backed up in the first place.
2. Cloning wasn’t done, and the only piece of media that holds a particular backup is broken.

Assuming the data was actually backed up, as a last thing to try*, you can recover filesystem data using scanner and uasm. This is documented in the man page for scanner – and also available as a PDF to Windows administrators in the command reference guide.

As per the man page, the way of running scanner and uasm to facilitate a recovery is as follows:

# scanner -S ssid device | uasm -r -v -m /source=/dest

or, if you prefer to avoid using the pipe,

# scanner -S ssid device -x uasm -r -v -m /source=/dest

Where in each of those commands:

• ssid is the saveset ID you want to recover from
• device is the path to the device where the volume holding that saveset can be accessed
• /source is the source path you’re recovering that you want to replace with:
• /dest is the replacement to the source path

Out of these, probably the “/source” and “/dest” makes the least sense, until you see an example.

Say for instance you have a backup of “/home” that you want to recover this way. Chances are, you don’t want to recover it on top of an existing /home; thus, you’d use “-m /home=/tmp/recovery”, or something like that. This instructs uasm to replace the “/home” path in the incoming data stream with “/tmp/recovery” when writing the data back out.

However, it’s not just a saveset ID argument that scanner takes in this mode; in fact, you can pass through most arguments in relation to restricting what is scanned – that is, clients, saveset names, and even file/record numbers.

If you’re dealing with potentially damaged media, the file/record numbers can sometimes be your saving grace. Say you’re doing a recovery and mid-way through the recovery NetWorker aborts saying there’s an error at file number 33 on the tape. At this point, you should at this point have a clone. Honestly, you really, really should have a clone.

Assuming though that for some reason you don’t have a clone, there’s a good chance your data is hosed** – recover, nwrecover, winworkr are not going to get you past that point.

Scanner might. There’s no guarantee, but it might. If the media error is limited to just a single portion of the tape, you may find that you can run a scanner/uasm combination that starts at the next file marker and hope that it gets your data back. Assuming we’re needing to get /home back for the client ‘baal’ on /dev/nst0***, this would make the command

# scanner -c baal -N /home -f 34 /dev/nst0 | uasm -r -v -m /home=/tmp/recovery

If you need to get even more data back, you could even use a starting record number. Say the recovery failed at file number 33, media record 1138; you might try the following:

# scanner -c baal -N /home -f 33 -r 1139 /dev/nst0 | uasm -r -v -m /home=/tmp/recovery

However, in each case, remember that the extent to which scanner and uasm will be able to recover your data will be limited to the amount of physical damage on the tape.

As I mentioned before, there’s no guarantees scanner is going to … ahem, save your bacon … if things have reached this point – but, what is there to lose by trying?

—
* This is also a valid technique where you have a decommissioned server and you just need to urgently pull back a few files without doing a full rebuild.

** And it’s not NetWorker’s fault of media or devices go bad, and it’s the only copy of the data!

*** If you’re needing to do this because of a media failure, make sure you use a different tape drive for your “last attempt”, just in case it was the tape drive that caused the failure.

Difficult ‘recover’ scenarios

Even though I usually avoid using GUIs for recoveries*, given my main workstation is a Mac, I don’t have the option of using a NetWorker GUI for personal recoveries anyway.

Over time I’ve become one of those users that many Unix sysadmins dislikes – I name files and folders with prefixes including:

–

*

and

#

Heck, I even use Ω and ∑ as directory prefixes.

It caught me by surprise then when I tried to recover a directory called “-Proposal”. My natural inclination was to go to the parent directory of “-Proposal” and type:

recover> add -Proposal

usage: add [-q] [filename] – add `filename’ to list of files to be recovered

As you can see, that didn’t particularly work.
Nor did the following:

recover> add — -Proposal

usage: add [-q] [filename] – add `filename’ to list of files to be recovered

Nor did:

recover> add ‘-Proposal’

usage: add [-q] [filename] – add `filename’ to list of files to be recovered

As you can imagine, it was starting to get a little bit frustrating.

To cut a long story short, in a scenario where you need to recover a directory that starts with a dash, you need to do something along the lines of the following:

1. If the directory still exists, change into that directory in the shell, and run recover from there, or
2. Add the parent directory, then exclude the files/directories you don’t need recovered, or
3. If the directory doesn’t exist, make the directory, change into that directory in the shell, and run recover from there.

None of these are ideal solutions, but they do work. I hope, if you need to recover such a directory, you manage to stumble across this tip or you remember it – there are few things worse than worrying that something you really need to recover seems an impossibility.

—
* If you can’t turn off a file-by-file selection when you’re adding 10,000,000 files, a GUI is painful.