NetWorker Blog

Commentary from a long term NetWorker consultant and Backup Theorist

  • This blog has moved!

    This blog has now moved to Please jump across to the new site for the latest articles (and all old archived articles).



  • Enterprise Systems Backup and Recovery

    If you find this blog interesting, and either have an interest in or work in data protection/backup and recovery environments, you should check out my book, Enterprise Systems Backup and Recovery: A Corporate Insurance Policy. Designed for system administrators and managers alike, it focuses on features, policies, procedures and the human element to ensuring that your company has a suitable and working backup system rather than just a bunch of copies made by unrelated software, hardware and processes.
  • This blog has moved!

    This blog has now moved to Please jump across to the new site for the latest articles (and all old archived articles).



  • Twitter

    Error: Twitter did not respond. Please wait a few minutes and refresh this page.

Posts Tagged ‘Mac OS X’

NetWorker 7.6 and Mac OS X 10.6 (Snow Leopard)

Posted by Preston on 2009-11-23

While still not appearing on the NetWorker compatibility lists, it would certainly appear from testing that NetWorker 7.6 plays a whole lot nicer with Mac OS X 10.6 (Snow Leopard) better than its predecessors did.

Previously when Snow Leopard came out, I posted that NetWorker 7.5 was able to work with it, but did qualify that if you were backing up laptops or other hosts that periodically changed location, NetWorker would get itself into a sufficient knot that it would become necessary to reinstall and/or reboot in order to get a successful backup. (In actual fact, it was rare that a reboot would be sufficient.) Fixed-point machines – e.g., servers and desktops – typically did not experience this problem.

It would seem that 7.6 handles location changes considerably better.

If you’re experiencing intermittent problems with NetWorker 7.5 not wanting to work properly on Mac OS X 10.6 hosts, I’d suggest upgrading to NetWorker 7.6 as a test to see whether that resolves your problems.

The usual qualifiers remain – read the release notes before doing an upgrade.

Posted in NetWorker | Tagged: , , , | Comments Off on NetWorker 7.6 and Mac OS X 10.6 (Snow Leopard)

Aside – Parallels Desktop 5 for Mac OS X

Posted by Preston on 2009-11-13

I’ve been using Parallels Desktop for the Mac for several years now – in fact, when I originally started using it, VMware were only talking about doing a desktop virtualisation product for the Mac.

That’s partly why I stay loyal to Parallels – they supported the platform sooner. The other reason is that after years of hearing tripe from VMware employees about why you couldn’t mix windows from both operating systems, Parallels went ahead and did it with Coherence. (Yes, VMware Fusion’s Unity functionality went there too, around the same time, but the amount of times I heard it wasn’t a feature they were interested in within Workstation, as an example, drove me nuts.)

So when Parallels Desktop v5 for the Mac came out, I jumped on the upgrade bandwagon. It’s given me one major positive – I can now run Solaris 10 AMD 64-bit on my Mac Pro; that was the one remaining OS I absolutely need to periodically run that I was being blocked from doing previously. After about 3 days of installing, reinstalling, downloading more recent versions of Solaris 10 AMD, I even finally managed to get networking operational within the VM too, which meant it was useful.

Other than that, Parallels v5 has been a bit of a disappointment. You see, currently for me, Coherence mode only works if I’ve got a single monitor attached to my machine. The only time that happens is when I’m using my laptop away from my desk, or when I’m traveling – and those are times I’m less likely to run VMs. Since Coherence doesn’t work for me 99% of the time, that means I can’t really try out the Crystal mode – though I’ll admit, I’m unlikely to use it heavily; I dislike the “shared apps” approach offered by Parallels, and the new Crystal view mode seems to rely heavily on this.

It does continue to annoy me that I don’t have the option of virtually turning the monitor off – why can’t I close the console for a running VM? Surely that’s not so huge a thing that it can only be limited to enterprise class virtualisation – aka ESX, VMware Server or Parallels Server. When you have 10+ VMs running at once, even minimised all those consoles start to get annoying.

While overall I’m pretty happy with Parallels performance in terms of memory and CPU, recent support cases have highlighted to me that when it comes to translated IO, Parallels struggles – it seems to peak at about 20MB/s per VM, regardless of the throughput capabilities of the attached device. I first noticed that under v4, and was unhappy to see no change in v5.

If you’re currently a Parallels Desktop for Mac user, and you haven’t yet upgraded, I’d suggest holding off until the next build of v5, rather than jumping into the initial build released. Hopefully by then they’ll at least have Coherence reliably working again.

[Edit, 2009-11-14]

I’d like to take back my comments about Parallels 5 still having mediocre IO performance. I just realised, a short while ago, that one of the VMs I’d been testing with had failed in its VMware Tools update. Now, with both VMs in this particular test config updated to Parallels Tools v5, instead of getting 14-17MB/s transfer speed between them as I’d been getting under Paralles v4, I’m now getting 45-57MB/s. Now that’s a performance improvement.

Posted in Aside | Tagged: , , , , , , | Comments Off on Aside – Parallels Desktop 5 for Mac OS X

NetWorker and Snow Leopard, Redux

Posted by Preston on 2009-09-18

When Snow Leopard first came out, I was reasonably impressed with how easily NetWorker continued to operate with it – and for desktop users and administrators of fixed-location servers, that should remain the case.

For laptop users though, it’s turning out to be slightly different story. My ongoing experience now is that if I switch locations repeatedly (e.g., home to work, work to customer site, customer site to work), the NetWorker client daemons eventually get so bogged down that it’s necessary to reboot to get back to working backups. In fact, a couple of times I’ve needed to go so far as to reinstall NetWorker on my laptop in order to get it running again smoothly. (That’s using NetWorker 7.5.1.)

If you’ve got mobile users upgraded to Snow Leopard who are now experiencing backup problems, a reboot (unfortunately) may be your first point of call – the nature of the daemon hang-up seems to prevent proper process shutdown, which in turn prevents the daemons from properly restarting. If the reboot fails, a client reinstall should fix it.

From my experience so far, it seems to only happen when locations are changed multiple times.

Posted in NetWorker | Tagged: , | Comments Off on NetWorker and Snow Leopard, Redux

NetWorker and Snow Leopard

Posted by Preston on 2009-08-28

I was rather pleased this morning to have a friendly FedEx courier drop off Mac OS X 10.6 – Snow Leopard.

Of course, my first thought was “will this work with NetWorker?”

I’m pleased to report – yes, yes it does. All of the following worked for me using NetWorker 7.5.1:

  • Recoveries from prior to the upgrade
  • Backups following the upgrade
  • Recoveries from new backups only
  • Recoveries that mixed old backups and new backups

(That’s just standard filesystem recoveries – I don’t use NetWorker for complete disaster recovery of Mac OS X as I feel that Time Machine is a much more appropriate system for those styles of recoveries.)

So, now we just need to wait for the software compatibility guide to be updated…

Note: The installer is very different under Snow Leopard, and no longer supports the old “Archive and Install” option; thus, when you finish the installation, there’s no need to actually re-install NetWorker; it remains there, and working, from the previous installation.

[Edit, 2009-09-04]

I can report one bit of odd experience with NetWorker under Snow Leopard. I was presenting a training course earlier in the week and wanted to change which hosts could backup the NetWorker client on my laptop. After editing the /nsr/res/servers file I found that the NetWorker client processes wouldn’t properly restart on the laptop. In the end, I did a new install of NetWorker onto the laptop, which fixed the problem. FYI, in case you notice similar things, reinstalling seems to fix the problem.

Posted in NetWorker | Tagged: , | 4 Comments »

Wishlist – Server/Storage Node for Mac OS X

Posted by Preston on 2009-08-13

For some time I’ve wished NetWorker would support both storage node and server functions on Mac OS X. When I had a PPC 17″ PowerBook, this mostly came from the glacially slow performance of running Linux within VirtualPC so as to run up a NetWorker server for testing. (Windows-within-Virtual PC was a dead-loss: the then-current version of NetWorker would not even start within VirtualPC.)

Since Apple made the jump to Intel machines, running a NetWorker server for lab work within a virtual machine has been far more efficient, given that now it’s just virtualisation rather than emulation. However, I’ve been thinking for a while that given the performance options available on Mac OS X, and the amount of data frequently stored on Mac OS X machines, not supporting at least a storage node is foolish.

Now that I have a Mac Pro, my personal belief is that it’s crazy not to support Mac OS X both as server and storage node.

Why, you may ask, would I think this? Is it just some weird combination of the “Mac Fan Boy” and “NetWorker Fan Boy” that I want them joined at the hip like some bizarre Doctor Frankenstein experiment?

[Here’s an aside. Why is it that people who defend Apple, and Macs, are immediately declared to be Apple Fan Boys, when PC/Windows users just as vehemently defend their own platforms declare themselves ‘realistic’? There’s only one answer: sad hypocrisy. Defending one platform is “hysterial frothing at the mouth buy-in to the reality distortion effect of Steve Jobs”, whereas equally defending another platform is “logical”. Please, spare me.]

So, jumping off that little soap box, I do actually have a method to my madness here. I honestly think, bang for buck, that the Mac Pro (using Apple’s high end machines as a reference point) represent the sort of significant processing and expansion capability often sought in backup servers. I snapped up a bargain previous generation Mac Pro that features that Intel Xeon 5400 CPUs rather than the current top-of-the-line Nehalem based processors, but this machine has serious processing power. The reason it’s called a workstation by Apple is because of it’s ability to handle complex graphics – but in reality it’s basically a server in a nice shell. With 8 x 3.2GHz cores and (currently) 12GB of RAM, this is a machine that just absolutely flies at data throughput. With expansion of up to 32GB of RAM, Mac Pros represent in one shiny shell more than enough processing power to run a backup server/storage node for any sized business*.

For companies that are space-conscious, there’s the “server” version of the Mac Pro, the Xserve, which is quite a powerful host in a 1RU enclosure.

Given the client software has already been ported to Mac OS X, the hard work has effectively already been done; server and storage node options are not going to take a significant amount of development effort.

Is there justification in porting server and storage node to Mac OS X? The cynical part of me wants to answer that there’s a hell of a lot better justification in porting server/storage node to Mac OS X than there was in porting the client to Linux PPC, but undoubtedly that would have been done to service some large-scale deal for EMC – i.e., there would have been significant business-incentive to do so.

Is there a business incentive for supporting more than client capabilities on Mac OS X? Well, market share is continuing to grow, as evidenced by Microsoft breaking what is almost universally acknowledged as the golden rule of advertising**. Then there’s the high frequency of use of Mac OS X systems in academia. This may not seem a compelling business case for EMC now, but let’s think a little longer-term – as more and more people become exposed to Macs again during their education (either secondary or tertiary), that exposure is going to influence them in their buying decisions as they move into employment. I.e., short of some catastrophic collapse***, Apple is going to see market share continue to increase – not flatten, not drop, but continue to increase.

In the short term though, another compelling reason is where Apple’s market share is at its highest – multimedia: graphic design, advertising, etc., all feature large amounts of data storage. While there’s some support for client software within Mac OS X, the backup server market in that arena is owned almost exclusively by Retrospect. (Retrospect is a good product, but it is still reasonably limited – definitely a workgroup, rather than an enterprise product.) In short, it seems mad that machines that routinely store tens or more terabytes of storage are denied storage node/dedicated storage node capabilities.

Now, some might argue that the lack of support for Sybase DBAnywhere (which powers GST, the back-end to NMC) would be sufficient cause to stop at the client (or at most, the storage node); after all, if you can’t run the GST/NMC server on the backup server, what’s the point? I have two (I believe valid) responses to this: first, it’s reasonably common to see separation between NMC/GST services and the NetWorker server, not only in environments that have multiple backup servers, but also just for reducing the potential for one service impacting the other. Secondly, there’s already examples of NetWorker server platforms that don’t have an accompanying NMC/GST server option – Solaris/AMD springs to mind immediately, and I know there’s other examples as well.

I do honestly think the point is rapidly approaching (if it is not already here) where there are more compelling reasons to port NetWorker server and storage node to Mac OS X than there are for not doing so. Architecturally, data storage volumes, increasing market share and an existing client all point to this having solid reasons.

* Note that I’m not saying that they are capable of being the sole backup server for a massive company; just like any other platform, in larger environments, the three-tier approach is always required.

** That rule is that the number one company in an industry should never refer to the number two company in the industry in their advertising.

*** With a market cap that now periodically bounces above Google’s, this seems somewhat unlikely now. (Apple’s market cap now exceeds the combined market cap of HP and Dell.)

Posted in Architecture, NetWorker | Tagged: , , , | Comments Off on Wishlist – Server/Storage Node for Mac OS X

Basics – Stopping and restarting NetWorker on Mac OS X

Posted by Preston on 2009-05-17

If you’re a NetWorker administrator, but not a Mac OS X administrator, you may be unfamiliar with the process to stop and restart NetWorker on that platform. It’s actually easy, but for someone who hasn’t come from a Mac OS X command-line background, it’s not something you’d immediately expect.

To stop and restart NetWorker, all you need to use is the SystemStarter command, which you run from Terminal (or another suitable command line login).

A typical stop/restart sequence will look like the following:

preston@archon ~$ sudo SystemStarter stop NetWorker
Stopping NetWorker Client.
preston@archon ~$ sudo SystemStarter start NetWorker
Starting NetWorker Client.

Or, you could simply run:

preston@archon ~$ sudo SystemStarter restart NetWorker

If you’re wondering where the NetWorker startup/shutdown script is installed, you’ll find it, along with other such scripts, installed in /Library/StartupItems.

Posted in Basics, NetWorker | Tagged: , | Comments Off on Basics – Stopping and restarting NetWorker on Mac OS X

Aside – My take on Time Machine

Posted by Preston on 2009-03-06


Being one of those freaky weird IT people who are passionate about backups*, when Apple first previewed Mac OS X 10.5 (aka Leopard), the number one thing I of course got excited about was Time Machine. Now, before anyone tells me that it’s “just a poor rip-off of VSS”, let me be blunt – analysts who started that talk have no clue what they’re talking about.

Yes, VSS is great on Windows systems – in fact, its great to see that standard VSS functionality has reached a point in NetWorker 7.5 that it’s just part of the Windows client for filesystem backups, rather than requiring additional licenses.

But VSS in itself is not in the same league as Time Machine for end user backup – and more importantly, recovery – and quite frankly, that’s more important when we’re talking about non-server backup systems.

Evaluating it as an end-user backup system

If you’re not fully across Time Machine, here’s how it works:

  1. You plug a new or otherwise unused hard drive into your Mac.
  2. The OS asks you if you want to use that drive for Time Machine backups.
  3. You answer Yes**.

That’s all there is to getting basic Time Machine backups running. At that point, Time Machine does a full backup, then from that point onwards does incremental backups making use of hard links, thus making very efficient use of space. Backups are taken every hour, and it manages backups such that:

  • Hourly backups are kept for 24 hours.
  • Daily backups are kept for a month.
  • Weekly backups are kept until the disk becomes full.

All pruning of space is automatically handled by the OS. For the system volume at least, Time Machine is an exclusive backup product – it backs up everything by default, and you have to explicitly tell it what you want excluded from the backup. This is a Really Good Thing. However, you can go into preferences and exclude other regions (e.g., I have a “DNB” (Do Not Backup) folder on my desktop that I drop stuff into for temporary storage), or explicitly include other drives attached to the system.

Overall the settings for Time Machine are simple – very simple:

Main preferences for Time Machine

Main preferences for Time Machine

The Options button is what allows you to manage exclusions for your backups:

Options pane for Time Machine

Options pane for Time Machine

To be honest though, who cares about backup? Desktop backup products abound, and in reality what we care about is whether you can recover. Indeed, for desktop products what we care most about is whether our parents, or our grandparents, or those people down the street who ask us for technical support simply because we’re in IT, can recover. Boy, can you recover.

Time Machine presents a visually beautiful way of browsing the backups. Unfortunately we won’t see it appear in other backup products because, well, according to Steve Jobs when it was first introduced, Apple took out a lot of patents on it***. The standard recovery browser will look like the following:

Time Machine Browsing Files

Time Machine Browsing Files

Equally importantly though, Time Machine isn’t just about facilitating file level recoveries, but also recoveries of other data that it understands – such as say, mail. Now, yes, enlightened readers will point out that Apple’s program stores mail in files and thus is easily browseable, but the files aren’t named in such a way that say, my father could work out which file needs to be recovered.

Here’s an example of what Time Machine looks like when browsing for recovery of mail:

Browsing mail with Time Machine

Browsing mail with Time Machine

To browse and retrieve email, the user simply browses through the folder structure – and the time of the backups – to pick the email(s) to be recovered. It’s incredibly intuitive, and takes less than 5 minutes to learn for the average user. As an enterprise backup consultant, honestly, I almost cried when I saw this and thought about how much of a pain message level recovery has been for so long. (Yes, getting better now, and has been for a while.)

Browsing back in time is straight forward – just scroll the mouse over the time bar on the right hand side of the screen and select the date you want:

Selecting alternate recovery time

Selecting alternate recovery time

This, quite honestly, is the epitome of simplicity. Going beyond standard backup and recovery operations, Time Machine is also an excellent disaster recovery tool – if you have serious enough issues that you need to rebuild your machine, the Mac OS X installer actually has the option of doing a rebuild and recovery from Time Machine backups.

To be blunt – as a backup utility for end users, Time Machine is an ace in the hole, and one of the most underrated features of Mac OS X.

There are some things that I think are lacking in Time Machine at the moment that will only come in time:

  1. Support for multiple backup destinations – savvy users want to be able to swap out their backup destination periodically to take it off site.
  2. Granular control of timing – some users complain that Time Machine affects the performance of their machine too much. Personally, I consider myself a power user and have not noticed it slowing me down yet, but others feel that it does, and don’t like the frequency at which it backs up. Being able to choose whether you want your most frequent backups done hourly, 2-hourly, 3-hourly, 4-hourly, etc., would be a logical enhancement to Time Machine, and one which I hope does arrive. Personally if this were available I’d more be seeking to keep daily backups for at least a month.
  3. Better application support – this actually isn’t an Apple issue at all, but one for third party software developers. Over time, I want to see any application that does database style storage, or storage where multiple files must remain consistent, to offer Time Machine integration. (The biggest failure in this respect is Microsoft Entourage – the monolithic database format makes hourly backups via Time Machine not only impractical, but unusable.)

Still, regardless of these deficiencies, Time Machine as it currently stands was a fantastic addendum to a robust operating system, one which puts easy recovery in the hands of average users.

(I have no idea what Apple intends to do with Time Machine at the server level – while Time Machine exists on Mac OS X Server, for the most part it’s to backup the server itself plus act as a repository point for machines on the LAN, much in the same way that Apple’s Time Capsule product works. However, if they added a little bit more – say, backing up multiple clients with file level deduplication across the clients, suddenly it would be very interesting.)

Comparing it to enterprise products…

Time Machine is great for providing a backup mechanism for end users, but it pales in comparison to what enterprise backup products such as NetWorker can do for an entire environment. As such, it’s not fair to compare it against those products – it’s not in their league, and it doesn’t pretend to be there. It doesn’t support remote storage, it doesn’t support true centralisation of backups, it doesn’t support removable media, … the list goes on, and on. Most importantly for any enterprise however, it doesn’t really support native backups of other operating systems. (Yes, you can shoe-horn it into say, backing up a SMB or CIFS share, but like any such form of backup, it’s not a true, integrated solution.)

As such, Time Machine isn’t something that’s going to replace your NetWorker environment. Chances are it won’t even replace your Retrospect environment. Used correctly though, it can act as a valuable enhancement in a backup environment, but if you’re a backup administrator, it isn’t going to put you out of a job today, next week, next year, or even in the next 5 years.

* Honestly, tell someone in a different discipline in IT that you specialise in data protection and that you enjoy it, and watch their eyes glaze over…

** Or in my case, since I can never resist the temptation, you answer no, and rename the disk to TARDIS, since if it’s going to be a Time Machine, it may as well be a good one.

*** Good for them. It’s tiresome watching what sometimes seems to be the entire computer industry using Apple as a free R&D centre.

Posted in Aside | Tagged: , , , , | Comments Off on Aside – My take on Time Machine

NetWorker 7.5 and Mac OS X

Posted by Preston on 2009-01-25

Released in December 2008, NetWorker 7.5 represents an incremental increase in NetWorker functionality mainly aimed at the following three things:

  • Virtualisation support – better VCBs, better awareness of virtual infrastructure, visualisation of virtual infrastructure, etc.;
  • Better integration with third party authority systems – e.g., LDAP, etc.;
  • Support for IPv6.

It’s the support for IPv6 that poses a particular challenge for Mac OS X clients. A bug currently exists with the NetWorker client for OS X that causes the startup and shutdown of the NetWorker daemons to take somewhere in the order of 5 minutes for the average machine. Given that IPv6 is enabled by default on Mac OS X, this means that a very high proportion of Mac OS X clients that are upgraded will experience this problem.

This doesn’t pose a problem in normal operations; this startup and shutdown normally occurs in the background for machine boot/reboot/shutdown, and thus doesn’t impact the time taken for a machine to start or shutdown – however, where it does pose an inconvenience is for an administrator working on the command line who needs to debug or test issues on a Mac OS X client. Indeed, the shutdown takes long enough that at least 50% of the time it times out and the nsrexecd processes need to be manually killed.

There are three options that administrators can choose to perform:

  • Delay deployment of NetWorker 7.5 on Mac OS X until the release of 7.5 SP1, where this issue is going to be fixed. (A 7.4.x client will communicate successfully with a 7.5 server.)
  • Accept the shutdown/startup delay for the time being and document it so that backup and system administrators are aware of the implications for the time being.
  • Disable IPv6 on affected clients until the release of NetWorker 7.5 SP1. This can be done using the command: ip6 -x  on the affected Mac OS X machines.

This isn’t a big inconvenience, but it’s one to be aware of.

Posted in NetWorker, Policies | Tagged: , , | Comments Off on NetWorker 7.5 and Mac OS X