Please change your channel: nsrd is moving!

I have greatly enjoyed running this site on WordPress for the last year, but I’ve decided I want to make a more complete site for NetWorker resources, and to do that, I need more than just a blog. So, as of this article, I’m moving the blog across to:

http://nsrd.info/blog

I’ll be leaving this blog around on WordPress in its current state for some time to come – there’s a lot of search engine results that link to it; however, if you’ve got links to this blog either in your bookmarks or on your website, I’d greatly appreciate a change of link across to nsrd.info/blog.

While I’m planning on hosting more than just a blog on the new site, for the moment I’ve focused on the blog transfer. The rest of the site is my summer holiday project*.

—
* Remember, I’m in the southern hemisphere, so that means something I’m actively working on now :-)

Seasons greetings from the NetWorker Blog

I’d like to take a moment to wish all the regular readers – and any new visitors – a very safe and happy holiday season. There’s too many of you to send cards out to everyone (even if I knew all your contact details) so I thought I’d give seasons greetings through a suitably NetWorker way:

Show me the man pages

As a long term Unix admin, it’s frustrating when there are commands on my systems for which there aren’t man pages. As a long-term NetWorker user, it’s equally frustrating when there aren’t man pages for particular NetWorker commands.

When I’ve discussed this in the past, I’ve usually had a response of “that’s because you shouldn’t be running that command”. That’s a bad response. The correct response should be something along the lines of “oops, we’ll write a man page for the next release that states:

That command is for internal NetWorker use only. It does X. It should not be run manually.”

Having undocumented commands that give no output, hang or produce strange results is just inviting frustration. Of just the nsr prefixed commands, on my current 7.6 lab server, the following commands are undocumented:

• nsravamar
• nsravtar
• nsrbmr
• nsrcatconfig
• nsr_cp_install
• nsrdmpix
• nsrdsa_recover
• nsrdsa_save
• nsrfile
• nsrfsra
• nsrlmc
• nsrndmp_2fh
• nsrrcopy
• nsrrcopy2
• nsrvcbserv_tool

So out of the 55 nsr prefixed commands I have on my server, 15 (or 27%) are undocumented.

Note to EMC: This does not produce a healthy level of trust. Please – get some documentation on these commands, even if that documentation gives us a one line overview of where they’re used and tells us not to run them ourselves.

Aside – My Number #1 Pet Peeve in Interface Design

At University, I had a fascinating lecturer. His typical mode of dress was a t-shirt, stubbies and to go barefoot around the campus. He had a great big bushy beard that barrelled along in front of him which at times looked like a mane. He had a reputation for reciting the entirety of The Ballad of Eskimo Nell (a rather ribald poem – I’m not providing a link) – though by the time I was at University, he could only ever be encouraged to let fly with a single verse.

None of this though made him fascinating.

What made him fascinating was his name. For your reference, his full name is:

Simon

That’s right, Simon. Just a first name, no last name. You see, at some point in the past Simon had decided to legally remove his surname. So he literally did not have a last name.

Simon was a fascinating case study in the implications of unexpected input in computer programmes. He was in fact a walking case study in the implications of unexpected input in computer programmes – almost exclusively due to his name. (This led me to having some joy in pointing out this XKCD cartoon to him a couple of years ago.) Every year, the people who made the phone book struggled to work out where to put him. He confounded registration systems everywhere, and turned compulsory fields on forms to rubbish. Simon was a walking lesson in the lessons of designing interfaces to handle unexpected inputs.

Not long after I finished University, I decided to change my name. Not anything so drastic as a removal of my surname; in fact, it was to add to my surname. You see, when my family emigrated to Australia several generations ago, they changed their surname from “de Guise” to just “Guise” so they could more easily assimilate. (So the story goes.)

Not being all that interested in blending in, and having an appreciation of the long term history of the name “de Guise”, I decided to reinstate it. (Some might question why I didn’t remove my middle name or at least change it from “Macdonald” – but that’s another story, to be told another time.)

It was at that point that I started to get an appreciation of the daily struggle Simon must have had in dealing with systems that were not adequately designed to work with non-conformist input.

I’ve learned therefore over the years that there’s far too many programmers with names like:

• Mary Jones
• Bob Smith
• David Peterson
• Jane Davidson

And far too few programmers with names like:

• Simon
• Preston de Guise
• Carlos de la Cruz
• Peter O’Toole

(I had already learned, by the way, that there were far too few companies that simultaneously employed a McDonald, Macdonald and MacDonald.)

So here’s my pet peeve in interface design, stated as examples:

• I am not Preston De Guise
• I am not Preston De guise
• I am not Preston de
• I am not Preston De
• I am not Preston Deguise
• I am not Preston DeGuise
• I am not De, Preston Guise
• I am not even, any longer, just Preston Guise (and I certainly don’t have a middle name of de).

There are too many lazy and/or inconsiderate programmers out there. (There’s also too many lazy and/or inconsiderate data entry operators as well.*)

If you’re a programmer, and want to get onto my good side in 2010, make sure your system gets my name right.

—
* In the past I have been guilty of name mutilation myself. In my last job I setup an account for someone, mistaking the first word of her surname as a middle name, and egregiously never got around to correcting it. It is actually something I genuinely regret.

What’s been happening this week?

It’s been a fairly interesting week overall in storage.

There’s still been a lot of chatter about FAST, EMC’s new system for having LUNs automatically moved between different tiers of storage. I clearly need to read up more about FAST, because so far I don’t see it so much as FAST but as 2nd gear. Sure, being able to automatically move whole LUNs around is nice, but I thought the magic was in sub-LUN migration years ago when I saw a Compellent demo on it. Clearly I’m missing something, because a lot of people have been getting very excited. Then again, my focus in storage has been protecting it rather than optimising primary access, so it’s likely I’ll get more interested in it as I read more about it.

Over at Search Storage, there was an interesting article about data reduction vs data deduplication and compression. This prompted me to pull my finger out, and so sometime soon I’ll have another blog article myself here called All this Deduplication is Making me Thirsty. I’ll leave it as an exercise to the reader what I’m talking about (or a surprise for the article.)

Moving on, I found this excellent article by Drew Robb over at ServerWatch called Tape vs Disk: Tape Refuses to be Evicted. Regular readers of my blog will know that my opinion on the “tape is dead” pronouncements we get every few months is to blow a big fat raspberry at move on. Drew’s article was pointedly useful, in that he dug into the IDC studies on tape sales. While a lot of the media is enjoying running around saying that IDC shows that tape sales are declining, they’re not telling the whole truth, as Drew points out. In actual fact, it’s the low end tape sales that are falling off quite a bit, but the enterprise stuff is still running along quite nicely. This is completely understandable – I’ve seen a lot of small businesses that used to rely on cheap and cheerful tapes like DAT transition to more reliable and longer lasting media, but I don’t see a lot of enterprises killing tape. (What’s the old saying from that old ad – linear serpentine, good for data; helical scan, good for parties … I think the party is fizzing out, but the data is still going strong…)

I think when it comes to handing out awards (if I were to do that) for the coolest storage blog entries this week, they have to go to Storagebod, with his 7 part series of letters to father christmas – including the final one asking for presents for other bloggers.

Over at Grumpy Storage, Ian penned a fantastic article called Show me the Money. I think this should be mandatory reading for every sales person and consultant in the tech industry – certainly in previous companies I’ve dealt with sales people who have been shot down and lost deals for failing to follow these rules that should be self evident.

On the lighter side, someone (apologies, I can’t remember who) twittered a link to Death By Powerpoint. This should be mandatory watching for everyone in business, full stop.

On a more local front, Brian over at Going Virtual has just done his wish list for updates to NetWorker for 2010 in relation to virtualisation support.

On a slightly different note, our Australian government has decided it’s going to attempt to introduce national mandatory net censorship laws next year. I would say what I think about such draconian subjects except it would undoubtedly be retroactively censored next year and this article deleted. I’d hate to upset my article count, so suffice to say, in a more polite way, that I hope they take notice of online polls that show upwards of 95% of people against the idea.

Over at Daring Fireball (yes, I know, not a storage blog), John Gruber has a rather excellent analysis of the garbage that’s been coming out of AT&T lately. Apparently they only want customers to buy, not use bandwidth. Silly, selfish users who expect to be able to buy and then use their services are apparently to blame for profit obsessed companies that have no interest in upgrading their infrastructure to meet needs.

Over on the The Register, there was a story about Stratus putting their money where their mouths are. Good on you Stratus.

Next to last, as a bit of self-advertising, I took time out from this blog to write up some thoughts on all those marketing slides that encourage people to do TCO calculations to compare pushing infrastructure out of the data centre and into the Public Cloud, and suggested that an alternate calculation needs to follow, one that I call Total Cost of Impact.

One final comment: please go and see Avatar. If you don’t, you’ll be missing out on the greatest block buster of all time (so far). If you’re a Star Wars fan, you doubly need to see it, so you can understand what a good movie looks like.

Introducing the Support and Services page

Regular visitors may note that there’s a new addition to the pages on this blog – one covering Support and Services.

I run this blog in my own time (probably using up a little too much of my own time to be quite truthful) and ask for no payment or reimbursement from my readers – well, other than an occasional pitch for people to buy my book, that is.

My day job however is a consultancy and support role at IDATA Resolutions, and the Support and Services page outlines some of the key things IDATA could do for you, if you happen to be looking for service, support, consulting or training for your environment.

If you’re looking for an independent review of your environment, or considering support options, looking at a new solution or needing some training (whether that’s one-on-one, customised or general), I’d invite you to check out the Support and Services page above to see what IDATA can do for you.

Pictorial representation of management happiness over OpEx cost savings

There is a profoundly important backup adage that should be front and centre in the mind of any backup administrator, operator, manager, etc. This is:

It is always better to backup a little more than not quite enough.

This isn’t an invitation to wildly waste backup capacity on useless copies that can never be recovered from – or needlessly generate unnecessary backups. However, it should serve as a constant reminder that if you keep shaving important stuff out of your backups, you’ll eventually suffer a Titanic issue.

Now, people at this point often tell me either that (a) they’re being told to reduce the amount of data being backed up or (b) it makes their manager happy to be able to report less OpEx budget required or (c) some combination of the two, or (d) they’re reluctant to ask for additional budget for storage media.

The best way to counter these oppressive and dangerous memes is to draw up a graph of how happy your manager(s) will be over saving a few hundred dollars here and there on media versus potential recovery issues. To get you started, I’ve drawn up one which covers a lot of sites I’ve encountered over the years:

You see, it’s easy to be happy about saving a few dollars here and there on backup media in the here and now, when your backups are running and you don’t need to recover.

However, as soon as the need for a recovery starts to creep in, previous happiness over saving a few hundred dollars rapidly evaporates in direct proportion to the level of the data loss. There might be minimal issues to a single lost document or email, but past that things start to get rather hairy. In fact, it’s very easy to switch from 100% management happiness to 100% management disgruntlement within the space of 24 hours in extreme situations.

You, as a backup administrator, may already be convinced of this. (I would hope you are.) Sometimes though, other staff or managers may need reminding that they too may be judged by more senior management on recoverability of systems under their supervision, so this graph equally applies to them. That continues right up the chain, further reinforcing the fact that backups are an activity which belong to the entire company, not just IT, and therefore they are a financial concern that need to be budgeted for by the entire company.

Fingers Crossed for some New Years Resolutions from EMC

Over at Storagebod’s Blog, Martin Glassborow has been providing a highly interesting and entertaining set of letters to Father Christmas, which I’d highly recommend reading.

So in the spirit of Martin’s postings, I’m going to do a slight variant – here’s a few things that I’ll be keeping my fingers crossed for that EMC will provide us in NetWorker in 2010.

Dear EMC,

Could you please add to your NetWorker New Years Resolution the following items?

• Windows 2008 Service Pack 2 support as soon as possible!
• Improvements for ADV_FILE devices:
  • Savesets can spill over from one disk backup unit to another should the first become full.
  • Better rotation/selection method for devices/volumes.
• Tapes:
• Ability to query the “offsite” mminfo flag.
• Cloning:
  • Inline cloning (simultaneous clone + original generation).
  • Fixing the “validcopies” flag!
• NetWorker Management Console:
  • Manual client backup and recovery operations integrated into the console.
  • Manual module backup and recovery operations integrated into the console.
• Resources/Configuration:
  • Ability to rename clients (and other resources).
  • Syntax for including another directive within a directive.
  • PIDs for nsrmmds stored within the device resource for easy viewing.
• Operations:
  • Client GUI manual backups to support pools other than Default.
  • More granular savesets.

I could think of a bunch of other enhancements, but these are the ones I’m hoping most are on your new years resolution list for NetWorker in 2010. I’m hoping I don’t have to put any of these on a wish list for your 2011 new years resolution list.

Thanks, and have a happy new year!

Preston de Guise.

The pros and cons of Legato License Manager

Bundled with NetWorker for some time now has been the peripheral product, Legato License Manager (LLM). (Now normally just referred to as “License Manager”.)

If you’ve never used LLM, you may wonder what use it serves. But to do that, we first need to look at the control zone, that region of space that encompasses one or more NetWorker datazones. This might resemble the following:

Both the NetWorker Management Console (NMC) server (typically the “gst” processes) and LLM reside in the control zone – that is, they exist to service multiple datazones. However, in the same way that many sites end up running the datazone and control zone (via NMC/GST) on the same backup server, there’s nothing preventing you from using LLM to manage licenses separately to the core NetWorker services.

Making this transition is relatively straight forward, and I’ll save doing an article on that aspect unless people would like to see one – instead, I’d like to discuss the pros and cons of having licenses outside of NetWorker but still referenced by a NetWorker server.

Advantages of LLM

It’s best to first understand what LLM brings to you. I’ll use the following keys beside the advantages to let you know where they’re applicable:

• (D+) – Useful for multiple datazones.
• (1D) – Useful for single datazones.
• (M) – Marketing advantage; touted as a bonus by EMC/Legato, rarely if ever used.

So, some of the advantages are:

• (D+) Licenses may be purchased and installed in a central location.
• (D+) Licenses may be reallocated between NetWorker servers as resource requirements/capabilities change (i.e., released from one server, and snapped up by another).
• (1D/D+) License authorisation codes are tied to the LLM server, not the NetWorker server. Therefore if you’ve got an environment where you’re planning on doing some NetWorker server migrations, you can move your licenses to LLM and not have to repeatedly do host transfers.
• (M) You can buy “bulk” licenses. (E.g., 10 x 5 Client Connection Licenses). This advantage seems to be minimising the number of licenses you need to enter. While this sounds cute in theory, I think it actually adds a layer to license complexity.
• (1D/D+) Licenses reported in NMC show the exact features they are being used for – e.g., instead of showing “NetWorker Server, Network Edition/125” to indicate that the server is licensed for 125 clients, you might see “NetWorker Server, Network Edition/93” to indicate that currently there are 93 client licenses being used.
• (M) LLM is where the full version of NMC is licensed, so you only have to learn one licensing administration system.

Disadvantages of LLM

The advantages of LLM don’t come without some disadvantages too. These are:

• Not all licenses work all the time with LLM. For example, historically there has been ongoing issues with creating dedicated storage nodes in an environment using LLM. Typically it’s been necessary to add both a dedicated and a full storage node license, create the dedicated storage node devices, then delete the full storage node license. (Messy.)
• When working outside of NMC, the command line access to LLM licenses (via lgtolic) is even more esoteric and preposterous to use than nsrcap and nsradmin.

Should you use LLM?

It depends on your site. If you’re a fairly small environment, I can’t see any purpose for switching from NetWorker licensing to LLM; however, if your site has a larger number of clients and is reasonably dynamic in client allocations, LLM may give you that extra easy oversight to justify switching to it even if you’ve only got one datazone in your environment.

Alternatively, if you’re planning to transition your backup server a few times over a shorter period (e.g., migrating from current old hardware to interim hardware to full new hardware), then moving licenses out of NetWorker and into LLM may save you the hassle of getting them re-authorised at each step.

Should your LLM server be considered “production”?

If you’re using LLM, the obvious question that some will ask is – can this just be plonked on any old desktop PC? The answer is no. Well, to qualify that answer a bit – nothing prevents you from doing this except common sense. By all means have this running as a virtual machine somewhere, but it should still be considered a production machine in the same way that a backup server is a production machine: it’s part of support production rather than primary production, but it’s still production.

15 crazy things I never want to hear again

Over the years I’ve dealt with a lot of different environments, and a lot of different usage requirements for backup products. Most of these fall into the “appropriate business use” categories. Some fall into the “hmmm, why would you do that?” category. Others fall into the “please excuse my brain it’s just scuttled off into the corner to hide – tell me again” category.

This is not about the people, or the companies, but the crazy ideas that sometimes get hold within companies that should be watched for. While I could have expanded this list to cover a raft of other things outside of backups, I’ve forced myself to just keep it to the backup process.

In no particular order then, these are the crazy things I never want to hear again:

1. After the backups, I delete all the indices, because I maintain a spreadsheet showing where files are, and that’s much more efficient than proprietary databases.
2. We just backup /etc/passwd on that machine.
3. But what about /etc/shadow? (My stupid response to the above statement, blurted after by brain stalled in response to statement #2)
4. Oh, hadn’t thought about that (In response to #3).
5. Can you fax me some cleaning cartridge barcodes?
6. To save money on barcodes at the end of every week we take them off the tapes in the autochanger and put them on the new ones about to go in.
7. We only put one tape in the autochanger each night. We don’t want <product> to pick the wrong tape.
8. We need to upgrade our tape drives. All our backups don’t fit on a single tape any more. (By same company that said #7.)
9. What do you mean if we don’t change the tape <product> won’t automatically overwrite it? (By same company that said #7 and #8.)
10. Why would I want to match barcode labels to tape labels? That’s crazy!
11. That’s being backed up. I emailed Jim a week ago and asked him to add it to the configuration. (Shouted out from across the room: “Jim left last month, remember?”)
12. We put disk quotas on our academics, but due to government law we can’t do that to their mail. So when they fill up their home directories, they zip them up and email it to themselves then delete it all.
13. If a user is dumb enough to delete their file, I don’t care about getting it back.
14. Every now and then on a Friday afternoon my last boss used to delete a filesystem and tell us to have it back by Monday as a test of the backup system.
15. What are you going to do to fix the problem? (Final question asked by an operations manager after explaining (a) robot was randomly dropping tapes when picking them from slots; (b) tapes were covered in a thin film of oily grime; (c) oh that was probably because their data centre was under the area of the flight path where planes are advised to dump excess fuel before landing; (d) fuel is not being scrubbed by air conditioning system fully and being sucked into data centre; (e) me reminding them we just supported the backup software.)

I will say that numbers #1 and #15 are my personal favourites for crazy statements.