Pictorial representation of management happiness over OpEx cost savings

There is a profoundly important backup adage that should be front and centre in the mind of any backup administrator, operator, manager, etc. This is:

It is always better to backup a little more than not quite enough.

This isn’t an invitation to wildly waste backup capacity on useless copies that can never be recovered from – or needlessly generate unnecessary backups. However, it should serve as a constant reminder that if you keep shaving important stuff out of your backups, you’ll eventually suffer a Titanic issue.

Now, people at this point often tell me either that (a) they’re being told to reduce the amount of data being backed up or (b) it makes their manager happy to be able to report less OpEx budget required or (c) some combination of the two, or (d) they’re reluctant to ask for additional budget for storage media.

The best way to counter these oppressive and dangerous memes is to draw up a graph of how happy your manager(s) will be over saving a few hundred dollars here and there on media versus potential recovery issues. To get you started, I’ve drawn up one which covers a lot of sites I’ve encountered over the years:

You see, it’s easy to be happy about saving a few dollars here and there on backup media in the here and now, when your backups are running and you don’t need to recover.

However, as soon as the need for a recovery starts to creep in, previous happiness over saving a few hundred dollars rapidly evaporates in direct proportion to the level of the data loss. There might be minimal issues to a single lost document or email, but past that things start to get rather hairy. In fact, it’s very easy to switch from 100% management happiness to 100% management disgruntlement within the space of 24 hours in extreme situations.

You, as a backup administrator, may already be convinced of this. (I would hope you are.) Sometimes though, other staff or managers may need reminding that they too may be judged by more senior management on recoverability of systems under their supervision, so this graph equally applies to them. That continues right up the chain, further reinforcing the fact that backups are an activity which belong to the entire company, not just IT, and therefore they are a financial concern that need to be budgeted for by the entire company.

Backup is a production activity

I routinely check (via the handy WordPress dashboard) what searches lead people to my blog. Often it’s for content that already exists on my site, but it also routinely helps me think of new topics to cover. (Occasionally it also provides some wry humour – for instance, someone a few weeks ago searched for “after the sun freezes”, which led them, I believe, to my posting on when I’d get around to running a search using Wolfram|Alpha.)

Interesting one today though was “why backups should not be on a production server”, and I thought that in this case, there’s a couple of distinct responses. These are:

1. Backups should not run on an existing production server (when configuring a new environment), because they should not be provisioned to share resources with existing services. Or more importantly, backups are a sufficiently important activity that one should not have to interrupt them to generate an outage for another system that shares the same system, or vice versa.
2. Backups are a production activity and they must be run on a production server.

There are obviously different levels of “production”; I’d suggest at bare minimum there are two styles of production systems for any enterprise:

• Operational production systems – Those systems that the business uses on a day to day business to fulfill standard business operations.
• Infrastructure support production systems – Those systems that the business uses at the “back end” to facilitate the success of the operational production systems.

Unless you’re a backup services provider, your backup server will never be an operational production system. However, in all other instances, your backup server will be part of the infrastructure support production systems.

You may consider that to be splitting hairs, but there are very simple yet important reasons why we need to consider backup systems as production systems. These include, but are not necessarily limited to the following:

• In many companies, non-production systems have a tendency to be “borrowed from” whenever there’s infrastructure overruns. For example:
  • A little bit of disk space here and there may be taken away for large image storage;
  • Redundancy on the system may be reduced if “production” systems need more storage;
  • New services may be “temporarily” placed on the server because there’s no other place for them.
• Outages or failures may be considered “acceptable” or not as closely monitored for non-production systems – thus backup systems that experience hardware faults overnight may not be suitably looked at;
• Systems profiles/allocation may be unsuitable for the performance requirements for enterprise production backups (in one extreme instance, I saw a desktop PC, years older than existing servers, used as a backup server!)
• CapEx/OpEx is improperly seen as something that should come from the IT budget rather than the operational budget of the company.

Let there be no uncertainty here – when it comes to production infrastructure support systems, your backup server, providing protection for your operational production systems, is equally as critical as all of the operational production systems it services.

Media, CapEx and OpEx

A common mistake I often see made, particularly when planning new system implementations, is to try to calculate out media costs over the entire planned growth period. That is, assuming a new backup system is going to be installed, accounting/management types will want to plan full tape requirements for the projected growth period the system was planned for (e.g., 3 years) from the outset.

The flaw of this approach is attempting to account for media costs as capital, rather than operational expenditure. This approach often results in unnecessary cost savings being made by cutting out other aspects of the system budget – software and hardware needed now are excluded from budget in order to make way for media that will be needed later.

This CapEx vs OpEx approach becomes most flawed when a system is being put in place making use of the “latest and greatest” media type. Let’s assume for the moment that LTO-5 has just been released, and a system with 4 x LTO-5 drives is installed, with planned capacity requirements for the next 3 years suggesting that 4,000 units of media will be required.

However, at just-released prices, media will be prohibitively expensive. Assume if you will that the RRP for each LTO-5 tape may be around $180 AU. Even with a bulk purchase discount bringing the price down to say, $100 AU per unit of media, that’s $400,000 of media if it is being purchased from the outset.

However, in the backup industry, we know that media gets progressively cheaper as it has been out for a while. Just look at all the LTO series of media. In Australia, each new format came out at around $180 RRP per cartridge. Now a simple search shows that I could pick up, on RRP alone, LTO-4 media for as little as $85 AU. (That’s just from one search, and for individual unit pricing.)

So going back to our not-yet-released LTO-5, assuming 4,000 units of media will be required across 3 years, the operational expenditure for that would be cheaper than the capital expenditure, and the media would only be purchased on an “as needs” or “near as needs” basis, ensuring media doesn’t sit on a shelf for lengthy periods of time before use.

Let’s say that media is purchased every 6 months for such a system, and an equal amount of media is purchased each time. So, every 6 months or so, one would need to order another 666 units of media for the system. Let’s round that up to 670 so we’re talking about packs of ten. We’ll assume an 11% decrease in the cost of the media every 6 months.

We’ll also assume that any bulk order (say, 300 units of media or more) will result in a 40% discount from the RRP. Let’s run a simple numbers game here then:

• First purchase – 670 @ RRP $180 / Discount $108, $72,360.00
• Second purchase – 670 @ RRP $160.20 / Discount $96.00, $64,400
• Third purchase – 670 @ RRP $142.58 / Discount $86.00, $57,316
• Fourth purchase – 670 @ RRP $126.89 / Discount $76, $51,012
• Fifth purchase – 670 @ RRP $112.94 / Discount $68, $45,400
• Sixth purchase – 670 @ RRP $100.51 / Discount $60, $40,406

That’s a total media OpEx budget over 3 years of just under $331,000, as opposed to $400,000 CapEx at the commencement of an implementation.

What’s more, because the media purchases are spread out over the course of the three years, rather than having to find $400,000 or even $331,000 up front, which would seriously put a dent in other budget activities, the most in any one financial year that would be required under OpEx for media would be in the first year, at a much lower $136,760.

Further, because backup is something that logically and operationally should source budget from the entire company, rather than this being OpEx out of the IT budget, it would be OpEx shared from all departmental budgets, or, if there’s a corporate overheads/OpEx budget, from that budget instead.

Contrary to popular belief, media purchases don’t have to be a nightmare or exorbitantly high.