IDATA Tools v4

For the last several weeks I’ve been doing a lot of development work, putting the finishing touches on IDATA Tools v4, and it’s being released 1 May (AU/NZ – 30 April US, etc.)

This represents a significant functionality bump over the last version of IDATA Tools, and I thought I’d spend a few minutes mentioning some of the new features that I think make the tool set both useful and very compelling. This is by no means a comprehensive list – there’s been a lot of improvements.

By the way, in the list below I’m saving the best to last, so make sure you follow it right to the bottom!

list-resources

This is a new utility with one purpose, that being to quickly list all the resources of a given type as defined on the NetWorker server. This is basic, and a lot of people have scripted similar things in the past, but included in this tool suite, it helps you write your own scripts. Using list-resources, you can quickly grab details of all the configured clients, and then step through each client name in your own script to run some task, or do something to each configured group, etc. It has two output modes – the default is “human readable”, designed for when you just want to get some information out of the server. The second is delimited format – your choice of delimiter. This means you can extract information about any resource (and any field within a resource) in one simple command, in the format you want, for use wherever else you want.

For example, if you wanted a list of all your groups, whether they automatically clone and which pool they clone to, it would be as simple as:

C:\> list-resources -t group -c, -F "clones,clone pool"
Default,No,Default Clone
Daily Desktops,Yes,Daily Clone
Daily Development,Yes,Daily Clone
Daily NMC,No,Default Clone

Or, if you wanted a list of your licenses with their enabler codes and authorisation codes, you could run:

# list-resources -t license -c, -F "enabler code,auth code"

I developed list-resources on a whim, because I got tired of firing up nsradmin (let alone NMC!) every time I wanted to quickly check out some configuration option or get a list of resources. I was pleasantly surprised when every beta tester for IDATA Tools v4 commented on how useful the utility was for their environment.

check-clients

This long-term utility has been a bit of a sleeper – it’s not used as often as I think it should be used. I do strongly recommend that you give it a go – it’s great at automating a lot of tests and checks against clients, meaning that if you’ve got even just one client that’s malfunctioning, you can set check-clients running against it and gather all results in one command. For example, even something as basic as reconfirming name resolution for all active clients is as simple as could be using check-clients:

[root@nox ~]# check-clients -t resolution -g all_active
=======================================================
Running test: resolution (Test/confirm name resolution)
=======================================================
    aralathan
        Name: aralathan (aralathan.lab.idata.com.au) (192.168.100.7)
        Name: aralathan.lab.idata.com.au (aralathan.lab.idata.com.au) (192.168.100.7)
        Addr: 192.168.100.7 (aralathan.lab.idata.com.au)
    archon
        Name: archon (archon.lab.idata.com.au) (192.168.100.1)
        Name: archon.lab.idata.com.au (archon.lab.idata.com.au) (192.168.100.1)
        Addr: 192.168.100.1 (archon.lab.idata.com.au)
    arcweld
        Name: arcweld (arcweld.lab.idata.com.au) (192.168.100.232)
        Name: arcweld.lab.idata.com.au (arcweld.lab.idata.com.au) (192.168.100.232)
        Addr: 192.168.100.232 (arcweld.lab.idata.com.au)

Now in v4, there’s several new checks and pseudo-reporting options available. One requested by a new IDATA Tools customer is a particular favourite – generating details of the amount of space occupied by all backups for each client. The output looks like the following:

[root@nox ~]# check-clients -t used_space -g all_active
============================================================
Running test: used_space (Calculates used space for backups)
============================================================
    Client                         Used Space (GB)    
    ----------------------------   --------------------
    aralathan                                 138.51074
    archon                                     53.18894
    arcweld                                     2.98269
    asgard                                      7.92337

(I’ve cut the output there – you don’t need to see a full list of all my lab machines to get the idea.) Note the “all_active” option there; the command line means: run the used_space test against all clients that happen to belong to groups that are active (autostart enabled). No fussing around with long lists of clients if you don’t want to.

Additionally, as a sanity check run once a week (or even once a day), how about checking to see whether there’s any clients with empty indices?

C:\> check-clients -t empty -a
=======================================================
Running test: empty (Report clients with empty indices)
=======================================================
    rhltst5 has an empty index
    nazgul has an empty index
    rhltst6 has an empty index
    shrike has an empty index
    rhltst3 has an empty index
    rhltst4 has an empty index
    grumpy has an empty index

Phew! Lucky it’s a lab server otherwise there’d be trouble at the station.

Reporting enhancements

Under reporting more generally, there’s a whole raft of enhancements:

1. Both client-report and review-res now allow you to specify additional fields to be reported on; thus, it doesn’t matter if your reporting requirements are different any more, you can just run the same reporting utility and add the extra required fields in. We also report on the client ID wherever possible too.
2. (In fact, given how important it is to regularly, routinely extract details of clients and their client ID fields, the check-clients utility also has a new test which extracts and prints this information.)
3. The idata-notify tool, always an excellent savegroup completion parser, has some extra options. First, it can now give you details of the last full backup date for each saveset listed in the savegroup completion report, and it can also now send all the additional information as a HTML attachment, such as in this example.
4. Additional options are available in the find-files utility to allow you to run faster queries.

Portion Control for Cloning and Staging

These are without a doubt the two biggest enhancements to IDATA Tools in this release. By “portion control” I mean automatically break up cloning and staging operations into smaller, more granular activities with pauses inbetween by running operating on savesets at any one time.

Where is this useful? Let’s examine each utility:

• In relation to cloning, sslocate supports portioning of the data to be cloned so you can set a large clone job running and know that periodically it’s going to stop, release resources, and in doing so give them up to anything that’s been sitting waiting for resources to become available. If a recovery is waiting for the tape you’re reading on to clone from, the recovery will get a chance to run before all cloning has finished, rather than you having to abort the cloning.
• In relation to staging, dbufree is significantly more useful at freeing up space faster. Consider the following – you’ve got 200GB of a backup left to run, your disk backup unit fills, and you have to wait for NetWorker to stage out 500 GB due to watermarks before you can finish your backup. It’s maddening, right? Well, with portioning, you could still tell dbufree to stage out 500 GB of data, but to do it in portions of approximately 100GB. Disk backup unit space will get released at the end of each portion, meaning that instead of waiting for 500 GB to be staged, you’ll see space reclaimed after approximately 100GB, then another 100GB, and so on.

For both cloning and staging, you also get a level of reporting not normally available. Everyone when they run cloning and staging eventually get to the point of wondering “where is this at? how much is left to go?” Now by keeping an eye on the portion based cloning and staging, you’ll have a much better idea of how much data has already been copied, and how much is left to run.

In Summary

We’ve really put a lot of new features into IDATA Tools for this new release. Existing users should find the upgrade reasons compelling, and those of you who haven’t tried out IDATA Tools yet should contact us to request a trial.