NetWorker Blog

Commentary from a long term NetWorker consultant and Backup Theorist

  • This blog has moved!

    This blog has now moved to nsrd.info/blog. Please jump across to the new site for the latest articles (and all old archived articles).
  •  


     


     

  • Enterprise Systems Backup and Recovery

    If you find this blog interesting, and either have an interest in or work in data protection/backup and recovery environments, you should check out my book, Enterprise Systems Backup and Recovery: A Corporate Insurance Policy. Designed for system administrators and managers alike, it focuses on features, policies, procedures and the human element to ensuring that your company has a suitable and working backup system rather than just a bunch of copies made by unrelated software, hardware and processes.
  • Advertisements
  • This blog has moved!

    This blog has now moved to nsrd.info/blog. Please jump across to the new site for the latest articles (and all old archived articles).
  •  


     


     

  • Twitter

    Error: Twitter did not respond. Please wait a few minutes and refresh this page.

Posts Tagged ‘Parallelism’

Top 5 Reflections

Posted by Preston on 2009-07-17

So this morning I was looking through the stats for this blog, and I generated the list of most popular posts thus far. I can’t say any of the results surprised me. Every single one of the top 5 comes from the “Basics” series.

Number 5, on that list, was Basics – Listing Files in a backup. There’s a lot of people out there who want to know how to use nsrinfo in general, and specifically want to know about pulling file lists for savesets. Net result? I think it would be greatly beneficial if in NMC users could double-click on browsable savesets and get a complete listing of files therein.

Number 4 was Basics – mminfo, savetime and greater than/less than. Now, I’m not going to pretend that every person who visited that article was looking for details about how greater than and less than works in mminfo in relation to savetimes, though I suspect a reasonable percentage of people new to mminfo found that interesting. My take on it is that it proves there’s not really enough documentation about mminfo, and that mminfo needs some expansion. My personal preference? Having a full SQL-like query engine for mminfo would greatly expand the options available to NetWorker administrators.

Number 3 on the list is Basics – Changing saveset browse/retention times. As regularly as possible I try to check the search strings that have brought people to my blog (as recorded by wordpress), and I can practically guarantee that every day there are multiple combinations to do with savesets, browse and retention times. Sometimes those combinations reference nsrmm, sometimes they don’t. Clearly, extending saveset browse/retention times in NetWorker needs to be more manageable from within the GUI as a bare minimum. I’ll get to the command line in a moment.

Moving on to number 2, we have something that I get search results for every day without fail. That’s Basics – Fixing “NSR Peer information” errors. It’s actually a reasonably simple error to fix, but sometimes finding the information about it is a bit like the old needle-in-a-haystack. I’m hoping that the posting on it has helped quite a few sites to clear out the warnings/errors in their logs and reduce the amount of clutter being reported.

Finally, for number 1, a topic I’m completely unsurprised to see at the top, we have Basics – Parallelism in NetWorker. Not because it’s difficult, but because there’s no absolute rules, parallelism is a topic in NetWorker that many administrators, regardless of length of time with the product, find challenging at times. Set too low, and backups may overrun. Set too high, and device contention, client slow-downs, recovery performance issues, etc., may come into play. Tuning parallelism in NetWorker has to take a lot into account.

The content of this list suggests a few things to me:

  • None of this information is out of reach in the product manuals, but, since the product manuals are (necessarily) lengthy, it is logistically is out of reach for a lot of users who don’t have time to read lengthy manuals.
  • EMC product management could take a few tips from the top 5 articles on my blog – I think they represent areas that could be improved within usability of the product. While parallelism is not something that can “solved” by changes within the GUI (it is, by necessity, complex), other options, such as improving mminfo search, making saveset contents more accessible within the GUI, etc., are readily fixable.
  • It seems there might be scope for a “Getting Started with NetWorker” style manual. I think a traditional book would (a) be too expensive and (b) be unsuitable. This is the sort of information that people want readily to hand on their desktops.

On the last point, I’m interested in writing such a manual. I obviously have some experience with writing – but more so than just the book, over the years I’ve written literally thousands of pages of NetWorker instructions as part of professional services documentation, training courses, etc.

So here’s a question – would people be interested in say, an eBook along the lines of “Getting Started with NetWorker” that gives basic operational and instruction usage so that rather than having to wade through the (close to 1000+) pages of the official documentation they had something shorter, and geared towards day to day operation?

Let me know what you think.

Advertisements

Posted in Basics, General thoughts, NetWorker | Tagged: , , , , , , , , , , | 10 Comments »

Oracle channels and NetWorker parallelism

Posted by Preston on 2009-06-30

If you’re backing up Oracle with the NetWorker module/RMAN, there are an extremely large number of options you can choose from. RMAN, after all, is a complete backup/recovery system in and of itself, and so when you combine RMAN and NetWorker you, well, find yourself swimming in options.

One such option is the allocate channel command within RMAN. If you’ve not seen a basic RMAN script before, I should put one here for your reference:

connect target rman/supersecretpassword@DB10;

run {
 allocate channel t1 type 'SBT_TAPE';
 send 'NSR_ENV=(NSR_SAVESET_EXPIRATION=14 days,
       NSR_SERVER=nox,NSR_DATA_VOLUME_POOL=Daily)';

 backup format '/%d_%p_%t.%s/'
 (database);

 backup format '/%d_%p_%t_al.%s/'
 (archivelog from time 'SYSDATE-2');

 release channel t1;
}

You’ll note that one of the first commands used in the script is the allocate channel command. This effectively tells RMAN to open up a line of communication with NetWorker. Now, you can consider an RMAN channel to be a unit of parallelism in NetWorker parlance. Thus, if you want to backup (larger) databases with higher levels of parallelism, you need to allocate more channels.

In many NetWorker/Oracle scenarios, the NetWorker administrator has very little, if no, control over the construction and the configuration of the RMAN script. (The introduction of v5 of the module may change this.)

As a consequence, there’s often a reduced level of communication between the NetWorker administrator and the Oracle DBA which can result in reduced performance or scheduling conflicts. One particular issue that can occur though is that the Oracle DBA, eager to have the database backed up as quickly as possible, will throw a lot of allocate channel commands in. That little script above may become something such as say:

connect target rman/supersecretpassword@DB10;

run {

 allocate channel t1 type 'SBT_TAPE';
 allocate channel t2 type 'SBT_TAPE';
 allocate channel t3 type 'SBT_TAPE';
 allocate channel t4 type 'SBT_TAPE';
 allocate channel t5 type 'SBT_TAPE';
 allocate channel t6 type 'SBT_TAPE';
 allocate channel t7 type 'SBT_TAPE';
 allocate channel t8 type 'SBT_TAPE';

 send 'NSR_ENV=(NSR_SAVESET_EXPIRATION=14 days,
       NSR_SERVER=nox,NSR_DATA_VOLUME_POOL=Daily)';

 backup filesperset 4
 format '/%d_%p_%t.%s/'
 (database);

 backup format '/%d_%p_%t_al.%s/'
 (archivelog from time 'SYSDATE-2');

 release channel t1;
 release channel t2;
 release channel t3;
 release channel t4;
 release channel t5;
 release channel t6;
 release channel t7;
 release channel t8;
}

However, there’s a catch to lots of channels being allocated – channel allocation has no bearing on or is in any way impacted by NetWorker client parallelism. You see, the NetWorker client instance has a single saveset – the RMAN script name (or equivilant thereof, when using the Wizard in v5). Thus, to NetWorker, any Oracle client instance only has one saveset. Thus, that client parallelism will not affect the number of channels that can be allocated, but instead the number of simultaneous instances of the client that can be initiated.

The net result? Consider a client with parallelism of 4, that has 6 databases to be backed up. This would have 6 client instances, one per database. Assuming they’re all in the same group*, then at any one instance NetWorker will only allow the backup for 4 of those instances to be running. However, each instance, or each Oracle RMAN script, can start as many channels as it wants. If each RMAN script has been “tweaked” to allocate say, 8 channels like the above script example, this would mean that backing up 4 instances simultaneously would potentially see the client trying to send 32 savesets simultaneously to NetWorker.

Thus, if using multiple Oracle channels in RMAN backups with NetWorker, and particularly if backing up multiple Oracle databases simultaneously, it’s very important to have the NetWorker administrator and the DBA responsible for the RMAN scripts to communicate effectively and plan overall levels of parallelism/number of channels to avoid swamping the NetWorker server, swamping the network, or swamping the Oracle server.


* There are other considerations for starting multiple Oracle backups on the same machine and at the same time. In other words I’m not necessarily calling this best practice, just using an example.

Posted in NetWorker, Scripting | Tagged: , , , | Comments Off on Oracle channels and NetWorker parallelism

Considerations for client parallelism for NetWorker server

Posted by Preston on 2009-06-24

While doing a few tests for this blog on a lab server, I noticed what looked like odd behaviour – I had started a manual save running on the NetWorker server for local data. That backup was writing to tape, and while it was going I kicked off a group for an altogether different client.

The backup for the client ran, but then seemed to hang on completion. As the backup-to-tape was merely to test filling tape, and therefore could be restarted at any time, I cancelled out on a hunch, and the savegroup completed almost immediately.

It was “hung” waiting for a free unit of parallelism for the NetWorker server in order to write the client indices. It turned out that I’d forgotten a change I’d made on Friday to test some other settings – that change being to reduce the parallelism of the client instance of the NetWorker server to 1.

With this in place, the backup server couldn’t complete the savegroup because it couldn’t write its indices, and it couldn’t write its indices because it was only allowed a client parallelism of 1, and that unit of parallelism was occupied writing to tape.

So it lead me to think – how easy would it be, given this, for companies to experience delays in their backups due to too low a setting for client parallelism for the NetWorker server? The answer – quite easy. After all, the first, most golden rule of client performance tuning on NetWorker is to eliminate client parallelism, to reduce it to 1, and work your way up based on client hardware and data configuration.

This means that it’s actually fairly critical that the NetWorker server have sufficient parallelism to ensure that index backups do not become an impediment to groups finishing. Based on this I’d recommend aiming for client parallelism for the NetWorker server to:

  • Never be set to 1.
  • For small environments (under 30 servers) be set to at least 4.
  • For medium environments (say, 31-100) be set to at least 8.
  • For larger environments (100+), be set to at least 8, but preferably one of:
    • The same as the actual server parallelism, or
    • The same as the highest group parallelism, if group parallelism is used.

Note that the above entirely assumes that the backup server is a dedicated backup server. If the backup server is also say, a file server*, then obviously different settings will need to be considered to avoid swamping the system.

In essence, while the main goal for regular clients is to achieve as low a client parallelism as possible – i.e., to optimise the balance between number of savesets and throughput, for the backup server the goal should be to have as high a client parallelism as necessary to ensure that index backups are not delayed, so as to ensure that groups finish when they are ready to finish.


* For what it’s worth, my recommendation is that in 100% of times, a backup server should be dedicated. That is, the primary and sole function of the server is to act as a backup server.

Posted in NetWorker | Tagged: , | 4 Comments »

Basics – Parallelism in NetWorker

Posted by Preston on 2009-02-17

Hi!

The text for this article has moved, and is now at the permanent NetWorker Information Hub Site. Please read it here.

Posted in Basics, NetWorker | Tagged: , | 19 Comments »