在点击 Activity Monitor 时，出现如下报错：

在点击 Media 时，出现如下错误：

故障原因： NetBackup Master Server 的数据库所在的分区空间使用满了，为保证数据库的数据完整性，关闭了 Media Manager server 数据库。

仔细查看，可以发现某些服务器在 Netbackup\db\images\ 的文件特别大，如下图：

两种方法解决：
1. 压缩 catalog
定位到 Host Properties/Master Server/Golbal Attributes/，选择 compress catalog interval after x days。
2. 使用 ALTPATH 修改 images 这个目录的路径，这个没有具体去研究

上述两种方法是官方推荐的方法，注意第一种方法压缩 catalog 时需要注意，如果您的 Master Server 版本是 6.0.3 之前的，会存在一个问题，具体查看 http://seer.entsupport.symantec.com/docs/283528.htm

不过以上两种方法，我目前都没有采用，我只是零时将一些 catstore 文件迁移到其他地方，然后使用 nbdb_admin -start 命令启动 NetBackup 的数据库，如下图：

之后，NetBackup Database server 恢复正常。目前在考虑迁移 Master Server 到一台新的服务器上，采用备份和恢复 catalog 的方法。

84 号错误是硬件读写方面的错误，查看了一下，最近的 Catalog 总是备份到硬盘 E:\catalog 目录上，而不是按照设置的“备份到磁带”和“备份到磁盘”依次间隔的方式。

分析过程：
1. 在 Catalog 备份方式的设置中，临时去掉备份到 Disk 的方式，只保留备份到 Tape 的方式，手工发起 Catalog 备份，备份成功；
2. 再在 Catalog 备份方式的设置中，临时去掉备份到 Tape 的方式，只保留备份到 Disk 的方式，手工发起 Catalog 备份，可以看到 E:\catalog 目录中先是生成了一个 2GB 大小的文件后，在生成第二个文件时删掉了第一个文件，最终的保留的文件大小约 550M。考虑到此时 E 盘所剩空间不足 1.9G，可以判断是硬盘空间不足问题造成了 catalog 备份到 Disk 时报 status 84 错误。

注：

1. 因为不能通过 bpmedialist -m Catalog_TapeID 和 通过 Reports 查看的方式取得 Catalog Image 文件备份到 Disk 上的大小，所以只能通过上面第二步备份到 Disk 的方式确定 Catalog 的具体大小。
2. 好像 Catalog 在备份到 Disk 错误后，下一次仍然会尝试备份的 Disk 上，造成连续的 Catalog 备份错误。（这是个人观察的，不知道是否备份到 Tape 错误后也是这样备份到相同介质。）

大概的流程图

+-----------+  collector   +-----------+           +-----------+
| NetBackup |   script     |  MySQL    |  Perl     |  Chart or |
| Catalog   |------------->| Database  |---------->|  Graph    |
+-----------+   query      +-----------+           +-----------+

The main concept is to run a collector script that queries the NetBackup catalog for backup statistics and stores that information in a MySQL database. A second program then uses the backup data in the MySQL database to generate a chart with Perl and the help of GD::Graph::bars. Alternatively, you can use SQL commands to query the MySQL database directly and produce any type of custom report. Figure 1 shows some example output.

Performance data gathered for each backup would include the backup ID, the duration, the client host, the schedule type (full or incremental), the amount of data backed up, the number of files, the kilobytes per second, the policy used, the date the backup started, and the server that initiated the backup. This data can then be queried to help answer such questions as:

• Which system is the best performer?

  mysql> select client,kbpersec from backperf order by kbpersec desc limit 1;
  +------------+----------+
  | client     | kbpersec |
  +------------+----------+
  | somehost   |    24612 |
  +------------+----------+
  	

• Which systems back up the most data?

  mysql> select client,kbytes from backperf order by kbytes desc limit 1;
  +---------+-----------+
  | client  | kbytes    |
  +---------+-----------+
  | ct01sts | 746335072 |
  +---------+-----------+
  

• What is the average number of files being backed up?

  mysql> select avg(numfiles) from backperf;
  +---------------+
  | avg(numfiles) |
  +---------------+
  |   102661.9459 |
  +---------------+
  

Other questions we could answer include: How long is client X taking to back up? What is the median/average backup time? How many files are the poor performers backing up? Which are the good performers?

These kinds of questions can be answered by analyzing the data shown in the previous examples. Figure 1 shows kilobytes/sec, which helps identify which systems might need attention and provides a good report to check every morning.

The Setup

Now let’s get to the nitty-gritty. The first piece in this setup is a MySQL database. This article assumes you have a MySQL server running. Using nbperf.sql(Listing 1), you can create a MySQL table called “backperf” to store your backup performance data.

#mysql -u root -p
mysql> create database nbperf
(control-d)
#cat nbperf.sql | mysql -u root -p nbperf

At this point you have a database (nbperf) with one table (backperf).

The next piece is to put data into the table on a regular basis. Ideally, this would occur as a cron job sometime after the backups are complete (early morning in most cases). Because the collector can run as often as you like, scheduling depends on your environment. The collector, co_backperf.pl (Listing 2), will need to be modified to fit your system environment. Set the $database and $hostname variables to the database you created in the above steps on the MySQL server. You will also need to modify @servers to contain the NetBackup servers on which you want to collect data (i.e., “server1″ and “server2″). Going through the listing, the next important thing to note is the $cmd variable. “bpimagelist” is the guts of the output we want to store; every time it runs, it produces a listing of everything in the NetBackup catalog since the “-d” option and includes several important fields. Typical output of this command looks like this:

IMAGE clientname 0 0 6 clientname_1054339491 policyname 0 *NULL* root
schedulename 0 5 1054339491 1324 1062374691 0 0 2659008 113017 1 1 0
policyname_1054339491_FULL.f *NULL* *NULL* 0 1 0 0 0 *NULL* 0 0 0 0 0 0 0
*NULL* 0 0 0

Although this looks confusing, co_backperf.pl will parse it. If you want to get a better “user” view of this data, you can add the “-U” flag to the command (but don’t change this in the script). If the output of this command contains lines with “IMAGE”, a split occurs on whitespace and several fields are noted. In particular, we grab these:

$backupid  = $fields[5];
$seconds   = $fields[14];
$client    = $fields[1];
$schedtype = $fields[11];
$kb        = $fields[18];
$num_files = $fields[19];
$kbpersec  = ($kb/$seconds);
$policy    = $fields[6];
$backdate  = $fields[13];
$backserver= $host;
$backdate  = &humantime($backdate);

After noting the important fields, co_backperf.pl will insert an entry into the database for this backup id (field 5 is the backup id). Running the script a second time will produce several failed inserts. This is because entries that are in the database will still appear in the catalog for some time until they expire, but the failed insert will prevent duplicate entries. This is done by a primary key set on the backupid field. Thus, even if the NetBackup catalog expires an entry, its historical performance data is still kept in the MySQL database. Additionally, one could write a co_driveindex.pl that would run bperror commands to collect the drive on which a specific backup id occurred. This is handy for checking the balancing of your drive usage for multiple tape drive configurations. (The drive_index field is unused in this article.)

Attention!!!

Note that I call the primitive rsh in my script, but you can use ssh or any other method. If you run the script locally on the backup server, you could remove the call altogether. One significant improvement would be to switch the command calls to use Net::SSH::Perl or a similar module.

Next, I will cover the representation of this data, which makes for good management meetings. mkimage_backperf.pl (Listing 3) creates an image using GD::Graph and a few other CPAN modules. In the listing, you’ll notice $database, $hostname, $user, and $password must modified as with co_backperf.pl. Then, we issue the following query:

SELECT backdate, client, kbpersec, policy,
 (to_days(now()) - to_days(backdate)) as age
 FROM nbperf WHERE
 (to_days(now()) - to_days(backdate)) < 2
 $policy_hunt
 ORDER BY policy DESC

Here we query the backup date, the client, the kilobytes per second, the policy that was used, and the "age", which is calculated with some MySQL functions. This produces a list of rows for backups that occurred within the last 24 hours. The order in which they will appear in the chart is by policy name, descending. Each row is used to build the X and Y axis of the chart, using the GD::Graph to produce the result (see Figure 1). That's it! From this point, you could modify mkimage_backperf.pl to have a different y_max_value if your drives are slower or faster than my drives. The resulting image will be written to an img/ directory, so you need to create that directory before running the script (see Figure 1). Note that, if you don't want all this fancy charting, you could simply run this on the backup server:

/opt/openv/netbackup/bin/admincmd/bperror -all -hoursago 24  \
  |grep "wrote backup" |  \
awk {'print $1 " " $15 " " $20'}  | sed 's/,//g'

This command would give you the backup jobs and speeds from the last 24 hours. You could then store or chart these results as you like. You could also use Excel or another charting program once you have the data in the database.

Assumptions

You'll need to install the Perl modules GD::Graph and DBI/DBD::mysql and have a MySQL installation available for use. Also, note that the kilobytes per second that is calculated does not take into account whether a tape needs to be switched. This field is calculated as kilobytes backed up / seconds. You could run bperror commands and change your collector to use the actual tape speeds, but in my case a library switches tapes rather rapidly and the value is close enough to get a good picture. Besides, if a backup were slow because a drive wasn't free, or the tape wasn't found for a while, I'd want to know.

Alternatives

Veritas has a product called NetBackup Advanced Reporter, which provides reporting capabilities for backup data. I did not review this portion of NetBackup since I didn't know it existed until after I began creating my own scripts. I think the product requires a separate license, and users looking for a more supportable solution might check into this product. The co_backperf.pl would break if future versions of NetBackup changed the format of the output, but then so would a lot of other systems administration scripts that have been written around NetBackup over the years.

Conclusion

Although Veritas NetBackup provides various command-line utilities for acquiring backup statistics, there is almost always the need to tie this data into another database or use open source tools to do something with the data. MySQL acts as a good starting point for storing the data because of its simplicity and availability. Perl makes sense for parsing the output of the various NetBackup commands, and GD::Graph works well to make simple graphs. The most difficult part of this setup is getting GD::Graph to install on a Solaris system but the module is used widely enough that you can find help to work through the quirks.

The Author

Jerry Uanino holds a B.S. in Computer Information Systems from Marist College and has worked for IBM, US Internetworking, Enterasys Networks, and most recently Factset Research Systems. He can be reached at: juanino@yahoo.com.

=========================================

nbperf.sql

--
-- Table structure for table 'backperf'
--

CREATE TABLE backperf (
  backupid varchar(50) NOT NULL default '',
  seconds int(11) NOT NULL default '0',
  client varchar(30) NOT NULL default '',
  schedtype varchar(30) NOT NULL default '',
  kbytes bigint(20) NOT NULL default '0',
  numfiles bigint(20) NOT NULL default '0',
  kbpersec int(11) NOT NULL default '0',
  policy varchar(40) NOT NULL default '',
  backdate datetime NOT NULL default '0000-00-00 00:00:00',
  backserver varchar(40) default NULL,
  drive_index int(11) default NULL,
  PRIMARY KEY  (backupid)
) TYPE=MyISAM;

Listing 2 co_backperf.pl

#!/usr/bin/perl -w

# purpose: collect bpimage output
# and store in db

use DBI;
use Data::Dumper;
use strict;

my $success = 0;
my $failure = 0;
my @failures;

# Connect to DB
my $database = 'nbperf';
my $hostname = 'databaseserver';
my $dsn = "DBI:mysql:database=$database;host=$hostname";

my $user = 'database_user';
my $password = 'password';
my $dbh = DBI->connect($dsn, $user, $password);

# list the systems to collect data on
my @systems = ('server1','server2');

foreach my $system (@systems) {
  my $host=$system;
  print "\n--> Trying $host \n";
  my $local_user="root";
  my $remote_user="root";
  # get all image data since 1980 aka everything
  my $cmd=" /opt/openv/netbackup/bin/admincmd/bpimagelist   -d 01/01/1980";

  my $query = qq{

  };

  my @lines;
  if (@lines = `rsh $host $cmd`) {
     $success++;
     print "Parsing, $host, stage 1:  ";
     print "Got " . scalar(@lines) . " lines \n";
     foreach my $line (@lines) {
         my ($backupid, $seconds, $client, $schedtype, $kb, $num_files);
         my ($kbpersec, $policy, $backdate, $backserver);
         if ($line =~ /IMAGE/) {
            my @fields;
            (@fields)  = split(/\s+/,$line);
            $backupid  = $fields[5];
            $seconds   = $fields[14];
            $client    = $fields[1];
            $schedtype = $fields[11];
            $kb        = $fields[18];
            $num_files = $fields[19];
            $kbpersec  = ($kb/$seconds);
            $policy    = $fields[6];
            $backdate  = $fields[13];
            $backserver= $host;
            $backdate  = &humantime($backdate);

            print "Updating database... \n";
            $query = qq{
                   INSERT INTO backperf
                     (backupid, seconds, client, schedtype, kbytes,
                      numfiles, kbpersec, policy, backdate, backserver)
                                      values
                     ('$backupid', '$seconds', '$client', '$schedtype',
                      '$kb', '$num_files', $kbpersec, '$policy',
                      '$backdate', '$backserver')
            };
            print $query;
            my $sth = $dbh->prepare($query);
            $sth->execute;
         } # end up line record for each image
     }

  } else {
     $failure++;
     push(@failures, $host);
  }

}

print "Total success: $success \n";
print "Total fails: $failure \n";
($failure>0) && print "Failed systems: @failures \n";

#################
sub humantime {
 # convert a unix time to human readable time
 my $time_req = shift;
 my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = \
     localtime($time_req);
 $year = 1900+$year;
 $mon++;
 my $humantime =  "$year-$mon-$mday $hour:$min:$sec \n";
 return $humantime;
}

Listing 3 mkimage_backperf.pl

#!/usr/bin/perl -w

# purpose: make graphs for backperf table

use DBI;
use Data::Dumper;
use GD::Graph::bars;
use CGI qw/:standard *table/;
use strict;

my $policy_hunt = $ARGV[0];
if ($policy_hunt) {
 $policy_hunt = qq{
                      AND policy = '$policy_hunt'
                  };
}

my $success = 0;
my $failure = 0;
my @failures;

# Connect to DB
my $database = 'nbperf';
my $hostname = 'databaseserver';
my $dsn = "DBI:mysql:database=$database;host=$hostname";

my $user = 'database_user';
my $password = 'password';
my $dbh = DBI->connect($dsn, $user, $password);

my @y;
my @x;
# get data from the last 1 days
my $timespan = " 24 hours ";
my $query = qq {
                     SELECT backdate, client, kbpersec, policy,
                      (to_days(now()) - to_days(backdate)) as age
                      FROM backperf WHERE
                      (to_days(now()) - to_days(backdate)) < 2
                      $policy_hunt
                      ORDER BY policy DESC
                   };
my $sth = $dbh->prepare($query);
$sth->execute;
my $rows = $sth->rows;
unless ($rows > 0) {
   $query = qq{
                     SELECT backdate, client, kbpersec, policy,
                      (to_days(now()) - to_days(backdate)) as age
                      FROM backperf WHERE
                      (to_days(now()) - to_days(backdate)) < 4
                      $policy_hunt
                      ORDER BY policy DESC
   };
   $sth = $dbh->prepare($query);
   $sth->execute;
   $timespan = " 3 days";
}
while (my $hash_ref = $sth->fetchrow_hashref) {
  my $backdate = $hash_ref->{backdate};
  my $client = $hash_ref->{client};
  my $kbpersec = $hash_ref->{kbpersec};
  my $policy = $hash_ref->{policy};
  push(@x, "$client - $policy");
  push(@y, $kbpersec);
}

# make and save graph
my $graph = GD::Graph::bars->new(800,400);
my $x_ref = \@x;
my $y_ref = \@y;
my @data = (
   $x_ref, $y_ref
   );

$graph->set(
      x_label           => 'host - policy',
      y_label           => 'kb/sec',
      x_labels_vertical => 1,
      title             => "Backup Speeds in Last $timespan",
      y_max_value       => 25000,
      y_tick_number     => 100,
      y_label_skip      => 30
);

my $format = $graph->export_format;
open(IMG,">img/backperf.png");
binmode IMG;
my $pic = $graph->plot(\@data);
print IMG $pic->png;

Master server 上包含以下 NetBackup catalog 文件：

• install_path\NetBackup\db 【 ntbbkp03 上大概 13.4 G 】
• install_path\NetBackup\var 【 ntbbkp03 上大概 23.6 K 】
• install_path\Volmgr\database 【 ntbbkp03 上大概 664 K 】

Media server 上包含以下 NetBackup catalog 文件：

• install_path\netbackup\db\media
• install_path\netbackup\var
• install_path\volmgr\database

我查看了一下，
install_path\NetBackup\db 这个目录下的 images 文件夹最大，包含了所有 clent 的 image 信息。

可以利用 bprecover 命令恢复 Catalog 信息（恢复 Catalog 前需要重新安装 NetBackup 软件。）

如何确定 Catalog 的最新备份呢？

Little Hint

Before you can recover the NetBackup catalogs, you must know which media ID has their latest backups. Without this media ID, you cannot accurately recover your catalogs and your only option is to use the NetBackup import feature to import all lost backup records into your NetBackup catalog (see the NetBackup System Administrator’s Guide, Volume I).

时刻得知最新的 catalog 备份的介质信息的方法是配置 email 通知，这样在每次 catalog 备份后会发一封邮件通知管理员。

如果事先知道具体是哪些 media 备份了 Catalog，但是不能确认到底哪个 media 上的是最新的，可以使用 bprecover -l 命令，列出具体某一个 media 上的备份信息包括具体的时间和数据信息。

示例 1：使用原始设备列出

假定将目录备份到磁带，但是目录的“介质管理器部分”丢失，因此介质管理器无法控制驱动器。

在这种情况下，请将介质插入到适当的驱动器中（假定原始设备路径为 \\.\Tape1）。然后，在拥有该驱动器的 NetBackup 服务器上执行以下 bprecover 命令。

bprecover -l -tpath \\.\Tape1
Database Backup Information from \\.\Tape1

Created: 03/31/97 11:31:34
Server: bphost
Block Size: 32768

	Path
	----
IMAGE1 D:\apps\VERITAS\NetBackup\db
IMAGE2 D:\apps\VERITAS\Volmgr\database

示例 2：使用介质管理器控制的驱动器列出

假定目录的“介质管理器部分”完好无损，且备份将存储到介质 ID 为 000001 的 dlt 磁带上。将该磁带插入到相应的驱动器中。然后，在拥有该驱动器的 NetBackup 服务器上执行以下bprecover 命令（NetBackup 设备管理器服务必须是活动的）。

bprecover -l -ev 000001 -d dlt
Database Backup Information from 000001

Created: 03/31/97 05:50:51
Server: bphost
Block size: 32768

	Path
	----
IMAGE1 D:\apps\VERITAS\Netbackup\db
IMAGE2 D:\apps\VERITAS\Volmgr\database
IMAGE3 D:\apps\VERITAS\NetBackup\var

示例 3：列出磁盘路径

假定目录备份的目标磁盘路径为 D:\apps\dbbackup，而此磁盘没有故障。执行以下 bprecover 命令列出备份信息。

bprecover -l -dpath D:\apps\dbbackup
Database Backup Information from D:\apps\dbbackup

Created: 03/31/97 11:31:34
Server: bphost
Block size: 32768

	Path
	----
IMAGE1 D:\apps\VERITAS\NetBackup\db
IMAGE2 D:\apps\VERITAS\NetBackup\var
IMAGE3 D:\apps\VERITAS\Volmgr\database

bprecover 命令语法

bprecover: -l -m media_ID -d density [-v]
           -l -tpath  [-v]
           -l -dpath  [-v]
           -l -opath  [-v]
           -r [ALL|image_number] -m media_ID -d density [-stdout] [-dhost ] -v]
           -r [ALL|image_number] -tpath  [-stdout] [-dhost ] [-v]
           -r [ALL|image_number] -dpath  [-stdout] [-dhost ] [-v]
           -r [ALL|image_number] -opath  [-stdout] [-dhost ] [-v]

恢复 NetBackup catalog 的过程

恢复目录所需的方法取决于：

• 包含 NetBackup 目录备份的介质类型（磁带或磁盘）。
• 那些目录的介质管理器部分是否仍完好无损。介质管理器目录文件通常在install_path\Volmgr\database 目录中。

Attention!!

介质管理器设备目录是二进制文件，您无法将它们恢复到不同类型的平台上。

恢复 Catalog 的准备工作

• 重新安装 NetBackup 软件（如果需要）。
• 查找包含最新目录备份的磁带。
• 确保您要恢复目录的磁盘包含 catalog 所在的目录。
  这是必需的，因为 bprecover 命令总是将 NetBackup 目录恢复到从中备份这些目录的路径（不允许使用其他路径进行恢复）。

在介质管理器目录丢失的情况下从磁带恢复目录

如果 NetBackup 目录备份在磁带上，且介质管理器目录已丢失，请在 bprecover 命令中指定原始设备路径。此方法涉及在驱动器中安装备份磁带和使用 -tpath 参数。

Attention!!!

如果您计划用于恢复的设备的配置丢失，请按照 Windows 系统文档中的说明重新安装该设备。

使用 robtest 得到的 ntbbkp03 上的 tape 信息

E:\Program Files\VERITAS\Volmgr\bin\tldtest.exe -r \\.\Changer0 -d1 Tape0 -d2 Tape1 -d3 Tape2 -d4 Tape3

E:\Program Files\VERITAS\Volmgr\bin\tldtest.exe -r \\.\Changer1 -d1 Tape4 -d2 Tape5 -d3 Tape6 -d4 Tape7

使用 robtest 得到的 ntbbkp04 上的 tape 信息

C:\Program Files\VERITAS\Volmgr\bin\tldtest.exe -r \\.\Changer0 -d1 Tape0 -d2 Tape1 -d3 Tape2 -d4 Tape3